To book on one of the PAR Courses please visit the courses page here

 

Objective surgical skill assessment: The diagonal operating matrix.

AUTHORS Sharif Al-Ruzzeh PhD, FRCS
Shishir Karthik FRCS
David O’Regan MBA, MD, FRCS C-Th

INSTITUTION Yorkshire Heart Centre,
Leeds General Infirmary,
Great George Street,
Leeds LS1 3EX.

CORRESPONDING Mr David J O’Regan MBA, MD, FRCS C-Th

AUTHOR The Yorkshire Heart Centre
Leeds General Infirmary
Great George Street,
Leeds LS1 3EX,
United Kingdom

E-mail: David.O'Regan@leedsth.nhs.uk
Telephone: 44-113-3925790
Facsimile: 44-113-3928092

KEY WORDS surgical skills, training, assessment

 

ABSTRACT
There is an urgent need for structured surgical training and assessment due to the
reduction in the training duration with the European Working Time Directive
(EWTD). We propose a model for objective skill assessment, the PAR- Diagonal
Operating Matrix (PAR-DOM) which breaks down the task of vascular anastomosis
into clearly defined skills.
The PAR-DOM is made up of a 3 x 5 table and progress is made along vectors
defined on the x-axis as PAR and on the y-axis as four levels. PAR defines three
skills at each level. Each skill is graded from 1 – 3 (This may be taken as below
average, average, above average). The skills at various levels are: Level 0 – Posture,
Address, Relaxation; Level 1 – Pick-up, Airtime, Rotation; Level 2 – Placing, Angles,
Rhythm; Level 3 – Precision, Adaptability, Reproducibility; Level 4 – Pace,
Awareness, Relations.
The PAR-DOM matrix provides a graphic representation of the progress of trainees
over their training period assigned for them to stay with the trainer and also help
identify individual strengths and weaknesses.

INTRODUCTION
Surgical training, in general, and cardiac surgical training, in particular, is passing
through a critical or “turbulent” time [1]. In united Kingdom, the training bodies have
become concerned about surgical training due to reduced number of years of training
with the adoption of the Calman system and reduced working hours with
implementation of European Working Time Directive (EWTD) [2]. There is an urgent
need to develop means of assessment in order to ensure the provision of an adequate
and satisfactory training.

We propose the PAR - Diagonal Operating Matrix (PAR-DOM) as an objective
assessment tool that breaks down the task of vascular anastomosis into clearly defined
skills.

MATERIALS AND METHODS
THE ASSESSMENT MODEL
The PAR-DOM model is made up of a 3 x 5 table defining 15 basic skills to be
assessed. Progress is made along vectors defined along the x-axis as PAR and along
the y-axis as four levels, starting from level 0 to level 4. The three letters P, A and R
represent the first letters of the words that define the threes skills in each level. Each
acronym is used to help memorise and communicate the ability to perform a clearly
defined skill which is scored between 1 and 3 (FIGURE 1). This may be taken as
below average, average, above average; poor, good, excellent; low, medium, high; or
any other classification that may be ascribed to the skill. Absence of a particular skill
is scored as 0, which simply implies that the trainer feels the trainee totally lacks the
defined skill. The model is assessed on cadaveric porcine tissue in a wet lab.

Level 0 defines the ability to control posture, dynamics of body and state of mind.

1. Posture: Assesses the ability of the trainee to attain the optimal posture and
height of the table, which will define the anatomical working angles of the
shoulders, the elbows and hands. The shoulders should be relaxed with straight
back, the elbows extended slightly beyond the 90° angle and the wrists relaxed in
slight palmar flexion (VIDEO 1).

2. Address: Assesses the ability to shift from right leg to left leg in order to alter the rotation of the hips, so the shoulders enable a forehand and backhand stitch
respectively with the ability to move around the operating table (VIDEO 2).

3. Relaxation: Assesses the ability to relax the body and move the arms, wrists and fingers independently as required as discrete separate movements. In other words, gross forearm or wrist movements do not substitute fine finger movements.

Level 1 defines the ability to handle the needle with an instrument in refined
successive movements.

1. Pick-up: Assesses the ability to pick up the needle accurately within the jaws of a
needle holder and at the correct angle relative to the tissue to be stitched.

2. Airtime: Assesses the amount of time the trainee spends to adjust the angle of the needle on the needle holder to get ready for the next stitch. The trainee is advised not to use fingers to adjust the needle in position but rather use an instrument or surface as previously suggested [3].

3. Rotation: Assesses the ability to rotate the needle round a central point, such that the needle circumscribes a perfect circle throughout its path. This requires
introduction of the needle at an angle of 90° on entering the tissue and on exiting
from it. This aims to avoid dragging the heel of the needle through the tissue that
can predispose to needle holes (VIDEO 3).

Level 2 defines the ability to use the needle properly.
1. Placing: Assesses the ability to place the needle in a correct position in relation to the tissue, as well as placing the suture accurately on the follow through
(VIDEO 4).

2. Angles: Assesses the ability to attain the suitable angle in relation to the needle
holder so that the needle can be accurately rotated through the tissues (VIDEO 4).

3. Rhythm: Assesses the ability to repeat the previous processes rhythmically. This
becomes dependent on the smooth rotation of the needle on entry into the tissue
and on exit to facilitate ease of pick up. The emphasis is on economy of time and
movement.

Level 3 defines a higher level of needle skills
1. Precision: Assesses the ability to do the previous needle exercises of introducing
the needle through the tissue and placing the suture accurately whilst using the
working angles precisely.

2. Adaptability: Assesses the ability to actually stitch in difficult positions with
variable anatomy and tissue resistance.

3. Reproducibility: Assesses the ability to stitch reliably and precisely all the time
and every time. For rating this skill, the three levels are defined as 30%-60%,
60%-90% and more than 90% of the time.

Level 4 defines the ability to perform the surgical tasks with awareness of the
surrounding environment including time and relations.

1. Pace: Assesses the ability to perform the task without hesitation, interruption,
deviation or repetition. This emphasises economy of time and movement, so the
operators at this level becomes attuned to their own natural rhythm. It is very
important to stress that pace does not equate to speed.

2. Awareness: Assesses the ability to stay aware of time, surroundings and
environment while performing the task. This aims to assess the ability of the
trainee keep a breast of the whole situation including real time monitoring, while
totally engaged in task.

3. Relations: Assesses the ability to communicate with the personnel around, which
would represent the theatre team in real life, while performing the task.
The PAR-DOM gives a graphic representation of the surgical ability of the trainee.
It highlights the deficiencies in needle skills and enables measurement of progression over time. Furthermore, the skills are distributed in the matrix so that progression can logically move diagonally from left upper corner down to the right lower corner of the model and does not move along either the x or y axis alone. Therefore, with experience, we further defined the level expected from each of the trainee in each of 6 training years of any cardiothoracic programme along this diagonal progression (FIGURE 2). This diagonal progression is deemed to be useful in assessing and planning further training, however, the skills are assessed concurrently and independently at the time of assessment.

DISCUSSION
This study proposes the PAR-DOM as a model of objective assessment of surgical
skills that can help plan training according to individual trainees’ requirements.

The conventional apprenticeship model in surgical education does not satisfy the
requirements of training at present nor in the future. It relies mainly on passive
unreliable observation of live operating in theatre as a mode of both transferring
knowledge by the trainer and its acquisition by the trainee [4]. Subjective assessment
of the trainer in this situation is global and not based on previously set or studied
criteria. However, this was still useful to give a limited assessment of the trainee’s
technical ability and progress and thus provide some feedback to trainees, but sadly
far from being structured or sufficient to formulate a staged training plan according to the trainee’s needs [5]. In fact, the subjective global rating scales by experts were still more reliable and valid than checklists alone in assessing the trainees’ performance of procedures, and the addition of the checklists to these scales did not improve either validity or reliability [6]. Nevertheless, with this type of subjective assessment some trainees were progressing to become independent operators without demonstrating enough objective evidence of technical proficiency even at basic surgical skills level  [7]. All this, in addition to the difficulties that surgical training has been facing lately [1, 2], has highlighted the urgent need for a structures objective assessment tool [5]. Patient safety during a surgical procedure remains a priority and a responsibility of the trainer to the extent it could create a conflict between the ethical duty of the trainer towards patients and his or her societal commitment to train new surgeons [8].

Furthermore, training or assessment under the stress of the operating room
environment is difficult and not an appealing experience for either the trainer or the
trainee. Therefore, the operating room is not a place of training or assessment but a
place of consolidation of an already gained and assessed training experience under
very controlled and strict conditions of safety and clinical governance [9].

The duty of the trainer or the training body starts with the process of selection of the suitable trainable trainees in the first place, which does not seem to follow clear criteria at present. One realises that this selection is a very complicated and difficult process.

However, it has been shown that the ability to be trained in surgical
specialties dose not rely solely on the academic knowledge but involves other noncognitive, demographic factors and personality characteristics [10]. The creation of objective assessment tools may also help in this difficult selection process at an earlier stage.

Several assessment tools have been developed over the past two decades in order to help assess surgical ability and progress of the surgical trainees. The Objective Structured Assessment of Technical Skills (OSATS) has been widely used and validated. It consists of six bench-top stations, where the trainee’s performance of a
procedure or task is assessed by a checklist with a global rating scale [11]. These bench-top stations are usually set up tasks on cadaveric porcine tissue that differ according to the speciality of the trainee. The bench-top models have been dealing with full tasks, which can be modified according to the surgical speciality of the
trainee. It could be an aortic cannulation for a cardiac trainee, but a ureteric
anastomosis for a urology trainee. The emphasis is on the performance of the task as a whole but not on the fine needle skills that are required to perform this task 6].
Objective assessment of skills and consequently training should pay attention to the
basic needle skills as they are the basic units that form the full procedure at the end.
This attention should not only be aimed for as a fine tuning exercise in order to
“polish the surgical technique of an already trained surgeon“, but as an objective tool with which the technical ability of the surgeon is assessed and may be improved. It has been repeatedly stated that a “skilfully performed operation is 75% decision making and 25% dexterity”.
Dexterity analysis systems are able to assess this surgical handicraft up to a certain
limit, however, they are still in the process of development and validation. When they are married with virtual reality they are able to assess and feedback about skill based errors [5]. Experience showed that assessment should marry training in order to get the benefit from both. Otherwise, it would be a useless exercise to diagnose trainees’ underperformances and not plan successful remedies for them. This highlights the ability to give and receive feedback in order to create progression cycle to move the trainee from a certain level to a higher one.
The recent evidence suggests that there should be a differentiation between the
performance profiles of a task and a skill in assessing surgical training. Future training strategies should aim to teach clearly defined surgical skills so the brain can build up the task from those skills every time it is required. It has been recently shown that surgeons were three times faster than non-trained medical students in performing a suturing task but not necessarily better [12].
The trainer (DOR) in this study, when diagnosed a technical deficiency, sent the
trainee away with a handful of simple exercises to practise at home out of hours
regularly. These exercises included performing needle skills on poached eggs, boiled
potatoes and bananas and practising fine quick movements as rice picking. Others
have thought and reported similar means of home based “out of hours” continuous
training on variable materials. Neurosurgeons could practise micro vascular techniques on a chicken wing artery [13], anaesthetists could practise on lemons and oranges to train on tissue resistance feedback for epidural insertion [14] and laparoscopic trainees could set an inexpensive “kitchen table” trainer or simulator at home using a lab-top, a web-cam and a box [15]. We aimed to propose and describe the model in this publication, however, the next step would be to assess its validity, reliability and reproducibility. This model also provides a framework by which the trainees can assess their own ability and hence understand the specific skill that is limiting their progress along the diagonal progression.

VIDEO LEGENDS

• VIDEO 1: Shows how the trainee can attain the optimal posture and height of the
table, which will define the anatomical working angles of the shoulders, the
elbows and hands. The shoulders should be relaxed with straight back, the elbows
extended slightly beyond the 90° angle and the wrists relaxed in slight palmar
flexion.

• VIDEO 2: Shows the ability to shift from right leg to left leg in order to alter the
rotation of the hips, so the shoulders enable a forehand and backhand stitch
respectively with the ability to move around the operating table.

• VIDEO 3: Shows the ability to rotate the needle round a central point, such that
the needle circumscribes a perfect circle throughout its path.

• VIDEO 4: Shows the ability to place the needle in a correct position in relation to
the tissue and the ability to attain the suitable angle in relation to the needle holder
so that the needle can be accurately rotated through the tissues.

REFERENCES

1. Jackson B. Turbulent times- the experiences of a college president. Am J Surg
2003;185: 22-25.
2. Chikwe J, de Souza A, Pepper J. No time to train the surgeons. BMJ 2004; 328:
418-9.
3. Hosseinpour A, Hilton G, Nashef S. Techniques of needle stance exchange in situ.
Interact CardioVasc Thorac Surg 2005; 4: 289-291.
4. Naftalia. Unfinished business and modernizing medical careers. BMJ 2003, 326:
S89-90.
5. Moorthy K, Munz Y, Sarker S, Darzi A. Objective assessment of technical skills
in surgery. BMJ 2003; 327: 1032-1037.
6. Regehr, G; MacRae, H; Reznick, R K; Szalay, D Comparing the psychometric
properties of checklists and global rating scales for assessing performance on an
OSCE-format examination. Acad Med 1998; 73:993-7.
7. Hance J, Aggarwal R, Stanbridge R, Blauth C, Munz Y, Darzi A, Pepper J.
Objective assessment of technical skills in cardiac surgery. Eur J Cardiothorac
Surg 2003; 28: 157-62
8. Bernstein M. Surgical teaching: how should neurosurgeons handle the conflict of
duty to today’s patients with the duty to tomorrow’s? Br J Neurosurg
2003;17:121-3.
9. Cox S, Swanson M. Identification of teaching excellence in operating room and
clinic settings. Am J Surg 2002; 183: 251-5.
10. Tett R, Jackson D, Rothstien M. Personality measures as predictors of job
performance: a meta-analytic review. Personnel Psychol 1991; 44: 703-42.
11. Martin J, Regher G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M.
Objective structured assessment of technical skill (OSATS) for surgical residents.
Br J Surg 1997; 84:273-8.
12. Pilitsis J, Redding G. The use of performance profiles in the design of a
systematic medical suturing training programme. Appl Ergon 1977; 8: 87-92.
13. Hino A. Teaching in microvascular surgery using a chicken wing artery.
Neurosurgery 2003; 52: 1495-8.
14. Brown T. An orange or lemon as a teaching aid. Anaesth Intensive Care. 2003;
31: 334-5.
15. Harris M, Hindmarsh J. Skills training and the role of skills centres. BJU Int 2003;
92: 332.

 

 

 

Wetlab Around the Globe

Click on the map to explore where we are based around the globe

Map

 

 

 

 

     

 

 

 

 

 

 

        

        

        

        

         

Latest News

The Par Aortic Course 2017

  After the success of last year's inaugural PAR Aortic course we are proud to announce the dates for The Par Aortic Course 2017. A fantastic aortic valve course... Read full article...

View all articles