Observational Teamwork Assessment for Surgery (OTAS)

Submitted by National Center... on Oct 7, 2016 - 1:48pm CDT

Instrument

Authors:

Hull, L.

Arora, S.

Kassab, E.

Kneebone, R.

Sevdalis, N.

Overview:

This observational tool was designed to assess the non-technical skills of surgical teams in real time in the operating theatre. It was based on the patient safety literature which found that lack of interpersonal skills (e.g., communication, teamwork, leadership) and cognitive skills (e.g., decision making, situational awareness, mental readiness) contributed to adverse patient complications and errors. Trained raters use the one-page form to observe and score three subteams separately (surgeons, anesthesiologists, and nurses) at three different time points in the procedure: pre-op, intra-op, and post-op. Subteam scores are aggregated for a total team score. Results are used to support team training. The OTAS has been used extensively with a range of general and specialty surgical procedures. Previous research reported strong inter-rater reliabilities and construct validity estimates. The study described here evaluates the content validity of the behavioral exemplars used to guide ratings, and presents the current instrument.

Link to Resources

Resource links:

Hull, L., Arora, S., Kassab, E., Kneebone, R., & Sevdalis, N. (2011). Observational teamwork assessment for surgery: Content validation and tool refinement. J Am Coll Surg, 212:234-243.

Sevdalis, N., Lyons, M., Healy, A.N., Undre, S., Darzi, A., & Vincent, C.A. (2009). Observational teamwork assessment for surgery: Construct validation with expert vs. novice raters. Ann Surg, 249(6): 1047-1051.

Undre, S., Sevdalis, N., Healy, A.N., et al. (2007). Observational teamwork assessment for surgery (OTAS): Refinement and application in urological surgery. World J Surg, 31: 1373-1381.

Undre, S., Healy, A.N., Darzi, A., & Vincent, C.A. (2006). Observational assessment of surgical teamwork: A feasibility study. World J Surg, 30: 1774-1783.

Descriptive Elements

Who is Being Assessed or Evaluated?:

Teams

Instrument Type:

Observer-based (e.g., rubric, rating tool, 360 degree feedback)

Source of Data:

Health care providers, staff

Other

Notes for Data Sources:

Two psychologists trained in and experienced with the OTAS observed teams and rated their performance. 15 clinical content experts from surgery, anesthesiology, and nursing performed qualitative reviews and contributed to refinement of the behavioral exemplars.

Instrument Content:

Behaviors / skills

Notes for Content:

The OTAS is based on a framework of five constructs:

Communication: quality and quantity of information exchanged among team members
Coordination: management and timing of activities and tasks
Cooperation / Back Up Behavior: assistance provided among members of the team, supporting others, and correcting errors
Leadership: management and timing of activities and tasks
Monitoring / Situational Awareness: team observation and awareness of ongoing processes

Some 114 behavioral exemplars are provided to guide raters in their scoring. Example from the Intra-Op, Surgeon form: "Asks team if all prepared to begin the operation."

Instrument Length:

The one-page tool contains the five constructs, a subset of behavioral exemplars relative to the stage of operation and the subteam, and a rating scale with descriptive anchors. The time required to complete the form is minimal, but it is based on full observation of the procedure. In this study, data were collected from 30 procedures which lasted between 30 and 240 minutes.

Item Format:

Check boxes for whether a particular exemplar was appropriate for the case, and whether or not it had been observed; and a 7-point rating scale (0 - 6) for assessing sub-team performance on the five constructs. A "6" indicated "exemplary behavior: very highly effective in enhancing team function." A "0" indicated "problematic behavior; team function severely hindered."

Administration:

Two trained raters observed 30 general surgery procedures and independently completed the instrument. Raters were present during the entire operation and had full visual access to the surgical field and team members. A total of nine separate forms are required: three (one per sub-team) at each of three phases of the operation.

Scoring:

Sub-team scores can aggregated for a total team score. This could be done at the operative phase level, or for the total procedure.

Language:

English

Norms:

None

Access:

Subscription (can be viewed in journal article)

Notes on Access:

Contact the author to obtain all nine forms.

Psychometric Elements: Evidence of Validity

Content:

The 130 exemplars developed for the original OTAS were based on ethnographic field notes, existing literature, best practice guidelines, OR documentation, and expert review. Exemplars were revised for a study of urologic surgery procedures. In the present study, the exemplars were tested in a 2-phase design. Data from 30 observations identified a subset of exemplars that were deemed appropriate in less than half the cases, and/or not observed in less than half the cases. 15 clinical experts reviewed these problematic exemplars and scored them for their relevance to teamwork and patient safety. Their scores created a rank-list. Inter-rater reliability and agreement estimates (see below) also identified some exemplars that were difficult to score. Using consensus methods among authors and clinical experts, a total of 21 behavioral exemplars were removed from the OTAS and and additional 23 were modified.

Response Process:

Observer agreement was high (Cohen's K > 0.41) for 84% of the exemplars; 61% were observed frequently with high agreement. Inter-rater reliability estimates were significant (p <0.001) for all five constructs: communication = 0.77, leadership = 0.73; cooperation = 0.71; coordination = 0.67; and team monitoring = 0.64.

Internal Structure:

See study by Undre, et al. (2007)

Relation to Other Variables:

See study by Sevdalis, et al. (2009)

Consequential:

None.

Assessment & Evaluation

Observational Teamwork Assessment for Surgery (OTAS)

Related Resources

User Collections

Start the Conversation