Multidisciplinary Team - Observational Assessment Rating Scale (MDT-OARS)

National Center for Interprofessional Practice and Education's picture
Submitted by National Center... on Sep 6, 2016 - 11:12am CDT

Taylor, C.
Atkins, L.
Richardson, A.
Tarrant, R.
Ramirez, A.

This observational assessment tool was designed for cancer multidisciplinary team (MDT) meetings. Specifically, it contains items related to 15 criteria associated with the team, infrastructure for meetings, meeting organization and logistics, and clinical decision making. The content is based on recommendations from a large-scale survey about effectiveness in MDT in the UK.  A pilot validation study conducted on 10 bowel cancer teams reported good inter-rater reliability (0.68 and above) for about half of the 13 quantitative criteria. The results are meant to assess MDTs alongside peer review and patient experience surveys.

Link to Resources
Descriptive Elements
Who is Being Assessed or Evaluated?: 
Instrument Type: 
Observer-based (e.g., rubric, rating tool, 360 degree feedback)
Notes for Type: 

This is an (in situ) observational tool. The authors suggest recording MDT meetings for observation to allow for multiple observers and replay of events.  Some of the criteria require integrated judgments based on materials (such as agendas) and objective aspects related to the organizational environment, as well as team behaviors exhibited during the MDT meetings.  

Source of Data: 
Health care providers, staff
Notes for Data Sources: 

The research team served as observers for the validation study. One oncology specialist registrar and one post-doctoral research psychologist were also employed as independent observers.

Instrument Content: 
Behaviors / skills
Notes for Content: 

15 criteria were developed to assess MDT meetings:

  1. The Team
  • Attendence
  • Leadership: chairing of meeting
  • Teamworking & Culture
    • Inclusion of relevant team members
    • Team sociability
    • Mutual respect
    • Tension and conflict
  • Personal development & training
  1. Infrastructure for meetings
  • Meeting venue
  • Technology & Equipment
  1. Meeting organization and logistics
  • Preparation prior to meetings
    • Agenda
    • Prioritization of complex cases
  • Organization/admin during meetings
    • Patient notes
    • Case presentation
  1. Clinical decision making
  • Patient centered care
  • Treatment plan

An in-vivo proforma sheet is also provided to collect information regarding the observational context. 

Instrument Length: 

15 ratings; ratings took between 45 and 160 minutes depending on meeting length (average = 93 minutes).

Item Format: 
11 items range from 1 (very poor) to 4 (very good); 3 items range from 1 (very poor/poor) to 3 (very good); and 1 item ranges from -4 (severe and sustained conflict) to 0 (no tension).
An observer and video camera record each cancer MDT meeting. In the validation study, no delays or interruptions occurred and teams reported negligible impact on team member behavior. The observer records meeting context on "in-vivo proforma" sheet (see above, attached). The recordings are then played back for observers, and recording may be replayed until a decision on each criteria has been made. Most characteristics are rated on a case-by-case basis, and then aggregated into a team rating via global judgments based on notes from individual cases. Patient centered care, as an exception, was based on the frequency with which patient information was considered. Attendance, meeting agenda, and meeting venue and technology ratings were based on objective information recorded by the observer recording the meetings.
A single score from 11-58 is calculated for the tool by summing quality ratings for each characteristic.
None described.
Subscription (can be viewed in journal article)
Notes on Access: 

The rating tool manual is shown in the BMC Cancer article, but you should contact the author to determine whether they or the journal owns the copyright.  The "in vivo proforma" sheet is attached.

Psychometric Elements: Evidence of Validity
The criteria were informed by the results of a national survey completed by over 2000 cancer multidisciplinary team members (MDTs) on effectiveness in MDTs. These initial criteria were then reviewed by an independent panel, which included a consultant radiologist, oncologist with bowel cancer expertise, nurse with bowel cancer expertise, and a senior NHS cancer manager.
Response Process: 
Team members participating in the study reported the presence of observer and recording equipment as causing no delay or interruption of usual flow of meetings with negligible impact on team member behavior. Teams were most likely to receive 'very good' ratings on administrative preparation, membership/attendance, and tension/conflict. Teams were least likely to receive 'very good' ratings on patient centered care and prioritization of cases on the agenda.
Internal Structure: 
Inter-rater reliability (IRR) was above 0.68 for seven of the 13 quantitative measures. It was between 0.70 -- 0.92 for mutual respect, personal development and training, meeting venue, technology & equipment, case presentation, and patient centered care. The IRR was lower for leadership, team sociability, preparation prior to the meeting, and organization/administration (0.47 -- 0.58) and lowest for treatment plan (0.32) and discerning tension/conflict (0.10).
Relation to Other Variables: 
None described.
None described.