A Validity Study of the Safety Attitudes Survey

Submitted by Connie C Schmitz on Feb 16, 2018 - 4:44pm CST

Instrument

Authors:

Davenport, D.L., Henderson, W.G., Mosca, C.L., Khuri, S.F., Mentzer, R.M. Jr. & Participants in the Working Conditions of Surgery Residents and Quality of Care Study.

Overview:

This large, national study examines the validity of the Safety Attitudes Questionnaire (SAQ) in predicting risk-adjusted morbidity and mortality outcomes for hospital patients on general/vascular services. The SAQ (Sexton JB, et al., 2006) is a well-known, psychometrically sound survey that has been used internationally for many years. It contains 30 items grouped into six scales measuring (1) teamwork climate, (2) safety climate, (3) working conditions, (4) job satisfaction, (5) recognition of stress effects, and (6) perceptions of management. Psychometric analysis of the SAQ demonstrated excellent reliability and factor structure for these scales, but research on their relationship to patient outcomes has been mixed. In this study reported here, Davenport and colleagues tested the SAQ along with a separate scale measuring provider burnout, and a separate set of items assessing the quality of communication and collaboration among different provider groups. This study included over 5,000 providers (surgeons, ICU physicians, nurses, residents, anesthesiologists, and others) at 52 teaching hospitals. It found that the only measure that predicted patient morbidity was the separate (non-SAQ) item measuring the quality of communication and collaboration with attending physicians. None of the SAQ scales, the burnout scale, or the communication / collaboration items correlated with patient deaths. The most recent version of the SAQ (2018) has incorporated additional items regarding the quality of collaboration among key providers and communication breakdowns. This tool could be helpful for measuring intermediary outcomes of IPECP interventions.

Link to Resources

Resource links:

Davenport, D.L., Henderson, W.G., Mosca, C. L., Khuri, S.F., Mentzer, R.M. Jr., & Participants in the Working Conditions of Surgery Residents and Quality of Care Study. (2007). Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration on surgical teams but not with scale measures of teamwork climate, safety climate, or working conditions. American College of Surgeons, 205(6): 778-784.

Sexton, J.B., Helmreich, R.L., Neilands, T.B., Rowan, K., Vella, K., Boyden, J., Roberts, R.R. & Thomas, E.J. (2006). The safety attitudes questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Services Research, 6:44.

Descriptive Elements

Who is Being Assessed or Evaluated?:

Organizations

Instrument Type:

Self-report (e.g., survey, questionnaire, self-rating)

Source of Data:

Health care providers, staff

Organizational records

Notes for Data Sources:

Individuals
- Attending surgeon, physician – 688 (11% of sample)
- Resident surgeon / physician – 1,193 (20%)
- Nurse (staff RN, NP, nurse anesthetists) – 3,432 (56%)
- Other (as determined by chief of surgery) – 770 (13%)

National Surgical Quality Improvement Program (NSQIP) data drawn for over 57,000 patients seen at 52 VA Medical Centers and academic medical centers in the U.S.A. Data reflecting morbidity and mortality outcomes were extracted for all patients enrolled during October 1, 2003 – September 30, 2004.

Instrument Content:

Reported perceptions, experiences of working relationships, teamwork

Organizational environment, culture

Notes for Content:

Perceptions of working relationships and organizational culture

SAQ
- Safety climate – perceptions of a strong and proactive organizational commitment to patient safety (e.g., “I would feel safe being treated here as a patient”)
- Teamwork climate – perceived quality of collaboration between personnel (e.g., “Nurse input is well received”)
- Working conditions – perceived quality of the work environment and logistical support (e.g., “This hospital does a good job of training new personnel”)
- Recognition of stress effects – acknowledgement of how performance is influenced by stressors (e.g., “I am less effective at work when fatigued”)
- Perceptions of management – approval of managerial action (e.g., “Hospital administration supports my daily efforts”
Burnout – 4 items from Maslach’s Burnout Inventory, 1996 (e.g., “I feel fatigued when I get up in the morning and have to face another day on the job”)
Quality of Communication / Collaboration with Colleagues – 4 items assessing the quality of communication / collaboration in working with providers from 4 different clinical roles, i.e.: Attending MDs, Resident MDs, Nurses, and Others

Instrument Length:

38 items total (30 = SAQ, 4 = Burn-out, and 4 = communication and collaboration with colleagues

Item Format:

SAQ and Burnout Scales: 5-point Likert-type scales ranging from strongly disagree (1) to strongly agree (5) Quality of Communication / Collaboration: 5-point qualitative scale ranging from very low (1) to very high (5)

Administration:

Survey: paper or online, administered between July 1, 2003 and September 30, 2004. Data collection was coordinated by an external contractor, stripped of identifying data, and returned to the PI for analysis. NSQIP data: extracted from medical records according NSQIP protocols by trained nurse extractors

Scoring:

To score the instruments, the authors calculated the percent of positive responses (4’s and 5’s) – e.g., slightly agree or strongly agree – for each item and scale. This was done in concordance with the Sexton study, which calculated these percentages for comparative use as benchmarks. For each hospital site, the authors used nationally established guidelines for calculating the risk-adjusted observed vs. expected (O/E) morbidity and mortality rates for all of the patients seen during the study time period. These were aggregated and averaged at the hospital site level. Additional information on scoring the most current version of just the SAQ can be found on the University of Texas / Texas Medical Center’s “Center for Health Care Quality and Safety”: https://med.uth.edu/chqs/surveys/safety-attitudes-and-safety-climate-questionnaire/.

Language:

English

Norms:

Benchmarking data were provided in Sexton at the clinical level for 203 different clinical areas. Descriptive data (means and standard deviations, floor and ceiling information) were also calculated, by country, for the three countries participating in the 2006 study (U.K., N.Z., and U.S.A.).

Access:

Open access (available on this website)

Notes on Access:

All of the items are shown in the Davenport article. Persons interested in the SAQ please note, however, that the most current version (2018) contains 36 items, including three that measure agreement with the statement, “I experience good collaboration with (nurses; staff physicians; pharmacists) in this clinical area;” and a fourth item measuring agreement with the statement, “Communication breakdowns that lead to delays in delivery of care are common.” This instrument can be downloaded from the University of Texas / Texas Medical Center’s Center for Health Care Quality and Safety: https://med.uth.edu/chqs/surveys/safety-attitudes-and-safety-climate-questionnaire/. The Center requests that users complete an information sheet as part of receiving permission to use the instrument.

Psychometric Elements: Evidence of Validity

Content:

The content development of the SAQ was not described in the Davenport article. The Sexton article details the evolution of the (2006) SAQ from a previous ICU Management SAQ—which was, itself, developed from principles of safety in the flight industry and an instrument called the Flight Management Attitudes Questionnaire (FMAQ). The SAQ version described in the 2006 article retained about 25% of the items in the FMAQ. Sexton and colleagues created about 100 new items through discussions with health care providers and subject matter experts. Retained items had some basis in a theoretical model for analyzing risk and safety (A. Vincent, BMJ, 1998), and/or a conceptual model for assessing quality (A. Donabedian, JAMA, 1988); and they could be grouped logically into proposed factors. These factors were tested via confirmatory factor analysis in their 2006 study, and subsequently, the survey was reduced from 40 to 30 items.

Response Process:

In the Davenport study being reported here, all of the VAMCs in the U.S.A. were invited to participate, and 51% agreed. Similarly, all of the academic medical centers in the U.S.A. were invited to participate and 57% agreed. Monitoring of the NSQIP-guided process of extracting morbidity and mortality data from medical records was done at each local hospital site. The inter-rater reliability of the nurse coding was maintained through “regular conference calls, sample case problems, and formal IRR testing of all nurses.”

Internal Structure:

Five of the six SAQ scales had acceptable to good internal consistency reliability (Cronbach’s alpha) of 0.71 to 0.82. One scale, “Perceptions of management,” was dropped from the analysis due to low reliability (0.54). The Burnout scale (0.82) was retained. Reliability information on the four communication/collaboration items (one per provider group) was not calculated / reported, likely because the 4 items were treated as separate items (one per provider group) and not expected to form a scale. The mean mortality O/E ratio was 1.08 (range 0.51 to 2.12) and the mean morbidity O/E ratio was 1.04 (range 0.47 to 1.59). These results demonstrated good variability for correlation analysis. These ratios were calculated at the hospital site level.

Relation to Other Variables:

Correlations between the SAQ and the Burnout scales were low and non-significant for both mortality and morbidity rates. The only significant correlation found in the analysis was between the item assessing communication and collaboration with attending MDs and morbidity (r=-0.38, p<0.01). The correlation between communication and collaboration with resident MDs and patient morbidity approached statistical significance (r=0.25, p=0.08).

Consequential:

The authors raise important questions about inferring better patient outcomes based on attitudinal surveys such as the SAQ. One challenge in establishing external (consequential) validity of attitudinal surveys such as the SAQ is that patient complications and deaths due to non-patient (i.e., safety) factors are relatively infrequent, making them more difficult to predict.

Assessment & Evaluation

A Validity Study of the Safety Attitudes Survey

Related Resources

User Collections

Start the Conversation