1. Introduction

The huge chasm that exists between the potential and the actual safety-based quality of care delivered by the health care industry is consistently wide across^1,2 most of the world. The World Health Organization(WHO) has formed an ‘Alliance for Patient Safety’. According to WHO patient safety is a Basic Human Right.

A most generally expressed and accepted view is that creation of culture of safety is a critical first step for healthcare organizations that aim to improve quality and safety.^3-6 This approach has been embraced by the National Quality Forum.⁷ The Joint Commission for accreditation of healthcare organizations appropriately included an annual assessment of safety culture in its 2007 Patient Safety Goals. Agency for Health Research and Quality(AHRQ) has developed surveys for assessment of patient safety culture in outpatient and inpatient settings.

It is important to point out that the WHO All Expert Working Group’s 2012 report⁸ calls for understanding of the relationship between safety culture and state of safety in any organization.

With support of an AHRQ grant the authors set out to:

1) examine the feasibility of objectively assessing the impact of a FMEA based Team Resource Management(TRM) intervention on reducing medication errors among geriatric patients in primary care settings, with and without Practice Enhancement Assistants(PEAs). At the heart of this TRM is a Safety Enhancement and Monitoring Instrument that is Patient Centred(SEMI-P).^9-13

2) examine changes in six safety attitude constructs measured with an Ambulatory Safety Attitudes Questionnaire(SAQ-A).^8,14-16

The associated hypotheses were that:

1) implementing an intervention will reduce Adverse Drug Event(ADE) incidence and severity over the comparison group,

2) implementing TRM will result in measurable changes in safety culture, and

3) the Practice Enhancement Assistant(PEA) will further reduce ADEs and improve safety climate beyond the effect of TRM intervention alone.

In this first section we present, very briefly, the TRM intervention and its effects on ADEs. Subsequent sections present detailed description and discussion of effects of this intervention on the safety attitudes.

1.1. SEMI-P Centered TRM

The authors have developed a bottom up Team Resource Management(TRM) approach for not only monitoring safety hazards but also for improving patient safety in an office setting.^10-13,17-19 This approach synergizes with the paradigm of complex adaptive systems and views each medical setting as a complex adaptive micro-system. The measurements that are used to identify and prioritize quality and safety problems must be trusted by the members of the system. The literature reflects scepticism regarding externally driven measures^20-22 and top-down recommendations for improvements, suggesting that, in their current forms, they may not be trusted by many physicians, nurses and other staff as fair and valid measures. An alternative approach, TRM, that solicits and encourages involvement of all team members to identify and prioritize safety and quality problems has SEMI-P at its heart and is prospective.^10,23 The science of observed systems is looped with that of observing systems in the development of this approach.^5,24

1.2. Effects of TRM on ADEs

Rate and severity of ADE’s and preventable ADE’s was measured using a Trigger Tool(TT)²⁵ for the 12-month periods before and after the start of the intervention. It is useful to note that a TT is likely to uncover the largest number of adverse evenets.²⁶ At baseline, among 1019 patients there were 341 ADE’s(33.5 per 100 patient-years), of which 39.3% were preventable. Paired T-tests using site-level rates showed that the rate of preventable ADE’s in the TRM intervention with PEA group was significantly lower after the intervention than before(11.0vs 15.9 per 100 patient-years, p=0.042). Preventable ADE’s most commonly occurred during prescribing and administration of medications.

The most common triggers as well as the largest contributors to ADE’s and to preventable ADE’s were: medication discontinuation, unplanned hospitalization, and emergency department visits.

Among preventable ADE’s

•38% were deemed to have minimal or no clinical effect(e.g., abnormal lab with no symptoms)

•30% were classified as “Severe”(resulting in either hospitalization, permanent disability or death)

The conclusion drawn from this part of the study was that the TRM intervention(when enhanced by a PEA) appeared to affect a significant improvement in medication safety.

1.3. Rationale and Background to the Constructs Safety Culture, Climate and Quantifiable and Unquantifiable Attitudes

Definitions of organizational and safety culture abound in the literature.^27,28 These constructs aspire to help analytical reasoning and practical research. It should be acknowledged that the causes and effects of an organization’s safety culture are intertwined. A highly reliable organization ‘has’ a safety culture and ‘is’ a safety culture, wherein the objective of cultivating this culture is to continuously enhance safety, advisedly, with self-empowered and motivated teams.¹⁰ It is also helpful to acknowledge and treat each ambulatory practice(organization) as a unique and complex adaptive micro-system.¹⁹ In the paradigm of complex adaptive systems, a culture of safety not only functions as a conceptual model but also as a ‘central attractor’ bringing order in disorder(i.e. reliability where there was risk).^17,27 It is interesting to note that a study by Quinn et al.²⁹ showed that physicians from practices that were involved in the evaluation of QI activities had significantly less isolation, stress, and dissatisfaction.

In broad terms, climate can be seen as the observable part of culture. Safety attitudes in turn are a subset of safety climate; they are the part of the climate that resides in individuals and may, therefore, be measurable via self-administered surveys. But it should be remembered that attitudes too are very impressionable and complexly dynamic. Figure 1 portrays the dynamic conceptual relationships between the different constructs.

Figure 2 portrays a framework that attempts to clarify the contributors to safety culture and the relationship of culture to climate and attitudes. This figure shows eight factors that, in our view, contribute to a safety culture.¹⁴ Although the relative importance of these contributors(and potentially others not identified in this framework) are not well understood at this time, it is important to note that each of these contributors interacts with and influences every other contributor(see narrow arrows); they worksynergistically to create a culture of safety. This culture is complex and hyper-dimensional and is the result of complex interactions between multiple players and their beliefs, attitudes, and behaviours over time.

Figure 1. Conceptual Dynamic Relationship between Various Constructs

Figure 2. Framework of interactive contributors to the construct of culture of patient safety:manifesting as safety climate,which expresses itselfpartly in measureable attitudes and perceptions with numerous cybernetic loops with the culture14

The manifest or observable aspects of safety culture are referred to as the safety climate and are in a cybernetic loop³⁰ with overall safety culture(via the wide arrows). interactions of goals, predictions, actions, feedback, and response within systems.30 In this context, climate is seen as a primary manifestation of culture which in turn influences and nourishes culture.^16,31-34

However, even climate is difficult to measure because, like culture, it exists largely not in individuals but in the interactions between them.

Self-administered questionnaires have been developed in a variety of industries as practical and convenient means of measuring quantitatively some of the important aspects of safety climate. These surveys, whether referred to as safety attitude, safety culture, or safety climate questionnaires, can only examine the aspects of climate that are quantifiable and expressible by individuals. These include individual attitudes and beliefs as well as perceptions about individual and group behaviours at any particular time.

It is the apparent ability to quantify safety climate or culture, albeit in a limited way, that has driven the development of self-administered safety attitudes questionnaires over the last 30 or more years for the expressed purposes of measurement, description, diagnosis, and design of interventions for safety. These formative measures should be seen in light of the fact that the mere process of measurement influences the measured, i.e. there is a time-sensitive cybernetic loop here also.

In attempting to describe safety climate quantitatively, a large number of variables can be identified. A number of these measurable variables are interrelated and measure aspects of the same underlying dimension of safety climate. A number of these dimensions, in turn, capture different aspects of the same underlying(unobservable)latent ‘factor’¹⁶ or ‘domain’³⁵ or group/category.³⁶ It is, therefore, possible to reduce/transform, successively, these variables to manageable(quantitatively) dimensions and factors. This transformation is usually done by using Factor Analyses.^16,35,37-40 Capturing the climate in terms of these factors helps to provide, hopefully, a clearer view of climate changes within and variations between different healthcare settings.

The SAQ, or Safety Attitudes Questionnaire is such a questionnaire and¹⁶ is a 60-item self-administered survey tool that was derived from a questionnaire used in commercial aviation, namely the Flight Management Attitudes Questionnaire.⁴¹ In a 2005 study comparing published healthcare safety attitude/climate/culture surveys, the SAQ appeared to be psychometrically the most robust.⁴² The SAQ has been successfully used in inpatient and ambulatory clinics.^16,43 It elicits attitudes through the following 6 scales(or ‘factors’): teamwork climate; safety climate; job satisfaction; perceptions of management; working conditions; and stress recognition. These scales(encompassing 30 of the 60 questions in the SAQ) were developed through multilevel factor analysis using data from 10,843 respondents from 203 clinical areas in three countries(USA, United Kingdom, and New Zealand). The 203 clinical areas included 179 ICU’s, 11 inpatient settings, 11 ambulatory clinics and 2 operating rooms.

A further study, using the ambulatory version of the SAQ(SAQ-A) in a single large multi-specialty academic outpatient practice(282 respondents), demonstrated good internal consistency-reliability for the same six factors (Cronbach’s alphas ranging from 0.68 to 0.86) .⁴² Although the outpatient care environment is very different from the mostly intensive care hospital settings where the factors were developed, the same six factors appeared to be robust in this setting. This may in part be due to the fact that this study was in a large academic practice that in some ways is organizationally similar to a hospital, with a centralized administrative infrastructure. The typical primary care outpatient setting, where the majority of outpatient care is provided, is very different from this. For example, the organizational structure is typically flatter, roles are sometimes less clearly defined(with more cross-coverage) and relationships between staff are usually closer. These and other differences might have significant effects on the performance of the SAQ-A in this type of setting.

As part of our AHRQ-funded study we tested the internal consistency-reliability of this attitudes questionnaire at the outset. The results are published elsewhere.¹⁴ The Cronbach’s Alpha for all respondents, including physician/extenders, nursing staff and administration staff, ranged from 0.58 to 0.77. The lowest(0.40) value of Alpha was recorded among nurses for perception of management and the highest, 0.89 and 0.90, among administration staff for teamwork climate and job satisfaction, respectively.

We concluded, at the outset, that further study was warranted, preferably with a larger sample size than ours, with the goal of developing a more robust instrument tailored to this setting.

2. Objective

The objective of the study presented in this paperwas to measure safety attitude changes in relation to the changes in ADEs by examining changes in the following safety attitude constructs as measured with a 30 item Safety Attitudes Questionnaire(SAQ-A):

a)stress recognition d) teamwork climate

b)perceptions of management e) safety climate

c)working conditionsf) job satisfaction

3. Methods

Table 1 shows the characteristics of the sites studied. This was a cluster randomized trial in which 12 Upstate New York Practice-based Research Network practices were each randomized to one of 3 states(4 practices each):(1) Team resource management intervention;(2) Team resource management intervention with PEA;(3) No intervention (comparison group).

All staff at the intervention sites were invited and encouraged to participate in TRM and respond to SAQ. The intervention period was 12 months.The study protocolwas approved by the Social and Behavioural Sciences Institutional Review Board.

Table 1. Site Characteristic

The data presented in the following are part, as stated earlier, of a larger study in which the authors presented pre-intervention SAQ-A results to each office’s staff(along with other data specifically related to medication safety) as a means of initiating discussions around change. Staff, in their respective sites, then worked together to design and implement changes.

3.1. Implementation

The SAQ-A was administered voluntarily and anonymously to all eligible staff at 8 primary care offices within the Upstate New York Practice-Based Research Network. To be eligible, staff had to have worked at the office(full- or part-time) for at least one month prior to survey administration. As shown in Table 1 the characteristics of the practices ranged from a rural solo practice to an urban academic residency practice site.

The main part of the survey consisted of a series of statements that respondents rate according to a 5-point Likert scale(1 = disagree strongly, 2 = disagree slightly, 3 = neutral, 4 = agree slightly, and 5 = agree strongly). Respondents could also indicate that an item was ‘Not Applicable.’ The survey took approximately 10-15 minutes to complete. Most surveys were distributed in person at brief informational meetings accompanied by a concise explanation of the purpose of the survey, instructions for completion, and assurances of anonymity. For those employees unable to attend the informational session, materials were left with brief written instructions. To help maintain anonymity and confidentiality, participants were instructed to refrain from placing any identifying information on the survey; a secure drop-box was provided for completed questionnaires. Surveys returned within two weeks were included in analysis.

3.2. Analysis

In keeping with the analytic technique of the originators of the SAQ, calculation of safety attitudes for each of the six safety factorswas performed by converting results from categorical to continuous variables as follows: Strongly disagree=0, Disagree=25, Neutral=50, Agree=75, and Strongly Agree=100. Some items were reverse scored so that a higher score always represents a more positive attitude. For each respondent, a mean score of ≥ 75 for the items in a particular factor denotes a “positive safety attitude” for that factor. Survey data were analyzed using SPSS, version 14.0(SPSS Inc., Chicago, IL).

4. Results

Table 2 shows the 30 item questionnaire response rates and mean scores on the six attitude subscales at the start and after 12 months of TRM intervention. Comparison of post versus pre data by ANOVA showed no significant differences with or without PEA despite anecdotal reports from multiple sites of improvement in attitudes as measured by SAQ-A.

Table 2. Response Rates and Mean Scores on the Six Subscales: pre- and post-intervention

We speculate that the reduction in the post intervention response rates may reflect reduced motivation to respond to the SAQ-A with perceived improving state of safety in practices.

5. Discussion

The theoretical framework described in Figure 2 includes several contributors to safety IT. Some of these contributors can be mapped to measurable attitudes, many of which are covered by the SAQ and SAQ-A. For example, ‘Creation of a Learning Environment’ maps very closely to the SAQ’s Safety Climate scale, ‘Creation of Non-hierarchical Teams’ corresponds to Teamwork Climate, Job Satisfaction, and Stress Recognition in the SAQ, and ‘Prioritization of Safety by Leadership’ is an area that is well addressed via the Perceptions of Management and Working Conditions scales. The framework reveals some areas that could be addressed further, such as ‘Design of the System for Recovery’ and ‘Adoption of a Proactive Approach’ that are not explicitly covered by the SAQ. It should also be recalled that all these 8 contributors manifest in complex multidimensional ways, some of which are either not measurable or require methods of assessment other than self-administered surveys.

In an effort to explain absence of direct correlation between reduction of ADEs due to TRM intervention and no significant change in measured safety attitudes, we have visualized threshold effects of TRM intervention, combined with concurrent administration of SAQ-A, on the dynamic interplays between total state of safety, perceived state of safety, the various constructs of culture/climate/attitudes, and the ADEs.

Figure 3 portrays the conceptual interplay, over a 12 month intervention period(t₁ to t₂), between the beneficial effect of TRM on ADEs and:(1) changes in total number of possible errors or total risks, and(2) changes in perceived number of errors or total risks. These relative changes are assumed to be due to improved situational awareness due to the formative effects of TRM and SAQ-A and the normative/summative effects of TRM. It is helpful to note that a number of studieshave shown that increase in error reporting can accompany overall safety improvements in a setting. A 2009 Report by AHRQ⁴¹ found that “hospitals with improvements over time in non-punitive response to error had slight increases in event reporting.”

Figure 3. Conceptual interplay between the beneficial effect of TRM on ADEs, changes in total number of possible errors or the total safety risks, and changes in perceived number of errors or the total safety risks

Figure 4. Interplays between the improvement in medication safety and the constructs of safety culture, climate, and the measured attitudes

For further clarification and understanding of the threshold effects we have visualized(Figure 4) the possible interplays between the improvement in medication safety and the(significantly affected) constructs of safety culture, safety climate, and the(insignificantly affected) measured safety attitudes. These interplays are thought to be caused by the formative and normative/summative influences, TRM intervention and the formative influences of SAQ-A.

As stated above only 30 of the 60 items are included in the six climate scales evaluated in this and prior studies. We have speculated above that the reduction in the post intervention response rates may reflect reduced motivation to respond to the SAQ-A with improving state of safety perceived in a practice affected by SEMI-P centered TRM intervention. SEMI-P centered TRM intervention is designed to inculcate mutual respect and trust, cooperation, and collaboration by generating common vision.

It may be helpful to introduce the concept of context-specificculture of safety. As stated in the first paragraph of Section 2, a highly reliable organization ‘has’ a safety culture and ‘is’ a safety culture, wherein the objective of cultivating this culture is to continuously enhance safety, advisedly, with self-empowered and motivated teams.¹⁰ Improvement of reliability in the domain of medication with TRM can therefore be viewed as improvement in the culture of safety in the context of medication. The TRM-based improvement methodology, after all, was informed by our framework of interactive culture formers¹⁴ illustrated in Figure 2.

It may also be helpful to note that in the general domain of safety there is lack of evidence to demonstrate that the currently available safety climate/attitude measurement tools/questionnaires adequately indicate the state of safety in the workplaces.^27,45 The 2009 AHRQ Report,⁴⁴ based on trends in hospital settings cautions that “survey scores might change, or not change, over time for a number of complex reasons” despite “patient safety actions”. Huang et al.⁴⁶ observe that “safety culture may influence patient outcomes but evidence is limited” and concluded, from their study in intensive care units, that safety climate(as measured with SAQ-ICU) was only moderately associated with patient outcomes. They recommend further work to develop methods of assessing safety culture. Weingart et al,⁴⁷ based on their study in multiple hospitals, concluded that independent indicators of patient safety did not line up neatly with safety culture surveys and that “the safety culture is a complex phenomenon that requires further study.” Waterson et al.⁴⁸ in their 2010 study found that measurement of safety culture and climate in healthcare is still a relatively immature stage of development and argue that there is need to further develop and construct theoretical models that are sensitive to the context-specific nature of healthcare environments.

We concur with views recently expressed by Guldenmund⁴⁹ that “culture is an intangible, fuzzy concept encompassing acquired assumptions that is shared among the members of a group.” We also hold the view that it is vital that the concept of culture is not deprived of its depth and subtlety and “morphed onto a grab bag of behavioural and other visible characteristics” and it should not be assumed to have normative/summative attributes.

The above discussion should be seen in the light of the fact that overall average Chronbach’s Alpha measures with SAQ-A for all respondents in our study ranged from as low as 0.58 to no higher than 0.77.

It may be worth considering use of recently developed AHRQ’s 12-dimensioned “Medical Office Survey on Patient Safety Culture” in future studies. As a word of caution it should be noted again that the trending data, obtained using 12-dimensional Hospital Survey on Patient Safety Culture, in the AHRQ 2009 Report [AHRQ 2009] shows that despite the “patient safety actions” taken by the trending hospitals there was only marginal increase in “overall perceptions of patient safety”, and that too in only thirty seven% of these hospitals.

In the authors’ view relation between safety and culture is often based on wrong assumption that people are the problem. The most famous quality guru Dr.W. Edwards Deming, who taught the theory of profound knowledge, always emphasized that 85% of the time, the problem is management, not the worker. Introducing a safe system is the responsibility of management. Introducing safety culture with wrong system in place is not going to work. That is why the patient safety movement has been a failure, according to Dr. Lucian Leape, the co-founder of patient safety movement more than a decade ago. We need to fix the system to prevent harm, not fight the problems after we created them. Preventing harm requires use of tools such as FMEA and Fault Tree analysis on ALL critical procedures.

6. Conclusions

1. At baseline SAQ-A revealed significant differences in safety attitudes between sites, attesting to the uniqueness of each setting.

2. Five of the six subscales had reasonable internal consistency-reliability in primary care offices but some performed poorly with some subgroups. Further work is needed to evaluate and refine the instrument for these settings.

3. Combined pre- and post-intervention scores of the safety attitudes constructs were: (a)stress recognition: 62 vs 64.8, (b)perceptions of management:64.3 vs 61.5, (c)working conditions: 68.1 vs 63.9, (d) teamwork climate: 75.4 vs 72.9, (e) safety climate: 73.3 vs 75.2, and (f) job satisfaction: 78.4 vs 77.0. Despite anecdotal reports to the contrary, the efficacious TRM intervention appeared to have had no significant effects on safety attitudes as measured by SAQ-A.

4. Measurement of safety culture and climate in healthcare is still a relatively immature stage of development and there is need to further develop and construct theoretical models that are sensitive to the nature of healthcare environments.

5. The concept of context-sensitive culture of safetyis,therefore,worthy of further development.

6. It may be worth considering use of the recently developed AHRQ 12-dimensioned “Medical Office Survey on Patient Safety Culture” in future studies.

7. Our work responds to WHO All Expert Working Group’s 2012 call to understand this interrelationship between safety culture and a successful intervention designed to improve safety.

6.1. Limitations

•Small number of sites

•Modest response rates(esp. post-intervention)

•Lack of a control group

•Confounding factors

•Changes in management, staff turnover, etc.

•Limitation of the measurement tool

•Some aspects of safety climate not covered

•Some issues not amenable to this approach

•SAQ-A is a formative tool; confounding the “before and after” comparisons, e.g., people’s threshold for agreement may change.

ACKNOWLEDGEMENTS

Authors are grateful for AHRQ Grant:1R21HS014867-01A1; Improving Geriatric Drug Safety in Underserved Practices.