Evaluation of the Canadian Patient Safety Institute Activities 2013-2014 to 2016-2017

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from April 2013 to March 2017 and included an assessment of the Canadian Patient Safety Institute's (CPSI) activities such as knowledge generation, synthesis and translation, competency building, leveraging the work of other organizations and building relationships.

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Results (2016) and considered the five core issues under the two themes of relevance and performance.^Footnote.

The Office of Audit and Evaluation acknowledges the consulting firm of Prairie Research Associates, which conducted the recipient-led evaluation of CPSI's activities.

Data for the evaluation was collected using various methods, including data, document and literature reviews, key informant interviews, survey of CPSI stakeholders, and case studies. More information on data collection and analysis methods is detailed in Appendix 3. In addition, data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

The most important limitation relates to the difficulty in attributing outcomes to CPSI. Attribution is a common challenge in many evaluations, particularly for intermediate and longer-term outcomes, since a multitude of factors may influence the achievement of outcomes, and experimental or rigorous evaluation designs (i.e., evaluation designs incorporating a counterfactual) are typically not feasible. While it is often possible to isolate the extent to which activities have produced immediate outcomes, which are expected to occur soon after activities are implemented, attribution becomes considerably more difficult as one moves along the results chain, where external factors are likely to impact the intermediate and final outcomes.

In light of this challenge, contribution analysis was used to draw conclusions about the extent to which CPSI activities have contributed to the achievement of intermediate and longer-term outcomes. According to the Treasury Board of Canada Secretariat, contribution analysis holds that "if an evaluator can validate a theory of change and account for external influencing factors, then it is reasonable to conclude that the intervention has made a difference".^{Footnote 9}  Contribution analysis involves assessing whether activities were implemented as planned, assessing whether the expected outcomes were observed, and accounting for the influence of external factors.

The data collection methods employed in the evaluation also had a number of limitations associated with them. For example, the literature review was limited in scope due to time and resource constraints, while the generalizability of the findings from the survey, key informant interviews, and case studies was limited by a non-random, non-representative approach to sampling and the possibility of self-selection bias among respondents. These limitations were mitigated through triangulation; that is, using the findings in conjunction with those from other lines of evidence in order to achieve greater confidence in the results.

Finally, as part of the data review, the data contained in the Patient Safety Metrics (PSM) system, through which CPSI has collected information on approximately 100 process and outcome indicators for Safer Healthcare Now! (SHN) interventions since the inception of SHN in 2005, were examined to determine their potential to contribute to the evaluation. The review determined that, while it would be theoretically possible to analyze these data using these techniques to determine which, if any, interventions were reliably linked to outcome improvements, such an analysis would require considerable time, as well as medical and statistical expertise, and was therefore determined to be outside the scope of the current evaluation.

4.0 Findings

4.1 Relevance: Issue #1 – Continued Need for the Program

More than a decade after patient safety first emerged as a major concern within the Canadian healthcare system, available data indicate that patient safety has not appreciably improved, suggesting an ongoing need to address the issue.

More than a decade after the Canadian Adverse Effects (CAE) Study first shed light on patient safety in the Canadian context and prompted policy-makers to act, the extent to which the Canadian healthcare system is safer for patients remains unclear. A significant challenge in assessing progress in this area — one that is not unique to Canada — is the absence of a single indicator or general measure of patient safety, and the consequent need to examine data pertaining to a range of relatively specific patient safety measures.^{Footnote ii}  The available Canadian data suggest that patient safety has not substantially improved:

• A study of patient safety in Canadian children's hospitals showed that in 2008, 9.2% of hospitalized children experienced an adverse event and 3.6% experienced a preventable adverse event.^{Footnote 10}  This compares with 7.5% of hospitalized adults who experienced an adverse event and 2.8% of adults who experienced a preventable adverse event in the 2004 CAE Study.
• A recent Canadian study examining adverse events during home care concluded that providing healthcare through home care programs "creates unintended harm to patients".^{Footnote 11}  In particular, the study found an annual adverse event incidence rate of 13.2%, concluding that a significant number of home care patients experience adverse events, of which one-third were preventable.
• The hospital standardized mortality rate (HSMR), which measures the number of hospital deaths compared to the expected number of deaths over time, fell from 108 in 2010 to 95 in 2014.^{Footnote 12}  While this is undeniably a positive outcome, the HSMR may not be a sensitive measure of patient safety.^{Footnote iii} 
• More granular indicators of patient safety identified by the Canadian Institute of Health Information (CIHI, see Table 2) do not show a pattern of significant improvement over time. Notably, in-hospital sepsis and falls, both of which have been targeted by the Safer Healthcare Now! (SHN) program, have remained relatively constant or have shown slight increasing trends.

Data available from the Organisation for Economic Co-operation and Development (OECD) allow for some comparison of the state of patient safety in Canada to the international context. Although health information is collected and used differently in every country, policy-makers, practitioners, and the public can use international comparisons to establish priorities for improvement, set goals, and motivate stakeholders to act. According to a 2014 CIHI report based on the OECD data, Canada falls behind the international average on five of the six measures of patient safety in the acute care setting.^{Footnote 13}  Although this result may be partially explained by Canada's reporting of adverse events, there is clear opportunity for improvement. Furthermore, the rate of harm captured by the Hospital Harm Measure, a collaborative project of CPSI and CIHI to address gaps in patient safety information, has remained stable at 5.6% between 2012–2013 and 2014–2015.^{Footnote 14} 

Overall, therefore, the available data suggest that patient safety should remain a concern for the Canadian healthcare system. Indeed, a recent report integrating evidence from the patient safety literature, international and Canadian patient safety experts, and Canadian healthcare leaders concluded that although some progress has been made, many healthcare organizations in Canada still have not adequately addressed important patient safety issues.^{Footnote 15} 

Furthermore, there is growing recognition in the Canadian and international literature of the barriers that exist when attempting to improve patient safety through the introduction of new clinical procedures. The amount of work required to implement a new procedure, as well as its apparent value to the team both strongly impact whether an effort will be adopted or sustained.^{Footnote 16}  Simply knowing that a set of procedures can be effective is not sufficient to engender change. An organization's "readiness to change" involves a number of factors, including available time and educational resources, team building activities, the ability to adapt workflows, and increased cross-disciplinary collaboration — all of which must be supported by governance bodies and administrators.^{Footnote 17 Footnote 18} 

These findings accord with an assessment of progress on patient safety in the US, Free from Harm: Accelerating Patient Safety Improvement Fifteen Years after To Err is Human.^{Footnote 19}  The central finding of this report was that sustainable improvements can be achieved only through a total systems approach to safety and the fostering of a culture of safety. Similar conclusions were reached by Baker and Black in their 2015 Canadian study, Beyond the Quick Fix: Strategies for Improving Patient Safety,^{Footnote 20}  and by the UK's National Institute for Health Research in its 2016 report, Patient Safety 2030.^{Footnote 21} 

These three studies all point to several tools that are integral to this systems approach, including the promotion of improved teamwork and communication across inter-professional teams; better measurement of patient safety at system-wide, organizational, and team levels; the building of a reporting and learning culture; attention to transitions across the continuum of care; stronger leadership at political, organizational, and clinical levels; engagement of patients and their families; and more coordination and collaboration across organizations and jurisdictions. Many of these tools aim to create higher reliability systems, in which all those involved in delivering and receiving care display greater awareness of the level of safety, more sensitivity to risks as they arise, and greater ability to make adjustments that reduce the incidence of harm.

One of CPSI's early efforts to improve patient safety, SHN, was organized around the promotion of best practices and quality improvements within particular clinical areas. In recent years, CPSI has recognized that these clinical best practices are necessary, but not sufficient, for change. Over the current funding agreement, CPSI has sought to take a more systems-level and systems-based approach to change by, for example, fostering greater collaboration across jurisdictions and organizations, embedding patient safety competencies in professional education across the healthcare system, sharing best practices in patient engagement, advancing system-level measures of patient safety, and fostering cultures that promote teamwork, reporting, and learning. Perhaps most notable in this effort is CPSI's transition from SHN to SHIFT to Safety, a new initiative for improving patient safety that seeks to address some of these elements of a more systems-based approach. SHIFT to Safety and CPSI's other new initiatives are described more fully in the remainder of this report.

4.2 Relevance: Issue #2 – Alignment with Government and Other Stakeholder Priorities

There is alignment between CPSI and federal priorities, and CPSI priorities are generally perceived by other stakeholders to be well-aligned with their own. While most stakeholders believe CPSI has adjusted effectively to changing needs and circumstances and that its activities address gaps and needs in the patient safety field, concerns were raised that CPSI lacks a clear strategic focus to guide its work and that other organizations are active on these issues.

CPSI priorities

CPSI's current strategic goals and priorities have been shaped, to a large extent, as a response to the recommendations of its last evaluation. That evaluation, completed in 2012, recommended that CPSI focus or rationalize its scope of activity, enhance engagement with the system, and embed ownership of major programs and activities within the system.^{Footnote 22}  CPSI's 2013–2018 Business Plan, Patient Safety Forward with Four,^{Footnote 23}  subsequently laid out, in broad strokes, its plan for responding to these recommendations. The Business Plan identified four strategic goals to focus and align CPSI's efforts over the five-year period:

• Strategic Goal #1: Provide leadership on the establishment of a National Integrated Patient Safety Strategy
• Strategic Goal #2: Inspire and sustain patient safety knowledge within the system, and through innovation, enable transformational change
• Strategic Goal #3: Build and influence patient safety capability at organizational and systems levels
• Strategic Goal #4: Engage all audiences across the health system in the national patient safety agenda

As a mechanism for achieving its overarching strategic goal of establishing a National Integrated Patient Safety Strategy, and consistent with the recommendation to enhance engagement with the system, CPSI convened the National Patient Safety Consortium (the Consortium) in January 2014. This coalition represents approximately 50 organizations across Canada, including Health Canada and the Public Health Agency of Canada (PHAC), provincial and territorial quality and patient safety councils and ministries of health, professional associations and other national organizations, and patient groups, among others. Its purpose is to drive a shared action plan for safer healthcare, with the desired outcome of safer healthcare in Canada. CPSI serves as the coordinating body for this effort and, as such, is responsible for maintaining overall strategic coherence and coordination of the Integrated Patient Safety Action Plan (IPSAP). The Consortium and the IPSAP are described in detail in Appendix 4.

Federal priorities

Historically, the Government of Canada  has demonstrated its commitment to supporting improvements to patient safety within the Canadian healthcare system through funding to CPSI and the Canadian Medication Incident Reporting and Prevention System, and through its efforts in the area of infection control and prevention and other activities, including development of legislation to improve the safety of medications and other health products.^{Footnote 24}  While there is an annual commitment to provide CPSI with $7.6 million in federal funding, the government's March 2016 Budget did not specifically mention CPSI or patient safety.^{Footnote 25}  Health Canada key informants explained that federal and departmental priorities are currently focussed on broader health system innovation. Within that context, patient safety is closely linked to some current federal priorities, such as legislation to improve medication safety (e.g., Vanessa's Law), but is not a specific priority or area of focus in its own right.

Priorities of other stakeholders

Development of the IPSAP took into account input from a wide range of stakeholders in the field of patient safety, and it was explicitly designed to focus on gaps and areas of high priority, as identified by the field. Results from the survey indicate that CPSI is reasonably well-aligned with the priorities of stakeholders, and is responsive to gaps and needs in the field. A majority (71%) of survey respondents believe CPSI's activities are well-aligned with their or their organization's patient safety priorities. The same proportion believes that CPSI's activities address identified gaps and needs in patient safety, though somewhat fewer (61%) think CPSI responds effectively to changing needs in the field (see Table 3).

Among key informants, there was consensus that CPSI has historically been very effective in responding to needs in the field by developing tools, resources, and strategies that could be used by healthcare organizations at the clinical and governance levels to address patient safety.

Evidence-based products were needed in Canada and they made them happen.

Numerous key informants reported that CPSI tools and resources, including SHN, the Canadian Disclosure Guidelines, the Incident Analysis Framework, and various educational offerings, have been extremely important and beneficial to advancing patient safety within their own organizations. The importance of these tools and resources to smaller organizations and jurisdictions with limited resources was particularly emphasized.

CPSI has helped us to ensure that we can meet [Accreditation Canada] standards. And CPSI’s adverse event and disclosure guidelines were a key part in rebuilding our incident management framework.

Safer Healthcare Now! impacted frontline provider care and therefore improved our patient outcomes within our regional health authorities (RHAs). SHN was a great program because people at the grassroots level could see the value in patient safety initiatives because they could see an improvement in patient outcomes. So that helped organizations within our province strategically target certain areas for improvement.

Pretty much all of CPSI’s offerings that are relevant to the acute care sector have been adopted by [my organization]. We’ve participated in Safer Healthcare Now!

Our patient safety incident management program is based on the foundation of the CPSI work in that area, and our disclosure processes are based on CPSI products.

Key informants noted that over time, healthcare organizations have matured and many have developed internal expertise in patient safety, so that they no longer need CPSI’s resources. In some cases, organizations have evolved in their internal patient safety capacity, such that they now provide support to CPSI.

As our organization has matured, the need for CPSI as a clinical resource has changed. […] We have this ongoing relationship around more strategic types of issues [and] some of us continue to be faculty to support CPSI, but as an organization using tools and techniques, not so much anymore.

Provinces and health systems have taken what CPSI has offered and they’ve ingrained it into their own activities. So the relevance of CPSI is less to them. CPSI is the shoulders upon which they stand rather than what is leading them in the new direction. That’s a real success, but it’s also the thing that puts you out of a job.

The case study of Alberta Health Services (AHS) provides a concrete example of an organization that has benefitted from CPSI’s work in the past, and that has now matured and developed its own internal capacity in patient safety, though it continues to maintain a collaborative relationship with CPSI.

Key informants noted that at the same time as some organizations have developed internal capacity to address patient safety, a variety of other organizations offering similar resources and supports have emerged and patient safety has become more closely linked to a number of other health quality issues, losing distinctiveness as an issue in its own right. Although not all key informants commented on CPSI’s effectiveness in responding to these changing needs and circumstances, those who did were divided. Some believe CPSI has adjusted effectively, citing its current business plan, the development of the IPSAP, and the recent transition away from SHN to SHIFT to Safety. Others reiterated concerns that CPSI continues to lack a clear strategic focus and may be attempting to do too many things, particularly given its size and budget. A more detailed discussion of these points is provided in section 4.5.

4.3 Relevance: Issue #3 – Alignment with Federal Roles and Responsibilities

CPSI’s mandate and activities are consistent with federal roles and responsibilities and, in particular, with Health Canada’s mandate under the Department of Health Act. CPSI meets expectations that Health Canada has for pan-Canadian health organizations to perform a coordinating function and drive pan-Canadian collaboration. However, there has been significant evolution in recent years.

The Constitution Act 1867 outlines provincial and territorial responsibility for delivering healthcare, but also identifies the Government of Canada’s roles and responsibilities in health promotion, disease prevention, knowledge sharing, research funding, and ensuring access to healthcare for specific populations. Health Canada’s funding to CPSI helps fulfill the government’s roles in disease prevention, knowledge sharing, and research funding.

Health Canada funding to CPSI is provided pursuant to its mandate under the Department of Health Act. Section 4 of the Act defines the Minister’s duties to include: the promotion and preservation of the physical, mental, and social well-being of Canadians; their protection against risks to health and the spreading of diseases; investigation and research into public health; the establishment and control of safety standards and safety information on requirements for consumer products; and the collection, analysis, interpretation, publication, and distribution of information relating to public health. More broadly, the Minister’s jurisdiction covers all matters related to the health of Canadians who have not otherwise been assigned by the government to another body.

CPSI describes its vision as “safe healthcare for all Canadians” and its mission as to “inspire extraordinary improvement in patient safety and quality.” As such, CPSI’s vision and mission align with Health Canada’s broad mandate to promote and preserve the physical, mental, and social well-being of Canadians, as described in the Department of Health Act.

In the context of Health Canada’s and CPSI’s broad organizational mandates, CPSI’s performance measurement strategy describes the purpose of Health Canada’s continued support for CPSI as:

to improve the quality of healthcare services by strengthening system coordination related to patient safety, including promoting national collaboration among key healthcare improvement partners. There continues to be a need for coordination and collaboration among the multiple organizations in the healthcare system in order to increase patient safety capacity.^{Footnote 26}

Insofar as CPSI collaborates with a variety of stakeholders across the country to develop and disseminate evidence-informed products and services, and to coordinate stakeholder efforts relating to patient safety, its activities align with Health Canada’s expectations for pan-Canadian health organizations to promote collaboration and coordination. Of course, Health Canada also expects CPSI to fulfill its mandate of improving patient safety, as set out in its policy and funding authorities.

Role and unique value of CPSI

Large majorities of stakeholders agreed that there is a continued need for a dedicated patient safety organization at the national level, and that CPSI brings unique value to the field. Stakeholders variously perceive CPSI’s unique value as being, among other things, the pan-Canadian focus it brings to bear on efforts to improve patient safety, its ability to bring together experts and stakeholders from across the country to work collaboratively on shared priorities, and its role as a clearinghouse for evidence-based patient safety resources, tools, and best practices. Some stakeholders believe CPSI has had difficulty carving out a clear role for itself within the patient safety landscape.

The Canadian patient safety landscape has evolved significantly since CPSI was first established. It is populated at the present time by a considerable number of organizations whose scope of activity encompasses patient safety — some of which, key informants noted, were established with support from CPSI. At the provincial and territorial level, every province except Quebec currently maintains an organization responsible for patient safety, with the Atlantic provinces working together through the Atlantic Health Quality and Patient Safety Collaborative (AHQPSC). Generally speaking, these organizations integrate patient safety into a broader health quality mandate, only the Manitoba Institute for Patient Safety (MIPS) has a mandate uniquely focussed on patient safety. Each provincial organization necessarily has a more local focus than CPSI.

Similarly, at the national level, while an exhaustive inventory of organizational mandates, roles, and responsibilities was beyond the scope of this evaluation, it is clear that numerous organizations are active in the area of patient safety. Some examples are Accreditation Canada, HealthCareCAN, Canada Health Infoway, the Institute for Safe Medication Practices (ISMP) Canada, and the Canadian Foundation for Healthcare Improvement (CFHI), to name just a few.

Recently, the Advisory Panel on Healthcare Innovation, formed by Health Canada in 2014, recognized pan-Canadian healthcare organizations like CPSI as the building blocks for a collaborative approach to healthcare innovation.^{Footnote iv Footnote 27}  However, the Panel recommended amalgamating several of the existing pan-Canadian healthcare organizations, including CPSI, CFHI, and Canada Health Infoway, into a new Healthcare Innovation Agency, in order to “reduce duplication, provide some economies of scale for the federal government, and streamline a crowded pan-Canadian health organization field.”

While the Panel was not tasked with identifying ways to improve patient safety, its report raised questions about whether a national organization dedicated to patient safety is still needed to achieve this outcome. On this issue, the evaluation found widespread agreement among CPSI stakeholders who were surveyed for this evaluation. A large majority of survey respondents (90%) agreed that there is a need for a national organization dedicated to patient safety in Canada; only 2% disagreed, while 4% were neutral, and a similar proportion did not know.

Moreover, a sizeable majority of respondents (72%) believe there are no other organizations in Canada that provide patient safety products and services similar to those offered by CPSI. This belief was significantly stronger in organizations that provide direct patient care (75.3%) than in those that do not (57.4%). The minority (28%) of respondents who disagreed most often mentioned provincial quality or patient safety councils or organizations such as the Institute for Safe Medication Practices (ISMP) Canada, Accreditation Canada, and the MIPS, among others. On the other hand, although 29% of respondents disagreed that another organization would step in and do the same work if CPSI did not exist, almost as many were neutral (25%), agreed (24%), or simply did not know (21%).

When asked about the value of CPSI’s work in specific areas, almost 80% of survey respondents believe the organization makes a valuable contribution to patient safety through its educational courses and programs (79%) and its efforts to encourage organizations to integrate patient safety standards into educational curricula, regulatory requirements, and professional standards (79%). About three-quarters (74%) believe CPSI makes a valuable contribution by promoting and supporting collaboration and coordination among stakeholders in the field (74%). Somewhat fewer, but still a majority (63%), believe CPSI makes a valuable contribution to patient safety through its commissioned research; notably, more than one-fifth of respondents (22%) did not know (see Table 4).

Furthermore, a large majority of survey respondents (83%) believe CPSI brings unique value to the field of patient safety (only 17% disagreed). Survey respondents offered a wide range of perspectives on what CPSI’s unique value is. Common responses were CPSI’s role as a source of evidence-based information on patient safety (20%) and the national/pan-Canadian perspective or focus that CPSI brings to bear on efforts to improve patient safety (14%). Both of these themes were identified by one-fifth of respondents or fewer, and a wide range of other responses were also given. Almost one-third of those who believe CPSI brings unique value did not elaborate (see Table 5).

As a group, therefore, survey respondents believe strongly in the ongoing importance of, and need for, a national organization dedicated to patient safety, and perceive CPSI as bringing unique value to the patient safety landscape in Canada. They offered a diverse range of perspectives regarding that unique value, and recognized that various other organizations are also active in the field.

Similar findings emerged from the key informant interviews. The vast majority of key informants believe CPSI brings unique value to the patient safety field in Canada, and, like survey respondents, offered a variety of perspectives when asked to explain what that unique value is. For some, CPSI’s unique value is that it is the only organization with a mandate focussed solely on patient safety, others mentioned its ability to bring together experts and stakeholders from across the country to work collaboratively on shared priorities, and still others highlighted its role as a clearinghouse for evidence-based patient safety resources, tools, and best practices. The latter role is seen as particularly valuable and cost-effective for smaller jurisdictions and organizations that do not have the capacity to develop or compile these resources on their own.

If CPSI weren’t there, I don’t think we would get that national focus. They also maintain the focus on patient safety. Their commitment and engagement to moving patient safety forward is remarkable.

CPSI is the organization that pulls all of the organizations together to come up with a plan […]. No other organization in Canada plays that coordinating role. That’s the value added of CPSI.

CPSI is an invaluable clearinghouse for patient safety resources and best practices. CPSI gives us a Canadian approach and standard to ascribe to. That is very valuable and helpful. It is not cost-effective for everyone to do this work on their own.

Similarly, like survey respondents, most key informants believe there is an ongoing need for a dedicated patient safety organization in Canada, even though virtually all recognized that many other organizations are also active in the field. Key informants argued that a dedicated national organization is still needed to provide leadership and maintain focus and attention on patient safety — particularly in light of technological innovations and the increasing complexity of the healthcare system, as well as the financial pressures it is facing — and to provide continuity with the patient safety work that has been done in the past.

There is still a tremendous amount of work to do with culture change, with patient safety education, with effective partnerships with patients, with patient safety measurement […]. There is no one else really that is pushing that mandate right now, other than CPSI, at a national level.

That said, some key informants suggested that a dedicated organization should have a more strategic focus than CPSI currently has.

There is still a need. The question is: What are the critical opportunities for CPSI? […] Patient safety still has relevance, but it’s not the premier issue it was a decade ago. The strategic focus for CPSI has to be: How do we contribute expertise and insights into the broader set of issues that the healthcare system is struggling with?

Pointing to its many intersecting partnerships and collaborations, as well as its ambitious work plan, particularly given its small size, these key informants were of the opinion that CPSI continues to experience challenges in carving out a concise, clear role and strategic mandate for itself within the patient safety landscape in Canada. A small number of key informants were of the view that a dedicated national patient safety organization is no longer necessary, given the evolution of the field over the past decade.

4.4 Performance: Issue #4 – Achievement of Expected Outcomes (Effectiveness)

In this section, we outline the extent to which key program outcomes have been achieved. The main focus of this evaluation is on the immediate and intermediate outcomes. Since 2013, and in response to the recommendations of the previous evaluation, CPSI has oriented its activities toward achieving system-level change through the pursuit of four strategic goals (see Appendix 4 for further details).

4.4.1 To what extent have the immediate and intermediate outcomes been achieved?

Increased evidence base to improve patient safety

Since it was established in 2003, CPSI has made available a range of tools and resources that are based in national and international research evidence, and has contributed to growing the patient safety evidence base through its own research activities. Some stakeholders believe CPSI could further contribute to the evidence base through greater emphasis on measurement and rigorous evaluation, including evaluation of its own products and interventions.

In the immediate term, CPSI’s work is expected to produce an increased evidence base to improve patient safety. CPSI key informants noted that all of the organization’s knowledge products and resources are grounded in national and international research evidence. Indeed, many key informants emphasized the important role that CPSI plays in compiling research evidence and translating it into practical tools and resources that can be used by frontline clinicians and other stakeholders. Between 2012–2013 and 2015–2016, CPSI developed 111 new resources and revised 62 existing resources.

Furthermore, CPSI routinely engages academic faculty in the development of its resources and delivery of its educational offerings, with a view to ensuring that the best possible evidence is used as the foundation for its products and that educational offerings are delivered by those best able to interpret and communicate the significance of this evidence. CPSI’s performance data indicate that in 2015–2016, CPSI engaged faculty in the development of 52 resources and in the delivery of 38 educational offerings.^{Footnote v}

In addition to its role in the area of knowledge translation, CPSI has also contributed directly to increasing the evidence base through its research activities. Alongside various funding partners, CPSI has commissioned and published 12 research studies (including five in the current evaluation period) on focussed patient safety topics, including patient safety in home care, primary care, emergency, and long-term care settings; the economics of patient safety; drug safety; incident analysis; governance of quality and patient safety in primary care; patient safety pertaining to new oral anticoagulants; and paediatric medical safety. Furthermore, since 2014–2015, CPSI has collaboratively sponsored three annual $50,000 research competitions focussed on contemporary challenges or questions in the field of patient safety.^{Footnote 28}  In addition, $5,000 patient safety studentship awards are offered annually to support student-driven research that directly contributes to the evidence base, while supporting the development of future research capacity.^{Footnote 29}  CPSI awarded four of these awards in 2013–2014 and five in 2014–2015.

CPSI key informants reported that the organization has, in the current funding period, deliberately scaled back its research activities, both in response to the emergence of other sources of funding for patient safety research^{Footnote vi}  and as part of its response to the recommendation of the 2012 evaluation to focus the scope of its activities. Accordingly, since 2013, CPSI has been more deliberate in targeting its funding and resources to its priority areas. Its most recent calls for commissioned research, for example, have focussed on the home care sector, while its studentship research focusses on advancing the work of the Integrated Patient Safety Action Plan
(IPSAP) and SHIFT to Safety priorities.

Survey respondents and key informants generally agreed that CPSI has contributed to an increased evidence base through its work. Among survey respondents, almost two-thirds (63%) agreed that the evidence base needed to improve patient safety has increased because of CPSI’s work, while only 1% disagreed and almost one-quarter did not know.

As for key informants, most believe that CPSI’s work has contributed to increasing the evidence base needed to improve patient safety, and some identified CPSI’s role as a clearinghouse for evidence-based resources and best practices as the unique value that CPSI brings to the patient safety field. However, several key informants offered a different view, arguing that although CPSI’s products are informed by published evidence, the organization itself has not contributed significantly to the evidence base. It was also pointed out that none of CPSI’s products have been rigorously evaluated for their impact on patient safety, which a few key informants considered a missed opportunity for CPSI to make a material contribution to the evidence base.

Although these key informants did not specifically define the term “rigorous evaluation”, they appeared to be referring to evaluation designs that enable comparison of outcomes achieved as a result of a program to outcomes achieved in the absence of the program (i.e., evaluation designs that incorporate a counterfactual). As a specific example, it was observed that through Patient Safety Metrics (PSM), CPSI has collected data on more than 100 process and outcome measures pertaining to Safer Healthcare Now! (SHN) interventions since the inception of that program in 2005, but has not, to date, published any analysis of these data in the peer-reviewed literature that could speak to the effectiveness of SHN interventions at improving patient safety.^{Footnote vii}  As another example, it was noted that CPSI has not rigorously evaluated Patients for Patient Safety Canada (PFPSC) and its activities to increase patient and family involvement in the health system.^{Footnote viii}  These key informants were of the view that CPSI should place a stronger emphasis on rigorous measurement and evaluation, including measurement and evaluation of its own products and interventions, as well as greater emphasis on the ongoing measurement of progress toward the long-term outcome of improved patient safety. The Hospital Harm Measure was identified as a positive step in this direction.

Evidence-informed patient safety curricula

Over the period covered by this evaluation, patient safety curricula across health disciplines have been increasingly informed by CPSI’s work. A key success has been the integration of the Safety Competencies into the CanMEDS educational framework of the Royal College of Physicians and Surgeons of Canada and the College of Family Physicians in Canada. As part of the IPSAP, a national Patient Safety Education Action Plan has been developed.

In the immediate term, CPSI’s work is expected to result in the formulation and delivery of evidence-informed patient safety curricula across health disciplines. CPSI has contributed to this outcome directly by creating a variety of learning modules on patient safety topics, which are targeted at undergraduate and graduate students, as well as practicing medical professionals, and are offered free through CPSI’s website and in-person learning sessions.

In addition, CPSI completed 15 curriculum mappings of the Safety Competencies Framework^{Footnote 30}  with nursing, pharmacy, and medical faculties, one specialty school, and seven national medical accreditation organizations, the purpose of which was to identify gaps in medical education surrounding patient safety.^{Footnote ix}  To date, the Competencies have been endorsed by a number of professional and health organizations, including the Canadian Nurses Association, the Royal College of Physicians and Surgeons of Canada, the Manitoba Institute for Public Safety (MIPS), the College of Licensed Practical Nurses of Alberta, the Canadian Physiotherapy Association, and the Canadian Association of Schools of Nursing (CASN).^{Footnote 31}

One of the most important outcomes of the mappings has been the integration of the Safety Competencies into the CanMEDS educational framework of the Royal College of Physicians and Surgeons of Canada, as well as the College of Family Physicians of Canada. As a result, the CanMEDS 2015 Framework will be embedded into the accreditation of all Canadian residency programs for medical and surgical specialties and family medicine, as well as in the training of undergraduate medical students, as part of the accreditation standards set by the Association of Faculties of Medicine of Canada.^{Footnote 32}  In addition, CASN has adopted key elements derived from the Safety Competencies into its accreditation standards for nursing education in Canada.^{Footnote 33}

According to CPSI, a number of health professions schools have used the Safety Competencies Framework as a roadmap for curriculum development in patient safety, and it has anchored the design of faculty development and professional development programs, including PSEP) in Canada. Researchers have reportedly used the Framework to develop assessment tools such as the Health Professionals Education in Patient Safety Survey, and the Framework has served as the foundation for patient safety Objective Structured Clinical Examination scenarios, or simulation scenarios, which determine whether learners can actually demonstrate key patient safety competencies such as disclosing a patient incident or enacting graded assertiveness in unsafe situations.

Additional examples of progress toward the outcome of evidence-informed patient safety curricula are provided below.

• In partnership with CASN, CPSI will be delivering an introductory-level webinar series that provides faculty members and nursing education administrators with information on how to integrate concepts of patient safety into curricula. The course will introduce faculty to the definitions and concepts of patient safety, educational programs, information, resources, and strategies intended to assist in the integration of meaningful learning experiences for students in all years of their nursing program and clinical practice. In addition, an advisory committee has been established with the task of developing guidelines for addressing patient safety in nursing curricula.
• In partnership with the Royal College of Physicians and Surgeons of Canada, CPSI offers the Advancing Safety for Patients in Residency Education (ASPIRE) program to medical educators and residents. ASPIRE focusses on incorporating patient safety, quality improvement, and resource stewardship content into residency training and faculty development; developing skills to lead educational change and integrate patient safety, quality improvement, and resource stewardship training in residency programs; and creating an educational plan for patient safety, quality improvement, and resource stewardship that can be implemented within programs or organizations.
• The Canadian Patient Safety Officer Course (CPSOC), in partnership with the former Accreditation Canada International, has been customized for the international community. The two-day education program employs various teaching methodologies while adhering to adult learning principles and is delivered by peer trainers. Participants of the Patient Safety Officer Course (PSOC) International education program are typically healthcare professionals who have a responsibility for disseminating patient safety principles and programs throughout their organization, as well as clinical educators who can develop and implement improvements, and have the capacity to provide training within their organization. These participants are given education materials that include a binder, as well as a USB toolkit with various CPSI tools and resources.
• CPSI has worked with the First Nations and Inuit Health Branch (FNIHB) of Health Canada to support the development of policies and procedures for the introduction of new incident reporting structures and a dashboard for monitoring patient safety incidents. CPSI is in discussions with the FNIHB Patient Safety Working Group to assist in the delivery of educational offerings in 2017.
• Key informants representing a variety of organizations and jurisdictions reported having accessed PSEP training, or being master PSEP facilitators which, within the PSEP HUB delivery model, allows them to provide PSEP training within their jurisdictions.
• The University of Toronto’s Centre for Quality Improvement and Patient Safety developed and introduced a Master of Science degree in Quality Improvement and Patient Safety, which key informants reported draws on CPSI resources.

CPSI, together with a variety of partners, continues to work toward the goal of embedding patient safety within curricula. In order to accelerate a consistent approach to patient safety and quality improvement education, CPSI hosted, in January 2015, a Patient Safety Education Roundtable that included 54 participants representing academic, accrediting, certifying and regulatory bodies, as well as provincial and national organizations interested in the education of health professionals. As part of the IPSAP, the Patient Safety Education Action Plan resulting from the roundtable includes goals that aim to strengthen faculty development on how to teach key concepts around partnering with patients and embed them into curricula, provides a Patient Safety and Quality Improvement content framework that enables educators to select from a menu of topics necessary for competency-based curricula, and identifies key educational elements required for senior leaders to purposefully influence a culture of patient safety.

Another primary goal of the roundtable was to lay the foundation for a National Patient Safety Education Network. In February 2016, CPSI hosted a second Patient Safety Education Roundtable to establish consensus on the future vision, purpose, and development of the proposed network. An Advisory Group composed of Roundtable participants has subsequently been convened to continue to connect the community of patient safety educators across the country and to develop a mechanism for knowledge translation and exchange of patient safety and quality improvement content, resources, and tools to inform health professional curricula.

Patient safety awareness and knowledge

Although the evaluation did not directly test stakeholder awareness and knowledge of patient safety issues, participants in the evaluation agreed that CPSI’s work has produced important gains in patient safety awareness and knowledge among stakeholders.

In the immediate term, CPSI’s work is expected to lead to increased knowledge and awareness of patient safety issues within the Canadian healthcare system. Since it is typically not feasible during the course of an evaluation to directly measure changes in stakeholder awareness and knowledge (i.e., by administering an objective test of awareness and knowledge at two or more points in time), proxy or indirect measures of change must be examined.

One way of indirectly measuring this outcome is by examining the reach of CPSI products. As shown in Table 6, performance measurement data maintained by CPSI show that its products, learning sessions and events were reaching stakeholders throughout the period covered by the evaluation. While none of the indicators show an increase, and others show variability or declines, a variety of possible explanations may account for these patterns. For example, CPSI introduced a charge in 2014–2015 for Canadian Patient Safety Week (CPSW) packages, which likely impacted the number of registrations the following year. It should be noted that following website architecture upgrades, an improved and more conservative reporting methodology for website downloads came into use in 2015–2016. Due to a lack of comparability, data pertaining to downloads of CPSI research reports and other resources are presented for 2015–2016 only.

Although output measures such as those described above are helpful because they provide evidence of CPSI’s reach to its intended audiences, more direct measures are arguably of higher importance when assessing impact on awareness and knowledge. Some direct measures of impact on awareness and knowledge are tracked by CPSI through its performance measurement framework. In particular, tracking data show that between 2013–2014 and 2015–2016, participants in CPSI learning modules reported an average 12% increase between pre- and post-assessments of knowledge following the completion of the modules. In that same time frame, 91% of participants per year reported that learning objectives were met over the course of the learning modules.
For the current evaluation, the survey of stakeholders is the primary source of direct evidence of the extent to which CPSI’s work has affected patient safety awareness and knowledge. Self-reported data from the survey show that respondents had varying levels of familiarity with a range of CPSI programs, resources, and events prior to completing the survey, as shown in Table 7.

• Of CPSI’s major programs and resources, a large majority (88%) were aware of SHN prior to the survey, with considerably fewer aware of the Canadian Incident Analysis Framework (51%) and the Global Patient Safety Alerts (46%).
• Pre-survey awareness of CPSI’s education programs was quite high for the Hand Hygiene Education Online Module (70%) and the CPSOC (66%), but somewhat lower for the PSEP (56%), as well as Effective Governance for Quality and Patient Safety (EGQPS) program and the Incident Analysis Learning Program (both 40%). Relatively few respondents (20%) were aware of the ASPIRE program prior to the survey, which is probably not surprising given its narrower target audience. Individuals from organizations that do not provide direct patient care were more aware of several of CPSI’s education programs, including the CPSOC (77% v. 64%), EGQPS (52% v. 37%), and ASPIRE (32% v. 17%).
• Pre-survey awareness of certain CPSI events was high, with 94% of respondents aware of CPSW and 84% aware of STOP! Clean Your Hands Day. About two-thirds were aware of Canada’s Virtual Forum and Falls Prevention Month.
• Generally speaking, survey respondents were least familiar with CPSI’s research reports. One-fifth or fewer were aware of the various reports published in the evaluation period. This may reflect CPSI’s deliberate decision to step back from its research activities after 2013 and the perceived relevance of its research activities to survey respondents, as well as other factors.

Most importantly, many respondents who have used or implemented specific CPSI programs or resources, participated in CPSI education programs, or read CPSI research reports, reported that doing so has led to increased awareness and knowledge of patient safety issues, either their own or within their organization. The proportion that reported increased awareness and knowledge varied considerably by program or resource.

• Among the more than two-thirds of respondents (69%, n=402) that have used or implemented SHN interventions within their own practice or organizations, 89% (n=356) reported that the use of these interventions has led to increased awareness and knowledge of patient safety issues within their organization.
• Similarly, among the one-third of respondents (35%, n=208) that have used or implemented the Canadian Incident Analysis Framework within their own practice or organization, 79% (n=164) reported that its use has led to increased awareness and knowledge.
• Although the survey did not specifically ask about the impact of Global Patient Safety Alerts on awareness and knowledge, the majority of respondents who indicated that the Alerts were directly applicable to their organization (n=214) reported reading them (91%), disseminating them to others (76%) and using them to identify potential patient safety issues within their organizations (79%). This suggests that the Alerts are indeed having an impact on awareness and knowledge of patient safety issues within respondents’ organizations.

Likewise, as shown in Table 8, participants in various CPSI education programs reported increases in awareness and knowledge, most notably as a result of participating in the Hand Hygiene Module and the Incident Analysis Learning Program (69% and 63%, respectively, reported moderate or substantial improvement in awareness and knowledge). Reported increases in awareness and knowledge were somewhat lower for CPSI’s other educational offerings, although it is important to note that in all of these cases, large proportions of respondents indicated that they did not know what the impact on awareness and knowledge was — presumably because others in their organizations, rather than they personally, had participated in these programs.

Between 47% and 75% of those who have read specific CPSI research reports noted moderate or substantial improvement in their own awareness and knowledge of patient safety issues. However, caution should be exercised when interpreting these results, since, in some cases, relatively few respondents had read the reports.

Virtually all key informants believe that CPSI has contributed to increased awareness and knowledge of patient safety within the Canadian healthcare system, and that because of its work, there is far more attention being paid to patient safety now than in the past. Some key informants highlighted specific areas and topics in which they believe awareness and knowledge have particularly increased, such as hand hygiene and medication reconciliation.^{Footnote x}

A resounding yes. People understand the scope of the problem better, they understand tools and tactics to advance in this area.

Numerous key informants reported that they personally, or others within their organization, have participated in CPSI’s education and training programs. Generally speaking, these key informants believe that participating in these programs is effective in increasing patient safety awareness and knowledge.

Our quality and risk managers have participated in the CPSOC at each of the regional health authorities (RHAs). It gave our quality and risk managers a solid background and it puts everyone on a common page. The PSEP program has had an impact across the RHAs. The effective governance gave board members a solid understanding and provided them with the knowledge and skills to ask the questions and make informed decisions at the board level regarding quality and patient safety.

We’ve been able to spread patient safety knowledge within our organization [as a result of participating in CPSI courses].

Some key informants observed that CPSI has been one of several influences on patient safety awareness and knowledge in Canada, or argued that improvements in this area are reflective of a general trend that has been occurring on a global scale over the past decade. Some qualified their remarks by observing that although patient safety awareness and knowledge have increased, culture change has been slower to take root. Among survey respondents, more than three-quarters (76%) believe that knowledge and awareness of patient safety issues within the Canadian healthcare system have increased because of CPSI’s work, only 1% of survey respondents disagreed, while 8% were neutral and 15% did not know.

Strengthened system coordination

Through the National Patient Safety Consortium and the IPSAP, CPSI has united a wide range of stakeholders in advancing a common patient safety agenda, while focussing on gaps and avoiding duplication of effort. The upcoming evaluation of the Consortium and IPSAP will shed light on the impact of these efforts.

In the immediate term, CPSI’s work is expected to lead to strengthened system coordination related to patient safety. As already described, CPSI has taken various steps during the evaluation period to strengthen system coordination. Its most prominent initiative in this area has been the formation of the National Patient Safety Consortium and the subsequent development of the IPSAP. In 2013–2014, 51 organizations came together to form the Consortium, and these 51 organizations are the formal participants in the Consortium to the present day. By 2015–2016, a total of 107 organizations, including the 51 Consortium participants, as well as 56 other organizations, had been involved in the Consortium some way (e.g., through attending summits or meetings) and by the same year, a total of 41 organizations had endorsed the National Integrated Patient Safety Strategy.

In addition, as shown in Table 9, as of 2015–2016, CPSI had one collaboration with government (down from five in the previous two years), and 17 collaborations with national and provincial institutions and operational organizations (down from 29 and 39 in 2013–2014 and 2014–2015, respectively).^{Footnote xi} 

Beyond its work with the Consortium and the IPSAP, CPSI met with the senior leadership of various pan-Canadian health organizations to identify collaborative opportunities to move patient safety forward, and in 2015, established an internal strategic partnership analysis and stakeholder map, with the objective of establishing new relationships and building on existing ones.

Among survey respondents, as already reported, just under three-quarters (74%) agreed that CPSI makes a valuable contribution to patient safety by promoting and supporting collaboration and coordination among stakeholders in the field. Furthermore, just over half (51%) agreed that coordination among actors in the field of patient safety has been strengthened because of CPSI’s work. A substantial proportion (26%) did not know, while 21% were neutral, and only 3% disagreed.

Like survey respondents, key informants differed on the question of CPSI’s contribution to strengthening system coordination. Some believe that through CPSI’s efforts to establish the National Patient Safety Consortium and lead the development of the IPSAP, progress has indeed been made in this area. They highlighted that through these efforts, CPSI has managed to unite a wide range of stakeholders in advancing a common patient safety agenda, while taking care to focus on gaps and avoid duplication of effort.

It has strengthened coordination in that it has brought many organizations together across the country and united them in a national set of goals and objectives.

Through the Consortium, they’ve harnessed the activities of a variety of players in a quite decentralized healthcare system in Canada, and got them working together on a variety of priority areas. They’ve got a work plan and appropriate leads for each priority area. They’ve done a lot of work to coordinate the various players to work together on an Integrated Patient Safety Action Plan. […] Duplication was clearly avoided through some of those conversations, and the items left in the work plan were new areas or areas that everyone agreed were gaps.

Others, however, were uncertain or did not know if CPSI’s activities have strengthened system coordination, believe that CPSI’s work has not achieved this outcome, do not see this outcome as part of CPSI’s role, or suggested that the meaning of the outcome itself is unclear. Still others felt they were not in a position to answer the question. As already noted elsewhere, some key informants were concerned that CPSI may be over-extending itself through its many intersecting partnerships and collaborations. A few, though supportive of CPSI’s work on strengthening system coordination, highlighted a need for better linkages with the provinces and territories, since they are responsible for the delivery of healthcare in Canada.

Positive patient safety culture

Over the past decade, legislative changes related to patient safety suggest that Canada has been shifting toward a more positive patient safety culture. While most stakeholders believe patient safety culture has improved in Canadian healthcare because of CPSI’s work, many noted that culture change is a lengthy process, and measurement and attribution are difficult.

In the intermediate term, CPSI activities are expected to engender a positive patient safety culture within the Canadian healthcare system. The importance of cultural change within organizations in bringing about improvements in patient safety has recently been acknowledged by experts in the field. The Institute of Medicine observed that “the biggest challenge to moving toward a safer health system is changing the culture from one of blaming individuals for errors to one in which errors are treated, not as personal failures, but as opportunities to improve the system and prevent harm”.^{Footnote 34}  This shift is at the heart of the current movement toward positive safety culture. Safety culture is the product of values, perceptions, attitudes, and patterns of behavior and is consequently difficult to quantify directly.^{Footnote xii Footnote 35}

Notwithstanding measurement challenges, there is some evidence that Canada is shifting toward a more positive patient safety culture. One indicator is legislative change related to patient safety. For example, data compiled by CPSI show that as of October 2016, nine of 10 provinces and two of three territories have enacted apology legislation since 2006. This legislation prevents apologies from medical professionals being used as expressions of legal culpability and is designed to create an environment more conducive to open dialogue with patients.

As of October 2016, Manitoba, Saskatchewan, Ontario, Quebec, and New Brunswick have legislation requiring mandatory disclosure and reporting of critical incidents, while Nova Scotia has legislation requiring mandatory reporting of serious reportable events. Similar legislation is currently being tabled in Newfoundland, the Yukon, and Northwest Territories.

Furthermore, of the seven provinces with quality and patient safety organizations, five (Alberta, Saskatchewan, Ontario, New Brunswick, and Quebec) also have formal provincial legislation on healthcare quality improvement, which includes patient safety, and all 13 provincial and territorial jurisdictions have some legislation relating to quality assurance. It should be noted that Saskatchewan’s legislation on healthcare quality improvement, as well as its legislation on mandatory disclosure and reporting of patient safety incidents, both predate the establishment of CPSI.

In most cases, the extent to which legislative developments may be attributable to CPSI cannot be determined, although it is reasonable to assume that CPSI has been one of the contributing influences.^{Footnote xiii}  One notable exception is CPSI’s influence on revisions to Ontario’s Quality of Care Information Protection Act (QCIPA), as described in detail below.

In 2016–2017, in order to support performance reporting related to health system safety culture, CPSI began formally monitoring two indicators derived from Accreditation Canada Required Organizational Practices (ROP) compliance data, namely indicators relating to patient safety incident disclosure and patient safety incident management (indicators in boldface font in Table 10 below). Compliance data for these ROPs indicate that compliance is generally high (in the range of 90%) and that small improvements are being realized; three of the five ROPs have trended upwards over time, and a new one was established in 2015.^{Footnote xiv}

Among survey respondents, 64% believe that patient safety culture has improved within the Canadian healthcare system because of CPSI’s work. Another 15% were neutral, 17% did not know and only 3% disagreed. As for key informants, while a minority believe that progress has been made toward a positive patient safety culture in Canada, the majority are either of the opinion that changes in patient safety awareness and knowledge have outstripped changes in patient safety culture, or have indicated that they simply do not know if culture change has occurred. Many observed that culture change within systems and organizations is extremely difficult and can take many years — even decades — to accomplish, and can be highly dependent on management and leadership at the local level.

Culture takes decades to move. In the micro-cultures where there is a leader who has taken a CPSI course or been able to influence the local level, there’s been success.

We’ve accomplished broader awareness, but we still haven’t accomplished behavioral change […] We’re on the early part of the bell curve and not the latter.

That said, CPSI was recognized for its work toward this goal, with several key informants mentioning, in particular, its work in relation to incident analysis and management, patient engagement, and governance, as well as its new focus on patient safety culture in its new initiative, SHIFT to Safety.

CPSI has helped through the development of the incident management model […] People see that we can do that kind of analysis and focus on improvement opportunities and not on individuals.

Health regions have really moved forward with board work on patient safety and quality. They have quality committees, they’ve changed agendas to have the patient voice; that’s directly from governance education. […] Roles and expectations of CEOs have changed [and] patient engagement has been affected […] That definitely has had impact from governance education and also the Consortium work. It’s sometimes hard to do an attribution but you know the influence is there, you can see the work of CPSI reflected in the conversation.

The experience of St. Michael’s Hospital is one example of the way in which CPSI’s work has influenced efforts to improve patient safety.

Some key informants emphasized the need to measure changes in patient safety culture, notwithstanding the associated challenges.

Positive patient safety practices

Canadian healthcare organizations have implemented positive patient safety practices as a result of CPSI activities, with an estimated 88% of eligible acute care facilities and 26% of long-term care facilities in Canada participating in Safer Healthcare Now! (SHN). Many survey respondents reported making changes to organizational practices as a result of SHN, as well as other CPSI resources such as the Canadian Incident Analysis Framework, Global Patient Safety Alerts, and educational offerings and events. Overall, 66% of respondents believe that positive patient safety practices have become more widespread in the Canadian healthcare system because of CPSI’s work.

In the intermediate term, CPSI activities are expected to result in increased implementation of positive patient safety practices within the Canadian healthcare system. One of the most prominent examples of this implementation relates to SHN. CPSI estimates that 88% of eligible acute care facilities in Canada, including mental health facilities, participate in SHN, as do 26% of eligible long-term care facilities.^{Footnote xv}  Furthermore, CPSI data show that as of 2016, 739 organizations with 1,683 distinct teams were enrolled with the program.^{Footnote 37}  CPSI performance data indicate that the proportion of teams that successfully met and maintained their SHN implementation goals within 24 months of first submitting data to Patient Safety Metrics (PSM) grew from 6% to 17% between 2013–2014 and 2015–2016.^{Footnote xvi}  While these proportions may seem low, it is likely that many teams made improvements in patient safety practices without necessarily reaching their goals.

Results from the stakeholder survey provide additional insight into the extent to which SHN interventions have been implemented within the Canadian healthcare system. Survey results show that, not including patients, 69% of respondents have used or implemented at least one SHN intervention, most frequently medication reconciliation, falls, and infection prevention and control. Close to 60% of those who have used SHN interventions have integrated SHN recommendations into written policies, while just under half reported increasing or improving incident tracking or reporting; using SHN information in communicating with clients, patients, and/or the public; and incorporating SHN recommendations or materials into mandatory training. About one-quarter incorporated SHN into optional training and one-fifth have changed practices and procedures, but not written policies (see Table 11).

Beyond SHN, the survey of stakeholders provides evidence of positive patient safety practices as a result of other CPSI programs, products, and interventions. For example, many respondents who have used the Canadian Incident Analysis Framework (n=208) reported that it has prompted changes to practice within their organization. Most often, users of the Framework reported changes in terms of investigating patient incidents (83%), with fewer, but still a majority, reporting changes in terms of developing recommended actions in response to patient safety incidents (69%), reporting patient safety events (69%), and sharing what was learned from patient safety investigations with others within their organization (55%). Half or fewer reported changes in the areas of ongoing monitoring of patient safety risks (49%), communicating with patients about safety events (49%), strategic planning around patient safety at the executive level (45%), involving patients in patient safety reporting and investigations (34%), and sharing what was learned from patient safety investigations with others outside of their organization (24%).

In addition, among those who were aware of Global Patient Safety Alerts and indicated that these were directly applicable to their organization (n=214), 79% reported using the Alerts to identify potential patient safety issues, while 76% reported using them to facilitate or implement changes in policies, practices, processes, or standards. And, as shown in Table 12, many participants in CPSI educational offerings and events likewise reported that they, or their organization, made changes to practice, policies, or procedures as a result of what was learned. For educational offerings, the proportion of participants reporting changes ranged from 63% for the Hand Hygiene Online Education Module and the Incident Analysis Learning Program to 27% for the Advancing Safety for Patients in Residency Education (ASPIRE) program.^{Footnote xvii}  For events, 56% of participants in STOP! Clean Your Hands Day reported changes, with fewer making changes as a result of Falls Prevention Month, Canadian Patient Safety Week (CPSW), and the Virtual Forum.

As already noted, in 2016–2017, CPSI began formally tracking several indicators of positive patient safety practices, based on ROP compliance data gathered by Accreditation Canada. CPSI worked in collaboration with Accreditation Canada to develop these ROPs, which have impacted over 1,000 accredited organizations across 6,000 health delivery sites.^{Footnote 38}  Data currently available from Accreditation Canada show levels of compliance exceeding 90% for three of the requirements, albeit with some variability from year to year and no clear upward trend. For two ROPs, namely those relating to hand hygiene and medication reconciliation, compliance levels have been somewhat lower and also variable over time (see Table 13).

Among survey respondents, 66% believe that positive patient safety practices have become more widespread in the Canadian healthcare system because of CPSI’s work. Another 15% were neutral, 18% did not know and only 1% disagreed. As for key informants, those representing organizations that provide direct care to patients described numerous ways in which their organizations have implemented positive patient safety practices based on CPSI resources and educational offerings, most notably through SHN, but also the Canadian Incident Analysis Framework, the Canadian Disclosure Guidelines, and CPSI’s work in relation to hand hygiene, to name just a few examples.

SHN has certainly made a big contribution. The two original health authorities in the province implemented most of the SHN interventions. The Canadian Incident Analysis Framework is very useful to the province. We’re going to have training on it and a standardized approach to incident management across the province. The Disclosure Guidelines helped us to develop government guidelines, and supported development of policies on disclosure at the regional health authority level as well.

The Safe Surgery Checklist has had major impacts in Alberta and Saskatchewan. Alberta healthcare institutions are being measured in terms of their use of the checklist.

Although many key informants felt they were not in a position to comment on the matter, among those who did there was also consensus that CPSI’s work has contributed to the implementation of positive patient safety practices within the Canadian healthcare system in general, with Accreditation Canada also seen as an important driver in this regard.

Most organizations want to achieve their accreditation status so they must meet the standards and it’s been great to see CPSI and Accreditation Canada move some of the key initiatives forward, so the ROPs are also many of the issues that CPSI has recognized and partnered with them to help move forward and to provide direction to people.

Accreditation Canada is mandated for a large number of organizations. So it’s a forced impact. […] That is really the driving force and what spearheaded our need to be involved in CPSI. CPSI has helped us to ensure that we can meet those standards.

Formal monitoring and reporting on patient safety

There is some evidence of progress in formal monitoring and reporting on patient safety in Canada due in part to CPSI’s work. The collaborative work of CPSI and the Canadian Institute for Health Information (CIHI) to develop the Hospital Harm Measure is seen as an important step forward.

In the immediate term, CPSI’s activities are expected to result in an increase in the formal monitoring of, and reporting on, patient safety in Canada. CPSI itself has been engaged in a number of activities to monitor patient safety. CPSI coordinates Canadian additions to the Global Patient Safety Alerts database, with 81 organizations contributing as of 2015–2016. Beginning in 2005, CPSI operated the Patient Safety Metrics (PSM) system, a central repository for capturing data from participating organizations about compliance with SHN interventions. As part of the transition to SHIFT to Safety, PSM was discontinued effective June 1, 2016. CPSI intends to work with organizations to assist with the migration of data to local storage.^{Footnote xviii Footnote 39}

CPSI has also undertaken a number of quality audits of data collected through PSM for SHN interventions, including venous thromboembolism (VTE), Medication Reconciliation, Hand Hygiene, and Falls Prevention. The most recent of these was a Surgical Site Infection (SSI) audit in collaboration with provincial quality and safety organizations in Alberta, the Atlantic provinces, BC, Ontario, and Saskatchewan in February 2016. The audit was “designed to establish a national perspective on the adoption and implementation of established process measures for preventing SSI”.^{Footnote 40}  The audit revealed that no regions exceeded 36% of cases with a perfect SSI score and two had scores of 0% (i.e., no surgical cases complied fully with all the evidence-based best practices). This effort to measure patient safety behaviors at the implementation level provides valuable and actionable data for safety improvements.

Finally, as already mentioned numerous times throughout this report, CIHI and CPSI have partnered to develop a national patient safety reporting resource called the Hospital Harm Measure. This measure uses existing hospital administrative data and captures hospitalizations involving at least one unintended occurrence of harm that could potentially have been prevented by implementing known evidence-informed practices. The measure is based on 31 clinical groups across four categories of harm: those associated with the delivery of healthcare or medication (e.g., pressure ulcers), healthcare associated infections (e.g., sepsis), patient accidents (e.g., falls), and harm associated with procedures (e.g., laceration or puncture).^{Footnote 41}  It will also provide evidence-based Improvement Resource, linking best practices directly to the measure. The Hospital Harm Measure provides, for the first time, a general measure of patient safety in acute care settings in Canada (excluding Quebec).

In addition to CPSI’s own patient safety monitoring and reporting initiatives, there is some evidence that the formal reporting and monitoring of patient safety has been increasing within the Canadian healthcare system in general. As already noted, four provinces (Manitoba, Saskatchewan, Ontario, and Quebec) have legislation requiring mandatory disclosure and reporting of patient safety issues, and data from Accreditation Canada indicate a high degree of compliance with various ROPs relating to adverse event reporting. While it is challenging to attribute these outcomes directly to CPSI, it is reasonable to assume that CPSI has contributed to their achievement through its activities.

The survey of stakeholders provides more direct evidence of the extent to which CPSI’s work has led to an increase in formal monitoring and reporting on patient safety. Results show that three major CPSI programs and resources — SHN, Global Patient Safety Alerts, and the Canadian Incident Analysis Framework — have prompted individuals and organizations to make changes to their approach to monitoring and reporting on patient safety. For example:

• 48% of respondents who have implemented SHN interventions reported that doing so has led to increased or improved incident tracking or reporting within their organizations;
• 79% of those who indicated Global Patient Safety Alerts are directly relevant to them or their organizations indicated that they use the Alerts to identify potential patient safety issues within their organizations; and
• Among respondents who have used the Canadian Incident Analysis Framework:
  • 83% have made changes with respect to investigating patient safety incidents;
  • 69% have made changes with respect to reporting patient safety events; and
  • 49% have made changes with respect to ongoing monitoring of patient safety risks.

Among survey respondents, 57% agreed that the formal monitoring and reporting of patient safety issues have become more widespread in Canada because of CPSI’s work. Almost one-quarter (23%) did not know, while 17% were neutral and 3% disagreed. Among key informants, some described how their own organization has used CPSI resources to improve its approach to monitoring and reporting on patient safety.

Before CPSI, there was barely any data collected here. CPSI created momentum to talk about patient safety […] and that’s when we said we need data. My hospital started to collect some data, and progressively we’ve been collecting more and more data. So CPSI significantly contributed to this.

Most key informants struggled to comment on the extent to which the formal monitoring and reporting on patient safety issues have become more widespread in Canada as a whole. Some suggested that specific CPSI products have had an impact such as the Global Patient Safety Alerts, the Canadian Incident Analysis Framework, and SHN. Others pointed out that while some advances in formal monitoring and reporting on patient safety have been made, these are probably most directly the result of legislative changes at the provincial or territorial level requiring such reporting, although these changes themselves may be, at least in part, attributable to CPSI’s influence.

CPSI’s collaborative work with CIHI on the Hospital Harm Measure was applauded as a significant step forward in monitoring and measuring advances in patient safety.

The Hospital Harm Indicator will require acute care organizations to collect new data and will allow for aggregated reporting at a provincial and national level that our data systems haven’t been sophisticated enough to do to have a meaningful conversation on. That’s a game changer on the horizon.

Policies, standards, and requirements informed by evidence

The evaluation found several examples of policies, standards, and requirements of professional associations and accreditation bodies informed by patient safety evidence. The Safety Competencies and the Canadian Disclosure Guidelines, in particular, have been widely used and adopted, and more than one-third of Accreditation Canada’s ROPs reference CPSI resources.

In the intermediate term, CPSI’s strategic partners, such as professional associations and accreditation bodies, are expected to introduce policies, standards of practice, and requirements that are informed by patient safety evidence. One important example, as already described, is the endorsement of CPSI’s Safety Competencies by a number of professional and health organizations, including the Canadian Nurses Association, the Royal College of Physicians and Surgeons of Canada, the Manitoba Institute for Patient Safety (MIPS), the College of Licensed Practical Nurses of Alberta, the Canadian Physiotherapy Association, and the Canadian Association of Schools of Nursing (CASN).^{Footnote 42}  Furthermore, the Competencies have been integrated into the CanMEDS 2015 educational framework of the Royal College of Physicians and Surgeons of Canada, as well as the College of Family Physicians of Canada, as a result of which they will be embedded into the accreditation of all Canadian residency programs for medical and surgical specialties and family medicine, as well as in the training of undergraduate medical students. In addition, the CASN has adopted key elements derived from the Safety Competencies into its accreditation standards for nursing education in Canada.^{Footnote 43}

As another example, the Canadian Disclosures Guidelines, which were originally published by CPSI in 2008 and updated in 2011, are “widely used in all of the Canadian provinces and territories to develop policy and approaches to disclosure”.^{Footnote 44}  The College of Physicians and Surgeons of British Colombia (2014) and the Canadian Medical Protective Association (CMPA) (2015) both cite the guidelines, and disclosure is one of the ROPs monitored by Accreditation Canada.^{Footnote 45}  It is important to note, however, that both the CMPA and Accreditation Canada were involved in the working group that developed the Disclosure Guidelines, and that components of the guidelines were derived from the World Health Organization’s (WHO) International Classification for Patient Safety framework.^{Footnote 46}  Thus, through publishing the Guidelines, CPSI has contributed to a process that has had significant impact on policy across Canada.

As a third example, CPSI and Accreditation Canada have collaborated to develop a number of ROPs,^{Footnote xix Footnote 47}  the subsequent implementation of which has impacted the policies and practices of accredited health organizations across Canada. CPSI has held a seat on seven advisory committees to inform ROPs and standards, as well as routinely engaging Accreditation Canada on CPSI working groups and providing ad hoc consultations during both organizations’ product development and product revision phases. Most recently, Accreditation Canada has included Recognition of the Deteriorating Patient, work that CPSI has done in collaboration with Healthcare Insurance Reciprocal of Canada (HIROC), in updates to its critical care and inpatient standards, which were released in January 2017 and come into effect in 2018. CPSI and HIROC are partnering on the development of resources to support organizations in their efforts to comply with the updated standard, to be released through SHIFT to Safety in mid-2017.

At present, just over one-third (36%, or 12 of 33) Accreditation Canada ROPs cite CPSI resources (see Table 14). According to CPSI, this alignment has made it possible to issue joint CPSI/Accreditation Canada reports during the current funding period on clinical topics where optimizing compliance with current ROPs and standards is supported by a roster of CPSI resources. For example, the 2014 report focussed on VTE, surgical safety, and homecare safety, while the 2015 report focussed on falls prevention.

A majority (62%) of respondents to the stakeholder survey believe that policies, standards, and requirements informed by patient safety evidence have become more widespread in the Canadian healthcare system because of CPSI’s work. Another 16% were neutral, 2% disagreed, and 20% did not know. Among key informants, some referred to the examples mentioned above, including the integration of the Safety Competencies into the CanMEDS framework, the widespread uptake of the Disclosure Guidelines, and CPSI’s work with Accreditation Canada on ROPs. Others, in particular those involved in direct healthcare delivery, described changes to policies, standards, and requirements made by their own organizations based on CPSI’s work. However, the examples given tended to be similar to those cited as examples of positive patient safety practices (e.g., changes made as a result of SHN or the Canadian Incident Analysis Framework).

Patient and family involvement in the health system

CPSI has been a pioneer in Canada in advocating for and supporting patient and family involvement in healthcare improvement. The extent of patient and family involvement within the healthcare system, as well as the impact of this involvement for patient safety outcomes, have not been measured or rigorously evaluated.

Although increased patient and family involvement is not a formal expected outcome within CPSI’s logic model, it has been a major focus of the organization’s work since its inception. Indeed, CPSI key informants reported that the organization has been a leader in Canada in patient and family involvement in the health system, and CPSI has certainly led by example by routinely integrating patient and family advisors into its own work. In 2013–2014, 68% of CPSI’s activities included a patient advisor, compared with 100% in 2014–2015 and 94% in 2015–2016.

CPSI has historically supported the volunteer patient group Patients for Patient Safety Canada (PFPSC), which is connected to the WHO’s global network of patient advisors, Patients for Patient Safety,^{Footnote 48}  and which is currently a program of CPSI. In 2015–2016, 37 of PFPSC’s 85 members held the WHO Patient Safety Champion designation, indicating their participation at a WHO-sponsored workshop that provided a specific orientation, where they committed to represent the patient perspective in patient safety improvements at all system levels as expressed in the London Declaration. PFPSC volunteers deliver speaking engagements at various events across Canada and engage with the healthcare system by acting as committee or board members and becoming involved with project development.

An independent evaluation of PFPSC in 2014 reported that the organization had succeeded in meeting its overall goal of championing the patient voice, and had realized some success in advancing the goal of safer healthcare in Canada and elsewhere.^{Footnote 49}  The evaluation concluded that good progress had been made toward increasing new relationships with health system partners, as well as increasing the strength of existing relationships. The evaluation also noted that PFPSC had “made a wide range of impacts in its short history,” citing its reach across the country (in particular with national organizations), its impact on individuals within the healthcare system and PFPSC members themselves, and its impact on CPSI, which has learned to engage more effectively with patients as a result of its connection with PFPSC. Finally, the evaluation noted that one of the most important impacts that PFPSC made was promoting the importance of patient engagement in order to improve patient safety, which resulted in healthcare organizations establishing their own mechanisms for identifying and engaging patients. Those interviewed as part of the evaluation generally believed that the PFPSC was successful at impacting patient safety, while acknowledging the difficulty in attributing change directly to its activities. The overall picture was of incremental success in involving the patient voice in healthcare and in fostering safer healthcare.

Most recently, CPSI’s new initiative, SHIFT to Safety, has an entire stream of resources devoted to patients and the public, and the organization is currently working on a Patient Engagement Guide. The Guide, launched in May 2017, is a comprehensive resource co-designed with patients and targeted to patients and their families and providers/organizations. It is based on evidence and leading practices, and aligned with, and complementary to, existing resources. The Guide aims to support effective partnerships with patients and their families, in order to accelerate patient safety in all settings and at all system levels. The topics covered include patient/family engagement in incident management and in improving patient safety and quality, as well as developing or sustaining structures and processes to support patient/family engagement. Originating from the Integrated Patient Safety Action Plan (IPSAP), the guide was developed with leadership and financial contributions from CPSI, the Atlantic Health Patient Safety and Quality Collaborative, and HQO, as well as support from an Action Team representing 15 organizations.

Among the 586 non-patient respondents to the stakeholder survey conducted as part of the current evaluation, a sizeable minority (38%) were aware of PFPSC prior to participating in the survey, and a similar proportion (36%) had accessed CPSI resources relating to patient involvement. Almost three-quarters (74%) of those who have accessed these resources have taken steps to involve patients and family members in activities intended to increase the quality and safety of patient care. Most frequently, these steps involved including patient advisors or representatives on committees, boards, or working groups (71%); soliciting input from patients into the assessment, design, development, or implementation of programs, processes, policies, strategies, or resources (68%); and inviting patients to share their experiences with staff, executives, board, or other patients (64%). However, it was considerably less common for respondents to have included patients on decision-making bodies responsible for direction setting and resource allocation (34%).

Respondents who had involved patients and family members described a number of positive impacts for their organization, including increased awareness, understanding, and knowledge of patient needs; changed organizational culture or thinking; and implementation of a specific process or organizational change. Other, less frequently mentioned, benefits include improved patient care, increased patient trust or confidence in the system, and improved transparency and communication.

Among all survey respondents, just under half (49%) agreed that patient and family involvement in the healthcare system has increased in Canada because of CPSI’s work. That said, only 3% disagreed, while substantial proportions were neutral (22%) or simply did not know (26%). As for key informants, many acknowledged that CPSI has been a role model and pioneer in Canada in advocating for and promoting patient and family involvement in the healthcare system, building on parallel initiatives undertaken by the WHO.

They were a notable leader in the country doing that. It takes leadership in a bunch of different ways to get that train started. […] They were certainly at the head of that curve.

We have come a long way in engaging staff at all levels of healthcare and across the spectrum around the importance of listening to patients and engaging patients. CPSI has been absolutely instrumental in moving that forward.

Some key informants described various ways in which CPSI’s work has influenced their own organizations to engage patients and family members, though not necessarily without challenges. It was observed that many boards now follow CPSI in beginning all meetings with a patient story or video, which several key informants observed is a powerful and compelling way of affecting the perspectives of those in attendance. The experience of St. Michael’s Hospital (SMH) provides an illustration of the impact that involvement with PFPSC and CPSI can have on organizations’ efforts to engage with patients and family.

Key informants also observed that increased emphasis on patient and family involvement has been a general trend in healthcare over the past decade, and thus, changes in this area cannot be fully attributed to CPSI.

There is more sophisticated awareness regarding patient involvement, but I can’t attribute it to CPSI. It could be a more general shift. This increasing role for patients is seen in all OECD countries.

I don’t think you can fully attribute growth in this to CPSI, but certainly some of it is due to CPSI’s work. CPSI was instrumental in the early days in its work with Patients for Patient Safety Canada and the Halifax Symposium. You could not help but be influenced by that work.

A few key informants noted that Accreditation Canada has incorporated the principles of patient engagement and client-centred care into its ROPs, and that, moving forward, these are likely to be the primary drivers of change in this area as healthcare organizations strive to comply with the standards in order to achieve or maintain their accreditation.^{Footnote xx}. However, it was also noted that CPSI is performing an important role by developing resource materials that will help organizations implement the standards.

A small number of key informants observed that there are no objective data or rigorous evaluations to support conclusions on the extent to which patient and family engagement has increased, nor any data on the impact that engagement may have had on patient safety outcomes. It was recommended that CPSI undertake research, such as a survey of patients (e.g., baseline with subsequent follow-ups), in order to address this gap.

Improved patient safety in Canada

Although there is no objective evidence that patient safety in Canada has improved since CPSI was established, most stakeholders believe that CPSI’s activities have contributed to improved patient safety in Canada.

In the long term, CPSI’s activities are expected to lead to improved patient safety in Canada. As was described in detail in section 4.1 of this report, there is no objective evidence that patient safety has improved appreciably in the 13 years since CPSI was established. Nevertheless, a majority of survey respondents (70%) agreed that CPSI has improved patient safety for Canadians, while 18% did not know, 11% were neutral, and only 2% disagreed. Moreover, many survey respondents who have used specific CPSI programs and resources reported that doing so has had a positive impact on patient safety within their organizations (see Table 15).

Two-thirds (67%) of those who have used the Canadian Incident Analysis Framework reported that it had either a substantial or moderate impact on patient safety within their organizations, as did 58% of those who have implemented SHN and 35% of those who have used Global Patient Safety Alerts. Respondents in organizations that provide direct patient care believed that both SHN (U=5496, p=.012) and Global Patient Safety Alerts (U=545.5, p=.0007) had led to more improvements in patient safety than those respondents in organizations that do not provide direct care.

Over 70% of those who have made organizational changes as a result of CPSI educational offerings reported either a substantial or moderate impact for patient safety within their organizations. The only exception was ASPIRE, although caution should be used due to the small sample size.

Virtually all key informants believe that CPSI has contributed to improved patient safety in Canada, although most acknowledged a lack of evidence to support their view. Several pointed out that, although patient safety has not improved since CPSI was established, it would be a mistake to conclude on this basis that CPSI has had no impact. On the contrary, it is possible that without CPSI, patient safety in Canada might have lost ground rather than remaining stable; particularly given a context of continuous technological and medical innovation, along with the increasing complexity of, and growing financial pressures on, the healthcare system.

Arguably, it could have been worse without CPSI. Health systems are under extraordinary financial pressure. We could speculate that if they'd not been doing their great work, we might have actually lost ground rather than staying the same.

4.5 Performance: Issue #5 – Demonstration of Economy and Efficiency

The Treasury Board of Canada's Policy on Results (2016) and guidance document entitled Assessing Program Resource Utilization When Evaluating Federal Programs (2013), define the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs, and expected results.

Observations on Economy

CPSI has operated in an economical manner over the three years covered by the evaluation. It has taken steps to minimize the cost of inputs by eliminating one office and downsizing another, increasing the number of staff working virtually, and managing staff remuneration, among other measures. CPSI could continue to explore cost recovery and other revenue-raising potential from sources other than Health Canada, and could articulate a comprehensive pricing model for its products and services.

As previously seen in section 2.4, CPSI's agreement with Health Canada allows it to carry forward up to 10% of any unused Government of Canada contributions in a given year to the following year, until the end of the Contribution Agreement. In addition, CPSI is entitled to retain surpluses from revenues raised from sources other than the Government of Canada contribution, provided that Government of Canada funds have not been used to pay for directly related expenses. As a result, CPSI's use of government funds has varied from the annual $7.6 million contribution due to the use of the carry forward provision, and it has also run small surpluses in some years by retaining surpluses arising from other revenue sources.

The accounting statement net income is adjusted to the basis on which CPSI reports to Health Canada. Under the Contribution Agreement, CPSI reports on a capital expenditure basis, so it backs out depreciation and includes capital expenditure in its reporting to Health Canada. The result is a difference in timing between the bottom line result shown in the accounting statements as compared to what is reported to Health Canada.

Several points and trends are worth noting in the financial information above. First, Health Canada's contribution constitutes a significant proportion of CPSI revenues: 97% in 2015–2016. Annual Health Canada funding declined from $8.00 million in 2012–2013 to $7.76 million in 2013–2014, and to $7.60 million for the two subsequent years (i.e., 2014–2015 to 2015–2016). It is expected to remain at this level for the remainder of the current funding agreement.^{Footnote 50}

Secondly, CPSI's revenues from other sources are highly variable and declining. Although revenues from sponsorships and registration are not insubstantial, amounting to $792,764 in 2013–2014, $578,013 in 2014–2015, and $190,444 in 2015–2016, they do appear to be highly variable.^{Footnote xxi}  This likely limits their value as a source of predictable funds to support CPSI's operations. CPSI staff indicate that, though there may be untapped sources of potential sponsorships, the long-term decline is impacted by broader structural and environmental factors, including the economic climate, fiscal retrenchment among governments, and the end of the SHN campaign mode and the integration of its regional nodes within CPSI. The development of an updated approach to raising other revenues, including sponsorships, is being considered alongside the development of a new strategic plan for the next funding period, as revenue-raising opportunities will depend highly on the particular strategies and lines of business emphasized in CPSI's next funding period.

Finally, as Government of Canada contributions have been held constant or reduced, and other revenues have declined, the difference has been made up by reductions in total expenses. Program expenses accounted for between one-quarter and one-third of all CPSI expenditures over the three years under consideration. Program expenses increased from 2013–2014 to 2015–2016; given that total revenues fell over the same period, this was made possible by a decline in operational expenses. CPSI's operational expenses consist largely of salary wages and benefits and other operating costs, as well as professional services, capital depreciation, costs associated with the Board of Directors, and travel and meetings. Staff compensation accounts for nearly two-thirds of all operating expenses.

Operating costs fell significantly over the period. Part of this decline was due to an internal accounting change that moved travel costs from the corporate budget to program budgets in order to encourage greater accountability for program-related staff travel. However, even after adjusting for this change, operation costs fell by $680,526 over the period. The biggest contributors included a 46% drop in office space and supply costs, a 49% decline in professional service costs, a 31% drop in corporate travel costs, and a 4% decline in salary wage and benefit costs. The two biggest contributors in absolute terms were the $319,279 decline in office space and supply costs, and the $147,714 decline in salary costs over the period.

CPSI Board members and staff, as well as Health Canada representatives, reported that several measures have been carried out over the evaluation period to minimize the cost of inputs:

• CPSI eliminated its Toronto office and downsized its Edmonton office; it currently maintains offices in Ottawa, where it shares space with other national organizations, and in Edmonton. Although CPSI has contemplated closing the Edmonton office altogether, CPSI key informants reported that the potential cost savings associated with closing this office would be relatively small compared to the benefits of maintaining it (including the ability to retain talented staff).
• CPSI has increased the number of staff working virtually (approximately eight or nine at the present time), which has reduced the need for office space, despite increasing the required investment in virtual infrastructure.
• Steps have been taken to manage remuneration for staff, including executives. CPSI staff indicated that salary reviews are undertaken every two to three years by an external evaluation consultant to ensure that compensation is in line with comparable private, non-profit, and public sector positions. The filling of several vacant positions was deferred, given CPSI's fiscal constraints. The senior management team was downsized by one position, and Board members further reported that they had consciously sought to limit the size of the retainer for the new CEO, as well as to streamline the organization's management structure.
• CPSI has moved to electronic distribution of Board and stakeholder resources to save on office supply costs, and has taken steps to control benefit costs and economize on travel.

A number of Board members and staff asserted that CPSI has already taken advantage of the most attractive opportunities to economize on its use of resources in carrying out its work. That said, CPSI has explored opportunities to finance its activities from non-Health Canada sources over the evaluation period. For instance, CPSI is examining the potential for cost recovery in relation to international stakeholders and clients. At present, some established educational programs delivered with partners, such as the Canadian Patient Safety Officer Course (CPSOC), are priced to recover all direct non-staffing costs, plus generate a small surplus to offset direct staffing costs.

Other courses, such as Patient Safety Education Program (PSEP)-Canada, are delivered primarily to recover direct non-staffing costs only, either through charges to the delivering partner or through direct participant registrations. Program development costs for the education programs are generally not recovered.

With respect to CPSI's other products, in some cases CPSI charges a nominal cost to partially offset delivery costs, such as in the case of the SHN learning series delivered in prior years. More recently, CPSI piloted a user fee for registration in Canadian Patient Safety Week (CPSW), but found that the fee proved a significant barrier to broad participation. In 2016, CPSW returned to a free pricing model, which is the approach used for most CPSI products and services that are intended to broadly be made available to the entire healthcare field, including all of its tools, resources, and research papers.

To some extent, the range of pricing reflected above, from cost recovery to free distribution, reflects the differentiation of products between those that are provided and customized for individual client benefit, versus those that are intended to be public resources; however, a pricing model has not been comprehensively enumerated or applied across CPSI.

CPSI representatives were somewhat ambivalent about expanding the scale or scope of cost recovery in relation to CPSI products and services. Some Board members argued that the potential benefits from cost recovery (i.e., revenue generation) must be balanced against the risk that this could cause other stakeholders to question the organization's commitment to patient safety. Both Board members and external key informants also argued that the current accessibility of CPSI products and services has significantly affected, in a positive manner, the perceived value-for-money proposition associated with their use and application. That said, CPSI could continue to explore revenue-raising potential from sources other than Health Canada, including cost recovery, and could articulate a comprehensive pricing model for its products and services.

Observations on Efficiency

CPSI has taken steps to optimize the quantity and quality of its outputs, including use of web-based technology for information dissemination and stakeholder engagement, and leveraging of in-kind contributions from stakeholders and external experts. The latter have been critical to CPSI's ability to undertake its activities and produce high-quality products and services. Some external stakeholders are concerned that CPSI is overextended, given its size, and recommended it focus on fewer, well-defined priorities in order to achieve greater impact. Other key suggestions included more strategic consideration to partnerships and collaborations, an emphasis on defining future directions in patient safety, and greater attention to measurement and evaluation.

CPSI representatives indicated that organizational resources have been allocated in ways designed to maximize the quantity and quality of the organization's outputs. For example, CPSI has used web-based technology to disseminate information (e.g., through improvements to CPSI's website, as well as expansion into social media) and to engage stakeholders (e.g., Canada's Virtual Forum on Patient Safety and Quality Improvement).

CPSI has also leveraged substantial participation and in-kind contributions from stakeholder organizations and external experts to work toward common objectives. For example, the work of the National Patient Safety Consortium is a coalition of the willing, relying heavily upon involvement from and engagement with dozens of stakeholders across Canada. CPSI has also leveraged provincial and local organizations to deliver education and training programs in particular jurisdictions while maintaining control over course content (the "hub and spoke" model), and routinely engages external experts in the design and review of its products and delivery of its educational offerings. As a final example, Patients for Patient Safety Canada (PFPSC) members are volunteers rather than paid staff.

While in-kind contributions from stakeholders appear to be quite significant — and probably even critical — to CPSI's ability to undertake a variety of its activities, the organization has not, to date, made any attempt to quantify these contributions. Tracking the amount of time contributed by stakeholders would enhance CPSI's ability to demonstrate the extent to which it has successfully strengthened coordination within the system, as well as its ability to demonstrate that it is operating in an efficient manner to produce high-quality outputs.

CPSI representatives did not identify additional opportunities to further increase the volume or quality of outputs produced, given the organization's available resources. Indeed, they suggested that the organization is already producing a large volume of outputs relative to its size and budget; a view that is also shared by some external key informants.

External key informants identified a number of opportunities to improve the achievement of CPSI's organizational outcomes. Four main recommendations emerged: 1) choose depth over breadth; 2) give more strategic consideration to partnerships; 3) focus on defining future directions for patient safety in Canada; and 4) emphasize measurement and evaluation.

Choose depth over breadth

Some key informants argued that CPSI should focus its efforts and limited resources on a small number of well-defined issues or priority areas, cautioning that the organization may be overextended at the present time, or may simply lack the requisite expertise or resources to generate measurable impact in any particular area.

The current four areas of focus are large buckets. My most important message is that they need to be focussed in their efforts if they want to have an impact. More intense focus on fewer areas.

There's patient safety, patient-centred care, and quality of patient care, and they are not the same thing. CPSI needs to more carefully articulate their part of that space. They can't do everything, especially with the resources that they've got. Dabbling is dangerous. Pick an area and do a really good job of that.

They have to be choosy in what they decide to do. Make a strategic decision on areas to focus on improving, or decide to work on understanding why we are so bad in certain areas. Tackle based on where the most harm is being done. Healthcare is a $220 billion operation — we are talking about CPSI tackling a huge issue with $8 million per year. You have to be pretty targeted to be able to see something for your efforts.

Notably, the perception that CPSI may be stretched too thin exists among some stakeholders, despite the organization's efforts to focus and rationalize its scope of activity in response to the findings and recommendations of the 2012 evaluation.

However, many external interviewees identified issues that have not, to date, received significant attention from CPSI, and that could be the focus of its activities in the future. These included, but were by no means limited to, public health (e.g., antimicrobial stewardship), primary care, palliative care, community support services (programs that provide assisted living, supportive housing, attended care, transportation, nutrition, hospital-to-home services, and friendly visiting services), and patient safety in relation to Indigenous populations.^{Footnote xxii}

Overall, few obvious common themes emerged, and stakeholders' suggestions for future areas of focus in many cases aligned with their own areas of work and interest. Although most external stakeholders did not perceive that CPSI is currently active in any areas in which it should not be engaged, a few questioned the appropriateness of, or expressed surprise to learn about, CPSI's involvement in certain areas, such as home care and mental health. Again, there was no obvious consistency in these responses. The diversity of perspectives that stakeholders offered regarding the appropriate future focus for CPSI's activities likely reflects the size and scope of the patient safety field in Canada.

In this context, it is important to note that key informants were asked general, open-ended questions about priorities, and that neither they nor survey respondents were asked to rank, prioritize, or choose from among possible areas of focus for CPSI. Nor were they asked if, in general, they supported a more restricted or an expanded role or scope of activity for CPSI. The responses to these questions could provide a clearer picture of stakeholder priorities. Such questions could be posed as part of stakeholder consultations undertaken to support a future strategic planning exercise.

More strategic consideration of partnerships

Related to the argument that CPSI should adopt a narrower focus for its work, some key informants argued that CPSI should give more strategic consideration to its partnerships, with a view to prioritizing those which are most relevant and most likely to advance patient safety.

We think CPSI should be more strategic. Rather than gathering up loads of partners, it would be more effective if they have a good understanding of why they are partnering with a certain organization and for what purpose. Usually it's better to have a narrow group of partners that are aligned, working together on something.

In addition, some external interviewees noted that there remain instances in which CPSI is perceived to be duplicating work being carried out by other organizations. The most frequently mentioned example was infection prevention and control,^{Footnote xxiii}  as well as patient experience mapping and standardized case definitions. These key informants argued that CPSI should improve coordination with stakeholders, so that it can work with or rely on stakeholders without alienating them.

When other partners are already in that space, they should respect that space. They don't have to lead on everything.

Define future directions

Some external stakeholders argued that CPSI should attempt to set the overall strategic direction for patient safety in healthcare in Canada, as embodied by the following assertions:

CPSI [needs] to be slightly ahead of our time — they need to push us from where we're comfortable to where we need to be in a few years.

CPSI has the opportunity to know where we should invest our time and energy two to five years from now. So it is actually setting the stage for improvement with regard to safety. There is a role for a national organization, CPSI in particular, to help define future directions.

To date, they've been responding to perceived or observed needs in healthcare rather than driving some of the advance of the field. They operate more on an enabling level. They develop tools and education and documents that people use, but I don't know that they are setting a tone or strategy that is needed in a national organization in patient safety. I would challenge them to be more of a strategic player, and to say, "Here are the top two or three priorities," and move people in that direction.

What Canada needs is a national vision for quality and safety that each of the provinces can endorse and move forward on. CPSI has not been able to lead this effort. They have been more successful with safety 101 education. They have basic expertise but we need a next generation of patient safety experts to make strong system changes that are required to foster better quality and safety.

Emphasize measurement and evaluation

Finally, some key informants suggested that CPSI should focus on improved measurement and evaluation of patient harm, patient safety interventions, and safety culture within healthcare organizations, with the explicit aim of articulating the impact of patient safety interventions, including its own.

We have never had a good impact analysis of any of these programs individually or collectively and I think that's one of the key things that's missing here — an evaluation.

CPSI should be paying more attention to evaluation, measuring outcomes, as a result of getting out into the marketplace with their different products. [So] they and the rest of us can learn about where we are getting the best outcomes and how their interventions lead to better outcomes.

There's not a vacuum in tools or training, there's a vacuum in measuring harm and knowing how safe people are. We could do more to really measure safety culture directly and to measure harm directly.

The whole idea of measurement is something that CPSI could have a big impact on if they were to champion that.

One key informant linked the concepts of greater strategic focus, strategic partnerships, and measurement by noting the following:

The more tightly you define what you are working on, the easier it is to develop partnerships and the approach to measurement.

As already noted elsewhere in this report, a majority of interviewees, as well as survey respondents, believe there continues to be a need for a dedicated national patient safety organization, even if, as the discussion above makes clear, there are varying perspectives on precisely what its strategic priorities should be.

Value for money

External stakeholders perceive CPSI's activities to have generated considerable value-for-money for their own organization and for others. Quantitative value-for-money analysis focussing on CPSI's medication reconciliation activities suggests that these have generated value-for-money by producing cost savings well in excess of the expenses required to sustain them. Implementation of medication reconciliation in one acute care facility has generated positive net benefits by averting the loss of patient welfare that accompanies preventable adverse drug events.

To the extent that CPSI's activities have accelerated the uptake of medication reconciliation in Canadian healthcare and ensured implementation in accordance with current best practice, CPSI is likely to have generated additional value-for-money for its stakeholders in terms of improved patient well-being. Assuming that all SHN interventions are designed in accordance with current best practices, it seems reasonable to assume the same is true of CPSI's activities in relation to SHN interventions more generally.

Generally speaking, external interviewees perceived value-for-money for their own organizations and other stakeholders from CPSI activities. Although key informants commonly remarked upon the challenges involved in quantifying the value generated by CPSI activities, some explicitly reasoned that the financial benefits associated with these activities, in terms of reduced patient harm (e.g., reduced length of stay), likely outweigh the costs of participation, particularly as many of CPSI's products and services require no financial outlay. The value-for-money of CPSI's products and services for smaller jurisdictions was specifically highlighted.

There just aren't the resources within the province to do some of these things. They either don't get done at all or they don't get done very well. So it's a valuable organization and money very well spent.

Specific examples of CPSI activities perceived to have generated value-for-money for external stakeholders include the following:

• Training and education: Among those individuals who referred to CPSI's activities related to training and education, there appears to be a strong sense that these activities provide value-for-money in the sense that the financial costs required to provide these services, or access them from alternate suppliers, would have been considerably higher.
• Tools and resources: CPSI tools and resources (e.g., SHN, Disclosure Guidelines) are highly regarded for both their quality and their accessibility.
• Miscellaneous: A few respondents highlighted other CPSI activities they perceived to have generated value-for-money over the evaluation period. These included CPSI's role in raising awareness of patient safety issues, in convening healthcare stakeholders to discuss and address these issues, and in drawing attention to the patients' experiences with, and perspectives on, safety incidents.

A few key informants asserted that although CPSI activities had generated substantial value-for-money in the past, this could become more difficult in the future, since much of CPSI's early work is now being undertaken by other organizations, or could be undertaken by these organizations if CPSI did not exist.

To supplement stakeholders' qualitative perceptions of the value-for-money associated with CPSI activities, the evaluation carried out a quantitative value-for-money analysis. One aspect of the analysis focussed specifically on examining the value generated by CPSI's medication reconciliation activities over the evaluation period, as they pertain to acute care facilities in Canada.^{Footnote xxiv}  This analysis conservatively assumed that:

• cost savings that acute care facilities realize due to the availability of CPSI's medication reconciliation tools, resources, and supports constitute the sole source of value stemming from CPSI's activities in this area;
• if CPSI had not provided these tools, resources, and supports, Canadian acute care facilities would have acquired some of them from alternative sources [such as the Institute for Healthcare Innovation (IHI) in the US],^{Footnote xxv}  but would have incurred additional costs (i.e., above and beyond those associated with the use of CPSI tools, resources, and supports) to preserve applicability and relevance to Canadian healthcare providers; and
• implementation of medication reconciliation is discontinued in a proportion of acute care facilities each year.

Based on these assumptions, the baseline value-for-money analysis determined that the present value of the benefits (i.e., cost savings) associated with CPSI's medication reconciliation activities over the period from 2013–2014 to 2015–2016 was approximately $8.1 million, while the present value of costs was about $0.3 million. This implies a net present value of $7.8 million for the evaluation period.

It is important to acknowledge that focussing on the cost savings attributable to the use of CPSI tools, resources, and supports arguably reflects a very narrow conceptualization of the value generated by CPSI's medication reconciliation activities in the context of acute care (as well as its activities more generally).^{Footnote xxvi}For instance, the discussion of the results of the key informant interviews above underscores the belief among certain stakeholders that in the absence of CPSI activities, some organizations may not have implemented certain patient safety interventions, or may have done so less effectively than they currently do. This, in turn, implies that these activities may have generated additional value for stakeholders by accelerating the uptake of patient safety measures in Canadian healthcare and ensuring implementation is carried out in accordance with current best practices. The value-for-money analysis does not account for either of these possibilities.

In addition to examining the value-for-money of CPSI's activities relating to medication reconciliation, the evaluation also included a value-for-money analysis of implementing medication reconciliation at Edmundston Regional Hospital (ERH), a medium-sized acute care facility located in northwestern New Brunswick. This analysis examined the impact of the interventions undertaken by ERH on the facility's costs, as well as on the incidence of preventable adverse drug events (pADEs) and, by extension, on the well-being of its patients.

For the purpose of this analysis, patient welfare was expressed in terms of quality-adjusted life years (QALYs), a construct commonly used in economic evaluations of healthcare programs that summarizes the impact of interventions upon both the length and quality of patients' lives.^{Footnote 51}  Implicit in the use of QALYs in this and many other economic evaluations is the notion that both length and quality of life are factors for which people and decision-makers are willing to sacrifice other things, which is to say that they are valued. Furthermore, such valuations are typically substantial: in a Canadian context, for example, thresholds ranging from $20,000/QALY to $100,000/QALY have been proposed.^{Footnote 52}

The value-for-money analysis found that implementing medication reconciliation at ERH is expected to reduce the number of pADEs occurring at the facility by an average of about 26 each year. Most of these incidents (~80%) would not have resulted in lasting harm to patients. However, a small number may have been life-threatening or fatal, or may have resulted in permanent injury. Despite the fact that severe, life-threatening, or fatal pADEs are relatively uncommon, they account for the vast majority of QALYs lost as a result of medication errors.

Medication reconciliation at ERH does not appear to be cost-saving. That is, from a financial standpoint, the costs of implementation to ERH ($151,770 in one-time costs, $179,829 in annual costs) are expected to exceed any savings it may enjoy due to reduced incidence of pADEs (estimated to be about $79,180 each year). However, upon accounting for the impact of pADEs on patient welfare, the analysis concluded that the present value of net benefits associated with the intervention is approximately $5.5 million for the period from 2007 to 2016, inclusive.

This indicates that although implementing medication reconciliation may not be cost-saving (in at least some settings), it is highly likely to generate value-for-money.^{Footnote xxvii}  It also suggests that to the extent that CPSI's medication reconciliation activities succeeded in accelerating the uptake of medication reconciliation in Canadian healthcare and ensuring that implementation is carried out in accordance with current best practices, CPSI is likely to have generated additional value-for-money for its stakeholders in terms of improved patient well-being. To the degree that all SHN interventions are designed in accordance with current best practice in the field of patient safety, it seems reasonable to assume the same is true of CPSI's activities in relation to SHN interventions more generally. With its recent transition away from SHN, and in light of potential future adjustments to its activities, CPSI will need to find new ways to generate and demonstrate value-for-money to stakeholders.

Observations on the Adequacy and Use of Performance Measurement Data

CPSI has taken steps to improve its approach to performance measurement, with an emphasis on identifying ways to measure its long-term impact on patient safety and enhancing its internal evaluation function. There are opportunities to further clarify and refine the theory or theories of change underlying its activities, and to make corresponding adjustments to the logic model and performance measure framework in order to facilitate performance reporting and demonstration of impact.

CPSI developed its current logic model and performance measurement strategy when it entered its current funding agreement with Health Canada. Within the strategy, the performance measurement framework identifies both output and outcome measures, and, based on a review of internal and external facing performance reporting, CPSI has done a good job of consistently tracking and reporting on these measures over the last three years.

CPSI key informants observed that while the organization has been able to identify appropriate indicators for its short- and medium-term outcomes, it has been considerably more challenging to develop suitable indicators to measure its long-term impact on patient safety. They reported that the organization is currently undertaking several initiatives with a view to enhancing its ability to track and report on the extent to which patient safety has improved over time.

First, CPSI is working with external partners to access systems data in order to help shed light on long-term improvements in patient safety. In particular, CPSI and the Canadian Institute for Health Information (CIHI) have worked collaboratively to develop the Hospital Harm Measure, which is based on existing administrative data that hospitals already collect on a routine basis. As already described, this initiative aims to provide a pan-Canadian framework for measuring patient safety across the entire care spectrum.

In addition, CPSI has incorporated indicators of impact into its approach to performance measurement that draw on Accreditation Canada Required Organizational Practice (ROP) compliance data. In particular, CPSI has formalized reporting of indicators related to its stated outcomes of increasing positive patient safety culture and increasing positive patient safety practices, using ROP compliance data from Accreditation Canada. Patient safety culture is monitored using ROP compliance data relating to patient safety incident disclosure and patient safety incident management, while patient safety practices are monitored using ROP compliance data related to safe surgery checklists, medication reconciliation, home safety risk assessment, hand hygiene compliance, and client safety education. CPSI began formal reporting on these indicators in Q2 2016–2017.

Finally, CPSI is working to enhance its internal program evaluation function and capacity in preparation for the next funding cycle.
While CPSI has made commendable efforts to improve its approach to performance measurement, there are opportunities to further clarify and refine the theory or theories of change underlying its activities, as well as its logic model and performance measure framework, to facilitate performance reporting and demonstration of impact.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

This evaluation confirmed an ongoing need to address patient safety in Canada. Recent reports of progress in patient safety in Canada, the US, and the UK have concluded that their respective systems have not appreciably moved the mark in patient safety, due to the complexity of the problem. Persistent rates of harm justify an ongoing focus on patient safety within Canada's healthcare system.

More than a decade after patient safety first emerged as a major concern within the Canadian healthcare system, available data indicate that patient safety has not appreciably improved, suggesting an ongoing need to address the issue.

While the need for a national patient safety organization clearly remains, it is less clear what role such an organization should play in the current environment. Over the past decade, CPSI has fulfilled a critical need in Canadian healthcare by promoting awareness and knowledge of patient safety issues and providing a diverse range of evidence-based tools, resources, and strategies that could be used by healthcare organizations, at the clinical and governance levels, to address patient safety. The availability of these tools and resources has been particularly important to smaller organizations and jurisdictions with limited resources.

However, many external interviewees identified issues that have not to date received significant attention from CPSI, and that could be the focus of its activities in the future. These included, but were by not limited to: public health (e.g., antimicrobial stewardship), primary care, palliative care, community support services (programs that provide assisted living, supportive housing, attended care, transportation, nutrition, hospital-to-home services, and friendly visiting services), and patient safety in relation to Indigenous populations.^{Footnote xxviii}

5.1.2 Alignment with Government Priorities

There is alignment between CPSI and federal priorities, and CPSI priorities are generally perceived by other stakeholders to be well-aligned with their own. While most stakeholders believe CPSI has adjusted effectively to changing needs and circumstances and that its activities address gaps and needs in the patient safety field, concerns were raised that CPSI lacks a clear strategic focus to guide its work.

5.1.3 Alignment with Federal Roles and Responsibilities

CPSI's mandate and activities are consistent with federal roles and responsibilities and, in particular, with Health Canada's mandate under the Department of Health Act. CPSI meets expectations that Health Canada has for pan-Canadian health organizations to perform a coordinating function and drive pan-Canadian collaboration.

As a national organization with a mandate to provide a leadership role in building a culture of patient safety and quality improvement in the Canadian health system, CPSI is well-placed to advocate for the importance of patient safety as a priority within the system. This mandate is bolstered by widespread stakeholder support for the continued existence of a national organization specifically dedicated to patient safety.

Stakeholders variously perceive CPSI's unique value as being, among other things, the pan-Canadian focus it brings to bear on efforts to improve patient safety, its ability to bring together experts and stakeholders from across the country to work collaboratively on shared priorities, and its role as a clearinghouse for evidence-based patient safety resources, tools, and best practices. Some stakeholders believe CPSI has had difficulty carving out a clear role for itself within the patient safety landscape.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

Evidence available to the evaluation indicates that progress has been made toward CPSI's expected immediate and intermediate outcomes.

• CPSI has contributed to an increased evidence base to improve patient safety, primarily by developing tools and resources grounded in national and international research evidence, and stakeholders widely acknowledge and appreciate this work. Some stakeholders argued that CPSI could further contribute to the evidence base through greater emphasis on measurement and rigorous evaluation, including evaluation of its own products and interventions.
• CPSI's work has led to the development of evidence-informed patient safety curricula. A key success has been the integration of the Safety Competencies into the CanMEDS educational framework of the Royal College of Physicians and Surgeons of Canada and the College of Family Physicians in Canada. As part of the Integrated Patient Safety Action Plan (IPSAP), work is ongoing with many stakeholders and partners to implement the priorities identified by experts via the Patient Safety Education Action Plan.
• CPSI's work has produced important gains in patient safety awareness and knowledge among stakeholders. Most survey respondents who have used or implemented specific CPSI programs or resources, participated in CPSI education programs, or read CPSI research reports, reported that doing so has led to increased awareness and knowledge of patient safety issues, either their own, or within their organization more generally.
• Through spearheading the establishment of National Patient Safety Consortium and the development of the IPSAP, CPSI has strengthened system coordination. These efforts have united a wide range of stakeholders in advancing a common patient safety agenda, while focussing on gaps and avoiding duplication of effort.
• Over the past decade, legislative and policy changes related to patient safety have suggested that Canada has been shifting toward a more positive patient safety culture. Most stakeholders believe this shift has occurred, in part, because of CPSI's work, although many noted that culture change is a lengthy process, and measurement and attribution are difficult.
• Canadian healthcare organizations have implemented positive patient safety practices as a result of CPSI activities, with an estimated 88% of eligible acute care facilities and 26% of long-term care facilities in Canada having participated in Safer Healthcare Now! (SHN).
• Progress has been made in formal monitoring and reporting on patient safety in Canada. Many survey respondents reported using CPSI resources such as the Canadian Incident Analysis Framework to make changes to their approach to managing patient safety. The collaborative work of CPSI and CIHI to develop the Hospital Harm Measure is seen as an important step forward.
• There is evidence of policies, standards, and requirements of professional associations and accreditation bodies informed by patient safety evidence. The Safety Competencies and the Canadian Disclosure Guidelines, in particular, have been widely used and adopted, and more than one-third of Accreditation Canada's Required Organizational Practices (ROPs) reference CPSI resources.
• CPSI has been a pioneer in Canada in advocating for and supporting patient and family involvement in healthcare improvement in Canada and internationally through its support for Patients for Patient Safety Canada (PFPSC) and other patient engagement activities. The extent of patient and family involvement within the healthcare system and the impact of this involvement for patient safety outcomes has not been measured or rigorously evaluated.

In the long term, CPSI's activities are expected to contribute to improved patient safety in Canada. Although there is no objective evidence that patient safety in Canada has improved since CPSI was established, most stakeholders believe that CPSI's activities have contributed to improved patient safety. Some argued that without CPSI, it is conceivable that patient safety in Canada might have lost ground rather than remaining stable, particularly given a context of continuous technological and medical innovation, along with the increasing complexity of, and growing financial pressures on, the healthcare system.

5.2.2 Demonstration of Economy and Efficiency

CPSI has operated in an economical manner over the three years covered by the evaluation. It has taken steps to minimize the cost of inputs by eliminating one office and downsizing another, increasing the number of staff working virtually, and managing staff remuneration, among other measures. Although Health Canada's contribution is CPSI's main source of funding and is likely to remain so in the future, CPSI could continue to explore cost recovery and other revenue-raising potential from sources other than Health Canada, and could articulate a comprehensive pricing model for its products and services.

Likewise, CPSI has taken steps to operate efficiently. It has sought to optimize the quantity and quality of its outputs through the use of web-based technology for information dissemination and stakeholder engagement, and leveraging in-kind contributions from stakeholders and external experts. The latter have been critical to CPSI's ability to undertake its activities and produce high-quality products and services. While some stakeholders identified potential additional areas for future CPSI activity (with little commonality or agreement among them), others raised concern that CPSI is already overextended, given its size, and recommended it focus on fewer, well-defined priorities in order to achieve greater impact. Other key suggestions included more strategic consideration to partnerships and collaborations, an emphasis on defining future directions in patient safety, and greater attention to measurement and evaluation.

External stakeholders perceive CPSI's activities to have generated considerable value-for-money for their own organization and for others. Furthermore, formal, quantitative value-for-money analysis, focussing on CPSI's medication reconciliation activities, suggests that these have generated value-for-money by producing cost savings well in excess of the expenses required to sustain them. Implementation of medication reconciliation in one acute care facility has generated positive net benefits by averting the loss of patient welfare that accompanies preventable adverse drug events.

To the extent that CPSI's activities have accelerated the uptake of medication reconciliation in Canadian healthcare, and ensured implementation in accordance with current best practices, CPSI is likely to have generated additional value-for-money for its stakeholders in terms of improved patient well-being. Assuming that all SHN interventions are designed in accordance with current best practices, it seems reasonable to assume the same is true of CPSI's activities in relation to SHN interventions more generally.

6.0 Recommendations

The recipient-led evaluation resulted in five recommendations directed at CPSI which can be found on the CPSI website. There are no recommendations directed at Health Canada.

Appendix 1 – CPSI's activities

CPSI's logic model identifies four broad categories of outputs that are expected to result from the organization's activities, namely interventions and programs, education, research and tools and resources.

Interventions and Programs

Over the period covered by this evaluation, CPSI established several major interventions and programs to assist individuals and organizations in reducing patient harm.

Safer Healt care Now! CPSI's flagship program, Safer Healthcare Now! (SHN) was established in 2005. It was a national program to support Canadian healthcare organizations in improving patient safety through the use of quality improvement methods and integration of evidence in practice. It had three primary goals: reducing harm, improving healthcare, and protecting Canadians. It accomplished these goals by developing and aiding in the implementation of a suite of evidence-based and empirically validated patient safety interventions for primary healthcare providers. The focus of SHN was on the implementation of interventions supported by measurement and evaluation. Each intervention had a corresponding Getting Started Kit, which was intended to serve as a comprehensive and pragmatic toolkit or change package for launching the intervention. Implementation was further supported by topic-specific information webinars, learning and action series, and improvement collaboratives. Discussions and documents could be accessed in online videos, tools, and resources.

In 2016, CPSI redirected some previous SHN resources to create and launch a new program called SHIFT to Safety. The change was prompted by evolving circumstances and needs in the practice environment, including the fact that many SHN interventions had become Accreditation Canada required practices and therefore had largely become embedded into routine practice across the country, the growing evidence that large-scale change requires systems-oriented approaches, and the recognition that patients and families must be positioned as full partners in care.

SHIFT to Safety is intended to be the source for patient safety information in Canada for members of the public, healthcare providers, and healthcare leaders. SHIFT to Safety is described further in Appendix 4, along with CPSI's other major new initiatives that have been implemented since 2013.

Global Patient Safety Alerts. Global Patient Safety Alerts was launched in 2011 and is maintained in collaboration with the World Health Organization (WHO) and international partner organizations from Australia, Canada, Denmark, England, Wales, the European Union, Hong Kong, Japan, and the US. It collects and stores publicly posted accounts of patient safety incidents, as well as lessons learned from those incidents and recommendations for actions to avoid or mitigate the consequences of such incidents. The database is available through a searchable web portal that supports advanced search filtering and browsing by category, and is also accessible via a free iOS application.

Patient Safety and Incident Management Toolkit. Launched in 2012, the toolkit provides practical strategies and resources to manage incidents effectively and keep patients safe. It integrates key CPSI resources, including the Canadian Disclosure Guidelines, Incident Analysis Framework, Global Patient Safety Alerts, and others. Co-designed with patient partners and drawn from the best available evidence and expert advice, the toolkit is intended for those responsible for managing patient safety, quality improvement, risk management, and staff training in any healthcare setting. There are three sections to the toolkit: 1) incident management — the actions that follow patient safety incidents (including near misses), 2) patient safety management — the actions that help to proactively anticipate patient safety incidents and prevent them from occurring, and 3) system factors — the factors that shape and are shaped by patient safety and incident management (legislation, policies, culture, people, processes and resources).

Canada's Hand Hygiene Challenge. Originally launched in 2007 as the Hand Hygiene Campaign, this initiative was relaunched as Canada's Hand Hygiene Challenge in 2010. It provides implementation tools to help reduce patient risk through increased hand hygiene. The Hand Hygiene Toolkit includes educational and promotional materials to promote hand hygiene. The Human Factors Toolkit includes tools organized around contemporary research on the human factors influencing compliance. The Patient and Family Guide informs and coaches patients on taking an active role in hand hygiene. A 15-minute online training module, with an associated certificate, has been completed by more than 14,000 healthcare workers and students across Canada. A variety of other materials including videos, scholarly articles, and advocacy links is also available on the webpage.

In addition to Canada's Hand Hygiene Challenge, CPSI is the Canadian host to the WHO Global Campaign, SAVE LIVES: Clean Your Hands. Each year on May 5, in partnership with Accreditation Canada, the Public Health Agency of Canada (PHAC), and Infection Prevention and Control Canada (IPAC), CPSI hosts STOP! Clean Your Hands Day. On average, over 1,000 Canadian organizations register each year for the event.

Patients for Patient Safety Canada (PFPSC). Initiated in 2005 after the WHO convened the first Patient for Patient Safety workshop, PFPSC was formed in 2006 and formally launched in 2007, with the goal of representing the patient perspective in patient safety improvements at all system levels. Patients are defined broadly to include clients, residents, customers, and family members (as specified by patients). The PFPSC membership is composed of volunteer patients or family members, many of whom have directly or indirectly experienced safety incidents stemming from avoidable patient safety issues. They represent the patient perspective in speaking engagements, working groups, committees, research consultations, and a variety of other fora. CPSI partners with patients in internal processes, such as recruitment of executives and strategic planning, and in external offerings. Several other organizations, across Canada and internationally, have requested PFPSC volunteers to partner in the design, development and delivery of their own programs.

Education

CPSI offers education and professional development to practicing professionals, as well as graduate and undergraduate students through several programs, as summarized below.

• Effective Governance for Quality and Patient Safety (EGQPS) provides a toolkit and educational sessions to support safe and effective governance of healthcare organizations.^{Footnote 53}  Targeted at Board members and trustees, the course is offered on a cost-recovery basis at a cost of approximately $600 per participant.
• The Canadian Patient Safety Officer Course (CPSOC), delivered in partnership with HealthcareCAN, provides an introduction to the techniques and tools required to advance patient safety for healthcare professionals responsible for administering or disseminating safety agendas within their organizations. The interactive course is available face-to-face or online using webinars, discussion fora, and access to faculty consultations.
• The Patient Safety Education Program (PSEP) – Canada is a Canadian adaptation of a program that began at Northwestern University and is now managed in the US by MedStar Health System. It aims to provide inter-professional teams with the tools necessary to become patient safety implementers and trainers within their own organizations — a 'train-the-training-team' approach. Over 35 separate modules are available, all of which are mapped to Accreditation Canada standards and CPSI's Safety Competencies Framework. Initial training is delivered at a conference by a team of Master Facilitators, and follow-up support is provided through online communities.
• Advancing Safety for Patients in Residency Education (ASPIRE) is an adaptation of the PSEP – Canada program and is a four-day workshop co-designed with the Royal College of Physicians and Surgeons of Canada to integrate with the established residency programs. It targets medical educators and residents with an interest in championing the pedagogy of patient safety by integrating process improvement and resource stewardship perspectives into patient safety education, fostering the skills development required to assume leadership roles to change education in residency programs, and formulating an education plan that promotes patient safety during residency.
• The Incident Analysis Learning Program is a set of learning modules that explore each of the eight sections of the Canadian Incident Analysis Framework. They are hosted on CPSI's website and provided free of charge.

CPSI also hosts and attends a variety of healthcare events across Canada, including lectures, panel discussions, public awareness campaigns, and workshops. Two large-scale annual events are Canadian Patient Safety Week (CPSW) and Canada's Virtual Forum on Patient Safety and Quality Improvement. CPSW is a national annual campaign started in 2005 to inspire improvement in patient safety and quality. Health facilities and health organizations across Canada participate in the week by holding events and activities to increase awareness of patient safety in their local area. The campaign is targeted at everyone involved in healthcare delivery in Canada, as well as those receiving care, including current and future healthcare professionals, decision-makers, patients, clients, and their families. CPSI provides participants with a CPSW package containing promotional products and materials to support local campaigns.

Research

CPSI supports research directly related to patient safety via competitions and collaborative funding, and indirectly by disseminating research findings, forming partnerships, and creating advisory panels. Since 2014–2015, CPSI has offered research funding through three posted competitions for $50,000.^{Footnote 54}  It also provides grants to students in health and allied disciplines for applied patient safety research.^{Footnote 55}  In addition to competitions, CPSI has commissioned 12 research projects, published between 2006 and 2014 (including five in the current evaluation period), that focus on understanding specific patient safety topics to inform its work.^{Footnote 56}

Tools and Resources

Finally, the CPSI website is a free, searchable repository of tools and resources such as toolkits, guides, frameworks, reports, checklists, presentations, and patient stories. Some examples include the Canadian Incident Analysis Framework, the Incident Management Toolkit, the Safety Competencies Framework, the Canadian Framework for Teamwork and Communication, and the Canadian Disclosure Guidelines, to name just a few.

A recent addition to CPSI's tools and resources was the publication, in October 2016, of a joint national report by CPSI and the Canadian Institute for Health Information (CIHI) entitled Measuring Patient Harm in Canadian Hospitals.^{Footnote 57}  This report provided the first results of a new patient safety measure, the Hospital Harm Measure, developed collaboratively by CIHI and CPSI in an effort to provide a national picture of patient safety. The measure uses data on discharges from Canadian acute care hospitals found in CIHI's Discharge Abstract Database, and captures hospitalizations with at least one unintended occurrence of harm that could potentially be prevented by implementing known evidence-informed practices. It includes 31 clinical groups based on four categories of harm, but excludes Quebec and select mental health diagnoses.^{Footnote xxix}

The measure provides, for the first time, a general measure of patient safety in acute care settings in Canada. The process of developing and refining the measure occurred in consultation with facilities and experts across Canada over a four-year period.

To complement the Hospital Harm Measure, CPSI has developed the Hospital Harm Improvement Resource (the Improvement Resource). Released in conjunction with the Measuring Patient Harm in Canadian Hospitals report, this resource links measurement and improvement by providing evidence-informed resources to support patient safety improvement efforts for each of the 31 clinical groups included in the measure. The Improvement Resource is intended to make information on improving patient safety easily available, so that teams spend less time researching and more time optimizing patient care. In partnership with CIHI, CPSI is supporting organizations in using the Hospital Harm Measure through briefings with stakeholders prior to the launch, a public launch of the national report with an expert patient safety panel, and subsequent national calls focussed on applying the measure and the Improvement Resource to patient safety improvement at the organizational level. CPSI consulted with clinical experts, Accreditation Canada, and patient advisors in the development of this resource.

Appendix 2 – Logic Model

Appendix 3 – Evaluation Description

An evaluation matrix was developed to address key questions of interest to senior management within CPSI and Health Canada and to align with the Treasury Board’s 2016 Policy on Results. The evaluation considered the five core Treasury Board issues under the two themes of relevance and performance (effectiveness, efficiency, and economy). Corresponding to each of the core issues, specific questions were developed based on program considerations, and these guided the evaluation process.

The evaluation drew on multiple lines of evidence, including a literature review, a review of documents and data, key informant interviews, a survey of CPSI stakeholders, and three in-depth case studies. A brief description of each of these data collection methods is provided below.

Literature, document, and data review

The literature review addressed evaluation questions relating primarily to relevance (i.e., continued need), while the document and data review provided historical and contextual information for CPSI’s activities and responded directly to virtually all of the evaluation questions. Relevant literature and documents were provided by CPSI or were accessed from publicly available sources. The data review considered financial information, as well as performance measurement data tracked by CPSI through its performance measurement framework.

Key informant interviews

Two rounds of interviews were conducted to support the evaluation. In the first round, 10 CPSI staff and Board members were interviewed in order to inform evaluation planning and design. In the second round, 40 individuals were interviewed, including CPSI staff and Board members (n=9); Health Canada representatives (n=3); and external stakeholders (n=28). External key informants represented a range of stakeholders, including provincial and territorial government organizations (e.g., ministries of health and quality councils), national and regional partner organizations, professional associations, healthcare providers, patient organizations, and academics. Many external key informants had multiple affiliations and fell into more than one of these stakeholder categories. Key informants were selected by CPSI for their knowledge of, and experience with, CPSI or the patient safety field in Canada. Interviews were digitally recorded with the permission of key informants to ensure the accuracy of the information collected.

Survey of CPSI stakeholders

A bilingual web-based survey of CPSI stakeholders was conducted. CPSI supplied the survey sample based on contacts within its contact database. All contacts within the database were included, with the exception of vendors, media, and CPSI personnel, resulting in a sample of 10,342 email addresses.

Overall, the survey achieved 596 completions, representing a completion rate of 6%. This is similar to response rates achieved in stakeholder surveys conducted by Prairie Research Associates of similar pan-Canadian organizations in the health field.^{Footnote xxx}  Overall, 90% of respondents (n=535) completed the survey in English, while 10% (n=61) completed it in French.

The survey results were analyzed using SPSS and R, two statistical data analysis software packages commonly used in social science research. Analysis consisted primarily of basic frequency tabulations. In addition, key questions were analyzed for differences among sub-groups based on respondents’ current positions or roles, the type of work they do in their organizations, the types of organization, and whether their organizations provide direct care to clients.^{Footnote xxxi}  For categorical outcomes, Chi-squared analysis was employed for single predictors or logistic regression in the case of multiple predictors. For ranked outcome variables (e.g., Likert scales), Kruskall-Wallis tests were employed with follow-up Mann-Whitney U tests. Where appropriate, Bonferroni corrections were made for multiple comparisons. Results significant at α = .05 are reported in the relevant sections.

Open-ended survey questions were analyzed using thematic analysis and tabulated in SPSS where the number of responses was sufficient to warrant it.

Case studies

Three case studies were conducted of organizations that had engaged with CPSI or implemented CPSI interventions, products, and resources. Three organizations of varying size and in different jurisdictions were selected by CPSI for case study: St. Michael’s Hospital in Toronto, Ontario; Vitalité Health Network in New Brunswick; and Alberta Health Services.

In all three cases, data collection consisted of document and data review, as well as a small number of interviews with organizational representatives. In addition, the case study of Vitalité Health Network included a quantitative value-for-money analysis, focussing on the value-for-money generated by CPSI’s medication reconciliation activities in acute care facilities. A detailed description of the methodology used to complete the value-for-money analysis is included in the Vitalité case study report.

Individual study reports were prepared for each organization and approved by the organizations prior to being submitted to CPSI. Key findings have been integrated into this report.

Appendix 4 – Major activities and response to previous evaluations

Since 2013, and in response to the recommendations of the previous evaluation to focus or rationalize its scope of activity, enhance engagement with the system and embed ownership of major programs and activities within system, CPSI has oriented its activities toward achieving system-level change through the pursuit of four strategic goals. Evidence suggests that CPSI has experienced challenges in achieving the increased focus or rationalization of activity called for by the previous evaluation. CPSI scaled back or even discontinued some existing program areas and activities, but it also realigned some existing activities under its new strategic goals and introduced several new initiatives, effectively increasing, rather than decreasing, the breadth of its programming. CPSI representatives noted that the organization has been in transitional phase since the last evaluation as it has worked to rationalize or focus its scope of activity.

As described in section 4.2, CPSI responded to the recommendations of the previous evaluation – to focus or rationalize its scope of activity through enhancing engagement with the system and embedding ownership of major programs and activities within system – by articulating a vision for increased focus on system-level change through the pursuit of four strategic goals. Its major activities in the evaluation period, including the development of the Consortium and the Integrated Patient Safety Action Plan (IPSAP), the transition from Safer Healthcare Now! (SHN) to SHIFT to Safety, and the development of the HUB model, have been oriented around these four strategic goals, and these activities are described below.

Development of the Consortium and the IPSAP

CPSI’s overarching strategic goal since 2013 has been the development of a National Patient Safety Strategy. As already described, CPSI spearheaded the development of the National Patient Safety Consortium in January 2014 as a mechanism for achieving this goal.^{Footnote xxxii}  Accordingly, CPSI is responsible for maintaining the overall strategic coherence and coordination of the IPSAP, which brings together six separate but interlocking areas of focus for patient safety into a three-year action plan affirmed by the Consortium. Four initial areas of focus have been identified, each with associated action plans: medication safety, surgical care safety, infection prevention and control and home care safety. In addition, the IPSAP includes action plans relating to patient safety education and the National Patient Safety Consortium itself. The IPSAP is depicted in Figure 1.

Figure 1: Integrated Patient Safety Action Plan

While the IPSAP began to take shape after the January 2014 Consortium meeting, the implementation horizon for the three-year plan runs predominantly from April 2015 to March 2018. Reflecting the collaborative nature of the IPSAP, a wide range of organizations from across Canada provide direct leadership or contribute actively as partners in the actions.

The Consortium is subject to a collaborative governance model proposed by CPSI and implemented by the collective. The governance model is comprised of various entities, key among which are a Steering Committee of executive leaders from partner organizations; Leads Groups to lead the planning, implementation, and coordination of IPSAP actions; and Action Teams, which are responsible for implementing the IPSAP actions.

The Consortium and the IPSAP will undergo a separate evaluation beginning in early 2017, based on an evaluation plan that was developed in fulfillment of an IPSAP action calling for evaluation of the work. Progress reporting on the IPSAP indicates that progress has been made in implementing all of the IPSAP action plans.^{Footnote xxxiii}  Overall, by the third quarter of its second year (i.e., by December 2016), 40% (n=34) of the 93 IPSAP actions were completed, 27% (n=23) were started, 32% (n=31) were scheduled to start at a later time, and 1% (n=1) were expected to start, but were delayed. Although progress appears to vary by action plan (see Table 16), it is important to note that action plans were created at different points in time over a 15-month period. Furthermore, Leads Groups had the flexibility to define their own actions, resulting in substantial variation in the number and complexity of actions across and within action plans. Finally, many of the actions identified in the IPSAP are laid out sequentially, necessitating the completion of some before others can begin. 

The establishment of the Consortium and the subsequent development of the IPSAP as a coordinated national effort to address patient safety were central elements of CPSI’s response to the recommendations of the previous evaluation to enhance stakeholder engagement and focus its scope of activity.^{Footnote xxxiv}  In pursuing this initiative, CPSI consulted with provincial and territorial jurisdictions and partners across the country to identify and validate system priorities, created a national collaborative governance and shared leadership structure, and secured the participation and commitment of a diverse range of organizations. Although CPSI reports that some challenges remain such as finding the best means to balance comprehensive involvement of partners with simplicity in governance and building a truly collective commitment (as demonstrated through financial investment by partners and strategic/operational plan alignment), it seems clear that advances in stakeholder engagement have been made through this initiative.

It is less clear whether or not the Consortium and the IPSAP represent a rationalization or focusing of CPSI’s activity. While it is certainly the case that the IPSAP addresses specific priorities and gaps identified through the consultative process – and can be thought of as “focussed” in that sense – CPSI representatives acknowledged that in many ways, the Consortium and the Action Plan were an add-on to CPSI’s other existing activities, and therefore increased, rather than decreased, the breadth of its programming.^{Footnote xxxv}

Transition from Safer Healthcare Now! to SHIFT to Safety

Another major activity for CPSI in the evaluation period has been the development and launch of a new program, called SHIFT to Safety, to replace its long-standing flagship program, SHN. Launched on July 20, 2016, SHIFT to Safety is a primarily web-based resource that curates best practice information from around the world for three target groups: patients and their families, healthcare providers, and leaders. CPSI key informants explained that, both in Canada and internationally, it had become evident that although clinical interventions such as SHN, which mostly targeted providers in acute care hospitals, were important, they alone were not sufficient to bring about desired improvements in patient safety, and that a broader approach targeting patients, families, and leaders, as well as healthcare providers, in a variety of settings, was necessary in order to have a real impact on patient safety.

SHIFT to Safety aims to answer the most pressing questions in patient safety, namely: 1) How do I prevent harm?, 2) How can I respond to harm when it happens?, and 3) How can I learn from harm that has already happened? The program offers evidence-based resources focussed on improving teamwork, communication, leadership, and patient safety culture and targeted at the specific needs of its three target audiences. Resources for the public are designed to empower patients and families to advocate for healthcare safety. Resources for providers are designed to encourage providers to prioritize safety, work as a team, respond effectively to patient safety incidents, and reduce the risk of patient injury. Resources targeting leaders are designed to assist leaders in taking steps to engender a positive patient safety culture.^{Footnote 58}  SHIFT to Safety is consistent with recent recommendations in the literature to foster a systems approach to safety that focusses on teamwork, culture, and measurement as key elements of safe systems.

SHIFT to Safety is envisioned by CPSI as an aspect of its transition toward an increasingly systems-based approach to change, and in this sense is a major element of CPSI’s response to the recommendations of the previous evaluation to focus its scope of activity and enhance engagement with the system. The development and implementation of SHIFT to Safety represented a significant change of direction for CPSI and was supported primarily by the scaling back and phasing out of CPSI’s long-standing flagship initiative, SHN, and the Patient Safety Metrics (PSM) system.

Development of the HUB model

A third major activity during the period covered by the evaluation has been the development of the HUB/Affiliation model for delivery of CPSI’s educational offerings. Development of the model was prompted by a desire to achieve greater reach, sustainability, and system embedment of these offerings. Under the HUB model, agreements between CPSI and partner organizations transfer most of the responsibilities for the planning and delivery of a mature educational offering to the partner organization, while CPSI retains responsibilities for maintaining core curricula, ensuring quality control, and monitoring overall program evaluation.

To date, CPSI has two national educational programs that have HUB offerings: Effective Governance for Quality and Patient Safety (EGQPS) and the Patient Safety Education Program (PSEP) – Canada. These programs have been licenced out to five organizations. The Association of Family Health Teams of Ontario and the Ontario Hospital Association (OHA) have agreements with CPSI to deliver EGQPS, while Queen’s University, the Manitoba Institute for Public Safety (MIPS), and the OHA have agreements to deliver PSEP – Canada. CPSI has an additional agreement with the Canadian Armed Forces to deliver PSEP – Canada.

CPSI reports that the organization is currently in the process of negotiating an agreement with Alberta Health Services (AHS) that would see AHS become a PSEP – Canada Affiliate in 2017, and is working with the New Brunswick Department of Health to establish a HUB affiliate partnership for EGQPS.
CPSI key informants reported that the development of the HUB model has been the primary way through which the organization has sought to address the previous evaluation’s recommendation to embed ownership of major programs and services within the system. Through the HUB model, partners have provided increased access to CPSI’s intended audience and allowed CPSI to further meet demands from the field, which would otherwise not be possible given CPSI’s resources and capacity. While the HUB model has realized success, CPSI key informants noted that it has been challenging to identify potential HUB affiliate organizations and vet potential partners. In addition, they noted that work remains to be done to ensure clarity regarding CPSI’s use of resources dedicated to HUB partnerships and to standardize some of the processes related to the HUB model while retaining flexibility in partnership arrangements. At present, according to CPSI, significant CPSI resources are required to support HUB partners, negating much of the resource savings gained from transferring the program out of CPSI’s direct control. While CPSI support requirements are reduced as HUB partnerships mature, allowing CPSI to shift its focus away from program delivery toward maintenance of curriculum and data collection related to session delivery, not all HUB affiliates reach this level of maturity at the same rate or as quickly as anticipated. Thus, the level of CPSI support required can vary from partner to partner. Moving forward, CPSI aims to ensure greater clarity regarding its ongoing role in supporting HUB partners in future partnership agreements. 

While CPSI has introduced HUB offerings for the EGQPS and PSEP for the reasons explained above, the organization continues to offer pan-Canadian access to these programs through its traditional delivery model.

Overall success in responding to the previous evaluation

CPSI responded to the recommendations of the previous evaluation – to focus or rationalize its scope of activity, enhance engagement with the system, and embed ownership of major programs and activities within the system – by developing its 2013–2018 Business Plan. The Business Plan laid out a vision for increased focus on system-level change through the pursuit of four strategic goals. In the evaluation period, CPSI has oriented its major activities, including the development of the Consortium and the IPSAP, the transition from SHN to SHIFT to Safety, and the development of the HUB model, around these four strategic goals. 

Despite the formulation of strategic goals to guide its work, CPSI has encountered challenges in achieving the increased focus or rationalization of activity called for by the previous evaluation. During the funding period, CPSI eliminated activity in two notable areas: suspension of Canada’s Forum on Patient Safety and the decommissioning of SHN and the PSM system. For the most part, however, CPSI scaled back existing program areas and activities or realigned them under its new strategic goals, rather than discontinuing them altogether, and it also introduced several new initiatives. Arguably, the overall effect was to increase, rather than decrease, the breadth of its programming.

In reflecting on CPSI’s efforts to respond to the recommendations of the 2012 evaluation, CPSI representatives noted that the organization wrestled internally with how to balance a more strategic orientation with historical programs that had been successful and were seen as crucial components of patient safety. In addition, they noted that CPSI’s establishment of internal operational divisions within each strategic goal, called Goal Areas, may have mitigated against achieving greater focus. Once included within a Goal Area, all programs were by definition seen as contributing to the larger strategic goal. As a result, the strategic goals could be, and evidently were, read very broadly to include almost any program activities. 

Endnotes