Evaluation of the Pan-Canadian Health Organizations 2018-19 to 2022-23

Prepared by the Office of Audit and Evaluation
Health Canada
March 2024

Table of contents

• List of acronyms
• Executive summary
• Background and context
• Evolution of the Pan-Canadian Health Organizations
• Impacts of the Pan-Canadian Health Organizations
• Impacts on behaviour change
• Impacts on health systems
• Factors facilitating and inhibiting progress on PCHO goals
• Health Canada's management of PCHO suite
• Conclusions and recommendations
• Management Response and Action Plan
• Appendix A: Description of the Pan-Canadian Health Organizations
• Appendix B: Common Logic Model, Pan-Canadian Health Organizations
• Appendix C: Recommendations in the Fit for Purpose Review
• Appendix D: Methodology
• Appendix E: Frequency of Meetings between Health Canada and the PCHO Leadership
• Appendix F: Health Canada PCHO Expenditures - Financial Data
• Appendix G: Examples of Collaboration
• End notes

List of acronyms

CADTH

Canadian Agency for Drugs and Technologies in Health

CCSA

Canadian Centre on Substance Use and Addiction

CFHI

Canadian Foundation for Healthcare Improvement

CIHI

Canadian Institute for Health Information

CPAC

Canadian Partnership Against Cancer

CPSI

Canadian Patient Safety Institute

HEC

Healthcare Excellence Canada

MHCC

Mental Health Commission of Canada

PCHO

Pan-Canadian Health Organization

Executive summary

This report presents the findings of the Evaluation of the Pan-Canadian Health Organizations (PCHO), covering the period from 2018-19 to 2022-23. The purpose of the evaluation was to examine how PCHOs have evolved to meet the changing needs of Canada's health systems, the individual and collective impacts PCHOs have made towards their shared goals, and the extent to which Health Canada's management of the PCHO suite has contributed towards efficiency and effectiveness.

Key findings

Evolution

PCHOs continue to play an important role in the Canadian context. In some cases, PCHOs are unique in fulfilling their mandate at the pan-Canadian level. In other cases, they complement the work of other non-governmental organizations through their position as national organizations, and as conveners of key players in their respective areas of focus. Over the past five years, PCHOs have taken the initiative to improve collaboration between themselves and with other non-governmental organizations.

PCHOs have pivoted to respond to emerging federal and national priorities, particularly during the pandemic response. Despite refocusing to work on these priorities, PCHOs were able to advance their work on longstanding initiatives. PCHOs and Health Canada have also taken steps to address areas of potential change highlighted in the 2018 Fit for Purpose review, such as program management and process improvements, and structural changes to increase efficiency and enhance alignment between the PCHOs.

Impacts

PCHOs have made collective and individual progress towards their shared goals of enhancing capacity in provincial-territorial health systems, changing the behaviour of health care decision makers, and supporting improvements in the health systems. There are opportunities to improve collective reporting across the PCHO suite on shared outcomes, such as by examining indicators to ensure they adequately demonstrate PCHO activities' impacts on their stated shared goals.

Health Canada's management

Over the past five years, there has been increased collaboration between Health Canada and the PCHOs. Health Canada regularly liaises with PCHO leadership and staff through a series of formal and informal mechanisms. This improved relationship has resulted in the PCHOs having greater clarity on federal priorities, and increased information sharing between the PCHOs and Health Canada, which helps the PCHOs better navigate government processes to allow them to focus on delivering on their mandates.

There are opportunities to improve internal coordination within Health Canada and the Health Portfolio to ensure a consistent strategic direction. There were challenges associated with information flow between senior management and operational staff, and between various branches of Health Canada and other members of the Health Portfolio that interact with the PCHOs. As a result, there were occasional multiple competing requests made of the PCHOs, without adequate understanding of the impact of these requests on their existing work. There is value in exploring internal coordination mechanisms for the Portfolio to help ensure prioritization of its requests to the PCHOs, particularly when they require organizations to pivot from their original workplans.

Recommendations

The evaluation has recommended improvements in three areas:

Recommendation 1: Examine opportunities to improve internal and Health Portfolio governance mechanisms to support Health Canada's approach to managing the PCHO suite.

Some internal interviewees noted gaps in the information flow between operational and senior management staff in Health Canada and between Health Portfolio members, which can create challenges for working level staff to effectively manage the funding agreement. Taking steps to address these gaps in information flow within Health Canada and the Health Portfolio may help streamline requests and support a more cohesive approach to management of the PCHO suite.

Recommendation 2: Implement strategies to prioritize requests from the Health Portfolio, with consideration of the potential impact on existing PCHO workplans.

Representatives from Health Canada and the PCHOs noted that there are opportunities for Health Canada and the Health Portfolio to prioritize their requests to the PCHOs and minimize the implications of adding requests to pre-planned activities. This is important, as the PCHOs must remain responsive to the needs of their various stakeholders and other funders, where applicable, as arms-length organizations. In addition, from the Health Canada perspective, greater prioritization reduces administrative burden, such as fewer potential contribution agreement amendments, and helps maintain more positive relationships with the PCHOs.

Recommendation 3: Work with the PCHOs to examine indicators used for collective reporting across the PCHO suite.

While the approach of developing a common logic model has been a step in the right direction towards a framework for collective reporting, some PCHO indicators related to the common logic model do not adequately demonstrate the impacts the PCHOs have made on their stated shared goals. To support continual improvement in the performance measurement framework for the PCHO suite, Health Canada should work with them to review indicators to help more efficiently and effectively describe the performance story of the PCHOs' collective impact.

Background and context

Overview of the Pan-Canadian Health Organizations

Health Canada funds seven independent non-profit Pan-Canadian Health Organizations (PCHOs) through contribution agreements. These organizations receive the majority of their funding from Health Canada and take a pan-Canadian approach to their activities and have Health Canada representation on their Board of Directors.

The Pan-Canadian Health Organizations include the following:

• Canada Health Infoway (Infoway)
• The Canadian Agency for Drugs and Technologies in Health (CADTH)
• The Canadian Centre on Substance Use and Addiction (CCSA)
• The Canadian Institute for Health Information (CIHI)
• The Canadian Partnership Against Cancer (CPAC)
• Healthcare Excellence Canada (HEC)
• The Mental Health Commission of Canada (MHCC)

See Appendix A for a description of each PCHO.

While each PCHO has a distinct area of focus, all PCHOs support and encourage sustainable and adaptable health systems by working with provinces, territories, and other partners on specific Canadian health systems needs and priority issues of national concern. The PCHOs work together and alongside other national mechanisms, including the following:

• federal organizations, programs, and legislation;
• intergovernmental processes;
• professional colleges and associations, First Nations, Métis, and Inuit organizations;
• and non-governmental and private sector organizations.

The mandates and activities of the various PCHOs have evolved since they were first created. Going forward, how Health Canada engages each PCHO in furthering the government's broad policy objectives will be adjusted as appropriate based on needs, priorities and the changing environment and context in which they operate.

The arms-length nature of the PCHOs, which exists because of their transfer payment funding relationship with the federal government, is intended to enable them to work more effectively with entities, partners, and communities on pan-Canadian issues where it would be difficult for the federal government to directly engage. This mechanism also enables the PCHOs to have more autonomy to align with federal, provincial, and territorial partners, in addition to alignment with pan-Canadian health system priorities.

Fit for Purpose review

In March 2018, the final report of an external review of the Pan-Canadian Health Organizations entitled "Fit for Purpose" was published.^{Footnote 1} The review provided potential scenarios for consideration to reconfigure the PCHOs to best serve federal health priorities and the country's health systems.

Fit for Purpose made 10 non-binding recommendations for the federal government on the PCHOs' ability to support future health systems. Recommendations were focused on better aligning the operations of the seven PCHOs with federal priorities and contemporary issues in Canada's health systems, improving coherence between the PCHOs, and implementing operational improvements in Health Canada's management of the PCHO suite (see Appendix C).

Scope and methodology

This evaluation was conducted to fulfill the requirements of the Financial Administration Act and to provide Health Canada management with information to help improve and adjust their activities and relationships with the PCHOs, as necessary.

The evaluation covered Health Canada's administration of the Pan-Canadian Health Organizations and PCHO activities from 2018–19 to 2022–23. In addition, the assessment of activities was based on the goals as outlined in the shared common logic model for the PCHO suite – see Appendix B. This evaluation was not designed to be a comprehensive assessment of all the activities of each of the seven PCHOs. It examined how the suite of the PCHOs contributed to the achievement of shared outcomes. Moreover, the evaluation took a retrospective view and did not assess the future role or the specific objectives of each PCHO.

The evaluation drew on evidence from multiple data sources, including a document and file review, a performance data review, interviews, an environmental scan, and a financial data review. Additional details are provided in Appendix D.

The evaluation addressed questions related to the following three areas of focus:

Evolution of PCHOs

How have the PCHOs evolved in response to the changing environment and changing national/federal priorities over the past five years?

Impacts of PCHOs

What progress have the PCHOs made towards their objectives, individually and collectively?

Management of PCHOs

To what extent is Health Canada's program management contributing to the effectiveness of the PCHOs?

Evolution of the Pan-Canadian Health Organizations

How have the PCHOs and Health Canada evolved over the past five years?

Rapid changes in the Canadian landscape driven by COVID-19, opioid-related deaths, and innovations in technology have been significant in the Canadian health care context, and thus in the priorities and activities of Health Canada and the PCHOs. The PCHOs and Health Canada have taken steps to address areas of potential change highlighted in the Fit for Purpose review, such as program management process improvements and structural changes to increase efficiencies and enhance alignment among the PCHOs.

The PCHOs have been more responsive to national priorities, including those of the federal government, and are increasingly working together to pursue common goals.

Adaptations in response to emerging trends in the Canadian context

The seven PCHOs were established with specific mandates that were defined by the federal government. In the 35 years since the first PCHO was established, both Health Canada and the PCHOs have adapted in response to changing trends, while continuing to advance their original mandates. Over the past five years, the PCHOs have made significant changes to their activities in order to respond to emerging issues in the Canadian landscape, namely COVID-19, the opioid epidemic, and the rapid pace of change in technology in the health care space. There are many examples of how the PCHOs have adapted to these emerging issues. CCSA has developed an opioid resource hub that shares resources developed by partner organizations, and is aligned with the pillars of the Canadian Drugs and Substances Strategy.^{Footnote 2} In response to demands for technological modernization in health care delivery, CIHI and Infoway collaborated together on the "pan-Canadian organ donation and transplantation data and performance reporting system project", including developing data standards, a data repository, and data access capabilities for decision makers.^{Footnote 3} Furthermore, as will be discussed throughout this report, when the COVID-19 pandemic started, nearly all the PCHOs pivoted their work to provide information and support to the national pandemic response.

Several interviewees from the PCHOs and subject matter experts familiar with the PCHO suite also identified increased focus on reconciliation and attention to First Nations, Métis, and Inuit issues; enhanced focus on diversity, equity, and inclusion; representation of people with lived or living experience and patients; and improved capacity to advance work related to sex- and gender-based analysis (SGBA Plus) as key changes in PCHO operations.

A review of strategic plans shows that the PCHOs establish their priorities following consultations with federal, provincial, and territorial governments, and other stakeholders.

Organizational changes within the PCHOs

The PCHOs have also implemented significant changes in how they work. There have been some changes in the structure or shifts in organizations to improve alignment, such as organizational amalgamations and formal collaboration agreements. For example:

Amalgamation of CPSI and CFHI to form HEC

In 2019-20, the Canadian Patient Safety Institute (CPSI) and the Canadian Foundation for Healthcare Improvement (CFHI) announced joint commitment to amalgamate to a new organization, Healthcare Excellence Canada (HEC). CPSI described the impetus for this amalgamation as an "opportunity to achieve safer, higher quality, more efficient, coordinated and patient-partnered health care" by creating "a single quality and safety organization with an expanded capacity to improve health care for everyone in Canada." CPSI noted that they and CFHI "have complementary mandates and goals as pan-Canadian quality and safety organizations, including shared stakeholders. The amalgamated organization will build on our collective responsiveness to the needs of federal, provincial, and territorial governments, as well as patients, families, communities, and valued health system stakeholders."^{Footnote 4}

Transfer of DSEN to CADTH

Based on the recommendations in the final report of the Advisory Committee on Healthcare Innovation, the functions associated with the Drug Safety and Effectiveness Network (DSEN) were transferred from CIHR to CADTH in February 2022. The launch of CADTH's new Post Market Drug Evaluation (PMDE) Program is designed to build on the research methods and analytical expertise across Canada to deliver timely, relevant answers to the post market information needs of decision-makers. CADTH is building on international partnerships and opportunities for common protocols, and common approaches to assessing post market queries.

Partnership between CCSA and MHCC on Mental Health and Substance Use

In 2020, in response to the increasing understanding of the connection between mental health and substance use, the Mental Health Commission of Canada (MHCC) and the Canadian Centre on Substance Abuse (CCSA) signed a Memorandum of Understanding to formalize their partnership and commitment to work collaboratively to address issues at the intersection between mental health and substance use.

Adaptations at Health Canada and within the PCHOs following the Fit for Purpose review

As noted in the "Background and context" section of this report, the 2018 Fit for Purpose review of the PCHOs presented a series of suggested recommendations for adaptations that the federal government should consider in order to improve the coherence of the PCHOs as a whole, and the efficiency with which they are managed by Health Canada.^{Footnote 5}

Since the release of the Fit for Purpose review, Health Canada has taken steps to address some of these suggestions, including an internal assessment of the PCHOs that identified recent changes which had taken place within the PCHOs, and areas for potential future work. Furthermore, in July 2020, Health Canada initiated a "process improvement" review, wherein Health Canada and the PCHOs collectively identified areas of concern in the management of the PCHO suite, along with potential short-, medium-, and long-term actions to address them.

According to many interviewees from both Health Canada and the PCHOs, there has been an increase in collaboration among the PCHOs, driven partly in response to recommendations from the Fit for Purpose review, but also due to the leadership of the respective PCHOs. Several PCHO interviewees also mentioned that the organizations have increased collaboration on back-office operations, such as examining shared office space and launching joint procurement for services like translation. Such efforts were implemented with the goal of achieving greater financial and administrative efficiency. Furthermore, several PCHO interviewees noted that the organizations have shared good practices on important cross-cutting topics or themes, such as patient engagement and engagement with First Nations, Métis, and Inuit communities and organizations. The sharing of such best practices has helped the various PCHOs improve their goal of being more inclusive and having more impact.

At the same time, many PCHOs were left with uncertainty about whether Health Canada would pursue significant changes in the funding agreements following the Fit for Purpose review, potentially affecting the sole or majority funding stream for all PCHOs. Some interviewees from the PCHOs and other subject matter experts reported that efforts to interpret and address the review's suggestions took away from efforts that could otherwise be focused on the organizations' front-line operations.

Are the PCHOs filling a needed role or gap in the Canadian landscape?

The PCHOs play different roles to address existing needs across Canada's health systems. While other organizations may fill similar functions at the provincial-territorial level or in complementary sectors (i.e., private sector), the PCHOs have continued to address gaps in lower-capacity regions, and by convening key players and the federal government on important topics.

Increased collaboration between the PCHOs in the past five years has helped ensure that they are leveraging each others' expertise and working together on shared priorities.

The role of the PCHOs in the Canadian landscape

It is important to note that, while the PCHOs may operate in the same space as other players who address similar issues, this overlap may demonstrate a profound need in a particular area, may reflect the need for diverse or regional responses within Canada's provincial and territorial health systems, or may simply represent multisectoral response across private, government, academic, and non-profit sectors.

Several interviewees from Health Canada, the PCHOs, and subject matter experts noted that the PCHOs fill capacity gaps in smaller or lower-capacity provinces and territories where these functions do not exist, either by providing national leadership or by funding specific initiatives at the regional level. This can be seen in the work of organizations like CCSA, which released a new version of Canada's low-risk alcohol drinking guidelines in early 2023. Furthermore, CPAC funds the implementation of innovative cancer screening programs, such as at-home screening for certain cancers, helping to ensure all regions are able to participate, regardless of capacity.

An environmental scan conducted for this evaluation showed that, while some of the PCHOs share similar functions with other non-governmental organizations in the Canadian landscape, they tend to bring unique strengths. For example, many of the PCHOs have acted as a "convener" of key stakeholders and the federal government, bringing together partners within their networks to address important issues in Canada's health systems through a multidisciplinary and pan-Canadian approach.

Furthermore, in some cases PCHOs leverage their unique expertise and experience to undertake key activities in areas of need.

Infoway

Infoway promotes health system interoperability, encourages uptake of various approaches to digital health, and provides digital health services. They work with health technology vendors in Canada to help ensure the uptake of technical standards required to facilitate the exchange of health data in Canada.

CADTH

CADTH is the only pan-Canadian organization that provides a full suite of evidence products and services to inform decisions about the optimal use of drugs and health technologies (e.g., medical devices, interventions) in our health care system. As a national organization who conducts health technology assessments and drug reviews, it informs the funding of pharmaceuticals including drug formulary decision making at a provincial, territorial and national level.

CIHI

CIHI establishes data standards to promote consistent collection of health information across Canada and analyzes data and information to accelerate improvements in health care, health systems performance, and population health across the continuum of care.

Collaboration between other PCHOs and complementary NGOs

Many interviewees from Health Canada and the PCHOs noted that the PCHOs tend to partner with organizations operating in their same priority areas. The PCHOs have taken great steps to ensure that their work is aligned with each other, that they leverage each other's expertise, and that they avoid duplication as much as possible. There are multiple tables (both standing and ad hoc) where the PCHOs collaborate with each other and share learnings. For example, many interviewees working with the PCHOs pointed to their cross-PCHO Coordination table, co-chaired by HEC and Health Canada, which provides opportunities to identify shared priorities and complementary areas of work.

There have been extensive instances of partnerships between the PCHOs and other non-governmental organizations. Many interviewees from all categories suggested that increased collaboration between the PCHOs, as well as collaboration between the PCHOs and their external partners, has been a hallmark change in the past five years. The PCHOs have

collaborated with each other to advance work addressing shared priorities, such as the collaboration between MHCC and CCSA on cannabis research, including a multi-part webinar series to showcase research findings. The PCHOs also worked with external partners to convene multiple actors to advance priority issues. For example, CPAC engaged over 7,500 people living in Canada, including provinces and territories, cancer agencies, First Nations, Inuit and Métis communities, governments and organizations, non-governmental organizations, patients and the public to develop the 10-year Canadian Strategy for Cancer Control.^{Footnote 6} See Appendix G for additional examples of collaboration.

Impacts of the Pan-Canadian Health Organizations

In 2021 and 2022, the Office of Grants and Contributions at Health Canada led the development of a shared logic model in collaboration with the PCHOs' Performance Measurement and Evaluation Community of Practice. The purpose of this logic model was to better articulate the alignment of goals of individual PCHOs with the collective work of the PCHO suite. Given the diversity of work undertaken by the seven individual PCHOs, not every one of them contributes to all expected outcomes in the logic model. From this logic model, a selection of performance indicators was identified and included for review in this evaluation. See Appendix B for the logic model.

While the development of a common logic model enables Health Canada to communicate the shared impact of the PCHOs more effectively across their diverse activities, some indicators included in the shared logic model do not clearly measure the outcome with which they are aligned. It may be beneficial to re-examine the selected indicators, remove those that do not describe the shared outcomes and replace them with new ones. Such efforts should help improve reporting to be better able to tell the collective performance story of the PCHO suite.

Impacts on capacity

The PCHOs were able to demonstrate progress with respect to increasing capacity among health care practitioners, governments, and health care delivery organizations despite the disruption of knowledge mobilization activities due to COVID-19.

As outlined in the common logic model, PCHO key activities are expected to increase capacity for health care practitioners, governments, and health care delivery organizations by building knowledge and skills among health practitioners, providing health care delivery organizations and government decision makers the means to make improvements in health care services, creating partnerships between health care delivery organizations to implement health care solutions, and engaging government decision makers to implement initiatives across health systems.

Health practitioners are equipped with knowledge and skills to address the population's health care needs

To build knowledge and skills, the PCHOs produced a variety of products that are accessible to key health care stakeholders. For example, CIHI produced reports focusing on priority themes such as seniors in transition, dementia, alcohol harm, children and youth with mental disorders, asthma hospitalizations among children and youth, and unnecessary care in Canada. Health system planners, decision makers, care delivery managers, and health researchers are among the users of CIHI data.^{Footnote 7} They use this data to inform policies, their research, models of care, and to aid in their decision-making processes. ^{Footnote 8, Footnote 9}In addition, in 2022-23, CADTH produced 99 drug reimbursement reviews, 113 health technology reviews and 57 implementation support tools for government decision makers and health care administrators on a wide range of topics such as the report on the treatment of Mpox.^{Footnote 10}

The PCHOs also developed learning opportunities and tailored knowledge products for health practitioners. For example, in response to COVID-19, HEC hosted the "Virtual Learning Together Series", a series of webinars connecting emergency shelters and substance use centres to share lessons learned and good practices. An evaluation of the initiative found that the program was increasing awareness of strategies to support clients and staff during times of crisis (88% of participants). Furthermore, CCSA partnered with the Alberta Family Wellness Initiative to launch the Brain Builders Lab project, aimed at mobilizing Brain Story science into action. The Brain Story is a training resource that explains how the brain develops through childhood, and how adverse childhood events can influence neurodevelopment and lead to the development of substance use disorders later in life. Across the 25 Brain Builders Lab projects, all reported increased knowledge and awareness among their stakeholders. Meanwhile, Infoway developed peer leader networks and clinician-in-training support programs that helped clinicians learn from one another as they adopted digital health tools. Infoway also partnered with Healthcare Excellence Canada to develop the Virtual Care Toolkit guide to support clinicians with their use and implementation of virtual care.^{Footnote 11}

The available performance data shows that the PCHOs tended to increase the reach of their products to health care providers and increased the knowledge of those stakeholders in a relatively high proportion of participants. For example:

CIHI

According to survey data in 2020-21, over 80% of stakeholders reported satisfaction with the usefulness of CIHI tools and products.

CPAC

Between 2018-19 to 2022-23, the number of approved requests for access to the harmonized Canadian Partnership for Tomorrow's Health (CANPATH) dataset and bio sample repository increased from two to 22.

MHCC

The percentage of stakeholders who report they are better equipped with the knowledge they need to address mental health and wellness issues as a result of knowledge products received remained high during the period covered by the evaluation: from 89.6% in 2018-19 for stakeholders in general to 86% for health care providers and organizations in 2021-22.

Health care delivery organizations and government decision makers are equipped with knowledge to make improvements in health care services

The PCHOs have undertaken activities to ensure that health care delivery organizations and government decision makers are equipped with knowledge to make improvements in health care. For example, to empower first responders, medical and public health professionals, and decision makers across Canada, CCSA provides up-to-date data on early warnings and trends in substance use, mainly through the Canadian Community Epidemiology Network on Drug Use (CCENDU) alerts and bulletins on drug use trends or topics of interest. CCSA also maintains the flow of key information for policy makers across Canada by regularly updating the Canadian Substance Use Costs and Harms Report to provide a national profile of estimated substance use costs by lost productivity, health care, criminal justice, and other direct costs, along with a jurisdiction-based data visualization tool for provinces and territories.

Other available performance data show some progress towards equipping health care delivery organizations and government decision makers with knowledge to make improvements in health care services. For example:

CADTH

The proportion of decision makers reporting that CADTH's evidence-based supports were useful in the context of decision making increased from 73% in 2018-19 to 95% in 2020-21.

CCSA

Between 2018-19 to 2021-22, the percentage of stakeholders reporting increased knowledge of the nature and harms of substance use and of best practices and other evidence-informed innovations increased from 86% to 89%.

HEC

Between 2018-19 to 2021-22, the percentage of health care leaders who reported knowledge acquisition in Quality Improvement participating in programing from the former Canadian Foundation for Healthcare Improvement (CFHI) and its successor organization, HEC, stayed relatively constant (over 90%), with the total number of leaders reporting knowledge acquisition increased from 274 to 1,495. In 2022-23, 95% of leaders participating in HEC activities (2,117/2,220) reported their engagement increased their preparedness to lead improvement, which is HEC's new measure equivalent to knowledge acquisition.

However, some organizations reported lower progress than anticipated in this area, based on indicators selected to measure progress on this outcome. For example, according to 2020-21 survey data, 18% of stakeholders reported an increased capability to use CIHI's products and services in their work setting. While this proportion is below the organization's target of 60%, results should be interpreted with caution due to a low response rate. It is also possible that the pandemic influenced stakeholder's capacity to seek out and use CIHI products.

Health care delivery organizations partner to implement health care solutions

Partnerships are often important to ensuring the success of an initiative. Documents show that the PCHOs have consistently partnered with various health care organizations. CCSA partnered with the Royal Ottawa Mental Health Centre and the Community Addictions Peer Support Association to raise awareness and improve wellness for those who use substances through a community-driven approach. In addition, MHCC collaborated with Fraser Health Authority to evaluate, replicate and scale-up the Trauma Informed Resiliency Program (TRIP), a program to reduce stigma and enhance resiliency among health care providers and provide improved service delivery for those who use opioids and other substances.

Furthermore, HEC collaborated with CPAC to work with paramedics in multiple provinces on the Paramedics and Palliative Care initiative. Through this initiative, HEC and CPAC supported partners to implement a training program which enabled paramedics to provide palliative care to patients in their homes. HEC's EXTRA executive training program is a team-based coaching initiative in which executives from health care organizations participate in peer-to-peer learning and guidance to support implementation of Quality Improvement projects and lead transformational change. Quality Improvement projects are systematic, data driven, and action-oriented projects aiming to develop and implement initiatives in order to make specific improvements in health care settings. Healthcare Excellence Canada supports various Quality Improvement projects, for example the LTC+ Acting on Pandemic Learning Together (LTC+) project.^{Footnote 12} The EXTRA program has been running for 17 years and has garnered participation from over 500 executives from 168 health care organizations.^{Footnote 13}

Other PCHOs have also enhanced capacity directly through partnerships with hospitals. For example, CIHI provided capacity building sessions with hospitals, including the Yukon Hospital Corporation, to increase awareness and understanding of CIHI's data sets and tools. Similarly, Infoway partnered with the Ottawa Hospital in August 2022 to undertake a pilot launch of PrescribeIT, including integration into their hospital information system. PrescribeIT is Infoway's national e-prescribing service that enables prescribers to send prescriptions and renewals electronically to a patient's pharmacy of choice, resulting in safer and more efficient patient care, as well as improved communication between clinicians.^{Footnote 14}

Limited performance data were available on the extent to which health care organizations partner to implement health care solutions as a result of PCHO projects/activities. However, the available data show that some organizations increased partnerships during the period covered by the evaluation. For example:

CPAC

CPAC established and enhanced partnerships with First Nations, Inuit and Métis governments/ territories, organizations and associations. For instance, CPAC signed a relationship agreement with the First Nations Health Managers Association and as part of its work on implementation of the Canadian Strategy for Cancer Control is providing funds to 29 partners, who are collectively working with over 130 First Nations, Inuit and Métis governments, organizations, and community partners to develop and implement Peoples-specific, First Nations, Inuit and Métis cancer priorities in each province and territory.^{Footnote 15}

MHCC

Since 2018-19, MHCC has expanded partnerships from 271 to more than 450 (with a peak of 555 in 2019-20).

Government decision makers are engaged to implement initiatives across health systems

A review of documents and performance data found that several of the PCHOs engaged with government decision makers to implement health care initiatives across Canada's health systems. The PCHOs accomplished this by developing tools and platforms that jurisdictions could use to inform decision making, and by engaging directly with decision makers through partnerships, meetings, and other mechanisms.

Infoway

Infoway worked with government representatives from Health Canada, the provinces and territories, and health sector stakeholders to develop the Shared Pan-Canadian Interoperability Roadmap – a long-term vision towards improving health information exchange and health care services for Canadians. The Roadmap aims to improve how all levels of government collect, share and use health information by adopting common standards and policies.

CIHI

CIHI has worked with various levels of government, as well as with stakeholders, measurement experts and the public, to select and develop a set of 12 pan-Canadian health systems performance indicators to measure improvements in access to home and community care, and to mental health and addictions services.^{Footnote 16}

CADTH

At the request of the federal government, CADTH struck a multi-disciplinary panel to develop three prototype medication lists and to propose principles and criteria for listing and delisting medications in a future national formulary.

CPAC

CPAC developed a modelling platform called "OncoSim," which is an evidence-based tool that allows decision makers to compare the costs and impacts on policy decisions on cancer rates, deaths, and health care costs. Over the last five years, OncoSim was used 155 times by researchers and analysts to inform policy decision-making in the cancer care system.

CCSA

CCSA's Competencies for Canada's Substance Use Workforce project provides a framework for workers, employers and administrators in the substance use and addictions field to assess current skill sets and identify ongoing learning needs to ensure best practices are followed and equitable, non-stigmatizing services are provided. These resources have been partially or fully adopted by numerous stakeholders and jurisdictions including the government of Ontario, Alberta, British Columbia, Nova Scotia, and New Brunswick.

HEC

First as CFHI and now as HEC, it has supported the Canadian Northern and Remote Health Network, which brings together senior decision makers, leaders, policy makers, and practitioners, to develop effective and sustainable solutions to improve health care and the health status of people living in northern and remote areas of Canada.

MHCC

MHCC developed a program called "Roots of Hope," a multi-site, community-led project that aims to reduce the impacts of suicide within communities across Canada. To support expansion of the initiative in interested communities, MHCC launched a complementary initiative called "Early Adopters" that supported more than 11 communities and the governments of New Brunswick and Yukon in implementing the program.^{Footnote 17} To date 18 communities and four provinces/territories are participating, including participation from 10 out of 13 provinces/territories.^{Footnote 18}

However, performance indicators specifically measuring the engagement of government decision makers were not always available during the evaluation period. In addition, several indicators currently included in the logic model do not clearly illustrate the achievement of this outcome.

One indicator did provide an indirect measure of the engagement from government decision makers to implement initiatives. CPAC's measure of this goal was the implementation of lung cancer screening by provinces and territories. The number of the 13 provinces and territories that were reported to have "taken steps towards implementing lung cancer screening programs" decreased from 11 to 10 between 2018-19 and 2021-22. Health Canada representatives noted that CPAC will continue to work closely with outstanding jurisdictions to gauge readiness to initiate planning for organized lung cancer screening.

Impacts on behaviour change

Performance data and evidence from internal documents and recipient evaluations suggest that PCHO activities have supported progress towards desired behaviour changes at individual and health systems levels.

At the individual level, positive changes included application of knowledge products to improve care, though results were varied across the PCHOs. At the health systems level, available performance data shows moderate increases in the adoption of certain health care initiatives and recommendations, as well as improvements in organizational culture.

Building on the PCHO activities described above to build capacity among health care practitioners, health care delivery organizations, and government decision makers, behaviour change is expected among individuals working in the health sector, and behaviour changes at the level of provincial and territorial health systems.

Individual level changes

At the individual level, expected behaviour changes targeted at health practitioners include:

• health practitioners offer health care services that are responsive to the needs of priority populations; and
• health practitioners offer person-centered health care in targeted areas.

PCHOs have shown mostly positive progress in the extent to which health practitioners offer health care services that are responsive to the needs of priority populations. For example, CIHI provides data on priority populations to meet the needs of their stakeholders. In CIHI's 2018-19 annual report, they noted that their strategic directions were tailored to meet the needs of priority populations identified by their stakeholders: seniors, those living with mental health and substance use issues, First Nations, Inuit, and Métis, and children and youth.^{Footnote 19}

In their performance data reporting, PCHOs have also highlighted progress in meeting the needs of priority populations by achieving increased satisfaction with virtual health options among patients, and increasing the proportion of outpatient cancer treatment centres that offer patients smoking cessation support. Infoway works with governments, health care organizations, clinicians, and patients to make health care more digital in order to facilitate faster, more seamless, and more secure information sharing. ^{Footnote 20} Virtual care can benefit a variety of patients, such as persons with mobility issues, residents living in remote areas, and busy parents and workers.^{Footnote 21} Cancer is a leading cause of deaths in Canada and associated treatment is more effective when a person with cancer stops smoking.^{Footnote 22} As such, CPAC included smoking cessation support as part of its work with partners to improve the quality of cancer treatment programs. In addition, MHCC focused some of its activities in the North and among Indigenous populations.

Infoway

The proportion of patients satisfied with virtual care, including tele-homecare, rose from 81% in 2018-19 to 86% in 2021-22. During the same period, the proportion of patients reporting satisfaction with using patient portals increased from 70% to 86%.

CPAC

The proportion of outpatient cancer treatment centres offering patients smoking cessation support increased from 66% in 2018-19 to 95% in 2022-23.

MHCC

Following e-mental health implementation in the NWT, between March 1, 2020, and October 27, 2022, 72% of service users surveyed were satisfied with wait times for counselling; 81% of Indigenous service users surveyed were satisfied with the safety of the counselling environment; and 73% of service users surveyed were satisfied with their overall counselling experience.^{Footnote 23}

However, some of the PCHOs reported decreasing or below-target achievement of indicators on this goal. For example, the proportion of stakeholders who agreed that MHCC knowledge products were used to inform mental health policies, behaviours, and practices in the area of substance use, suicide prevention, stigma reduction, and population-specific initiatives decreased from 90% in 2018-19 to 35% in 2021-22. While MHCC met its target of 85% in 2018-19, it did not meet its target of 50% in 2021-22.

MHCC noted that this drop may be in part due to the timing of survey distribution and the COVID-19 pandemic. Due to the pandemic, MHCC had fewer opportunities to promote their knowledge products to intended audiences. MHCC took steps to address this, including convening an internal working group. In the final year that data was available (2022-23), the PCHO improved the results and was close to target (47%).

Evidence from documents show that the PCHOs advanced several initiatives aimed at encouraging health practitioners to offer person-centred health care in targeted initiatives. For example, CADTH and Choosing Wisely Canada convened an expert panel of clinicians, policy experts, and patient advisors to identify actions that health systems could take to improve post-pandemic care.^{Footnote 24} Another targeted area of work advanced by HEC was Appropriate Use of Antipsychotics (AUA) in long-term care facilities. This person-centred approach to caring for seniors living with dementia involves reviewing the use of antipsychotic medication and discontinuing or reducing prescriptions if they are no longer needed to reduce negative health outcomes and improve patient care experiences.^{Footnote 25} During each year of the evaluation period, the AUA initiative was scaled up into hundreds of new care home facilities and new jurisdictions in New Brunswick, Newfoundland, PEI, Vancouver, and in Quebec under the name "Optimizing Practices, Use, Care and Services for Antipsychotics, or "OPUS-AP".

Furthermore, CPAC and HEC's Paramedics and Palliative Care initiative also serves as an example of scaling up implementation of a person-centred model of care. To avoid unnecessary trips to the emergency room and allow patients to receive care aligned with their preferences, this initiative trained thousands of paramedics to provide palliative and end-of-life care in homes. An evaluation of the initiative found it to be highly effective: in approximately 7,000 palliative care 911 calls, only 47% required hospital transport, as compared to about 90% before program implementation.^{Footnote 26}

Performance metrics measuring the extent to which health care providers adopted person-centred models of care show improvements over the evaluation period, for example:

CPAC

The proportion of paramedics trained in eligible regions and jurisdictions to provide palliative care in a patient's home increased from 4% in 2018-19, to 98% in 2021-22.

HEC

The proportion of improvement teams that reported making improvements in the health of patients and residents resulting from their Quality Improvement project rose from 89% to 94% in 2021-22. Similarly, the proportion of CPSI project teams reporting improved outcomes rose from 82% to 92% during the years where data was available (2019-20 and 2020-21).

Health systems level changes

At the health systems level, PCHO goals are related to changes in health care delivery organizations, governments, and among researchers and other stakeholders. These include:

• Health care delivery organizations testing health care solutions;
• Health care delivery organizations and government decision makers adopting and implementing proven health care solutions in targeted areas;
• Government decision makers widely implementing health care solutions; and
• Health stakeholders and researchers having access to pan-Canadian data and research findings to support their decision-making processes.

Where it was available, performance data shows a moderate increase in adoption of PCHO tools and evidence, use of tools and evidence by decision makers, and improvement in organizational culture. Organizations like CPAC and Infoway demonstrated broad adoption of evidence and tools, while HEC showed that their improvement teams were seen to improve organizational culture related to health care. For example:

Infoway

Between 2018-19 to 2021-23, the number of clinicians providing virtual care increased from 2,934 to 164,000 and Electronic Health Record system users increased from 330,000 to 418,000.^{Footnote 27} Between 2018-19 to 2021-22, the number of physicians using Electronic Medical Record systems increased from 35,940 to 42,900.

CPAC

The number of provinces and territories that have used CPAC information to implement strategies to reduce disparities in cancer screening increased from nine provinces and territories in 2019-20 to 11 in 2021-22.

HEC

The proportion of improvement teams that reported advancements in their organization's culture related to health care practices and delivery tended to hover between 96% and 100%.

Limited performance data was available on the extent to which health care institutions are testing solutions, however evidence from documents suggests that some of the PCHOs have piloted new approaches to health care and tracked progress on these new solutions. For example, Infoway supported the MyHealth Records project, which was launched province-wide in Alberta in March 2019 and is now fully deployed and has been used by over 1.5 million Albertans.^{Footnote 28} MyHealth Records gives citizens access to some of their health information from Alberta Netcare, the provincial electronic health record. Furthermore, CPSI, the predecessor to HEC, worked with partners to implement and evaluate measurable and sustainable "safety improvement projects" that align with pan-Canadian priorities in order to demonstrate what works in patient safety.^{Footnote 29} Performance data shows that the number of teams involved in safety improvement projects stayed around 30 in the years covered by the evaluation, exceeding their goal of 20 teams.^{Footnote 30} Performance data from some of the PCHOs shows that government decision makers had implemented health care solutions due to PCHO activities, for example:

Infoway

Infoway has worked with provinces and territories to co-invest in foundational systems for EHR in health care settings and support pan-Canadian projects for a common architecture, such as the EHR blueprint, launched in 2006 and updated in 2016.^{Footnote 31, Footnote 32} Between 2018-19 and 2020-21, the proportion of physicians participating in projects reporting that they can share patient health information with providers outside their practice via electronic health records increased from 24% to 35%, and the proportion that could share via electronic medical records increased from 25% to 36%.

HEC

In 2020-21, 45% of bodies targeted by CPSI adopted a policy, standard or regulation incorporating evidence-informed patient safety requirements, and the proportion of HEC and CFHI improvement teams reporting creation of new, updated, or revised policies, standards, or guidelines from a project increased from 42% to 78% between 2018-19 and 2020-21.

The PCHOs made some contributions to improved access to pan-Canadian data and research findings for health stakeholders and researchers to support their decision making. For example:

CADTH

The pan-Canadian Pharmaceutical Alliance, which conducts joint provincial, territorial and federal negotiations for brand name and generic drugs in Canada, took up 99% of CADTH's recommendations for full or conditional reimbursement.

CIHI

CIHI provided access to pan-Canadian data on a range of health indicators. As of December 31, 2022, there were 28 data holdings which researchers, decision makers and health managers could use to request specific data to suit their information needs.^{Footnote 33} These data holding categories include data related to the continuum of care, such as data on finances, pharmaceuticals, community care, and patient experience, among others.^{Footnote 34}

CPAC

CPAC increased the number of provinces and territories contributing data to cancer data systems performance reports from 3 to 13 in the period covered by this evaluation.

Impacts on health systems

Evidence from documents and some performance data shows that, together, the PCHOs have made some progress towards their goals related to improved health care experiences for people who live in Canada, the spread of health care improvements, and use of PCHO products by decision makers to inform decisions.

The ultimate goals for the PCHOs described in the common logic model are related to expected improvements in patient experience, evidence for decision making, and health care delivery that come from the desired changes in capacity and behaviour.

Canadians experience safe, timely, appropriate, person-centred, inclusive, and coordinated care

Available performance data shows some improvements in Canadians' actions resulting from mental health training, and improved efficiency in health delivery organizations resulting from increased uptake of good practices. For example, as a result of MHCC training, the proportion of Canadians who reported taking action to improve their mental health increased from 69% in 2018-19 to 72% in 2021-22. Furthermore, Canadians have had improved access to health services through virtual care and implementation of electronic health information resulting from Infoway's activities.

Infoway

As of March 31, 2021, 3.5 million Canadians and more than 91,000 health care providers had logged more than five million uses of virtual care solutions as a result of Infoway investment in 34 projects across all jurisdictions.^{Footnote 35, Footnote 36} Infoway reported an increase from 36% to 45% of patients who were able to avoid a trip to the doctor or emergency room after accessing virtual care.

Health care improvements are spread and sustained across health systems

While there was limited performance data available for this indicator, available data from HEC (CFHI) shows that Quality Improvement projects continued to be scaled beyond their original sites. The proportion of teams that reported further spreading of Quality Improvement projects beyond their original implementation site increased from 43% to 87% between 2018-19 and 2020-21. In 2021-22, 60 teams (72%) reported the spread of the project beyond its original implementation site.^{Footnote 37} Additionally, the proportion of teams demonstrating improvement in project-specific patient safety practices increased from 10% in 2018-19 to 53% in 2020-21. The following year, as CPSI transferred to HEC, 95% of improvement teams reported making improvements to patient, residential, and essential care partners' experience of care. In 2022-23, HEC demonstrated that 87% of settings who reported on the indicator were formally working to spread a practice or behaviour and 81% sustained a practice or behaviour change for six months following implementation.

In addition, the MHCC has worked to spread and sustain Stepped Care 2.0 which integrates e-mental health interventions with traditional in-person programming on a provincial scale. It was tested in Newfoundland and Labrador and has since been expanded to Nova Scotia, NWT, PEI and discussions are underway in other provinces/territories to implement it.^{Footnote 38}

Due to limitations in the relevance of indicators and the availability of performance data, it was not possible to fully assess the extent to which other PCHOs have contributed to progress in this area.

Health care decision makers use evidence and data to make improvements and investments in the health care system

Stakeholders are using PCHO products to inform decision making in health systems. For example:

CADTH

Decision makers took CADTH's recommendations for drug reimbursement more than 90% of the time during the period covered by the evaluation.

CIHI

In 2021-22, 87% of stakeholders reported using evidence from a CIHI knowledge product or service to support decision making. For example, Labrador-Grenfell Health, a regional health authority) used CIHI data to identify high rates of usage of antipsychotics and restraints, and high rates of falls in long-term care facilities. As a result of this information, Labrador-Grenfell Health set annual targets to decrease the use of antipsychotic medications and restraints in order to reduce the incidence of falls in long-term care facilities.

CPAC

All provinces and territories were using data from the Prevention Policies Directory and CAREX Canada to inform decision making for policy change in the years covered by this evaluation. The Prevention Policies Directory, created by CPAC, made it easier for public health researchers and policy specialists to find policies related to cancer and chronic disease prevention.^{Footnote 39} CAREX Canada, primarily funded by CPAC, provides information about Canadians' exposures to known and suspected carcinogens where they live and work.^{Footnote 40}

Factors facilitating and inhibiting progress on PCHO goals

Progress on PCHO goals was facilitated by strong relationships among the PCHOs, their partners, and with Health Canada. While the COVID-19 pandemic caused delays to some PCHO planned activities, it provided an opportunity to demonstrate responsiveness to emergent public health issues. Inflationary pressure and increased competition for human resources also created some challenges in recruiting and retention, which may have slowed work in some areas.

Factors facilitating success

Interviewees from Health Canada and the leadership of the PCHOs were asked about factors that helped facilitate progress towards their goals. Both groups identified the following factors:

• Regular collaboration among the PCHOs, and between the PCHOs and Health Canada.
• Strong and deliberate partnerships and collaborations with health care and provincial and territorial government stakeholders, and with patients and people with lived and living experience.
• Improved relationships with First Nations, Métis, and Inuit communities and organizations.
• Ongoing funding for most of the PCHOs, as well as elements of the funding agreement, including the ability to retain unexpended funds in the following fiscal year, and the shift from activity-based reporting to outcome-based reporting, allowed flexibility for the PCHOs to pivot to new activities in response to emerging priorities.

Staff from Health Canada also noted the benefits of the PCHOs' arms-length position. Through this model, Health Canada provides funding through a contribution agreement to help deliver on identified health system priorities, but organizations can define their specific activities and operations with autonomy. As a result, interviewees felt that the PCHOs were able to be nimbler and pursue progress in areas that would not be possible in a federal government organization.

Furthermore, PCHO leaders noted that improved direction from Health Canada and ongoing support and guidance from the Office of Grants and Contributions on issues related to contribution agreements, such as reporting and renewals, were facilitators to success, along with the expertise and contributions of staff working at the PCHOs.

Barriers to success

Some interviewees from the PCHOs and Health Canada also described challenges that inhibited achievement of PCHO goals in some areas. Notably, multiple demands from the federal government sometimes led to PCHO overcommitment. According to some PCHO interviewees, they did not want to stop working on legacy programs and they also wanted to respond to shifting priorities. In some cases, such overcommitment presented challenges in having adequate capacity to deal with all demands, which sometimes resulted in project delays and shifting of deadlines.

Interviewees from the PCHOs also described issues related to funding for human resources. For several organizations, unchanged funding for staffing, combined with wage increases due to inflation and scheduled salary increases, diminished capacity to recruit and retain staff. Others commented that inflationary pressures and unchanged funding was instead eroding the funding available for programs.

Impacts of the COVID-19 Pandemic on planned work

Many interviewees from Health Canada, the PCHOs, and PCHO subject matter experts highlighted that the COVID-19 pandemic provided both opportunities and obstacles to the achievement of their goals.

Due to COVID-19, the PCHOs prioritized activities that responded to the pandemic, which in some cases meant that it was necessary to pause or delay other planned work. For example, some of the PCHOs reported that they had to delay strategic planning work scheduled to take place in 2019-20 or 2020-21, due to the fact that in-person engagement with stakeholders was no longer feasible because of travel restrictions.^{Footnote 41} Interviewees from the PCHOs and Health Canada described delays in some key initiatives, such as finding hospital partners to participate in health care improvement collaborations, or delays in work like CIHI and Infoway's initiative on Organ and Tissue Donation and Transplantation as clinicians, stakeholders, and provinces and territories were less available to be involved in the project.

In general, these delays were beyond the control of the PCHOs, as staff were redeployed, some external PCHO-initiative partners (i.e., provincial and territorial governments, health care organizations) had reduced availability due to increased workload, and new projects were initiated to support the national pandemic response. Interviewees from Health Canada and some of the PCHOs reported that planned work initiated in response to the Fit for Purpose review (i.e., program management process improvements, work to pursue structural changes in some organizations) was deprioritized at Health Canada due to competing demands to support the pandemic response.

It appears the PCHOs were able to continue to progress on planned work in addition to new demands associated with the pandemic by shifting timelines and, in some cases, carrying over funding that could not be spent (i.e., for travel, in-person engagement, in-person knowledge mobilization) into the following fiscal years when public health measures were relaxed.

At the same time, COVID-19 provided an opportunity for the PCHOs to shine a light on issues like mental health and substance use, and demonstrate their ability to step up and address unmet needs at the national level. PCHO representatives noted that COVID-19 also provided an opportunity to clarify where joint priorities existed between Health Canada and the PCHOs and this led to an openness to try new things. Documents and interviews illustrated many new initiatives launched in response to information needs during COVID-19. Examples of such work include:

Infoway

In response to COVID-19, Infoway led the development of pan-Canadian standards on secure messaging and videoconferencing for digital health and virtual care solutions to help provinces and territories in the implementation of new initiatives on virtual care and digital health in 2020.

CADTH

CADTH conducted evidence appraisal, rapid synthesis, and horizon scanning related to emerging therapies and products for COVID-19 to help inform discussion on the implementation of new technologies, and engaged in knowledge exchange activities, including a virtual COVID-19 lecture series attended by over 18,000 participants.^{Footnote 42}

CCSA

CCSA built and maintained a COVID-19 resource hub, which synthesized evidence, best practices, and expert insights on the impact of COVID-19 on substance use from more than 150 resources from across Canada and around the world to inform decision makers, care providers, and the Canadian public.^{Footnote 43}

CCSA also collaborated with MHCC to conduct public opinion research on the impact of the COVID-19 pandemic on mental health and substance use among people living in Canada.

CIHI

CIHI developed an interactive "Health System Capacity Planning Tool" to help decision makers understand expected health resource demands and supply shortfalls related to the COVID-19 pandemic.^{Footnote 44} This tool was used in several jurisdictions to demonstrate the positive impact that public health measures could have on hospital capacity.

MHCC

MHCC developed an online COVID-19 hub and published resources specifically addressing the impact of COVID-19 on specific populations, such as impacts on early childhood mental health, people living with serious mental illness, and rural and remote mental health and substance use.^{Footnote 45}

Health Canada's management of PCHO suite

How effectively have Health Canada's management processes and practices supported the PCHOs?

The relationship between Health Canada and the PCHOs, and among the PCHOs, has evolved significantly over the past five years, with the addition of an extensive set of formal and informal mechanisms to facilitate collaboration and information exchange, particularly relative to the period before the Fit for Purpose review was completed.

Health Canada's management processes and practices supported the PCHOs by improving information sharing between Health Canada and the PCHOs and helping the PCHOs better navigate government processes to allow them to focus on delivering their mandates.

Health Canada's management of the PCHO suite has improved over the evaluation period. The evaluation found that there are now many formal mechanisms in place to help support the management of the PCHOs. For example, Health Canada has a full voting member on most PCHO boards of directors. Where the Department does not have a voting member, they have ex-officio or 'observer' membership. According to many interviewees from both Health Canada and the PCHOs, the Health Canada representatives on boards are active participants in board meetings, including sharing information and participating in decision-making discussions.

In addition to these formal mechanisms for Health Canada to share information and contribute to PCHO decision making, the PCHOs have also established various tables to facilitate collaboration and information sharing between each other, sometimes with Health Canada's participation. Participation in these groups has enabled Health Canada to better align with PCHO work. For example, through the Community of Practice on Performance Measurement and Evaluation, Health Canada led the development of a common logic model using indicators already collected by the PCHOs.

Several informal mechanisms have been established to support the relationship between Health Canada and PCHO management, such as bi-weekly or monthly meetings between the PCHOs and Health Canada, along with regular ongoing working relationships at the operational level via emails and informal communication. There are also regular and ad-hoc bilateral meetings between the senior leadership at Health Canada and the Chief Executive Officers of the PCHOs. See Appendix E for details on the frequency of meetings between Health Canada and PCHO leadership.

Overall, interviewees from the PCHOs expressed generally positive views on Health Canada's roles in governance bodies and their support to the PCHOs, noting that they felt Health Canada's participation was consistent and meaningful. They noted that the relationship between Health Canada and the PCHOs has supported improved collaborative decision making and information sharing, and has provided a venue for interpretation of messages coming from the federal level. When asked about the benefits of the formal governance bodies described above, a few interviewees working with the PCHOs and in Health Canada noted that they enable the Department to help the PCHOs navigate government processes and can help communicate upcoming federal priorities and mandates. For example, when CFHI and CPSI were amalgamating, Health Canada was able to support their engagement with the Canada Revenue Agency, in order to ensure that progress continued during COVID-19 administrative slow-downs.

To what extent has Health Canada efficiently and effectively managed for results with the PCHO programming?

Health Canada has taken steps in the past five years to improve the efficiency with which they manage contribution agreements for the PCHOs. This work has included improvements, such as aligning conditions for funding in Contribution Agreements and shifting from activity-based reporting to outcome-based reporting in order to maximize flexibility for the PCHOs.

In the past five years, there has been significant work undertaken to adjust and align the conditions under which the contribution agreements are administered across the PCHOs. While some variations still exist, changes have been introduced to improve flexibility for the PCHOs.

The structure of the Contribution Agreements for the PCHOs are set up so that they report on their outcomes, rather than only on their activities. According to many interviewees from both PCHOs and Health Canada, this structure allowed for greater flexibility for the PCHOs to pivot their activities throughout the duration of their funding agreements. While this feature of contribution agreements was generally viewed positively, one PCHO interviewee noted that the organizations do not always have latent capacity to pivot to new requests, and therefore need to increase staff when Health Canada issues a new request for support in particular policy areas. They reported that this was particularly challenging due to delays between the request and issuance of new funding.

Health Canada also supported flexibility by allowing the PCHOs to retain unexpended funds from their contribution agreements at the end of the fiscal year. Several interviewees from both PCHOs and Health Canada viewed this feature positively, noting that it allows for longer-term planning and helps avoid extraneous expenditures at the end of the fiscal year. According to these interviewees, this was particularly useful during COVID-19, when many planned expenditures were delayed due to public health measures limiting travel and in-person services, and recruitment and retention of staff was difficult due to their movement to other responsibilities or competing work in the pandemic response. While Health Canada has released funding equal to planned budgets (see Appendix F), records of cash flow and expenditures provided by the PCHOs indicate that several of the PCHOs have used the option to carry over unspent funding from one fiscal year to the next. Several organizations, including Infoway, CADTH, CCSA, CPAC, and HEC, reported spending between 70-85% of their total planned budgets for core funding for one or more years in between 2019-20 to 2021-22, in part due to COVID-19 and subsequent disruption in knowledge mobilization activities, travel, recruiting, and human resources.

Health Canada approved retention of unexpended funds for all of the PCHOs in at least one year covered by the evaluation. Approved amounts of retained funding ranged from 1% of total planned budget up to 20% of total planned budget for core funding in a given year, though in most cases, the amount was below 15%.

According to records of expenditures provided by funding recipients in 2018-19, MHCC and CCSA were unable to spend most of their planned additional funding for Cannabis due to delays in the release of the funding. Further delays in acquiring licensing agreements were beyond the control of the two PCHOs and caused additional delays. In the following years of 2019-20 and 2020-21, CCSA spent under 40% of their planned budget for Cannabis due to external delays related to research work resulting from COVID-19. In 2018-19, both MHCC and CCSA received approval to retain unexpended balances of 50% or more of their planned budget due to these delays which allowed them to move some activities to future years.

What opportunities for improvement, if any, exist?

Overall, representatives from the PCHOs were positive in their assessment of Health Canada's management of their contribution agreements and their overall relationship. At the same time, areas for continued improvement were identified: addressing instability during funding renewal, prioritizing requests to the PCHOs, and strengthening internal coordination within Health Canada.

Representatives from Health Canada and the PCHOs were generally quite positive about the relationship between Health Canada and the PCHOs, noting that Health Canada's management of the PCHO suite has improved, particularly relative to the period before the Fit for Purpose review. However, interviewees from both Health Canada and the PCHOs identified some areas where there were opportunities for continued improvement.

Prioritizing requests from Health Canada and the Health Portfolio

Health Canada interviewees noted that, because the PCHOs are arms-length organizations, there is a need to balance how much direction is given so that the organizations have the autonomy to respond to the needs of their other stakeholders. However, some of the PCHO representatives expressed that there could be improved clarity on what issues are a priority from the federal perspective.

There has been a great demand for the PCHOs to respond to federal requests in the past five years due to COVID-19 and other emerging issues. The PCHOs have viewed this as an opportunity to demonstrate their responsiveness and support the national response as members of the "team." However, when priorities shift, there is an opportunity to clarify how these requests affect the PCHO's progress on their workplans and balance these requests, particularly for organizations that are funded only in part by Health Canada, and whose workplans must also respond to the needs of their other funders. However, there was a desire expressed by some interviewees from the PCHOs for recognition that responding to additional rapid requests from Health Canada with short turnaround times can detract from progress on the activities in their workplans. Furthermore, while the PCHOs were generally able to continue their planned work in addition to new COVID-19 requests, interviewees from the PCHOs noted that this created considerable pressure on staff within the organizations, who were often responding to more work with the same resources. This has contributed to staff recruitment and retention challenges experienced by some PCHOs.

Key informants noted that there are also opportunities for Health Canada to improve internal coordination and ensure consistent strategic direction to the PCHOs when working with multiple groups across the Department, and across the Health Portfolio (this primarily includes Health Canada, the Public Health Agency of Canada, and the Canadian Institutes of Health Research). Some interviewees from the PCHOs noted that they occasionally receive communications and requests from multiple groups across Health Canada and PHAC, and it appears that their key contact in HC may not always be aware of competing requests coming from other areas. There may be value in enhancing internal coordination in order to address these issues. For example, some Health Canada interviewees mentioned that an internal working group committee that had met until 2018-19 had previously been beneficial to supporting internal communication between the branches working with PCHOs, and across the Health Portfolio. Several representatives from one PCHO indicated that Health Canada could consider developing "mandate letters" for PCHOs, which would clearly specify the federal government's priorities for the coming year.

In addition, according to a couple of Health Canada representatives, clearer prioritization also could benefit Health Canda itself. Having a better sense of what the priorities are and focusing on these priorities would help reduce the administrative burden for the organization, as it would most likely lead to having to complete fewer amendments to the respective contribution agreements. Furthermore, providing greater clarity to the PCHOs on Health Canada and Health Portfolio priorities would help ensure continued positive relationships between Health Canada and the PCHOs, as there would be less confusion and more effective working relationships between the two.

Improve communication to address instability during funding renewal

As with most grant and contribution programs, uncertainty during the renewal period for funding agreements caused challenges for the PCHOs. While most of the PCHOs have ongoing funding established within Health Canada's budget, they are subject to contribution agreements with terms and conditions typically limited to a five-year period. Several interviewees from the PCHOs reported that, when organizations approached the end of a funding period without a renewal in place, they experienced challenges in recruitment and retention of staff. Furthermore, uncertainty about if and when the funding agreement would be renewed and the associated funding level was also seen to jeopardize ongoing relationships with program delivery partners. Health Canada has taken steps to address this perceived instability, such as, for example, by extending existing agreements if renewal was slower than anticipated. However, representatives from the PCHOs and subject matter experts who have worked on the PCHO boards both reported there may still be opportunities to improve communication during this period to provide indications of future timing and levels of funding, which might help address uncertainty and minimize disruption in the PCHOs' activities.

Conclusions and recommendations

Conclusions

Evolution

The PCHOs have responded to emerging issues, including the COVID-19 pandemic, while uniquely filling a needed role in the Canadian health system, and collaborating with partners and each other to advance their respective mandates. Structural changes and organizational adaptations at both Health Canada and the PCHOs have addressed emerging issues in the Canadian health system and allowed both Health Canada and the PCHOs to make progress in addressing issues identified in the previous Fit for Purpose review.

Impact

The PCHOs have made collective and individual progress towards their shared goals of enhancing health system capacity, changing the behaviour of health care decision makers, and supporting improvements in the health systems. The PCHOs also demonstrated flexibility by pivoting activities to respond to the pandemic while continuing to advance longstanding initiatives. Going forward, there will be opportunities to improve collective reporting across the PCHO suite on shared outcomes, such as examining indicators to ensure they adequately describe the impact of PCHO activities on their stated shared goals.

Management

Over the past five years, Health Canada and the PCHOs have had an improved collaborative relationship, supported by formal and informal governance and communication mechanisms. This has resulted in better information sharing between Health Canada and the PCHOs.

Internal coordination at Health Canada and the Health Portfolio could be enhanced, such as by increasing communication and the consistency and quality of briefings, and by exploring internal coordination mechanisms. Health Canada could improve the prioritization of their requests to the PCHOs, particularly when they require organizations to pivot from their original workplans. Consistency of messaging to the PCHOs should also be sought. Coordination and prioritization should also extend to other members of the Health Portfolio that interact with the PCHOs.

Recommendations

Recommendation 1: Examine opportunities to improve internal and Health Portfolio governance mechanisms to support Health Canada's approach to managing the PCHO suite.

Some internal interviewees noted gaps in the information flow between operational and senior management staff in Health Canada and between Health Portfolio members which can create challenges for working level staff to effectively manage the funding agreement. Taking steps to address these gaps in information flow within Health Canada and the Health Portfolio may help streamline requests and support a more cohesive approach to management of the PCHO suite.

Recommendation 2: Implement strategies to prioritize requests from the Health Portfolio, with consideration of the potential impact on existing PCHO workplans.

Representatives from Health Canada and the PCHOs noted that there are opportunities for Health Canada and other departments in the Health Portfolio to prioritize their requests to the PCHOs and minimize the implications of adding requests to pre-planned activities. This is important, as Health Canada is the largest funder for many of the PCHOs, but not the only one. Requests from Health Canada reflect the needs of the federal government, but as an arms-length organizations, the PCHOs must also remain responsive to the needs of other stakeholders, such as provinces and territories and health care organizations, among others. In addition, from the Health Canada perspective, greater prioritization reduces administrative burden, such as fewer potential contribution agreement amendments, and helps maintain more positive relationships with the PCHOs.

Recommendation 3: Work with the PCHOs to examine indicators used in collective reporting across the PCHOs.

While the approach of developing a common logic model has been a step in the right direction towards a framework for collective reporting, some of the PCHO indicators related to the common logic model do not adequately demonstrate the progress the PCHOs have made on their stated shared goals. To support continual improvement in the performance measurement framework for the PCHO suite, Health Canada should work with them to review indicators on outcomes from the shared logic model. While improvements should seek to minimize any additional reporting burden on individual PCHOs by leveraging data which has already been collected, it may be worthwhile to explore methodologies to align their performance measurement and facilitate stronger collective reporting. These further improvements could help more efficiently and effectively describe the performance story of the PCHOs' collective impact.

Management Response and Action Plan

Recommendation 1

Examine opportunities to improve internal and Health Portfolio governance mechanisms to support Health Canada's approach to managing the PCHO suite.

Management response

Management agrees with this recommendation.

Action plan

Develop a governance framework to guide horizontal policy and program management work within Health Portfolio and among the suite of Pan-Canadian Health Organizations. The framework will inform optimal steps and approaches to achieve strategic alignment and consistency in direction-setting among programs and with PCHOs.

Deliverables

Governance Framework to guide horizontal policy and program implementation relating to the PCHO mandates and their impact. It will include (but not limited to):

• Roles
• Accountabilities
• Mechanisms (e.g., a forum)

Expected Completion Date

November 2024

Accountability

ADM(s) responsible for PCHOs

Resources

This recommendation will be completed using existing human and financial resources.

Recommendation 2

Implement strategies to prioritize requests from the Health Portfolio, with consideration of the potential impact on existing PCHO workplans.

Management response

Management agrees with this recommendation.

Action plan

Health Portfolio adopts a principle-based approach to engagement and communication, which acknowledges the roles, functions, and priorities of program areas and the PCHOs.

Deliverables

• Summary document outlining current practices regarding Health Portfolio's cyclical/planned and ad-hoc requests to PCHOs.
  • Expected completion date: May 2024
• Best practice guidelines for alignment and efficiency when engaging PCHOs on priorities, as articulated in the governance framework (see Recommendation 1).
  • Expected completion date: November 2024

Accountability

ADM(s) responsible for PCHOs

Resources

This recommendation will be completed using existing human and financial resources.

Recommendation 3

Work with PCHOs to examine indicators used in collective reporting across the PCHO suite.

Management response

Management agrees with this recommendation, with the condition that collective and individual performance measurement is considered.

Action plan

Health Portfolio adopts a principle-based approach to engagement and communication, which acknowledges the roles, functions, and priorities of program areas and the PCHOs.

Deliverables

• Assessment report on existing performance reporting indicators and mechanisms, with consideration of:
• How individual or common indicators contribute to collective outcomes and reporting
• Utility of information and existing formats (for audiences including senior management, Canadian public)
• Expected completion date: July 2024
• Updated collective reporting which could include performance indicators as required (based on the assessment report)
• Expected completion date: January 2025

Accountability

ADM(s) responsible for PCHOs

Resources

This recommendation will be completed using existing human and financial resources.

Appendix A: Description of the Pan-Canadian Health Organizations

Table 1 : Descriptions

Organization	2022-23 Health Canada Funding (Millions)	Key Functions
Canada Health Infoway	$38.5	Infoway works with governments, health care organizations, clinicians, and patients to help accelerate the development and adoption of digital health technologies, drive health innovation, and improve health outcomes. Infoway collaborates with partners in public and private sectors on digital health projects, including e-prescribing, pan-Canadian interoperability, and virtual care.
Canadian Agency for Drugs and Technologies in Health	$28.5	The Canadian Agency for Drugs and Technologies in Health (CADTH) is responsible for providing Canadian health care decision makers with objective evidence to help inform decisions on the optimal use of drugs and medical devices in the health care system. Key activities include reimbursement reviews, health technology reviews, horizon scans, and offering scientific advice.
Canadian Centre on Substance Use and Addiction	$12.0	The Canadian Centre on Substance Use and Addiction (CCSA) provides national leadership on substance use and to advance solutions to address alcohol- and other drug-related harms. CCSA is focused on synthesizing research, driving collaboration efforts across Canada, and bridging the gap between knowledge and action to achieve an accessible and inclusive continuum of services and supports for people who use substances, and achieving evidence-informed responses to reduce substance use harms.
Canadian Institute for Health Information	$99.0	The Canadian Institute for Health Information (CIHI) provides comparable and actionable data and information that are used to accelerate improvements in health care, health systems performance, and population health across Canada. CIHI maintains a broad range of health system databases, measurements, standards, and evidence-based reports and analyses to inform decision-making processes.
Canadian Partnership Against Cancer	$47.5	The Canadian Partnership Against Cancer (CPAC) was founded in 2007 to steward the Canadian Strategy for Cancer Control and aims to effect systemic change through collaboration to improve the experience of people in Canada with cancer and their families. They aim to reduce the incidence of cancer, reduce the likelihood of dying from cancer, and improve the quality of life of those affected by cancer.
Healthcare Excellence Canada	$25.1	Healthcare Excellence Canada (HEC) is a new organization, formed in 2021 by the amalgamation of two formerly separate PCHOs: the Canadian Foundation for Healthcare Improvement (CFHI), and the Canadian Patient Safety Institute (CPSI). Through collaboration with patients, caregivers, and people working in health care, HEC promotes broad implementation of proven innovations in health care and patient safety. Current areas of focus are care of older adults, bringing care closer to home with safe transitions, and supporting pandemic recovery and resilience.
Mental Health Commission of Canada	$15.8	The Mental Health Commission of Canada (MHCC) leads the development and dissemination of innovative programs to support the mental health and wellness of Canadians, and supports federal, provincial, and territorial governments in implementing sound public policy.

Appendix B: Common Logic Model, Pan-Canadian Health Organizations

Note: Measurement of work related to First Nations, Inuit and Métis peoples should be done in consultation with First Nations, Inuit and Métis leaders and communities led by them, and will be the responsibility of each PCHO in their contribution to the logic model.

Logic Model: Long Description

Figure 1 consists of a logic model that explain the expected outputs and outcomes of the programs in a progressive manner.

There are 8 PCHOs that can apply to different sections of the logic model. They are:

• CADTH
• CCSA
• HEC – (CHFI)
• HEC – (CPSI)
• CIHI
• CPAC
• Infoway
• MHCC

The first section of the logic model is Activities:

• Policy coordination on shared priorities
• Stakeholder engagement
• Co-design products
• Enter collaborative partnerships
• Consultation across shared groups
• Generate and synthesis knowledge
• Knowledge mobilization (dissemination and implementation)
• Developing tailored products

The next section is Outputs:

• Learning opportunities
• Evidence-informed products
• Tools
• Data
• Technology solutions
• Advice and recommendations
• Physical resources
• Funding
• Partnerships, networks, social supports.

Next are two activities: Disseminate and support implementation of knowledge and Collaboration and partnerships.

These lead into the Immediate Outcomes – Increase in capacity. These outcomes are as follows:

• 1.1 Health practitioners are equipped with knowledge and skills to address the population's health care needs. This indicator applies to all eight PCHOs.
• 1.2 Health care delivery organizations and government decision-makers are equipped with knowledge to make improvements in health care or social services This indicator applies to all PCHOs except Infoway.
• 1.3 Health care delivery organizations partner to implement health care solutions This indicator applies to HEC- (CFHI), HEC – (CPSI), CPAC and MHCC.
• 1.4 Government decision-makers are engaged to implement initiatives across health care systems. This indicator applies to all PCHO organizations except HEC (CPSI).

This leads to another section describing Intermediate Outcomes – Changes in behaviour. These outcomes are separated into two categories: Individual-level impact and Health systems level impact.

Individual-level impacts:

• 2.1 Health practitioners offer health care or social services that are responsive to the needs of priority populations. This indicator applies to all eight PCHOs.
• 2.2 Health practitioners offer person-centered health care or social services in targeted areas. This indicator applies to all PCHOs except CADTH and CIHI.

Health systems level impacts:

• 2.3 Health care delivery organizations test health care solutions. This indicator applies to CCSA, MHCC, HEC (CHFI) and HEC (CPSI).
• 2.4 Health care delivery organizations and government decision makers adopt and implement proven health care solutions, in targeted areas. This indicator applies to all eight PCHOs.
• 2.5 Government decision makers widely implement health care solutions. This indicator applies to all eight PCHOs.
• 2.6 Health stakeholders and researchers have access to pan-Canadian data and research findings to support their decision making. This indicator applies to all PCHOs except HEC (CHFI), HEC (CPSI), and Infoway.

The Individual-level impacts and Health systems level impacts are interrelated and can affect each other.

The Intermediate outcomes are also linked to the Outputs, denoting the continuous learning that occurs between Outputs and Outcomes.

The final section is Ultimate Outcomes:

• 3.1 People who live in Canada experience safe, timely, appropriate, person-centred and inclusive, coordinated health care and social services. This indicator applies to all eight PCHOs.
• 3.2 Health care system improvements are spread and sustained. This indicator applies to CCSA, HEC (CFHI), CPAC and Infoway.
• 3.3 Health care decision makers use evidence and data to make improvements and investments in the health care system. This indicator applies to all eight PCHOs.

Logic Model: Assumptions

Direct benefits assumptions:

• Health system enhancements prove to be practical.
• Initiatives respond to the existing needs of Canadians.
• The target populations are aware that service availability has improved.

Practice assumptions:

• Health care practitioners use knowledge.
• Partners create opportunities for knowledge use.
• Health care practitioners and their co-workers are receptive to change.
• Organization representatives reached can influence internal support to dedicate resources and implement changes (senior management and manager support, $, motivation).
• PT-based initiatives align with PTs priorities and allocation of resources.
• Initiatives developed in collaboration and co-creation with First Nations, Inuit, and Métis communities.

Capacity assumptions:

• Health care delivery organizations:
  • Proposed initiatives are responsive to needs.
• Health care practitioners:
  • Find the knowledge products and learning opportunities useful (relevant, credible, timely) and appropriate for their intended purpose.
  • Receive internal support to dedicate time to learning and training.
• PTs
  • Receive political support.
  • Senior management understands the issues and need for the initiative.
• Common/All target audiences or intermediaries:
  • Effective reach to appropriate health practitioners and health care delivery organizations and they access knowledge products and learning opportunities.
  • Stakeholders want to use the knowledge and skills, and find it timely and useful.
  • Understand varied experiences and complexities in priority populations.
• Stakeholders are provided with the physical and social supports they need.

Reach assumptions

• Stakeholders that can make or lead changes within their organization are reached.
• Partners promote the outputs.
• Target populations are aware of the outputs.

Appendix C: Recommendations in the Fit for Purpose Review

The Fit for Purpose review, published in 2018, provided 10 non-binding recommendations directed at the federal government to improve the Pan-Canadian Health Organizations suite. These recommendations are provided verbatim below:

1. As part of its recognition and affirmation of Indigenous rights, the Government of Canada should task Health Canada and the Department of Indigenous Services Canada with initiating a dialogue with the National Indigenous Organizations to identify how the PCHOs could serve Indigenous communities' health priorities. Including the leadership of the PCHOs and other groups in those discussions will likely become necessary to ensure the transition to a new model and a new culture.
2. The Government of Canada should adopt a long-term vision for the future of Canadian health care and articulate the role it intends to play in pursuing that future. The Minister and the organizations in the federal health portfolio - including the PCHO suite - should assume explicit responsibility for establishing and pursuing a common set of priorities.
3. As a principle, the Government of Canada should support partnerships with clear goals and objectives addressing defined health systems issues. Such partnerships should engage provincial and territorial governments, the Canadian public, National Indigenous Organizations, the research and public health communities, administrators, and providers.
4. Health Canada should implement process improvements to clarify the purpose and functions of the PCHOs; streamline their governance; establish joint strategic and work planning; put in place measurable objectives and indicators for health system improvement; ensure independent impact evaluations in areas of PCHO activity; and require that the PCHOs build truly pan-Canadian and bilingual teams.
5. The status quo for the PCHO suite will not serve the needs of Canadians in the 21st century. The federal government must recast, re-mandate, and redirect the PCHOs to make optimal use of all resources available at the federal level to support health quality and system transformation.
6. Health Canada should instruct the PCHO suite to partner with the provinces and territories to accelerate the emergence of comprehensive, integrated publicly funded health systems centred in primary care. Spread and scale of system innovations, using all levers including policy and regulation, must be a shared strategic priority.
7. Three PCHOs have achieved the purpose for which they were established. Their ongoing existence in their current form contributes to siloing and undermines health system integration efforts. The Government of Canada should phase out these organizations in their current forms and encourage new and better use of their functions and resources.
   • CCSA - The required functions and roles related to substance use and addiction can be assumed by a national network focused on discovery, innovation, and implementation, built on the foundation of the Canadian Research Initiative in Substance Misuse (CRISM).
   • CPAC - As a result of CPAC's work, the cancer community in Canada has developed strong collaborations. A group of mature cancer agencies across the country now has the capacity to support the shared goal of providing world-class cancer prevention, care, and survivorship.
   • MHCC - Mental health is now "out of the shadows". The integration of mental health care services into the core of Canadian health systems requires a different type of leadership, capable of driving a bottom-up approach in which patients and families, providers, researchers, and the broader mental health community come together to break down silos.
8. The Government of Canada should take responsibility for the ongoing data governance and management needed to support learning health systems across Canada. Such systems must be built on fully interoperable electronic health records that can be accessed by patients and their circle of care.
9. The Government of Canada should establish an integrated infrastructure for prescription drug policy that connects approvals, assessment, pricing, purchasing, and post-market surveillance. The assessment of health technologies should be carried out by leveraging the existing network of qualified agencies across the country.
10. In service of equity, the Government of Canada must put in place an ongoing mechanism to define standards for the modernization of the basket of publicly funded services in partnership with the provinces and territories. The definition of medical necessity requires an ongoing pan-Canadian process informed by evidence and public engagement; the PCHOs can be important participants in this work.^{Footnote 46}

Appendix D: Methodology

Scope

The present evaluation was conducted to fulfill the requirements of the Financial Administration Act and to provide Health Canada management with information to help improve and adjust their activities and relationships with the PCHOs, as necessary.

The evaluation covered Health Canada's administration of the Pan-Canadian Health Organizations and the PCHO activities from 2018-19 to 2022-23.

The evaluation used a SGBA Plus lens in its assessment of the Pan-Canadian Health Organizations and Health Canada, including consideration of the importance of responsiveness to the needs of priority populations when examining the evolution theme. Furthermore, performance measures used to inform the examination of impacts of the program included consideration of SGBA Plus factors. Although official languages were not specifically examined, they were not found to be an issue for the program's activities. Additionally, an examination of the Sustainable Development Goals was not applicable for this evaluation.

In conducting the evaluation, a single window was identified from the Office of Grants and Contributions at the Strategic Policy Branch, with whom the Office of Audit and Evaluation worked closely throughout the evaluation. The scope for this evaluation was shared secretarially with the Performance Measurement and Evaluation and Results Committee (PMERC) in October 2022. The preliminary findings were presented at PMEC on June 29, 2023, and the final report will be presented at PMERC on October 12, 2023.

The evaluation addressed questions related to three areas of focus: Evolution, Impacts, and Management. The questions addressed in the evaluation were:

1. Evolution: How have the PCHOs evolved in response to the changing environment and changing national and federal priorities over the past five years?
   1. How have the PCHOs and Health Canada evolved over the past five years?
   2. Are the PCHOs filling a needed role or gap in the Canadian landscape?
2. Impacts: What progress have the PCHOs made towards their objectives?
   1. What progress have the PCHOs had on their immediate objectives?
   2. What progress have the PCHOs had on their intermediate objectives?
   3. What progress have the PCHOs had on their ultimate objectives?
   4. What factors or challenges have influenced potential impacts (e.g., COVID, funding, HR capacity)?
3. Management: To what extent is HC's program management contributing to the effectiveness and efficiency of PCHOs?
   1. How effectively have HC management processes and practices supported PCHOs?
   2. To what extent has HC efficiently and effectively managed for results with the PCHO programming?
   3. What opportunities for improvement, if any, exist?

Data Collection Sources and Methods

Interviews

• OAE conducted key informant interviews with 42 respondents from Health Canada, representatives of the Pan-Canadian Health Organizations, and subject matter experts familiar with the program. In total, OAE interviewed:
  • Employees Working with Health Canada: 14
  • Staff in Leadership Positions at the PCHOs: 21
  • Interviewees that are Subject Matter Experts working with the PCHOs: 7 (The majority of subject matter experts were former or current board members of the PCHOs, including some board chairs)
• Interviews were coded and summarized using NVIVO qualitative data analysis software.

Environmental scan

• An environmental scan was conducted comparing the mandate and activities of the PCHOs to 29 non-governmental organizations with similar focus in Canadian health systems.
• The environmental scan examined areas of duplication and complementarity, as well as partnerships undertaken with similar organizations.

Document review

• OAE collected and reviewed approximately 900 documents provided by Health Canada programs and by PCHO representatives.
• Documents provided information related to the evolution of the PCHOs and progress towards expected outcomes.
• Documents included annual reports, evaluations, and administrative records, among many others.

Performance data review

• OAE worked with program representatives to select a sample of key performance indicators from the common logic model.
• Results from these indicators were analyzed and compared to performance targets, where available, to describe progress towards expected outcomes of the PCHO suite.

Financial data review

• OAE used financial data from Health Canada's Branch Senior Financial Officer (BSFO). OAE also reviewed the PCHO reports of cash flows, expenditures, and approvals for retention of unexpended balances to identify variances between planned and actual expenditures within the fiscal year by recipient organizations.

Limitations

OAE identified findings by comparing and combining information gathered from the various sources listed above. The use of multiple sources of information is meant to increase the accuracy and authority of any conclusions made in this report. Still, many evaluations face conditions that limit their accuracy and may be important to consider. The following table lists the limitations for this evaluation and actions taken to address them.

Limitation: Lack of direct feedback from the PCHO beneficiaries

Impacts: In an effort to reduce the reporting burden associated with the evaluation, OAE did not collect any direct feedback with beneficiaries of the PCHO activities, such as health care delivery organizations, health care providers, and government decision makers. As such, it was not possible to include any first-hand data from these stakeholders on the extent to which the PCHO activities impacted their work.

Mitigation strategy: OAE leveraged existing survey data and feedback collected by the PCHOs with their stakeholders, wherever available, to illustrate how stakeholders leveraged the PCHO products and activities, and the impacts on their work. Additionally, OAE interviewed subject matter experts who worked with the PCHOs, several of whom are currently or had previously worked in government positions and health care settings, leveraging the PCHO expertise.

Limitation: Limitations in performance data for shared outcomes in logic model.

Impacts: The PCHO suite is comprised of seven distinct organizations with diverse mandates. Recently, Health Canada led the development of a shared logic model which leverages existing performance measures already collected by the PCHOs to demonstrate progress towards shared outcomes. Because these indicators were selected based on the performance frameworks of the respective PCHOs, they do not all clearly demonstrate progress on stated outcomes in the shared logic model as intended. Furthermore, some indicators selected for inclusion in this evaluation did not have complete data for the full period covered within the scope.

Mitigation strategy: Where possible, missing, irrelevant or low-quality performance metrics and data were omitted, and evidence from documents, including recipient-led evaluation reports, was used to describe impacts of the PCHO's activities on their shared outcomes.

Limitation: Small number of key informant interviews for diverse set of organizations.

Impact: In an effort to reduce the reporting burden for the PCHOs participating in the evaluation, and in an effort to manage the scope of data collection for this project, OAE conducted a small number of interviews with each organization, up to four respondents per organization.

Mitigation strategy: OAE selected targeted interview respondents in senior leadership roles who would be best able to address questions related to the evolution of the PCHOs and Health Canada's management of the PCHO contribution agreements. As previously mentioned, interviews were triangulated with other lines of evidence to substantiate or refute emerging themes from these interviews.

Appendix E: Frequency of Meetings between Health Canada and the PCHO Leadership

Table 2: Meetings

PCHO	Health Canada participation in Board of Directors	Regular bilateral meetings	Ad-hoc bilateral meetings
CCSA	DM is an ex-officio member of the board	Quarterly meetings with DM and CEO	CEO and Minister, CEO and ADM SPB and CSCB CEO and DG of Controlled Substances and Cannabis Branch (CSCB), DG of Health Programs and Strategic Initiatives Directorate (HPSI)
MHCC	ADM SPB is a full voting member of board	Quarterly meetings with DM and CEO	CEO and Minister CEO and ADM, CEO and DG of HPSI
HEC	ADM SPB is a full voting member of the board	Monthly meetings with ADM and CEO	CEO with DM CEO and SPB-ADM Quarterly meetings with DM and CEO
CADTH	ADM SPB is a full voting member of board	Monthly meetings with ADM and CEO	CEO and DM
CIHI	DM is full voting member of the board	Monthly meetings with DG and CEO	CEO and Minister, CEO and DM, CEO, ADM, and Assoc. ADM SPB
Infoway	Two federal appointees, including ADM SPB, are full voting member of the board	None Indicated (Ad Hoc)	CEO and DM meet semi-annually (as needed) CEO and SPB ADM meets at least monthly
CPAC	ADM SPB is a full voting member of board. Director of Programs, HCPPD SPB is an observer on the board.	Triweekly meetings with ADM and CEO Biweekly meetings with Director of Programs, HCPPD and CEO	CEO and Minister CEO and DM

Appendix F: Health Canada PCHO Expenditures - Financial Data

The following financial tables reflect Health Canada's total planned and actual disbursements for grants and contributions to the Pan-Canadian Health Organizations (PCHO) between 2018-19 and 2022-23. The financial data below is provided by Health Canada's Chief Financial Officer Branch. Planned Grants and Contributions (G&Cs) funding is based on program authorities maximum eligible funding that can be disbursed to the recipient. Funding is considered disbursed when it is released from Health Canda to the recipient.

Note the financial tables below do not include the following funding:

• Funding to PCHOs from other sources e.g., provincial and territorial governments
• Funding transferred from one PCHO to another PCHO for joint work
• Unexpended balances that were retained from one fiscal year to be spent in the following fiscal year
• Return of funds to Health Canada after the fiscal year

Explanations for variances above approximately 5% of planned vs. actual disbursements are included in the footnotes below the table.

Table 3: Infoway financial data

Year	Planned	Actual	Variance ($)
	Total Planned G&Cs	Actual G&Cs Disbursed from Health Canada to PCHO	Variance ($)	% of Planned G&Cs
2018-19	$50,000,000	$49,844,676	-$155,324	99.7%
2019-20	$75,000,000	$74,647,172	-$352,828	99.5%
2020-21	$77,000,000	$84,380,788	$7,380,788	109.6%Footnote 1
2021-22	$85,554,000	$91,242,085	$5,688,085	106.6%Footnote 1, Footnote 2
2022-23	$47,580,000	$38,508,646	-$9,071,354	80.9%2
Total	$335,134,000	$338,623,367	$3,489,367	101%
Note: In addition to the funding from the 2017 Contribution Agreement, Infoway received additional grants and up-front multi-year funding between 2001 and 2010, and funding from other previous agreements that were used to help achieve some of the outcomes noted in the report. Total expenditures over the five-year period as per Infoway audited financial statements was $461.7M. Footnote 1 Variance to the distribution of funding to support activities and deliverables associated with the Prime Minister's May 2020 announcement of $240.5M, of which Infoway received $50M, to accelerate the use of virtual tools and digital approaches in response to the COVID-19 pandemic. Return to footnote 1 referrer Footnote 2 Variance related to revisions to the funding related to the Organ Donation and Transplantation initiative. Return to footnote 2 referrer

Table 4: CADTH financial data

Year	Planned	Actual	Variance ($)
	Total Planned G&Cs	Actual G&Cs Disbursed from Health Canada to PCHO	Variance ($)	% of Planned G&Cs
2018-19	$20,058,769	$20,058,769	$0	100.0%
2019-20	$23,058,769	$23,058,769	$0	100.0%
2020-21	$26,058,769	$26,058,769	$0	100.0%
2021-22	$29,058,769	$29,058,769	$0	100.0%
2022-23	$31,284,352Footnote 1	$28,467,897	-$2,816,455Footnote 2	91%
Total	$129,519,428	$126,702,973	-$2,816,455	98%
Footnote 1 Health Canada received $5.2M from the Canadian Institutes for Health Research (CIHR) to provide to CADTH to transfer the operation of the Drug Safety and Effectiveness Network (DSEN) program from CIHR to CADTH. Return to footnote 1 referrer Footnote 2 A portion of 2022-23 planned funding ($1.1M) was retained by Health Canada to recover unspent funding from 2021-22 when CADTH experienced staffing challenges during the COVID-19 pandemic. Separately, $1.7M was retained by Health Canada from the CIHR transfer as CADTH was able to reallocate other unspent funding as a one-time source of funds to launch the new Post-Market Drug Evaluation program, which replaced DSEN. Return to footnote 2 referrer

Table 5: CCSA financial data

Year	Planned	Actual	Variance ($)
	Total Planned G&Cs	Actual G&Cs Disbursed from Health Canada to PCHO	Variance ($)	% of Planned G&Cs
2018-19	$10,000,000	$9,797,872	-$202,128	98.0%
2019-20	$12,330,000	$11,910,040	-$419,960	96.6%
2020-21	$12,405,000	$11,815,806	-$589,194	95.3%Footnote 1
2021-22	$12,350,000	$13,350,000	$1,000,000	108.1%Footnote 2
2022-23	$12,330,000	$12,029,868	-$300,132	97.6%
Total	$59,415,000	$58,903,586	-$511,414	99.1%
Financial data includes both core funding and funding for cannabis research and public education, both funded under Health Canada's Substance Use and Addiction Program. Footnote 1 Funding for Cannabis research was time-limited (end date March 2023). Cannabis research funding variances due to COVID-19 affected their funded partner's capacity and access to academic and hospital research facilities, as well as other partner-dependent delays (e.g., HC cannabis licensing and CIHR awarding grants), creating challenges to completing CCSA's research activities and subsequent knowledge mobilization. Return to footnote 1 referrer Footnote 2 Additional funding for Phase I work on CCSA's 2023 Guidance on Alcohol and Health. Return to footnote 2 referrer

Table 6: CIHI financial data

Year	Planned	Actual	Variance ($)
	Total Planned G&Cs	Actual G&Cs Disbursed from Health Canada to PCHO	Variance ($)	% of Planned G&Cs
2018-19	$83,808,979	$83,948,979	$140,000	100.2%
2019-20	$87,658,979	$88,158,979	$500,000	100.6%
2020-21	$92,658,979	$99,593,979	$6,935,000	107.5%Footnote 1
2021-22	$101,373,979	$101,373,979	$0	100.0%
2022-23	$97,324,479	$99,014,979	$1,690,500	101.7%
Total	$ 462,825,N/A395	$472,090,895	$9,265,500	102%
Footnote 1 The variance between planned and actual disbursements is mainly due to an allocation of funds in support of the Organ Donation and Transplantation initiative and Safe Restart work following the pandemic. Return to footnote 1 referrer

Table 7: CPAC financial data

Year	Planned	Actual	Variance ($)
	Total Planned G&Cs	Actual G&Cs Disbursed from Health Canada to PCHO	Variance ($)	% of Planned G&Cs
2018-19	$43,100,000	$42,971,939	-$128,061	99.7%
2019-20	$51,000,000	$50,846,986	-$153,014	99.7%
2020-21	$51,000,000	$50,943,947	-$56,053	99.9%
2021-22	$52,500,000	$52,453,123	-$46,877	99.9%
2022-23	$47,500,000	$47,476,048	-$23,952	99.9%
Total	$245,100,000	$244,692,043	-$407,957	99.8%

Table 8: HEC financial data

Year	Planned	Actual	Variance ($)
	Total Planned G&Cs	Actual G&Cs Disbursed from Health Canada to PCHO	Variance ($)	% of Planned G&Cs
2018-19	$24,600,000	$24,600,000	$0	100.0%
2019-20	$24,600,000	$24,600,000	$0	100.0%
2020-21	$24,600,000	$31,300,000	$6,700,000	127.2%Footnote 1
2021-22	$26,300,000	$28,400,000	$2,100,000	108.0%Footnote 1
2022-23	$24,600,000	$25,100,000	$500,000	102.0%
Total	$124,700,000	$134,000,000	$9,300,000	107.5%
Note: Financial data includes funding for CPSI and CFHI (2018-19 to 2020-21 prior to merger), and HEC (2021-22 to present). Footnote 1 Health Canada transferred additional funding to HEC to support its Long Term Care Plus program, which supported pandemic readiness and mitigation in long term care homes across Canada. Funding was provided from internal sources as well as $6.35M allocated to the program from the 2020 Fall Economic Statement ($4.65M in 2020-21 and $1.7M in 2021-22). Return to footnote 1 referrer

Table 9: MHCC financial data

Year	Planned	Actual	Variance ($)
	Total Planned G&Cs	Actual G&Cs Disbursed from Health Canada to PCHO	Variance ($)	% of Planned G&Cs
2018-19	$ 14,250,000	$ 14,754,000	$504,000	103.5%
2019-20	$ 15,616,000	$ 15,616,000	$0	100.0%
2020-21	$ 17,561,000	$ 17,561,000	$0	100.0%
2021-22	$ 17,512,000	$ 17,512,000	$0	100.0%
2022-2023	$ 15,807,000	$ 15,807,000	$0	100.0%
Total	$ 80,746,000	$ 81,250,000	$504,000	100.6%
Includes core funding, as well as the Cannabis Public Education Initiative funding under Health Canada's Substance Use and Addiction Program.

Appendix G: Examples of Collaboration

Infoway played a leadership role working with provinces and territories and health systems stakeholders to develop a Shared Pan-Canadian Interoperability Roadmap, endorsed by the conference of Deputy Ministers on March 30, 2023. They also led a coalition of partners, including Health Canada, provinces and territories, and health systems stakeholders to launch and scale up PrescribeIT. Furthermore, they collaborated with MHCC to initiate Stepped Care 2.0.

CADTH has had an on-going collaboration with Choosing Wisely Canada (CWC) to reduce post pandemic low value care. It has also collaborated with the Institute for Health Economics and the Canadian Association for Population Therapeutics to offer a workshop entitled "Defining Decision Grade Real-World Evidence and its role in the Canadian Spirit Context: A Design Sprint."

CCSA has developed a partnership with the Alberta Family and Wellness Initiative to build the Brain Builders Lab. CCSA also collaborated with the Inuit Tapiriit Kanatami to develop resources addressing substance use, mental health, and suicide among Inuit.

CIHI worked with the Ontario Ministry of Children and Youth Services interRAI, a pan-Canadian database which captures standardized clinical, demographic, administrative, and resource use information on publicly funded home care and facility-based long-term care (LTC) services. This is part of a larger Care Planning Tool initiative, which comprises Momentum Healthware, the First Nations Alberta Technical Services Advisory Group, interRAI, Alberta Health Services, Alberta Health, and six First Nations communities.

CPAC collaborated with provincial and territorial governments, as well as cancer agencies, the not-for-profit sector, and patient and survivor groups to develop the Canadian Strategy for Cancer Control. They also worked with HEC to develop resources geared towards paramedics to provide palliative care to patients in their home.

HEC's 'LTC+: Acting on the Pandemic, Learning Together' was delivered in partnership with several organizations, namely the British Columbia Patient Safety and Quality Council, Shared Health (previously Manitoba Institute for Patient Safety), the New Brunswick Association of Nursing Homes, and CADTH.

MHCC collaborated with CCSA on the Health Standards Organization committee, which develops standards for substance use and mental health services. Furthermore, MHCC partnered with CCSA on the cannabis research including a multi-part webinar series and the end-of-grant 3 day event to showcase research findings.

End notes