Evaluation of the Health Care Policy and Strategies Program - 2018-19 to 2022-23

Prepared by the Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

January 2024

Table of Contents

• List of acronyms
• Executive summary
  • Background
  • Conclusions
  • Recommendations
• Program description
  • Background
  • Program description
• Evaluation description
  • Evaluation scope
• Evaluation findings
  • Addressing demonstrated needs and priorities
• Progress toward achievement of HCPSP expected outcomes
• Program oversight
• Program delivery
  • Factors affecting project success
• Program Spending
• Conclusions and recommendations
  • Conclusions
  • Recommendations
• Management Response and Action Plan
• Annex A – Evaluation methodology and limitations
• End notes

List of acronyms

Executive summary

Background

The Health Care Policy and Strategies Program (HCPSP), formerly the Health Care Policy Contribution Program (HCPCP), was launched in 2002. The HCPSP's core budget provides up to $25.7 million per fiscal year in contribution funding to address health care system priorities, through funding for projects in four priority areas: mental health care, home and community care, palliative and end-of-life care, and other federal, provincial/territorial, and emerging priorities.

Conclusion

The HCPSP continues to address key needs within the Canadian health care system. The HCPSP's priority areas are consistent with Government of Canada priorities, including those related to the health care system and supporting groups experiencing health inequalities. The HCPSP is generally complementary to the work of other programs, and there are both formal and informal processes in place to help prevent or limit duplication of other programs' activities.

HCPSP-funded projects were largely effective in generating, accessing, and sharing knowledge products. These included standards and guidelines on health system approaches, guidance documents, literature reviews, training modules and learning supports, and formalized knowledge networks. The evaluation also found some evidence that target audiences were applying these products and that this had led to health care system changes, including changes to guidelines and policies and improvements in treatment and care for patients. Organizational readiness and project leadership within funded organizations, as well as Health Canada direction and support to funding recipients, were key factors that contributed to project success, while capacity limitations within some funded organizations created impediments for some projects. The COVID-19 pandemic created some delays and challenges for projects but also led to innovation.

At the program level, the HCPSP has established some of the necessary oversight and accountability elements for projects managed in part by other groups within Health Canada and where other groups use the HCPSP's authorities to flow funding for Budget-related initiatives. This includes governance committees and informal understandings on respective roles and responsibilities with other groups. However, it is lacking a formalized description of roles and responsibilities for other groups using the HCPSP's authorities.

Overall, the support from Health Canada to funding recipients throughout the project cycle, including the application, implementation, performance measurement, and reporting processes, is a strength of the Program. The Program has made extensive improvements to the HCPSP's performance measurement approach and has introduced various enhancements for reporting results. However, Health Canada does not share information on projects or lessons learned with funding recipients or internally. Furthermore, the use of this information in Health Canada's decision making appears minimal. Sharing this information could allow projects to learn from one another, avoid working in silos, prevent duplication, and identify gaps. Internally, program efficiency could be improved by better understanding and sharing the lessons learned from projects.

The flexibility of the HCPSP was important for its success. The priority area of other federal, provincial/territorial, and emerging priorities allowed the Program to respond to a variety of unanticipated priorities and fund innovative projects and was particularly useful in light of the COVID-19 pandemic. In addition, the expanded use of the Program's authorities for Budget-related initiatives allowed for a wider variety of issues, activities, and organizations to be funded, and for the Department to more easily respond to Ministerial and government-wide priorities and initiatives. However, human resource capacity issues within the HCPSP Unit contributed to feelings of burnout and limited the Program's ability to complete some planned work. Work areas affected included the sharing of lessons learned and the finalization of a roles and responsibilities document.

Recommendations

Evaluation findings led to the following recommendations.

Recommendation 1: Increase the sharing of information both externally and internally, including project information and lessons learned.

Overall, the support Health Canada provided to funding recipients was a strength of the Program. However, the lack of information sharing on projects and lessons learned was a gap in this support. This issue was also identified as an area for improvement in the previous evaluation. Sharing this information with funding recipients and internally within Health Canada could benefit funding recipients in their projects and could benefit the overall program.

Recommendation 2: Clarify program accountabilities, which could include formal articulation of key process steps and associated roles and responsibilities for projects managed by those outside of the HCPSP Unit.

The HCPSP includes some of the necessary elements to help ensure effective oversight and accountability, including oversight through governance committees and informal understandings of respective roles and responsibilities with other groups. However, it does not have documented roles and responsibilities at a more operational level for other groups using the HCPSP's authorities. This has led to a few instances of confusion about roles. Formal accountabilities are important for effective governance and reducing risk to the Program.

Program description

Background

The Health Care Policy and Strategies Program (HCPSP), previously called the Health Care Policy Contribution Program (HCPCP), was launched in 2002. It is a national program, led by Health Canada's Strategic Policy Branch, that provides contribution funding to projects that address health care system priorities. The Program was developed as one of several mechanisms to respond to issues identified by the Standing Senate Committee on Social Affairs, Science and Technology, and the Commission on the Future of Health Care in Canada in 2002. The HCPSP also aimed to address health care system priorities, as identified in the First Ministers' 2003 Accord on Health Care Renewal and the First Ministers' 10 Year Plan to Strengthen Health Care (2004), often referred to collectively as the Health Accords. More recently, the HCPSP's policy and funding priorities were aligned with key areas identified in the Common Statement of Principles on Shared Health Priorities (CSoP)^{Footnote 1}, agreed to by federal, provincial, and territorial (FPT) Health Ministers in 2017.

Program description

The HCPSP is intended to contribute to the Government of Canada's role in health care by supporting the development of policies and strategies to address evolving health care system priorities. The HCPSP's core budget provides up to $25.7 million per fiscal year in contribution funding to projects related to four priority areas:

• mental health care;
• home and community care;
• palliative and end-of-life care; and
• other federal, provincial/territorial, and emerging priorities.

The Program has undergone significant changes since the previous evaluation in 2018, including a change in one of the funding priority areas from health care system design and optimization of the health work force to other federal, provincial/territorial and emerging priorities. In addition, the Program streamlined governance and administrative processes and established a standardized performance measurement and reporting framework. The use of HCPSP authorities to flow funding for Budget initiatives also increased over this time period.

The Terms and Conditions (T&Cs) for the HCPSP were updated in 2020 and 2021. In addition to changing the Program's name and the fourth priority area, other key changes included allowing for funding and research awards to research institutions, allowing for funding of other expenditures deemed necessary to achieve program results, and clarifying distribution of funding to third party organizations by funding recipients. In addition, while the HCPSP Unit remains responsible for program coordination and management functions, the updated T&Cs indicate that other Health Canada directorates may also take responsibility for certain program management functions.

Over the period of the evaluation, the HCPSP funded 60 projects under the four priority areas. See Figure 1 below.

Figure 1 - Text description

Over the evaluation period, the HCPSP funded 60 projects under the four priority areas: 10 for mental health care, 10 for home and community care, 14 for palliative and end-of-life care, and 28 for other federal, provincial, territorial and emerging priorities.

Figure 1 is a white pie chart on a blue background showing the percentage of projects funded under each of the four priority areas.

• mental health care: 10 projects, 17% of funding
• home and community care: 8 projects, 13% of funding
• palliative and end-of-life care : 14 projects, 23% of funding
• other federal, provincial, territorial, and emerging priorities: 28 projects, 47% of funding

The Program's authorities were also used to flow funding for a number of federal Budget initiatives subsequently confirmed through Treasury Board submissions, such as the Terry Fox Research Initiative, Ovarian Cancer Canada, the Sexual and Reproductive Health Fund, Medical Assistance in Dying Policy and Practice, Mental Health Standards, and the implementation of the Department's Action Plan on Palliative Care. Between 2018-19 and 2022-23, several Budget commitments that were confirmed by subsequent Treasury Board decisions used the HCPSP's authorities to flow a total of $59.6 million in funding from sources other than the Program's core budget. In addition, money was transferred to other groups in Health Canada for projects or initiatives outside of the HCPSP. All of these initiatives were not part of the scope of this evaluation.

Evaluation description

Evaluation scope

This evaluation covers activities from 2018-19 to 2022-23. It was conducted to satisfy Financial Administration Act (FAA) requirements and to inform program management. Projects funded by HCPSP core program funds were evaluated. The evaluation did not cover funds that were transferred to other Health Canada groups for projects under different program authorities, or projects that used the HCPSP's authorities but received funding from Treasury Board decisions related to Budget initiatives. Evaluation evidence reflected in this document is based on a review of program documentation, including detailed project-level documentation for 11 of the 60 funded projects, a review of financial information, a focused literature review, interviews with Health Canada staff and funding recipients, and a survey of funding recipients. See Annex B for more information.

The evaluation addressed the following issue areas:

• Does the HCPSP address demonstrated needs and priorities?
• What progress has the HCPSP made in achieving its program objectives?
• How has Health Canada's management of the HCPSP impacted program efficiency?
• What processes and/or accountabilities should/could be in place to ensure proper oversight of activities outside of HCPSP's core funding?

Evaluators considered the planned HCPSP outcomes and condensed them to three generalized outcomes.

Evaluation findings

Addressing demonstrated needs and priorities

The HCPSP continues to address Canadian health care system needs. The HCPSP's priority areas are consistent with Government of Canada priorities, including those related to the health care system and supporting groups experiencing health inequalities. The priority area of other federal, provincial/territorial, and emerging priorities was particularly important during the COVID-19 pandemic, as it allowed Health Canada to quickly support the pandemic response. Overall, the HCPSP is complementary to the work of other programs and there are both formal and informal processes in place to help prevent or limit duplication.

Alignment with Canadian health care system needs

Mental health care

A significant number of Canadians are affected by poor mental health and mental illness. In any given year, one in five people in Canada will personally experience a mental health problem or illness. Mental illness also indirectly affects all Canadians at some time through a family member, friend, or colleague.^{Footnote 2}

In a recent study, 22% of Canadians reported their needs related to mental health were only partially met, with 21% reporting that their needs were fully unmet.^{Footnote 3} Furthermore, one in ten residents had a wait time for mental health services of between 34 and 260 days, varying across Canadian regions.^{Footnote 4}

High demands for mental health care and barriers to accessing services contribute to emergency department overcrowding, as an increasing number of people seek help for mental health issues there. Better access to mental health services would not only benefit people with mental health conditions, but also relieve the burden on the wider health care system.^{Footnote 5}

Home and community care

Home and community care services help people to receive care at home, rather than in a hospital or long-term care facility, and to live as independently as possible in the community^{Footnote 6}. The increasing population of older adults and those with chronic conditions is likely to result in a greater demand for home care services.^{Footnote 7} It is projected that by 2050 the number of older Canadians needing support from unpaid caregivers will double, yet there will be 30% fewer family members potentially available to provide support.^{Footnote 8}

Health care services delivered in homes may help meet the desires of individuals to remain at home, and may reduce costs associated with more expensive institutional options.^{Footnote 9} Unmet home care needs have been linked to negative consequences such as poorer health, increased use of other health services, admission to nursing homes, and reduced emotional well-being.^{Footnote 10} In 2021, 419,800 households reported they needed home care services but did not receive them.^{Footnote 11}

Long-term care facilities, also known as nursing homes, continuing care facilities, and residential care homes, provide a range of health and personal care services for Canadians with medical or physical needs who require access to 24-hour nursing care, personal care, and other therapeutic and support services^{Footnote 12}. By 2031, almost a quarter of Canada's population will be over the age of 65, and demand for long-term care will grow further^{Footnote 13}.

Palliative and end-of-life care

Palliative care is specialized medical care that focuses on providing relief from pain and other symptoms of serious illnesses. End-of-life care refers to care for people in decline who are expected to die in the near future. It can include medical assistance in dying (MAID), which is a process that allows eligible individuals to receive assistance from medical practitioners in ending their lives.

There is an increasing need for palliative and end-of-life care in Canada. This is due in part to the ageing population, with over 861,000 people aged 85 and older in Canada, according to the 2021 Census.^{Footnote 14} By 2046, the number of individuals aged 85 and older could triple to almost 2.5 million people.^{Footnote 15} Furthermore, improving treatment options are contributing to longer lifespans and allowing for more gradual declines in health.^{Footnote 16} These factors lead to an increasing number of individuals facing limitations and long-term health challenges^{Footnote 17}, putting increasing pressure on all levels of government to ensure adequate support in areas such as palliative care.^{Footnote 18}

In terms of health care service preparedness, three out of five doctors feel unprepared to support people needing palliative care^{Footnote 19}, and while 75% of Canadians would prefer to die at home, only 15% have access to palliative home care services.^{Footnote 20} Early access to palliative care has been shown to reduce the burden on emergency departments and reduce intensive care unit stays at end of life.^{Footnote 21}

Other federal, provincial, territorial, and emerging priorities

Beyond the three priority areas above, there is an array of other health care system priorities. These priorities include issues for which HCPSP projects were funded, such as the need for organ and tissue donations and transplants, various health workforce challenges, and the COVID-19 pandemic. Examples of other health care system needs include:

• The need for organ and tissue donations is increasing. There are more than 2,200 solid organ transplants each year; however, despite improvements in the number of organ donations since 2012, the number of people in need of a transplant has also increased. Each year, 250 people on the waiting list die while waiting for an organ.^{Footnote 22}
• According to the Canadian Medical Association, the Canadian health care system is in crisis, in large part due to health workforce shortages.^{Footnote 23} A recent study noted that one in six Canadians is lacking a regular family physician and less than half of Canadians are able to see a primary care provider on the same or next day.^{Footnote 24}
• During the time period assessed by this evaluation, a key emerging priority was the response to the COVID-19 pandemic. The pandemic resulted in health care system challenges of its own and exacerbated existing issues related to mental health care, home and community care, long-term care, palliative and end-of-life care, primary care, digital and virtual health, and health human resources. An overabundance of information, some of which was inaccurate, also was a major issue during the pandemic when Canadians needed sources of reliable information on COVID-19. Misinformation endangered the population's health, especially when related to false prevention measures or treatments, or when undermining trust in health services and public or political institutions.^{Footnote 25}

Groups experiencing health inequalities

The HCPSP includes consideration for groups experiencing health inequalities, through formalized Sex and Gender Based Analysis Plus (SGBA+) requirements. In program documents, language related to SGBA+ considerations can be found in the Program authorities, contribution agreements, applicant guidance documents, and reporting templates for funding recipients. Updates to the Program's T&Cs in 2020 resulted in stronger language related to consideration of SGBA+ issues, moving from "encouraging" to "requiring" recipients to consider such issues. In addition, the updated T&Cs added consideration of diversity factors.

The HCPSP funded projects that target groups experiencing health inequalities. For instance, the HCPSP 2020-21 Annual Results Report notes that in 2020-21, of the 41 funded projects, 22 were targeting underserved populations, some of which included multiple target groups.

Figure 2 - Text description

Figure 2 is a vertical bar chart with blue bars on a white background depicting the number of projects targeting groups experiencing health inequalities in fiscal year 2020-21.

• Indigenous: 10 projects
• Children/youth: 6 projects
• Seniors : 5 projects
• Linguistic minorities: 5 projects
• Cultural/ethnic minorities: 5 projects
• Rural communities: 4 projects
• LGBTQ2+: 2 projects
• Women: 2 projects
• Homeless/vulnerable: 1 project

Source: HCPSP 2020-21 Annual Results Report

Alignment with Government of Canada priorities

HCPSP priority areas continue to align with Government of Canada priorities. HCPSP priorities are aligned with the 2017 Common Statement of Principles on Shared Health Priorities, which notes the importance of improving access to mental health and home and community care, including palliative and end-of-life care.^{Footnote 26} The Program also supports the implementation of the Action Plan on Palliative Care.

Recent Speeches from the Throne (2019 to 2021) mentioned mental health as a priority. In addition, the Minister of Health's 2021 mandate letter discusses the need to expand the delivery of mental health services, including for prevention and treatment.^{Footnote 27} In 2021, the Government of Canada also created a new federal cabinet position: the Minister of Mental Health and Addictions. Ensuring that mental health care is treated as a full and equal part of Canada's universal health care system is a key responsibility of the Minister. In February 2023, FPT First Ministers agreed on shared health priorities to improve integrated health care for Canadians. Shared priorities included supporting health workers and reducing backlogs, improving access to quality mental health and substance use services, modernizing the health care system with standardized health data and digital tools, and helping Canadians age with dignity, closer to home, with access to home care or a safe long-term care facility.^{Footnote 28} The HCPSP has funded projects that address each of these areas.

Complementarity and duplication

Health Canada has several contribution programs that relate to health care and address issue areas similar to those of the HCPSP, including health care improvement, quality and patient safety, home and community care, and substance use and addictions.

Most internal and funding recipient interviewees indicated that the HCPSP is complementary to other programs. The HCPSP funds work that complements other funding programs, such as PHAC's funding of mental health promotion and mental health literacy. Seventy-eight percent of funding recipients surveyed agreed or strongly agreed that their HCPSP-funded activities complement those of other organizations operating in their field, and 81% of respondents disagreed or strongly disagreed that their activities overlap or duplicate those of other organizations.

A few funding recipients and internal interviewees stated that there are some potential areas of duplication. For example, in the palliative care priority area, two sets of similar trainings for caregivers were reportedly funded. In addition, a few internal interviewees mentioned that the priority area of other federal, provincial/territorial, and emerging priorities is very broad and might benefit from additional direction to help prevent future overlap with other programs.

The evaluation found that there are both formal and informal ways in which the Program avoids duplication and overlap with other programs. These processes include working closely with other groups, including policy units, other federal departments, such as Employment and Social Development Canada's (ESDC) Workforce Sector Solutions Program and Social Programs (e.g., ESDC's Aging Well at Home Program, New Horizons for Seniors Program), and applicants. In addition, the Program consults existing guidance documents, such as the Action Plan on Palliative Care. Furthermore, the HCPSP's Funding Opportunities Template requires policy leads to outline the rationale for the funding opportunity and explain how it links with previous projects or other projects, as well as how it aligns with Health Canada's policy objectives. In addition, the application guide provided to potential funding recipients asks them to consider other programs and to indicate how the project complements, aligns with, addresses gaps in, or builds on similar initiatives in other jurisdictions or at other levels.

Progress toward achievement of HCPSP expected outcomes

HCPSP-funded projects were largely effective in generating, accessing, and sharing knowledge products, such as standards and guidelines, guidance documents, literature reviews, training modules and learning supports, and formalized knowledge networks. The evaluation also found some evidence that target audiences were applying these products and that this had led to health care system changes, including changes to guidelines and policies and improvements in treatment and care for patients.

Key factors for project success were support from Health Canada officials to funding recipients and organizational readiness and project leadership within funded organizations. Impediments included funding recipient capacity issues due to organizational size and competing priorities. The COVID-19 pandemic created some delays and challenges for projects but also led to innovation.

Stakeholders generate, access, and share knowledge products, tools, standards, and training.

A variety of products, tools and trainings were developed by HCPSP-funded projects, including:

• Standards and guidelines on health system approaches;
• Recommendations and guidance documents for health care professionals;
• Scoping and literature reviews;
• Training, learning modules and learning supports; and,
• Formalized knowledge networks.

Projects disseminated knowledge products using a variety of mechanisms including webinars, workshops, conferences, meetings, engagement sessions, social media platforms, and email lists. There was a shift from in-person to virtual delivery when the COVID-19 pandemic emerged.

There were specific examples of stakeholders generating, accessing, and sharing knowledge products from HCPSP projects:

• The Health Standards Organization (HSO) project held the Improving Integrated Care for Youth (IICY) Initiative to launch their Learning Collaborative. The aim was to make sustainable improvements through the integration of community-based mental health and addiction services using co-design, testing, and adoption of evidence-based standards and implementation tools that promote integrated care best practices. This symposium was attended by 121 participants from a wide variety of health agencies across the country.
• The Canadian Home Care Association (CHCA) project, Operational Excellence: Home-based Palliative Care, identified leading operational practices in home-based palliative care to equip decision makers with the information and skills to advance change management. The project generated knowledge products to identify Opportunities and gaps in home-based palliative care. In addition, a total of 824 participants attended subject matter expert webinars, which aimed to support different organizations across Canada in implementing leading practices in their communities.
• A project led by the Pallium Foundation of Canada worked to scale up the design, development, and delivery of inter-professional palliative care training, tools, and resources, including their Learning Essential Approaches to Palliative and End-of-Life Care (LEAP) courses. Between 2018 and 2020, Pallium conducted 1,143 LEAP training sessions, with almost 20,000 participants. In addition, as of March 2021, they had delivered six webinars with a total of 1,106 participants.
• The McMaster University project, DIVERT ECHO (Detection of Indicators and Vulnerabilities of Emergency Room Trips scale Education, Change, Outcomes), created knowledge products to provide informal caregivers with general skills to better support patients, thereby reducing caregiver distress and helping to avoid unnecessary emergency room visits. Their reported webpage views for these products increased from 2020-21 to 2021-22, sometimes significantly, as seen in Table 1.

• The Canadian Virtual Hospice Tools project expanded web-based offerings and developed new resources and services to meet the needs of underserved communities including Francophones across Canada, the 2SLGBTQI+ community, and families caring for a dying child. Project reporting noted increases of 17% to 52% in website traffic in the first quarter since the project concluded. By March 31, 2022, 68,000 users, across every province and territory in Canada, had accessed their tools.
• Primary Health Care Integrated Geriatric Services Initiative, an Alberta Health Services project, worked to enhance integrated health care in Alberta to support older adults with chronic health issues, including dementia, and their caregivers. In 2021-22, they held a panel discussion with 46 participants, a conference display with 35 participants, six presentations with an average attendance rate of 55 participants, and nine workshops and webinars with an average of 28 participants each.

Stakeholders apply knowledge products, tools, standards, and training

Only some of the reviewed projects were able to report on the degree to which target audiences were applying or using their knowledge products. Furthermore, several of the project reports included evidence that was vague or didn't include any qualifying or explanatory information. For example, evidence of use in one project was described as follows: "The remaining teams are using the tools and learnings in other practices." Another project noted that "metrics were not collected for all knowledge products for reasons including burden on network partners and reporting expected at later dates."

Some projects still in progress reported on how their stakeholders intend to apply knowledge products once they became available for use. For example, for the Health Standards Organization (HSO) project, 72% of surveyed stakeholders indicated that they intend to use the Integrated Care Assessment Tool to review and update current strategies, policies, and procedures, and to benchmark progress.

At the same time, evidence of knowledge products being used was found. According to the HCPSP 2020-21 Annual Results Report, most projects for the 2019-20 and 2020-21 fiscal years reported that their target populations applied training, implemented standards, or used products (95% in 2019-20 and 78% in 2020-21).

Examples of reported use of knowledge products included:

• For Pallium's LEAP training sessions, 92% of participants reported that they acquired the knowledge and skills to implement improvements in their priority areas, and 72% of stakeholders, including physicians, nurses, pharmacists, social workers, paramedics and others, reported using the data to inform their work.
• The mobile and web-based BC COVID-19 Support Application was developed by the BC Ministry of Health as a communication channel that provided information and provincial resources related to the pandemic. It also contained a self-assessment tool that provided recommendations regarding further assessment by a health care provider or facility based on symptoms provided by the user. The project team surveyed 243,000 users, and 12% reported using the knowledge they gained from the App to inform their personal behaviours during the pandemic, exceeding the target of 10%. After using the self-assessment tool, approximately 8% of the users were reportedly able to avoid a health care interaction.
• The Canadian Home Care Association Partners in Care project, Preparing Family Caregivers in Supporting Patients of Home Restorative Programs, reported that 75% of the 20 stakeholders surveyed had applied new emotional intelligence competencies to improve care for frail seniors at home who are recovering from hospital visits.

In addition, a few projects reported that stakeholders found knowledge products to be useful. For example, 80% of stakeholders indicated that the Canadian Virtual Hospice tools were helpful, and almost half of stakeholders for the Canadian Home Care Association project reported that the knowledge products and learning opportunities were useful. According to the HCPSP 2020-21 Annual Results Report, overall, the percentage of stakeholders reporting that training or standards were useful in modernizing the health care system was 90% in 2019-20 and 89% in 2020-21.

Health care system changes

Of the projects reviewed, the average length of the contribution agreement was approximately three to four years. A few HCPSP projects, like Choosing Wisely Canada and the Canadian Medication Incident Reporting and Prevention System Program, were continuations of earlier projects. Because many of the projects were still in progress, either due to planned schedules or as a result of COVID-19 pandemic-related delays, evidence on the achievement of longer-term outcomes was limited. Furthermore, for many projects, long term outcomes would be expected in the future, following project completion, rather than during or immediately after the project ends.

In terms of reporting on longer-term outcomes, the mid-year and year-end progress report templates only ask for outputs and do not include sections for outcomes. As a result, information on project impacts may not be available until the final project performance measurement reports are completed. For longer-term and completed projects, there were some examples of health care system improvements:

• Pallium's LEAP training sessions were used by health care professionals to make changes in their workplaces, such as earlier initiation of palliative care, improved use of opioids and other medications, increased patient screening, better pain management strategies, and advanced care planning. This training was also reported to have helped change organizational cultures within health care settings, such as in long-term care and hospitals, because it led to more health care professionals speaking a common language on palliative care.
• The BC COVID-19 Support App allowed British Columbians to contribute to the pandemic mitigation effort. The online self-assessment and mobile applications were used as tools in the province's COVID-19 response, providing a direct line of communication to connect with and educate British Columbia residents, while simultaneously contributing to the reduction of health care use.
• The McMaster University project, Building Community Paramedicine into the Canadian Healthcare Landscape: An Incremental Approach to Making 'Community Paramedicine at Clinic', helped expand the role of paramedics in the delivery of home care to seniors in subsidized housing to reduce unnecessary hospital visits.
• The Foundation for Advancing Family Medicine (FAFM) project, Re-evaluation of the College of Family Physicians of Canada Route to Certification without Examination for Internationally Trained Family Physicians, allowed physicians from Australia, Ireland, the United Kingdom, and the United States to receive expedited certification to practice in Canada. This contributed to the recruitment of family physicians in Canada. While in 2021, 235 physicians were certified through this route, the number increased to 354 in 2022, and continues to rise in 2023 (numbers are still incomplete).
• The Institute for Safe Medication Practices Canada (ISMP) project, Canadian Medication Incident Reporting and Prevention System (CMIRPS), receives medication incidents from across the country, and this information is used to augment the evidence base for medication safety improvements in all health care settings. An external project-level evaluation noted that CMIRPS has positively impacted the Canadian health system by contributing to greater medication safety, reduced harmful medication incidents, and improved outcomes for patients. Further, it concluded that CMIRPS results in significant savings to the Canadian health care system.

Other projects described anticipated health care system improvements. For example, an economic analysis done for the Connecting People and Community for Living Well project, by Alberta Health Services, found that, if the model spreads to seven additional rural Alberta communities, the expected annual return on investment is estimated to be between $3.90 and $9.40 for every dollar invested. Similarly, for the Development of a Needs-Based Planning Model for Mental Health and Substance Use/Addiction Services and Supports across Canada project, led by the Centre for Addiction and Mental Health (CAMH), five provinces have either begun to or have already implemented a core services framework based on the national needs-based planning model. This will allow key decision-makers across Canada to better estimate the resources required to address the need for mental health and substance use services in their jurisdictions.

Factors affecting project success

Health Canada support

Many funding recipients noted that the support and collaborative approach of Health Canada were key factors for project success. In interviews with funding recipients, support from Health Canada staff was one of the most commonly cited factors in project success. A few funding recipient interviewees indicated that the HCPSP Unit is the best relationship they've had with a federal funder. Many noted that relationships with individual Health Canada employees were a strength of the Program, particularly when the relationships were longstanding. Support from Health Canada was reported to be timely and constructive, providing help in problem solving, planning, and working with shared goals in mind. Health Canada staff were seen to be available when needed and open to engagement with funding recipients.

Funding recipient survey respondents were also positive about the direction and support they received from Health Canada, with 88% agreeing or strongly agreeing that, overall, they were satisfied with the support and guidance provided by Health Canada. The vast majority also agreed that this support and guidance was useful, timely, clear and consistent (see Figure 3). Furthermore, almost 70% of funding recipient survey respondents reported that HCPSP Unit's guidance and support contributed to project success.

Figure 3 - Text description

Figure 3 is a horizontal bar chart with blue bars on a white background showing percentages of funding recipient survey respondent views on aspects of Health Canada's support and guidance.

• Consistent: 78%
• Clear: 81%
• Timely: 88%
• Useful: 94%
• Overall satisfaction: 88%

Many funding recipient interviewees indicated that the Program's flexibility in providing amendments, where necessary, was a critical factor during the pandemic. For example, both Canadian Virtual Hospice and Pallium received amendments to adapt activities to the COVID-19 context, including moving to online webinars and modules. One funding recipient interviewee noted that quick pivoting and the corresponding results were only possible due to the collaborative approach taken by their group and Health Canada.

Financial support from Health Canada was also seen by funding recipients as critical to their projects. Survey data found that almost 70% of funding recipients thought the funding provided contributed to project success. Some funding recipients indicated that support from the provinces and territories was also useful for projects, and that having Health Canada as a funder added credibility to their projects. At the same time, several funding recipient interviewees did note that the level of funding provided did not allow for scalability and sustainability, and 15% of survey respondents reported that funding levels impeded their project success.

In addition, the length of the contribution agreements was generally seen as enabling project success among funding recipient interviewees and for 56% of survey participants. However, a few funding recipient interviewees noted delays in setting up their contribution agreements or receiving funding because of process delays.

While support from Health Canada was seen positively overall, several funding recipient interviewees mentioned that turnover in Health Canada staff resulted in a loss of Health Canada's institutional knowledge, which sometimes led to challenges for projects. For example, a few noted that it required more time to introduce new Health Canada staff to the projects, and to rebuild rapport and understanding.

Organizational readiness

Organizational readiness, including project leadership, was seen as a precursor to project success, with some funding recipient interviewees noting the advantages of having a positive organizational reputation and solid networks in place. Additionally, 87% of funding recipient survey respondents agreed that project leadership contributed to project success. Almost 60% reported that having a clear workplan and defined objectives were also critical factors. A few funding recipient interviewees noted that they struggled with capacity issues due to their organization's size, the pandemic, and competing external priorities.

COVID-19 pandemic

The COVID-19 pandemic resulted in challenges for projects, including project staff having to refocus on family priorities and clinical work, delays in project deliverables, and shifting timelines. It was the most common impediment to project success identified by funding recipient survey respondents (40%). As a result of the pandemic, all projects examined by the evaluation had amendments to their contribution agreements, such as time extensions, changes to funding arrangements, and removal of key deliverables, like in-person events.

Nonetheless, adjustments due to the pandemic contributed to the success of some projects. Some funding recipient interviewees noted that the shift to virtual delivery of events expanded their reach beyond their initial objectives. The HCPSP 2020-21 Annual Results Report noted that virtual delivery models allowed participation from anywhere, thus increasing the reach of the learning activities to the target audiences. It also noted the increased demand for virtual learning events on the topics covered by the priority areas, such as palliative and end-of-life care, and COVID-19 skills for graduating and new nurses, all of which attracted a greater number of participants than originally expected.

Program oversight

The HCPSP has established some of the necessary oversight and accountability elements for projects managed in part by other groups within Health Canada, and where other groups use the HCPSP's authorities for funding for Budget-related initiatives. These elements include oversight through governance committees and informal understandings among groups. However, there is a lack of formal articulation of roles and responsibilities at a more operational level for other groups using the HCPSP's authorities.

While some projects are managed entirely by the HCPSP Unit, different groups within Health Canada manage some key aspects of other projects using the HCPSP's authorities. There were changes to the T&Cs in 2020 to allow this approach. Updated wording indicates that while the HCPSP Unit is responsible for program coordination and management functions, other directorates may also take responsibility for certain program management functions.

At minimum, the HCPSP Unit is involved in issuing the solicitation of all projects using the HCPSP's authorities. Depending on the initiative in question and the informal arrangements in place with other units, the HCPSP Unit may also be involved in other aspects of project cycles, such as planning, review process, funding recommendation, performance measurement and reporting, and recipient auditing. Generally, the projects can be divided into four categories:

• Projects that receive HCPSP core funding, use HCPSP authorities and are managed entirely by the HCPSP Unit;
• Projects that receive HCPSP core funding and use HCPSP authorities, but where some aspects of the project cycle are managed by other groups;
• Projects that use HCPSP authorities but not the Program's core funding, and that are managed entirely by the HCPSP Unit; and,
• Projects that use HCPSP authorities but not the Program's core funding, and where many aspects of the project cycle are managed by other groups.

The HCPSP Unit works with staff from other directorates on the projects that are managed outside of HCPSP, including the Mental Health Directorate and the Health Programs and Strategic Initiatives Directorate. In addition, they work collaboratively with the Office of Grants and Contributions to align with other programs and departmental policy, as necessary. For all programs within the Strategic Policy Branch, the Office of Grants and Contributions undertakes quality assurance reviews of project materials using their Quality Assurance Review Checklist for Contribution Agreements.

According to a recent Health Canada audit, effective governance in the context of G&Cs should include clear roles, responsibilities, and accountabilities, as well as the provision of leadership, oversight, and challenge functions^{Footnote 29}. It appears that the HCPSP has established some of these elements, including oversight through the Program's governance committees and informal understandings on roles and responsibilities. However, the Program is lacking a more formalized and detailed documentation of roles and responsibilities at an operational level for other groups using the HCPSP's authorities.

Governance

The HCPSP's T&Cs provide overall guidance for the Program and outline the types of organizations and activities eligible for funding. The governance structure for the HCPSP consists of a Program Management Committee (PMC) and Program Advisory Committee (PAC). The PMC is intended to provide strategic oversight and direction for the HCPSP. The PAC includes representation from groups across the Strategic Policy Branch, including those connected to the priority areas and groups managing projects under the HCPSP's authorities. It is intended to provide input and advice to support key priority issues and make recommendations to the PMC. This work includes identifying policy priorities, potential projects, and improvements for internal and external program communication.

The two governance committees have not been meeting as regularly as planned in their respective Terms of Reference. Some internal interviewees suggested that the regular meetings should resume. In addition, some interviewees also noted that the PMC could play a more strategic role for the Program in activities such as identifying priorities.

Roles and responsibilities

While each governance committee has adopted formal Terms of Reference, other formalized governance processes are limited. The HCPSP's Performance Measurement Strategy outlines roles and responsibilities specifically related to performance measurement and evaluation, for three categories of projects under the HCPSP's authorities. However, this strategy is currently only a draft. No additional documents outlining roles and responsibilities were available. Internal interviewees confirmed that there is no formal process in place for oversight of activities associated with the HCPSP's authorities wherein one or more phases of the project cycle are managed outside of the HCPSP Unit. The majority of internal interviewees felt that there should be such a process.

Key arguments for and against establishing more formalized accountabilities are outlined in this section.

Support for more formalized accountabilities

• The establishment of more formal accountabilities is a best practice for effective governance of G&Cs programs.
• A lack of clear roles and responsibilities for projects is a potential risk to the Director General responsible for the HCPSP.
• There have been instances where there was a lack of clarity regarding who was responsible for a specific task, such as preparing a memorandum to the Minister, or working with recipients to develop performance measurement plans and reporting.
• The current lack of documentation could present issues for new employees.
• There are operational benefits that come from having a map of program processes, an articulation of clear roles and responsibilities, and an interpretation guide for the T&Cs.
• Treasury Board submissions using HCPSP project authorities outline program commitments, but do not outline operational roles and responsibilities.

Reasons more formalized accountabilities are not seen to be required

• There are already informal processes in place. For example, discussions take place with groups using the T&Cs about roles and responsibilities at the outset of each project. No major issues have arisen to date.
• Some aspects of the project process are consistently undertaken by the HCPSP Unit, such as issuing calls for proposal.
• Informal relationships among staff help with program accountability.
• There are currently checks and balances, such as funding approval forms that require ADM-level approval for all projects, as well as memoranda to the Minister for information about when a call for proposals is to be launched, and before funding for a project is approved.

Internal interviewees noted that an accountability document had been started but was stalled due to human resource capacity issues and the complexity of various informal arrangements currently in place. Going forward, it would be important to consider the ideal level of detail in such a document, in the context of the Program's capacity and the level of risk involved.

Program delivery

Overall, the support from Health Canada to funding recipients is a strength of the Program. The Program has made extensive improvements in performance measurement and has introduced various enhancements for reporting results. However, Health Canada does not share information on projects or lessons learned with funding recipients or internally. Furthermore, use of this information in Health Canada's decision-making appears minimal. Sharing this information could benefit funding recipients in their projects and the Program as a whole.

The flexibility of the HCPSP was important for success. The priority area of other federal, provincial/territorial, and emerging priorities allowed the Program to respond to unanticipated priorities and fund innovative projects. It was particularly useful in light of the COVID-19 pandemic. In addition, the expanded use of the HCPSP's authorities for Budget-related initiatives that are mostly managed outside the HCPSP Unit allowed for a wider variety of issues, activities, and organizations to be funded.

Direction and support for funding recipients

Health Canada provides direction and support to funding recipients throughout the project cycle, including for the application, implementation, performance measurement, and reporting processes. For the application process, potential funding recipients receive an application package from Health Canada. This package includes a standardized application form and a detailed HCPSP Guide for Applicants, which provides step-by-step instructions and guidance on completing the HCPSP application documents. It also contains key background information on the Program, the application process, and eligible expenses and activities. According to program representatives, these documents were developed as part of their efforts to streamline the application and review processes to better support applicants.

Once projects are approved for funding, Health Canada provides direction and support to funding recipients in the management of the contribution agreement, for example, determining eligible expenses and negotiating amendments. In addition, Health Canada provides support for performance measurement planning and reporting through the Performance Measurement Guide for Recipients, which is described in the section below.

Performance measurement and reporting

To address the need for better collection and use of project performance information, as identified by the previous evaluation, the Program updated the HCPSP's performance measurement approach, using an iterative process, starting in 2020. This led to a standardized logic model, a Performance Measurement Planning and Reporting Template, and a Performance Measurement Strategy, including standardized performance indicators, a standardized recipient performance measurement and reporting tool, and guidance documents for recipients and program officers. The Performance Measurement Guide for Recipients includes general background on performance measurement, definitions of key terms, the HCPSP's logic model, guidance on selecting project-relevant indicators, baseline measures, and targets. In addition, there is a step-by-step guide to completing the Excel-based performance planning and reporting template, as well as a section of frequently asked questions based on previous questions and feedback from recipients.

In addition, the HCPSP Unit includes a performance measurement expert who provides guidance to funding recipients on issues related to project performance measurement and reporting. Most funding recipient interviewees highlighted this support, describing staff as being available, collaborative, knowledgeable, and helpful.

The Office of Grants and Contributions has used the HCPSP's performance measurement support documents as best practice examples to provide advice to other groups within Health Canada. Several funding recipient interviewees commented on the changes to Health Canada's performance measurement and reporting approach over the years. The majority praised the changes, noting that the new approach works better for them. One interviewee noted that they had received positive feedback on the HCPSP reporting and evaluation templates after having shared them with an international working group. Other funding recipient interviewees raised concerns about the approach. For example, some believed the approach is overly onerous, contains redundancies, focuses on outputs instead of impacts, and disadvantages small organizations, due to its complexity and the level of effort required.

In addition to the new performance measurement approach, the Program also began developing products to share program results. This work included experimenting with various formats for an Annual Results Report and infographics, and undertaking case studies. An external expert was contracted to produce two case study reports, providing a more in-depth examination of two projects in 2020-21 and three projects in 2021-22. The case studies were only produced for two years because the HCPSP Unit determined that the studies did not provide the in-depth analysis that was expected. In 2022-23, the HCPSP Unit introduced new post-project questionnaires oriented to program officers and policy officers involved in projects. It is hoped that these questionnaires will help identify best practices and lessons learned from completed projects, which can inform program decisions on future projects.

Use of HCPSP project performance information

In terms of the use of project results within Health Canada, several internal interviewees noted that project performance information is used primarily for corporate reporting responsibilities and accountability. A small number also suggested it was used to inform policy work within the department, to identify lessons learned for the future, and to recognize any unintended consequences that could inform future projects.

Although the Project Performance Measurement Guide for Recipients contains information on the use of performance information, no funding recipient interviewees knew if Health Canada was using the performance information and reporting they provide, and several noted that Health Canada was a 'black box' in this regard. Others believed the information was collected strictly for accountability purposes. Meanwhile, less than half of funding recipient survey respondents (44%) agreed that Health Canada used the information they provided through project reporting.

Sharing lessons learned and connecting projects

While there is some information on projects funded by the HCPSP on Health Canada's proactive disclosure website and in the Departmental Results Report, the Health Canada webpage for the HCPSP is very limited and only provides brief descriptive information on the Program, including the four priority areas, eligible recipients, and a contact email address. It does not contain any information on projects that have been funded under the HCPSP or any project results or lessons learned.

In terms of sharing lessons learned with project recipients, a section of the Performance Measurement Guide for Recipients includes Frequently Asked Questions, which are based on lessons learned through earlier projects. However, there was no evidence of Health Canada sharing this type of information further, or of sharing collated information on projects with funding recipients, a view shared by most funding recipient interviewees and survey respondents. These findings are consistent with those from the previous evaluation, which noted an opportunity for improvement in terms of collating and sharing project information.

Both internal and funding recipient interviewees noted that sharing of information on other HCPSP projects could help projects to learn from one another, preventing projects from working in silos, preventing duplicative work, and helping to identify gaps or areas needing more attention. Several funding recipient interviewees mentioned that without a feedback loop, it is difficult to understand the value of their projects, how to build upon past projects, and how to mobilize and translate the knowledge gained from their projects. Furthermore, some internal interviewees mentioned that understanding the lessons learned from project funding recipients and placing an emphasis on knowledge translation could improve program efficiencies.

Several internal and funding recipient interviewees, as well as funding recipient survey respondents, noted that Health Canada should take on more of a leadership role to improve connections and information sharing among projects. For example, by arranging a regular meeting or symposium to bring groups together, and by expanding the HCPSP's website to provide information for funding recipients and overview information on projects. At the same time, some internal interviewees noted the challenge of identifying generalizable lessons learned given how different projects are from one another, as well as capacity constraints.

As mentioned above, the Program is starting to conduct post-project questionnaires for Health Canada staff to identify lessons learned; however, this process is still in progress.

Factors affecting program success

Program flexibility

The flexibility of the HCPSP was identified by internal and funding recipient interviewees as a key factor for program success. This flexibility allowed funding recipients to make amendments, adjust timelines, and shift finances, in response to challenges encountered or changes in how the projects were unfolding. Furthermore, the increased flexibility and breadth of the HCPSP's authorities, including the use of the T&Cs to flow funding for various budget-related initiatives, contributed to program success because it allowed for a wider variety of issues, activities, and organizations to be funded. Many also noted that the fourth priority area of other federal, provincial/territorial, and emerging priorities was invaluable in terms of allowing the Program to respond to a variety of unanticipated priorities, and enabled program staff to move quickly to fund innovative projects. This priority area was particularly useful given the emergence of the COVID-19 pandemic. The HCPSP was able to quickly fund projects to support the pandemic response, like the BC COVID-19 Support Application project that created a communication channel to provide information and provincial resources during the pandemic, and the COVI Canada project that developed a COVID-19 mobile alert application to support users in making decisions about their activities during the pandemic.

The Program was also considered flexible because funding was not formally allocated across the four specific priority areas. This flexibility was seen as being advantageous because it allowed the Program to fund more projects in one priority area or another, depending on gaps and opportunities, rather than arbitrarily limiting priority areas to a set proportion of the funding. However, a few internal interviewees noted that the lack of targets made it unclear at times what funding might be available. That said, the Program sought balance by encouraging those working in each of the four priority areas to propose project ideas for funding consideration.

Some Health Canada interviewees described the HCPSP as an incubator program, allowing for quick injections of money that give projects the support to build up when needed. However, some noted that this flexibility can only exist if the HCPSP funds are available and not fully committed.

Human resource capacity

The most frequently mentioned factor impeding program success was human resource capacity within the HCPSP Unit. According to program representatives, the Unit was consistently understaffed over the last few years. Based on a review of quarterly organizational charts for the last five years, on average, the HCPSP Unit was 26% below full capacity. This ranged from 11% in some quarters to 50% in others, and there were no periods with a full staff complement.

As a result of these capacity challenges, internal interviewees reported that it was sometimes necessary to leverage resources from other groups. Capacity issues were reported to contribute to feelings of burnout for existing staff and to limit the amount of work that could be completed on rolling up and sharing project information, developing oversight and guidance documents, engaging with recipients as thoroughly as desired, and briefing senior staff. Some also mentioned that limited capacity in the HCPSP Unit put the Program at risk of losing the nimbleness and responsiveness that is seen as such a highlight for users, because staff may not be able to respond quickly to opportunities or challenges.

Many internal interviewees noted that staff retention is an ongoing issue and that finding replacements is difficult because of the limited number of employees across the federal government with G&Cs knowledge, which is critical for the work. Interviewees noted that this is an issue across the government and not just within the HCPSP Unit. One internal interviewee also noted that the Canada School of Public Service ended their G&Cs training program during the COVID-19 pandemic, making it extremely difficult for staff to access required basic training. Furthermore, this means that existing staff must take on the role of trainer for new staff.

Program approach

Overall, the Program approach is effective, with 84% of funding recipient survey respondents agreeing that the HCPSP is a well-run program. As discussed above, the increased use of program authorities was seen as a success factor. However, this issue was also an impediment since it created additional work and responsibilities for the HCPSP Unit. Internal interviewees indicated that the breadth of project funding sources, budget announcement commitments, and differing groups managing projects led to a complex program environment that put additional pressure on staff.

Many internal interviewees described symbiotic relationships between the different groups and between policy and program staff as enablers for the Program's success. Some noted that they had worked effectively together, with policy groups being well-positioned to identify key stakeholders, subject area expertise, and potential projects, and with the Program delivering large amounts of funding to projects to help address priorities.

The Program's use of targeted calls for proposals was also identified as a success factor, particularly in light of the Program's human resource capacity challenges. Several internal interviewees noted that Health Canada lacks a system to efficiently run open calls for proposals, and that to do so without some form of automation would be overwhelming. The HCPSP Unit has used, in limited circumstances, an open-targeted proposal approach where a small number of organizations may compete for funding. Targeted calls allow for the identification of organizations that can deliver relevant projects and provide flexibility for emerging issues. The associated processes and timelines are much more streamlined than they would be for open calls for proposals. A few internal interviewees indicated that a risk of this approach is that they could miss out on potential new innovative projects and other groups capable of delivering quality projects. As such, interviewees noted that an ideal approach would include a mix of targeted and some kind of open process for solicitation.

Some internal interviewees also noted that multi-year planning, including projects with longer contribution agreements, has helped the Program function effectively in terms of capacity challenges by minimizing the lapsing of funds and helping to manage projects more efficiently.

Program spending

Information on program spending was for G&Cs funding only, given that the Program and other groups using HCPSP funding and authorities do not track other resources separately across initiatives for which they are also responsible.

Over the period 2018-19 to 2022-23, the total core funding for the HCPSP was $128.5M (see Table 3). With respect to actual spending, approximately $80.1M was spent on projects under HCPSP's authorities and were the focus of this evaluation. In addition, due to reasons of efficiency (e.g., other contribution agreements already in place with certain organizations), or through special requests to meet needs in other areas with funding challenges, approximately $15.7M was transferred to other groups in Health Canada for projects related to HCPSP objectives, but under different authorities. For example, projects related to long-term care standards, and organ and tissue donation and transplantation were funded through these transfers. Furthermore, $21.6M was transferred to projects addressing other Health Canada priorities. This included a transfer of $12M in fiscal year 2018-19 related to the move of the First Nations and Inuit Health Branch to the newly formed Indigenous Services Canada. HCPSP Unit representatives say they have made significant efforts to identify funding opportunities for project spending on HCPSP objectives that relate more closely to departmental policy priorities.

Conclusions and recommendations

Conclusions

The HCPSP continues to address key needs within the Canadian health care system. The HCPSP's priority areas are consistent with Government of Canada priorities, including those related to the health care system and supporting groups experiencing health inequalities. The HCPSP is generally complementary to the work of other programs, and there are both formal and informal processes in place to help prevent or limit duplication of other programs' activities.

HCPSP-funded projects were largely effective in generating, accessing, and sharing knowledge products. These included standards and guidelines on health system approaches, guidance documents, literature reviews, training modules and learning supports, and formalized knowledge networks. The evaluation also found some evidence that target audiences were applying these products and that this had led to health care system changes, including changes to guidelines and policies, and improvements in treatment and care for patients. Organizational readiness and project leadership within funded organizations, as well as Health Canada direction and support to funding recipients, were key factors that contributed to project success, while capacity limitations within some funded organizations created impediments for some projects. The COVID-19 pandemic created some delays and challenges for projects but also led to innovation.

At the program level, the HCPSP has established some of the necessary oversight and accountability elements for projects managed in part by other groups within Health Canada and where other groups use the HCPSP's authorities to flow funding for Budget-related initiatives. This includes governance committees and informal understandings on respective roles and responsibilities with other groups. However, it is lacking a formalized description of roles and responsibilities for other groups using the HCPSP's authorities.

Overall, the support from Health Canada to funding recipients throughout the project process, including application, implementation, performance measurement, and reporting, is a strength of the Program. The Program has made extensive improvements to the HCPSP's performance measurement approach and has introduced various enhancements for reporting results. However, Health Canada does not share information on projects or lessons learned with funding recipients or internally. Furthermore, the use of this information in Health Canada's decision making appears minimal. Sharing this information could allow projects to learn from one another, avoid working in silos, prevent duplication, and identify gaps. Internally, program efficiency could be improved by better understanding and sharing the lessons learned from projects.

The flexibility of the HCPSP was important for its success. The priority area of other federal, provincial/territorial, and emerging priorities allowed the Program to respond to a variety of unanticipated priorities, fund innovative projects, and was particularly useful in light of the COVID-19 pandemic. In addition, the expanded use of the Program's authorities for Budget-related initiatives allowed for a wider variety of issues, activities, and organizations to be funded, and for the Department to more easily respond to Ministerial and government-wide priorities and initiatives. However, human resource capacity issues within the HCPSP Unit contributed to feelings of burnout and limited the Program's ability to complete some planned work. Work areas affected included the sharing of lessons learned and the finalization of a roles and responsibilities document.

Recommendations

Recommendation 1: Increase the sharing of information both externally and internally, including project information and lessons learned.

Overall, the support Health Canada provided to funding recipients was a strength of the Program. However, the lack of information sharing on projects and lessons learned was a gap in this support. This issue was also identified as an area for improvement in the previous evaluation. Sharing this information with funding recipients and internally within Health Canada could benefit funding recipients in their projects and could benefit the overall program.

Recommendation 2: Clarify program accountabilities, which could include formal articulation of key process steps and associated roles and responsibilities for projects managed by those outside of the HCPSP Unit.

The HCPSP includes some of the necessary elements to help ensure effective oversight and accountability, including oversight through governance committees and informal understandings of respective roles and responsibilities with other groups. However, it does not have documented roles and responsibilities at a more operational level for other groups using the HCPSP's authorities. This has led to a few instances of confusion about roles. Formal accountabilities are important for effective governance and reducing risk to the Program.

Management Response and Action Plan: Evaluation of the Health Care Policy and Strategies Program (HCPSP)

Recommendation 1

Increase the sharing of information both externally and internally, including project information and lessons learned.

Management response

HCPSP Management agrees with the recommendation.

Action Plan

Develop and implement a knowledge translation strategy to increase the sharing of information both internally and externally.

Deliverables

• Summary document with results from consultations with external and internal stakeholders on their information needs and potential options for information sharing.
  • Expected Completion Date: July 2024
• Knowledge translation strategy, with defined timelines for implementation, that would include activities related to sharing project information and lessons learned, both internally and externally.
  • Expected Completion Date: March 2025

Accountability

Director General, Health Care Programs and Policy Directorate, and Program Manager, HCPSP Unit (Strategic Policy Branch)

Resources

Existing resources

Recommendation 2

Clarify program accountabilities, which could include formal articulation of key process steps and associated roles and responsibilities for projects managed by those outside of the HCPSP Unit.

Management response

HCPSP Management agrees with the recommendation.

Action Plan

Clarify program accountabilities and formally articulate key process steps and associated roles and responsibilities for projects managed by those outside the HCPSP Unit.

Deliverable

Finalize DG-level agreements outlining key process steps along with the respective roles and responsibilities for projects using HCPSP authorities and either "core" or "non-core" funding.

Expected Completion Date

December 2024

Accountability

Director General, Health Care Programs and Policy Directorate, and Program Manager, HCPSP Unit (Strategic Policy Branch)

Resources

Existing resources

Annex A – Evaluation methodology and limitations

The evaluation team collected data using various sources and methods, as described below

Document and Performance Information Review
OAE undertook a review of key program documents and performance information, such as T&Cs, performance measurement guidance documents, available annual results reports, case studies, and infographics. In addition, evaluators reviewed detailed project-level documentation for 11 projects, including:

• Alberta Health Services, Primary Health Care Integrated Geriatric Services Initiative;
• Canadian Home Care Association, Operational Excellence: Home-Based Palliative Care;
• McMaster University, DIVERT ECHO (Education, Change, Outcomes) Project: Scaling Up Emerging Innovations in Home-based Chronic Disease Management across Canada;
• NCE-IKTP YOUTH MENTAL HEALTH (operating as Frayme/Cadre, Canada's Youth Mental Health and Substance Use Services and Systems Support Network;
• Health Standards Organization, Improving Integrated Care for Youth Initiative;
• British Columbia Ministry of Health, BC COVID-19 Support App;
• COVI Canada, COVI – An AI-enabled Health App to Fight COVID-19;
• Unity Health Toronto, Choosing Wisely Canada – Phase 3;
• Cree Board of Health and Social Services of James Bay, End of Life Care for Cree Patients from Eeyou Istchee, Quebec;
• The International Centre for Dignity and Palliative Care Inc. / Canadian Virtual Hospice, CaregiversCAN: Building on Progress to Support Caregivers; and,
• Pallium Foundation of Canada, Building and Bridging: Palliative Care is Everyone's Business.

Interviews
OAE undertook interviews with a total of 35 interviewees. This included:

• 19 internal to Health Canada; and
• 16 funding recipients.

Financial Analysis

Available financial information for the period 2018-19 to 2022-23 (Grants and Contributions funding only) was analyzed.

Literature Review

OAE conducted a focused literature review, which included academic papers and grey literature. This review focused on health care needs in Canada.

Survey of funding recipients

An electronic survey was distributed to 75 HCPSP funding recipients whose email addresses were provided by the Program. A snowball sampling approach was used, with known recipients being asked to share the survey with other potential respondents. In total, 32 funding recipients participated in this survey. Most respondents had been engaged with HCPSP as funding recipients for more than three years (n = 20 or 63%). Half of respondents (50%) were located in Ontario and 41% worked in multiple provinces and territories or nationally. In terms of priority areas, approximately 38% were from the other federal, provincial/territorial and emerging priorities area, 25% were from the mental health care area, 19% were from palliative and end-of-life care, and 19% were from home and community care.

The evaluation team used triangulation to analyze data collected by these various methods in order to increase the reliability and credibility of the evaluation findings and conclusions. The evaluation considered the SGBA+ lens in its assessment of the HCPSP, including consideration of groups that may experience health inequalities in projects funded. Official languages were not specifically examined, and they did not emerge as an issue for the Program's activities. Furthermore, an examination of the Sustainable Development Goals was not applicable for this evaluation.

In conducting the evaluation, a single window was identified from the Strategic Policy Branch, with whom the Office of Audit and Evaluation worked closely throughout the evaluation. The scope for this evaluation was shared with the Performance Measurement, Evaluation and Results Committee (PMERC) in January 2023. The preliminary findings were presented to the Executive Committee in October 2023, and the final report will be presented at PMERC in January 2024.

Limitation: Key informant interviews are retrospective in nature, providing only a recent perspective on past events.

Potential impact: This could influence the validity of respondents' assessment of activities or results that may have changed over time.

Mitigation strategy: The other lines of evidence were triangulated with the data received from interviews to substantiate or provide further information. Document review also provided corporate knowledge.

Limitation: There were some challenges with available performance information. There was limited performance data available for medium and longer-term outcomes. Furthermore, several of the project reports included evidence that was vague or did not include any qualifying or explanatory information.

Potential impact: This limited the evaluation's ability to report on the longer-term results of the HCPSP.

Mitigation strategy: Available data was used to provide examples of progress towards longer term outcomes.

Limitation: Complete planned and actual financial data for the Program was not available, for example salary, operations and management, and capital. Only planned and actual G&Cs expenditures were provided.

Potential impact: There is a limited ability to assess the efficiency of program activities quantitatively.

Mitigation strategy: Triangulation of other lines of evidence was used to substantiate or provide further information on program value.