Evaluation of the Health Care Policy Contribution Program 2013-14 to 2017-18

List of Tables

• Table 1:  Number of Projects Funded by Recipient Type 2013-2014 to 2017-2018
• Table 2:  HCPCP Contribution Funding 2013-14 to 2017-18

List of Acronyms

CIAF

Canadian Incident Analysis Framework 

CMIRPS

Canadian Medication Incident Reporting and Prevention System 

FAA

Financial Administration Act 

FMRI

Family Medicine Residencies Initiative 

HCPCP

Health Care Policy Contribution Program  

HCSIF

Health Care System Innovation Fund 

HHRS 

Health Human Resource Strategy  

IEHP

Internationally Educated Health Professionals  

IEHPI

Internationally Educated Health Professionals Initiative  

ISMP

Institute for Safe Medication Practices Canada 

PRA

Practice Ready Assessment

PT

Provincial/Territorial

Executive summary

This evaluation assessed the relevance and effectiveness of the Health Care Policy Contribution Program (HCPCP) for the period of 2013-14 to 2017-18. The evaluation was required by the Financial Administration Act and the Treasury Board of Canada's Policy on Results (2016). Due to changes occurring within the program during the evaluation timeframe, it was agreed that the evaluation would be focused in its scope and methodological approach.

Launched in 2002, the HCPCP is a national program that provides contribution funding to projects that address health care system priorities. Current HCPCP priority areas include palliative care, home care, mental health, and health care system innovation.

Evaluation Conclusions

In terms of relevance, the evaluation found that HCPCP priorities have evolved over time and there are continuing needs within the health care system related to current priority areas, including palliative care, home care, mental health, and health care system innovation.   

There was evidence of progress towards all three program outcome areas (stakeholders are aware of knowledge products and tools, stakeholders are using knowledge products and tools, and there are improvements in the health care system). Although direct measurement of awareness was limited, there was other evidence related to stakeholder awareness, including the involvement of some target audiences in knowledge product development, and the subsequent use of some of these products. Use varied, ranging from the development of frameworks and guidance documents based on knowledge products stemming from projects, to participation in training programs.  

Some projects led to improvements in the health care system, ranging from the adoption of professional standards, practices, and policies, to more physicians with rural and remote experience. However, evidence was limited on the impacts of these changes. Several of the projects that were able to demonstrate progress towards longer-term outcomes were continuations of previously funded projects, sometimes with multiple earlier phases.

Areas for Consideration

Given that the HCPCP continues to implement program redesign, this evaluation presents areas for consideration that the program can continue to build upon or take into account as it moves forward, rather than providing recommendations.

• At the project level, collaboration with relevant partner organizations and strong project leadership were seen to be the most crucial elements for ultimate project success, along with longer-term contribution agreements, evidence-based planning, and support from Health Canada (i.e., encouraging buy-in from stakeholders, and providing project guidance and advice).
• At the program level, the evaluation identified Health Canada's role in knowledge translation and strategic direction as areas for improvement, including use of project performance and progress reports and following up on completed projects to better determine the achievement of longer-term outcomes, as well as program support for innovative projects.

1.0 Evaluation purpose

The purpose of the evaluation was to assess the relevance and effectiveness of the Health Care Policy Contribution Program (HCPCP) for the period of 2013-14 to 2017-18. The evaluation was required by the Financial Administration Act (FAA) and the Treasury Board of Canada's Policy on Results (2016).

2.0 Program Description

Launched in 2002, the HCPCP is a national program that provides contribution funding to projects that address health care system priorities. The HCPCP is managed by the Health Care Programs and Policy Directorate of Health Canada's Strategic Policy Branch.

The HCPCP was developed as one of several mechanisms to address issues identified by the Standing Senate Committee on Social Affairs, Science and Technology, and the Commission on the Future of Health Care in Canada in 2002. The HCPCP also aimed to address health care system priorities, as identified in the First Ministers' 2003 Accord on Health Care Renewal and the First Ministers' 10 Year Plan to Strengthen Health Care (2004), often referred to collectively as the Health Accords.

The HCPCP is intended to support the Government of Canada's role in health care by supporting the development of policies and strategies to address evolving health care system priorities. The HCPCP is based on a theory of change, including the concept of knowledge translation, which is the active process of synthesizing, disseminating, exchanging, and implementing knowledge to ultimately improve the health of Canadians. 

During the timeframe of the evaluation, the HCPCP funded projects through three components: 1) the Health Human Resource Strategy; 2) the Internationally Educated Health Professionals Initiative (IEHPI); and, 3) the Health Care System Innovation Fund. Funding recipients included provincial and territorial (PT) governments, and non-governmental organizations (NGO), such as educational institutions and not-for-profit organizations (see table below).

Table 1:  Number of Projects Funded by Recipient Type 2013-2014 to 2017-2018^{Footnote a}

Year	PT	NGO	Total
2013-14	13	8	21
2014-15	14	14	28
2015-16	12	11	23
2016-17	2	13	15
2017-18	0	12	12
Table 1 - Note a For each fiscal year, these numbers include all active projects, including new contribution agreements, as well as ongoing projects, many of which received funding over multiple years. Return to Table 1 - Note a

Expected Outcomes

In cooperation with program representatives, the evaluators considered the various HCPCP outcomes and generalized them to three key outcome areas ^{Reference i}:

• Stakeholders are aware of knowledge products and tools;
• Stakeholders are using knowledge products and tools; and
• Improvements in the health care system.

Program Resources

The table below presents the HCPCP's financial data for 2013-14 to 2017-18.

Table 2:  HCPCP Contribution Funding 2013-14 to 2017-18

YEAR	Planned	Actual	Variance	% Variance
2013-14	34,504,000	19,712,535	14,791,465	42.9
2014-15	26,359,000	20,382,789	5,976,211	22.7
2015-16	25,709,000	17,839,928	7,869,072	30.6
2016-17	25,509,000	9,284,670	16,224,330	63.6
2017-18	26,874,000	8,737,838	18,136,162	67.5
TOTAL	138,955,000	75,957,760	62,997,240	45.3

There were a number of reasons for the differences between planned and actual expenditures over the period of the evaluation, including the following:

• extended timelines required to establish contribution agreements with PTs;
• the underspending of funds by some recipients under the IEHPI, as well as the winding down of this initiative starting in 2015-16; and
• the realignment of the program to identify and address new priorities, which started in 2013 and was ongoing at the time of the evaluation.

3.0 Evaluation Description

The scope of this evaluation covered the period from 2013-14 to 2017-18, and included all activities under the HCPCP. Given that program redesign activities were undertaken concurrently with the evaluation, the evaluation was calibrated and focused on meeting FAA requirements, including the examination of the core issues of relevance and effectiveness, while also exploring factors that contributed to or impeded program success. The questions below guided the evaluation.

Relevance

• 1. How has the program evolved over time to meet demonstrated needs/changing priorities?

Effectiveness

• 2. What progress has the HCPCP made in achieving its expected outcomes?
• 3. What elements or factors have contributed to, or impeded, project and/or program successes?

Data for the evaluation was collected using various methods, including the following:

• A focused literature review;
• A document review that included an in-depth review of 10 project files. A cross-section of projects were selected by taking into consideration the year, funding component, and value of the contribution agreements. There was also a focus on projects that were completed or that had been underway for several years;
• Key informant interviews that included nine interviewees (five internal and four external); and
• Case studies of two projects:
  • The Community-Integrated Palliative Care Initiative: The Way Forward, which included an in-depth review of project files and two key informant interviews (one external and one internal).
  • The National Assessment Collaboration Practice Ready Assessment (PRA) for International Medical Graduates in Canada, which included an in-depth review of project files and three key informant interviews (two external and one internal).

The key limitations that this evaluation faced were a lack of collated performance information, minimal data on long-term impacts, and a limited methodology. In addition, the majority of outcome data was self-reported by the projects or consultants contracted by the projects. To respond to these limitations, the evaluation used multiple lines of evidence to gather a variety of perspectives from within Health Canada and from funding recipients. Documentary evidence at both the program and project levels was included. Due to the number of project files reviewed (12 in total), results are not representative of all HCPCP projects. As a result, illustrative examples are provided rather than reporting proportions of projects.

4.0 Findings

4.1 Relevance – How has the program evolved over time to meet demonstrated needs/changing priorities?

The evaluation found that HCPCP priorities have evolved over time and that there are continuing needs within the health care system related to current priority areas, including palliative care, home care, mental health, and health care system innovation.   

Evolution of Priorities

During the timeframe covered by the evaluation, the HCPCP funded projects through three program components: the Health Human Resource Strategy (HHRS), the IEHPI, and the Health Care System Innovation Fund (HCSIF). These components were based on priority areas stemming from federal commitments made in the 2003 and 2004 Health Accords.

The previous evaluation of the HCPCP noted that, with the expiration of the Health Accords in 2014, the program would be at a “watershed moment without a similar statement of federal priorities to guide future health care contribution funding”^{Footnote 1}. In anticipation of the Health Accords ending, the HCPCP initiated an exercise aimed at refocusing policy and funding priorities, and redesigning the comprehensive operational framework and management approach for the program^{Footnote 2}. Program representatives developed a detailed Project Charter to guide the redesign process. This process took into account a wide variety of issues and information, including:

• emerging Government of Canada health care policy priorities;
• provincial and territorial government priorities;
• other national stakeholder perspectives (including the Health Council of Canada, the Canadian Medical Association and Canadian Nursing Association, the College of Family Physicians of Canada, and the Canadian Federation of Nursing); and
• expert reports^{Footnote 3}.

In 2015, the following policy areas were confirmed as HCPCP priorities: High Users of Health Care, System Adaptation to Needs of Aging Population, Optimizing the Health Work Force, and Supporting Changing Role of Patients. The HCPCP continued to fund projects under the three existing program components, although the new priorities were used for project selection^{Footnote 4}.

The HCPCP priorities were updated again in 2017 to reflect key areas identified in the Common Statement of Principles on Shared Health Priorities. The program also considered other government programs in order to help prevent overlap and ensure complementarity^{Footnote 5}. According to the most recent Health Canada Departmental Plan, current HCPCP priorities include health care system innovation, as well as palliative and end-of-life care. In addition, other program documentation and internal key informants cited home care and mental health as priorities. To support these priorities, the 2018 Call for Proposals objectives were to: 

• optimize the health, well-being, and functional independence of home care clients;
• promote the integration of community mental health services with primary care; and
• improve access to palliative and end-of-life care.

The HCSIF component is now the primary funding stream for HCPCP projects, due to the sunsetting of the IEHPI component. According to internal key informants, a decision was made to exclude PTs from the current Call for Proposals due, in part, to the fact that PTs will be receiving $11 billion over ten years under the Common Statement of Principles on Shared Health Priorities for commitments related to home care and mental health. Several key informants also noted challenges in managing projects with Provincial/Territorial (PT) recipients given delays associated with approval processes within multiple bureaucracies that led to lapsing of funds. It is worth noting that, while PTs were not eligible for the 2018 Call for Proposals, they remain eligible funding recipients under the program's Terms and Conditions and it is possible that they will be recipients again in the future.

Evidence of Need

Documents and literature reviewed as part of this evaluation found that there are continuing needs within the health care system related to current HCPCP priority areas, including palliative care, home care, mental health, and health care system innovation.

In terms of home care and palliative care, there is evidence that there are unmet needs for Canadians. For example, a recent study found that during a 12-month period, 2.2 million Canadians received help or care at home due to a long-term health condition, disability, or a problem related to aging, with seniors being the most likely to receive care at home. Approximately 15% of those who received home care did not receive all the help needed, and nearly half a million Canadians who needed help or care for a chronic health condition in the 12 previous months did not receive it^{Footnote 6}. Another study found that almost half of all adult cancer deaths in Canada (45%) happen in acute care hospitals, though many people would have benefited from palliative care at home or in hospices^{Footnote 7}.

Furthermore, given that the population in Canada is aging (with people aged 85 years and over making up the fastest growing age group in Canada, and for the first time, there are more people aged 65 and older than there are children aged 0-14 years)^{Footnote 8}, these unmet needs are likely to increase.

With regard to mental health, a significant number of Canadians are affected by poor mental health and mental illness. According to the Canadian Mental Health Association, in any given year, one in five people in Canada will personally experience a mental health problem or illness, and mental illness indirectly affects all Canadians at some time through a family member, friend, or colleague^{Footnote 9}. Another study concluded that gaps in mental health services continue to exist, and that policy and practice solutions are needed to address unmet needs^{Footnote 10}.

A recent advisory panel highlighted the need for health system innovation by indicating that federal action was important to promote innovation and enhance both the quality and sustainability of Canadian health care^{Footnote 11}. The need for innovation has also been emphasized through recent Government of Canada budgets^{Footnote 12,Footnote 13,Footnote 14} and in the Common Statement of Principles on Shared Health Priorities^{Footnote 15}.

Although the Canadian Medication Incident Reporting and Prevention System (CMIRPS) Program does not fall within the identified priority areas, it is an HCPCP project that receives ongoing funding. As a result, the evaluation examined evidence related to this project, and found that there is a continuing need. For example, the World Health Organization indicated that medication errors are a global issue^{Footnote 16}, and a recent study found that harmful events associated with medications (including getting the wrong medication) occurred frequently during Canadian hospital stays^{Footnote 17}. The University Health Network indicated that preventable medical errors killed more than 30,000 Canadians in 2014, which means that preventable deaths occurring in acute care kill more Canadians than stroke, diabetes, Alzheimer's, and kidney disease combined^{Footnote 18}.

4.2 Effectiveness – What progress has the HCPCP made in achieving its expected outcomes?

4.2.1 Stakeholders are aware of knowledge products/tools

Projects were generally effective in producing and disseminating knowledge products. Direct measurement of awareness was limited, although there was other evidence related to awareness, including the involvement of some target audiences in knowledge product development, and subsequent use of some knowledge products.

Overall, projects examined as part of the evaluation were successful in producing planned knowledge products. However, some projects were behind schedule and, at the time of their final progress reports, a small number of projects had not produced all deliverables as planned. A variety of knowledge products developed, including the following:

• frameworks, standards and guidelines on health care system approaches;
• recommendations and guides for specific health care issues;
• accreditation standards and processes;
• environmental scans and discussion documents;
• data on medical incidents, safety bulletins, and recommendations regarding medication safety;
• training enhancements, learning modules, and learning supports;
• assessment tools; and
• formalized knowledge networks.

Projects disseminated knowledge products using a wide variety of mechanisms including webinars, workshops, meetings, newsletters, networks, engagement sessions, virtual forums, Twitter, Facebook, email lists, and conference presentations, booths, and posters. Some projects measured the type and number of outreach and dissemination activities, including access, and provided figures in their progress reports related to these activities. There was little overall or rolled-up data, although a few projects did measure reach. For example, the Health Force Integration Research and Education for Internationally Educated Health Professionals (IEHPs) project, which aimed to accelerate and expand the assessment and integration of IEHPs, reported that it had reached “a large audience of IEHPs (1,408 client outreach encounters) and organizations (1,850 organization outreach encounters)”^{Footnote 19}. No trend analysis was available for any of the projects.   

Measurement of awareness of knowledge products was limited. A few projects did not include ‘awareness' as part of their outcome statements or objectives, so measurement of awareness would not have been expected for those projects. Some project progress reports indicated that they had intended to raise target audience awareness, and that awareness had been measured; however, supporting information did not necessarily provide a measure of awareness. For example, one project reported the result of “stakeholders continue to be engaged”, while another reported “overall satisfaction with both content and delivery modules was high”, both without further explanation or information on awareness.

Available data on awareness included the following:

• The Way Forward project, which was intended to help move the health system towards a community-integrated approach to palliative and end-of-life care, measured target audience awareness, as well as disseminating discussion documents and the National Framework: A Roadmap for an Integrated Palliative Approach to Care. They found that 96% of respondents were aware of the discussion documents. For the Framework, 88% of respondents had read it. In addition, participants at a 2013 workshop were asked if their understanding of an integrated palliative approach to care increased after the workshop and 60% gave a rating of 4 or 5, while 34% gave a rating of 3 (where 1 is not at all and 5 is significantly)^{Footnote 20}.
• The Future of Medical Education in Canada Postgraduate Project aimed to develop smoother and more effective transitions from medical school to residency, and from residency into clinical practice. The project-level evaluation found that, on average, 69% of respondents were aware of projects related to transitions from residency to clinical practice, with awareness levels ranging from 31% for one initiative to 100% for another^{Footnote 21}.

The collaborative nature of projects meant that for some, key players involved in the project were also among the target audience for the project. As such, they were aware of the knowledge products. For example:

• The Way Forward was led by the Canadian Hospice Palliative Care Association and the Quality End-of-Life Care Coalition of Canada (which is made up of 34 member organizations), and included other stakeholders including federal, provincial and territorial governments, policymakers, health care professionals, home care associations, primary/acute care and long-term care associations, and organizations representing Canada's First Nations.
• The National Assessment Collaboration Practice Ready Assessment (PRA) for International Medical Graduates project aimed to develop a pan-Canadian process to assess International Medical Graduates' readiness for practice under provisional licensure. A steering committee that included medical regulatory authorities, certification colleges, the Federation of Medical Regulatory Authorities of Canada, and the Medical Council of Canada led the project. The project also had working groups for areas of medical speciality (e.g., family medicine, psychiatry, internal medicine), and included stakeholders representing PRA programs, PT Ministries of Health, the Association of Faculties of Medicine in Canada, and the Canadian Medical Association. The final project report indicated that “programs were engaged in all prioritization, design, and research and development activities. By engaging programs in the creation of standards, knowledge transfer was increased regarding what the standards meant.”^{Footnote 22}

Also of note is that awareness is a prerequisite to uptake/use, and many projects examined as part of this evaluation were able to demonstrate use of some of the knowledge products they had produced (see Section 4.2.2, below).   

4.2.2 Stakeholders are using knowledge products/tools

The evaluation found evidence that some target audiences were using knowledge products stemming from projects. Use varied, ranging from the development of frameworks and guidance documents based on knowledge products, to participation in training programs.  

Given that not all knowledge products had been produced or finalized at the time of project reporting, evidence of use is limited to some knowledge products only. However, many progress reports, project evaluations, and final project reports included information on uptake and use of knowledge products by the target audiences. Examples of use ranged from using frameworks and resources, to participating in family medicine training programs.

Several project reports included evidence that was vague or non-specific and appeared to be based on informal input from stakeholders. For example, evidence of use in one project report was, “report used by various stakeholders to inform policy discussions”, while another project report indicated, “anecdotal evidence and monitoring program changes”.

However, more concrete evidence of use was available for many projects. The case study of The Way Forward found that, in 2017, members of the Quality End-of-Life Care Coalition and PT representatives were surveyed to assess uptake and use of The Way Forward's knowledge products, including the National Framework: A Roadmap for an Integrated Palliative Approach to Care. Eighty-nine percent of Quality End-of-Life Care Coalition respondents and 83% of PT respondents indicated that they had used the National Framework for guidance^{Footnote 23}. Specific examples of the use of knowledge products from The Way Forward included:

• Nova Scotia used an early iteration of the National Framework to assist in the development of a draft provincial framework.
• The Ontario Palliative Care Steering Committee used the discussion documents and Framework as input into their palliative care work.
• Ontario used the framework and quality indicators discussion document to inform the provincial “Declaration of Partnership” table to develop performance indicators for Ontario.
• Alberta used the framework and survey results to inform their provincial steering committee activities.      

For the Future of Medical Education in Canada Postgraduate Project, the project evaluation reported that three colleges (College of Family Physicians of Ontario, Collège des médicins du Québec and Royal College of Physicians and Surgeons of Canada) had begun realignment of their Postgraduate Medical Education accreditation standards at the time of the project evaluation^{Footnote 24}.

Evidence was also available for the Choosing Wisely Canada project, a campaign aimed at helping clinicians and patients engage in conversations about unnecessary tests and treatments, and make smart and effective care choices. Two practical toolkits from Choosing Wisely Canada were used by hospitals and one was used in a randomized trial:  

• Give the Test a Rest, a toolkit for reducing unnecessary emergency department lab testing, was used by North York General Hospital.
• Less is More with T3 and T4, a toolkit on reducing free thyroid hormone testing, was used by Women's College Hospital.
• Drowsy Without Feeling Lousy, a toolkit for de-prescribing of benzodiazepines and other sedative hypnotics in primary care, was used in a cluster randomized trial in Montreal that engaged patients at various pharmacies.

The National Assessment Collaboration PRA for International Medical Graduates project reported that, “Coupled with their [stakeholder] involvement in common material development, the adoption of these materials into current practices increased. In addition to the programs' in-kind support of the initiative with their commitment of professional time, they shared materials and tools that were adopted and/or adapted for use across programs and now reside in the resource library.”^{Footnote 25}

The evaluation examined two Family Medicine Residencies Initiatives (FMRI) projects that aimed to enhance training for family medicine residents in rural and underserved areas, one in Newfoundland and one in Nunavut. Use, in the context of these projects, was participation in the training programs. Overall, the two projects supported the training of 44 residents. One project was slightly below its target number of residents trained, as no residents were trained in the first year, reportedly due to the timing of receipt of initial funding from Health Canada. The second project was above their target number of residents trained; however, this was due to more residents being sent to the training site for shorter periods of time, rather than having a smaller number of residents at the sites for longer periods, as had been planned.  

A few internal and external interview respondents indicated that Health Canada is a user of the knowledge products created by some projects. For example:

• Data from the Canadian Post-Medical Doctor Education Registry (CAPER), which provides longitudinal data on the numbers and types of physicians moving through Canada's postgraduate medical training system, has been used by Health Canada for a health workforce program and by the Federal/Provincial/Territorial Committee on Health Workforce for resource planning.
• Health Canada has used knowledge products from CMIRPS and worked with the Institute for Safe Medication Practices Canada (ISMP) to develop guidance on a naming policy for drugs, as well as for labelling and packaging of prescription and non-prescription drugs, and natural health products.
• Health Canada is currently using knowledge products from The Way Forward to inform legislative work and the federal strategy on palliative care.

4.2.3 Improvements in the health care system

There is evidence that some projects led to improvements in the health care system, ranging from the adoption of professional standards, practices, and policies, to more physicians with rural/remote experience. However, evidence is limited on the impacts of these changes. It is notable that several projects that were able to demonstrate progress towards longer-term outcomes were continuations of previously funded projects, sometimes with multiple earlier phases.

Of the projects reviewed, the average contribution agreement length was approximately three years. As a result, some final project reports and evaluation reports had limited evidence on the achievement of longer-term outcomes. Those that ended several years ago might only now be demonstrating impacts at the system level; however, although Health Canada had permission to revisit the achievement of longer-term outcomes two to three years after project completion (as per a question in the progress reporting template that was required for most projects examined by the evaluation), there was no evidence that this has been done. Several internal key informants identified this as an area the program should focus on for improvement. However, the progress reporting template (developed by the Office of Grants and Contributions) has been updated and no longer includes such a question. This could present a challenge if Health Canada wanted to follow up on the longer-term results of projects in the future. HCPCP representatives indicated that they will look into this issue further to see if there are ways to strengthen recipient reporting on achievement of outcomes and impacts over the long-term, beyond the funding period of the project. A few internal key informants also noted that projects funded over a shorter period do not necessarily go beyond the output or immediate outcome level. Furthermore, they noted that, given the overall amount of funding for projects relative to overall health care system funding, it is very challenging to measure the specific impact of projects.

There was, however, evidence of health care system improvements for some projects. Of note is that a number of projects examined as part of the evaluation were continuations of earlier projects funded by the HCPCP; for example, CMIRPS, the National Assessment Collaboration PRA for International Medical Graduates, Choosing Wisely Canada, and Pour une meilleure intégration au Québec des médecins diplômés hors du Canada et des États-Unis. For these projects, evidence of longer-term results may have been, at least in part, the result of work that was started many years earlier. For example, the case study of the National Assessment Collaboration PRA, found that the project was able to demonstrate changes to the health care system in terms of additional health care practitioners with provisional licensure. The project reported that in 2016 and 2017, 428 individuals succeeded in obtaining provisional licensure through a harmonized PRA, at both a lower cost and in a shorter timeframe than traditional residency. At the same time, one of the other key objectives of the PRA was that internationally educated physicians would be “safe and competent” to practice in Canada. However, data was unavailable as to whether patient safety improved as a result.

For the CMIRPS project, a recent case study conducted by an independent contractor noted that the best example of uptake was when ISMP Canada recommendations and medication safety products and tools have influenced professional standards. Accreditation standards require compliance, so ISMP Canada's influence has system-wide effects when its medication safety recommendations and strategies are incorporated into these standards. In 2016, ISMP confirmed that 72 recommendations from incident analyses continued to be included in Accreditation Standards and Required Organizational Practices. For example, Accreditation Canada's Required Organizational Practices handbook for 2014 includes two topics that relate to ISMP Canada's work on opioids. Additional examples of health care system improvements stemming from CMIRPS include:

• New Ontario College of Pharmacists ‘Community Pharmacy Assessment Criteria', which helps ensure pharmacies have operational processes in place to ensure safety, and includes two criteria that direct the user to CMIRPS safety bulletins (2018).
• Good Label and Package Practices Guides for both Prescription Drugs, and Non-Prescription Drugs and Natural Health Products, which provide industry with direction on how to comply with the new requirements set out as part of the Plain Language Labelling Regulations, and have contributed to changes in packaging and labelling (2017).
• Acceptable Abbreviations for Prescription Health Product Labels in Canada, which creates consistency in the use of select abbreviations on prescription health product labels in Canada (2017).
• Physician order forms integrating recommendations from a Safety Bulletin from CMIRPS that indicates to prescribers the abbreviations not to use and abbreviations that can be used when writing orders (2017).
• Canadian Incident Analysis Framework (CIAF), co-authored by ISMP in 2012, is a resource to help support individual and organizational learning, as well as quality improvement, in response to patient safety incidents. Although the development of the CIAF was outside the scope of this evaluation, it was considered given that implementation of the framework is ongoing. An evaluation undertaken by the Canadian Patient Safety Institute in 2017 highlighted the effects of the CIAF^{Footnote 26}. It found that of the organizations that had used the CIAF, a large proportion had made changes to practices, policies or procedures, and most reported that this had impacted patient safety substantially or moderately. Users of the CIAF reported changes in a variety of areas such as investigating patient incidents, developing actions in response to incidents, reporting patient safety events, communicating with patients about safety events, and involving patients in reporting and investigations.

Choosing Wisely Canada is another project that demonstrated longer-term results. As of October 2017, the project reported 61 instances of reductions in unnecessary tests and treatments, stemming from resources produced by the project^{Footnote 27}. Examples include: 

• Using the approach in the ‘Give the Test a Rest' toolkit, North York General Hospital in Ontario achieved a 23% reduction in lab testing over two years.
• Using the approach in the ‘Less is More with T3 and T4' toolkit, Women's College Hospital reduced free thyroid hormone testing by 54%.
• The Cumming School of Medicine reported that for one intervention, “a 4% reduction in test ordering was observed post-intervention equating to $4,600 in annual savings with a projected provincial savings of $40,000 for all emergency departments in Alberta.”
• McGill University Health Centre reported that for one intervention, “the average cost per admitted patient decreased from $117 to $66, with an estimated savings of $50,657 over 985 admissions. After adjusting for fiscal period and the presence of our intervention, there was a significant reduction in the per-patient number of total tests, complete blood counts, and electrolyte panels performed. Length of stay decreased during the intervention to five days from seven days.”

The Way Forward project conducted a follow-up survey in 2017 and found that, in nine of the 12 jurisdictions who responded to the survey, there is now a provincial policy framework for hospice palliative care, supported by strategies for training and skills development of health care providers (in BC, AB, SK, MB, ON, NS, PEI, NWT, and YK)^{Footnote 28}.

The two FMRI projects reviewed as part of this evaluation led to additional physicians with rural/remote experience. Memorial University's Family Medicine Residency Program produced 16 family medicine graduates with rural/remote training over the project's duration. The project-level evaluation found that 54% of graduates went into rural practice, with five graduates returning to practice in the community in which they were trained, or in nearby communities, and with two graduates signing long-term contracts. However, it also noted that as the two pilot sites constituted a small proportion of the province's population, there was no observable impact on physician recruitment to rural/remote areas of Newfoundland^{Footnote 29}. The Nunavut Family Medicine Residencies Project produced 28 additional residents with rural/remote experience. However, eight trainees were meant to have a “return of service” commitment of two years^{Footnote 30}, and only two graduates appear to have signed contracts (one for one year and another for three years)^{Footnote 31}.

The FMRI was the only program area to have been reviewed internally by the program. This 2015 review found that the FMRI was successful in increasing residency spots and/or  meeting the needs of the jurisdictions, estimating that 144 of the targeted 145 residency spots had been created, thus improving access to health care, especially in underserved areas. However, the review noted that, while the results were positive and were generally in line with the contribution agreements, it was difficult to ascertain the impact of federal funding on achieving these results, because some of the FMRI projects were part of larger initiatives by the PTs to address rural and remote physician recruitment and retention. In some cases, the PTs also funded residents as part of the project and they reported on all the activities, rather than just those that were federally funded.  

4.3 Effectiveness:  What elements/factors have contributed to or impeded project success?

The evaluation identified several areas that contributed to or impeded the success of projects, including collaboration, project leadership, and the length of the contribution agreements. In many cases, the same factor was seen to be either an impediment or contributor, depending on the situation or perspective.

A number of themes emerged for factors that contributed to or impeded project success. The evaluation focused on factors that all projects could learn from, rather than on those that would only be relevant to extremely similar projects. Key themes are described below.

Collaboration
The importance of collaboration with relevant partner organizations throughout project planning and implementation was the most often cited issue in final project reports, evaluations, project progress reports and by external key informants, as well as by a majority of internal key informants. In some projects, collaborative activities were said to have slowed or impeded progress, but ultimately collaboration was seen to be the most crucial element for project success. Issues related to collaboration included the importance of engaging a wide variety of perspectives and disciplines, building strong relationships with relevant stakeholders, including through face-to-face interactions, and transparent communication.

Project leadership
The theme of project leadership was the factor cited by the largest number of internal key informants and a large number of external key informants as either contributing to or impeding success, depending on the effectiveness of the leadership. Issues related to leadership included the importance of the organization leading the project having:  a dedicated project leader or team, governance structures in place, and adequate capacity, in terms of time and knowledge or experience.  

Length and flexibility of contribution agreements
There was general agreement among internal and external key informants that longer-term contribution agreements better enabled project success, while shorter-term contribution agreements were an impediment to success. Virtually all internal and external key informants who commented on the length of contribution agreements indicated that a five-year agreement would be ideal. In addition, a few key informants noted that a multi-phased approach to contribution agreements for projects that demonstrate success would lead to better results. The potential for greater innovation with longer-term contribution agreements was also noted in one project report, and by a few internal key informants. In addition to contribution agreement length, several external key informants and a few project progress reports indicated that increased flexibility in contribution agreements would be preferable, for example, to allow recipients more discretion in the timing of spending.    

Evidence-based planning
A number of project reports and many external key informants discussed issues related to the availability of evidence for planning. They cited lack of relevant evidence as an impediment for some projects. However, several projects overcame this obstacle by undertaking research and environmental scans to address gaps in knowledge and to improve planning for the projects as they moved forward. A clear project plan with defined objectives/outcomes was seen as important for project success, in particular by internal key informants. Finally, a few external key informants discussed the value of using lessons learned or evaluation findings to inform future planning.

HCPCP input 
According to a few external and internal key informants, engagement with HCPCP staff to align projects with program or broader Health Canada priorities/needs was a success factor for some projects, as this helped ensure they were relevant and useful. In addition, several external key informants, many project progress reports, and one final project report noted that HCPCP involvement and support were helpful for projects. Examples include encouraging buy-in from stakeholders and providing project guidance and advice. However, a number of project progress reports indicated that Health Canada could do more information sharing, such as with other HCPCP project activities and lessons learned.

4.4 Effectiveness:  What elements/factors have contributed to or impeded program success?

Re-centralizing the HCPCP and support from the Office of Grants and Contributions were identified as the key factors contributing to program success. The program's role in knowledge translation and strategic direction, including use of project performance and progress reports, as well as program support for innovative projects, were identified as areas for improvement.

A few key themes emerged as factors that have affected success at the program level. These are described below.     

Program Structure and Support

A few internal key informants talked about the structure of the HCPCP, indicating that re-centralization of the HCPCP in 2016 was a positive move for the program. These key informants saw re-centralization as a key factor in improving the consistency of program delivery and for consolidating both project and program management. As part of re-centralization, the HCPCP redesigned its strategic plan and direction and made changes to business processes to help ensure management integrity and consistency. This included updating governance processes, identifying new policy priorities that aligned with ministerial priorities, and approving new Terms and Conditions. The program designed a Call for Proposals to align with new priorities and expand the reach of the program to recipients across the country. To deliver on the Call for Proposals, new business processes and tools were developed and implemented. Program representatives indicated that the program placed an emphasis on performance measurement and provided support to recipients to help ensure accountabilities were met and deliverables achieved. In addition, they indicated that there will be a lessons learned exercise following the completion of the Call for Proposals process to provide an opportunity to review and improve program management practices.

The Office of Grants and Contributions is a Health Canada group tasked with improving performance measurement and knowledge translation within a number of grants and contributions programs. Many internal key informants cited the Office as an important support for program success, because they are able to provide assistance and advice to the program and funding recipients regarding issues such as performance measurement and evaluation. Many external key informants and project progress reports also mentioned the value of the workshops that the Office of Grants and Contributions delivered (discussed further below).

Program Role in Knowledge Translation and Strategic Direction

The program's role in knowledge translation and strategic direction emerged as an area for improvement. Most internal key informants highlighted the underuse of project performance information, with some indicating that its use should be increased in order to improve knowledge translation and to inform policy development and the identification of strategic priorities and investments.

Progress reporting is mandatory for all projects funded by the HCPCP, although specific requirements are based on risk, which, according to program representatives, is assessed annually, as per the corporate risk tolerance strategy. For example, high-risk projects may require quarterly reporting, while lower-risk projects may only require annual reporting, and final evaluation reports are only required for some projects. However, the evaluation found issues with the information contained in some of the progress reports, including incomplete reporting and a lack of verifiable information related to outcomes. Furthermore, the program did not appear to be systematically tracking the reporting requirements of the various projects, for example, into a shared database.

In addition, there was only very limited evidence of the program collating findings from project evaluations and project progress reports. This could be used to report on outcomes in strategic planning, or to identify themes for lessons learned and best practices, both at the project level and in regard to Health Canada support. The FMRI, which included nine projects over the period 2009-10 to 2016-17, was the only instance of the program collating project reporting. For this set of projects, the program used project progress reports, project evaluations, and other relevant project documents in order to assess the following:

• activities conducted in relation to contribution agreements;
• results achievement;
• plans for sustainability post-Health Canada funding;
• best practices; and
• lessons learned.

Draft versions of the report and a related two-page summary document were provided for this evaluation, although it was not clear whether they were finalized or how the program used them.  

Several external key informants commented on the project reporting templates, indicating that they were generally comfortable with them, although they were time-consuming. Some project progress reports also provided suggestions for changes to the template, such as streamlining the questions and having some questions only included in annual or final progress reporting. One interviewee questioned what Health Canada had done with the information from the reports, particularly given the time invested by the recipients in completing it, and one final project report indicated that the templates were difficult to work with and that, although Health Canada had asked for feedback, it did not seem to use it.

One area where the program did appear to have taken on a knowledge translation role was in offering sessions or workshops to funding recipients. According to program representatives, the sessions were primarily offered to Health Canada staff and funding recipients working under the IEHPI, although based on external key informant feedback, the audience also included project recipients funded under HHRS and HCSIF. Program representatives indicated that the Office of Grants and Contributions organized most of the sessions, following requests from the program. Recent examples include a session entitled “Moving Knowledge into Practice, Programs and Policy in the Context of Grants and Contributions” on October 5, 2017 and a Performance Measurement Learning Event on February 13, 2018. Many external key informants and project progress reports mentioned these sessions as being positive and supportive. Recipients saw the sessions, along with annual meetings, as good opportunities for knowledge translation and learning about specific subjects, such as performance measurement and evaluation, and gender-based analysis. Recipients also saw these as opportunities for communicating with other funding recipients.

Beyond the provision of such sessions, there was little evidence of the program taking on a proactive role in knowledge translation. Several external key informants and project progress reports, as well as one final project report, noted that the program could improve its strategic direction and contribution to knowledge translation. Suggestions for improvement included funding projects that were explicitly designed to be shared and replicated across the country, and being more strategic and timely with sharing information on other projects to help prevent duplication and encourage learning. Several internal key informants shared these views, indicating that the program could be less operational and more strategic if it improved knowledge translation and exchange, provided more leadership, and shared information such as project results. A few internal key informants also suggested that the program should develop a knowledge translation strategy.

This focus on knowledge translation and use of performance information is consistent with the theory of change upon which the program design is based. A key feature of this theory is feedback and learning loops between inputs, activities, outputs, and outcomes. The loops indicate opportunities when performance information can be used to improve the likelihood that the outcomes will be achieved^{Footnote 32}.

Encouraging Innovation

A few internal key informants suggested that the program could do more to encourage innovation, citing what they perceive to be a risk-averse approach to project selection as something that stifles innovation. They suggested that the program should help test and build the evidence base by funding more innovative projects.

5.0 Conclusions and Areas for Consideration

5.1 Conclusions

HCPCP priorities have evolved over time and there are continuing needs within the health care system related to current priority areas, including palliative care, home care, mental health, and health care system innovation.   

Projects were generally effective in producing and disseminating knowledge products, and there was evidence of progress towards all three program outcome areas. Although direct measurement of awareness was limited, there was other evidence related to awareness, including the involvement of some target audiences in knowledge product development and subsequent use of some knowledge products. Use varied, ranging from the development of frameworks and guidance documents based on knowledge products, to participation in training programs.  

Some projects led to improvements in the health care system, ranging from the adoption of professional standards, practices and policies, to more physicians with rural/remote experience. However, evidence was limited on the impacts of these changes. It is notable that several projects that were able to demonstrate progress towards longer-term outcomes were continuations of previously funded projects, sometimes with multiple earlier phases.

5.2 Areas for Consideration

Given that the HCPCP continues to implement program redesign, this evaluation presents the following areas for consideration that the program can continue to build upon or take into account as it moves forward, rather than providing recommendations:

• At the project level, collaboration with relevant partner organizations and strong project leadership were seen to be the most crucial elements for ultimate project success, along with longer-term contribution agreements, evidence-based planning, and support from Health Canada (i.e., encouraging buy-in from stakeholders and providing project guidance and advice).
• At the program level, Health Canada's role in knowledge translation and strategic direction, including use of project performance and progress reports and following up on completed projects to better determine the achievement of longer-term outcomes, as well as program support for innovative projects, were identified as areas for improvement.