Evaluation of the Health Care Policy Contribution Program

(PDF Version - 284 K)

Prepared by
Evaluation Directorate
Health Canada and the Public Health Agency of Canada

September 2013

List of Abbreviations

ACHRHR

Advisory Committee on Health Delivery and Human Resources

ADM

Assistant Deputy Minister

CBOC

Conference Board of Canada

CIHI

Canadian Institute for Health Information

CMIRPS

Canadian Medication Incident Reporting and Prevention System

CPSI

Canadian Patient Safety Institute

FLMM

Forum of Labour Market Ministers

FMRI

Family Medical Residencies Initiative

FPT

Federal/Provincial/Territorial

FQR

Foreign Qualifications Recognition

FY

Fiscal Year

G&C

Grant and Contribution

GCMIS

Grants and Contributions Information Management System

HCPCP

Health Care Policy Contribution Program

HCPPD

Health Care Programs and Policy Directorate

HCSIF

Health Care System Innovation Fund

HHRS

Health Human Resource Strategy

IEHPI

Internationally Educated Health Professionals Initiative

ISMP Canada

Institute for Safe Medication Practices Canada

NGO

Non-governmental organizations

NWTI

National Wait Times Initiative

PMAU

Program Management and Accountability Unit

PMC

Program Management Committee

PMIP

Program Management Implementation Plan

PWTGPPF

Patient Wait Times Guarantee Pilot Project Fund

RRET

Recipient Reporting and Evaluation Template

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1.0 Introduction
  • 1.1 Background and context
  • 1.2 Program components
  • 1.3 Program objectives, activities and outcomes
  • 1.4 Governance
  • 1.5 Resources
• 2.0 Evaluation Methodology
  • 2.1 Purpose and scope of the evaluation
  • 2.2 Evaluation design and data collection methods
  • 2.3 Limitations and mitigation strategies
• 3.0 Findings - Relevance
  • 3.1 Is the HCPCP aligned with federal roles and responsibilities?
  • 3.2 Is the HCPCP aligned with the Government of Canada and departmental priorities?
  • 3.3 Is there a continued need for the HCPCP?
• 4.0 Findings - Performance
  • 4.1 Program Implementation
  • 4.2 Effectiveness
  • 4.3 Efficiency and Economy
• 5.0 Conclusions and recommendations
• Annex 1 Program components
• Annex 2 Program Logic Model
• Annex 3 References

Executive Summary

The evaluation covered the Health Care Policy Contribution Program (HCPCP or the Program) for the period from 2008-09 to 2012-13. The evaluation was undertaken in fulfillment of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009).

Evaluation Purpose, Scope and Design

The evaluation assessed the relevance and performance (effectiveness, efficiency, and economy) of the Program. The evaluation took place between April 2012 and August 2013 and drew on several lines of evidence, including a literature and program-level document review, a project-level document review, a survey with contribution agreement recipients (primary contacts), and Health Canada key informant interviews. This is the second evaluation of the HCPCP, having been previously evaluated in 2007.

Program Description

The HCPCP is a national program that supports the development of policies and strategies to address identified health care system priorities. The HCPCP uses contribution funding to increase knowledge and application of evidence and best practices, which were ultimately intended to improve health care system planning and performance.^{Footnote 1} It also fosters collaboration and coordination of responses to health care system priorities among federal, provincial, and territorial governments; other health care policy-makers; service providers; users; researchers; and other stakeholders. It has funded 163 projects across five component areas: the Health Human Resource Strategy (HHRS); the Internationally Educated Health Professionals Initiative (IEHPI); the Health Care System Innovation Fund (HCSIF); the National Wait Times Initiative (NWTI); and the Patient Wait Times Guarantee Pilot Project Fund (PWTGPPF). The total budget allocated to the HCPCP between 2008-09 and 2012-13 was $250M and actual expenditures totalled $201M.^{Footnote 2} Funding for HCPCP is divided into two streams: Vote 10 funds project recipients and Vote 1 funds program O & M in Health Canada.

Evaluation Conclusions and Recommendations

Conclusions - Relevance

The HCPCP aligns with the federal role and responsibilities for health care, as well as with federal and departmental priorities. The Program serves the federal role in health care by providing leadership, promoting health care system innovation, and supporting federal/ provincial/territorial coordination and collaboration to improve the health care system. In addition, the HCPCP aligns with Health Canada's first departmental strategic outcome of "a health system responsive to the needs of Canadians" (Health Canada, 2011a). The Program's alignment with federal priorities initially came from the federal commitments made in the Health Accords. Although the federal government has not indicated an intent to negotiate a new Health Accord, it has continued to emphasize its broad commitment to the improvement of the health care system in the most recent Speeches from the Throne (2013) and Budget Speeches, as well as in other public statements.

With the Health Accord set to expire in 2014, the HCPCP is at a watershed moment where it will need to define the federal role and funding priorities without a similar statement of federal priorities.

Conclusions - Program implementation

HCPCP made progress over the period assessed in the evaluation in implementing its planned activities. In particular, the Program instituted a new governance structure, engaged in strategic planning and coordination and moved to improve performance measurement data and monitoring. The three governance committees were found to have addressed many of their responsibilities. However, areas for improvement were identified, such as overlapping responsibilities and lack of engagement in some areas of responsibilities. While the Program made efforts to conduct strategic planning since 2007, concerns were still voiced that much of its focus had been diverted to reallocating resources to reduce lapsing, rather than providing strategic direction for the Program. The Program has made progress in performance reporting through the development of the Recipient Reporting and Evaluation Template (RRET). However, at the time of this evaluation, the Program did not yet have consistent performance information from all of its funded projects

Findings demonstrated that the Program was largely successful in implementing planned calls for proposals. The method (targeted and unsolicited) of conducting calls for proposals was an area of potential improvement. To ensure that the proposal process is transparent, the HCPCP should keep systematically-collected information on calls for proposal and unsolicited proposals.

The Program, through Vote 1 funding ($51M over 5 years) was supposed to contribute to policy/research development and implementation, as well as facilitation of collaboration, through funding projects that were actively engaged in these activities. The Program was not able to demonstrate its efforts in these areas because it does not have a mechanism for performance reporting at this level. The Program mainly focused on its role (selecting and managing projects) as a funder and less on how it contributed (facilitation, knowledge translation/uptake) to the overall HCPCP.

Conclusions - Performance

There was limited evidence to suggest that HCPCP was achieving its outcomes. While funded projects were successful in producing outputs, there was little evidence to demonstrate the impact of their efforts. Many funded projects ended with the development of a knowledge product or with its dissemination (which was often passive, e.g. posting online). As a result, projects could not always report on outcomes and HC does not track project outcomes once funding has ended. There was also little in the way of evidence to demonstrate Health Canada's role, besides that of funder, in having an impact on the health care system. Health Canada key informants acknowledged that the collection of performance data to demonstrate the achievement of outcomes had been a weakness for the HCPCP.

As a funder, it is in Health Canada's interest to fund projects that achieve measureable and meaningful health system outcomes. The department may benefit in conceptualizing its contribution beyond funding and managing projects, producing reports and organizing meetings. This would involve determining what its role is within the HCPCP and how its activities contribute to improving the health system. The evaluation pointed to the need for Health Canada to determine the scale of its intended impacts (i.e., jurisdictional vs. pan-Canadian). A pan-Canadian focus was not consistently maintained in project selection. Health Canada may wish to chart a new direction in terms of programming, determine the scope and scale of stewardship that it wants to engage in, the value the Department can add to the efforts of the projects the Department funds and report on these efforts. Health Canada may want to re-examine the HCPCP logic model where the Program has outlined its activities, outputs and outcomes.

Based on internal key informant data, HCPCP would benefit from developing a clear understanding of breadth of the intended impacts, e.g. jurisdictional vs. pan-Canadian impacts, and  knowledge translation. Interviews indicated that there was some disagreement on whether pan-Canadian impacts were sought and that there was inconsistency with regard to the intended uptake to demonstrate knowledge translation. If the Program intends to make knowledge translation a focus of performance measurement, building a common understanding of expectations for knowledge translation and buy-in from Program staff and other stakeholders will be important.

Recommendation 1

Health Canada should explore ways to optimize the investment in the HCPCP in order to achieve program outcomes. In this context, the HCPCP should refine its priorities, adjust program objectives and associated program design, building on lessons learned over the past decade.

Recommendation 2

Systematic project and program tracking to achieve program results should be strengthened to inform program decision-making and reporting on success.

Performance - Efficiency and economy

The issue of economy requires the ability to consider Program cost in relation to the outcomes produced, and without systematic performance data, the evaluation could not conduct this analysis. The ability to respond to the issue of efficiency was affected by the inability to track how HCPCP Vote 1 money had been used, as it was combined with other Vote 1 funds for the Directorate. That being said, the evaluation was able to consider issues of efficiency in terms of whether Program resources were used as planned and whether decision making was efficient and economical.

The Program experienced funding lapses each fiscal year covered by the evaluation (2008-09 to 2012-13). The lapses were attributed to several challenges experienced by the Program, including the need to hold funds aside for the Family Medical Residencies Initiative (FMRI); difficulties in getting information on delays from provinces and territories in time to reallocate funds; late proposal submissions and lengthy review processes that reduce the ability to use funds in the first year of a contribution agreement. Given the need to demonstrate efficiency and economy within the Program, the HCPCP should consider steps to reduce future funding lapses.

The evaluation identified other areas of program management where the HCPCP could improve its efficiency, and the HCPCP had already undertaken efforts to improve in all of these areas. The proposal approval process within Health Canada was sometimes lengthy, which was one reason why a substantial percentage of contribution agreements required amendments. The Program had undertaken methods to improve the efficiency of the proposal approval process through improved communication with senior Health Canada officials about upcoming proposals. Previous efforts to improve the objectivity and transparency of the proposal review and approval process could be expanded upon to improve Program functioning and ensure that funding priorities are clearly articulated, that funded projects better target identified needs/gaps, and that funds expended have greater potential to produce the intended outcomes.

Management Response and Action Plan

Health Care Policy Contribution Program 2008-2012
September 26, 2013

Recommendation	Program Response/ Activities	Outputs/ Deliverables	Responsibility Centre	Timeframe
Recommendation 1: Health Canada should explore ways to optimize the investment in the HCPCP in order to achieve program outcomes. In this context, the HCPCP should refine its priorities, adjust program objectives and associated program design building on lessons learned over the past decade.	Agree. The Program will conduct a collaborative strategic planning process to inform Program redesign that will include: a review of Program objectives and authorities development of Program parameters, e.g. scope, eligibility, funding allocations.	Discussions with key stakeholders to inform program objectives and priorities. Revised Program objectives, Terms and Conditions; funding criteria and related proposal assessment criteria.	Director General (DG), Health Care Programs and Policy Directorate (HCPPD)	June 2014
		Modified solicitation documents (e.g. Call for Proposals, Guide for Applicants) that require applicants to demonstrate measurement of project outcomes and alignment with Program policy and funding priorities.)	DG, HCPPD	October 2014
Recommendation 2: Systematic project and program tracking to achieve program results should be strengthened to inform program decision-making and reporting on success.	Agree. The Program will develop and implement a HCPCP Performance Measurement Strategy (PMS) to capture information on financial and non-financial Program activities.  The tools and processes for data collection, analysis, and reporting will be modified as required to support improvements in performance measurement and financial monitoring for the Program. The data will inform Program planning, corporate reporting (e.g. RPP, DPR), and decision-making for future funding priorities. The Program will explore options for development of improved tools to assess longer term program impacts	A Performance Measurement Strategy and supporting documents, including revised operational guidelines and training materials for staff and managers.	DG, HCPPD	December 2014
		Program work plans, input into corporate reporting, and Records of Decisions from Program priority-setting discussions	DG, HCPPD	March 2015
		Options for measurement of program impact post expiry of funding agreements.	DG, HCPPD	March 2015

1.0 Introduction

The Health Care Policy Contribution Program (HCPCP) is a national program that supports the development of policies and strategies to address health care system priorities as identified in the First Ministers' 2003 Accord on Health Care Renewal and the First Ministers' 10 Year Plan to Strengthen Health Care in 2004.  Between 2008-09 and 2012-13, the HCPCP has funded 163 projects to increase knowledge and application of evidence and best practices, which are ultimately intended to improve health care system planning and performance.^{Footnote 3} The objective of the Program is to foster collaboration and coordination of responses to health care system priorities among federal, provincial, and territorial governments; other health care policy-makers; service providers; users; researchers; and other stakeholders. The total budget allocated to the HCPCP between 2008-09 and 2012-13 was $250M and $201M for actual expenditures.

The evaluation of the HCPCP was a scheduled evaluation on the Public Health Agency of Canada/Health Canada's Five-Year Evaluation Plan. The evaluation assessed the relevance and performance (effectiveness, efficiency, and economy) of the Program, in accordance with the Treasury Board Policy on Evaluation (2009) and to fulfill requirements of the Financial Administration Act. The evaluation covered the 2008-09 to 2012-13 fiscal years (FY). The results will inform the implementation of current and future activities of the HCPCP.

The evaluation took place between April 2012 and August 2013 and drew on several lines of evidence, including a literature and program-level document review, a project-level document review, a survey with contribution agreement primary contacts, and key informant interviews. This report presents the evaluation findings, draws conclusions, and identifies two recommendations.

1.1 Background and context

Just over a decade ago, two major reviews of Canada's health care system were released. The impetus for these reviews was concern with sustainability and accessibility of the health care system. The Standing Senate Committee on Social Affairs, Science and Technology, chaired by Michael Kirby, engaged in a two-year study of the health care system and released its final report in October 2002 (Standing Senate Committee on Social Affairs, Science and Technology, 2002). The federal government had also created a Commission on the Future of Health Care in Canada headed by former Saskatchewan premier Roy Romanow to conduct an inquiry into the issues facing the health care system, which released its reports in November 2002 (Commission on the Future of Health Care in Canada, 2002). Both reports provided recommendations for health care renewal. While the reports differed in many respects, both included recommendations that considered accessibility, sustainability, and accountability.

Federal, provincial, and territorial governments were simultaneously setting priority areas for health care renewal. The 2000 Communiqué on Health, which resulted from a First Ministers' meeting, committed to an increase in the federal health transfer payment, as well as targeted federal funding to provinces and territories (Canadian Intergovernmental Conference Secretariat, 2000). Subsequently, two additional agreements set out specific priority areas. The First Ministers' 2003 Accord on Health Care Renewal and the First Ministers' 10-Year Plan to Strengthen Health Care in 2004 committed additional funding through the Canada Health and Social Transfer, as well as targeted funding to achieve certain pan-Canadian objectives. Among these objectives were improving access to primary care, including access to care in the North; reducing wait times in priority areas; improving the planning and management of health human resources, including integration of internationally educated health professionals; improving access to home care and end-of-life care; and promoting health care innovation to enhance access and quality (First Ministers, 2003, 2004). Better data collection and reporting on results to Canadians were also key components of these commitments, often referred to collectively as the Health Accords.

The federal government used a variety of mechanisms, including the HCPCP, to address these priority areas identified in the Health Accords. Through use of its contribution funding, the HCPCP targeted many of the areas listed in the Accords, as is shown in the description of the Program's components provided in the following sections.

1.2 Program components

The HCPCP funds projects through five program components (detailed description provided in Annex 1): the Health Human Resource Strategy (HHRS); the Internationally Educated Health Professionals Initiative (IEHPI); the Health Care System Innovation Fund (HCSIF); the National Wait Times Initiative (NWTI); and the Patient Wait Times Guarantee Pilot Project Fund (PWTGPPF) (Health Canada, 2012a).^{Footnote 4} Two of the five components -- the NWTI and the PWTGPPF -- have sunset, but were included in the evaluation, as they received funding during the evaluation period of 2008-09 to 2012-13. Depending on the component, funding recipients included provincial/territorial governments and non-governmental organizations as presented in Table 1.

Table 1 Contribution agreements by HCPCP components and type of funding recipient, 2008-09 to 2011-12

Program component	Type of funding recipient						Total actual funding for contribution agreements
	Province or territory		Educational institution		Not-for-profit organization
	#	$	#	$	#	$	#	$
HHRS	9	3,711,856	22	9,689,512	39	23,340,149	70	36,741,517
IEHPI	25	43,721,910	2	856,706	8	2,520,587	35	47,099,203
HCSIF	0	0	4	149,538	35	15,597,526	39	15,747,064
NWTI	0	0	0	0	8	2,849,489	8	2,849,489
PWTGPPF	11	19,274,891	0	0	0	0	11	19,274,891
Total	45	66,708,657	28	10,695,756	90	44,307,751	163	121,712,164
Source: (Health Canada, 2012b) Two HHRS projects and three IEHPI projects were approved but did not receive funding until FY 2012-13.

1.3 Program objectives, activities and outcomes

This section describes the program logic for the HCPCP and descriptions of the key elements, which represent the theory behind the HCPCP. These descriptions explain the expected results of Program activities. The visual diagram depicting the linkages between HCPCP activities, outputs, and expected outcomes appears in Annex 2. The evaluation findings in Section 4 assess whether the Program activities are being implemented as planned and whether expected outcomes are, in fact, being achieved.

1.3.1 Program activities and outputs

The HCPCP consisted of four principal activities:

• Strategic planning includes identifying health care system needs and gaps, and setting the Program's direction and funding priorities.
• Management and accountability includes instituting a robust governance structure for the Program and providing Program oversight through performance monitoring, audit, and evaluation.
• Facilitating collaboration involves supporting the building and maintaining of collaborative approaches to bring together combinations of researchers, health care program managers, health care professionals, and policy-makers to identify and prioritize needs, conduct research, and collect performance data that feed into policy development and implementation activities.
• Policy and research development and implementation includes funding projects, synthesizing findings, and identifying promising practices and strategies (Health Canada, 2008c, p. 18).

These activities produced the following outputs:

• knowledge products, such as research reports, databases, reference materials, and planning tools; strategies, approaches, or models; synthesis of research, policy options, and advice; and dissemination mechanisms
• collaborative relationships with and among stakeholders, including recipient organizations, professional associations, governments, and policy-makers
• identification of barriers and enablers to the creation of knowledge products, dissemination and use of knowledge, and renewal and innovation in the health care system (Health Canada, 2009b, p. 22)

1.3.2 Outcomes

The implementation of the activities identified above corresponds with specific immediate, intermediate, and longer-term outcomes. The immediate outcomes included the following:

• Increased awareness and understanding of knowledge tools, products, approaches, models, innovations, health care system performance, and health care system reform issues. Using appropriate information dissemination mechanisms and collaborative networks of policy-makers, researchers, and other stakeholders will aim to build awareness among target users of evidence related to health care system performance and potential opportunities for improvement.
• Improved collaboration and coordination among health care system stakeholders to minimize identified barriers and utilize identified enablers. HCPCP activities include identification of barriers and enablers and development of strategies to minimize the effects of these barriers and utilize enablers. This will facilitate improvements in health care system policies and practices.
• Successful and/or increased use of knowledge, approaches, models, strategies, or promising practices. HCPCP activities include the development of new evidence and knowledge, including new or enhanced tools, approaches, practices, and models. This outcome considers whether the projects resulted in increased use or, for new knowledge products for which a baseline of usage would not exist, successful use at the level of project partners.

In addition, Program activities and the achievement of the immediate outcomes were expected to foster several enabling factors that will support future health care system reform. Collaborative relationships with and among stakeholders were to lead to improved overall collaboration and coordination. By decreasing identified barriers to health care system reform, the HCPCP was to enable knowledge development, translation, and use. These, along with increased awareness of knowledge tools, products, approaches, models, innovations, and health care system reform issues, were to encourage a supportive organizational culture that is ready for change and has strong leadership.

The achievement of these immediate outcomes and the establishment of these enabling factors were expected to lead to the intermediate and, ultimately, the long-term outcomes:

• Intermediate Outcome: Broader adoption by relevant health care system stakeholders of knowledge or innovations resulting in changes in policy and/or practice. HCPCP activities include dissemination mechanisms that will broaden the potential reach of the evidence and knowledge tools developed by funded projects. This intermediate outcome considers the success of the Program in the uptake of new or enhanced evidence and knowledge tools by the broader target audience.
• Longer-term Outcome: Improvements in health care system policies, practices, and performance in areas targeted by the HCPCP. The adoption of new or modified policies and strategies should address the health care system priorities targeted by the HCPCP. Evidence produced, disseminated, and applied with support from the HCPCP should help improve health care system planning and performance, which is the long-term outcome for the HCPCP.

1.4 Governance

Health Canada's Strategic Policy Branch manages the HCPCP through the Health Care Programs and Policy Directorate (HCPPD or the Directorate) (Health Canada, 2010a). The Program Management and Accountability Unit (PMAU)^{Footnote 5} provides central program management and coordination functions for the Program, while project management is decentralized -- that is, takes place within the Program component unit (e.g., Health Human Resource Strategy and the Internationally Educated Health Professionals Initiative) or within the policy area responsible for the project (e.g., Chronic and Continuing Care Unit, Office of Pharmaceutical Management Strategies).

Prior to 2008, the components operated relatively independently from each other, subject to the HCPCP Terms and Conditions. However, several circumstances created the need to change the governance structure of the HCPCP from a more decentralized structure to one intended to facilitate strategic planning and overall coordination, as well as improve performance measurement and reporting. The 2007 evaluation of the Program recommended more centralized strategic planning and program coordination (Health Canada, 2008b). With the 2008 renewal of the Program, transfers among components were now permissible to prevent lapsing of funds, which necessitated more coordination among the components. The Government of Canada Action Plan to Reform the Administration of Grant and Contribution (G&C) Programs stressed the need for mechanisms to ensure that transfer payments through G&Cs are managed in a transparent, accountable, and risk-sensitive manner (Health Canada, 2010a; Treasury Board of Canada Secretariat, n.d.).

The resulting structure included three committees:

• a Program Management Committee (PMC) that oversaw the HCPCP and was chaired by the Director General of the HCPPD,
• an Accountability Committee that was responsible for performance measurement, and risk identification and management activities, and
• an Operations Committee that coordinated general program management functions (e.g., communications, finance, and standard operating procedures and tools).

The governance structure of the HCPCP will be discussed in more detail in Section 4.1.

1.5 Resources

During the period covered by the evaluation (2008-09 to 2012-13), the HCPCP had an adjusted budget allocation of $197M for Vote 10 (Contribution funding) and $52M for Vote 1 (Operations and Maintenance funding). Actual expenditures for Vote 10 were significantly lower ($149M). Vote 1 funding was not tracked. Consequently the evaluation was not able to compare adjusted budget allocation to actual expenditures. The issues of tracking actual Vote 1 expenditures and lapsed funding are discussed further in Section 4.3.

2.0 Evaluation Methodology

2.1 Purpose and scope of the evaluation

The goal of the evaluation was to provide senior management with a neutral and evidence-based assessment of the relevance and performance of the Program. The following questions guided the collection of data and addressed the five core issues required under the 2009 Treasury Board Policy on Evaluation. Key findings for each question were then synthesized into broader conclusions.

Relevance

• 1. Is there a continued need for the Program?
• 2. Does the Program align with Government of Canada and departmental priorities?
• 3. Is the Program aligned with federal roles and responsibilities?

Performance - Effectiveness

• 4. To what extent has the Program been implemented as planned?
• 5. To what extent has the Program contributed to the achievement of its identified outcomes?
• 6. Were there any unintended consequences, either positive or negative, resulting from the Program?

Performance - Efficiency and Economy

• 7. Does the Program demonstrate efficient and economical decision making and implementation?
• 8. Is the Program managed in the least costly way, or are there alternative methods that are more economical?

2.2 Evaluation design and data collection methods

A preliminary review of Program documents, as well as four initial interviews with Health Canada key informants, informed the evaluation design. In addition, an evaluation working group consisting of representatives of the HCPCP assisted by reviewing data collection instruments. An evaluation matrix, which lists the evaluation questions, indicators, and lines of evidence, was used to guide the study.

The evaluation consisted of the following data collection methods:

• Literature and program-level document review. The literature review and program-level document review addressed evaluation questions related to Program relevance and effectiveness. Relevant peer-reviewed (i.e., academic) and grey literature considered in the review were located through online searches. Program-level documents included government documents primarily produced by Health Canada that pertained to the Program overall, as opposed to specific projects.
• Project-level document review. Initial evaluation plans assumed that data from Recipient Reporting and Evaluation Template (RRET) would be available to analyze project-level activities, outputs, and outcomes.While the RRET system has been developed largely as planned, project coverage was low. Also, the HCPCP decided to roll out the RRET with new projects funded after December 2009; existing projects were not required to change their reporting format. In addition, provinces and territories were not required to use the RRET (although the Program has reported that most provinces and territories with new contribution agreements are using the RRET). The PMC was cautioned that this approach would provide a limited pool of data, and it was suggested by the PMAU that this approach be reconsidered (Health Canada, 2010b); however, the implementation of the RRET in April 2010 proceeded as planned. As a result of the method chosen to introduce the RRET, only 33 of the 163 projects funded during the time period covered by the evaluation have used the RRET. As these projects were initially funded in 2010-11 or 2011-12, they had only been using the RRET for about one full year when the current evaluation began. The results of these decisions meant that the HCPCP did not have many projects with systematic and comparable performance information gathered through the RRET. Consequently, a project-level document review was undertaken.
  
  The project-level document review involved a systematic review of 45 project files out of the 163 projects funded between 2008-09 and 2012-13. The evaluation used a stratified random sample of 30 project files based on the component, project status, and budget. The sample was supplemented with 15 completed project files to maximize the ability to obtain outcome information and cover a larger proportion of the HCPCP funding between 2008-09 and 2012-13. None of the project files reviewed contained completed RRETs.
  
  The review focused on key documents that addressed implementation and performance for each project. Evaluators used a template to ensure that the information on each element of the Program logic for the HCPCP was captured systematically. The review addressed evaluation questions related to relevance, effectiveness (implementation and outcomes), and efficiency and economy.
• Survey with contribution agreement recipients. The online survey targeted representatives of contribution agreements that received funding between 2008-09 and 2012-13. Some organizations received more than one contribution agreement and listed the same primary contact for all agreements. As a result, the final sample included 120 unique individuals representing the 162 contribution agreements.^{Footnote 6} The survey solicited performance information on each contribution agreement's activities and outcomes, which required sending some individuals multiple survey requests. Although attempts were made to reduce respondent burden and identify alternative contacts in these cases, this was not always possible and may have affected the response rate. After one month online, the survey achieved 95 responses, for a response rate of 58.6%. The survey addressed effectiveness, as well as efficiency and economy.
• Interviews with Health Canada stakeholders. Two rounds of interviews with Health Canada representatives were conducted for the evaluation. The first round of preliminary interviews (n=4) occurred in July 2012. This round of interviews provided the evaluation team with background and contextual information to assist with the design of the methodology and data collection instruments, as well as addressing some evaluation questions. The second round involved 13 interviews with 22 Health Canada representatives and occurred in December 2012 and January 2013. The interviews were the final step in data collection and were intended to give Health Canada an opportunity to respond to the preliminary findings from the other lines of evidence and to fill any gaps in information or understanding. The interviews addressed relevance, effectiveness, and efficiency.

2.3 Limitations and mitigation strategies

The evaluation experienced several methodological limitations, which are described in Table 4. To respond to the limitations, the evaluation used multiple lines of evidence to gather feedback from stakeholders with different perspectives from within Health Canada (e.g., managers and staff, policy and Program officers), and funding recipients. Evaluators reviewed project- and program-level documentary evidence. By using triangulation of findings from these different sources, the evaluation was able to provide strengthened conclusions. In addition, the evaluation took specific steps to mitigate each limitation, as described in Table 2 below.

Table 2: Limitations and Mitigation Strategies

Limitation	Challenge	Mitigation Strategy
Lack of systematic performance data	The evaluation experienced challenges due to the lack of systematic performance data for all projects funded between 2008-09 and 2012-13. While the HCPCP has begun to collect standardized performance data through its RRET, the Program does not have complete performance information. The RRET is available for a small subset of the projects funded between 2008-09 and 2012-13; a pilot was launched in April 2010 and is mandatory for new contribution agreements after that date, with the exception of provinces and territories, for which it is optional. As a result, during the period of this evaluation few projects have used the RRET.	To address this challenge and gather more performance information at the project level, the evaluation took two actions. First, it expanded the project-level document review to include more files, with a focus on closed files with larger budgets. The additional files selected focused on the completed, larger-budget projects for each component, as these projects can be expected to have a more substantial impact and, as completed projects, should be able to report on outcomes. Second, the evaluation conducted a survey that solicited information on project performance from all of the primary contacts of contribution agreements that received funding during 2008-09 and 2012-13.
Self-selection bias	The survey with funding recipients is affected by self-selection bias, which is bias based on who responds and who chooses not to respond.	The evaluation undertook several steps to encourage responses from all recipients. A pretest was conducted to ensure relevance, clarity, and ease of response. Reminders were provided and the survey deadline extended to increase the response rate. The survey gathered information on a few key characteristics (component, type of funding recipient, level of organization, and contribution agreement status) to allow the evaluation to determine whether self-selection bias might have affected survey findings. A comparison of respondents to the overall sample on these characteristics showed that the survey respondents were representative of the sample of contribution agreements funded between 2008-09 and 2012-13.
Response bias	The survey and interviews with Health Canada stakeholders also have the possibilities of self-reported response bias, which occurs when individuals are reporting on their own activities and may want to portray themselves in the best light, and strategic response bias, whereby the participants answer questions with the desire to affect outcomes.	The mitigation strategy for this limitation was to use documentary evidence where possible in order to confirm or supplement stakeholder opinions.
Selectively deposited data/incomplete project-level files	A template was developed to capture information to assess the completeness of project files so that the results could be judged on the basis of the availability of documentation. The project-level file review found that several files were incomplete, which affected the ability to assess implementation and/or performance of those projects. The evaluation team found that even when documents were available, the content was not always clear.	A review based on this relatively small number of completed project files limits the ability of the evaluation to address program success. As a result, the sample was increased from 30 to 45 complete project-level files. Contribution agreement recipient survey data were used to compare and contrast similar data from the project-level document review.

3.0 Findings - Relevance

3.1 Is the HCPCP aligned with federal roles and responsibilities?

As a targeted funding initiative, the HCPCP is aligned with the federal role in health care by providing leadership, promoting health care system innovation, and supporting federal/provincial/territorial coordination and collaboration to improve the health care system.

Jurisdictional authority for health care in Canada is largely determined by the Constitution Act, 1867. While the provinces and territories are primarily responsible for the delivery of health care, the federal government supports the provincial/territorial role by providing transfer payments, establishing national standards through the Canada Health Act, and undertakes other health-related functions outlined in the Department of Health Act, such as conducting public health research and cooperating with provinces and territories on efforts to improve public health (Government of Canada, 1985; Health Canada, 2006, 2011c). The federal role in supporting other health-related functions includes several types of policy levers, such as grant and contribution programming, where the federal government provides funding to provinces, territories, or other organizations to pursue particular policy commitments, promote innovative practices, and generally provide federal leadership on health-related issues. Health Canada is the department responsible for administering these federal activities (Jackman, 2000, pp. 98-99).

3.2 Is the HCPCP aligned with the Government of Canada and departmental priorities?

The HCPCP initially aligned with the Government of Canada and departmental priorities as defined in the First Ministers' meetings that led to the Health Accords. Since then, the federal government has continued to emphasize its commitment to protecting the health of Canadians and to improving the health care system.

Federal commitments to improving the health care system were demonstrated in the two most recent Speeches from the Throne and the 2008-09 and 2011-12 Federal Budgets (Government of Canada, 2008, 2009, 2010a, 2010b, 2011a, 2011b, 2012a, 2013). Although the HCPCP was not directly mentioned, these statements of federal priorities supported similar goals of modernization of the health care system, improved accessibility, increased supplies of health care providers, and more rapid responses to identified needs. For example, the 2011 Speech from the Throne reiterated the federal commitment to investing in health care "while respecting provincial jurisdiction and working with the provinces and territories to ensure that the health care system is sustainable and that there is accountability for results" (Government of Canada, 2011a).

Budget 2012 included references to several priority areas also covered by the HCPCP, including:

• improved use of health human resources to make the health care system more cost-effective
• further improvements to foreign credential recognition through the Pan-Canadian Framework for the Assessment and Recognition of Foreign Qualifications, which includes health care professionals among its target occupations
• actions to strengthen health care in rural and remote communities (Government of Canada, 2012b, Chapter 3.1 and 3.3)

In addition, in two recent government responses to Standing Committee reports, the federal leadership role in supporting targeted initiatives was reiterated, which implicitly reinforces the appropriateness of policy levers such as the HCPCP as an expression of that leadership. In the federal government's response to a report by the Standing Committee on Health, it recognized the importance of health human resources in ensuring accessibility and responsiveness in the health care system. The response stated that the federal government "plays a leadership role by supporting a range of targeted projects and initiatives of national significance" (Government of Canada, 2010c). Similarly, the federal government response to the report of the Standing Senate Committee on Social Affairs, Science and Technology that reviewed the 2004 Health Accord noted that the contributions funded through the HCPCP comprised one of the federal measures in addressing the priority health care system issues (Health Canada, 2012d).

While the above statements demonstrate a continued federal commitment to the improvement of the health care system, recent events indicate some uncertainty about future federal priorities for health care renewal. The 2011 Speech from the Throne included a specific reference to renewing the Health Accord, but federal priorities have since shifted. The federal government took a different approach from the process of federal/provincial/territorial talks that led to the previous Health Accords by announcing, at the end of 2011, a new formula for health care transfers effective fiscal year 2017-18 with no mention of negotiating a new Health Accord (Bailey & Curry, n.d.; Government of Canada, 2011c). The Minister of Health indicated a federal interest in discussing performance measurement and accountability without compromising the ability of provinces and territories to establish their own regional approaches (Health Canada, 2011d). At the most recent meeting of the Council of the Federation, the provinces and territories announced their own plans for continued health care reform by establishing future priority areas without federal input (Health Care Innovation Working Group, 2012). The federal government has stated that it "remains committed to collaborating with the provinces and territories and other partners to implement change and to play a leadership role in areas where it has clear jurisdiction" (Health Canada, 2012d, p. 19). As of 2013, the federal government has not indicated its intent to negotiate a new Health Accord but it has continued to emphasize in the most recent Speeches from the Throne (2013) its broad commitment to the improvement of the health care system.

The Health Accords provided a clear set of priorities. Given that they are soon to expire, the HCPCP is at a watershed moment without a similar statement of federal priorities to guide future health care contribution funding. The issue of the continued need for federal involvement in the current HCPCP funding areas will be addressed in Section 3.3.

The evaluation found that the HCPCP aligns with Health Canada departmental priorities. It supports the departmental strategic outcome "a health system responsive to the needs of Canadians" (Health Canada, 2008d, 2009c, 2010c, 2011a). The HCPCP also aligns with the expected result for the first strategic outcome, which is to serve "as a catalyst to address current and emerging health issues and priorities," as demonstrated by:

• increased adoption of new approaches, models, and best practices in the health care system;
• increased numbers of policies, practices, and proposals targeting the needs of the health care system; and
• improved collaborations between Health Canada and other stakeholders in the health care system (Health Canada, 2011a, p. 9 and 20).

3.3 Is there a continued need for the HCPCP?

Based on the available literature and health care system data, there remain health care system needs in each of the priority funding areas of the HCPCP.

A brief review of the current health care system issues is provided below by program component, with a concluding section on the continued need for innovation and knowledge translation.

Health Human Resource Strategy: While the adequacy of health human resources remains a key theme in the literature, the available data show that some areas of shortage are being addressed. In particular, the supply of physicians has grown in recent years (CIHI, 2011a). A 2010 CIHI report showed that growth in the number of physicians in Canada consistently outpaced population growth from 2005 to 2010, growing by 6.8% (CIHI, 2011a). The number of medical degrees awarded in Canada increased 30% from 2005 to 2010, and the number of international medical school graduates grew 18%. The percentage of physicians leaving Canada also dropped 16% from 2006 to 2010 (CIHI, 2011d, pp. 39-40).

In contrast to physicians, the number of registered nurses, pharmacists, and physiotherapists had not seen comparable growth between 2005 and 2010. For nurses, one recent study estimated a 113,000 shortfall nationally by 2016 (Ferguson-Paré, Mallette, Zarins, McLeod, & Reuben, 2010). Data from CIHI also found evidence of nursing shortages, as the number of registered nurses had barely kept pace with population growth since 2000. Between 2005 and 2010, the number of licensed practical nurses per 100,000 population improved, but only after experiencing a decline in the previous five years. For pharmacists, the reverse trend was seen, with the improvement in 2000 to 2005 replaced by maintenance in the number of pharmacists per 100,000 population between 2005 and 2010. The number of physiotherapists did not see any change in the 2005 to 2010 period.

With all health professionals, distribution remains an issue, although substantial improvements have occurred. The most recent CIHI data show that the number of physicians in rural areas increased by 9.8% between 2007 and 2011, while in comparison, the population in rural areas increased by 1.7% between 2006 and 2011 (CIHI, 2011b, p. 7). This has resulted in 15% of family physicians working in rural areas where 18% of the population live (CIHI, 2011b, p. 3). The distribution of nurses in rural areas, however, has greater room for improvement, as in 2010, over one tenth (10.8%) of nurses worked in the territories or rural or remote areas (CIHI, 2012b, p. 34). The increase of physicians has also not been distributed evenly across the provinces and territories. In particular, between 2007 and 2011, the number of physicians per 100,000 population declined in the Northwest Territories (-21.4%) and Yukon Territory (-8.6%), while it increased in the other provinces and territories (CIHI, 2011b, p. 8)(CIHI, 2011b, p. 8).

Other areas funded by HHRS include effective use of human resource skills, which has focused on interprofessional collaboration, particularly in terms of education and training. In the literature, interprofessional collaboration is still an area of sustained interest, particularly because team-based care is seen as an important method of improving access to primary care, as well as home and community care (Grimes & Tholl, 2010, p. 2). The 10-Year Plan's goal of 50% of Canadians having round-the-clock access to a multi-disciplinary team by 2011 has not been met (CBOC, 2012, p. 20).

Furthermore, a review of the literature identified several knowledge gaps that the HCPCP could address with future funding, such as developing an evidence base on the most effective models of interprofessional collaboration, the cost-effectiveness of interdisciplinary teams, and how this approach affects patient outcomes (Barrett, Curran, Glynn, & Godwin, 2007; Chan & Wood, 2012; Grimes & Tholl, 2010). Enabling health care professionals to work at their full scope of practice has been identified as essential to improving access to care and minimizing the impact of health human resource shortages (HCC, 2012, p. 10)(CBOC, 2012, p. 20). While some projects related to scope of practice were funded between 2003-04 and 2007-08, none have been funded since the Program's renewal in 2008-09, which could be considered a gap.

Health human resource issues remain on the agenda based on numerous reports and other statements of priorities. In analyzing 18 major national and provincial health care reform reports released in the last 15 years, the Conference Board of Canada (CBOC) identified health human resources as one of the seven recurring health care reform themes, with 14 of the 18 reports making recommendations related to health human resources (CBOC, 2012). Other major reports since the CBOC study continued to identify health human resources as a priority area for health care renewal. A 2012 Standing Senate Committee review of the 2004 Health Accord found progress on health human resources but identified future areas for federal involvement in the areas of strategic health human resource planning (e.g., creation of a national observatory on health human resources); the distribution of health human resources, particularly for rural and remote areas; and interprofessional training to support the expansion of team-based practice (Standing Senate Committee on Social Affairs, Science and Technology, 2012). The Council of the Federation's Health Care Innovation Working Group included recommendations on clinical practice guidelines, team-based models of care, and health human resource management (Health Care Innovation Working Group, 2012).

Internationally Educated Health Professionals Initiative: The review of the literature found that internationally educated health care professionals still experience barriers. As the Forum of Labour Market Ministers (FLMM) pointed out, Canadian occupational regulatory systems were designed to respond to Canadian-educated applicants. The literature identified several barriers that appear to be fairly consistent across categories of health care professionals, including: supplying and validating credentials, language skills, lack of support networks, achieving educational equivalence, cost, and difficulty with passing certification examinations (Baumann & Blythe, 2009; Johnson, Baumal, & CSMLS, 2011; Keith Johnson, 2007). Bridging and adaptation programs, which provide assistance such as skill and educational assessments, skills training, preparation for licensure examinations, and language training, are seen as critical supports to increase the integration of internationally educated health care professionals and are currently not available in all jurisdictions (Baumann & Blythe, 2009). Beyond licensure, internationally educated health care professionals also continue to experience challenges in obtaining employment in their chosen health care field and becoming successfully integrated into the workforce (Johnson et al., 2011).

Health Care System Innovation Fund (HCSIF): The HCSIF covers a wide array of health care issues that include some overlap with the other HCPCP components. The HCSIF funding objectives include wait times, access to health care, chronic and continuing care, and end-of-life care. In addition, the HCSIF funds the Canadian Medication Incident Reporting and Prevention System (CMIRPS) project. Given the variety of issues that the HCSIF can fund, assessing the continued need for its priority areas is difficult.

The aging population has led to increasing attention on access to long-term care, end-of-life care, and home care. Several authors have suggested that governments augment access to long-term care, palliative care, and end-of-life care (CLHIA, 2009, p. 9; CMA, 2010, pp. v-vi; Standing Senate Committee on Social Affairs, Science and Technology, 2012). Without adequate home or continuing care, the acute care sector is relied upon instead (Brasset-Latulippe, Verma, Mulvale, & Barclay, 2011). Government support for additional home care services and for their integration with primary health, as well as acute and long-term care, has been urged as a priority in health care reform since 2002 (CIHI & Statistics Canada, 2002; Fooks & Lewis, 2002; HCC, 2008, 2011; Soroka, 2007)(CIHI & Statistics Canada, 2002; Fooks & Lewis, 2002; HCC, 2008, 2011; Soroka, 2007). The review of the 2004 Health Accord recommended new federal initiatives in home care, in particular (Standing Senate Committee on Social Affairs, Science and Technology, 2012). Chronic and continuing care, end-of-life care, and home care are priority areas that have received comparatively less attention by the HCPCP and should be considered in any review of HCPCP funding priorities.

The CMIRPS project focused on medication safety errors, which the available evidence shows are not isolated occurrences. A recent study found that6% of all Canadians and 11% of Canadians with chronic care needs or more acute, intensive care needs had experienced a medication error in the previous two years (Blendon, Schoen, DesRoches, Osborn, & Zapert, 2003). Another study of 14 Ontario hospitals identified 4,243 errors (0.86 errors per bed, or 0.25 errors per thousand doses of medication dispensed) over a 12-month period. These errors resulted in 120 incidents that possibly contributed or did contribute to patient harm (adverse drug events), and 685 errors that reached patients, but did not cause harm (potential adverse drug events) (Marshman, Lam, & Hyland, 2006).These incidents have as their cause a systemic problem that can be addressed through better processes, procedures, medication labels, and other interventions (CIHI, 2009; Etchells, Juurlink, & Levinson, 2008). Thus, the literature supports the CMIRPS approach to identifying system responses to reduce medication errors.

National Wait Times Initiative (NWTI) and the Patient Wait Times Guarantee Pilot Projects Fund (PWTGPPF): Improving Canadians' timely access to health care has been a formalized federal/provincial/territorial government priority since the Health Accords. Diverse academic and grey literature has continued to cite wait times as an issue hindering accessibility of health care (CMA, 2005, p. 2, 2011, p. 5; Duckett & Kempton, 2012, pp. 98-99; HCC, 2008; Kreindler, 2008, pp. 129-131; Saryeddine & Brimacombe, 2009, pp. 1-3). While the NWTI and the PWTGPPF have both ended, these components, along with other federal funding commitments (e.g., the Wait Times Reduction Fund and the Patient Wait Times Guarantee Trust) have addressed wait times in Canada through various means, including by targeting five clinical areas: cancer care, hip and knee replacement, cardiac care, diagnostic imaging, and cataract surgeries. Benchmarks were established by provinces and territories for hip/knee replacement, hip fracture repair, cataract surgery, coronary artery bypass graft surgery, and radiation therapy for cancer patients (CIHI, 2011c).

As noted above, the distribution of health care professionals means that Canadians in remote or rural areas continue to experience issues with access to care. Recent studies that compared Canadian perceptions of health care performance to other OECD countries demonstrated continued issues with access. Almost 35% of Canadian adults reported waiting six days or more to see a doctor when sick or needing medical attention in 2010, up from 25% in 2004, which was substantially higher than Australia, the United Kingdom, or the United States (Duckett & Kempton, 2012).

The literature showed a continued need to address wait times. Although there was evidence of improvements for some targeted clinical areas (see Section 4.2.2), few provinces completed 90% or more of three targeted procedures - hip replacement, knee replacement, and cataract surgery - within a clinically-appropriate time frame (CIHI, 2012c). Other areas noted for long wait times have included: diagnostic imaging (particularly MRI scans), coronary revascularization, gastroenterology, and emergency rooms (Armstrong et al., 2008, p. 160; Gutkin, 2011, p. 256; HCC, 2011; Leatherman et al., 2010, pp. 12-13; Leddin et al., 2008, p. 166; Southern et al., 2011, pp. 262.e25-26).

On a system-wide level, the literature shows a continued need to address wait times, although there is also evidence of improvements for some targeted clinical areas. In 2013, CIHI reported that approximately eight out of ten patients across Canada received procedures in the targeted clinical areas within clinically-recommended time frames or benchmarks, although the results varied by province and type of procedure (CIHI, 2013). CIHI determined a target of 90% of patients receiving procedures within the benchmarks to be reasonable given the realities of other circumstances, such as patient illness. Nine out of ten provinces met the target for radiation therapy, while no provinces met the target for the other priority clinical areas. In addition, the CIHI report found that progress appears to have plateaued, with no improvements since 2010.

Innovation and knowledge translation: Fundamental to the HCPCP's program theory is supporting innovative approaches to address enduring health care policy issues and facilitating the translation of this new knowledge to a pan-Canadian audience. The literature stressed the continued need for a focus on innovation and research in the health care system (CMA & CNA, 2011, p. 3; CMA, 2010, p. v; HCC, 2011). Successful innovations must be shared in order to have a broader impact, which means that the health care system needs to focus more on demonstrating the application of existing evidence and best practices (Saryeddine & Brimacombe, 2009, p. 26; Standing Senate Committee on Social Affairs, Science and Technology, 2012). Knowledge translation is not only important for sharing best practices but also for building an evidence base of barriers and enablers to knowledge translation. A synthesis of the knowledge transfer literature found a gap in understanding best practices in implementing knowledge transfer strategies for health policy: "there is actually very little evidence that can adequately inform what [knowledge transfer and exchange] strategies work and in what contexts" (Mitton, Adair, McKenzie, Patten, & Perry, 2007, p. 756). While it is necessary for the HCPCP to focus on creating useful tools via project-funded contribution agreements, it is only part of the process that contributes to improving the health care system. Disseminating knowledge of the tools and monitoring uptake is a needed part of the HCPCP process.

4.0 Findings - Performance

4.1 Program Implementation

In general, the evaluation found that the HCPCP made progress during the evaluation period in implementing its planned activities; for example, a new governance structure and centralized committee for strategic planning were put in place and a performance measurement tool was developed.

The evaluation found that projects had clear objectives and were generally implemented as planned. However, Health Canada did not establish clear expectations for all of its activities at the program-level. Instead, the Program focused mainly on its role as funder rather than information broker or facilitator. While work plans were developed for the HCPCP (Program Management Implementation Plan), the identified actions did not align with the logic model activity areas. In particular, expectations for Vote 1 activities in policy and research development and facilitation of collaborations was not developed, making an assessment of whether these activities have been implemented as planned difficult.

The following discussion considers the main activity areas of the Program as identified in the logic model: management and accountability; strategic planning; policy and research development; facilitation of collaborations; and funding projects.

4.1.1 Management and accountability

A major Program activity during the evaluation period involved instituting and refining a new governance structure. Prior to 2008, the HCPCP had a decentralized structure where components operated more or less independently. However, the HCPCP revised its governance structure in response to the 2007 evaluation, the renewal of the Program in 2008, and the Government of Canada Action Plan to Reform the Administration of G&C Programs. Taken together, these developments emphasized the need for more coordinated strategic planning, as well as the need for more transparent and accountable processes.

Three committees were formed: the Program Management Committee, the Operations Committee, and the Accountability Committee. The terms of reference for the new committees were approved in February 2008(Health Canada, 2008c), and the structure was effectively in place as of April 2008, although some committees did not meet until later in the year.

Program Management Committee (PMC)

The current members of the PMC include the Director General and managers from the HCPPD, the manager of CMIRPS, and the Director, Health Care System Division. Based on its terms of reference, the PMC's functions were wide-ranging and include:

• overseeing accountability activities (including performance measurement and evaluation);
• providing overall policy and Program direction (including identifying health care policy priorities and monitoring activities to provide future policy/Program directions);
• ensuring comprehensive risk management;
• supporting the development/dissemination of contribution information and achievements;
• ensuring coordination in contribution management (as a forum for sharing information); and
• providing "sound financial management of contributions", including decisions on reallocating contribution funds (Health Canada, 2009d, p. 2).

A review of PMC minutes and supporting materials, as well as a 2010 progress report, demonstrated activity in all of the above areas, although the level of activity varied (Health Canada, 2010d).

In terms of serving its centralizing function, the evaluation found that over the 2008-09 to 2012-13 time-period, the PMC served as a regular forum for coordinating contribution management and sharing information. During this time, the PMC was the primary body for financial management of the HCPCP, particularly for reallocating funds among components. Strategic planning occurred more sporadically for reasons discussed in more detail below, but there were several sessions in 2008-09, as well as a forward planning session in 2011 to help identify overall policy and program direction.

The PMC was less active in the area of supporting the development of dissemination of contribution-funded project information and achievements. In early years, the focus was on corporate reporting (e.g., Reports on Plans and Priorities, Departmental Performance Reports), and the annual report, which provided a summary of projects and results for the HHRS and IEHPI components of the HCPCP and which was posted on the Health Canada website. As of 2010-11, the Program's annual report was expanded to include HCSIF projects.

Health Canada key informants expressed some concern about whether the role of the PMC is being fulfilled, noting that the terms of reference may be too ambitious for one committee. Several key informants believe that the PMC is taking on a more operational role, particularly around the proposal review and approval process, and expressed a desire that the PMC be a more strategic body. Others raised questions about continued representation on the PMC of component areas that have sunset.

Operations Committee

According to its terms of reference, the Operations Committee has three main functions: financial management; program delivery; and communications (Health Canada, 2008e). The Operations Committee initially focused its efforts on program delivery and financial management, but activity occurred in all three areas.

The 2007 evaluation recommended that the Program develop guidelines for assessing project proposals and a solicitation package for proposals. In addressing this recommendation, the Operations Committee developed several standard tools, including the Guide for Applicants, a template for proposal review and assessment, and a contribution agreement template. In addition, the Operations Committee offered training on the use of the new Guide for Applicants. According to the Terms of Reference for the Committee and Health Canada key informants, two key Operations Committee activities appeared to remain largely unaddressed: conducting periodic reviews of project proposals to monitor quality and relevance, and identifying the need for and recommending of contribution management training. As the PMC has the same language with respect to conducting periodic reviews of project proposals in its terms of reference, this is a clear area of overlap that should be considered in a future review of the governance structure.

Based on the available information, the Operations Committee has been less involved in communications. It has ensured that knowledge transfer and dissemination activities were included in the Guide for Applicants, and established the HCPCP website. The HCPCP website has become relatively dormant, as appears to be considered a site for funding opportunities rather than for sharing knowledge from funded projects.

Based on documents and key informant interviews, the Operations Committee became less active over time. While some major activities, such as development of the Guide for Applicants, were accomplished, there were other areas within the Committee's terms of reference, such as communications, that were either not well-defined or had not had substantial activity. In each of the fiscal years 2010-11 and 2011-12, formal meetings of the Committee had only occurred two or three times. Instead, it was reported that ad hoc meetings occurred as needed. As a result, there was little detailed information on this Committee's recent activities.

Accountability Committee

The Accountability Committee's terms of reference included coordination and communication; performance measurement and evaluation; risk identification and management activities; and promotion of a policy/program integration model. The policy/program integration model was intended to use program evaluation findings to inform the development of policy and to identify strategic priorities (Health Canada, 2008f).

Based on Program documentation, the Committee's primary focus had been on performance measurement and risk management activities. For risk management, the work plans and minutes of the Accountability Committee documented the work undertaken to support the accountability activities under the Health Portfolio Action Plan, which included implementing the Enterprise Risk Management Agreement/Recipient Risk Assessment Tool. The effectiveness of that work was confirmed during the 2011 Health Canada audit of the management control framework for contributions programs, which concluded that "sound practices and tools are in place to help ensure monitoring decisions are based on recipient risk levels" (Health Canada, 2011e, p. 14). The Accountability Committee also oversaw eight recipient audits between 2008-09 and 2011-12, which exceeded the stated goal in the work plan of one audit per year.^{Footnote 7}

The evaluation found that the Accountability Committee made progress with respect to developing the RRET which collects performance measurement information from projects in a common way. This is discussed in more detail in the following section.

4.1.2 Performance reporting

The 2007 evaluation of the HCPCP recommended that the Program improve performance measurement data and monitoring. The recommendations included developing a systematic approach to collecting performance information. The Program developed the RRET to collect performance information (e.g., activities, outputs and outcomes) from the funded recipients. Two main factors have limited the evaluation's ability to use the performance information collected through the RRET. First, the HCPCP used an approach to roll out the RRET (described below) that meant that most ongoing projects during the evaluation were not using the RRET. Second, while initial plans were for an HCPCP-specific automated performance measurement reporting system, this work was overtaken by the development and implementation of a reporting system at the Branch level, which was not yet available during the evaluation period. As a result, more than five years later, the Program did not have comprehensive and consistent performance information from all of its funded projects, nor did it have a mechanism for collecting and reporting program-level (Vote 1) activities, such as strategic planning, facilitating collaborations, and policy and research development and implementation.

Project-level reporting

Efforts in improving performance reporting focused on the development of the RRET in order to determine if funded projects were achieving measureable results and meaningful health system outcomes. However, RRET participation was only mandatory for new contribution agreements after 2009, with the exception of provinces and territories, for which it is optional. As a result, the RRET contained information on 33 projects out of the 163 that were funded over the period that this evaluation was examining. The usefulness of the RRET for program management and evaluation purposes was limited because the Program chose not to require all funded projects to use it. Due to the focus on the RRET and its phased roll-out, there was limited performance information on HCPCP outputs and outcomes for projects funded between 2008-09 and 2012-13. The lack of comprehensive and robust performance data appears to have affected the Program's ability to apply the policy/program integration model, which was intended to use program evaluation outcomes to inform the development of policy and the identification of strategic priorities (Health Canada, 2008f). Program documentation did not provide evidence that project outcomes were informing the development of policy and identification of strategic priorities (discussed more below under "Strategic planning").

The evaluation sought feedback on the RRET from Health Canada key informants, as well as from funding recipients through the survey. Both lines of evidence indicated that stakeholders generally consider the RRET to be a positive step toward better performance measurement, although areas for improvement were noted (e.g. more focus on outcomes and increased emphasis on alignment with HCPCP objectives)

The RRET's dual purpose of assisting recipients with reporting on their projects' performance and supporting program-wide performance measurement meant that a key objective of the RRET was to link project outcomes with Program objectives. However, there is the perception that the RRET does not yet achieve this objective. Health Canada key informants commented that the funding recipients are not seeing the link between the outcomes identified in their projects' performance plans with the broader Program outcomes captured in the RRET. In some cases, recipients have difficulty understanding the linkage between the RRET and the performance measurement requirements for their project.

Health Canada key informants suggested that the RRET needs to be supported through training for both funding recipients and Program staff on performance measurement. The training would enhance Program staff's ability to assist funding recipients with performance reporting, as well as improve their ability to assess the adequacy of the information provided. Key informants expressed concern that without better training of funding recipients and monitoring by Program staff, the RRET may result in a laundry list of activities/outputs and not capture the value of these activities and what has been achieved.

Program-level reporting

Very little performance measurement information was collected for Health Canada's Vote 1 program-level activities. Even though the HCPCP's logic model identifies program-level activities, outputs, and outcomes, the Program does not track activities, outputs, and outcomes at this level. Program documents do not define performance indicators that are linked directly to the program-level activities identified in the logic model. Health Canada key informants acknowledged that expectations of the program-level contribution to performance reporting are not clear. Moreover, the ability to track and monitor program-level activities, outputs, and outcomes is compromised by the merging of program-level funds into the broader Directorate-level activity and funding.

4.1.3 Strategic planning, including priority/direction setting and identifying needs and gaps

According to its terms and conditions, the HCPCP is intended to be a national program that uses strategic and evidence-based decision making. At the same time, it is intended to be a program that can respond to emerging health care system issues quickly (Health Canada, 2008b). Striking a balance between strategic planning and flexibility has been an issue for the Program over the years.

The 2007 evaluation found coordination lacking at the national level and recommended that the HCPCP improve its strategic planning. The suggested types of activities included giving consideration to the level of funding to be allocated to specific priorities; conducting periodic reviews of the projects being funded to ensure that they align with program and component objectives; and reviewing the status of knowledge translation within the priority areas (Health Canada, 2008b). The Program agreed with these recommendations and incorporated them into its Management Action Plan and its 2009 Program Management Implementation Plan (PMIP) (Health Canada, 2008g, 2009e). The new governance structure with a central body (the PMC) to conduct strategic planning was a key component of enhancing the coordination and management of the Program.

In accordance with the PMIP, the HCPCP began to take a more structured and deliberate role in strategic planning, with seven PMC priority-setting and planning sessions taking place between December 2008 and April 2009. These sessions included presentations by components on potential future directions and funding priorities, opportunities for collaboration within the Program, and proposed solicitation processes (Health Canada, 2010b). The sessions did not include some of the activities envisioned by the evaluation recommendations, such as a systematic review of funded projects to align them with Program priorities and identify gaps, or to assess whether/how to build on the evidence base created. The data from the RRET were intended to contribute to this process. The sessions did not review the status of knowledge translation within priority areas. The sessions did, however, speak generally to achievements and potential future directions, and included work on developing criteria for determining funding priorities, which the PMC encouraged components to use, in developing their calls for proposals.

Despite these early efforts at strategic planning, the evaluation found that the Program continued to face challenges in balancing a structured planning process with the need to respond to emerging policy priorities. In particular, the 2011 announcement of the Family Medical Residencies Initiative (FMRI) involved a commitment of $39.5 million over approximately four years. According to some key informants, as early as 2008-09, the HCPCP began to hold funds aside for this potential commitment, although the first mention by the PMC of the need to reserve funds for the FMRI came in 2009 and referred to the 2010-11 and 2011-12 FYs. Several key informants noted that projects like the FMRI, which consume a large proportion of resources, restrict flexibility elsewhere in the Program by limiting funds available for new projects, and thereby affect the ability to conduct broader strategic planning.

After the financial implications of the FMRI were clear, the PMC reinvigorated its strategic planning with a "forward planning" process in early 2011. In addition, the Program began making a more concerted and systematic effort to develop funding priorities and to link them to the Program logic model. The PMC began using a template for assessing proposals that aligned the proposals directly with the components' strategic priorities and provided information on policy relevance and outcomes, sustainability, and stakeholder/province/territory reaction (i.e., opportunities for collaboration or synergies with other initiatives). In 2012 discussions of funding priorities for HCSIF, the PMC used a checklist that tied the exercise to the Program logic model to identify policy priorities (e.g., Are the policy priorities based on evidence? Do they address identified gaps? Can outcomes be well-defined? Is there strong likelihood that the knowledge will have broad application and that the work will have a demonstrable impact on the health care system?). According to key informants, these activities, although sometimes contentious, did result in more objective decision making.

In general, the flexibility of the HCPCP was considered an asset by most key informants. The Program's flexibility allowed the HCPCP to respond to shifting government priorities and accommodate projects like the FMRI. That flexibility, however, could also cause the Program to become reactive to operational challenges and less strategic. According to key informants, the HCPCP's focus shifted from setting funding priorities by assessing gaps in knowledge and identifying areas of greatest potential for effecting change to managing potential funding lapses and identifying projects quickly to reduce lapsing. As a result, key informants noted that the coherence of the Program may have been affected when it did not engage in more proactive contingency planning.

4.1.4 Policy/research development and implementation (including synthesizing findings and identifying promising practices) and facilitation of collaboration

In addition to strategic planning and management and accountability, Health Canada was supposed to engage in collaboration and facilitation, policy and research development and implementation activities. Some work occurred directly in these program areas. However, the available evidence found that this was limited to annual or synthesis reports and funding meetings or conferences. Also, Vote 1 funding was reallocated throughout the Directorate and as it was not tracked and reported, it is difficult to determine whether this funding was used for HCPCP activities.

As identified in the logic model, it was expected that the Program would contribute to the above mentioned activities independent of funding projects. However, there was very little in the way of Program data to demonstrate Vote 1 funded activities. The Program did not track its own activities and outputs and in some instances staff did not see how their efforts were intended to support these activities at the program level or build on the efforts of the funded projects. Part of the reason was that the Program found limited ability to engage in this work except through contribution funding. According to Health Canada key informants, the use of Vote 1 (Operations and Maintenance) funds shifted during the period under evaluation, with a larger proportion of it covering salaries. As a result, there was less funding available for conducting research, policy development, knowledge synthesis, and facilitating collaboration at the program level.

In addition, because HCPCP Vote 1 funds were a major source of funding for the entire Directorate, this made it difficult to identify how that funding contributed directly to HCPCP activities, outputs, and outcomes as distinct from the overall Directorate work. However, Health Canada key informants noted that, while the Directorate's work was not always, strictly speaking, conducted for the HCPCP, it could contribute to it indirectly.

At the program level, several activities under policy/research development and implementation were undertaken. The following provide some examples of the outputs of these activities.

• The Health Human Resources Strategy Projects: Policy Synthesis Report (2011) included a review of 20 projects and identified key outputs/outcomes for each strategic direction of the component (Health Canada, 2011f).
• The Patient Wait Times Guarantee Pilot Project Fund Summative Evaluation (2011) reviewed documents and gathered information from the project leads on activities and outcomes for each of the 11 funded projects under the PWTGPPF (Health Canada, 2011b). The PWTGPPF evaluation was distributed to project leads to share the results of the other PWTGPPF projects. It was also made available to the public upon request (via the Health Canada website).
• The Synthesis of Results of the National Wait Times Initiative (2009) reviewed all 33 projects funded over the life of the NWTI and presented results of projects by 10 key themes (PWGSC, 2009). According to Health Canada key informants, the NWTI synthesis report was disseminated within Health Canada and to stakeholder groups, such as the CIHI. An overview was also made available on the Health Canada website.
• Annual reports describing projects funded by HHRS and IEHPI and listing their anticipated results are posted online and provide an opportunity for interested stakeholders to review the funded projects and determine if there are opportunities for synergy or for learning best practices. Contact information for the projects is provided (Health Canada, 2010e, 2011g).
• IEHPI work on the Foreign Qualifications Recognition (FQR) was supported by Vote 1 monies. This work includes participating on task teams, sub-committees, and working groups related to the Pan-Canadian Framework for the FQR, whose objectives have been mapped to IEHPI objectives. In addition, because all of the IEHPI Unit's work was tied closely to the HCPCP, key informants consider Vote 1 funds used for professional development and consultations with provincial and territorial recipients to support policy and research development.

Health Canada key informants identified several examples of the HCPCP contribution to facilitating collaborations. Examples included ongoing collaboration on curriculum and training with the Association of Faculties of Medicine of Canada and with the Canadian Medical Association for the Taming of the Queue conferences.

• In 2008, the IEHPI organized an Internationally Educated Nurses Meeting to discuss harmonization of approaches to registration of nurses and ideas on how the IEHPI could address provincial/territorial needs. The attendees at the session included Chief Nursing Officers for each province and representatives of key nursing stakeholders from across Canada, such as nurse regulator groups.
• Two federal/provincial/territorial consultations occurred in 2010 to discuss funding priorities and project ideas in a roundtable format. The consultations provided jurisdictions with an opportunity to consider how they might collaborate, as well as share insights on common challenges and how to address them (Health Canada, 2010f).
• Vote 1 funds were used to support IEHPI annual meetings with provincial and territorial partners and meetings of the Federal/Provincial/Territorial Advisory Committee on Health Delivery and Human Resources(ACHDHR).
• Vote 1 funds were also used to host a two-day in-person meeting for PWTGPPF projects.
• Key informants mentioned that there have been many stakeholder meetings funded by Vote 1 for policy development, knowledge translation, and other activities. A specific example given was the 2011 Collaborative Learning Environment Roundtable, a one-day session with projects involved in interprofessional collaboration so that they could share experiences and results.

The evaluation found the Program contribution in these activities to be an area for possible future improvement. While Health Canada key informants gave examples of activities or outputs, performance information, such as how the outputs were used and contributed to HCPCP outcomes, was not tracked. Some stakeholders noted that Operations and Maintenance monies have not been used to generate new knowledge or to understand what the HCPCP priorities should be. They noted that while some Vote 1 funds have been used to produce synthesis reports, in many instances, these reports, as well as the findings from project reports, have not been well-circulated or used for policy and research development. Building that link between project results and broader policy and research development was considered a weakness by some key informants. These issues point to the need for Health Canada to conceptualize its contribution to the health system beyond funding and managing projects, producing products and organizing meetings.

4.1.5 Funding projects

This section considers whether the program-level activities for funding projects were implemented as planned and whether funded projects were, themselves, implemented as planned. The main program-level activities for funding projects involve conducting calls for proposals and selecting projects to fund, which are discussed below.

Calls for proposals

All of the calls for proposals were targeted calls, although the Conferences and Events call was considered broad-based by some key informants, because it was sent to all organizations in the components' lists of stakeholders.^{Footnote 8} The call was not promoted on the HCPCP website, as the website was not yet available. Based on interviews, there appears to have been a division within the HCPCP between those who prefer open, or broad-based, calls for transparency reasons and others who prefer the targeted approach due to capacity issues both within Health Canada (to manage open calls) and the proponent organization (to conduct the work).

That being said, there was a strong desire that the method of solicitation be strategic and not just expedient. This concern was noted particularly for unsolicited proposals. While unsolicited proposals must still be reviewed and are required to meet funding criteria for the component and the HCPCP, Health Canada key informants pointed out that organizations that are more knowledgeable about the HCPCP may be more likely to submit an unsolicited proposal. Consequently, this approach may not result in funding the most innovative or promising idea. The Program did not track the number of proposals by method of funding (broad-based, targeted, unsolicited), but over one quarter (28%, or n=27) of survey respondents indicated that they had submitted an unsolicited proposal.

Overall, during the evaluation period, the HCPCP conducted three calls for proposals and funded 35 new projects (see Table 3).

Table 3: Calls for proposals by HCPCP components, 2008-09 to 2011-12

Program component	Broad-based	Targeted	Projects approved
HHRS		No calls for 2009 but a call was issued in 2010-2011	8 funded projects in 2011-2012 resulted from the 2010-2011 call for proposals
IEHPI		2010: Call for proposals for new provincial and territorial agreements that span 2011-12 to 2015-16	13 funded projects
HCSIF		2009-10: for Conference and Events	23 proposals/14 funded projects
		2009-10: HCPCP Thematic Approach (funding could be for FY 2009-10 and/or 2010-11)	The call for proposals was deferred for future consideration
NWTI	N/A - no new funding allocated
PWTGPPF
Source: Health Canada internal documents and interviews

Alignment of projects with Program and component objectives

A content analysis was done of project-level documentation to determine if those funded projects' activities, outputs, and outcomes aligned with those of the Program. According to the findings of the project-level document review and survey, funded projects aligned with Program and component objectives. However, the Program objective most directly related to the knowledge gained by identifying barriers and enablers to health care reform was addressed by a minority of the contribution agreements: "increase knowledge of factors determining the performance and responsiveness of the health care system and its responsiveness to users' needs" (40% of survey respondents and 33% of project files reviewed aligned the project with this objective). See Table 4 for complete results.

Table 4: Alignment with HCPCP objectives

Objectives	Total survey respondents (n=95)		Total project files reviewed (n=45)
Identify, assess, and promote new approaches, models, and best practices that respond to identified health care system priorities	72	76%	27	60%
Contribute to improvements in the accessibility, responsiveness, quality, sustainability, and accountability of the health care system	64	67%	27	60%
Increase knowledge and application of evidence and best practices, leading to improved health care system planning and performance	60	63%	27	60%
Increase collaboration on, and coordination of, responses to health care system priorities amongst federal, provincial, and territorial governments, other health care policy-makers, service providers, users, researchers, and other stakeholders	52	55%	26	58%
Foster the development and implementation of health care system policies and strategies to address identified health care system needs	46	48%	30	67%
Increase knowledge of factors determining the performance and responsiveness of the health care system and its responsiveness to users' needs	38	40%	15	33%
None	3	3%	--	--
Note: Multiple responses allowed; column sums to more than 100%. Sources: Survey with contribution agreement primary contacts and project-level document review.

Based on project-level document review and survey results, funded projects aligned with component objectives. While almost all component objectives are addressed by at least one project, there is clustering of projects around certain objectives. For example, under HHRS, more projects address using human resource skills effectively, and fewer projects address more health care providers, although the projects related to the supply of health care providers tended to have larger budgets.

Project implementation

The survey and content analysis of contribution-funded projects also focused on understanding project implementation.

Activities: The project-level document review and survey results indicated that almost all project activities were implemented as planned but with delays for many in completion. For instance, 69% of the document review projects and 40% of the survey respondents had amendments to their contribution agreements to change the timetable of the project.The most common reasons for changes to the projects indicated in the survey were project delays, a determination that the original approach was not feasible, and a lack of capacity among stakeholders. Previous project reporting has not captured this information, as the project files reviewed did not provide much information on the reasons for delayed or uncompleted project activities. Gathering the reasons for difficulties in implementing the projects should be an important part of project selection, management and reporting. If properly completed, the RRET will be collecting this information.

Based on the project-level document review results, projects appear to have more difficulties in completing later stage activities such as knowledge transfer, evaluation, and final reporting, where about half of the activities planned were completed, compared to two thirds of other planned activities (project management, development of project content, implementation, dissemination).

Outputs: The HCPCP expects projects to produce at least one of the following three kinds of outputs: knowledge products; collaborative working arrangements; and identification of barriers and enablers of knowledge development, translation, and use of health care system reform. The evaluation evidence (project-level document review and survey with funding recipients) found substantial success in the development of knowledge products and collaborative work arrangements. Almost all projects had developed knowledge products (about 95% for both lines of evidence). Based on survey results (88%) and the review of project files (82%), most projects reported collaborative working arrangements. However, the evaluation findings are less clear on the success in the third output ¾ the identification of barriers and enablers. Understanding barriers and developing strategies to address them through enablers are critical factors in any knowledge translation strategy. Just over three quarters of survey respondents indicated that their project was successful in identifying barriers and enablers. In comparison, no barriers were identified in 40% of the project files reviewed, and very few projects identified enablers. A cursory review of RRET data found that projects often confused health system barriers (e.g. access to health care) with project activity barriers (e.g. hiring staff, timely receipt of funding). The inconclusiveness of the findings for the identification of barriers and enablers may indicate that: reporting on this output does not reflect what has been accomplished; some projects have not focused on this particular output; some projects are unclear about the meaning of the term; or some projects have not progressed to the point where this type of identification has occurred.

4.2 Effectiveness

This section is organized primarily at the project level around the outcomes identified in the logic model. In order to maximize the ability to obtain outcome information and cover a larger proportion of the HCPCP funding, the project-level document review focused on larger, completed projects, however, none of these projects had used the RRET. The evaluation collected performance information through the survey with contribution agreement primary contacts and Health Canada key informant interviews. In addition, various reports produced for the HCPCP were reviewed, including the PWTGPPF Summative Evaluation Final Report (Health Canada, 2011b), the Synthesis of Results to the NWTI Final Report (PWGSC, 2009), the HHRS Projects Policy Synthesis Report (Health Canada, 2011f), and the Knowledge Translation Review Final Report (Moore et al., 2012).

Several outcomes only had limited performance information in support of the evaluation. Health Canada key informants acknowledged that the collection of performance data to demonstrate the achievement of outcomes has been a weakness for the HCPCP. This was also identified in the 2007 evaluation. The project-level document review results, presented in the following sections, confirmed this opinion, with many projects not identifying HCPCP outcomes and therefore failing to report on them. In addition, the ability of projects to demonstrate outcomes rests, in part, on sound theory or logic where activities produce specified outputs, which logically will result in identified outcomes. While approximately half the files reviewed (n=23) included clear linkages between project activities and outcomes, this meant that the other half did not have a theory that explained how the planned activities were to result in the expected outcomes (see Table 5). The review identified confusion in reporting on activities and outcomes. Although the project work plans and progress reports included an "outcomes" column, this column was used more to describe the importance of the planned activity than project outcomes. A final observation, which is also mentioned in Section 4.1, is that the HCPCP is not collecting performance data on its program-level activities. Therefore, the following discussion about the achievement of outcomes focuses on the contribution of funded projects.

Table 5: Program logic in planning documents

Link between project activities and outcomes (n=45)			Link not established (n=21)
Yes	23	51%	Expected outcomes not defined in planning documents	15	71%
No	21	47%	Outcomes were outputs	7	33%
No information	1	2%	Activities were linked to outputs only	7	33%
			No linkage of activities to outputs/outcomes	1	5%
			Outcomes not defined until evaluation	1	5%
			Other	3	14%
			No information/no planning documents	2	10%
Note: Multiple response; column sums to more than 100%.

4.2.1 Achievement of Expected Outcomes

To what extent have the immediate outcomes been achieved?

Immediate Outcome #1: Increased awareness and understanding of knowledge tools, products, approaches, models, innovations, health care system performance, and health care system reform issues

The evidence to demonstrate progress on increasing awareness and understanding as a results of the products that projects produced was not strong. While project recipients believed they were effective in increasing awareness, there were no other lines of evidence to support this view.

The efforts of the funded projects were expected to lead to changes in awareness and understanding at two levels: among funded recipient and stakeholders directly involved in the projects through participating in research studies, consultations, and other methods used to inform the development of knowledge products; and 2) among the larger target audience through the dissemination of the knowledge created by the project. Projects should be able to report on results on at least the first level. In addition, as this outcome represents an early step in knowledge translation, most projects should be able to identify this as an outcome.

The success of the HCPCP in achieving this outcome was difficult to determine and was likely underreported. Because progress reports used at the time did not require projects to align their outcomes with the HCPCP's, this potential outcome may have been simply overlooked by projects. Less than half (44%, or n=20) of the project files identified increased awareness and understanding of knowledge products, tools, or innovations as an outcome. In contrast, over three quarters (78%) of survey respondents believe that the projects within their contribution agreement have been effective or very effective in increasing awareness of these outputs (Table 6). The evaluation of the PWTGPPF supported the survey results, as it found through interviews, that project stakeholders believed the project had increased their understanding of the knowledge tools and approaches that could address patient wait times, as well as operational issues that affected wait times (Health Canada, 2011b, pp. 16-17).

Table 6: Increasing awareness

How effective is/was the project(s) in increasing awareness of its knowledge products, tools, or innovations?
	Total respondents (n=95)
Very effective	32	34%
Effective	42	44%
Not effective	1	1%
Too early to tell	10	11%
Not applicable to project	6	6%
Don't know	4	4%

One reason for the seemingly inconsistent results between the survey and the project-level document review points to a future challenge for the Program in obtaining robust performance information. Based on project reporting in the files reviewed, many projects that identified increased awareness and understanding as an outcome ended shortly after the knowledge products were either developed or implemented. This meant that only the evidence of initial dissemination was included in project reporting or evaluations. These projects essentially presumed that their dissemination activities would result in increased awareness and understanding of knowledge tools and products. For example, posting information or tools on websites and presenting research findings and tools at conferences (examples of outputs) were offered as evidence of increased awareness.

Ideally, projects would be able to demonstrate increased awareness and understanding through pre-post surveys of stakeholders, but none of the projects in the project-level document review conducted this type of study. Some projects, however, were able to provide evidence through surveys with participants after exposure to the knowledge products, or the projects inferred increased awareness or understanding through increased use of knowledge products. In all of these instances, projects showed some degree of increased awareness and/or understanding, which aligns with the survey and PWTGPPF evaluation results.

Immediate Outcome #2: Improved collaboration and coordination among health care system stakeholders to minimize identified barriers and utilize identified enablers

Evidence suggests that the many of the projects engaged in activities that focused on collaboration and coordination. However, the evaluation could not determine if these activities resulted in minimizing barriers and identifying enablers.

This outcome will be considered in two parts: 1) how effective projects were in improving collaboration and coordination; and 2) how effective they were in minimizing barriers and addressing enablers. This approach was taken because projects did not necessarily link collaboration solely to addressing barriers and using enablers.

Interjurisdictional collaboration was considered a desired, but not essential, criterion in evaluating proposals, which may explain why the project-level document review had few projects (n=15) that identified outcomes specifically linked to improving collaboration. However, most projects have collaborative working relationships (82% in the project-level document review and 91% of survey respondents). As a result, the opportunity certainly existed for projects to identify outcomes related to these collaborations and report on whether the collaborations resulted in improvements. The HCPCP appears to have missed an opportunity to encourage projects that clearly had a collaborative component to identify improvements in collaboration and coordination as an outcome.

The evaluation can infer some level of improvement that goes beyond the 15 projects that identified improved collaboration and coordination as an outcome from the fact that 35 projects had new collaborative arrangements. In addition, these new collaborative arrangements involved organizations with varying geographic scope, including local (n=10), provincial/territorial (n=19), and pan-Canadian/national (n=18). These results indicate that projects built new relationships with a variety of stakeholders that have the potential to effect broader change.

The most common roles in the collaborative arrangements were providing advice (57%), assisting with dissemination (40%) and/or research (37%), providing access to the policy process (26%), and participating in the development of products (24%). In addition, for most of the projects for which information was available, the collaborative arrangements were maintained throughout the project's life, and for one third of completed projects, the collaborations were expected to continue once the HCPCP funding ended.

The success in establishing collaborative relationships was confirmed in the survey. Eighty-two percent of respondents reported that the projects within the contribution agreement have been effective or very effective in establishing collaborative working relationships. In addition, the PWTGPPF evaluation found evidence of successful interjurisdictional collaboration:

• Prince Edward Island and New Brunswick developed a memorandum of understanding for recourse related to radiation therapy among the four Atlantic provinces;
• Yukon collaborated with British Columbia authorities in the development of protocols for recourse to facilities in British Columbia;
• Manitoba shared its findings within and across jurisdictions, and stakeholders reported that collaboration within jurisdictions increased as a result of the projects. (Health Canada, 2011b, p. 18)

Health Canada key informants pointed to several examples of successful collaboration for the other components:

• The IEHPI has funded two regional forums: the Western and Northern Forum (Manitoba, Saskatchewan, Alberta, British Columbia, Yukon, Northwest Territories, and Nunavut) and the Atlantic Connection (Nova Scotia, New Brunswick, and Prince Edward Island).
• The Western and Northern Forum pooled 20% of their IEHPI funding for collaborative work.
• The Atlantic Connection pooled resources for collaborative projects.
• HHRS funded the Pan-Canadian Health Human Resource Network to host national and regional consultations with health human resources researchers and decision-makers to inform the development of an interactive website and range of research, knowledge exchange, and tools for decision making and implementation.
• NWTI contribution funds supported broader dissemination through the Taming of the Queue conference, which facilitates information sharing across jurisdictions.

The evaluation found limited evidence that projects were successful in addressing barriers and utilizing enablers. In the project-level document review, 58% (n=26) of projects identified a total of 66 different barriers. Only seven projects identified enablers. For both, there was limited information on the strategies the projects undertook to respond to the barriers or enablers, or whether they were successful.^{Footnote 9} The documentation in the project files was unclear on whether efforts to address the barriers were successful for 61% or 40 of the 66 barriers identified. For the 26 barriers for which information was available, the strategies were at least partially successful. Project files did not document the success in efforts to utilize enablers. In contrast to this, most survey respondents consider their project(s) to be effective or very effective in addressing barriers (68%) and enablers (66%). See Table 7.

Table 7: Addressing barriers and utilizing enablers

How effective is/was the project(s) in addressing barriers/utilizing enablers to knowledge development, translation, and use?
	Total respondents: Barriers (n=95)		Total respondents: Enablers (n=95)
Very effective	23	24%	19	20%
Effective	42	44%	44	46%
Not effective	5	5%	4	4%
Too early to tell	11	12%	11	12%
Not applicable to project	8	8%	8	8%
Don't know	6	6%	9	10%
Note: Some columns do not sum to 100% due to rounding. Source: Survey with contribution agreement primary contacts.

As noted in Section 4.1, the lack of information on barriers and enablers is a missed opportunity for the Program, as learning about how to minimize barriers and utilize enablers is intended to be a key feature of the HCPCP. The evaluation also found that the Program had not collected information, to support knowledge development and transfer related to the identification of barriers and successful strategies to address them. This was also identified as an issue in the report on knowledge translation activities within the HCPCP, which found that few projects assessed barriers or reported on adaptations adopted to address them (Moore et al., 2012, p. 15).

Immediate Outcome #3: Successful and/or increased use of knowledge, approaches, models, strategies, promising practices

The data found that projects were successful in fulfilling the obligations under their contribution agreements ( i.e. the development of a product ) and there was some project-level survey data that demonstrated that these products  were successful in increasing the use of these new approaches, models and promising practices.

According to the program theory, success at the project level would be seen when there is uptake of a knowledge product by, for instance, a province or territory and/or an increased use among project partners. The intermediate and long-term outcomes consider the broader application and impacts of the uptake of projects' products. Given that successful and/or increased uptake is essentially a catch-all outcome, every project should be able to provide evidence of small-scale, local success/impact. However, 26 of the 45 projects in the project-level document review had a related outcome, leaving over one third of projects reviewed (n=19) without any performance information directly related to this outcome. Evidence of achievement of the outcome was weak in the project files reviewed. For the 69 unique outcomes that aligned with the successful and/or increased use of knowledge products, tools, or innovations, available documentation did not support an assessment of success for 20 (or 29%) of the outcomes. For about half of these outcomes, however, there was at least partial success.

Again, project reporting may be the issue, as, based on survey responses, most projects appear to be able to report on this outcome. Only 4% of survey respondents indicated that the outcome was not applicable to their project. In fact, most (75%) believe that the projects within the contribution agreement have been completely or partially successful in achieving the use of knowledge products by the original project participants. See Table 8.

Table 8: Use of knowledge products

How successful is/was the project(s) in achieving the use of its knowledge products, tools, or innovations by the original project participants?
	Total respondents (n=95)
Completely successful	38	40%
Partially successful	33	35%
Not successful	0	--
Too early to tell	15	16%
Not applicable to project	4	4%
Don't know	5	5%
Source: Survey with contribution agreement primary contacts.

Survey results also indicated that projects could provide a assessment of success, with about half of respondents reporting that they obtain baseline information on use (52%) or collect ongoing performance data (49%). Over one third of respondents reported conducting a pre-post analysis (38%) or having a defined strategy to assess the impact of the project at its targeted scope (37%). Given these responses, project reports that were reviewed for the project-level document review should have provided more robust performance data. Most evidence found in project reports for this outcome was based on formal or informal feedback. Some projects conducted surveys asking if participants either had used or would use the knowledge products, and most responded affirmatively. Other projects reported on informal positive feedback they had received on their knowledge products. As with the other outcomes, some projects reported on activities as evidence of outcomes based on the presumption that dissemination of materials or participation in workshops or conferences was evidence of increased use of knowledge.

Projects may not be able to report on this outcome because many contribution agreements concluded with the development of knowledge products or early dissemination. As a result, it may be too early to assess whether the knowledge products led to changes in health care delivery, policies, or practices by the project participants. Health Canada key informants noted this as an issue for performance measurement, and survey results seem to confirm this view. When project results move from use of knowledge products to changes in health care delivery, policies, or practices by project participants, the percentage of respondents who consider it too early to assess success rises from 16% to 39% (compare Table 10 and Table 11). While 42% of respondents reported at least some success, just over one tenth of respondents did not believe the project(s) in their contribution agreement were attempting to achieve changes in health care delivery, policies, or practices (see Table 9). This lack of identification with an intended outcome for the Program, reported by about 10% of contribution agreement primary contacts who responded to the survey, should be a concern for the HCPCP.

Table 9: Changes in health care delivery, policies, or practices

How successful is/was the project(s) in achieving changes in health care delivery, policies, or practices as a result of project knowledge products, tools, or innovations by the original project participants?
	Total respondents (n=95)
Completely successful	11	12%
Partially successful	29	31%
Not successful	0	--
Too early to tell	37	39%
Not applicable to project	10	11%
Don't know	8	8%
Note: Column does not sum to 100% due to rounding. Source: Survey with contribution agreement primary contacts.

4.2.2 To what extent has the intermediate outcome been achieved?

Intermediate Outcome: Broader adoption of knowledge/innovations resulting in changes in policy and/or practice

While some projects were able to demonstrate success on this outcome, a number of projects felt that it was too early for an intermediate outcome to be seen.

Increased awareness, improved collaborative relations, and increased use of knowledge are expected to lead to the intermediate outcome of broader adoption of the knowledge or innovations by relevant health care system stakeholders resulting in changes in policy and/or practice. This outcome goes beyond the immediate project participants and considers the adoption by the larger pan-Canadian target audience identified for the project. This outcome reflects the emphasis of the HCPCP on knowledge translation: the Program intends to fund specific projects that have the potential to lead to pan-Canadian system improvements, which can only occur if the knowledge generated at the project level is shared with stakeholders involved in the larger health care system. Broader adoption beyond the direct project participants is one step further in the knowledge translation cycle.

The issues discussed for the immediate outcomes regarding identifying appropriate outcomes, providing evidence of achievement, and encountering limitations in reporting on this outcome due to the nature of the project had even greater effects on the ability to demonstrate success for this intermediate outcome. The project-level document review found fewer projects (n=17) with outcomes aligned with the HCPCP's intermediate outcome of broader adoption of knowledge/innovations. In addition, for 30% of knowledge products identified in the project-level document review, the documentation did not provide evidence of uptake. For just over one quarter of knowledge products (26%, or n=37), the project had not progressed far enough to provide evidence of uptake or concluded with the development of the knowledge product. As a result, information on uptake was available for less than half of the knowledge products produced (45%, or n=65).

Many projects provided evidence of uptake via:

• surveys or interviews in which target audience stakeholders reported that they considered the knowledge products useful and/or their organization had used the knowledge products;
• the number of hits or downloads from websites; and
• registration in courses or workshops developed by projects.

Some projects focused on uptake by project participants. For example, projects that involved developing knowledge products for use within piloted sites reported on the uptake and use within the pilots; dissemination or uptake with a broader potential target audience was beyond the scope of the contribution agreement.

The project-level document review found that uptake or use of knowledge products is not well-monitored by projects. Knowledge dissemination was often relied upon as an indication of the use or uptake of knowledge products. These findings support the conclusion in the Knowledge Translation Review Final Report, which found that projects focused on dissemination and "few projects monitored knowledge use" (Moore et al., 2012, p. 3). This report noted, and the file review confirmed, that knowledge dissemination was often provided as an indication of the use or uptake of knowledge products.

In contrast, survey respondents indicated greater success than found in the review of project files; however, a greater proportion of survey respondents consider it either too early to be able to assess success on the intermediate outcome (as compared to the immediate outcomes) or consider this outcome inapplicable to their project. As shown in Table 10, 55% of respondents considered the project(s) in the contribution agreement to have been completely or partially successful in achieving use of the knowledge products by the broader target audience, while 26% reported it too early to tell. In comparison, 32% reported success in achieving changes in delivery, policies, or practices, and 41% considered it too early to tell (see Table 11).

Table 10: Broader adoption of knowledge products

How successful is/was the project(s) in achieving the use of its knowledge products, tools, or innovations by the broader target audience?
	Total respondents (n=95)
Completely successful	20	21%
Partially successful	32	34%
Not successful	1	1%
Too early to tell	25	26%
Not applicable to project	7	7%
Don't know	10	11%

Table 11: Changes in health care delivery, policies, or practices

How successful is/was the project(s) in achieving changes in health care delivery, policies, or practices as a result of project knowledge products, tools, or innovations by the broader target audience?
	Total respondents (n=95)
Completely successful	9	10%
Partially successful	21	22%
Not successful	1	1%
Too early to tell	39	41%
Not applicable to project	13	14%
Don't know	12	13%
Note: Column does not sum to 100% due to rounding. Source: Survey with contribution agreement primary contacts.

The evidence to demonstrate broader uptake was not strong. A number of projects considered the product and dissemination of the product as an outcome, as opposed to an output and activity, respectively. This may be due to the fact that many projects indicated that the completion of an activity or development of a product was an indicator of success. As a result, success at doing something and the impact of that activity were conflated. And as previously stated, many projects claimed that it was too early to report on their intermediate outcomes. This observation may point to a program design issue. Health Canada may be funding projects that cannot achieve meaningful results within the timelines of the contribution agreement. Either way, the overall achievement of the intermediate outcome could not be fully determined, or, in some cases, may not be occurring.

4.2.3 To what extent has the long-term outcome been achieved?

Longer-term Outcome: Improvements in health care system policies/practices and performance

Few projects stated they had achieved or partially achieved their long-term outcomes. Many projects surveyed felt that not enough time had elapsed to have the desired impact. Some components of the HCPCP however, were able to demonstrate outcomes at this level (i.e. broader adoption by stakeholders of a practice).

In the long term, the goal of the HCPCP was to effect improvements in health care system policies, practices, and performance in areas targeted by the HCPCP. The outcome was based on the premise that the development of knowledge products and their successful use by the project coupled with the adoption of the knowledge by the broader target audience would result in improvements to health care system policies and practices. Consequently, the program logic or theory should support the inference of achievement of long-term outcomes if the projects can demonstrate evidence of achievement of immediate and intermediate outcomes.

The priority areas on which the federal government was focusing its health care system improvement efforts were patient wait times, health human resources, and patient safety. Data from project-level documents and survey data with project participants were mainly used to assess this outcome. Fourteen projects in the project-level document review identified 21 unique outcomes that align with the HCPCP long-term outcome. Documentation of success was not available for 12 (57%) of these long-term outcomes. The survey results confirmed the project-level document review findings in the sense that evidence of achievement for the long-term outcome was not available for most projects. As might be expected, when asked about success in achieving the HCPCP's long-term outcome, many survey respondents (39%) believed that it was too early to tell whether the project(s) in their contribution agreement had been successful in improving health care system polices/practices. Just under one fifth (17%) of respondents did not think the long-term outcome was applicable to their contribution agreement, and another 15% could not provide a response. See Table 12.

Table 12: Improvements in health care system policies/practices

How successful is/was the project(s) in having an effect on the intended change in the health care system (e.g., reduced wait times, increased supply of targeted health care professionals, increased access to targeted types of health care)?
	Total respondents (n=95)
Completely successful	10	11%
Partially successful	17	18%
Not successful	1	1%
Too early to tell	37	39%
Not applicable to project	16	17%
Don't know	14	15%
Note: Column does not sum to 100% due to rounding. Source: Survey with contribution agreement primary contacts.

Internationally Educated Health Professionals Initiative: The project-level document review showed that IEHPI projects had experienced increased participation in programming; low attrition rates; and some positive results in certification, licensing examinations, and residency placements. Key informants also pointed out that the IEHPI has brought together regulatory bodies for different professions across the provinces and territories, so there was less variation in processes and the processes were more streamlined. For example, the Medical Council of Canada established a national assessment process for international medical graduates wishing to enter residency training.

Health Care System Innovation Fund: ISMP Canada received HCPCP funding for CMIRPS, a system that aimed to reduce and prevent harm from medication incidents by managing and sharing voluntarily-reported information about medication incidents. The main objective of the system was to monitor and identify prescription drug errors. Through CMIRPS, ISMP Canada received practitioner reports of medication incidents and then conducted root cause analyses and developed and disseminated information bulletins with recommendations for preventative strategies. Among its HCPCP-funded activities, ISMP Canada also conducted workshops on medication safety and developed a medication safety self-assessment program designed to assist health care organizations in evaluating the safety of their medication systems by identifying areas requiring improvement and developing strategies for systems enhancement.

Through its CMIRPS activities, ISMP Canada demonstrated several ways in which its activities had broader system effects:

• Its medication safety self-assessment program and tool achieved a high rate of adoption in Canadian hospitals.
• Safety bulletins that provided recommendations on how to address medication safety incidents were integrated into health care practice among many health care organizations surveyed.
• Pharmaceutical companies changed aspects of packaging, labelling, and/or naming in response to ISMP Canada information on medication incidents caused by confusion of products created by the visual aspects of the product.
• ISMP Canada influenced Accreditation Canada Required Organizational Practices in areas related to medication safety.

National Wait Times Initiative: The Hip and Knee Surgery Wait Times Strategy Phase III created a tool kit and a knowledge translation network to manage wait times. A report posted by the Government of Newfoundland and Labrador indicates that a key recommendation in the tool kit -- to have a centralized intake system -- has reportedly been used successfully in six provinces (Newfoundland and Labrador, 2012, p. 8). This project appeared to benefit from the ongoing support of a national organization (Bone and Joint Canada) and the success in developing a knowledge translation network to support the use and continued development of the knowledge tool beyond HCPCP funding.

Patient Wait Times Guarantee Pilot Project Fund: The previous internal evaluation of the PWTGPPF found that a few of the pilot projects could clearly demonstrate reductions in wait times. In particular, a process instituted in Saskatchewan to identify patients who were reaching their maximum wait time for coronary bypass surgery resulted in a reduction in wait times over the course of the pilot project. Similarly, a pilot project in the Yukon significantly reduced cataract wait times. The introduction of a new mammography unit in the Northwest Territories project improved access for Hay River residents and resulted in the attainment of the project goal of over 70% of eligible patients being screened in 36 months. This goal was met within only 18 months of operation under the project. As noted in the evaluation, project evaluation reports had limited information on the ability of projects to reduce wait times.

4.2.4 Addressing challenges in demonstrating the achievement of outcomes

Some features of the HCPCP make it challenging to determine/achieve long-term outcomes. For instance, there was a lack of consistency amongst HC key informants on the overall program theory, on the intended scale and scope of projects (pan-Canadian vs jurisdictional) and on the role that HC staff should play with respect to knowledge translation and project monitoring.

Building a common understanding of Program outcomes. To demonstrate achievement of its intermediate and long-term outcomes, the HCPCP must show effective knowledge translation that has led to large-scale pan-Canadian impacts. This section considers both outcomes, as the evaluation found some tension within the Program about how these apply to the HCPCP. In addition, the evaluation evidence indicates that there is room for improvement in supporting knowledge translation and, ultimately, the ability to achieve pan-Canadian impacts.

Based on Program documents, the theory behind the HCPCP was to fund projects to develop knowledge products, tools, approaches, and best practices that could lead to pan-Canadian impacts. While each contribution agreement might not have a pan-Canadian target audience, the HCPCP clearly has expectations that funded projects will either "increase knowledge on a pan-Canadian scale or be implemented on a pan-Canadian scale" (Health Canada, 2009f, p. 13). Program documents indicate that the pan-Canadian impacts can be achieved through dissemination of results and knowledge translation. The HCPCP has recognized the importance of knowledge translation for achieving its outcomes, as evidenced by commissioning a review of its knowledge translation activities to identify opportunities to improve and expand them (Moore et al., 2012).

Interviews with key informants from Health Canada indicated differences of opinion on how to measure success, in particular whether pan-Canadian impacts are required and what is necessary to demonstrate knowledge translation. Some Health Canada key informants noted that there was never the intention for all funding to be pan-Canadian in nature.^{Footnote 10} According to these key informants, when provinces and territories were the funding recipients, the intention was to build capacity in the proponent's jurisdiction; these projects did not have to show a pan-Canadian impact. For these projects, broader adoption of the knowledge products of a project was not a measure of project success. Rather, the delivery of intended products and successful results within the project were sufficient indicators of success. Interviews revealed differing understandings on what was expected for knowledge translation at the project level; some believed it required evidence of uptake of knowledge products, while others were satisfied with dissemination. Health Canada key informants held different views about whether it was the projects' or the Program's role to support knowledge translation beyond production and dissemination. While some key informants considered knowledge translation to be a role of both the Program and the projects, others believed that if a project is successful, other interested stakeholders will adopt its knowledge products if they are aware of the project's existence. These issues point to the need for Health Canada to determine what its role is within the HCPCP, the scope and scale of the intended impacts of the program, and how its activities contribute to improving the health system.

Broadening the forms of knowledge translation in contribution agreements. Key informants believed that the current approach of funding projects that end with the development of knowledge tools and dissemination hampers the HCPCP's ability to demonstrate outcomes of broader adoption. There is less evidence as one moves through the knowledge translation cycle from development to dissemination to implementation (Moore et al., 2012). According to key informants, the HCPCP will have to direct applicants to integrate a broader range of knowledge translation activities into proposals so that information on uptake can be gathered. Several suggestions were provided:

• Fund projects in phases so that particularly successful projects that appear most suitable are funded for knowledge translation activities that go beyond dissemination. A few key informants cautioned that this would require the HCPCP to undertake more careful mid-term analyses of projects, which is currently not occurring.
• If the original funded project does not have the capacity to engage in pan-Canadian knowledge translation or change management, consider signing a separate agreement with an organization skilled at knowledge translation and/or change management.
• Require larger projects to have an advisory committee that includes a knowledge translation expert.
• Provide longer-term funding (beyond five years) to assess uptake of knowledge products.

Post-contribution funding project follow-up of longer-term outcomes. The Program had not incorporated project-level follow-up, the systematic and formal monitoring of projects and their outcomes once the project funding has expired, even though some projects continue beyond HCPCP funding. Follow-up with projects could allow the HCPCP to gather information on the knowledge translation activities that have occurred since funding ended and possible pan-Canadian health system impacts. Follow-up would also allow the Program to understand what project activities work, which could inform future project selection and funding.

Training. Some key informants suggested training for funding recipients and Program staff in knowledge translation and uptake and performance measurement and reporting. Training in these areas would help HCPCP staff understand the Program objectives and use this knowledge to assist projects with incorporating these objectives via knowledge translation and measuring their performance. A standard performance measurement presentation tool for HCPCP staff was suggested. It was also suggested that sharing strong evaluations internally will give HCPCP staff members a better understanding of the level of reporting desired.

Program-level reporting. Key informants believed that a broader synthesis of project results would allow the Program to conduct a more strategic analysis of the effectiveness of funding in different priority areas, types of projects, and categories of funding recipients.

4.3 Efficiency and Economy

The evaluation noted that to improve the program's efficiency and economy, 1) a more streamlined project approval process is necessary, 2) measures to reduce lapses in funding should be implemented and 3) more realistic project timelines should be encouraged in order to reduce project amendments.

This section considers the efficiency and economy of the HCPCP. Some of the evaluation questions related to efficiency and economy could not be addressed with the available evidence. The Program does not maintain its financial and non-financial program and project information in a manner that would allow for an analysis of efficiency and economy. As noted earlier (see Section 4.1), Vote 1 (Operations and Maintenance) activities and funds could not be separated from broader Directorate-level activity and funding. This limited the ability to conduct an analysis of the efficiency of the HCPCP. The evaluation can show that Vote 1 funds (operating expenditures such as salaries, benefits, Operations and Maintenance) increased as a proportion of total HCPCP expenditures due to the decline in Vote 10 expenditures (Contributions funding) (see Table 13). In 2008-09, Vote 1 constituted 24% of HCPCP expenditures, while in 2011-12, it was 36%. These are only estimates, as actual expenditures for HCPCP Vote 1 could not be provided.

Table 13: Planned and Actual HCPCP Funding, 2008-09 to 2012-13

Type of Funding	2008-09		2009-10		2010-11		2011-12		2012-13		Total
	Adjusted Budget Allocation	ActualsTable 13 footnote *	Adjusted Budget Allocation	ActualsTable 13 footnote *	Adjusted Budget Allocation	ActualsTable 13 footnote *	Adjusted Budget Allocation	ActualsTable 13 footnote *	Adjusted Budget Allocation	ActualsTable 13 footnote *	Adjusted Budget Allocation	ActualsTable 13 footnote *
Vote 10 ($)
HCSIF	1,001,010	3,544,334	951,010	5,320,898	700,760	1,243,341	1,000,760	540,422	1,800,760	N/A	3,653,540	10,648,995Footnote ‡
CMIRPS	1,600,000	1,216,433	1,600,000	1,481,637	1,600,000	1,200,000	1,600,000	1,200,000	1,600,000	N/A	6,400,000	5,098,070Footnote ‡
HHRS	14,356,237	6,567,888	14,883,137	9,214,128	14,233,137	12,666,973	14,325,637	8,292,528	14,233,185	N/A	57,798,148	36,741,517Footnote ‡
IEHPI	15,577,555	13,042,818	15,577,555	12,891,823	16,677,555	12,882,299	16,677,555	8,282,263	16,677,555	N/A	64,510,220	47,099,203Footnote ‡
PWTGPPF	13,488,750	10,181,421	13,338,750	9,093,470	--	--	--	--	--	--	26,827,500	19,274,891
NWTI	3,750,000	2,849,489	--	--	--	--	--	--	--	--	3,750,000	2,849,489
Total - Vote 10 (S)	49,773,552	37,402,383	46,350,452	38,001,956	33,211,452	27,992,613	33,603,952	18,315,213	34,311,500	N/A	197,250,908	149,112,165Footnote ‡
Vote 1 (S)	11,526,835	N/A	10,439,784	N/A	10,114,697	N/A	10,114,697	N/A	10,114,697Table 13 footnote ±	N/A	52,310,710	N/A
Total (Vote 1 and 10) ($)	61,300,387	N/A	56,790,236	N/A	43,326,149	N/A	43,718,649	N/A	44,426,197	N/A	249,550,118	201,422,875Table 13 footnote ^
Source: Health Canada. Table 13 footnotes Table 13 footnote * Vote 1 is based on the original approved amount. Health Canada does not have the ability to track the actual spending of Vote 1 funds by initiative, so actuals are not available. Return to the first table 13 footnote * referrer Table 13 footnote + Estimates from Draft 2012-13 DPR Return to table 13 footnote + referrer Table 13 footnote ± Used estimate from 2011-12 Adjusted Budget Allocation Return to table 13 footnote ± referrer Table 13 footnote ^ Assumes all of Vote 1 was spent. Return to table 13 footnote ^ referrer Table 13 footnote ‡ includes 2008-09 to 2011-12 amounts only. Return to the first table 13 footnote ‡referrer Table 13 footnote N/A Not available: Reporting for 2012-13 Actuals, as of August 30, 2013, is incomplete because the Program has not closed all payouts for this FY. Return to the first table 13 footnote N/A referrer

The issue of economy requires the ability to consider Program cost in relation to the outcomes produced, and without systematic performance data, the evaluation could not conduct this analysis. However, the evaluation can address the issue of whether the Program resources were used as planned (comparison of planned versus actual expenditures), and whether the HCPCP demonstrated efficient and economical decision making and implementation.

One measure of efficiency available to the evaluation is the lapsing of funding. According to the available information, the Program lapsed funds consistently between 2008-09 and 2012-13 (see Table 2). The following are areas where lapses occurred:

• HCSIF experienced apparent lapses of CMIRPS funds, which was the result of CIHI's decision to self-fund the project. Half of the freed funds were reallocated to ISMP Canada. HCSIF also had a deficit in the "core" funding stream, as the Paediatric Wait Times Initiative received funding through it. Unspent funds from the PWTGPPF and the NWTI were used to cover the deficit in HCSIF.
• HHRS held funds aside in 2008-09 and 2009-10 while awaiting the government's decision to launch the Family Medical Residencies Initiative. In 2011-12, two events caused lapsed funds: Saskatchewan withdrew from the Medical Residencies Initiative, and there was a delay in the Quebec agreement.
• IEHPI lapsed funds typically because projects funded by the provinces and territories under the initiative lapsed funds. In 2011-12, however, the unspent funds increased more than twofold from previous years due to late proposal submissions and a lengthy review and approval process (Health Canada, 2012e). Since the 2008 renewal of the Program, forecasted funding surpluses within a FY can be transferred across components to address the issue of lapsing funds.

Key informants identified several reasons for lapsed funding, some of them being outside the scope of HCPCP. Most believed that the majority of lapsed funds were for provincial and territorial projects, which was confirmed in an analysis of lapsed funds that was conducted as part of the HCPCP's strategic plan. Key informants noted that the provinces and territories often did not provide information on potential lapses, so funds could not be reallocated in a timely manner. Key informants noted that some NGOs were overly optimistic in what they could accomplish in the first year. However, several key informants also mentioned that the lag time created by Health Canada's approval process (see below) meant that projects had less time than originally anticipated in the first FY. While key informants noted that the HCPCP should encourage/require projects to reduce their first year budget to compensate for the lost time, this did not always happen.

The following are measures recently taken by the Program to minimize lapsed funding by projects:

• engage in more proactive communication with recipients to ensure they are aware of potential lapsed funds sooner,
• assess proposals and encouraging projects to be more realistic in their first year projections of activities and funding, and
• have projects adjust funding if the approval process is delayed beyond the anticipated start date.

Health Canada key informants did not believe that these measures have been entirely effective in reducing lapses. Given the many potential areas of need (see Section 3.3), the HCPCP should consider what steps could be taken to reduce funding lapses in the future.

The evaluation found evidence of efficient and effective decision making and implementation, however, areas for improvement pertain to the approval process and the contribution agreement management.

Proposal approval process. The consensus among Health Canada key informants was that the approval process within the Department created delays. This was confirmed by survey findings: over one third (39%) of survey respondents indicated that they experienced a delay in signing their contribution agreement, with the Health Canada approval process being the most common reason cited, followed by the proposal development process with the HCPCP (i.e., the need for revisions). Once the new Grants and Contributions Information Management System (GCMIS) is in use, the HCPCP will be able to assess systematically at what point in the process delays occur so that efforts can be focused on addressing them. Further to this, Health Canada staff stated that delays occurred not during the review process within the HCPCP but during the funding approval process within higher administrative levels of Health Canada, and provided examples of proposals taking 6-12 months to be approved.

Health Canada key informants believed that the efficiency and effectiveness of the proposal process improved in the last two years. Efforts had been made to improve consistency and objectivity in the review and approval process, such as templates and checklists; training of program officers; and more informed discussions by the PMC. Several key informants believed that the proposal review and approval process could build on these efforts and be more objective, as well as efficient and economical (i.e., projects would better target identified needs/gaps and have greater potential to produce outcomes).

The HCPCP had undertaken efforts to reduce delays in the proposal review and approval process through improved engagement with senior management, in order to address potential issues/questions proactively. According to key informants, these efforts had informed the proposal checklist so that supporting materials (funding approval form, briefing note) should address many common questions of senior management. Program staff were also considered to have improved at providing senior management with a succinct rationale for funding a particular project and with more advance information about the proposals that were being developed (subject matter, potential outcomes, anticipated budget, etc.).

Contribution agreement management. Based on the project-level document review and the survey, a substantial percentage of the contribution agreements required amendments (40% of survey respondents and 69% of project files reviewed).^{Footnote 11} Most Health Canada key informants believed that the proportion of contribution agreements that required amendments indicates the need to improve efficiency through better project management by Health Canada. For example, while the delay created by the proposal approval process was provided as a reason for contribution agreement amendments, Health Canada key informants also noted that the Program should be able to manage these challenges by reorienting project funds between Year 1 and 2 in the initial agreement. The above finding also needs to be considered in the broader context, as the evaluation identified (as discussed in Section 4.1) that the Program has demonstrated efficiency in that most project activities were completed as planned and mostly on schedule.

5.0 Conclusions and recommendations

This section of the report summarizes the main findings from the evaluation, draws conclusions, and makes recommendations.

Relevance

The HCPCP aligned with the federal role and responsibilities for health care, as well as with federal and departmental priorities. The Program served the federal role in health care by providing leadership, promoting health care system innovation, and supporting federal/provincial/territorial coordination and collaboration to improve the health care system. In addition, the HCPCP aligned with Health Canada's first departmental strategic outcome of "a health system responsive to the needs of Canadians" (Health Canada, 2011a). The Program's alignment with federal priorities initially came from the federal commitments made in the Health Accords. Although the federal government has not indicated an intent to negotiate a new Health Accord, it has continued to emphasize its broad commitment to the improvement of the health care system in its most recent Speeches from the Throne (2013) and Budget Speeches, as well as in other public statements. The evaluation assessed the continued need for the Program through a review of the literature and found that there remain health care system needs in each of the HCPCP's priority funding areas, e.g. health human resource, internationally educated health practitioners and in health system innovation.

With the Health Accord set to expire, the HCPCP is at a watershed moment where it will need to define the federal role and funding priorities without a similar statement of federal priorities.

Performance - Program implementation

The evaluation found that the HCPCP made progress during the evaluation period in implementing its planned activities. In particular, the Program has instituted its new governance structure (including three new committees), which enabled it to move to a more centrally-coordinated structure, respond to the 2007 evaluation, as well the Government of Canada Action Plan to Reform the Administration of G&C Programs. The three committees were found to have addressed many of their responsibilities. However, areas for improvement were identified, such as overlapping responsibilities, and lack of engagement in some areas of responsibilities.

The evaluation found that the PMC made efforts to conduct strategic planning, but concerns were voiced that much of its focus had been diverted to reallocating resources to reduce lapses rather than providing strategic direction for the Program. In addition, the strategic planning that occurred did not yet include a systematic review of funded projects to identify gaps or to assess whether/how to build on the evidence base created, which would have responded more directly to the 2007 evaluation recommendations. It also focused on funding priorities and not performance-related issues, such as whether certain types of projects had greater evidence of success or how best to achieve broader impacts at the health care system level (e.g., supporting knowledge translation). Defining these new policy priorities as part of an evidence-based strategic review process may be important in the post-Accord era.

The 2007 evaluation of the HCPCP recommended that the Program improve performance measurement data and monitoring. The recommendations included developing a systematic approach to collecting performance information. Although the evaluation found that the Program has made progress in performance reporting through the development of the RRET, the focus on the RRET and its gradual roll-out with projects has meant that more than five years after the 2007 evaluation, the Program does not have comprehensive and consistent performance information from all of its funded projects. The Program also does not have a mechanism for performance reporting of program-level (Vote 1 or Operations and Maintenance) activities.

The evaluation findings demonstrate that the Program was largely successful in implementing its planned activities with respect to funding projects. Three of the four planned calls for proposals occurred between 2008-09 and 2012-13, and 35 new projects were funded. One call for proposals was deferred for future consideration. Funded projects during the evaluation period all aligned with HCPCP and component objectives. In addition, the project-level document review and survey results demonstrated that almost all project activities were implemented as planned and mostly on schedule, taking into consideration the revised schedule in contribution agreement amendments. Projects also largely produced expected outputs, with almost all developing at least one knowledge product and having a collaborative working arrangement.

In terms of funding projects, the method of conducting calls for proposals was identified as an area of potential improvement. Although the Program did not track the number of proposals by method of solicitation (broad-based, targeted, or unsolicited), other lines of evidence (documents, survey, interviews) indicate that most proposals are received through targeted solicitations, with just over one quarter being unsolicited. The HCPCP had not conducted any open or broad-based calls. There was some divergence of views within the Program between those who prefer broad-based calls for reasons of transparency and those who prefer targeted calls due to capacity issues both within Health Canada (to manage open calls) and the proponent organization (to conduct the work). To ensure that the proposal process is transparent, the HCPCP should keep systematically-collected information on calls for proposals and unsolicited proposals.

The Program has contributed to policy/research development and implementation, as well as facilitation of collaboration, by funding projects that are actively engaged in these activities. At the program level, there is also the expectation that the HCPCP will support these activities with use of Vote 1 funds, which it has through activities such as providing an annual report on funded projects (initially for HHRS and IEHPI but now program-wide), conducting synthesis reports, and hosting a variety of stakeholder meetings that facilitate information sharing. That being said, the evaluation found that there were no clearly-defined expectations for the HCPCP's role in these activities, and this role did not appear to be a major part of the HCPCP's strategic planning process. The program logic model and evaluation plan did not provide indicators for how these program-level activities support outcomes, and, as a result, the Program was not collecting performance data on these activities. In addition, tracking the activities that were funded by the HCPCP was made difficult as a result of combining Vote 1 funds from various sources to fund Directorate activities.

Performance - Achievement of outcomes

There was limited evidence to suggest that HCPCP was achieving its outcomes. Health Canada key informants acknowledged that the collection of performance data to demonstrate the achievement of outcomes had been a weakness for the HCPCP. The issue appeared to be twofold. There was a need for improved understanding at the Program and project level of performance measurement, and a need for better performance reporting. The intention of the RRET was to improve performance reporting by making it more systematic, but without an improved understanding of performance measurement among Program and project staff, the results were still inadequate for assessing the projects' and Program's performance. Based on past project reporting, some funding recipients tended to report on activities and outputs rather than outcomes. In addition, outcome evidence was often weak or not provided. The HCPCP recently became more proactive in assisting projects with performance reporting; however, Health Canada key informants believed that both Program staff and funding recipients would benefit from performance measurement training.

The ability of the HCPCP to demonstrate effectiveness in achieving its outcomes was also affected by the nature of the funded projects, given the identified outcomes of the Program. Many funded projects ended with the development of a knowledge product or with its dissemination (which was often passive -- e.g., posting online). As a result, the projects could not always report on immediate outcomes, such as increased use of knowledge, approaches, models, or strategies developed by the project, not to mention longer-term outcomes, such as broader adoption of their knowledge/innovations or improvements in health care system policies/practices. Nor does Health Canada follow-up on a project's potential outcomes once contribution funding has ended. The HCPCP might want to consider alternatives (in addition to the RRET), such as changing the structure of contribution agreements in order to fund and otherwise support post-contribution funding knowledge translation activities; including additional follow-up activities with projects, such as exit interviews and one-year follow-up; and focusing performance reporting efforts on strategic evaluations of priority projects.

Based on evaluation findings, HCPCP would benefit from building a common understanding of what is required for pan-Canadian impacts and knowledge translation in the Program. Interviews indicated that there was some disagreement on whether pan-Canadian impacts were required and what was necessary to demonstrate knowledge translation (i.e., was dissemination sufficient?). If the Program intends to make knowledge translation a focus of performance measurement, building a common understanding of expectations for knowledge translation and buy-in from Program staff and other stakeholders will be important.

As a funder, it is in Health Canada's interest to fund projects that achieve measureable and meaningful health system outcomes. The department may benefit in conceptualizing its contribution beyond funding and managing projects, producing reports and organizing meetings. This would involve determining what its role is within the HCPCP and how its activities contribute to improving the health system. The evaluation pointed to the need for Health Canada to determine the scale of its intended impacts (i.e., jurisdictional vs. pan-Canadian). A pan-Canadian focus was not consistently maintained in project selection. Health Canada may wish to chart a direction in terms of programming, determine the scope and scale of stewardship that it wants to engage in, the value it can add to the efforts of the projects it funds and report on these efforts. Health Canada may want to re-examine the HCPCP logic model where the Program has outlined its activities, outputs and outcomes.

Recommendation 1

Health Canada should explore ways to optimize the investment in the HCPCP in order to achieve program outcomes. In this context, the HCPCP should refine its priorities, adjust program objectives and associated program design, building on lessons learned over the past decade.

Recommendation 2

Systematic project and program tracking to achieve program results should be strengthened to inform program decision-making and reporting on success.

Performance - Efficiency and economy

The issue of economy requires the ability to consider Program cost in relation to the outcomes produced, and without systematic performance data, the evaluation could not conduct this analysis. The ability to respond to the issue of efficiency was affected by the inability to track how HCPCP Vote 1 money had been used, as it was combined with other Vote 1 funds for the Directorate. That being said, the evaluation was able to consider issues of efficiency in terms of whether Program resources were used as planned and whether decision making was efficient and economical.

The Program experienced funding lapses each fiscal year covered by the evaluation (2008-09 to 2012-13). The lapses were attributed to several challenges experienced by the Program, including the need to hold funds aside for the Family Medical Residencies Initiative (FMRI); difficulties in getting information on delays from provinces and territories in time to reallocate funds; and late proposal submissions and lengthy review processes that reduce the ability to use funds in the first year of a contribution agreement. Given the need to demonstrate efficiency and economy within the Program, the HCPCP should consider steps to reduce future funding lapses.

The evaluation identified other areas of program management where the HCPCP could improve its efficiency, and the HCPCP had already undertaken efforts to improve in all of these areas. The proposal approval process within Health Canada was sometimes lengthy, which was one reason why a substantial percentage of contribution agreements required amendments. The Program had undertaken methods to improve the efficiency of the proposal approval process through improved communication with senior Health Canada officials about upcoming proposals. Previous efforts to improve the objectivity and transparency of the proposal review and approval process could be expanded upon to improve Program functioning and ensure that funding priorities are clearly articulated; that funded projects better target identified needs/gaps; and that funds expended have greater potential to produce the intended outcomes.

Annex 1 Program components

1. Health Human Resource Strategy (HHRS)

The HHRS was created in 2004 to improve the national planning, recruitment, retention, education, and coordination of health human resources. Between 2008-09 and 2011-12, HHRS funded 70 contribution agreements for a total of $36.7 million. Initially, the HHRS consisted of three initiatives: Pan-Canadian Health Human Resource Planning, Interprofessional Education for Collaborative Patient-Centred Practice, and Recruitment and Retention, which included the Healthy Workplace Initiative.

In 2008, the HHRS reframed its priority areas to focus on four directions that were still very much related to the original three initiatives, but also incorporated internationally educated health professionals. The four priority areas were: needs-based planning; internationally educated health professionals (increasing access to assessment and training programs, and workplace integration); health workforce optimization (retention, alignment of education with health care system needs, and effective use/optimal scope of practice); and practice environment revitalization (interprofessional collaborative practice and improved work environments) (Health Canada, 2008a, pp. 4-6). The most recent description of the HHRS priorities remains very connected to these themes of resource planning, healthy workplaces, and effective use of the health workforce:

• using human resource skills effectively
• creating healthy, supportive, learning workplaces
• more health care providers
• more effective planning and forecasting (Health Canada, 2012c, pp. 2-3)

In 2011, the federal government announced the $39.5 million Family Medical Residencies Initiative (FMRI). This funding was provided through the HHRS and contributes to one of the four key directions noted above (more health care providers). The initiative is to address physician shortages and maldistribution by adding about 100 residency positions in underserved rural or remote areas. Currently, eight provinces and territories (Newfoundland and Labrador, New Brunswick, Quebec, Ontario, Manitoba, Saskatchewan, British Columbia, and Nunavut) have benefited from the initiative.

2. Internationally Educated Health Professionals Initiative (IEHPI)

The IEHPI was created in 2005 to facilitate the development of programs and supports for the integration of internationally educated health professionals into the Canadian health work force. Between 2008-09 and 2011-12, IEHPI funded 35 contribution agreements for a total of $47.1 million.

The initiative focused on eight different professional areas: licensed practical nurses, medical laboratory technologists, medical radiation technologists, registered nurses, occupational therapists, pharmacists, physiotherapists, and physicians (Health Canada, 2012c, p. 3). The funded projects targeted one or more of the six strategic outcomes of the IEHPI:

• Internationally educated health care professionals will be able to gain access to clear, timely information about paths to licensure.
• Fair and transparent mechanisms for assessing credentials, knowledge, and clinical skills of internationally educated health care professionals will be established.
• Programs that increase the capacity of faculty and clinical educators to work effectively with internationally educated health care professionals will be created.
• Access to a range of bridge training and remediation programs will be increased.
• Internationally educated health care professionals will be better integrated into the health care workplace.
• Regional collaboration to maximize the impact of available resources will be enhanced. (Health Canada, 2012c, p. 3)

The IEHPI distributed funds through two streams. The provincial/territorial stream received the majority of the IEHPI funding. The funding distribution was based on a funding formula that included a base amount for each province/territory plus a per capita allocation. While the contribution agreement was with the province or territory, each jurisdiction chose the projects to fund, giving consideration to their alignment with the IEHPI objectives. As a result, one contribution agreement with a province or territory could fund multiple projects. The second stream funded pan-Canadian projects with non-government, not-for-profit organizations, and educational institutions.

3. Health Care System Innovation Fund (HCSIF)

The HCSIF was intended to accommodate various health care system policy issues and encourage innovation in the health care system (Health Canada, 2008b, p. iii). Between 2008-09 and 2011-12, HCSIF funded 39 contribution agreements for a total of $15.7 million.

Since 2008, the HCSIF provided two funding streams. The first stream offered $1.2 million per year in project funding to support the Canadian Medication Incident Reporting and Prevention System (CMIRPS) and $400,000 per year to support Health Canada's participation in CMIRPS. CMIRPS was a collaborative effort of the Institute for Safe Medication Practices Canada (ISMP Canada), the Canadian Institute for Health Information (CIHI), the Marketed Health Products Directorate of Health Canada, and the Canadian Patient Safety Institute (CPSI). HCSIF funding supported ISMP Canada's activities under CMIRPS, which included its core initiatives of analyzing medication incident data and enhancing knowledge translation activities for medication safety interventions, in addition to other initiatives in the area of medication safety.

Through its second stream of approximately $1 million,^{Footnote 12} the HCSIF funded projects in priority areas, including:

• wait times;
• access to health care;
• chronic and continuing care; and
• end-of-life care. (Health Canada, 2012c, pp. 3-4)

During the 2008-09 to 2011-12 period, the priority areas for the second stream fit under the two main themes: "health care policy conferences and events" that are to encourage dialogue among health care system stakeholders and support knowledge translation, and "health care system responsiveness to population aging," which focused resources on projects that considered the implications of the aging population on the health care system and possible pan-Canadian, system-level solutions (Health Canada, 2009a). While a call for proposals occurred in 2009-10 for conferences and events, the call for proposals for the theme on the health care system responsiveness to population aging was deferred for future consideration.

4. National Wait Times Initiative (NWTI)

The federal government announced the NWTI in February 2005. Its main objective was to support the commitments to reduce wait times made by First Ministers in the 2004 10-Year Plan to Strengthen Health Care. Between 2008-09 and 2011-12, the NWTI funded 8 contribution agreements for a total of $2.8 million.

The NWTI was designed to support research, knowledge development, and dissemination to inform the development of policies, best practices, and programs and services aimed at improving access to care and reducing wait times (Health Canada, 2012c, p. 4). Initially, the themes of the NWTI were to support projects that would:

• demonstrate progress on benchmarks and indicators;
• inform the public on wait times, educate providers on wait time management tools, and educate patients waiting for a procedure; and
• offer knowledge development, transfer, and dissemination to support the development and implementation of innovative approaches to wait time management (PWGSC, 2009, p. 1).

In 2006, the NWTI shifted focus to emphasize the third theme after the government made Patient Wait Time Guarantees a priority. This new focus was to enable the NWTI to support the implementation of the Patient Wait Times Guarantees (PWGSC, 2009, p. 1). The NWTI ended on March 31, 2009.

5. Patient Wait Times Guarantee Pilot Project Fund (PWTGPPF)

In 2007, the Prime Minister announced that the provinces and territories would establish wait time guarantees in at least one clinical area of their choice by 2010. The clinical areas selected by the provinces and territories included: cancer care, hip and knee replacement, cardiac care, diagnostic imaging, cataract surgeries, or primary care (Government of Canada, 2007). To support this policy direction, the government set aside time-limited funding of over $600 million for the Patient Wait Times Guarantee Trust, $30 million for the PWTGPPF, and $400 million to Canada Health Infoway for health information technology that would support the establishment and implementation of Patient Wait Time Guarantees. The Trust and the funding to Canada Health Infoway are not part of this evaluation. However, the purpose of the PWTGPPF was to support this broader project of Patient Wait Time Guarantees by funding provincial and territorial projects to develop, implement, and test innovative approaches to reducing wait times and establish wait time guarantees (Health Canada, 2011b, p. iii). Between 2008-09 and 2011-12, PWTGPPF funded 11 contribution agreements for a total of $19.2 million.

In particular, the PWTGPPF supported provinces and territories in projects aiming to:

• develop and assess tools to establish and meet guarantees (including the provision of recourse for patients when guaranteed time frames for treatment are exceeded);
• identify and address policy and operational issues associated with establishing guarantees; and
• encourage collaboration and the exchange of best practices. (Health Canada, 2012c, p. 4)

The PWTGPPF ended on March 31, 2010 (Health Canada, 2012b).

Annex 2 Program Logic Model

Program Logic Model

Logic Model Components	Stream 1	Stream 2		Stream 3		Stream 4	Enabling Factors
Activities	Strategic Planning Priority/direction setting Identifying needs and gaps	Management and Accountability Governance Performance monitoring, audit and evaluation		Facilitating Collaboration Collaborating with and engaging stakeholders Building and maintaining collaborative working relationships		Policy and Research Development and Implementation Funding projects Synthesizing findings Identifying promising practices and strategies	Collaboration and coordination Supportive organization culture Readiness for change Strong leadership Decrease barriers to: Knowledge development, translation, and use Health system reform
Outputs	Knowledge tools, products and innovations Including: research reports; databases; reference materials; planning tools; strategies, approaches or models; synthesis of research; policy options and advice; and dissemination mechanisms		Collaborative relationships Development and maintenance of relationships with and among stakeholders including recipient organizations, professional associations, governments, and policy-makers		Identified barriers and enablers of: Knowledge development, translation, and use Health system reform
Immediate Outcomes	Increased awareness and understanding of knowledge tools/products, approaches, models, innovations, health care system performance, and health system reform issues		Improved collaboration and coordination among health care system stakeholders to minimize identified barriers and utilize identified enablers		Successful and/or increased use of knowledge, approaches, models, strategies, or promising practices
Intermediate Outcomes	Broader adoption by relevant health care system stakeholders of knowledge or innovations resulting in changes in: Policy Practice Organizational structure
Long-Term Outcomes	Improvements in health care system policies, practices, and performance in areas targeted by the HCPCP

Annex 3 References

• Armstrong, D., Barkun, A. N., Chen, Y., Daniels, S., Hollingworth, R., Hunt, R. H., & Leddin, D. (2008). Access to specialist gastroenterology care in Canada: The Practice Audit in Gastroenterology (PAGE) Wait Times Program. Canadian Journal of Gastroenterology, 22(2), 155.
• Bailey, I., & Curry, B. (n.d.). In surprise move, Flaherty lays out health-spending plans til 2024. The Globe and Mail. Retrieved August 29, 2012, from http://www.theglobeandmail.com/news/politics/in-surprise-move-flaherty-lays-out-health-spending-plans-til-2024/article4247851/
• Barrett, J., Curran, V., Glynn, L., & Godwin, M. (2007). CHSRF synthesis: Interprofessional collaboration and quality primary healthcare. Retrieved from http://www.chsrf.ca/Migrated/PDF/SynthesisReport_E_rev4_FINAL.pdf
• Baumann, A., & Blythe, J. (2009). Integrating internationally educated health care professionals into the Ontario workforce. Nursing Health Services Research Unit, McMaster University.
• Blendon, R. J., Schoen, C., DesRoches, C., Osborn, R., & Zapert, K. (2003). Common Concerns Amid Diverse Systems: Health Care Experiences In Five Countries. Health Affairs, 22(3), 106-121. doi:10.1377/hlthaff.22.3.106
• Brasset-Latulippe, A., Verma, J., Mulvale, G., & Barclay, K. (2011). Provincial and Territorial Health System Priorities: An Environmental Scan. Ottawa: CHSRF.
• Canadian Intergovernmental Conference Secretariat. (2000, September 11). News Release - First Ministers' Communique on Health. Retrieved February 1, 2012, from http://www.scics.gc.ca/english/conferences.asp?a=viewdocument&id=1144
• CBOC. (2012). The Canadian Health Care Debate: A Survey and Assessment of Key Studies. CBOC. Retrieved from http://www.conferenceboard.ca/temp/e4261f6c-d336-42f5-86d0-78054e3f863c/12-305_CanadianHealthCareDebate_RPT.pdf
• Chan, A., & Wood, V. (2012). Preparing tomorrow's healthcare providers for interprofessional collaborative patient-centred practice today. Retrieved from http://www.ubcmj.com/pdf/ubcmj_1_2_2010_22-24.pdf
• CIHI. (2002). Health Indicators 2002. Ottawa: CIHI. Retrieved from http://secure.cihi.ca/cihiweb/products/HRinsert2002eng.pdf
• CIHI. (2007). Health Indicators 2007. Ottawa: CIHI.
• CIHI. (2009). Health Care in Canada 2009: a decade in review. Ottawa: CIHI.
• CIHI. (2011a). Physician supply increasing twice as quickly as Canadian population. News release. Retrieved April 16, 2012, from http://www.cihi.ca/cihi-ext-portal/internet/en/document/spending+and+health+workforce/workforce/physicians/release_15dec11
• CIHI. (2011b). Supply, Distribution and Migration of Canadian Physicians, 2011.
• CIHI. (2011c). Wait Times in Canada - A Comparison by Province, 2011. Ottawa: CIHI. Retrieved from http://secure.cihi.ca/cihiweb/products/Wait_times_tables_2011_en.pdf
• CIHI. (2011d). Supply, distribution and migration of Canadian physicians, 2010. Ottawa: CIHI. Retrieved from http://site.ebrary.com/lib/celtitles/docDetail.action?docID=10527460
• CIHI. (2012a). Health indicators 2012. Ottawa: CIHI. Retrieved from http://site.ebrary.com/id/10478722
• CIHI. (2012b). Regulated Nurses: Canadian trends, 2006 to 2010. Retrieved from https://secure.cihi.ca/free_products/RegulatedNursesCanadianTrends2006-2010_EN.pdf
• CIHI. (2012c). Wait Times in Canada--A Summary, 2012. Ottawa: CIHI. Retrieved from http://secure.cihi.ca/cihiweb/products/WaitTimesSummary2012_EN.pdf
• CIHI. (2013). Wait Times for Priority Procedures in Canada, 2013. Retrieved from https://secure.cihi.ca/free_products/wait_times_2013_en.pdf
• CIHI, & Statistics Canada. (2002). Health Care in Canada 2002. Ottawa: CIHI. Retrieved from https://secure.cihi.ca/free_products/HR2002eng.pdf
• CLHIA. (2009). CLHIA Report on Health Care Policy: Towards a Sustainable, Accessible, Quality Public Health Care System. Toronto: CLHIA.
• CMA. (2005). Meeting the health care needs of Canadians. Retrieved from http://www2.cna-aiic.ca/CNA/documents/pdf/publications/Meeting_Health_Care_Needs_e.pdf
• CMA. (2010). Health Care Transformation in Canada. Ottawa: CMA.
• CMA. (2011). 11th Annual National Report Card on Health Care. Retrieved from http://www.cma.ca/multimedia/CMA/Content_Images/Inside_cma/Media_Release/2011/reportcard/2011National-Report-Card_en.pdf
• CMA, & CNA. (2011). Principles to Guide Health Care Transformation in Canada. Retrieved from http://www.cma.ca/multimedia/CMA/Content_Images/Inside_cma/Advocacy/HCT/HCT-Principles_en.pdf
• Commission on the Future of Health Care in Canada. (2002). Building on values: the future of health care in Canada. Saskatoon: CFHCC. Retrieved from http://publications.gc.ca/collections/Collection/CP32-85-2002E.pdf
• Duckett, S., & Kempton, A. (2012). Canadians' Views about Health System Performance. Healthcare Policy, 7(3), 85-101.
• Etchells, E., Juurlink, D., & Levinson, W. (2008). Medication errors: the human factor. Canadian Medical Association Journal, 178(1), 63-64. doi:10.1503/cmaj.071658
• Ferguson-Paré, M., Mallette, C., Zarins, B., McLeod, S., & Reuben, K. (2010). Collaboration to change the landscape of nursing: A journey between urban and remote practice settings. Nursing Leadership, 23(Special Issue), 90-100.
• First Ministers. (2003). 2003 First Ministers' Accord on Health Care Renewal.
• First Ministers. (2004). A 10-Year Plan to Strengthen Health Care (First Ministers' Meeting on the Future of Health Care).
• FLMM. (2009). A pan-Canadian framework for the assessment and recognition of foreign qualifications. Ottawa: FLMM.
• Fooks, C., & Lewis, S. (2002). Romanow and Beyond: A Primer on Health Reform Issues in Canada. CPRN. Retrieved from http://www.cprn.org/documents/15525_en.pdf
• Government of Canada. (1985). Canada Health Act , R.S.C., 1985, c. C-6. Retrieved from http://laws-lois.justice.gc.ca/PDF/C-6.pdf
• Government of Canada. (2007). Canada's New Government announces Patient Wait Times Guarantees. Retrieved June 11, 2012, from http://pm.gc.ca/eng/media.asp?id=1611
• Government of Canada. (2008). Responsible leadership: The Budget Speech 2008. Ottawa: Department of Finance Canada. Retrieved from http://www.budget.canada.ca/2008/pdf/speech-discours-eng.pdf
• Government of Canada. (2009). Canada's economic action plan: The Budget Speech 2009. Ottawa: Department of Finance Canada. Retrieved from http://www.budget.canada.ca/2009/pdf/speech-discours-eng.pdf
• Government of Canada. (2010a). Speech from the Throne to open the Third Session Fortieth Parliament of Canada. Retrieved April 10, 2012, from http://www.parl.gc.ca/Parlinfo/Documents/ThroneSpeech/40-3-e.html
• Government of Canada. (2010b). Canada's economic action plan, year 2, the Budget Speech 2010: leading the way on jobs and growth. Ottawa: Department of Finance Canada. Retrieved from http://www.budget.canada.ca/2010/pdf/speech-eng.pdf
• Government of Canada. (2010c). Government Response to the Standing Committee on Health's Report: Promoting Innovative Solutions to Health Human Resource Strategies. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/411/soci/rep/Response25sep12-e.pdf
• Government of Canada. (2011a). Here for all Canadians, Stability, Prosperity, Security: Speech from the Throne to Open the First Session of the Forty-First Parliament of Canada. Retrieved from http://www.speech.gc.ca/local_grfx/docs/sft-ddt-2011_e.pdf
• Government of Canada. (2011b). The next phase of Canada's Economic Action Plan, a low-tax plan for jobs and growth - The Budget Speech. Retrieved from http://www.budget.canada.ca/2011/speech-discours/speech-discours-eng.pdf
• Government of Canada. (2011c). Backgrounder on major transfer renewal. Retrieved August 29, 2012, from http://www.fin.gc.ca/n11/data/11-141_1-eng.asp
• Government of Canada. (2012a). Jobs, Growth and Long-Term Prosperity: Economic Action Plan 2012 - The Budget Speech. Retrieved from http://www.budget.canada.ca/2012/rd-dc/speech-eng.pdf
• Government of Canada. (2012b). Budget Plan. Retrieved from http://www.budget.canada.ca/2012/plan/pdf/Plan2012-eng.pdf
• Grimes, K., & Tholl, B. (2010). Toward Building a Better Business Case for Team-Based Health Care in Canada. Canadian Policy Network of the University of Western Ontario. Retrieved from http://www.cp-net.ca/site/ywd_dd_76/assets/pdf/FinalReportBBC.pdf
• Gutkin, C. (2011). Unacceptable emergency wait times. Canadian Family Physician, 57(2), 256.
• HCC. (2008). Rekindling reform: health care renewal in Canada, 2003-2008. Toronto: Health Council of Canada. Retrieved from http://epe.lac-bac.gc.ca/100/200/301/hcc-ccs/rekindling_reform_summ-e/H174-15-1-2008E.pdf
• HCC. (2011). Progress Report 2011: Health Care Renewal in Canada. Toronto: HCC.
• HCC. (2012). Progress Report 2012: Health care renewal in Canada. Toronto: HCC. Retrieved from http://healthcouncilcanada.ca/tree/ProgressReport2012_FINAL_EN.pdf
• Health Canada. (2006, May 9). Government of Canada's role in Aboriginal health care - Fact Sheet - First Ministers' Meeting on Health Care September 2004. Retrieved January 20, 2012, from http://www.hc-sc.gc.ca/hcs-sss/delivery-prestation/fptcollab/2004-fmm-rpm/fs-if_02-eng.php
• Health Canada. (2008a). HHRS Strategic Planning Workshop summary report, May 8, 2008.
• Health Canada. (2008b). Summative Evaluation Report of the Health Care Strategies and Policy Grant and Contribution Programs (2002 - 2007). Departmental Executive Committee - Finance, Evaluation and Accountability.
• Health Canada. (2008c). Health Care Policy Contribution Program: Results-based Management and Accountability Framework.
• Health Canada. (2008d). Health Canada 2008-2009 Report on Plans and Priorities.
• Health Canada. (2008e). Health Care Policy Contribution Program Operations Committee - Terms of Reference Final: June 24, 2008.
• Health Canada. (2008f). HCPCP Accountability Committee Terms of Reference (ToR) September 24, 2008.
• Health Canada. (2008g). Health Care Policy Directorate - Evaluation Recommendations and Management Action Plans.
• Health Canada. (2009a). Health Care System Innovation - Health Care Policy Contribution Program (2007-2010). Retrieved January 20, 2012, from http://www.hc-sc.gc.ca/hcs-sss/finance/hcpcp-pcpss/hcsi-isss-eng.php
• Health Canada. (2009b). Health Care Policy Contribution Program Guide for Applicants. Ottawa: Health Canada.
• Health Canada. (2009c). Health Canada 2009-2010 Report on Plans and Priorities.
• Health Canada. (2009d). Terms of Reference: Program Management Committee, Health Care Policy Contribution Program - Revised January 30, 2009.
• Health Canada. (2009e). Program Management Implementation Plan (PMIP) 2009/2010 Final April 8, 2009.
• Health Canada. (2009f). HCPCP Guide for Applicants English, February 10, 2009.
• Health Canada. (2010a). Terms of Reference: Horizontal Audit of Health Canada's Contributions Control Framework.
• Health Canada. (2010b). HCPCP Program Management Implementation Plan (PMIP) Progress Report Draft, April 27.
• Health Canada. (2010c). Health Canada 2010-2011 Report on Plans and Priorities.
• Health Canada. (2010d). Health Care Policy Contribution Program Program Management Implementation Plan - Progress Report. Health Canada.
• Health Canada. (2010e). Health Human Resource Strategy and Internationally Educated Health Professionals Initiative 2008/09 Annual Report. Retrieved June 13, 2012, from http://www.hc-sc.gc.ca/hcs-sss/pubs/hhrhs/2009-ar-ra/index-eng.php
• Health Canada. (2010f). Internationally Educated Health Professional Initiative: Federal/Provincial/Territorial Consultations Summary Report. Ottawa.
• Health Canada. (2011a). Health Canada 2011-2012 Report on Plans and Priorities.
• Health Canada. (2011b). Patient Wait Times Guarantee Pilot Project Fund Summative Evaluation. Ottawa: Health Canada.
• Health Canada. (2011c). Canada's Health Care System. Retrieved from http://www.hc-sc.gc.ca/hcs-sss/pubs/system-regime/2011-hcs-sss/index-eng.php
• Health Canada. (2011d). Canadian Medical Association Annual General Assembly - Minister's Speech 2011-08-22. Retrieved August 29, 2012, from http://www.hc-sc.gc.ca/ahc-asc/minist/speeches-discours/_2011/2011_08_22-eng.php
• Health Canada. (2011e). Final Audit Report: Audit of the Management Control Framework for Contribution Programs.
• Health Canada. (2011f). Health Human Resource Strategy Projects Policy Synthesis Report.
• Health Canada. (2011g). Health Human Resource Strategy and Internationally Educated Health Professionals Initiative: 2009/10 Annual Report. Ottawa: Health Canada.
• Health Canada. (2012a). HCPCP Full list of projects 08-09 to 12-13 March 7.
• Health Canada. (2012b). Evaluation - HCPCP Multi-Year Forecast 2008-2013 April 2012.
• Health Canada. (2012c). Request for Proposal: Evaluation of the Health Care Policy Contribution Program.
• Health Canada. (2012d). Government response to the review of the 2004 Health Accord: Taking action to improve the health of Canadians. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/411/soci/rep/Response25sep12-e.pdf
• Health Canada. (2012e). Financial Chart (Funding History 2008-12) June 2012.
• Health Care Innovation Working Group. (2012). From Innovation to Action: First report of the Health Care Innovation Working Group. Council of the Federation.
• Jackman, M. (2000). Constitutional Jurisdiction over Health in Canada. Health Law Journal, 8, 95-117.
• Johnson, K., Baumal, B., & CSMLS. (2011). Assessing the Workforce Integration of Internationally Educated Health Professionals (Final Report). CSMLS. Retrieved from http://www.camrt.ca/certification/international/iemrtreports/Workforce%20Intergration%20of%20IEMRTs.pdf
• Keith Johnson. (2007). Integrating internationally educated physiotherapists. Canadian Alliance of Physiotherapy Regulators. Retrieved from http://www.alliancept.org/pdfs/cred_iiep_integrate_eng.pdf
• Kreindler, S. A. (2008). Watching Your Wait: Evidence-Informed Strategies for Reducing Health Care Wait Times. Q Manage Health Care, 17(2), 128-135.
• Leatherman, S. T., Sutherland, K., CHSRF, CPSI, CIHI, & Statistics Canada. (2010). Quality of healthcare in Canada: a chartbook. Ottawa: CHSRF.
• Leddin, D., Armstrong, D., Barkun, A. N., Chen, Y., Daniels, S., Hollingworth, R., ... Paterson, W. G. (2008). Access to specialist gastroenterology care in Canada: Comparison of wait times and consensus targets. Canadian Journal of Gastroenterology, 22(2), 161-167.
• Marshman, J. A., Lam, R. W., & Hyland, S. (2006). Medication error events in Ontario acute care hospitals. The Canadian Journal of Hospital Pharmacy, 59(5). Retrieved from http://cjhp-online.ca/index.php/cjhp/article/viewArticle/268
• Mitton, C., Adair, C., McKenzie, E., Patten, S., & Perry, B. (2007). Knowledge transfer and exchange: Review and synthesis of the literature. The Milbank Quarterly, 85(4), 729-768.
• Moore, J., Khan, S., Marquez, C., Garg, R., Stein, S., Chan, W., & Strauss, S. (2012). HCPCP Knowledge Translation Review: Final Report. Health Canada.
• Newfoundland and Labrador. (2012). A strategy to reduce hip and knee joint replacement surgery wait times in Newfoundland and Labrador. Retrieved from http://www.health.gov.nl.ca/health/wait_times/orthopedic_wait_times_strategy.pdf
• PWGSC. (2009). Synthesis of Results of the National Wait Times Initiative: Final Report (Prepared for Health Canada). Ottawa: PWGSC.
• Saryeddine, T., & Brimacombe, G. G. (2009). "Wait" watchers III: Order & Speed... Improving access to care through innovations in patient flow. Ottawa: ACAHO.
• Soroka, S. (2007). Canadian perceptions of the health care system: a report to the Health Council of Canada. Toronto: HCC.
• Southern, D. A., Izadneghadar, M., Humphries, K. H., Gao, M., Wang, F., Knudtson, M. L., ... Ghali, W. A. (2011). Trends in Wait Times for Cardiac Revascularization. Canadian Journal of Cardiology, 27(2), 262.e21-262.e27. doi:10.1016/j.cjca.2010.12.030
• Standing Senate Committee on Social Affairs, Science and Technology. (2002). The Health of Canadians - The Federal Role. Volume Six: Recommendations for Reform (Final Report on the state of the health care system in Canada). Ottawa: SSCSAST.
• Standing Senate Committee on Social Affairs, Science and Technology. (2012). Time for Transformative Change: A Review of the 2004 Health Accord. Ottawa: SSCSAST.
• Treasury Board of Canada Secretariat. (n.d.). Working Together to Achieve Real Change: Implementing the Government Action Plan to Reform the Administration of Grants and Contributions.