Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations 2008-2009 to 2012-2013

Prepared by
Evaluation Directorate
Health Canada and the Public Health Agency of Canada

March 2014

Executive Summary
Management Response and Action Plan
1.0 Evaluation Purpose
2.0 Program Description
3.0 Evaluation Description
- 3.1 Evaluation Scope, Approach and Design
- 3.2 Limitations and Mitigation Strategies
4.0 Findings
5.0 Conclusions
- 5.1 Relevance Conclusions
- 5.2 Performance Conclusions
6.0 Recommendations
Appendix 1 - Logic Models
Appendix 2 - References

List of Tables

Table 1: Financial data
Table 2: Core Issues and Evaluation Questions
Table 3: Limitations and Mitigation Strategies
Table 4: Progress Along the Knowledge Translation or Change Continuum

Executive Summary

Description of Evaluation

The purpose of the evaluation was to assess the relevance and performance of certain transfer payments to pan-Canadian organizations, which were created and sustained by Health Canada, for the period of April 2008 to November 2013. The evaluation was conducted to fulfill requirements under section 42.1 of the Financial Administration Act (FAA) and the Treasury Board of Canada's Policy on Evaluation (2009) for departments to conduct an evaluation of all ongoing grant and contribution programs every five years. Departmental evaluations help ensure that credible, timely and neutral information on the ongoing relevance and performance of programs is available to support evidence-based decision making on policy, expenditure management and program improvements. Pan-Canadian organizations were also required under their funding agreements to conduct their own independent evaluations of their achievement of the objectives of the funding agreement and these independent evaluations were used as a line of evidence in this evaluation, thereby reducing duplication of effort.

The transfer payments to pan-Canadian organizations included in this evaluation were: the Canadian Agency on Drugs and Health Technologies, the Canadian Patient Safety Institute, the Health Council of Canada, the Mental Health Commission of Canada, and Canada Health Infoway. Other sustaining transfer payments to pan-Canadian organizations managed by Health Canada's Strategic Policy Branch were not included in this evaluation given they had other departmental evaluation commitments on different time schedules.

The methodology used in the evaluation included literature and document reviews and key informant interviews (18 in total). Independent evaluations and audits were used as a line of evidence in assessing the relevance and performance of the transfer payments to pan-Canadian organizations over the past five years. The economy and efficiency analysis provided an opportunity to examine the suite of organizations collectively and largely focused on Health Canada's engagement with, and management of, the transfer payments to pan-Canadian organizations. The analysis also took into consideration the context of each pan-Canadian organization (all differed in size, scope, and maturity), and comparisons between pan-Canadian organizations were not made.

Background Information

In Canada's decentralized healthcare system, these pan-Canadian organizations were intended to provide national coordination and collaboration on addressing system-wide priorities. The role and scope of these organizations differ, but all of them tend to provide either system-wide services/support (such as creating evidence-based advice to support decision making) or act as a catalyst for improvement in priority areas (e.g. providing leadership and intensifying progress on an issue).

Conclusions - Relevance

Overall, transfer payments to pan-Canadian organizations are aligned with federal priorities and fulfil an appropriate federal role. The organizations provide a unique, national coordination function that individual jurisdictions would not be able to accomplish. Although there are many organizations involved in the renewal of the healthcare system in Canada, provinces, territories or other stakeholders look to the pan-Canadian organizations for leadership and for providing a national perspective on system-wide priorities.

Besides the Health Council of Canada, whose mandate was directly tied to the health accords which will soon expire, the issues addressed by the other pan-Canadian organizations are still relevant today; and, while progress has been made, there is a clear need to continue to improve the healthcare system issues addressed by CADTH, CPSI, MHCC and Infoway.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

With the support of transfer payments from Health Canada, the pan-Canadian organizations were found to be producing a large number of outputs and achieving outcomes aligned with the stages along the knowledge translation continuum as well as their own objectives. While all organizations were working towards improving certain aspects of the healthcare system, some were not able to provide concrete examples of broad changes that had been achieved at this point, particularly for the newer pan-Canadian organizations. Annual reports and independent evaluations were useful in providing evidence of immediate and some intermediate outcomes, but contained fewer details about longer-term outcome achievement.

It should be noted that all of the pan-Canadian organizations were able to provide evidence of collaboration and coordination with a number of different organizations. The pan-Canadian organizations, as a whole, have been able to demonstrate progress towards their outcomes and should theoretically be able to achieve longer-term outcomes in time.

Demonstration of Economy and Efficiency

Overall, the transfer payments to pan-Canadian organizations were well managed in terms of monitoring progress and ensuring compliance with terms in the funding agreements. However, efforts to monitor funding requirements and to manage the funding agreements without dedicated program management staff meant that less time was available to provide strategic policy information for decision making. The evaluation identified opportunities to improve collaboration and coordination among policy areas managing pan-Canadian organizations as well as the need to share lessons learned, which would help minimize duplication (for example, in the development of control frameworks or processes for providing feedback on workplans) and increase efficiency in the approach to managing funding agreements.

There was less coherence across transfer payments in terms of the nature and level of engagement between the department and the pan-Canadian organizations, as well as some concern about the consistency of messaging in communications with the organizations. Establishing a common branch-wide approach to engagement with pan-Canadian organizations with a clear shared understanding of the interests underpinning that engagement, as well as a more coordinated Health Portfolio approach to interactions with them may help to ensure that expectations and information are being communicated, and communicated in a consistent manner, within the branch, within the portfolio and then to the pan-Canadian organizations. Ensuring a common approach that aligns program and relationship management with the department's strategic objectives, and strengthening communications both within the government and externally are key to optimizing the outcomes of federal engagement with these pan-Canadian organizations.

More engagement, and greater coherence, would provide an opportunity to encourage the alignment of the strategic direction or approach pursued by the pan-Canadian organizations with the priorities of the federal government, while taking into consideration the perspectives of other stakeholders involved. However, given that this evaluation only examined a subset of pan-Canadian organizations managed by Health Canada, consideration could be given to exploring how and whether these findings are relevant to the other pan-Canadian organizations supported by sustaining transfer payments.

Recommendations

The evaluation identified the following four recommendations:

Recommendation 1
The evaluation noted a desire for greater clarity on the part of program staff at various levels, about what the departmental expectations are regarding their appropriate role in, and approach to encouraging the alignment of the organizations' strategic direction and activities with federal priorities, while taking into consideration the perspectives of other stakeholders involved.

Articulate Health Canada's strategic approach to the use of pan-Canadian organizations, given their importance as policy levers, and implications of this approach for Health Canada's engagement with each of these organizations.

Recommendation 2
It was highlighted in the evaluation that collaboration and coordination among Health Canada staff managing the department's relationships with the pan-Canadian organizations could be enhanced in order to share lessons learned which would help minimize duplication, optimize effectiveness and increase clarity around how to engage with the pan-Canadian organizations.

Strengthen coherence and collaboration among Health Canada staff involved with managing the department's relationships with the pan-Canadian organizations to effectively implement the departmental strategic approach and foster the exchange of experiences and lessons learned in this regard.

Recommendation 3
Having more of a "single window" approach through the policy areas managing the department's relationships with the pan-Canadian organizations would provide a unified federal perspective when communicating with pan-Canadian organizations.

For each pan-Canadian organization, strengthen coordination within and across the department, portfolio, and other federal partners (where appropriate) to enhance internal communications and ensure a coherent federal perspective when interacting with that organization.

Recommendation 4
The evaluation noted that reporting requirements were often burdensome and the monitoring of these requirements detracted from broader policy related efforts. Therefore, it would be important to align the program management approach and reporting requirements with departmental information need to ensure sufficient, but not excessive, rigor.

Align/calibrate the approach to program management, including the type(s) and content of reporting requirements set out in funding agreements in support of pan-Canadian organizations to ensure it is coherent and appropriate to support effective program monitoring, management and decision making by Health Canada, while recognizing variations across the organizations (e.g. size, scope, funding model, governance, objectives).

Management Response and Action Plan

Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations
Recommendations	Response	Action Plan	Deliverables	Expected Completion Date	Accountability
1. Articulate Health Canada's strategic approach to the use of pan-Canadian organizations, given their importance as policy levers, and implications of this approach for Health Canada's engagement with each of these organizations.	Agree	Complete a policy review to identify options for a senior management discussion on Health Canada's approach and its implications for engagement with each of the pan-Canadian organizations.	Summary report of the policy review Record of Decision from senior management discussion	August 2014 - Summary report October 2014 - Senior management discussion	Director General, Health Care Strategies Directorate, and ADM, Strategic Policy Branch, in collaboration with other Director Generals responsible for pan-Canadian organizations
2. Strengthen coherence and collaboration among staff involved with managing the department's relationships with the pan-Canadian organizations to effectively implement Health Canada's strategic approach and foster the exchange of experiences and lessons learned in this regard.	Agree	Through the Policy Forum on Pan-Canadian Healthcare Organizations established in October 2013, maintain a community of practice to foster engagement and support implementation of Health Canada's strategic approach.	Community of practice Revised terms of reference for the community of practice, to support direction from senior management	Ongoing November 2014	Director General, Health Care Strategies Directorate, and ADM, Strategic Policy Branch, in collaboration with other Director Generals responsible for pan-Canadian organizations
3. For each pan-Canadian organization, strengthen coordination within and across the department, portfolio, and other federal partners (where appropriate) to enhance internal communications and ensure a coherent federal perspective when interacting with that organization.	Agree	For each pan-Canadian organization, develop a plan that builds on existing networks and processes, where applicable, to coordinate department, portfolio, and other federal partners' interactions with the health pan-Canadian organizations.	Coordination plans and supporting mechanisms	September 2014	Director Generals responsible for pan-Canadian organizations, and ADM, Strategic Policy Branch
4. Align/calibrate the approach to program management, including the type(s) and content of reporting requirements set out in funding agreements in support of pan-Canadian organizations to ensure it is coherent and appropriate to support effective program monitoring, management and decision making by Health Canada, while recognizing variations across the organizations (e.g. size, scope, funding model, governance, objectives).	Agree	Review program management approach and reporting requirements of each of the pan-Canadian organizations and based on that review develop a plan to clarify and support program management improvements, as required.	Coordination mechanism to undertake review and support ongoing knowledge exchange. Plan to strengthen management of programs supporting pan-Canadian organizations.	March 2015	Executive Director, Health Programs and Strategic Initiatives, and ADM, Strategic Policy Branch in collaboration with other Director Generals responsible for a pan-Canadian organization

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of certain transfer payments to pan-Canadian organizations, which were created and sustained by Health Canada, for the period of April 2008 to November 2013. The evaluation was conducted to fulfill Financial Administration Act (FAA) and the Treasury Board of Canada's Policy on Evaluation (2009) requirements for departments to conduct an evaluation of all ongoing grant and contribution programs every five years. Departmental evaluations help ensure that credible, timely and neutral information on the ongoing relevance and performance of programs is available to support evidence-based decision making on policy, expenditure management and program improvements. Pan-Canadian organizations were also required under their funding agreements to conduct their own independent evaluations of their achievement of the objectives of the funding agreement and these independent evaluations were used as a line of evidence in this evaluation, thereby reducing duplication of effort.

2.0 Program Description

2.1 Program Context

Over the last 25 years, the Government of Canada has created and sustained nine pan-Canadian organizations addressing a variety of healthcare issues. In some cases, the establishment of this arm's length machinery to address national interests in healthcare was the result of federal/provincial/territorial agreement, while others reflect federal initiative in a particular healthcare area. Some are solely funded by the federal government, while others, though predominantly federally supported, also receive funding from provinces and territories. Each pan-Canadian organization has a distinct shared governance model, involving different mixes of governments, experts, and stakeholders.

In Canada's decentralized healthcare system, these pan-Canadian organizations were intended to function at arm's length from the federal government, providing national coordination and collaboration on addressing system-wide priorities. The role and scope of these organizations differ, but they tend to provide either system-wide services/support (such as creating evidence-based advice to support decision making) or act as a catalyst for improvement in priority areas (e.g. providing leadership and intensifying progress on an issue). These organizations work with governments, stakeholders, experts and the public at the regional, provincial/territorial and federal levels. The section below will describe the pan-Canadian organizations and associated Health Canada transfer payments examined in this evaluation.

2.2 Program Profile

In total, there are nine transfer payment program for pan-Canadian healthcare organizations funded by Health Canada, each with its own policy objectives, terms of reference, and funding profile. The five pan-Canadian organizations receiving transfer payments that were examined in this evaluation include:

the Canadian Agency for Drugs and Technologies in Health (CADTH),
the Canadian Patient Safety Institute (CPSI),
the Mental Health Commission of Canada (MHCC),
Canada Health Infoway (Infoway), and
the Health Council of Canada (HCC).

Other transfer payments to pan-Canadian organizations managed by Health Canada's Strategic Policy Branch, such as the Canadian Partnership Against Cancer, the Canadian Institute for Health Information, the Canadian Centre on Substance Abuse, and the Canadian Foundation for Healthcare Improvement, were not included in this evaluation because they either recently completed, or will soon complete, a separate departmental evaluation.

Canadian Agency for Drugs and Technologies in Health
CADTH was established in 1989 by the Ministers of Health and received funding from the federal, provincial and territorial governments. CADTH was mandated to deliver evidence-based information to Canada's healthcare decision makers on the effectiveness and efficiency of drug and non-drug technologies in health (e.g. drugs, devices, systems, and services), and to support the adoption of these technologies. It was expected that CADTH would produce evidence and advice which would inform healthcare decision making and support the sustainability of the healthcare system.

From 2008 to 2013 Health Canada's grant to CADTH provided $87.5M of the organization's overall funding in order to continue support for its core business activities, namely:

Health Technology Assessment (HTA), which evaluates the clinical and cost effectiveness of health technologies (drugs and non-drug) while considering patient input. Its reports provide comprehensive peer-reviewed assessments of health technologies.
Common Drug Review (CDR), which through the independent Canadian Expert Drug Advisory Committee provides guidance to participating jurisdictions on whether, and pursuant to which criteria, to publicly reimburse drugs.
Canadian Optimal Medication Prescribing and Utilization Service (COMPUS), which identifies and promotes evidence-based, clinical and cost-effectiveness information on optimal drug prescribing and use. Strategies, tools, and services are provided to encourage the use of this information in decision making among policy makers, healthcare providers and consumers.

In 2013, CADTH's funding was changed from a grant to a contribution agreement. While the core business activities remains similar (Health Technology Assessment, Optimal Use and the Common Drug Review) the focus has shifted to Health Technology Management - not only the production of evidence but the sharing of the evidence and advice to support the clinical and cost effective adoption of technology in the healthcare system.

The focus as demonstrated by CADTH's most recent logic model, is on Health Technology Management and now includes brokering (building partnerships, outreach, and capacity building to share evidence, information and lessons learned) and producing HTAs (including producing formulary reviews, environmental scans, and knowledge mobilization activities). CADTH not only develops quality evidence but also supports how HTA evidence and the experiences of healthcare decision makers can be shared to support the clinical and cost effective adoption of healthcare technologies. The intended reach for CADTH's outputs included drug plan managers, healthcare providers, academics, and other healthcare decision makers. (See Appendix 1 for a detailed description of the logic model developed under the contribution agreement.)

Canadian Patient Safety Institute
CPSI was established by the federal government in 2003 in response to growing public concern about medical errors and a shared federal/provincial/territorial commitment to address patient safety in the 2003 Health Accord. CPSI was given $40M in grant funding from Health Canada over the period of 2008 to 2012.

CPSI was responsible for raising awareness and facilitating the implementation of innovations and best practices to improve patient safety. More specifically, CPSI was mandated to undertake work in four main activity areas/streams. The Education stream focused on providing education and training to health professionals to increase their awareness of patient safety issues. The Research stream focused on developing and supporting research projects on patient safety topics, and facilitating the uptake of the findings into healthcare policy and procedures. The Tools and Resources stream focused on providing tools and guidelines to healthcare leaders to improve patient safety practices. Lastly, the Interventions and Programs stream, which focused on engaging front-line healthcare workers involved in the delivery of healthcare in clinical care, patient safety, quality improvements to reduce patient variations and increase performance reliability. This stream also includes the flagship program Safer Healthcare Now!, which developed tools and resources to prevent harmful incidents in areas including hand hygiene, surgical site infections, and medication reconciliation.

These activities and outputs were expected to lead to strengthened coordination related to patient safety and a stronger culture of patient safety, resulting in improved healthcare safety and quality. (See Appendix 1 for a detailed description of the logic model).

Mental Health Commission of Canada
The MHCC was created by the Government of Canada through Budget 2007 and was provided with $130M in funding over ten years. In 2008, the federal government provided an additional $110M over five years through a separate grant to support five research demonstration projects in mental health and homelessness (At Home/Chez Soi). It is important to note that this evaluation includes both transfer payments.

The MHCC has a ten year mandate (2007 - 17) to act as a national focal point for mental health issues, as well as a catalyst for improving the mental health system and changing the attitudes of Canadians regarding mental health. It was created, in part, in reaction to a Senate recommendation that there be a national mental health commission for Canada. This recommendation was contained in the Senate Committee on Social Affairs, Science and Technology's 2006 report on mental health, mental illness and addiction.

Over the period examined by this evaluation, core funded activities of the MHCC focused on developing a National Mental Health Strategy (Changing Directions, Changing Lives released in May 2012); undertaking an anti-stigma campaign (Opening Minds); establishing a Knowledge Exchange Centre to facilitate access to reliable information on mental health; and conducting research projects in the area of mental health and homelessness (At Home/Chez Soi "Housing First" project).

Under these core activities, the MHCC pursued multiple projects and initiatives that were not explicitly identified in their grant agreements. These included activities relating to workplace mental health, such as the Mental Health First Aid training program and the National Standard for Psychological Health and Safety in the Workplace (released in January 2012), as well as emerging priority areas, such as suicide prevention.

The MHCC's activities were expected to lead to an increased awareness and understanding of mental health and mental illness by all people living in Canada (including health professionals, employers, and the general public), and reduced stigma and discrimination related to mental illness. Evidence from MHCC research was expected to inform the development of policy and service delivery and increase the capacity of decision makers to implement mental health policies. Overall, MHCC was expected to enhance the integration and coordination within the mental health system, leading to improvements in policies and practices. (See Appendix 1 for a detailed description of the logic model).

Canada Health Infoway
Established in 2001, after First Ministers agreed to make eHealth a priority, Infoway was created to accelerate the development and adoption of electronic health information systems with compatible standards across Canada. Through five separate federal government transfer payments, Infoway received over $2.1 billion over the past 12 years to fund projects with the provinces and territories that support the development and implementation of electronic health records (EHRs), electronic medical records (EMRs), a pan-Canadian Health Surveillance System, telehealth solutions, and other electronic health information and communication technology.

Infoway works with provincial and territorial governments to advance shared priorities and to promote alignment of each jurisdiction's eHealth strategies with national priorities. A shared initial focus of Infoway's activities was the establishment of a common architecture, standards and EHRs. As progress was made on this front, Infoway's scope of activities was expanded to include clinician adoption of EMRs at points of care. Infoway also develops analytical reports, and helps foster the adoption of health information technologies by clinicians. Increasing in number over time to reflect Infoway's expanding scope, Infoway now has 12 investment programs, including Infostructure, Interoperable EHR, Registries, Diagnostic Imaging Systems, Drug Information Systems, Laboratory Information Systems, Telehealth, Public Health Surveillance, EMR and Integration, Innovation and Adoption, Consumer Health Solutions and Patient Access to Quality Care. Under these programs, each province and territory can develop projects best suited to their respective circumstances and needs. Based on these programs, a jurisdiction will bring projects forward to Infoway for consideration.

The work funded by Infoway aims to improve access to health services, quality of healthcare delivery and productivity in the health system. Other anticipated impacts include more timely delivery of healthcare, increased productivity and interoperability, improved access to, and sharing of information, the creation of sustainable knowledge-based jobs, and a strengthened knowledge infrastructure within the healthcare system. (See Appendix 1 for a detailed description of the logic models).

Health Council of Canada
The HCC was established by the federal government in 2003 through a grant of $10M annually, to monitor and report on progress in implementing the Health Accords (particularly its accountability and transparency provisions), and report on the health status and health outcomes of Canadians. As a result of a Corporate Members' review^{Footnote A1} in 2010, HCC's mandate was expanded to focus on identifying and disseminating best practices and highlighting innovation in healthcare renewal. In the context of Budget 2013, and to coincide with the pending expiry of the health accords, the federal government decided to wind down funding to HCC.

The HCC provided a system-wide perspective on healthcare reform in Canada. HCC's immediate outcomes included: providing its target audiences with access to relevant information on innovative practices in healthcare, aiding its target audiences in becoming more informed and aware of the progress of health system renewal, and collaborating with governments, other healthcare organizations and the public. HCC's intermediate outcomes included: supporting policy, program and service change through information sharing, improving sharing and collaboration among F/P/Ts in progressing on healthcare renewal, and improving accountability of F/P/T governments. The Council's ultimate outcome was a strengthened and renewed Canadian health system.

The intended reach for the program was healthcare decision makers and the general Canadian public. (See Appendix 1 for a detailed description of the logic model).

2.3 Program Alignment and Resources

Transfer payments to pan-Canadian organizations were part of Health Canada's Program Alignment Architecture (PAA): Strategic Outcome 1.0 "A health system responsive to the needs of Canadians", under program 1.1 "Canadian Health System Policy", and sub-program 1.1.1 "Health System Priorities".

The transfer payments' financial data for the years 2008-09 through 2012-13 are presented in Table 1, which includes the federal funding amounts only and excludes any additional funding that these organizations may have received from other stakeholders. The materiality of each grant varied greatly. Collectively, these organizations received federal funds totalling approximately $840M over the five years examined in this evaluation. It should be noted that the allocated resources for these organizations did not include any additional or dedicated Health Canada operating and maintenance resources to manage these agreements.

Table 1: Financial data
Recipients	Planned Budget ($ millions)
Recipients	2008-09	2009-10	2010-11	2011-12	2012-13	Total
CADTH	16.9	16.9	17.9^{table 1 Footnote *}	18.9^{table 1 Footnote *}	16.9	87.5
CPSI	8.0	8.0	8.0	8.0	8.0	40.0
MHCC	117.5^{table 1 Footnote **}	12.0	15.0	15.0	15.0	174.5
Infoway^{table 1 Footnote ***}	123.0	64.5	91.8	87.9	122.5	489.7
HCC	10.0	10.0	10.0	10.0	8.0	48.0
Total:						839.7
Footnote * Includes $3.0M over two-years for a special project on isotopes. Return to table 1 footnote * referrer Footnote 2 Includes up front multi-year funding of $110M provided in 08/09 for the 'At Home/Chez Soi' homelessness and mental illness research project grant. Return to table 1 footnote referrer Footnote 3 Infoway received $1.2B as lump-sum transfer payments between 2001 and 2004. An additional $400M was allocated in 2007 and $500M allocated in 2009 (confirmed in 2010) consisted of up-front multi-year funding disbursed to Infoway based on annual / semi-annual cash flow requests. Return to table footnote * referrer

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from April 1, 2008 to November 30, 2013, and included the grant-funded activities of five pan-Canadian organizations: CADTH, CPSI, HCC, MHCC, and Infoway. The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) five core issues related to relevance and performance. Corresponding to each of the core issues, specific questions were developed (see Table 2), based on program and senior management information needs.

Table 2: Core Issues and Evaluation Questions
Core Issues	Evaluation Questions
Relevance
Issue #1: Continued Need for Program	Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians. What healthcare system needs are these pan-Canadian organizations addressing? What would happen if these needs were not met through these organizations?
Issue #2: Alignment with Government Priorities	Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes. Do the transfer payments' objectives continue to align with Government of Canada priorities? Do the transfer payments' objectives continue to align with Health Canada's strategic outcomes?
Issue #3: Alignment with Federal Roles and Responsibilities	Assessment of the role and responsibilities for the federal government in delivering the program. Are the transfer payments' objectives aligned with the federal and departmental roles and responsibilities?
Performance (effectiveness, economy and efficiency)
Issue #4: Achievement of Expected Outcomes (Effectiveness)	Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes. To what extent did the transfer payments achieve their expected outcomes?
Issue #5: Demonstration of Economy and Efficiency	Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Were the costs of outputs and outcomes reasonable in light of the program context for each of the transfer payments? (Examination of the cost/effort to manage each grant) Are there alternative approaches to meeting the transfer payments' objectives?

Since all of the pan-Canadian organizations were required to conduct an independent evaluation, the approach was to use the independent evaluations as the basis for this synthesis evaluation. Data for this evaluation were collected using various methods, including a literature review, a document review (including the pan-Canadian organizations' recent independent evaluations), and key informant interviews (n=18). Primary data collection was targeted and therefore limited to departmental staff. Organizations were not contacted for additional information in order to reduce their reporting burden (given they had all recently undergone independent evaluations).

The literature review analysis included both peer-reviewed and grey literature. An iterative review strategy was implemented, where articles from peer-reviewed journals and key government reports were initially sought. This was followed by a search of government/organization websites, using key words for each pan-Canadian organization. The information gathered was used primarily to address the issue of continued need for the pan-Canadian organizations.

Program documentation, including: pan-Canadian organizations' annual reports, progress reports, workplans, strategic plans, independent evaluations, audits, special studies, cashflow requests, funding agreements, as well as internal briefing notes and monitoring frameworks, were examined.

Key informant interviews were conducted with 18 departmental representatives, including Analysts, Managers, Directors, Director Generals, and Associate Deputy Ministers, to obtain information about their level of engagement with, and management of, the pan-Canadian organizations.

Data were analyzed by triangulating information gathered from the different methods listed above. Organizations were assessed individually for relevance and effectiveness, and resulting themes were aggregated. The economy and efficiency analysis provided an opportunity to examine the suite of organizations collectively and largely focused on Health Canada's engagement with, and management of, the transfer payments to pan-Canadian organizations. The analysis also took into consideration the context of each pan-Canadian organization (all differed in size, scope, and maturity), and comparisons between pan-Canadian organizations were not made. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Table 3: Limitations and Mitigation Strategies
Limitation	Impact	Mitigation Strategy
Limited primary data collection. Reliance on secondary data (e.g. independent evaluations and audits).	Secondary data might be missing/ unavailable to fully assess the proposed evaluation questions.	Mitigated by reviewing independent evaluation methodology and critically assessing conclusions, as well as, validating data with program theory, and using key informant interviews to fill data gaps.
Access to pan-Canadian organizations was limited in order to reduce reporting burden.	Most recent performance information may not be included in the analysis.	Validation of information was sought from Health Canada staff who manage the funding agreements to the pan-Canadian organizations.

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Transfer Payments to Pan-Canadian Organizations

There is a continued need to address most of the healthcare system issues supported by the transfer payments to pan-Canadian organizations.

Pan-Canadian organizations address specific areas of national interest and federal priority in Canada's healthcare system and these areas are complex, requiring sustained, long-term effort. The issues addressed by the pan-Canadian organizations were still relevant at the time the evaluation was conducted:

CADTH: There is a continuing need for evidence-informed decision making concerning the adoption of healthcare technologies (drug and non-drug).
CPSI: There is a continuing need for patient safety improvement efforts since Canada remains below the OECD average on a number of patient safety indicators.
MHCC: There is a continuing need for mental health activities because the prevalence and economic burden of mental illness remains significant in Canada: 1 in 5 Canadians will experience a mental illness every year, costing the Canadian economy an estimated $50 billion annually.
Infoway: There is a continuing need to harness the potential of electronic health technologies to improve healthcare and health system performance, for example through increased use in clinical settings and interoperable systems of records.
HCC: There was a need to monitor and publicly report on the government's progress under the Accords. However, with the Accords set to expire in 2014, reporting on progress of its implementation will no longer be necessary. It should be noted that HCC's corporate members have already agreed that the organization will cease operations as of June 30 2014 and HCC is working with other organizations to try to pass on some of their promising projects regarding innovation in healthcare.

The pan-Canadian organizations are well positioned to address national healthcare issues since there are no other national organizations with similar objectives. There is a need for national organizations in these areas since they provide pan-Canadian leadership, build on best practices and avoid duplication of efforts across jurisdictions. There are some organizations at the provincial/territorial level that have functions similar to those of the pan-Canadian organizations, such as the provincial quality councils and the provincial health technology agencies, but their scope is limited to the jurisdictional level.

Canadian Agency for Drugs and Technologies in Health
CADTH contributes to the quality and sustainability of healthcare in Canada by providing decision makers with evidence-based information, advice, and recommendations needed to make informed decisions regarding the appropriate use of drugs and other health technologies. The literature supports the fact that health technology assessments are a "critical component of evidence-informed policy decision making".^{Endnote 1}

The Canadian healthcare system is constantly evolving and the challenge remains to balance the implementation of new technologies to improve the quality of care while containing the cost of care. An independent evaluation found that CADTH stakeholders expect an "escalation in the volume and complexity of technology appraisals", causing them to look to CADTH for leadership in choosing those technologies that are most promising. Federal, provincial and territorial healthcare decision makers rely on CADTH's evidence-based advice to make informed policy decisions. There are 18 publicly funded drug plans covering 7.8 million^{Endnote 2} Canadians that rely on a pan-Canadian approach to make decisions about funding and listing pharmaceuticals. CADTH's Common Drug Review ensures that there is a consistent approach to the review of new drugs across Canada.

Part of the Common Drug Review's value to the healthcare system is that it eliminates duplication, maximizing expertise and resources. Key informants for this evaluation also noted that CADTH's reviews provide an important piece of evidence upon which drug reimbursement decisions are based. Even though there are larger jurisdictions that do have some capacity to conduct similar activities, CADTH continues to fill this need for smaller jurisdictions that may not have the capacity to perform this function.

Canadian Patient Safety Institute
The CPSI was created in 2003, at a time of growing concern for patient safety issues, particularly because of increases in awareness of adverse events including those related to medical interventions, devices, medications, infections acquired through the provision of healthcare, and broader system issues such as standards and practices. Although some progress has been made, for example several patient safety tools and interventions have been created, there is an ongoing need to carry on with the work of the CPSI. Notably, a recent report entitled Benchmarking Canada's Health System: International Comparisons (2013) states that Canada ranks below the Organisation for Economic Cooperation Development (OECD) average on several patient safety indicators. In particular, Canada is among the worst for leaving objects such as sponges or medical equipment in patients after surgery (ranking 17th out of 20 countries). Obstetric trauma was also cited as a key problem area. The OECD measures obstetric trauma with an instrument (deliveries with forceps or vacuum extraction) and without an instrument. Compared to other OECD countries, Canada performs poorly with respect to obstetric trauma both with and without an instrument (20th and 18th out of 21 respectively).^{Endnote 3,} ^{Endnote 4,} ^{Endnote 5}

Key informants for this evaluation noted that patient safety issues are spread across the country; therefore it is important to have a pan-Canadian focus with standard interventions, resources and tools. The independent evaluation found that there is a continued need for CPSI, with interviewees and survey respondents noting that there would be significant losses to the system should CPSI no longer carry out its patient safety activities. Some of the impacts of this loss would be the risk of declining patient safety, decreased awareness of patient safety issues, and a slowdown or loss of momentum for improving patient safety. Furthermore, the independent evaluation found that there are varying capabilities and capacities within and between sectors, governments, and agencies to address this need. The provinces and territories do not necessarily have access to the network of experts that CPSI has, nor do all of them have the capacity to develop and implement patient safety interventions; however, CPSI fills this need.

CPSI is seen by its stakeholders as delivering value and filling a role within the healthcare system as a distinct and leading patient safety organization. Further, the independent evaluation showed strong support for CPSI's role and a clear need for having a Canadian organization solely focused on patient safety. It found that there are a lot of organizations and initiatives focused on quality and patient safety, yet CPSI continues to be viewed as having an important contribution: 73 per cent of those interviewed consider CPSI to be the leading, or one of the leading organizations in patient safety in Canada because of its efforts to build a culture of patient safety and quality improvement. The evaluation found that CPSI is viewed as being "instrumental in raising awareness on patient safety issues and providing relevant, evidence based, timely and quality patient safety resources to the broader health community".^{Endnote 6}

Research continues to show that adverse events come at significant costs, both to the healthcare system, as well as human costs. The Economics of Patient Safety in Acute Care: Technical Report estimated that the economic burden of adverse events in Canada for 2009-10 was nearly $1.1 billion, including approximately $397 million for preventable adverse events.^{Endnote 7} These costs could potentially be decreased with improvements in patient safety.

Mental Health Commission of Canada
In May 2006, the Senate Standing Committee on Social Affairs, Science and Technology released a report on mental health and mental illness. One of the key recommendations of that report was the creation of the Mental Health Commission of Canada (MHCC) to provide a national focal point for addressing mental health issues.^{Endnote 8,} ^{Endnote 9} Key informants for this evaluation reaffirmed the fact that the MHCC continues to appropriately fulfill its mandate to act as a national focal point for mental health issues, as well as a catalyst for improving the mental health system and changing the attitudes of Canadians regarding mental health.

Mental health problems often start in childhood. It is estimated that 70 per cent of adults diagnosed with a mental illness first began to experience it in childhood or adolescence.^{Endnote 10,} ^{Endnote 11,} ^{Endnote 12} In any given year one in five Canadians will experience a mental health problem or illness, costing the Canadian economy an estimated $50 billion.^{Endnote 13} In 2012 approximately 2.8 million people, or 10 per cent of Canadians aged 15 and older, reported symptoms consistent with at least one of six mental health or substance use disorders in the past 12 months.^{Endnote 14} Mental health problems have far reaching effects. For example, in any given week, at least 500,000 employed Canadians are unable to work due to mental health problems.^{Endnote 15}

Mental illnesses can have tragic consequences. Nearly 4,000 Canadians die by suicide each year, an average of 11 suicides a day.^{Endnote 16,} ^{Endnote 17} After accidents, suicide is the second leading cause of death for young Canadians aged 10-19. Youth suicide is a serious problem^{Endnote 18} with Canada's youth suicide rate ranked third highest in the industrialized world.^{Endnote 19} First Nations youth die by suicide about 5 to 6 times more often than non-Aboriginal youth in Canada. Suicide rates for Inuit youth are among the highest in the world, at 11 times the national average.^{Endnote 20} Every suicide is a tragedy that has a far reaching impact on family, friends and the community long after a person has died.^{Endnote 21}

The MHCC has made significant progress in its four core activity areas; however, work remains as the burden of mental illness continues to be high. Many people who have a mental health problem suffer in silence due to the stigma associated with mental illness. The World Health Organization declared stigma the single most important barrier to overcome.^{Endnote 22} Stigma remains a barrier not only to diagnosis, but also to treatment. It is estimated that almost 49 per cent of those who feel they have suffered from depression or anxiety have never sought treatment because of the stigma attached to the illness.^{Endnote 23} There is public stigma based on deep seated prejudices, but there is also self-stigma which includes feeling ashamed and blameworthy, which can lead to concealing an illness.^{Endnote 24,} ^{Endnote 25}

Canada Health Infoway
Canada Health Infoway was created as a catalyst for innovation in the healthcare system, doing so through the development and implementation of electronic health technologies, such as EHRs, EMRs, and telehealth. eHealth is seen as an important way to modernize healthcare in Canada and while progress has been made in eHealth (e.g. EHRs, EMRs), the systems are not fully utilized and connected. Further advancement on these fronts would harness the potential of eHealth for healthcare by enabling seamless access to information that supports coordinated and quality care.

It has been reported that between 2006 and 2012, the use of EMRs has more than doubled, going from 23 per cent to 56 per cent, resulting in estimated savings to the medical system of $1.3 billion.^{Endnote 26} These savings were realized in increased administrative efficiencies, increased health system benefits, improved chronic disease management and illness prevention, and improved communication among healthcare providers and with patients. Despite this progress, Canada still lags behind several comparator countries in terms of primary care physician adoption of EMRs (ranking 9th out of 10 countries, with the top four countries having 97 per cent or more of its primary care physicians using EMRs).^{Endnote 27,} ^{Endnote 28}

There is a need for a pan-Canadian approach that sets up national standards. A recent survey commissioned by Infoway noted that 96 per cent of Canadians think it is important that the healthcare system make use of digital health tools and capabilities.^{Endnote 29} Infoway provides that pan-Canadian approach to move the eHealth agenda forward, allowing for eventual integration of electronic health records across Canada. Infoway helps to foster and accelerate the development of EHRs across Canada, working with the provinces and territories, which are responsible for delivering healthcare. The 2010 Auditor General audit reported that Infoway sets the national direction and ensures that provincial and territorial strategies are aligned with national priorities.^{Endnote 30}

An independent evaluation noted that in the absence of Infoway, there would be a drastic slowdown of progress on EHRs. The evaluation found that the pan-Canadian approach to eHealth would be lost without Infoway, making seamless patient care, particularly between provinces, almost impossible.

Health Council of Canada
A number of F/P/T priority areas and action items were identified in the 2003 and 2004 Accords. Therefore, the Health Council of Canada, arm's length pan-Canadian organization, was established to monitor and report on the implementation of these priority areas and action items through its annual reports and other reports on the health status and outcomes of Canadians. The establishment of the Health Council was a key commitment of the Accords. With the Health Accord set to expire in 2014, there is no longer a need for the Health Council to report on Canada's progress on its implementation. There are other organizations, including provinces and territories and the Canadian Institute for Health Information, that also report on health system performance. In addition, there are six provincial quality councils that report on the health system.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

The activities of the pan-Canadian organizations align with Government of Canada and Health Canada priorities.

The objectives of the transfer payments to pan-Canadian organizations aligned with the federal government's overall priorities as well as with Health Canada's PAA strategic outcome 1.0 "A health system responsive to the needs of Canadians".

Objectives of the transfer payments were highlighted in recent Budget announcements and Speeches from the Throne. The 2010 Speech from the Throne stated that "protecting the health and safety of Canadians and their families is a priority of our Government." Budget 2012 also committed to "Supporting families and communities by protecting the health and safety of all Canadians and their communities". In addition to the 2010 Speech from the Throne, the 2011 Speech from the Throne highlighted the health and safety of Canadians as an important priority for the government, including a sustainable healthcare system. More recently, Budget 2012 and 2013, as well as the 2013 Speech from the Throne reiterated that supporting families and communities by protecting health, and specifically patient safety, is a key priority for the current federal government. These statements confirmed the importance that the Government of Canada placed on the need to protect and sustain the healthcare system. Given that these improvements in the healthcare system will require longer term efforts with a national focus, the funding of pan-Canadian organizations is aligned with the overall federal government strategy in the area of healthcare.

Canadian Agency for Drugs and Technologies in Health
As mentioned above, the 2011 Speech from the Throne highlighted the need to have a sustainable healthcare system and with technologies and pharmaceuticals being a huge cost driver for healthcare, CADTH's work to optimize the use of effective technologies (both drug and non-drug) is key to cost reductions in the delivery of care. In addition, the government reaffirmed its support of CADTH's activities in Budget 2013, or more specifically in the Economic Action Plan 2013, which highlighted the government's ongoing support of innovation through its commitment to advanced research and technology.

Canadian Patient Safety Institute
CPSI's priorities (i.e. to raise awareness and facilitate the implementation of innovations and best practices to improve patient safety) are aligned to patient safety themes that were set out in the October 2013 Speech from the Throne, which announced the government's intent to have new patient safety legislation. As a follow up, in December 2013, the Minister of Health introduced new patient safety legislation delivering on the 2013 Speech from the Throne commitment on Mandatory Adverse Drug Reaction Reporting, Recalls and Accurate Labels.

Mental Health Commission of Canada
In Budget 2007, the Government of Canada made a commitment to fund the establishment of the Mental Health Commission of Canada (MHCC) as a national focal point for addressing mental health issues. The Economic Action Plan 2013 noted the importance of enhancing mental health services in First Nations Communities, ensuring a culturally appropriate approach as identified in the Mental Health Strategy. The Action Plan also referenced the time limited funding granted to the At Home/Chez Soi component of the MHCC and its continuation and transfer to Employment and Social Development Canada. The MHCC remained relevant at the time the evaluation was conducted by continuing to respond to emerging federal priorities. For example, suicide prevention became a priority for the Commission, as outlined in its business plan, through both informal and formal consultations with PHAC and Health Canada.

Canada Health Infoway
Emerging more than a decade ago as a shared priority for federal, provincial and territorial collaboration following the First Ministers' 2000 agreement, the federal government has made multiple investments in the area of eHealth. This is evidenced by the many rounds of federal funding provided to Infoway to support the evolution of Infoway's activities, which first focussed on EHR architecture and has now been expanded to include clinician adoption of electronic records at points of care.

In addition, Budgets 2009 and 2010 both linked the investment in Infoway to important federal government priorities related to the health of Canadians and Canada's economy. Budget 2009 highlighted the importance of EMRs and EHRs in achieving an efficient and effective healthcare system, which was further reinforced in Budget 2010. More recently, Budget 2013 and the Economic Action Plan 2013 confirmed the government's commitment to the increased use of technologies such as telehealth, particularly in First Nations Communities.^{Endnote 31} In addition to the Budgets, EHRs and telehealth were clearly stated as being key to health system renewal, particularly in rural and remote communities.^{Endnote 32,} ^{Endnote 33}

Health Council of Canada

In Budget 2004, the federal government committed funding to healthcare and healthcare reform activities in support of the 2003 Health Accord, which included funding for the creation of the Health Council of Canada to monitor and report on progress made in fulfilling Accord commitments in areas such as, primary and home care, health human resources, pharmaceuticals and health innovation, care in the North, and Aboriginal Health. Under the 2004 Accord, HCC's mandate was expanded to cover commitments in other areas, such as wait times, access to care in the north and public health. More recently, the 2011 Speech from the Throne reiterated the government's commitment to ensuring accountability for results while respecting provincial jurisdiction for healthcare.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

The objectives of Health Canada's transfer payments to pan-Canadian organizations are well-aligned with federal and departmental roles and responsibilities.

Under the Department of Health Act (S.C. 1996, c.8), the Minister of Health has a mandate to promote the health of Canadians and protect Canadians against health risks, including:

the promotion and preservation of the physical, mental and social well-being of the people of Canada,
the protection of the people of Canada against risks to health,
the establishment and control of safety standards and safety information requirements for consumer products and for products intended for use in the workplace,
the collection, analysis, interpretation, publication and distribution of information relating to public health, and
the cooperation with provincial authorities with a view to the coordination of efforts for preserving and improving public health.

The Department of Health Act (1996) sets out the Minister of Health's duties such as the collection, analysis and dissemination of information relating to public health, which was reinforced in the Kirby Report (2003): "An effective health protection and promotion infrastructure also requires a strong capacity to communicate authoritative, evidence-based, information."^{Endnote 34}

In Canada, provincial and territorial governments are primarily responsible for managing and delivering health care. Through the principles and conditions set out in the Canada Health Act (R.S.C. 1985, c. C-6) and by encouraging national strategies and national coordination to ensure universality and the reduction of fragmentation in areas of need, Health Canada influences health care policy across Canada. Furthermore, the June 2011 Speech from the Throne reiterated that the Government of Canada is committed to respecting provincial jurisdiction and working with the provinces and territories to ensure that the healthcare system is sustainable and that there is accountability for results. As noted in the Government Response to the Senate Review of the 2004 Health Accord, the federal government is taking action to improve healthcare and public health in a way that respects jurisdictional roles and responsibilities, and that acknowledges the invaluable contribution of healthcare providers and other stakeholders. Through the pan-Canadian organizations, the federal government makes investments to help the provinces and territories, and other health system actors, accelerate change in areas of shared priority.

Health Canada manages these transfer payments to key pan-Canadian partners, contributing to priority health issues requiring national leadership and strong partnership. The Government of Canada is the primary funder of a number of these pan-Canadian health organizations, which work in partnership with provinces and territories, experts, and health care providers to support the translation and application of knowledge for use in the healthcare system.

Canadian Agency for Drugs and Technologies in Health
CADTH delivers timely, evidence-based information to healthcare leaders about the effectiveness and efficiency of health technologies, including pharmaceuticals, which aligns with the federal government role of providing evidence-based information. Health Canada continues to fund CADTH for activities such as the Common Drug Review because the federal government recognizes that the appropriate use of drugs is an essential part of a safe and cost-effective healthcare system. Furthermore, key informants for this evaluation noted that CADTH's role in providing recommendations to the drug plan managers for federal populations, such as First Nations Peoples, is essential. This, in addition to its primary role of minimizing duplication of expertise, is an important federal role.

Canadian Patient Safety Institute
Enhancing patient safety is an essential part of Health Canada's commitment to ensure a quality healthcare system for all Canadians. Health Canada's role, as a catalyst for system change, with respect to patient safety includes regulation of products, surveillance, research and policy development, information dissemination, and direct service provision for certain populations (e.g. First Nations and Inuit populations). Health Canada supports CPSI to improve the quality of healthcare services by strengthening system coordination related to patient safety, including promoting national collaboration among key players.

Mental Health Commission of Canada
In 2006, the Standing Senate Committee on Social Affairs, Science and Technology released Out of the Shadows at Last, the final report of its two-year study of mental health and mental illness in Canada. The report outlined a number of recommendations related to the federal role including the need for the creation of a Canadian Mental Health Commission to provide a national focal point for addressing mental health issues. Activities of the MHCC are aligned with the federal government role of providing leadership in the development of national mental health strategy, as well as, facilitating the exchange of information on best practices and assisting in the coordination across sectors and jurisdictions.

Canada Health Infoway
Infoway is mandated to foster and accelerate the development and adoption of electronic health information systems with compatible standards on a pan-Canadian basis. National leadership was essential to determine standards collectively and facilitate the adoption of electronic health systems across Canada. Infoway's other work as a catalyst for innovation continues to align with the federal role in health.

Health Council of Canada
HCC has a mandate to monitor and publicly report on the government's progress under the Health Accords, as well as on health outcomes and the health status of Canadians. However, this mandate will come to an end with the expiry of the Accords in 2014. While monitoring and reporting on the health status of Canadians remains an appropriate role for the federal government, there are other departments/organizations, such as Statistics Canada and the Canadian Institute for Health Information, that fulfill this role.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent were pan-Canadian organizations' outcomes achieved?

Strong evidence that pan-Canadian organizations were developing their outputs and achieving immediate outcomes; less evidence of longer-term outcome achievement.

While each pan-Canadian organization was created separately, supported by a discrete Health Canada program, and each had its own set of specific objectives, they were all working towards improving various aspects of the healthcare system. In this evaluation, the theory of knowledge translation was applied to describe how the organizations' activities could lead to this type of broad change. The Canadian Institutes of Health Research defines knowledge translation as the complex process of the "exchange, synthesis and ethically sound application of research findings (knowledge) within a complex set of interactions among researchers and knowledge users." In the literature, the concept of 'knowledge' is broader to include any type of product created by the organization (herein referred to in this evaluation as 'knowledge products'), such as tools, guidelines, standards, technologies, etc. The pan-Canadian organizations' objectives and outcomes generally aligned with the following stages along the knowledge translation or change continuum:

Creation and dissemination of knowledge products,
Increased awareness and understanding of knowledge products,
Increased collaboration and coordination,
Increased use and/or adoption of knowledge products, and
Improvements to policies and/or practices.

The evaluation examined the extent to which the pan-Canadian organizations' activities focused on different stages along the continuum and whether they were able to demonstrate contributions to the longer-term outcome of improving the healthcare system (e.g. quality, safety, sustainability, accessibility, accountability). Based on an analysis of the evidence from the document review and interviews, the diagram below depicts where each of the organizations was found to lie on this continuum. Where the organization is placed along the continuum directly relates to the focus of the activities undertaken by the pan-Canadian organization and its ability to create change, which is impacted by several factors, including the maturity of the organization (i.e. some organizations have been in existence for over 15 years, others just over six years) and the scope of the issue being addressed.

Table 4: Progress Along the Knowledge Translation or Change Continuum
Creation and dissemination of knowledge products	Increased awareness and understanding	Increased collaboration and coordination	Increased use and/or adoption	Improved policies and/or practices	Improvements in the health care system
Progression	Progression	HCC
Progression	Progression	Progression	Progression	MHCC
Progression	Progression	Progression	Progression	Progression	CADTH CPSI Infoway

The document review and interviews revealed strong evidence that pan-Canadian organizations were producing a large number of outputs, and were frequently achieving their immediate outcomes. However, less evidence was available detailing progress towards longer-term outcome achievement. Objectives in workplans were activity-based, so progress and annual reports (which report on progress relative to the workplans) often explained what activities were completed and which outputs were created, but provided fewer details regarding who used these outputs (i.e. increased use and adoption of knowledge products) and what improvements had been achieved (i.e. improvements to policies and/or practices, or in the healthcare system more broadly). This is somewhat expected as the longer-term outcomes relate to broader system changes, which typically take 10+ years to achieve.

A systematic effort would be required by the pan-Canadian organizations to be able to track and capture evidence of longer-term outcome achievement. Not all pan-Canadian organizations included in this evaluation were required to develop or implement a performance measurement framework (including tracking and reporting on longer-term outcomes) as part of their funding agreement; however, in 2013-14, this has become a requirement for renewed/new agreements.

Evidence of each pan-Canadian organizations' progress toward outcome achievement, as they relate to the stages along the continuum, will be described in the next sections.

Canadian Agency for Drugs and Technologies in Health
CADTH's key objectives were to build capacity to promote the optimal use of drugs and other health technologies, as well as to serve as a broker, helping to create and nurture an environment for evidence generation and adoption. CADTH achieved this through its two activity streams: the Common Drug Review, and the Health Technology Assessment/Optimal Use (HTA/OU).

Under the grant, the main outputs created under the CDR were formal drug plan formulary listing recommendations for participating public drug plan managers (except Quebec). In 2012-13, CDR produced 33 recommendations, in line with CADTH's target of between 30-35 outlined in their workplan. The HTA/OU stream created a broader array of outputs, including: Rapid Response Service reports, Reports in Brief, Bulletins, Webinars and presentations at continuing education events, newsletters, environmental scans, and an annual CADTH symposium.

While the level of awareness and understanding about CADTH's products was not reported in the progress or annual reports, there were other proxy measures to suggest that CADTH has increased awareness/understanding in some areas. For instance, the number of participants in the CADTH Symposium increased to over 600 in 2012 (up from over 300 in 2008). It also highlighted the top 10 downloaded documents in 2012-13, with the most popular document (the 2013 Preliminary Symposium Program) downloaded over 5200 times. It may be useful to know in the future which audiences are accessing these reports (e.g. health professionals, decision makers) and how they are using them. According to the independent evaluations, some users were not fully aware of CADTH's products, resulting in missed opportunities for CADTH to have an impact on healthcare system decision making.

In terms of increasing collaboration and coordination, CADTH was actively engaged with several different organizations, both within Canada (including health professional associations, provincial quality councils, pan-Canadian organizations, Health Canada) and with international networks. CADTH established several committees that brought experts together to collaborate on common issues: Canadian Drug Expert Committee, Drug Policy Advisory Committee, a Policy Forum, and the Exchange (included representatives from seven provinces).

This collaboration and coordination helped CADTH understand the needs of their knowledge users and broader healthcare issues/challenges, which helped increase the level of use and/or adoption of their CDR recommendations. Over the last five years, participating jurisdictions' drug plan listing decisions were congruent with CDR recommendations on average about 90 per cent of the time, with Atlantic provinces typically experiencing higher congruency rates than jurisdictions with their own HTA body (e.g. ON and BC).

Several examples of how HTA products were used and adopted across Canada were identified in the document review. However, it was too early for the impacts of these to be reported. Here are a couple of illustrative examples:

Fraser Health (in BC) adapted CADTH patient pamphlets and other tools on hip protectors to include in local information and evidence.
A Regional Health Authority in Saskatchewan incorporated CADTH's evidence/tool into a long term care falls prevention strategy.

One of the most significant HTA reports was on the use of Self-Monitoring Blood Glucose (SMBG) test-strips, which demonstrated that the health of patients with type 2 diabetes (not using insulin) did not change with reduced usage of test strips. While no policy changes have occurred yet, several jurisdictions are using the report and associated tools to create educational campaigns and to create a patient education portal. These jurisdictions have also developed structured academic detailing programs that focus on changing clinical practice around test strip usage. One jurisdiction used the report to support a previous decision to limit test strip usage, and another jurisdiction reversed its decision to limit test strip usage due to advocacy group push back (demonstrating that other barriers for jurisdictions to adopt CADTH's recommendations exist). As a result, CADTH reported that costs for test strips in these jurisdictions either decreased (-4 per cent) or experienced declines in growth (+3 per cent vs. +7 per cent previously). In comparison, costs for test strips in jurisdictions without academic detailing programs were growing at a rate of more than five per cent a year between 2008 and 2011.

CADTH's Proton Pump Inhibitor report led to policy changes in approximately half of participating jurisdictions. According to one of the independent evaluations, there has been a decrease in spending on proton pump inhibitor (PPI) drugs since the release of the report, even while diffusion of PPI drugs has increased (resulting in improved access for patients). Jurisdictions have also simplified the reimbursement process for PPI drugs by listing them on the general formulary (thereby increasing the efficiency of the healthcare system).

Some improvements to the healthcare system related to CADTH's work have been observed. Most notably, the CDR recommendations have led to increased consistency of drugs listed on jurisdictional formularies, as well as increased efficiency by reducing duplication of efforts and helping jurisdictions allocate resources efficiently. Also, these recommendations allowed drug plans to derive maximum value for tax dollars spent (e.g. recommending a drug be listed, but at a lower price), which contributed to the sustainability of the healthcare system.

Canadian Patient Safety Institute
CPSI made progress in achieving its outputs and outcomes, with some examples of how its work contributed to improvements to the safety and quality of the healthcare system identified in the course of the evaluation. In this section, some examples of outputs in each of CPSI's four streams will be described, and outcomes achieved will be highlighted as they relate to the stages along the knowledge translation continuum.

Education:
For example, the Patient Safety Education Program was a 2.5 day event that certified members of interprofessional teams as patient safety trainers. In 2011-12, the core curriculum was enhanced to include five new modules on patient safety in mental health. Levels of awareness and understanding were not directly measured, but several activities, such as four Patient Safety Education Program Canada conferences held in 2012-13 with more than 200 trainers in attendance, aimed to increase participants' awareness and understanding of patient safety issues.

In collaboration with the Royal College of Physicians and Surgeons of Canada and many key content experts, CPSI developed an information technology-based mapping tool and process to help educators in integrating safety competencies into undergraduate curricula. As of 2013, eight faculties of pharmacy, nursing, and medicine, and a pediatric specialty program used this tool and process to integrate patient safety content into their curricula. The Canadian Medical Protective Association mapped the safety competencies to their Good Practices Guide. In addition, these competencies were mapped to the accreditation standards for the Canadian Association of Schools of Nursing.

Tools and Resources:
Some examples of outputs under this stream included: Global Patient Safety Alerts, the Canadian Incident Analysis Framework, and the Patients for Patient Safety Canada initiative, which was a network of patients (56 in 2012-13) that provided CPSI with credible learnings and patient insights into patient safety and quality initiatives.

The document review and interviews identified the development of the Canadian Disclosure Guidelines (developed in 2008, updated in 2011) as one of CPSI's major accomplishments. The Guidelines are CPSI's most accessed publication. The independent evaluation noted that "33 pan-Canadian and provincial/territorial organizations endorsed the Guidelines, which have been credited with changing disclosure policies and protocols across Canada, including the introduction of apology legislation now in several provinces and territories".

Research:
CPSI has commissioned over 70 research projects on various patient safety topics (e.g. patient safety in homecare, economics of patient safety). However, no details describing who used these reports or what decisions these helped inform were included in the documentation reviewed.

Interventions and Programs:
Safer Healthcare Now! (SHN) was CPSI's flagship program, with between 35 and 50 per cent of its program expenditures dedicated to it in fiscal years 2010-11 through 2012-13. Tools and resources developed for SHN interventions were designed to prevent harmful incidents. SHN brought providers and experts together to make improvements in 11 areas, including hand hygiene, surgical site infections, and medical reconciliation.

In 2012-13, 719 organizations across Canada were voluntarily enrolled in SHN. These organizations were located in hospitals and community care settings and had 1,638 distinct teams implement interventions that contributed to improved patient safety. The medication reconciliation (i.e. a formal process to consistently review patients' medication history across transitions of care) intervention was the most highly subscribed to intervention in SHN, in part due to the alignment of Accreditation Canada's required organizational practices with CPSI's guidelines for medication reconciliation. CPSI, in collaboration with the Institute of Safe Medication Practices - Canada, developed a National Medication Reconciliation Strategy, which was endorsed by ten national organizations through a joint consensus statement related to the impact of communication failures in medication information.

Improvements in policies and practices were achieved through the SHN initiatives. Results from institutions participating in two SHN surgical infections-related interventions revealed that the percentage of surgical patients who received timely antibiotic administration (a practice known to reduce surgical site infections) increased from 57 per cent in 2006 to an average of 98 per cent in 2013. The SHN interventions also contributed to improvements in patient outcomes whereby the percentage of clean surgery patients who later acquired surgical infections decreased from seven per cent in 2006 to two per cent in 2013, and the rate of central-line associated bloodstream infections was reduced from 3.8 infections/1000 central-line days in 2006, to 0.6 infections/1000 central-line days in 2013.

CPSI also had a STOP Infections Now! initiative, including a STOP Clean Your Hands Day educational campaign (with 1000 sites registered in 2013), a Hand Hygiene Tool Kit, and the STOP Infections Now Collaborative with 22 teams.

The independent evaluation noted that system users were seeing policies, programs, and other changes related to an increased focus on patient safety and many were able to recognize the role of CPSI in delivering these, from the front line to management. Of the front line staff engaged during the independent evaluation, many were either using, or involved in supporting, CPSI's products and services. The independent evaluation also reported that 80 per cent of survey respondents indicated complete or partial agreement that CPSI had improved patient safety for Canadians.

Mental Health Commission of Canada
The MHCC mainly demonstrated progress towards achievement of their immediate outcomes, with some anecdotal evidence to support use and adoption of their products. This is not surprising given that this pan-Canadian organization was created in 2007, and longer-term outcomes tend to require more time to be achieved. In addition, its last independent evaluation was completed in 2011, just four years into its 10-year mandate. As such, the evaluation did not assess their long-term outcomes.

This section will outline outputs in each of MHCC's activity streams, and immediate/ intermediate outcomes achieved will be highlighted as they relate to the stages along the knowledge translation continuum.

Since one of the MHCC's objectives related directly to raising awareness of mental health and the work of the MHCC, some early findings were available in the independent evaluation. Most respondents to a 2011 survey of MHCC target audiences (n= 463) were aware of the work of the MHCC (85 per cent), and understood its mandate (81 per cent). Some respondents, particularly the media (80 per cent), MHCC volunteers (67 per cent), family members (64 per cent), and those with lived experience (64 per cent), expressed a desire to know more about the MHCC. Some of the activities undertaken by the MHCC since the survey was conducted included: presentations at conferences and professional associations, free webinars, and engagements with people with lived experience. These activities were expected to increase clients' awareness of the MHCC's work.

Another area where the MHCC increased awareness of mental health and mental illness was through their Mental Health First Aid program. This program helps participants recognize the signs and symptoms of mental health problems, provide initial help, and help guide a person towards appropriate professional help. As of March 2013, more than 70,000 individuals and 800 instructors have participated in the program and have become more aware and better able to help individuals experiencing mental health issues. The MHCC developed specific components for various target populations, including youth, seniors, Northern Peoples, and First Nations. In 2012-13, more than 100 agencies had contacted the MHCC about training opportunities for their staff, and about 70 were finalizing plans for training.

The At Home/Chez Soi research demonstration projects offered Housing First programs to people with mental illness experiencing homelessness in five pilot cities: Vancouver, Winnipeg, Toronto, Montreal, and Moncton. Between November 2009 and March 2013, over 1,000 people had been provided housing through these projects. By 2011, rates of homelessness decreased in Vancouver, with the At Home/Chez Soi project cited as one of the reasons. The Housing First approach projects were beginning to capture the attention (and increase awareness and understanding of mental issues) of stakeholders in other cities and internationally. Knowledge gained from the Housing First approach was shared with stakeholders and the wider community through presentations at local and provincial forums, national and international conferences, and in numerous papers and publications. However, findings from the final project report were not available in time to include in this evaluation, so any impacts of this were not included. Early findings suggested that this project was beginning to help guide plans, budgets, and decisions to address homelessness at local, provincial, and federal levels. For example, while the grant ended in 2013, the pilot projects were transitioned to Human Resources and Skills Development Canada and provinces and territories have continued to run the pilot projects in their original sites, and some have expanded the project to other sites. This demonstrates increased use and adoption of knowledge, yet it is too early to tell if this will result in broader policy changes within jurisdictions.

In 2012, the MHCC released one of their most significant accomplishments - the development of a National Mental Health Strategy, which involved extensive public consultations. The Strategy was supported by the majority of mental health stakeholders. The Strategy identified 26 priorities and 109 recommendations for action to transform the mental health system in Canada. The MHCC reported that media coverage of the release of the National Mental Health Strategy was 'unprecedented'. More than 6,600 copies of the Strategy and 8,600 summary copies were distributed, and articles about the Strategy were published in a variety of academic, professional consumer and knowledge exchange publications. The Strategy (and its earlier framework) served as a catalyst for all but two provinces/territories to have updated, developed, or begun developing their own provincial/territorial mental health strategies, and inspired one jurisdiction to develop tools for teachers and students across 72 school boards.

Another initiative making progress towards increased use and adoption of knowledge products is the Opening Minds (OM) anti-stigma campaign. Established in 2009, this campaign aimed to address stigma among four target groups: healthcare providers, youth, the workforce and the media. Some of the work in this initiative focused on evaluating existing anti-stigma programs across the country to identify promising practices, such as the use of 'contact-based education', where health professional curricula incorporate opportunities for students to interact socially with a person living with mental illness. Partnerships were forged with governments, universities, and other organizations (e.g. colleges, CMHA-ON, Mental Health Works, WHO, regional health authorities, Nova Scotia College of Family Physicians) to evaluate and implement anti-stigma programs. In total, 95 programs were evaluated. Already some of these programs were being replicated across the country, for example, the Ontario Shores Mental Health Centre and their Talking About Mental Illness (TAMI) program for youth was linked to a group of territorial government leaders to assist them in providing help to youth in the north. Also, an Ontario organization with a program to reduce stigma among hospital staff was used recently in all of the interior BC hospitals.

The National Standard of Canada for Psychological Health and Safety in the Workplace (herein referred to as the Workplace Standard) was highlighted as another key accomplishment of the MHCC. The Workplace Standard was a set of "tools not rules" providing guidelines, tools, and resources to help employers promote good mental health in the workplace. As of March 2013, the Workplace Standard had been downloaded almost 10,000 times from 15 different countries. Media coverage on the day the Workplace Standard was launched resulted in more than 168 references in print, online, radio and live video coverage. Early adopters of the Workplace Standard included: Bell Canada and the Centre for Addiction and Mental Health. Furthermore, in 2012, the Standards Council of Canada approved the Workplace Standard, confirming that it meets quality benchmarks for a national standard.

Lastly, the Knowledge Exchange Centre is aimed to advance the MHCC's work by facilitating collaborative partnerships with stakeholders that enable the sharing of knowledge and best practices. It experienced less progress than the other initiatives, in part due to a delayed start. The SPARK (Supporting the Promotion of Activated Research and Knowledge) Training Institute was launched in 2012, to help ensure that sharing and exchanging knowledge stays high on the agenda. However, it is too early to tell what impact this had.

Canada Health Infoway
Overall, Infoway made progress in achieving its intended outcomes outlined in its funding agreements, particularly in terms of advancing the use and adoption of electronic health information systems across Canada, which has resulted in some early improvements in the healthcare system.

In terms of increasing awareness and understanding of eHealth technologies, Infoway launched an education campaign (Knowing is Better) in order to increase the awareness and understanding of, and support for, electronic health information systems among the general public. The campaign generated more than 101 million media impressions (a measure of the number of times an ad was seen) and 315,000 unique web site visits. Infoway reported that awareness and support of EHRs increased from 72 per cent to 87 per cent among the general public over the course of the campaign. In 2012, the campaign was expanded for clinicians in collaboration with clinical engagement groups. Resources (such as videos, presentations, and articles) were disseminated through more than 25 national/provincial organizations and reached approximately 190,000 clinical stakeholders.

Since its inception, Infoway has developed strong collaborations with the provinces and territories and successfully co-invested in over 380 projects across their 12 investment programs. The shared projects helped position eHealth as a shared priority among provinces and territories.
Another successful example of increased collaboration was the Standards Collaborative, which provided leadership, expertise and core services to support the development, maintenance, and implementation of pan-Canadian health information systems. The Standards Collaborative included over 450 members, who together developed five new Canadian approved standards and six updates to existing standards.

Given the nature of electronic health systems, Infoway also completed work on privacy and security issues, including a public opinion survey assessing Canadians' attitudes towards digital health and privacy, and the completion of a discussion paper on consent management issues.

Most of the information on Infoway's progress related to the extent to which electronic health systems had been adopted by various groups. This focus is reflective of the fact that systems have to be in place and adopted/used to achieve longer-term outcomes, including system transformation. Annual reports included details regarding adoption rates for each jurisdiction, and progress reports included details of adoption of EHRs and EMRs in Canada. Adoption rates varied across the country, with some territories experiencing lower adoption rates than other parts of the country, though these rates do not reflect overall progress with other important advances, such as EMRs or telehealth.

In March 2011, Infoway achieved its goal of having more than 50 per cent of Canadians with an EHR available to authorized providers, just three months shy of its original target of December 2010. As of March 2013, 55.4 per cent of Canadians had an EHR available to their healthcare provider, of which three provinces had reached 100 per cent availability. Infoway aims to reach its goals of having 58-60 per cent of all Canadians with an EHR by the end of 2013-14, and 100 per cent by the end of 2015-16.

Between the time the 2010 funding agreement was established to March 2013, more than 12,000 community-based clinicians enrolled in provincial EMR programs that were supported by Infoway. The number of these clinicians using the EMRs to support clinical activities increased from 23 per cent in 2006 to 56 per cent in 2012.

More than a third of the EMR systems designated for use in ambulatory care settings were deployed, supporting between 7,100 to 7,500 clinicians. These systems were also being connected to hospital systems and the EHR to enable the exchange of clinical data. Results of a benefits evaluation of EMR use in community-based settings revealed several efficiencies in the healthcare system resulting from EMR use. For example, savings related to efficiencies in laboratory and diagnostic test management and reductions of 'chart pulls' were estimated at $177M in 2012; and savings related to fewer duplicate tests and adverse drug events (due to prescription legibility) were estimated at $123M in 2012 and these savings would be ongoing into the future. Other improvements to lab/diagnostics test management (e.g. digital results, use of EHRs) led to a reduction in time to access diagnostic imaging from two weeks to two to three days.

Similarly, telehealth solutions enabled more people to access healthcare (particularly for those living in the north). The results of a benefits evaluation of telehealth use in Canada revealed that these telehealth solutions reduced the cost of Canadian healthcare by approximately $55 million in 2010. It should be noted that savings were in part attributed to the reduced need for individual patient travel and associated reduction in the utilization of medical travel subsidies and grants, as well as the avoidance of emergency department visits and other inpatient hospital costs.

Health Council of Canada
In this section, HCC outputs achieved and progress towards outcome achievement will be highlighted as they relate to increased awareness and understanding of knowledge products and increased collaboration and coordination as the HCC produced several reports and collaborated with a number of organizations. Less evidence was available to demonstrate individual use or adoption of these products.

HCC produced annual Progress Reports on the implementation of commitments made under the Health Accords, hosted three National Symposia, released eight Canadian Health Care Matters Bulletins (analyzing results of the Commonwealth Fund International Health Policy Survey), and launched an Innovation Portal (a web-based searchable database of best practices). The Council also collaborated with several organizations, including: jurisdictions (to seek input before launch of a report); and the six provincial quality councils (including joint purchasing of Commonwealth Fund Survey data in order to increase the relevance of the findings for the jurisdictions).

These outputs helped raise awareness and understanding among Canadians about the performance of the Canadian healthcare system to a varying degree: independent evaluation results indicated that 77 per cent of survey respondents (n= 181 potential clients of HCC's reports, such as representatives of regional, provincial/territorial, and national healthcare-related organizations; healthcare service delivery organizations; professional associations; etc.) and 52 per cent of interviewees (n= 23 representatives from provincial/territorial governments, health agencies, and the media) felt that Health Council's information influenced their understanding of the status of health system performance and innovative practices.

According to HCC's Annual Report in 2012-13, there were 759 news stories about HCC reports; 377 participants in the National Symposium on Integrated Care; almost 55,000 downloads of HCC reports; and over 12,000 searches on the Health Innovation Portal.

In terms of increased use and adoption of knowledge products, it was noted in the independent evaluation that between 45 and 63 per cent of interview/survey respondents had used HCC reports or activities on health system performance. More survey respondents (71 per cent) than interviewees (33 per cent) thought the reports/activities were useful to support decision making. As a suggestion to improve uptake, the independent evaluation noted that HCC could do more consultations to identify the information requirements of the primary audiences, but that would require more resources.

There was limited evidence in the documents reviewed describing how the information generated by HCC was used or who used it. HCC reported that its reports were used to help inform the Standing Senate Committee on Social Affairs, Science and Technology's review of the 2004 Health Accord titled Time for Transformative Change. The independent evaluation noted that the HCC's report Seniors in Need, Caregivers in Distress: What are the Home Care Priorities for Seniors in Canada? was used as a reference source in a number of contexts (in the areas of health human resources; home, community, and primary care; and continuing care), with one group using some of the HCC measures identified in the report to compare their progress to that of other provinces.

In terms of the impact of Health Council's work on organizations, 24 per cent of the interviewees from the independent evaluation said that the Health Council's reports and activities had a moderate to great impact on health policy, programs, or services within their organization. Within the group of interviewees, government respondents provided lower ratings than other stakeholder groups.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

As the financial information from the pan-Canadian organizations did not provide output/outcome specific costing data (and were not required to under their funding agreements), this evaluation chose a different approach to assessing economy and efficiency. An examination of the management of the funding agreements, including Health Canada's level of engagement with recipients on policy and program issues as well as participation on pan-Canadian organizations' governance structures, was carried out. In addition, a cursory examination of alternative approaches to meeting the transfer payments' objectives is discussed in terms of its impact on efficiency.

4.5.1 Were the costs of outputs and outcomes reasonable in light of the program context for each of the transfer payments (Economy)?

This was the original evaluation question, however, given the above data constraints, the evaluation focused on Health Canada's management of the programs through which the transfer payments to pan-Canadian organizations were provided, including the level of engagement with the pan-Canadian organizations in order to facilitate outcome achievement. This engagement occurs in order to ensure compliance with funding agreement conditions, as well as to encourage strategic alignment of pan-Canadian organizations' activities with federal priorities (within the existing parameters identified in the funding agreement while respecting the involvement of other stakeholders.)The evaluation also considered Health Canada's participation in the governance structures of the organizations as a parallel stream of engagement and its interplay with the program management function.

Within the timeframe of the evaluation, there has been a concerted effort to increase the level of engagement on both the compliance and strategic alignment fronts, due to a variety of factors including: results from audits, an increased desire from senior management for more information on the status/progress of the transfer payments, a strengthened interest in ensuring that federal investments are achieving the intended policy objectives and/or the transition from a grant to a contribution funding model (which can entail more reporting and accountability requirements). Overall, interviewees viewed the increased engagement as a positive direction, though also recognized the increased resource implications.

Monitoring Progress and Compliance
The transfer payments to pan-Canadian organizations were managed well in terms of monitoring progress and ensuring compliance with the terms in the funding agreements (e.g. ensuring reporting obligations were received in a timely manner). Funding agreements had fairly similar reporting requirements, including: an independent evaluation, cashflows, and audited financial statements. Some funding agreements also required an evaluation framework, a performance measurement framework, internal mid-year progress reports, and a performance audit. However, while reporting requirements looked similar on paper, sometimes the content or level of detail contained in the reports differed. Some independent evaluations and annual reports included examples of longer-term outcome achievement, while others reported mainly on activities and outputs.

Most of the policy areas managing these funding agreements had developed formal monitoring frameworks, complete with checklists to help track reporting requirements and timelines. These checklists also included some questions regarding the content of Annual Reports at a high level. One policy area was working on developing an annual report template in order to outline reporting requirements for each year, and to help track progress in certain activity areas from year to year. However, the organizations' annual workplans tended to outline specific objectives and activities for the upcoming year, which were not explicitly linked to indicators of outcome achievement, and checklists contained no details regarding the degree of outcome achievement (e.g. who used the outputs or what impact these had on the healthcare system). As a result, while these frameworks successfully enabled monitoring of compliance with funding agreements, they did not provide significant information about whether expected outcomes or broader changes had been achieved. Given the department's interest in documenting achievement of longer-term outcomes, there was some desire from staff to clarify how this assessment should be done.

There was some evidence to suggest that reporting requirements were burdensome to both recipients and Health Canada staff. For example, some organizations reportedly kept a separate performance measurement system or set of accounting books to reflect how Health Canada required them to report. One pan-Canadian organization completed four independent evaluations, in accordance with the terms of its various funding agreements, within the last five years.

Some interviewees at various levels noted how time consuming funding agreement management was on staff, particularly for policy staff without a lot of program management experience. Also, as mentioned before, resources were not formally allocated to the management of funding agreements in the cabinet documents establishing authorities for these programs and therefore, these activities had to be resourced within existing directorate budgets. Some staff suggested having dedicated program management staff to assist with the monitoring of funding agreements and allowing policy analysts to focus more attention on policy/strategic issues associated with the work of the organizations, and related program decisions, would be beneficial.

While the reporting requirements were useful in terms of having current information about activities and progress, a balance needs to be struck between reporting burden and information requirements (for both monitoring purposes as well as to support policy and program decision making at Health Canada). In an effort to reduce unnecessary burden on both Health Canada and the pan-Canadian organizations, it would be beneficial to explore ways to align (and calibrate) these reporting requirements with the information needs of the Strategic Policy Branch, and Health Canada more generally, while ensuring the availability of performance data to inform future evaluations and to inform departmental policy decisions.

Strategic Relationship Management
It was acknowledged by some senior management in the policy areas managing the funding agreements that Health Canada should be engaging with the pan-Canadian organizations, beyond solely for monitoring purposes, as an opportunity to better align pan-Canadian organizations' strategic priorities with the federal policy objectives underpinning the programs through which they are funding and with evolving departmental policy directions.

The level of strategic engagement between Health Canada staff and the pan-Canadian organizations varied. For instance, as organizations expanded their scope of activities and/or adjusted strategic directions in response to emerging health system needs, some organizations proactively included Health Canada senior management in strategic planning discussions while others did not. There was also some evidence to suggest that less engagement with pan-Canadian organizations was related to: poorer quality/depth of briefing notes to senior management, less opportunity for Health Canada to provide feedback to pan-Canadian organization's on their workplans/strategic plans, less Health Canada involvement with the independent evaluations, and limitations in the performance measurement frameworks.

A number of interviewees indicated that the results of Health Canada's engagement with pan-Canadian organizations could be optimised, if interactions happened at various levels within the department - from the program officer/analyst level to senior management - and were aligned with departmental participation on the pan-Canadian organizations' governance structures (e.g. Board of Directors or Corporate Members). It was essential that each of these levels was conveying a consistent message to the pan-Canadian organizations regarding the department's policy directions. Not all Health Canada staff were aware of a departmental policy direction for each of these organizations, or how and when they were expected to engage with pan-Canadian organizations in order to promote departmental policy directions. For example, some policy areas provided feedback on workplans in terms of how the activities/approaches were described, rather than strategic feedback on which activities/approaches were taken. And still, others did not provide comments on strategic plans or workplans at all (this was done solely at the Board level). Several staff raised the fact that they were mindful that these organizations were arms length autonomous legal entities, and were the experts in their fields, therefore the department should not be too directive or prescribe priorities for them.

The evaluation revealed that the policy areas that manage the funding agreements were not typically identified as target audiences or clients of the pan-Canadian organizations' work, but other branches within Health Canada or organizations within the health portfolio were. Often times, these departmental, portfolio and other federal departmental groups would interact, collaborate and/or consult with the pan-Canadian organizations without including or acknowledging the policy areas responsible for managing the funding agreements within the Strategic Policy Branch. These different federal groups may have different information needs or priorities on which they are trying to engage the pan-Canadian organizations. While this is not, in itself, problematic, there is a risk of communicating contradictory messages or leaving the responsible policy areas out of the loop. As mentioned earlier, providing a consistent vertical (all levels within the policy areas managing the transfer payment - analyst, director and senior management) and horizontal (branch, departmental, portfolio and in some instances, federal) message is key to optimizing the outcomes of federal engagement with these pan-Canadian organizations.

Participation in Governance Structures
While distinct from program management, at the Board of Directors or Corporate Members level, Health Canada's level of engagement also varied, partly due to variations in how the governance structures of the pan-Canadian organizations were initially set up. Variations were identified in terms of the seniority of the member, the position of the member (internal to the department or external), as well as member's voting responsibilities. The seniority of the members ranged from Director General to the Minister (and everything in between), often depending on the requirements set out in the organization's by-laws and/or the level of strategic intervention needed to respond to departmental objectives and to the scope and scale of the organization. Health Canada staff noted that strategic engagement occurred both during Board meetings and outside them, and that Board members who had more seniority (DM or Minister) did not necessarily engage more in strategic discussions than Board members at other levels.

The document review and interviewees both confirmed that some of the responsibilities of Board/Corporate Members differed across pan-Canadian organizations. While all official Board/Corporate Members had voting rights (the observer did not), three of the four Board/Corporate Members chose not vote on items of a financial nature.

For some pan-Canadian organizations' Board of Directors, Health Canada had appointed a representative from outside the department (i.e. an external representative) either presently or in the past. In these cases, departmental briefing materials were not provided to the Board Member and debriefs after the meetings from the external Board Member were not common practice. However, even when the members were internal to Health Canada, debriefs to staff were not always provided regularly or to all levels of staff involved with managing the pan-Canadian organization. One pan-Canadian organization with an external Board Member also had a representative from Health Canada who attended Board meetings as an observer, which helped keep departmental staff informed of the issues discussed at this level, and what other Board Members' views were on various system issues or challenges while respecting confidentiality. Another policy area, with a vacant Board Member position, communicated directly with the pan-Canadian organization's Board Secretariat to obtain debriefs of Board meetings (in addition to the formal meeting minutes).

In terms of Board of Director and Corporate Member expectations, a branch-wide, and even a department-wide approach to participation and the provision of debriefs may be helpful to ensure that information is being communicated in a consistent manner to staff about the broader context of issues that are arising and/or challenges that the Board/Corporate Members (e.g. other provincial/territorial representatives) are facing. This intelligence helps departmental staff know what decisions have been taken or not (and why), which helps them monitor the pan-Canadian organization's progress, and provides more strategic feedback on organizational priorities. It also enables staff to craft strategic policy briefings and advice for future Board/Corporate and/or senior management meetings.

4.5.2 Are there alternative approaches to meeting the transfer payments' objectives? (Efficiency)

This section examines alternative approaches to meeting the transfer payments' objectives by discussing the appropriateness of the funding mechanism, and assessing how the department manages the suite of pan-Canadian organizations more generally.

In terms of assessing the appropriateness of the funding mechanism, there was general agreement that funding the creation and operation of these pan-Canadian organizations was an appropriate way to advance federal health system priorities. Having an independent organization facilitate collaboration among many stakeholders, including provinces and territories, while respecting their jurisdiction in healthcare was seen as a positive approach to create national system-level changes. It was also seen as more efficient having one national organization focusing on a particular issue than creating a program with multiple, potentially disjointed, time-limited projects, which would not have been able to leverage the resources required to achieve national-level changes.

Recently, as some of the funding agreements were renewed, there was a trend to change the type of transfer payment from a grant to a contribution, allowing for increased accountability and reporting requirements and a stronger departmental role in shaping deliverables. It also tended to facilitate the increased level of engagement the department had with the pan-Canadian organization. While some transfer payments were already increasing accountability and reporting requirements, such as requiring more frequent cashflow requests, these requirements tended to help monitor progress on a more regular basis, but did not necessarily impact the level of departmental engagement in terms of aligning strategic policy directions.

Whether funding to pan-Canadian organizations should be managed individually or together as a program was briefly examined. There were advantages to examining the suite of pan-Canadian organizations to share lessons learned and to identify potential integration and/or coordination of activities amongst the organizations. Some staff suggested that the collaboration and coordination among policy areas involved with program and relationship management with the pan-Canadian organizations could be enhanced. Some areas were collaborating significantly on the policy-related issues through established networks/committees in particular policy areas (e.g. eHealth network) and through the recently established Policy Forum on Pan-Canadian Healthcare Organizations, but there was less interaction on other monitoring and funding agreement management issues (e.g. funding renewals, changes in type of funding mechanism, changes to the Canada Not For Profit Corporations Act, etc.). Policy areas could share lessons learned which would help minimize duplication (e.g. in the development of control frameworks or processes for providing feedback on workplans) and increase efficiency in the approach to managing funding agreements. However, the pan-Canadian organizations still require a degree of individual management given the differences in their policy objectives, business and governance models, and their size, scope and maturity.

5.0 Conclusions

5.1 Relevance Conclusions

Overall, Health Canada's transfer payments to pan-Canadian organizations are aligned with federal priorities and fulfil an appropriate federal role. The organizations provide a unique, national coordination function that individual jurisdictions and regional groups would not be able to accomplish on their own. Although there are many organizations involved in the improvement of the healthcare system in Canada, provinces, territories or other stakeholders look to the pan-Canadian organizations for leadership and for providing a national perspective on system-wide priorities.

Besides the Health Council of Canada, the areas of focus addressed by the other pan-Canadian organizations are still relevant today; and, while progress has been made, there is a clear need to continue to improve the healthcare system issues addressed by CADTH, CPSI, MHCC and Infoway.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes

Pan-Canadian organizations were producing large number of outputs and achieving outcomes aligned with the stages along the knowledge translation continuum, as well as, their own individual objectives. While all organizations were working towards improving certain aspects of the healthcare system, some, but not all, were able to provide concrete examples of broad changes that had been achieved at this point, particularly for the newer pan-Canadian organizations. Annual reports and independent evaluations were useful in providing evidence of immediate and some intermediate outcomes, but contained fewer details about longer-term outcome achievement.

It should be noted that all of the pan-Canadian organizations were able to provide evidence of collaboration and coordination with a number of different organizations. In summary, the pan-Canadian organizations, as a whole, have been able to demonstrate progress towards their outcomes and have the success factors necessary to continue progress towards longer-term outcomes.

5.2.2 Demonstration of Economy and Efficiency

Overall, the transfer payments to pan-Canadian organizations were managed well in terms of monitoring progress and ensuring compliance with terms in the funding agreements. However, efforts to monitor funding requirements meant that less time was available to provide strategic policy information for decision making. It was highlighted that the collaboration and coordination among policy areas managing pan-Canadian organizations could be enhanced. Policy areas could share lessons learned which would help minimize duplication and increase efficiency in the approach to managing funding agreements.

There was less consistency across transfer payment programs in terms of the level of engagement and messaging between the department and the pan-Canadian organizations. Establishing a common branch-wide or Health Portfolio-wide approach to managing interactions with pan-Canadian organizations may be helpful to ensure that information is being communicated, and communicated in a consistent manner, within the branch, within the federal government, and then to the pan-Canadian organizations. Providing a consistent message is key to optimizing the outcomes of federal engagement with these pan-Canadian organizations.

More engagement would provide an opportunity to align the strategic direction or approach pursued by the pan-Canadian organization with the priorities of the federal government. However, given that this evaluation only examined a subset of pan-Canadian organizations managed by the Strategic Policy Branch, it would be necessary to examine Health Canada's level of engagement with the other pan-Canadian organizations as well to see if similar impacts exist, and to help determine what the optimal level(s) of engagement should be for the various organizations.

6.0 Recommendations

The evaluation identified the following four recommendations:

Articulate Health Canada's strategic approach to the use of pan-Canadian organizations, given their importance as policy levers, and implications of this approach for Health Canada's engagement with each of these organizations.

Strengthen coherence and collaboration among Health Canada staff involved with managing the department's relationships with the pan-Canadian organizations to effectively implement the departmental strategic approach and foster the exchange of experiences and lessons learned in this regard.

For each pan-Canadian organization, strengthen coordination within and across the department, portfolio, and other federal partners (where appropriate) to enhance internal communications and ensure a coherent federal perspective when interacting with that organization.

Align/calibrate the approach to program management, including the type(s) and content of reporting requirements set out in funding agreements in support of pan-Canadian organizations to ensure it is coherent and appropriate to support effective program monitoring, management and decision making by Health Canada, while recognizing variations across the organizations (e.g. size, scope, funding model, governance, objectives).

Appendix 1 - Logic Models^{Footnote A2}

Description of the Logic Model for Canada Health Infoway's 2010 Funding Agreement

The logic model identifies Infoway's clients and stakeholders as follows: Canadian residents; federal, provincial, and territorial ministries; health care providers and practitioners; NGOs; industry - vendors; Deputy Ministers of Health; and Infoway Management and staff

The mission and vision of the logic model:

Mission: To foster and accelerate the development and adoption of electronic health information systems with compatible standards and communication technologies on a pan-Canadian basis with tangible benefits to Canadians. The Corporate plan will build on existing initiatives and pursue collaborative relationships in pursuit of its mission.
Vision (Current): Healthier Canadians through innovative e-health solutions
Vision (Former): A high quality, sustainable and effective Canadian health care system supported by a pan-Canadian health infostructure that provides residents of Canada and their health care providers with timely, appropriate and secure access to the right information whenever and wherever they enter the health care system. Respect for privacy is fundamental to this vision.

The logic model identifies the following outcomes from the 2010 Funding Agreement: With respect to EHR solution... development of reusable Health Information Building Blocks. Development of EHRs for fifty (50%) of Canadians by the end of 2010; With respect to EMR solutions only...enhanced vendors and jurisdictional capacity...Increased use of EMRs at points of service; implementation of intra-operable solutions which enable information flow between various points of the health system; With respect to efforts in support of Secondary Use of Data...collaborate with stakeholders to identify policy, privacy and communication issues and lead the specification of technical components needed to facilitate secondary use of data;With respect to the development of reusable tools/assets to address the human factor in EHR and EMR adoption...change management and process transformation and knowledge transfer and sharing; pan-Canadian, multi-jurisdictional approach; subject to, and conditional upon, the collaboration of recipients of Infoway funding... Including Provincial and Territorial jurisdictions...Vendors and other stakeholders; expected that funding will result in more timely delivery of health care, increased productivity and interoperability, improved access to, and sharing of information, the creation of sustainable knowledge-base:! jobs; and a strengthened knowledge infrastructure within the health care system"

Outcomes are to be achieved in accordance with the Purpose and Use of Up-Front Multi-Year Funding as per the sixteen (16) statements in 4.1to 4.16 inclusive of the 2010 Funding Agreement.

The logic model identifies the outputs as follows:

EHRs Blueprint - the business and technical architecture for an interoperable EHR framework across Canada
Investment strategies and program
Investments in investment projects
Benefits evaluations
Partnerships and alliances
Policies and standards
Annual summary, corporate plans, annual reports, and other plans, reports and communications products

Description of the Logic Model for Infoway's 2007 Funding Agreement

The mission and vision of the logic model:

Mission: To foster and accelerate the development and adoption of pan-Canadian electronic health information systems. These are systems with compatible standards and communication technologies, providing tangible benefits to Canadians.
Vision: Better care through timely access to secure health information when and where it's needed

The activities outlined in the logic model are as follows:

Further develop, update and refine EHRS Blueprint
Develop/refine strategies
Conduct technical reviews in support of investments
Implement and monitor Infoway investments in projects within the investment programs
Conduct benefits evaluations of investments
Develop and maintain various relationships
Develop/refine standards
Prepare and distribute plans, reports and communications

The activities lead to the following outputs:

EHRS Blueprint - the business and technical architecture for an interoperable HER framework across Canada
Investment strategy
Investments in six program areas - diagnostic imaging systems; drug information systems; laboratory information systems; interoperable EHR; public health surveillance; patient access to quality care (four areas not in scope are the registries; telehealth; infostructure; innovation and adoption
Partnerships and alliances
Policies and standards
Business plans; annual reports; other plans, reports and communications products

The logic model identifies the outcomes from the 2007 Funding Agreement as follows:

Grant Funding provided to Infoway under this Funding Agreement shall be used by Infoway solely for the purpose of providing funding to committed PWTG jurisdictions or organizations located within such Committed PWTG jurisdictions to assist them in the development and implementation if EHICT projects.
Principles - support development and use of EHICT...needed to support wait time reductions, improved access to health care and assist provinces and territories in implementing Patient Wait Time Guarantees; support publicly funded health care system consistent with the Canada Health Act; foster collaboration by building F/P/T partnerships, in conjunction with other stakeholders; support interoperability; adopt a pan-Canadian approach; and alignment of F/P/T and Infoway investments, directions and plans in EHICT
Outcomes - achieve the expected outcomes specified in the annual Corporate Plans...in respect of the outcomes listed below over Infoway's next five Fiscal Years: reusable Health Information Building Blocks; interoperable electronic health record solutions; reusable tools/assets to address the human factor; pan-Canadian, multi-jurisdictional approach; achieve the outcomes contemplated above in a manner that complies with the applicable laws enacted to protect privacy, confidentiality and security of personal health information and appropriate privacy principles

Description of the Logic Model for Canada Health Infoway

The logic model identifies Infoway's clients and stakeholders as follows: Canadian residents; federal, provincial, and territorial ministries; health care providers and practitioners; universities, research institutes and NGOs; industry; Deputy Ministers of Health; and Infoway Management and staff

The mission and vision of the logic model:

Mission: To foster and accelerate the development and adoption of electronic health information systems with compatible standards and communication technologies on a pan-Canadian basis with tangible benefits to Canadians.
Vision: A high quality, sustainable and effective Canadian health care system supported by a pan-Canadian health infostructure that provides residents of Canada and their health care providers timely, appropriate and secure access to the right information whenever and wherever they enter the health care system. Respect for privacy is fundamental to this vision.

The activities outlined in the logic model are as follows:

Develop strategies and plans
Maintain relationships and alliances with stakeholders
Develop business and technical solution architecture
Develop and apply standards
Conduct technical reviews in support of investments
Implement and monitor Infoway investments in projects within the investment programs
Manage and administer finance, IMT and procurement
Prepare plans and reports
Manage and administer communications
Manage and administer human resources
Manage and administer legal affairs

The activities lead to the following outputs:

EHRS Blueprint - the business and technical blueprint for an interoperable EHR framework across Canada
Investments programs - infostructure, registries, drug information systems, diagnostic imaging systems, laboratory information systems, telehealth, public health surveillance systems^{Footnote A3}, interoperable EHR, adoption and innovation
Partnerships and alliances
Policies and standards
Business plans; annual reports; other plans and reports

The logic model identifies the outcomes as follows:

achieve the expected outcomes specified in the annual corporate plans...in respect of the outcomes listed below
Information and Technology Standards; EHR Solution Architecture; basic elements of EHR Health Infostructure and Security Infostructure; reusable health Information Building Blocks; interoperable electronic health record solutions; expansion of Telehealth in Canada; reusable tools/assets to address the human factor; pan-Canadian, multi-jurisdictional approach
achieve the outcomes...complies with all applicable laws enacted to protect privacy, confidentiality and security of personal health information and appropriate privacy principles
Overall, Infoway should be able to demonstrate that its strategic investments over the next three years will have contributed to absorbing a material part of the fixed development costs of the EHR Health Infostructure

Description of the Logic Model for Canadian Agency for Drugs and Technologies in Health (CADTH)

Funding streams:

Health Technology Management (Drugs and other health technologies) - Health Technology Assessment (HTA); Optimal Use (OU)
Common Drug Review

Inputs: FTEs (e.g. Researchers, Liaison Officers, Analysts, Scientists), F/P/T Resources, Research Material, Contracting for expertise

Inputs for the Health Technology Management funding stream lead to three activities: Produce HTA/OU products; generate advice through expert committees; maintain relationships to broker HTA/OU knowledge

Inputs for the Common Drug Review funding stream lead to the following activity: Conduct reviews of submitted drugs that assess their clinical and cost-effectiveness

All activities for HTA/OU lead to the following outputs:

HTA/OU knowledge products, reports, services and advice; partnerships/relationships; brokering events (e.g. conferences, forums, or liaison communications)

Activity for the Common Drug Review leads to the following output:

Formulary listing recommendations

The logic model identifies CADTH's immediate outcome for HTA/OU as: increased awareness and understanding of HTA and OU evidence. It also identifies the immediate outcome for the Common Drug Review as follows: increased awareness of evidence for listing decision; and increased transparency across jurisdictions.

Intermediate outcomes:

Increased utilization of evidence-based information in health care decision-making
Improved coordination of drug and other health technology reviews
Improved collaboration among health care system stakeholders, including other producers and users of evidence

Long-term outcome:

Increased evidence-based decisions on the optimal use of health technologies to support improved health outcomes and health care system sustainability

Description of the Overall Logic Model for CADTH and the Optimizing Health Systems Efficiency Initiative

Activities for the CADTH logic model include:

Manage the operation of the Health Technology Analysis Exchange which produces the following output: mechanism for the exchange of ideas and information by HTA producers in Canada. The logic model identifies the immediate, intermediate and long-term outcomes as follows:

Immediate outcome - effective utilization of existing health technology assessment and review capacity and resources
Intermediate outcome - improved coordination of national HTA priorities and capacity
Long-term outcome - evidence-based decision-making on health technologies across the Canadian health care systems

Assessments of approaches, policies and best management practices for medical isotope usage produces outputs that include: evidence assessment reports; national guidance documents and intervention tools. The logic model identifies the immediate, intermediate and long-term outcomes as follows:

Immediate outcomes - uptake and utilization of CADTH programs, products and services; and increased capacity of CADTH primary and secondary stakeholders to make evidence-based decisions
Intermediate outcomes - more effective use of scarce resources (HTA skill sets and health dollars); and increased uptake and utilization of CADTH programs, products and services.
Long-term outcome - evidence-based decision-making on health technologies across the Canadian health care systems

Identify and monitor CADTH Primary Stakeholder health technology needs and identify opportunities for the continuous improvement of CADTH products and services; and conduct assessments and reviews and deliver CADTH evidence-based products and services produce the following outputs: relevant, high quality, timely evidence-based products and services; reports, information and advice; intervention tools and resources; and accessible databases. The logic model identifies the immediate, intermediate and long-term outcomes as follows:

Immediate outcomes - uptake and utilization of CADTH programs, products and services; and increased capacity of CADTH primary and secondary stakeholders to make evidence-based decisions
Intermediate outcomes - more effective use of scarce resources (HTA skill sets and health dollars); and increased uptake and utilization of CADTH programs, products and services.
Long-term outcome - evidence-based decision-making on health technologies across the Canadian health care systems

Promote CADTH and its products and services produces the following outputs: targeted educational events, workshops, seminars, pod-casts, web-casts, presentations. The logic model identifies the immediate, intermediate and long-term outcomes as follows:

Immediate outcomes - greater awareness, understanding of, and receptivity to, CADTH products and services by CADTH primary and secondary stakeholders; and broad recognition of CADTH as a leader in the health technology assessment field
Intermediate outcomes - more effective use of scarce resources (HTA skill sets and health dollars); and increased uptake and utilization of CADTH programs, products and services
Long-term outcome - evidence-based decision-making on health technologies across the Canadian health care systems

Facilitate knowledge exchange with national and international health technology organizations with similar mandates produces the following outputs: linkages with national and international bodies with related mandates; ongoing/continuous communication with and feedback from CADTH primary and secondary stakeholders; improved methodologies and processes; and increasingly relevant products and services from CADTH and its partners. The logic model identifies the immediate, intermediate and long-term outcomes as follows:

Immediate outcomes - greater awareness, understanding of, and receptivity to, CADTH products and services by CADTH primary and secondary stakeholders; and broad recognition of CADTH as a leader in the health technology assessment field
Intermediate outcome - increased uptake and utilization of CADTH programs, products and services
Long-term outcome - evidence-based decision-making on health technologies across the Canadian health care systems

Facilitate discussion among F/P/T policy makers on health technology issues and policy development produces the following outputs: effective operation of policy forum, Advisory Committee on Pharmaceuticals, Pharmaceutical Directors Forum, COMPUS Advisory Committee and Devices and Systems Advisory Committee. The logic model identifies the immediate, intermediate and long-term outcomes as follows:

Immediate outcome - improved communications and a common approach to decision-making across F/P/Ts on Canadian health technology policy issues
Intermediate outcome - implementation of coordinated policy initiatives b F/P/Ts
Long-term outcome - evidence-based health technology policy decision-making in Canada

All activities contribute to the following ultimate outcome: appropriate and effective utilization of health technologies within the Canadian health care system.

Description of the Logic Model for Mental Health Commission of Canada - Level 1 (as of December 2010)

The vision for this logic model is a society that values and promotes mental health and helps people living with mental health problems and mental illness to lead meaningful and productive lives.

Assumptions:

The MHCC is responsible to people with lived experience of mental illness and their families, service providers, researchers and governments in Canada.
The MHCC and the mental health system have a responsibility related to the mental well-being, mental health promotion and mental illness prevention for all people living in Canada, including children, youth, adults and seniors.
The implementation of a mental health strategy for Canada relies not just on the development of the strategy by the MHCC but the combined support and collaboration of all stakeholders to make this a reality.
People living in Canada support the work of the MHCC.
Communities and service providers are responsive to and working collaboratively to support the work of the MHCC.
People in the mental health community (including PWLE, families, caregivers, mental health service providers and other stakeholders) who are aware of the MHCC, have high expectations including an expectation of real and concrete deliverables.

Inputs/Resources:

Funding - HC ($130M over 10 years); HC for At Home/Chez-soi ($110M over 5 years); and other sources
Accountability - Governance Board; and Government of Canada/Health Canada
Human resources - MHCC Executive and staff; contracted staff and agencies; and volunteers (Advisory Committees and others)
Partners/collaborators - PWLE of mental illness; families and caregivers; Government (FPT) stakeholders; NGO stakeholders; service provider stakeholders; researchers; educators; international partners; national and local media; people living in Canada; and local communities

All inputs/resources lead to the following key activities:

Five key initiatives - Mental Health Strategy for Canada; Opening Minds Anti-Stigma/Anti-Discrimination Initiative; Knowledge Exchange Centre; Partners for Mental Health; and At Home/Chez-soi multi-site mental health and homelessness research demonstration projects
Advisory Committees - Implementation of 8 Advisor Committees (ACs); and Priority initiatives/projects undertaken to support the 5 key initiatives
Engagement/Raising Awareness/Communication - Establish, maintain partnerships; linkages with partners for dissemination opportunities; and develop communication plan with key messages, communications vehicles, priorities and detailed communications strategies for each strategy
Program delivery - Mental Health First Aid (MHFA)
Corporate Management and Governance - Board governance; policies and procedures; operating model; organizational structure; policy and research team support to - ACs and other priority projects; performance management structure developed; secure additional funding for initiatives into the future; and establishing the MHCC as a role model workplace - a mentally and physically safe workplace environment for staff and volunteers

Activities for the five key initiatives lead to the following outputs:

Work plans developed for each strategy; programs or frameworks developed; environmental scans/surveys completed; nationwide consultations implemented; and production and dissemination of reports and other materials

Activities of the Advisory Committees lead to the following outputs:

Advice and support for the 5 key initiatives; 24 Advisory Committees' projects commenced/awarded; and production and dissemination of reports, frameworks, documents and workshops

Activities of engagement and raising awareness communication leads to the following outputs:

Meetings with governments (FPT) and other stakeholders; national/international conferences/symposia and roundtables held; communication plan in place; data and information through website; information via key communication channels of partners; and information via national, local media and news media

Activities of program delivery lead to the following outputs:

MHFA trainings provided; and adaptation of MHFA curriculum for the NWT government

Activities of the corporate management and governance lead to the following outputs:

Board reports; strategic and business plans; policies and procedures; organizational structure documents; priority projects such as Risk Analysis Restraint and Seclusion, and others; performance management logic model developed; evaluations implemented and reporting on evaluation; funding and other resources secured; and code of conduct developed

The key activities and outputs are intended to reach the following audiences: PWLE (including homeless)/families/caregivers; all people living in Canada; mental health professionals/service providers/NFP groups; researchers/academics/educators; federal, provincial, territorial ministries and authorities; federal, provincial, territorial decision and policy makers; the Minister of Health and Health Canada.

The logic model identifies the following initial outcomes (2-4 years):
Increased:

Awareness and understanding of mental health and mental illness by all people living in Canada
Awareness of the MHCC by partners and collaborators
Dissemination of evidence-informed knowledge to governments and stakeholders
Knowledge base: sharing/exchanging knowledge
Understanding of stakeholders' views on mental health and mental illness
Positive reporting and decreased negative reporting on mental health and mental illness by the media
Collaboration and participation of service providers, governments, educators and researchers
Stakeholder utilization of MHCC resources and products
Engagement of PWLE and families
Access to voice of PWLE, their families and caregivers
General awareness of stigma and its impacts on PWLE
Involvement of people living in Canada in Partners for Mental Health
Improved delivery of services for individuals who are mentally ill and homeless in 5 selected communities in Canada
Inclusive workplace environment at MHCC

The logic model identifies the following intermediate outcomes (5-8 years):

Reduced stigma and discrimination related to mental illness
Improved collaboration among partners and collaborators
Improved awareness of issues and evidence-informed best practices to address those issues
Increased utilization of MHCC research impacting the development of policy and service delivery
Enhanced integrated and collaborative mental health system
Increased capacity of decision makers to implement policies

The logic model identifies the following ultimate outcomes (9-10 years):

Contribute to : System outcomes - a transformed mental health system and transformed Canadian society as outlined by the 7 goals of the Mental Health Strategy for Canada and evidenced by effective and efficient delivery of services
Contribute to: PWLE outcomes - active engagement for improved health outcomes/quality of life and able to live meaningful, productive lives

Description of the Logic Model for Mental Health Commission of Canada

Inputs: A $130M 10-year funding agreement with Health Canada; and a $110M 5-year funding agreement with Health Canada (Homelessness Demonstration Projects)

All inputs lead to four activities:

Development and diffusion of the Mental Health Strategy will produce the following outputs
Creation of the Knowledge Exchange Centre (KEC)
Reduction of stigma and discrimination faced by people with lived experience (Opening Minds)
Development, management and reporting of major homelessness research projects (At Home/Chez-soi)

Development and diffusion of the Mental Health Strategy produces the following outputs: a plan to promote the Mental Health Strategy for Canada and its recommendations to health care professionals and jurisdictions; and a Mental Health Strategy. The logic model identifies the short and medium-term outcomes as follows:

Short-term outcome - strategy being adopted by P/T governments
Medium-term outcomes - reduced levels of stigma associated with mental illness; and enhanced knowledge and understanding of mental health issues

Creation of KEC produces the following outputs: sharing research and knowledge; SPARK (Supporting the Promotion of Activated Research and Knowledge) Training Institute; and the Knowledge Exchange Centre. The logic model identifies the short and medium-term outcomes as follows:

Short-term outcomes - increased awareness of MHCC, its products and resources; and increased collaboration among Canadian leaders in the area of suicide
Medium-term outcomes - reduced levels of stigma associated with mental illness; and enhanced knowledge and understanding of mental health issues

Reduction of stigma and discrimination faced by people with lived experience (Opening Minds) produces the following outputs: evaluation of existing anti-stigma programs in Canada; leading practices report on knowledge acquired to-date on what works in reducing stigma; and symposia/conferences related to stigma associated with people living with mental illness. The logic model identifies the short and medium-term outcomes as follows:

Short-term outcomes - campaign to reduce stigma of mental illness in Canada; youth, workers, media and health professionals trained on anti-stigma; reduction in workplace mental health issues; and improved access to crisis intervention to prevent suicide
Medium-term outcomes - reduced levels of stigma associated with mental illness; and enhanced knowledge and understanding of mental health issues

Development, management and reporting of major homelessness research projects (At Home/Chez-soi) produces the following outputs: knowledge exchange related to Housing First; reports and policy papers; and At Home/Chez-soi final national report. The logic model identifies the short and medium-term outcomes as follows:

Short-term outcomes - continued engagement of participants; adoption of Housing First approaches by P/T governments; and improved health and social outcomes for participants
Medium-term outcomes - reduced levels of stigma associated with mental illness; and enhanced knowledge and understanding of mental health issues

All activities contribute to the ultimate outcome: a society that values and promotes mental health and helps people living with mental health problems and mental illness to lead meaningful and productive lives.

Description of the Logic Model for Canadian Patient Safety Institute

Inputs: Financial resources such as Health Canada contribution agreements and other funding sources; human resources including staff across Canada; and external resources such as patients and families, faculty networks and other organizations.

All inputs lead to five activities:

Generate knowledge - Identify gaps in knowledge about patient safety issues; fund research; and create high-quality patient safety knowledge
Build competence - Pre-service education and professional development
Synthesize and translate knowledge - Gather knowledge from others; assess the quality of knowledge; and develop tools/resources to address patient safety priorities
Leverage the work of other organizations - Influence patient safety priorities; recommend regulatory and/or financial levers to promote system level patient safety improvement; and support integrated approaches to improve patient safety
Build relationships - Engage audiences across the health system; and facilitate strategic alignment across health system audiences

All activities lead to the following outputs:

Education; research; tools and resources; and interventions and programs

The logic model identifies CPSI's immediate, intermediate and long-term outcomes as follows:

Short-term outcomes (1 to 3 years) - a growing evidence base to improve patient safety; evidence-informed patient safety curricula across health disciplines; increased patient safety awareness and knowledge; and strengthened system coordination related to patient safety
Intermediate outcomes (3-5 years) - increase in positive patient safety culture; increase in positive patient safety practices; patient safety is formally monitored and reported; and policies, standards and requirements of strategic partners are informed by patient safety evidence
Long-term outcome (more than 5 years) - patient safety in Canada is improved

Description of the Logic Model for Logic Model for the Health Council of Canada

Situation statement: To report on the progress of renewal of Canada's health care system, focusing on best practices and innovation.

The logic model depicts the results chain for the Health Council of Canada, identifying the priorities as follows: informing, communicating, collaborating and having an impact.

All priorities lead to three activities:

Research, analysis and reporting; stakeholder relations and collaborations; and communication and public outreach

These activities are intended to reach health care policy and decision influencers; government decision makers; and the Canadian public via the media.

All activities lead to the following outputs:

Health accords reporting (outcomes and status); reporting on innovative health care practices, policies, programs and services; new/ongoing relationships, partnerships and networks; support materials (media, social media, multimedia, web), symposia and e-newsletters

The logic model identifies the Health Council's immediate outcomes (impact) as follows: target beneficiaries have access to relevant information on innovative practices in health care; target beneficiaries are more informed and aware of the progress of health system renewal; and target beneficiaries collaborate with Council and participate in Council activities (symposia, workshops).

The logic model identifies the Health Council's intermediate outcomes (impact) as follows: information is used to support policy, program and service change; improved sharing and collaboration among F/P/Ts in progressing on health care renewal; and improved accountability of F/P/T governments.

Ultimate outcome: strengthened and renewed Canadian Health System

The logic model also identifies the theory of change for the logic model (including external influencing factors) with the assumptions and risks.

Assumption: The Council has a niche area and some unique reporting lines
Risk: The Council functions within a crowded health care information landscape

Assumption: Evidence is available to support reporting activities
Risk: Necessary evidence is not available

Assumption: HCC products are useful to target beneficiaries

Risk: Supply and demand for products is not balanced

Exhibit II-1 - Outcomes to be evaluated (Section 4.2 and 6.1.8of the Addendum Agreement)

4.2 Outcomes

4.2.1 ...Infoway shall endeavour to achieve the expected outcomes specified in the annual corporate plans referred to under section 5.2.1.6 to be developed by Infoway and approved by its board of directors and presented to its Members in respect of the outcomes listed below in sections 4.2.1.1 to 4.2.2.2 over Infoway's next five fiscal years commencing April 1, 2004. In this regard, it is acknowledged that the ability of Infoway to achieve these outcomes is subject to, and conditional upon

the collaboration of F/P/T jurisdictions within Canada in which Infoway operates or seeks to operate,
the results of the needs assessment referred to in section 4.2.1.2 and
the availability of i-PHIS for use as contemplated in the needs assessment referred to in section 4.2.1.2.

4.2.1.1 Through improved sharing of information across multi-disciplinary teams, and within and across regions, provinces and territories, accurate and timely health surveillance information will be available for:

improving health outcomes related to the identification and management of infectious diseases;
better infectious disease case, contact and quarantine management;
better identification, tracking and management of infectious disease outbreaks and risks to health related to infectious diseases; and
research and analysis to support improved preparedness for future infectious disease outbreaks and risks to health related to infectious diseases.

4.2.1.2 The development of a needs assessment, which includes a costed implementation strategy, for a pan- Canadian Health Surveillance System which takes into account the EHR Solution Architecture.

4.2.1.3 In achieving the outcomes in sections 4.2.1.1 and 4.2.1.2, Infoway shall consider the need for:

high quality, timely surveillance data at the regional, provincial/territorial and federal levels drawn from feeder systems that support health delivery;
the ability to send and receive data from laboratories;
the ability to send and receive data from hospital infection control;
outbreak case, contact and quarantine management for infectious diseases;
immunization records; and
water quality and food inspections.

4.2.1.4 Based upon the results of the costed implementation strategy developed as part of the needs assessment referred to in section 4.2.1.2, implementation, to the extent possible within the limits of the Amount, of a pan-Canadian Health Surveillance System built upon appropriate Information and Technology Standards.

4.2.2 Infoway shall achieve the outcomes contemplated above in a manner that complies with all applicable laws enacted to protect privacy, confidentiality and security of personal health information and appropriate privacy principles. In the event that Infoway collects, uses or discloses personal health information, it shall comply with all applicable privacy legislation as well as appropriate privacy principles and assess and document its compliance.

6.0 Acknowledgements

6.1.8 In order to achieve the outcomes contemplated in section 4.2 of this Addendum Agreement, the collaboration of F/P/T Member jurisdictions is required on an equal basis and each Member of Infoway, and no individual Member or jurisdiction represented by any Member, has an oversight role in respect of Infoway.

Appendix 2 - References

Canadian Institute for Health Information. (2013-11-21). Benchmarking Canada's Health System: International Comparisons. Retrieved on December 4, 2013.

Canadian Medical Association. (2014-01-07). Canada ranks low on patient safety in international comparison. Canadian Medical Association Journal, 186(1), E12. Retrieved on December 2, 2013.

Canadian Mental Health Association. (2014-01). Fast Facts About Mental Illness. Retrieved on January 3, 2014.

Canadian Patient Safety Institute. (2012-07). The Economics of Patient Safety in Acute Care Technical Report. Retrieved on December 3, 2013.

Canadian Patient Safety Institute. (2012-06-21). Independent Evaluation of the Canadian Patient Safety Institute. Retrieved on January 3, 2014.

CBC News. (2013-04-22). E-health records saved medical system $1.3B in 6 years. Retrieved on January 3, 2014.

Centre for Addiction and Mental Health. (2014-01).Mental Illness and Addiction Statistics. Retrieved on January 3, 2014.

Centre for Addiction and Mental Health. (2014-01).September 10 is World Suicide Prevention Day. Retrieved on January 3, 2014.

Chalkidou, K. et al. (2013-12-21).Health technology assessment in universal health coverage. The Lancet, 382(9910), e48-49. Retrieved on December 2, 2013.

Chief Public Health Officer. (2011). Report on the State of Public Health in Canada. Retrieved on December 2, 2013.

Government of Canada. (2013-03). Jobs, Growth and Long-Term Prosperity: Economic Action Plan 2013. Retrieved on January 3, 2014

Government of Canada. (2006). The Human Face of Mental Health and Mental Illness in Canada. Retrieved on November 29, 2013.

Health Canada. (2013-12-06). Harper Government Introduces New Law to Protect Patients - Delivers on 2013 Throne Speech Commitments on Mandatory Adverse Drug Reaction Reporting, Recalls and Accurate Labels. Retrieved on January 3, 2014.

Health Canada. (2013). 2013-14 Report on Plans and Priorities. Retrieved on January 3, 2014

Health Canada. (2004-09-16). First Minister's Meeting on the Future of HealthCare 2004: A 10-year plan to strengthen healthcare. Retrieved on January 31, 2014.

Mental Health Commission of Canada. (2013-11-18). Opening Minds Interim Report. Retrieved on January 3, 2014.

Office of the Auditor General. (2010-04). Electronic Health Records in Canada. Retrieved on January 31, 2014.

Risk Analytica (for the Mental Health Commission of Canada). (2011-12). The Life and Economic Impact of Major Mental Illnesses in Canada: 2011 to 2041. Retrieved on December 2, 2013.

Scottish Government. (2013). Suicide Prevention Strategy 2013-2016. Retrieved on January 3, 2014.

Schoen, C. et al. (2012-11-15). A Survey of Primary Care Doctors in Ten Countries Shows Progress in Use of Health Information Technology, Less in Other Areas. Health Affairs, 31(12), 2805-2016. Retrieved on January 31, 2014,.

Skinner, R. and McFaull, S. (2012-06-12). Suicide among children and adolescents in Canada: trends and sex differences, 1980-2008. CMAJ, 184(9). Retrieved on November 29, 2013, from http://www.cmaj.ca/content/184/9/1029.full

Statistics Canada. (2013-09-18). Canadian Community Health Survey: Mental Health, 2012. Retrieved on December 3, 2013.

Statistics Canada. (2009). Mortality, Summary List of Causes. Retrieved on November 29, 2013.

The Standing Senate Committee on Social Affairs, Science and Technology. (2012-03). Time for Transformative Change: A Review of the 2004 Health Accord. Retrieved on January 31, 2014.

The Standing Senate Committee on Social Affairs, Science and Technology. (2006-05). Out of the Shadows At Last: Transforming Mental Health, Mental Illness and Addiction Services in Canada. Retrieved on December 2, 2013.

The Standing Senate Committee on Social Affairs, Science and Technology. (2005-10). A Proposal to Establish a Canadian Mental Health Commission. Retrieved on November 29, 2013.

The Standing Senate Committee on Social Affairs, Science and Technology. (2003-11). Reforming Health Protection and Promotion in Canada: Time to Act. Retrieved on November 29, 2013.

Workplace Safety and Prevention Services. (2013-10-29). Challenges and Benefits in Implementing Health Risk Assessment Programs. Retrieved on January, 3, 2014.

List of Acronyms

ADM: Assistant Deputy Minister
CADTH: Canadian Agency for Drugs and Technologies in Health
CDR: Common Drug Review
CMHA: Canadian Mental Health Association
COMPUS: Canadian Optimal Medication Prescribing and Utilization Service
CPSI: Canadian Patient Safety Institute
EHR: Electronic Health Record
EMR: Electronic Medical Record
F/P/T: Federal/Provincial/Territorial
HCC: Health Council of Canada
HTA/OU: Health Technology Assessment/Optimal Use
Infoway: Canada Health Infoway
MHCC: Mental Health Commission of Canada
OECD: Organisation for Economic Cooperation Development
OM: Opening Minds
PAA: Program Alignment Architecture
PHAC: Public Health Agency of Canada
PPI: Proton Pump Inhibitor
SHN: Safer Healthcare Now!
SMBG: Self-Monitoring Blood Glucose
SPARK: Supporting the Promotion of Activated Research and Knowledge
TAMI: Talking About Mental Illness
WHO: World Health Organization

Footnotes

Footnote A1

The work of the HCC is overseen by 13 independent councillors appointed by participating jurisdictions, and the F/P/T Ministers of Health (with the exception of Quebec) are the HCC's Corporate Members.

Return to footnoteA1 Referrer

Footnote A2

To obtain a copy of the Logic Model graphic please use the following e-mail "Evaluation_Reports_HC_-_Rapports_Evaluation@hc-sc.gc.ca"

Return to footnoteA2 Referrer

Footnote A3

Public Health Surveillance Systems are outside of the scope of the Performance Evaluation.

Return to footnoteA3 Referrer

Endnotes

Footnote 1

Chalkidou, K. et al. (2013-12-21). Health technology assessment in universal health coverage. The Lancet, 382(9910), e48-49. Retrieved on December 2, 2013, from http://www.sciencedirect.com/science/article/pii/S0140673613625593

Return to footnote1 Referrer

Footnote 2

CADTH. (2013). A Decade of the CADTH Common Drug Review 2003-2013. Retrieved on December 3, 2013, from http://www.cadth.ca/media/pdf/CADTH-Annual-Report-2012-13-e.pdf.pdf

Return to footnote2 Referrer

Footnote 3

Canadian Medical Association. (2014-01-07). Canada ranks low on patient safety in international comparison. Canadian Medical Association Journal, 186(1), E12. Retrieved on December 2, 2013, from http://www.cmaj.ca/content/186/1/E12.full.pdf+html

Return to footnote3 Referrer

Footnote 4

Canadian Institute for Health Information. (2013-11-21). Benchmarking Canada's Health System: International Comparisons. Retrieved on December 4, 2013, from https://secure.cihi.ca/free_products/Benchmarking_Canadas_Health_System-International_Comparisons_EN.pdf

Return to footnote4 Referrer

Footnote 5

OECD. (2013). Health at a Glance 2013: OECD Indicators. Retrieved on January 3, 2014, from http://www.oecd.org/els/health-systems/Health-at-a-Glance-2013.pdf

Return to footnote5 Referrer

Footnote 6

Canadian Patient Safety Institute. (2012-06-21). Independent Evaluation of the Canadian Patient Safety Institute. Retrieved on January 3, 2014, from http://www.patientsafetyinstitute.ca/English/About/Documents/CPSI%20Evaluation.pdf

Return to footnote6 Referrer

Footnote 7

Canadian Patient Safety Institute. (2012-07). The Economics of Patient Safety in Acute Care Technical Report. Retrieved on December 3, 2013, from
http://www.patientsafetyinstitute.ca/English/research/commissionedResearch/
EconomicsofPatientSafety/Documents/Economics%20of%20Patient%20Safety%20-
%20Acute%20Care%20-%20Final%20Report.pdf

Return to footnote7 Referrer

Footnote 8

The Standing Senate Committee on Social Affairs, Science and Technology. (2005-10). A Proposal to Establish a Canadian Mental Health Commission. Retrieved on November 29, 2013, from http://www.parl.gc.ca/Content/SEN/Committee/381/soci/rep/rep16nov05-e.pdf

Return to footnote8 Referrer

Footnote 9

Return to footnote9 Referrer

Footnote 10

Centre for Addiction and Mental Health. (2014-01). Mental Illness and Addiction Statistics. Retrieved on January 3, 2014, from
http://www.camh.ca/en/hospital/about_camh/newsroom/for_reporters/Pages/
addictionmentalhealthstatistics.aspx

Return to footnote10 Referrer

Footnote 11

Government of Canada. (2006). The Human Face of Mental Health and Mental Illness in Canada. Retrieved on November 29, 2013, from http://www.phac-aspc.gc.ca/publicat/human-humain06/pdf/human_face_e.pdf

Return to footnote11 Referrer

Footnote 12

Chief Public Health Officer. (2011). Report on the State of Public Health in Canada. Retrieved on December 2, 2013, from http://www.phac-aspc.gc.ca/cphorsphc-respcacsp/2011/index-eng.php

Return to footnote12 Referrer

Footnote 13

Risk Analytica (for the Mental Health Commission of Canada). (2011-12). The Life and Economic Impact of Major Mental Illnesses in Canada: 2011 to 2041. Retrieved on December 2, 2013, from http://www.mentalhealthcommission.ca/English/node/5024

Return to footnote13 Referrer

Footnote 14

Statistics Canada. (2013-09-18). Canadian Community Health Survey: Mental Health, 2012. Retrieved on December 3, 2013, from http://www.statcan.gc.ca/daily-quotidien/130918/dq130918a-eng.pdf

Return to footnote14 Referrer

Footnote 15

Workplace Safety and Prevention Services. (2013-10-29). Challenges and Benefits in Implementing Health Risk Assessment Programs. Retrieved on January, 3, 2014, from
http://www.conferenceboard.ca/Libraries/CASHC_PUBLIC/
nov2013_Implementing_employee_health_risk_assessment_programs_Elizabeth_Mills.sflb

Return to footnote15 Referrer

Footnote 16

Statistics Canada. (2009). Mortality, Summary List of Causes. Retrieved on November 29, 2013, from http://www.statcan.gc.ca/pub/84f0209x/84f0209x2009000-eng.pdf

Return to footnote16 Referrer

Footnote 17

Return to footnote17 Referrer

Footnote 18

Return to footnote18 Referrer

Footnote 19

Centre for Addiction and Mental Health. (2014-01). September 10 is World Suicide Prevention Day. Retrieved on January 3, 2014, from
http://www.camh.ca/en/hospital/about_camh/newsroom/
news_releases_media_advisories_and_backgrounders/current_year/Pages/September-10-is-World-Suicide-Prevention-Day.aspx

Return to footnote19 Referrer

Footnote 20

Return to footnote20 Referrer

Footnote 21

Scottish Government. (2013). Suicide Prevention Strategy 2013-2016. Retrieved on January 3, 2014, from http://www.scotland.gov.uk/Resource/0043/00439429.pdf

Return to footnote21 Referrer

Footnote 22

Mental Health Commission of Canada. (2013, 11-18). Opening Minds Interim Report. Retrieved on January 3, 2014, from http://www.mentalhealthcommission.ca/English/initiatives-and-projects/opening-minds/opening-minds-interim-report?routetoken=db076c6017d9a9801591223f2ef720fd&terminitial=211

Return to footnote22 Referrer

Footnote 23

Canadian Mental Health Association. (2014-01). Fast Facts About Mental Illness. Retrieved on January 3, 2014, from http://www.cmha.ca/media/fast-facts-about-mental-illness/

Return to footnote23 Referrer

Footnote 24

Return to footnote24 Referrer

Footnote 25

Return to footnote25 Referrer

Footnote 26

CBC News. (2013-04-22). E-health records saved medical system $1.3B in 6 years. Retrieved on January 3, 2014, from http://www.cbc.ca/news/politics/e-health-records-saved-medical-system-1-3b-in-6-years-1.1384119

Return to footnote26 Referrer

Footnote 27

The Commonwealth Fund. (2014-01). A Survey of Primary Care Doctors in Ten Countries Shows Progress in Use of Health Information Technology, Less in Other Areas: Synopsis. Retrieved on January 31, 2014, from http://www.commonwealthfund.org/Publications/In-the-Literature/2012/Nov/Survey-of-Primary-Care-Doctors.aspx

Return to footnote27 Referrer

Footnote 28

Return to footnote28 Referrer

Footnote 29

Canada Health Infoway. (2014-03-03). New Survey Reveals Canadians Want Access to Digital Health Tools. Retrieved on March 3, 2014, from https://www.infoway-inforoute.ca/index.php/news-media/2013-news-releases/new-survey-reveals-canadians-want-access-to-digital-health-tools

Return to footnote29 Referrer

Footnote 30

Office of the Auditor General. (2010-04). Electronic Health Records in Canada. Retrieved on January 31, 2014, from http://www.oag-bvg.gc.ca/internet/docs/parl_oag_201004_07_e.pdf

Return to footnote30 Referrer

Footnote 31

Government of Canada. (2013-03). Jobs, Growth and Long-Term Prosperity: Economic Action Plan 2013. Retrieved on January 3, 2014, from htt p://www.budget.canada.ca/2013/doc/plan/budget2013-eng.pdf

Return to footnote31 Referrer

Footnote 32

Health Canada. (2004-09-16). First Minister's Meeting on the Future of Health Care 2004: A 10-year plan to strengthen health care. Retrieved on January 31, 2014, from http://www.hc-sc.gc.ca/hcs-sss/delivery-prestation/fptcollab/2004-fmm-rpm/index-eng.php

Return to footnote32 Referrer

Footnote 33

The Standing Senate Committee on Social Affairs, Science and Technology. (2012-03). Time for Transformative Change: A Review of the 2004 Health Accord. Retrieved on January 31, 2014, from http://www.parl.gc.ca/content/sen/committee/411/soci/rep/rep07mar12-e.pdf

Return to footnote33 Referrer

Footnote 34

The Standing Senate Committee on Social Affairs, Science and Technology. (2003-11). Reforming Health Protection and Promotion in Canada: Time to Act. Retrieved on November 29, 2013, from http://www.parl.gc.ca/Content/SEN/Committee/372/soci/26app-e.pdf

Return to footnote34 Referrer

Page details

Date modified:: 2014-07-14

Synthesis Evaluation of Transfer Payments to Pan-Canadian Organizations 2008-2009 to 2012-2013

Table of Contents

List of Tables

Executive Summary

Description of Evaluation

Background Information

Conclusions - Relevance

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

Demonstration of Economy and Efficiency

Recommendations

Management Response and Action Plan

1.0 Evaluation Purpose

2.0 Program Description

2.1 Program Context

2.2 Program Profile

2.3 Program Alignment and Resources

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

3.2 Limitations and Mitigation Strategies

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Transfer Payments to Pan-Canadian Organizations

4.2 Relevance: Issue #2 - Alignment with Government Priorities

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent were pan-Canadian organizations' outcomes achieved?

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

4.5.1 Were the costs of outputs and outcomes reasonable in light of the program context for each of the transfer payments (Economy)?

4.5.2 Are there alternative approaches to meeting the transfer payments' objectives? (Efficiency)

5.0 Conclusions

5.1 Relevance Conclusions

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes

5.2.2 Demonstration of Economy and Efficiency

6.0 Recommendations

Appendix 1 - Logic ModelsFootnote A2

Description of the Logic Model for Canada Health Infoway's 2010 Funding Agreement

Description of the Logic Model for Infoway's 2007 Funding Agreement

Description of the Logic Model for Canada Health Infoway

Description of the Logic Model for Canadian Agency for Drugs and Technologies in Health (CADTH)

Description of the Overall Logic Model for CADTH and the Optimizing Health Systems Efficiency Initiative

Description of the Logic Model for Mental Health Commission of Canada - Level 1 (as of December 2010)

Description of the Logic Model for Mental Health Commission of Canada

Description of the Logic Model for Canadian Patient Safety Institute

Description of the Logic Model for Logic Model for the Health Council of Canada

Exhibit II-1 - Outcomes to be evaluated (Section 4.2 and 6.1.8of the Addendum Agreement)

Appendix 2 - References

List of Acronyms

Footnotes

Endnotes

Page details

Appendix 1 - Logic Models^{Footnote A2}