Evaluation of the Innovation Strategy 2009-2010 to 2013-2014

Prepared by
Office of Evaluation
Health Canada and the Public Health Agency of Canada

March 2015

Table of Contents

• Executive Summary
• Management Response and Action Plan
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Logic Model and Narrative
  • 2.4 Program Alignment and Resources
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and Mitigation Strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 - Continued Need for the Program
  • 4.2 Relevance: Issue #2 - Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)
  • 4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency
• 5.0 Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6.0 Recommendations
• Appendix 1 - Innovation Strategy Projects
• Appendix 2 - Logic Model
• Appendix 3 - Summary of Findings
• Appendix 4 - International Comparison
• Appendix 5 - Evaluation Description

List of Tables

• Table 1: Overview of Innovation Strategy Funding Phases
• Table 2: Financial Data (2009-2010 to 2013-2014)
• Table 3: Limitations and Mitigation Strategies
• Table 4: Stakeholder reach of Phase 1 interventions
• Table 5: Reported classification of Innovation Strategy project partnerships
• Table 6: Summary of knowledge product and knowledge activity outputs
• Table 7: Examples of Knowledge Product Use
• Table 8: Examples of Improved Health Outcomes
• Table 9: Assessment of Innovation Strategy-funded SEAK project's readiness for scale up
• Table 10: Financial Data (2009-2010 to 2013-2014)
• Table 11: Summary of Innovation Strategy leveraged and in-kind resources
• Table 12: Summary of loaned staff and volunteer hours for Innovation Strategy projects

Executive Summary

This evaluation covered the Public Health Agency of Canada's Innovation Strategy for the period from 2009-2010 to 2013-2014. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the Innovation Strategy. This was a scheduled evaluation for 2014-2015 on the Health Canada and the Public Health Agency of Canada Five-Year Evaluation Plan.

Program Description

The Public Health Agency of Canada's Innovation Strategy, delivered by the Centre for Health Promotion, is designed to test and deliver innovative population health interventions to reduce health inequalities and address priority complex public health problems and their underlying factors. Employing an intervention research approach, the Innovation Strategy supports the delivery of a set of population health interventions to improve health outcomes, while generating critical knowledge through monitoring and evaluation on (1) how the intervention brought about change and (2) the context in which the intervention worked best and for which populations. Currently, the program supports intervention research projects in the public health priority areas of mental health and healthy weights. Through intersectoral project partnerships and knowledge exchange activities, the Innovation Strategy disseminates and shares the knowledge generated from the projects in support of program and policy development nationally. Interventions that are determined to be appropriate for scale up (expanded, replicated, adapted and sustained) to reach a greater number of people and reduce health inequalities are identified.

Over the last five years (2009-2010 through 2013-2014), the Innovation Strategy had total Gs&Cs expenditures of approximately $54 million.

CONCLUSIONS - RELEVANCE

Continued Need

Complex population health issues such as mental illness and obesity remain a public health concern. To effectively respond to them, evidence-based population health interventions which act upon the social determinants of health are required. Currently, there is limited evidence in Canada related to effective population health interventions. As a result, there is a continued need for population health intervention research to generate knowledge about policy and program interventions that have the potential to act at a population level.

Alignment with Government Priorities

Reducing health inequalities is identified by the federal government as a priority. In the past five years, Canada has signed federal and international declarations on this issue. Government of Canada strategic policy and planning documents, including the Speech from the Throne, also reflect this as a priority area. Within the Public Health Agency of Canada, Strategic Horizons 2013-2018 identifies the population health approach, including reducing health inequalities, as the model that will guide the Agency's focus for its priorities and strategic directions for the next five years. The current Innovation Strategy priority areas of mental health and healthy weights are also aligned with federal and Agency priorities as outlined in planning and strategic policy documents including the Speech from the Throne and the Agency's Strategic Horizons.

Alignment with Federal Roles and Responsibilities

The Agency's role in health promotion and protection, and disease prevention is outlined in the Department of Health Act and the Public Health Agency of Canada Act. There is a clear federal role to promote the overall health and well-being of Canadians, particularly for public health issues of national concern such as obesity and mental health. The objectives and activities of the Innovation Strategy, which include reducing health inequalities through population health intervention research, are aligned with this federal role. The Innovation Strategy does not appear to duplicate the role of other stakeholders.

CONCLUSIONS - PERFORMANCE

Achievement of Expected Outcomes (Effectiveness)

Through the implementation, delivery, monitoring and evaluation of population health interventions, the Innovation Strategy has achieved its intended immediate outcomes. Population health interventions have been developed and are supported by intersectoral partnerships. A range of knowledge products have been developed, disseminated and accessed at the project level, and while knowledge products have also been developed at the program level, the extent to which they have been disseminated and accessed is less clear.

Progress has also been made by the Innovation Strategy towards most of its intermediate outcomes. At the project level, there is some evidence that knowledge generated through the program is informing policy and program changes; however, knowledge uptake has not been systematically tracked at the project or program level. A more consistent and systematic approach to tracking this information would facilitate the assessment of the program's progress towards this outcome. There are indications that the program is contributing to improved health outcomes, particularly in the Innovation Strategy priority area of mental health, which is further along in program delivery. These outcomes include improved knowledge and skills in children and families, including those in Aboriginal settings. Additional progress towards improved health outcomes is expected as projects complete Phase 2 and move into Phase 3, where full scale up will occur. Currently, there are indications of Phase 2 interventions in both priority areas demonstrating readiness for scale up.

Demonstration of Economy and Efficiency

The program design of the Innovation Strategy, including the phased approach to project funding and the staggered delivery of each priority area, has contributed to program efficiencies. Through the phased approach, only those projects demonstrating promise or effectiveness are funded in subsequent phases, thereby limiting the funding of less effective interventions. The staggered delivery of the priority areas has allowed for lessons learned from Mental Health Promotion projects to be applied to the delivery of Achieving Healthier Weights projects. In addition, the partnerships developed through the delivery of the Innovation Strategy have allowed projects to leverage approximately $5.7 million, and receive approximately $5.6 million of in-kind support since project delivery commenced (equal to 31% of project funding over that time frame).

Additional efficiencies could be gained through increased collaboration and information sharing within the Health Promotion and Chronic Disease Prevention (HPCDP) Branch and the Health Portfolio, specifically the Canadian Institutes of Health Research (CIHR) Institute for Population and Public Health (IPPH). While there have been informal connections made between the Innovation Strategy and these areas, few formal mechanisms currently exist to facilitate joint work planning, or information sharing.

The Innovation Strategy program has been active in collecting and using performance information, however, additional performance measurement is needed to better understand the impact of Innovation Strategy knowledge products, and to support the periodic follow up of interventions post-Innovation Strategy funding.

RECOMMENDATIONS

Recommendation 1:

Identify and action potential opportunities for strategic coordination of efforts and increased collaboration to leverage expertise and maximize efficiencies in the area of population health intervention research and in the two policy priority areas (mental health and healthy weights):

• Within the HPCDP Branch of the Public Health Agency of Canada; and
• With the CIHR - IPPH, and other areas of the Health Portfolio as appropriate.

Evaluation evidence (i.e., key informant interviews, document review) consistently revealed that while there have been informal connections made between the Innovation Strategy and relevant program areas, many of these connections have not been formalized. As a result, there may have been missed opportunities to enhance program efficiencies and effectiveness through collaboration related to joint work planning, and/or coordination of activities. The evaluation therefore recommends that, to the extent possible, these relationships be formalized, or that tools and processes to support collaboration are identified to support overall efficiencies and contribute towards advancing progress in population health intervention research and in the two policy priority areas.

Recommendation 2:

Develop a formal strategy to guide the sharing of information and lessons learned through the Innovation Strategy program, related to the policy priority areas (currently mental health and healthy weights) and population health intervention research.

Overall, the evaluation identified that although the Innovation Strategy has resulted in the development of a number of knowledge products, there has been no formal strategy guiding the knowledge dissemination efforts of the program. To support the achievement of program outcomes related to stakeholders accessing and using Innovation Strategy-generated evidence, a formal strategy to guide information sharing is recommended.

Recommendation 3:

Enhance performance measurement activities related to:

• Tracking of information uptake and use to better measure and understand program impact, particularly in the policy priority areas; and
• Follow-up of projects post-Innovation Strategy funding to determine if interventions were sustained.

In order to fully measure and understand the effectiveness of the Innovation Strategy, improvements to performance measurement activities are needed. Currently, knowledge product uptake is not systematically tracked at either the project or program level. As a result, it is not clear if or how Innovation Strategy knowledge products are being used. Additionally, to gain perspective on intervention sustainability post-funding, it is recommended that periodic follow up of projects post-Innovation Strategy funding take place.

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Public Health Agency of Canada's Innovation Strategy for the period of April 2009 to August 2014.

This was a scheduled evaluation as per the Public Health Agency of Canada/Health Canada approved Five-Year Evaluation Plan 2013-2014 to 2017-2018, in fulfillment of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

2.0 Program Description

2.1 Program Context

The Innovation Strategy originated from the former Population Health Fund (PHF), a grant and contribution program that funded voluntary not-for-profit organizations and educational institutions to develop knowledge and program models to address the determinants of health. Established in 1997, the PHF program design was based on accumulating evidence in support of the theory that aspects of the economic, social and physical environments (determinants of health) affect the health status of populations. The goal of the PHF was to increase community capacity for action on or across the determinants of health by funding projects to develop community-based models for applying the population health approach, increase the knowledge base about population health, and increase partnerships and intersectoral collaboration in Canada. The PHF administered transfer payments of approximately $12.1 million per year.

A 2007 review of the Promotion of Population Health Grants and Contributions (including the PHF) identified several weaknesses in the PHF approach including inconsistent priorities across the regions, smaller one-off projects with insufficient time to generate evidence of what worked, limited project funding for evaluation, and little sharing of lessons learned. In addition, there was an increased recognition of the complexity of public health issues and of the knowledge gaps about how to act effectively, providing additional incentive for an updated approach. As a result of the review, the Agency adopted the recommendation that the PHF be transformed into the Innovation Strategy in order to address several of the weaknesses identified with the PHF design.

The Innovation Strategy was established in 2009, based on a population health intervention research approach to build the evidence base to support interventions that focus on innovation and reducing health inequalities in Canada. At the outset of the program, the majority of public health research, population health research in particular, focused on the description of public health problems rather than the identification of potential solutions. Consequently, there was little evidence available for public health policy-makers and practitioners to inform decision-making regarding effective interventionsEndnote 1. Key features of the Innovation Strategy design included: providing a longer term, phased, funding approach; funding a smaller number of larger projects that covered several regions; dedicating a larger percentage of project funding to evaluation; ensuring that projects had sufficient capacity to carry out comprehensive evaluations; and developing a program knowledge exchange strategy to ensure that lessons learned were shared.

Previously delivered by the Strategic Initiatives and Innovations Directorate, accountability for the delivery of the Innovation Strategy was moved within the Health Promotion and Chronic Disease Prevention (HPCDP) Branch to the Centre for Health Promotion (CHP) in February 2013. Over the past five years (2009-2010 to 2013-2014) the program has administered approximately $9.7 million per year in transfer payments.

2.2 Program Profile

The Public Health Agency of Canada's Innovation Strategy is designed to test and deliver innovative population health interventions to reduce health inequalities and address priority complex public health problems and their underlying factors. Employing an intervention research approach, the Innovation Strategy supports the delivery of a set of population health interventions to improve health outcomes, while generating critical knowledge through monitoring and evaluation on (1) how the intervention brought about change and (2) the context in which the intervention worked best and for which populations. Stated more simply, the Innovation Strategy studies and promotes 'what works' and ‘what doesn't work’ in public health. Through intersectoral project partnerships and knowledge exchange activities, the Innovation Strategy disseminates and shares the knowledge generated from the projects in support of program and policy development nationally. Interventions that are determined to be appropriate for scale up (i.e., expanded, replicated, adapted and sustained) to reach a greater number of people and reduce health inequalities are identified.

To support the development, implementation and evaluation of innovative population health interventions, the Innovation Strategy provides funding support to external organizations in a variety of sectors such as health and education. These organizations receive Innovation Strategy funding to study interventions in the current Innovation Strategy priority areas of mental health and healthy weights. Funded projects focus on actions to address underlying environmental, social, demographic and economic conditions related to these priority areas, including a focus on northern and remote communities, children and youth, and those with low incomes. Each project is required to involve local and community partners and is encouraged to collaborate with the research/academic community, the health sector,  non-governmental organizations, all levels of government, social services and, where possible, the private sector. The Innovation Strategy priority areas reflect complex public health problems (obesity and mental illness) and were determined through internal and external consultation processes, availability of existing evidence and alignment with Agency and Branch priorities.

Intervention research in each Innovation Strategy priority area is funded through a “phased” approach. In the first phase (innovation phase), funding is provided for a 12-18 month period to support early development and implementation of population health interventions. In the second phase (learning phase), funding is provided for up to four years towards the full implementation and evaluation of the population health interventions. The intervention research approach used by the Innovation Strategy requires projects to submit an extensive evaluation of the intervention and synthesis of results by the end of Phase 2. It is anticipated that, by the end of the second phase, interventions with the greatest potential for scale up would be identified and Phase 3 would focus on scale up the interventions^{Endnote 2}. Scale up refers to a deliberate effort to increase the reach and impact of successfully tested population health promotion interventions to benefit more people and foster sustainable policy/program development across diverse populations and communities. The delivery of Innovation Strategy projects has been staggered by priority area with Mental Health Promotion project delivery having started in 2009-2010 and Achieving Healthier Weights having commenced in 2012-2013.

At the outset of each phase of Innovation Strategy funding for both priority areas, calls for proposals are issued by the Innovation Strategy program. A formal review committee is assembled to assess proposals submitted by external organizations for relevance, innovation merit, readiness for full implementation and evaluation, and knowledge development and exchange plans. Committee members are external to the Agency and have expertise in the identified health priority, population health interventions and public policy. When recruiting committee members, the Innovation Strategy program attaches priority to the selection of researchers with population health intervention experience and expertiseEndnote 3. As each phase of funding ends, projects are invited to submit a letter of intent, signifying their desire to continue in the next phase. Those projects demonstrating the most merit or promise, as assessed by the review committee, are invited to submit proposals to participate in the subsequent phase. Table 1 provides an overview of the two priority areas in terms of start dates and number of projects funded per phase. Of note, between 2010-2011 and 2012-2013, the Innovation Strategy also funded five Phase 1 projects related to Managing Obesity across the lifecycle (separate from the projects in the Achieving Healthier Weights priority area); however, these interventions were not funded beyond Phase 1 and are not included within the scope of this evaluation. A summary of Phase 2 funded Innovation Strategy projects is included in Appendix 1.

Note: In addition to the above, $1.77M in one time funding was provided for Phase 1 of projects related to Managing Obesity across the lifecycle (not included in evaluation scope).

Source: Innovation Strategy program files

During Phase 1, projects typically received between $250,000 and $400,000 total for the implementation of their interventions. During Phase 2, where interventions expand to additional communities, funding increased to $1.5 to $2.5 million total per project.

Governance

Overall accountability for the achievement of Innovation Strategy program outcomes rests with the Centre for Health Promotion. Key functions for the Innovation Strategy include:

• Ensuring effective delivery of contribution projects, including managing calls for proposals, assessing and recommending proposals to the Minister, monitoring recipient reporting, and overseeing funded projects.
• Supporting knowledge development and exchange related to population health, intervention research, and priority areas (mental health and healthy weights);
• Building and maintaining partnerships related to population health, intervention research and priority areas;
• Managing linkages with regions; and
• Managing program budgets and expenditures.

Regional Offices also support delivery of the Innovation Strategy program. Key activities or functions for regional staff include providing support for performance measurement, knowledge development and exchange, and regional linkages for the Innovation Strategy in accordance with the Agency Centre-Regional Operations Agreement and Business Management Model. Of note, within the Public Health Agency of Canada, the Centre for Chronic Disease Prevention is the policy lead for healthy weights; and the Centre for Health Promotion is the policy lead for mental health.

2.3 Program Logic Model and Narrative

The long term or ultimate outcome for the Innovation Strategy focuses on national adoption or uptake of population health interventions to contribute to improved health outcomes and reduced health inequalities for Canadians across the life course. The activity areas, outputs, immediate and intermediate outcomes to achieve this final outcome are described below.

Program Activities

The Innovation Strategy program carries out activities in two key areas to support the achievement of intended outcomes: 1) Implement and test innovative population health interventions and 2) Knowledge development and exchange. As part of the former, the program funds, supports and monitors organizations to design, develop, implement, adapt and evaluate community-based population health interventions that address complex public health issues facing children, youth and families. As part of the latter, the program collaborates with stakeholders to develop and disseminate evidence-based knowledge products and lessons learned about population health interventions.

Outputs and Immediate Outcomes

Through the implementation and testing of population health interventions, it is expected that population health interventions will be developed or adapted, and that tools, approaches and models to support their implementation will be identified. As a result, it is expected that promising population health interventions will be implemented and evaluated. Additionally, through these activities, it is expected that funded projects will participate in new and existing partnerships, and that methods and models to promote intersectoral partnerships will be identified. These partnerships are expected to support the delivery of the population health interventions. Knowledge development and exchange activities are expected to lead to the production of knowledge syntheses, research papers, training materials, presentations, webinars, case studies, and summary reports. Once produced, it is expected that stakeholders (health practitioners, researchers, and other policy makers within and outside of the health sector) will access knowledge products and synthesized learnings to advance population health policy and practice.

Intermediate Outcomes

Three intermediate outcomes have been articulated for the Innovation Strategy program. Progress towards these outcomes is expected to be measurable approximately four years following initial funding of projects. At this time, it is expected that population health interventions will contribute to improved protective factors, reduced risk behaviours, and improved health outcomes for individuals, families and communities. Consistent with the timing of the Innovation Strategy phases, it is also expected that population health interventions will demonstrate readiness for scale up. With respect to knowledge exchange, it is expected that stakeholders will use knowledge products, intervention research evidence and synthesized learnings from the Innovation Strategy to advance population health policy and practice.

The connection between the program activity areas and the expected outcomes is depicted in the program logic model (see Appendix 2). The evaluation assessed the degree to which the defined outputs and outcomes have been achieved.

2.4 Program Alignment and Resources

The program is part of the Agency's Program Alignment Architecture (PAA): Program 1.2 Health Promotion and Disease Prevention, Sub-Program 1.2.2 Conditions for Healthy Living.

The program's expenditures for the years 2009-2010 through 2013-2014 are presented below (Table 2). Overall, the program had a budget of $54 million over five years. A summary of the program's planned versus actual spending is reviewed in section 4.5.

Note: The Innovation Strategy managed several one-time Grants and Contributions throughout the five year evaluation period, that are outside of the scope of this evaluation, but for which financial data is included in the above table. Funding of these one-time Grants and Contributions totalled $6,952,572.

Source: Public Health Agency of Canada, Office of the Chief Financial Officer

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from 2009-2010 to 2013-2014, and included Phases 1 and 2 of the Mental Health Promotion and the Achieving Healthier Weights funding of the Innovation Strategy. The following activities were considered out of scope for this evaluation: mental health promotion activities carried out by the Centre for Health Promotion that are not part of the Innovation Strategy and will be reviewed as part of the Evaluation of Mental Health and Mental Illness activities scheduled for 2015-2016, and the promotion of healthy weights activities carried out by the Centre for Chronic Disease Prevention's Healthy Living Fund that were included in the current Evaluation of the Agency's Chronic Disease Prevention Activities. Even though these activities were out of scope, these programs were consulted as part of this evaluation. Also outside the scope of this evaluation are the one-time Grants and Contributions managed through the Innovation Strategy that were not part of the Mental Health Promotion or Achieving Healthier Weights components of the Strategy.

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 3. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned.

The Policy on Evaluation (2009) guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the policy. A non-experimental design was used based on the evaluation matrix, which outlined the evaluation strategy for these activities.
Data collection activities were carried out between June and November 2014. Data for the evaluation was collected using various methods, including: literature review, document and file review, financial data review, international review and key informant interviews - both internal and external. More specific details on the data collection and analysis methods are provided in Appendix 2. Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. Table 3 outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

4.0 Findings

This section provides a summary of the findings organized under two broad headings:

• Relevance: the need, priorities, and federal public health role in preventing chronic diseases.
• Performance: the effectiveness, efficiency and economy of the Agency's activities in this area.

4.1 Relevance: Issue #1 - Continued Need for the Program

The Public Health Agency of Canada's Innovation Strategy contributes to addressing a need for effective population health interventions to mitigate the health inequalities that contribute to complex public health issues such as mental health and obesity.

The World Health Organization defines good health as a state of complete physical, social and mental well-being, and not merely the absence of disease^{Endnote 4}. This paradigm suggests that at every stage of life, health is determined by complex interactions between social and economic factors, the physical environment and individual behaviour. These factors are referred to as the determinants of health^{Endnote 5}. The determinants of health include a wide range of personal, social, economic and environmental factors such as education, employment, income, social status, housing, gender and culture^{Endnote 6}. It is suggested that differences in health status from the combination and interaction of health determinants result in health inequalities between individuals and among various population groups^{Endnote 7}. The resulting challenge lies in how to use knowledge of the determinants of health to understand how to improve the health of Canadians^{Endnote 8}. Population health refers to health outcomes and their distribution in the population^{Endnote 9}. The population health approach aims to improve the health of the entire population, and to reduce health inequalities among specific population groups^{Endnote 10}.

Population health interventions are used to address the complex interaction between the social determinants of health. A 2008 Senate Subcommittee report recognised that disease issues are complex, largely due to the diverse social determinants of healthEndnote 11; therefore, a population health approach is needed to create change at the population level. Population health intervention research uses scientific methods to produce knowledge about policy and program interventions that operate within or outside the health sector and have the potential to impact health at the population levelEndnote 12,Endnote 13. While the majority of public health research focuses on the prevention and causes of public health problems, intervention research focuses on solutions to learn more about what works, under what conditions, and howEndnote 14 thereby increasing the understanding about the impact of policies and programs to improve population health and reduce health inequalitiesEndnote 15. Population health intervention research generates relevant, contextually sensitive, credible and timely knowledge that enables decision makers to continually improve programs and policies. An assessment of population health intervention research supported through CIHR concluded that even though interest in evidence-informed decision making related to population health has been growing, there is a lack of population health intervention research being funded in CanadaEndnote 16. In addition, a 2009 paper highlighting the Canadian Cancer Society's Centre for Behavioural Research and Program Evaluation's experiences in advancing the goals of the Population Health Intervention Research Initiative for Canada (PHIRIC) noted that the majority of research to date has been descriptive in natureEndnote 17 and needs to identify effective solutionsEndnote 18.

In Canada, there are questions about which policy, program and/or intervention will improve the health of particular populations. However, evidence-based information is often lacking, and there is limited capacity for conducting the kind of research needed to provide answers relevant to the Canadian context^{Endnote 19}. As a result, there is little evidence available for public health policy-makers and practitioners to inform decision-making regarding effective interventions. In addition, there is little information available to demonstrate how a promising intervention can be scaled up, sustained, and eventually replicated by others. The Innovation Strategy seeks to address this gap in knowledge by funding population health intervention research to generate knowledge about policy and program interventions that have the potential to impact health at a population level^{Endnote 20}.

Obesity and mental health are complex health problems that are influenced by a range of biological, social, physical and economic factors. As such effective population health interventions are required to help address these ongoing public health problems.

The following statistics highlight the prevalence and economic burden of mental illness and obesity.

• Every year, one in ten Canadians will experience a mental illness, and one in three will experience one sometime in their lives^{Endnote 21}. In 2008, the direct cost of mental illness in Canada was estimated to be at least $7.4 billion^{Endnote 22}. As for vulnerable populations, Aboriginal youth commit suicide five to six times more often than non-Aboriginal youth^{Endnote 23}.
• It is estimated that approximately two out of every three adults in Canada are overweight or obese, and almost one in three children and youth is overweight or obese^{Endnote 24}, costing the Canadian economy up to $7.1 billion each year^{Endnote 25}. In 2007-2010, Aboriginal adults had higher obesity rates than non-Aboriginal adults: First Nations (26%), Inuit (26%), Métis (22%) versus non-Aboriginal (16%)^{Endnote 26}. A recent study of socioeconomic-related inequalities in obesity risk among Canadian adults found that obesity is more prevalent among economically disadvantaged women^{Endnote 27}.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

While there is no specific link between federal priorities and population health intervention research per se, reducing health inequalities through the development of evidence-informed population health interventions aligns with federal priorities.

In October 2011, the World Conference on Social Determinants of Health was held with participants from 125 Member States including Canada. The goal of the conference was to gather support for action on the social determinants of health, with a focus on developing strategies for reducing health inequalities^{Endnote 28},^{Endnote 29}. At the conference, the Rio Political Declaration on Social Determinants of Health was adopted, reaffirming that health inequalities within and between countries are politically, socially and economically unacceptable, as well as unfair and largely avoidable^{Endnote 30}. The Rio declaration underscores the principles set out in the WHO Constitution and the 1986 Ottawa Charter that both promote a focus on health equity^{Endnote 31},^{Endnote 32}.

In the January 2009 Speech from the Throne, the Government of Canada committed to protect vulnerable populations including the unemployed, lower-income Canadians, seniors and Aboriginal Canadians. In the 2010 Speech from the Throne, the government stated that protecting the health of Canadians is a priority. More recent Budgets and Speeches from the Throne committed to protecting the health of Canadians, in particular the health and well-being of Aboriginal peoples.

Within the Public Health Agency of Canada, Strategic Horizons 2013-2018 identifies the population health approach, including reducing health inequalities, as the model that will guide the Agency's focus for its priorities and strategic directions for the next five years. Strategic Horizons 2013-2018 also identifies knowledge development and exchange, enhanced information sharing, strengthened partnerships and collaboration as key priorities. Knowledge development and exchange is built in as a requirement of Innovation Strategy project funding. Partnerships and collaboration are key components of funded projects. As noted in the Health Promotion and Chronic Disease Prevention Branch strategic and operational plans, there is alignment between Agency priorities and current activities of the Innovation Strategy.

The Innovation Strategy priorities of mental health and healthy weights are also aligned with broader Government of Canada and Public Health Agency of Canada priorities. The Economic Action Plan (2012) and the October 2013 Speech from the Throne highlighted the government's commitment to work with the provinces and territories, private and not-for-profit sectors to encourage young Canadians to be more physically active. The plan also proposed funding for mental health research.

The Government of Canada, along with the provincial and territorial Ministers of Health, reinforced that the prevention of chronic diseases and related conditions such as healthy weights was a priority by collaborating with its provincial and territorial counterparts to develop national agreements. The 2005 Pan-Canadian Healthy Living Strategy was strengthened in 2010, and pointed to new areas of opportunity, including preventing obesity. To this end, two initiatives were endorsed between 2010 and 2011:

• Creating a Healthier Canada: Making Prevention a Priority - A Declaration on Prevention and Promotion from Canada's Ministers of Health and Health Promotion and the Sport Recreation and Physical Activity Ministers, which was a public statement to work together, and with others, to make the promotion of health and prevention of disease, disability and injury a priority for action; and
• Curbing Childhood Obesity: A Federal, Provincial and Territorial Framework for Action to Promote Healthy Weights, which focused on reducing prevalence of childhood obesity in Canada and advancing strategies for Federal, Provincial and Territorial collaboration.

Within the Agency, Strategic Horizons 2013-2018 identified healthy weights, obesity, mental health and intervention research as priorities. Intervention research was identified as a key priority to help the Agency to achieve its Strategic Outcome - Protecting Canadians and empowering them to improve their health. The Agency's Corporate Risk Profile also identified obesity and mental health as key priorities. In addition, the Public Health Network Council (of which the Agency's Chief Public Health Officer is the federal co-chair) has listed mental health and healthy weights as priorities in its current strategic priorities documents.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

There is a federal public health role to conduct research, provide leadership and promote overall health. Intervention research as defined by the Innovation Strategy aligns with this role.

The Agency's role in health promotion and protection, and disease prevention is outlined in the Department of Health Act and the Public Health Agency of Canada Act. There is a clear federal role to promote the overall health and well-being of Canadians, particularly for public health issues of national concern such as obesity and mental health. The Agency's more specific roles to conduct research, provide leadership and support knowledge dissemination are presented under various program authorities.

The Innovation Strategy grants and contributions fall under the Terms and Conditions for Promotion of Population Health and are intended to promote the health of Canadians, while reducing health inequalities. The Innovation Strategy focuses on dissemination of knowledge of the results of promising population health interventions, and sharing these practices across the country with the aim of reducing health inequalities in its two priority areas. The Innovation Strategy focuses on the following four interrelated pillars that align with federal role in public health:

• Development of partnerships and collaboration;
• Evaluation and performance measurement;
• Development of tools and resources; and
• Knowledge development, exchange and uptake.

Further, the Constitution Act, 1867 outlines the provincial and territorial responsibility for delivering health care, but also identifies the federal government's roles and responsibilities such as ensuring access to health care for specific populations (e.g., Aboriginal peoples), health promotion, disease prevention, knowledge sharing, and funding health research. As an agency of the federal government, the Public Health Agency of Canada fulfills these roles to fund research, and promote overall health. Similarly, the objectives of the Innovation Strategy align with the Agency's role in reducing health inequalities^{Footnote i}, and promoting the physical, social and mental well-being of Canadians^{Endnote 33},^{Endnote 34}, in addition to sharing knowledge of effective population health interventions^{Endnote 35},^{Endnote 36}.

While the Public Health Agency of Canada funds intervention research through the Innovation Strategy, it is not the only program within the Agency that is funding the delivery of interventions with a particular focus on vulnerable populations, nor is it the only federal government department/agency supporting population health intervention research. A challenge to identifying those involved in intervention research however lies in the fact that intervention research is defined differently within and outside the Agency. For instance, within the Agency, there are programs that fund the delivery of population health interventions (e.g., Aboriginal Head Start in Urban and Northern Communities, Community Action Program for Children, Canada Prenatal Nutrition Program, Fetal Alcohol Spectrum Disorder Initiative, Integrated Strategy on Healthy Living and Chronic Disease, Community Associated Infections and the Federal Initiative to Address HIV/AIDS)^{Footnote ii}. The extent to which the delivery of these interventions constitutes or includes intervention research however, may be subject to interpretation.

Outside the Agency, other government departments that appear to support the delivery of population health interventions include Health Canada's First Nations and Inuit Health Branch (FNIHB), the Department of Employment and Social Development Canada, and the Canadian Institutes of Health Research (CIHR)^{Footnote iii}. It is not clear if the population health intervention work of Health Canada or Employment and Social Development Canada include an intervention research component. CIHR is the Government of Canada's agency responsible for funding health research. This includes funding intervention research. CIHR has a mandate to create new scientific knowledge and enable its translation into improved health, more effective health services and products, and a strengthened Canadian health care system. CIHR provides leadership and support to health researchers through its 13 institutes. Each institute is made up of a network of researchers brought together for a specific area of focus, such as the Institute of Population and Public Health that supports research into the determinants of health to improve the health of Canadians while promoting health equity^{Endnote 37},^{Endnote 38},^{Endnote 39}.

Overall, the Innovation Strategy does not appear to duplicate the work of CIHR (discussed in section 4.5). For example, while the Innovation Strategy funds the development, implementation, evaluation, and scale up of interventions, CIHR's IPPH primarily funds research on population health interventions. The Pathways to Health Equity for Aboriginal Peoples is an initiative that is led by CIHR, in partnership with the Public Health Agency of Canada and Health Canada's FNIHB. This initiative aims to develop the evidence base in the design and implementation of programs and policies that promote health and health equity in the following four priority areas: suicide prevention, tuberculosis, diabetes/obesity and oral healthEndnote 40. Possible overlap with the Innovation Strategy may exist given that this initiative is involved in the development, implementation, evaluation and scale up of interventions; however, this project is solely focused on Aboriginal peoples and funding is only used for research. Unlike the Innovation Strategy, funding of the intervention is explicitly excluded in the case of the Pathways Initiative.

Internationally, Australia, the United Kingdom, the United States and Canada are similar in that they all have government organizations responsible for health research, with a particular focus on addressing/reducing health inequalities and improving health outcomes. Each country also appears to place importance on knowledge development and exchange. Most similar to the Innovation Strategy, the U.S.' National Institute on Minority Health and Health Disparities Community-based Participatory Research Initiative uses a phased approach to support the delivery and research of population health interventions. Both initiatives incorporate the use of community partnerships to improve health outcomes and reduce health inequalities. The programs differ in that the U.S. Initiative provides funding for up to 11 years and funding recipients are generally academic researchers. Comparable programs in Australia and the United Kingdom were not identified as part of this international web review; however, the Australian Centres of Research Excellence and the UK Clinical Research Collaboration have coordinated research partnershipsFootnote iv,Footnote v to bring together all relevant components of the health system (discussed in more detail in Appendix 4).

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the immediate outcomes been achieved?

Immediate outcome #1: Population health interventions are implemented and evaluated

Through the Innovation Strategy, population health interventions aimed at promoting mental health and reducing barriers to healthy weights have been implemented and evaluated as planned. Tools, approaches and methods to support the implementation of interventions have been developed.

As discussed in Section 2.2, the first phase of Innovation Strategy funding is intended to support projects through the early development and implementation of population health interventions. Overall (both priority areas combined), a total of 52 interventions received Phase 1 funding (15 Mental Health Promotion interventions, 37 Achieving Healthier Weights interventions). Of these, a total of 51 population health interventions were implemented. One intervention funded under the Mental Health Promotion priority area was unable to complete reporting for phase 1 and therefore is not included in this report. As per funding requirements, all of the population health interventions involved in Phase 1 were identified as being new and/or adapted interventions.

Interventions funded under the Innovation Strategy priority area of Mental Health Promotion were organized under three clusters: interventions that aimed to address parenting competencies and family cohesion, school-based interventions that aimed to influence risk and protective factors, and interventions that sought to increase community capacity to address child and youth mental health. Interventions funded under the Innovation Strategy priority area of Achieving Healthier Weights were organized under five clusters: interventions that aimed to address food security (access to food), family based initiatives that aimed to support early childhood and youth, interventions that aimed to promote healthy school programs and environments, northern community-based initiatives and interventions that aimed to promote support environments (encouraging healthy lifestyles).

Projects in both priority areas covered rural, remote, urban, and inner-city areas across the country and targeted populations of varying socio-economic status, literacy level, and cultural background.

As part of their reporting requirements for Phase 1, Innovation Strategy-funded projects were required to report on the reach of their interventions (Table 4). The 14 Mental Health Promotion interventions that were implemented during Phase 1 reached approximately 5,000 individuals between 2009-2010 and 2010-2011. Within the Achieving Healthier Weights priority area, the 37 interventions that were implemented in Phase 1 reached approximately 84,000 individuals between 2011-2012 and 2012-2013. In general, these interventions reached a range of program stakeholders including: individuals who face specific risk factors, public health practitioners, policy makers and the public. In both priority areas, the majority of individuals reached (68% for Mental Health Promotion and 70% for Achieving Healthier Weights) were individuals who face specific risk conditions or risk factors. In Phase 2, the Innovation Strategy projects increased the numbers of intervention sites (discussed in more detail in section 4.4.2) and are now reaching close to 550,000 individuals across the country.

Source: Innovation Strategy, Core Indicator Reports

The Innovation Strategy was developed to explore innovation and learning in population health interventions in order to reduce inequalities. As such, as a component of Phase 1, projects were asked to report on innovative tools, approaches, and processes that were used to support the implementation of population health interventions and models. A total of eight tools/approaches were identified and deemed innovative by the program in the Mental Health Promotion priority area, and a total of 23 tools/approaches were identified and deemed innovative in the Achieving Healthier Weights priority areas during Phase 1. Examples of innovative tools/approaches used in the Mental Health Promotion priority area include: the use of multiple strategies to develop intersectoral partnerships, adaptation of interventions (to local context), emphasis on community engagement, and implementation of interventions in multiple settings. Examples of innovative tools/approaches used in the Achieving Healthier Weights priority area include: multi-sectoral and multilevel partnership development, tailoring and targeting of communications, leveraging of existing partnerships, and a community development approach.

As intended, all population health interventions that participated in Phase 1 of the Innovation Strategy were evaluated. On an annual basis, all projects were required to submit a completed “Project Evaluation and Reporting Tool”. This tool is used to monitor progress towards objectives and collect key performance data. In addition, all Phase 1 projects in both priority areas were required to submit final evaluation reports. These reports provided a summary on the implementation of the project and the results achieved.

Immediate outcome #2: Partnerships support delivery of interventions

A range of intersectoral partnerships have been established by all projects that support the delivery of the population health interventions.

The development of intersectoral partnerships was common to the delivery of all Innovation Strategy projects, across each phase and in both priority areas. Indeed, 100% of the funded Innovation Strategy projects to date have engaged in a range of partnerships to support the delivery of their interventions. Looking at each priority area individually, the ratio of partners to projects has increased over time. For Mental Health Promotion projects, 148 partners were reported in Phase 1, and 289 partners were reported in Phase 2. Given that there were 15 projects in Phase 1 and nine in Phase 2, the ratio of partners to projects rose from approximately 10:1 in the first phase to 32:1 in the second phase. Within the Achieving Healthier Weights priority area, 599 partners were reported in Phase 1, and 262 partners were reported in Phase 2. Given that there were 37 projects in Phase 1 and 11 in Phase 2, the ratio of partners to projects rose from approximately 16:1 in the first phase to 24:1 in the second phase^{Footnote vi}.

Partners have offered various forms of support to the projects including financial and in-kind resources (loaned staff, volunteers) (discussed in section 4.5 on efficiency and economy), support to enhance community engagement, expertise, support to create change in policy and support to disseminate information. In the Mental Health Promotion priority area, a series of interviews (n=31) were conducted with project partners to classify the nature of their relationship with the project implementers. In the Achieving Healthier Weights priority area, project staff were asked to categorize the nature of their relationships (n=262) with their partners. The results for both are shown in Table 5. Of note, despite the various ways in which the data was collected, approximately 20% of partnerships in each priority area are classified as ‘collaborative', characterized by frequent communication, mutual trust, and consensus reached on all decisions. Further, the majority of partnerships were categorized by project staff and partners as being at least ‘cooperative', characterized by formal communication channels, independent decision making and somewhat defined roles.

Source: Innovation Strategy Core Indicator Report, Partnership Survey Report

As noted by key informants, the complex population health issues being addressed by the Innovation Strategy are influenced by a range of social determinants, many of which extend beyond the domain or influence of public health. For this reason, all Innovation Strategy projects have prioritized and engaged in intersectoral partnerships. According to program documentation, partnerships are intersectoral when the project partner is working with the project implementers to meet objectives and goals and where the partner is either a different type of organization (i.e., private, public, not-for-profit), operates at a different geographic level or scope (i.e., international, national/federal, provincial/territorial) and/or has a different primary area of focus (i.e., health, education, social services, academia/research) than that of the funded project. Consistent across both priority areas, the projects have predominately engaged with the public and not-for-profit sectors, and to a lesser extent with the private sector. With respect to geographic level, projects in both priority areas have worked fairly evenly across the local/municipal, regional, provincial/territorial, and national levels. The most common areas of focus for partners in both priority areas were health, education, social services, and Aboriginal issues. Examples of intersectoral partners include: school boards, provincial ministries, local municipalities, grocery stores, Inuit community governments, local community wellness groups, law enforcement, parent organizations and Aboriginal friendship centres.

Immediate outcome #3: Stakeholders access knowledge products

A range of knowledge products have been developed, disseminated and accessed at the project level, and while knowledge products have also been developed at the program level, the extent to which they have been disseminated and accessed is less clear.

Over the course of both phases in each priority area, Innovation Strategy projects have been active in developing and disseminating knowledge products, informed by learnings through the implementation and delivery of their interventions. A variety of knowledge products have been produced, including manuals/training kits, brochures/pamphlets/posters, web sites, newsletters and position papers/research summaries. These products have been disseminated to project stakeholders through project websites, targeted mail outs, and knowledge activities. Indeed, Innovation Strategy projects participated in a range of knowledge activities including workshops/presentations, activities to influence policy, and community events. To date, Innovation Strategy projects have produced approximately 730 knowledge products and participated in approximately 2,721 knowledge activities (Table 6). Further, in both priority areas, the ratio of knowledge products produced to number of projects has increased from Phase 1 to Phase 2.

Note: Given that each priority area is at a different stage of delivery, the Phase 2 data in Table 5 includes three years of data for Mental Health Promotion projects, but only one for those related to Achieving Healthier Weights.
Source: Innovation Strategy, Core Indicator Reports

Performance measurement data collected by the Innovation Strategy projects included estimates of the degree to which knowledge products and activities were accessed. Estimates were informed by attendance tracked at knowledge events, or through tracking of dissemination efforts including website hits, newsletter mail outs, and knowledge product downloads. In general, project level knowledge products appear to be well accessed by program stakeholders. In the Mental Health Promotion priority area, knowledge products and activities were made accessible to an estimated 22,395 stakeholders in the first phase, approximately 5,294,766^{Footnote vii} stakeholders in the first two years of Phase 2, and to approximately 273,682 stakeholders in the second reporting year for Phase 2. In the Achieving Healthier Weights priority area, knowledge products and activities were made accessible to an estimated 557,070 stakeholders in Phase 1, and to approximately 107,779 stakeholders in Phase 2.

Innovation Strategy program staff have also developed and disseminated knowledge products. At the program level, a total of 50 knowledge products have been developed over the last five years. These include summaries of the Innovation Strategy program to inform stakeholders (5), project summaries including details on findings (5), compilations of project performance information (summative across program) (15), research summaries (evidence reviews) to support project implementation and enhance knowledge around Innovation Strategy priorities (25), and knowledge events (15). To date, the program has produced a limited number of rolled up analyses (informed by Innovation Strategy projects) related to implementation science in general. However, key informants noted that such products would be more informed if produced in Phase 3.

In general, program level knowledge products have been shared with all projects, and internal program stakeholders. However, the extent to which they have been disseminated and accessed is relatively unknown as there has been inconsistent tracking of knowledge product dissemination at the program level. Key informants (including program staff and management) noted that program-level knowledge products have not been accessed as much as they would like. The program has recently begun to develop a knowledge dissemination and uptake plan that will introduce more consistent tracking of knowledge exchange activities.

4.4.2 To what extent have the intermediate outcomes been achieved?

Intermediate outcome #1: Stakeholders use knowledge products

While there are examples of knowledge products being used by stakeholders, particularly at the project level, the precise extent to which use is occurring, and how, has generally not been well documented at the project or program level.

The extent to which Innovation Strategy knowledge products (project and program level) are used, and how, is not well understood because the overall uptake of Innovation Strategy knowledge products has not been systematically or consistently tracked across the Innovation Strategy program. For example, details of how the projects tracked this information or for what purposes stakeholders used each of the knowledge products they produced was not required as part of the reporting. In each priority area, project staff reported on whether knowledge products had been used. In Phase 1of the Mental Health Promotion priority area, 21% of projects reported that stakeholders had used their knowledge products. In Phase 2 (2012-2013 reporting year), this number rose to 78%. In the priority area of Achieving Healthier Weights, 73% of projects reported that their knowledge products had been used by stakeholders in Phase 1, and in the first year of Phase 2, this percentage has remained at 73%.

Despite the limitations on how this information was tracked, examples of knowledge product use were available in both priority areas (Table 7). In some cases, the knowledge products were discussed or cited in stakeholder practice and policy forums. Examples also existed of knowledge products having been used by stakeholders to inform practice and policy. In some cases, Innovation Strategy knowledge products have even led to the implementation or adaptation of an intervention, policy or program outside of Innovation Strategy funding.

Source: Innovation Strategy program files

Similarly, at the program level, the use of knowledge products has not been systematically tracked. However, the program has recently conducted a series of surveys with individuals (n=36) who have requested knowledge products from the program in 2013-2014. To date, these surveys reveal that approximately 86% of these individuals have used the Innovation Strategy knowledge products they requested. Of these, approximately one third have used the Innovation Strategy knowledge product to support decision-making, while approximately one fifth have used knowledge to inform their work or practice. The program is currently developing a knowledge dissemination and uptake plan to support a more systematic and consistent approach to tracking this information at both the project and program level.

Intermediate outcome #2: Improved health outcomes

There are early indications that the population health interventions funded by the Innovation Strategy are contributing to improved health outcomes. These include improved knowledge and skills in children and families in diverse populations including Aboriginal settings.

Measureable progress towards this outcome is not expected until four to six years after projects begin Phase 1. While projects in the Mental Health Promotion priority area have been delivered for this duration of time, projects in the Achieving Healthier Weights priority area have not. As a result, most evidence in this section relates to projects from the priority area of Mental Health Promotion.

Each year, as part of their annual reporting requirements, Innovation Strategy projects report on progress made towards this outcome. More specifically, projects report on whether their activities or interventions have contributed to: improved health practices, skills, and/or outcomes, improved protective factors, and/or reduced risk factors in their target populations. In general, changes in target populations were measured by project staff using a mix of methods, including: pre-post surveys, retrospective surveys, focus groups, and observational data.

In the Mental Health Promotion priority area, progress towards this outcome has been made over time. In Phase 1 reporting, data was not available on this outcome as many projects had not yet even attempted to measure it. In the first year of Phase 2 reporting (2012-2013), 44% of projects reported that they had contributed to improved health practices, skills and/or outcomes, while the remainder noted that it was still too early to make an assessment. In the most recent year of Phase 2 reporting (2013-2014), 78% of projects reported that they had contributed to positive changes in health outcomes. For projects in the Achieving Healthier Weights priority area, similar trends were observed. Data was not available in Phase 1, and in the first year of reporting for Phase 2, 36% of projects reported that they had contributed to improved health outcomes.

Overall, changes in health outcomes, protective factors and risk factors have been measured at both the individual, family and community levels. In the Mental Health Promotion priority area, changes at the individual (child and youth) level have related to factors such as resilience, self-esteem and self-image, coping and social skills, and communication and conflict or problem-solving skills. In Aboriginal settings, emphasis was placed on factors such as identity, and sense of connection with family, community and culture. Outcomes for families have included improved parental skills and positive family interactions and cultural connections. In the school environment, observations suggested changes in teachers' approach and a culture shift in school environment. Specific examples of improved health outcomes in both priority areas are provided in Table 8.

Source: Innovation Strategy program files

Intermediate outcome #3: Readiness for scale up

There are early indications of interventions demonstrating readiness for scale up, which is Phase 3 of the Innovation Strategy.

As noted in section 2.2, the third phase of the Innovation Strategy will support the scale up of effective population health interventions. Phase 3 for projects in the Mental Health Promotion priority area is scheduled to begin in 2015-2016, and for projects in the Achieving Healthier Weights priority area, not until 2017-2018. In both priority areas, there are already early indications of interventions demonstrating readiness for scale up.

A recently produced Innovation Strategy program knowledge product entitled “Understanding Scale Up”^{Endnote 41} presents key factors for effective scale up of interventions, informed through the delivery of Phase 1 and 2 of the Innovation Strategy. Key factors for scale up include the following:

• Overall Readiness: Is the intervention ready for scale up?
• Capacity: Is there established organizational infrastructure, with the resources required for scale up? What is the best way to achieve scale up?
• Partnership Development: Are there vested partners with sustainable networks who are engaged in the project?
• System Readiness: What are the institutional impediments to scale up? What are the type and quality of resources required for scale up?
• Community Context: Are the sites ready with adequate resources, knowledge, development and exchange capacity and evaluation? Is the cultural context within the community supportive of the intervention?
• Evaluation and knowledge, development and exchange: How will the scale up process be evaluated? What are the knowledge, development and exchange mechanisms needed to share results?
• Cost Factors: What are the costs associated with the interventions compared to the costs of no action? How will scale up be sustained?

To help inform which Innovation Strategy projects will move forward to Phase 3, the Innovation Strategy program will assess each project against these criteria. While the evaluation was not able to report on all of these key factors, evidence was available for some of them. In terms of overall readiness, feedback from a range of key informants (program management and staff, Agency staff) indicated that there are interventions in both priority areas demonstrating readiness for scale up. With respect to capacity, particularly organizational infrastructure, it's clear that projects have moved from implementation and design to intervention delivery as they have expanded to additional locations, helping to prepare them for scale up. There is at least one Innovation Strategy intervention being delivered in all provinces/territories in a range of communities (up to 47 communities for one intervention). There is at least one Innovation Strategy project being delivered in all provinces/territories in a range of communities (up to 47 communities for one project). In Phase 2, nine Mental Health Promotion projects are currently delivered in 240 communities across Canada. Similarly, the eleven Phase 2 Achieving Healthier Weights projects are currently delivered in 100 communities across Canada.

In terms of partnership development, the success of the projects in this outcome area has already been discussed in section 4.4.1. However, it is also noteworthy that in both priority areas, projects are demonstrating the formation of committed (sustainable) partnerships. Approximately 60% of the project partnerships in the Mental Health Promotion priority area have been sustained for over three years, and while projects in the Achieving Healthier Weights priority area have not been established as long, 33% of these partnerships have been sustained for at least three years.

With respect to system readiness, key informants noted that P/T bodies play an important role related to project scale up and sustainability. Project level partnership activities are currently self-directed; however, the program encourages partnerships with P/T stakeholders. To date, 65% of projects (overall) have partnered with P/T bodies. During data collection with P/T representatives (n = 2), the notion of P/T engagement to facilitate system readiness was reinforced. It was suggested by these key informants that P/T bodies should be engaged early in project implementation to further facilitate project delivery and scale up. Failure to engage P/T bodies at project outset was identified as a potential impediment to scale up.

An example of an Innovation Strategy intervention demonstrating readiness for scale up is the Mental Health Promotion project entitled “Socially-Emotionally Aware Kids” (SEAK), led by the Canadian Mental Health Association - Nova Scotia. The SEAK project teaches elementary school children aged 5-12 the Promoting Alternative Thinking Strategies (PATHS) program: How to deal with emotions, pro-social behaviours, how to succeed in school, ways to prevent bullying, what to do when being the target of physical, social, emotional, or psychological harm, how to recognize suicide warning signs, how to reach out for immediate help and professional treatment. Table 9 provides an overview of ways in which the SEAK project has demonstrated readiness for scale up, based on the key factors discussed above.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

Innovation Strategy projects have successfully leveraged funding to facilitate more economical delivery of the program. Efficiencies could be gained through improved collaboration and knowledge exchange with other areas of the HPCDP Branch and the Health Portfolio, specifically CIHR-IPPH. In general, Innovation Strategy activities are well managed and work has begun to improve performance measurement related to tracking the impact of knowledge products.

The Treasury Board Policy on Evaluation (2009) and the guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.

The evaluation provided observations on economy and efficiency based on findings from the document review, key informant interviews and available relevant financial data. In addition, the findings below provide observations on the adequacy and use of performance measurement information to support economical and efficient program delivery and evaluation.

Observations on Economy

In general, over the past five years, the Innovation Strategy has spent 98% of their planned resources (not including leveraged resources). As illustrated below (Table 10), there were some variances between planned and actual spending during the period evaluated. The largest variance was a surplus of 13.4% that occurred in 2009-2010, attributed largely to delays in program start up and implementation, while the smallest variance was a deficit of 1.9% that occurred in 2013-2014.

Note: The Innovation Strategy managed several one-time Grants and Contributions throughout the five year evaluation period, that are outside of the scope of this evaluation, but for which financial data is included in the above table. Funding of these one-time Grants and Contributions totalled $6,952,572.

Source: Public Health Agency of Canada, Office of the Chief Financial Officer

As previously discussed under the outcome related to partnerships supporting program delivery (4.4.1), Innovation Strategy projects have leveraged funds from external sources and obtained in-kind resources. Overall, Innovation Strategy projects have leveraged approximately $5.7 million, and received approximately $5.6 million of in-kind support (Table 11). Combined, this represents approximately 31% of total IS funding to date. Leveraged funds have most often been received from other federal government departments, P/T governments, regional health authorities, not-for-profit organizations and universities.

Of note, not all Innovation Strategy projects have been successful in leveraging additional funds. In the Mental Health Promotion priority area, three of the fifteen projects were responsible for all leveraged funds in Phase 1, while four of the nine funded projects successfully leveraged funds in Phase 2. In the Achieving Healthier Weights priority area, funds were leveraged by 22 of the 37 projects in Phase 1 and by five of the eleven projects in Phase 2. A greater percentage of projects in both priority areas were involved in obtaining in-kind resources. Approximately 80% of all (both phases) Mental Health Promotion projects and 90% of all Achieving Healthier Weights projects contributed to obtaining the in-kind resources. Factors that contributed to projects' ability to leverage funds or in-kind resources included project design, including nature and number of stakeholders, as well as geographic characteristics of project sites, including specific location, community size and infrastructure.

Source: Innovation Strategy program files

In addition, Innovation Strategy projects have also benefitted from loaned staff and volunteers (Table 12). Overall, Innovation Strategy projects have benefitted from approximately 110,000 hours of loaned staff time and at least 28,700 hours of volunteer time. In both priority areas, the majority of projects have been successful in obtaining these additional supports. In the Mental Health Promotion priority area, approximately 75% of projects in both phases have been successful in obtaining these resources (both loaned staff and volunteers). In the Achieving Healthier Weights priority area, approximately 90% of projects in each phase have benefitted from loaned staff, while approximately 70% of projects in each phase have obtained volunteer support. Common contributions made by volunteers included: marketing and writing press articles, peer mentorship, event organizing, teaching workshops, steering committee participation, program implementation, the distribution of evaluation materials and involvement in data collection for evaluation reporting.

Source: Innovation Strategy program files

Observations on Efficiency

The phased approach to the delivery of Innovation Strategy projects, combined with the on-going monitoring and evaluation of projects, supports program efficiency. With this approach, only those projects that demonstrate effectiveness or promise through intervention evaluations are moved to subsequent phases of funding. This helps prevent the funding of less effective interventions, while fostering those with the most potential. The staggered delivery of the individual priority areas has also resulted in program efficiencies. This has allowed the program to collect and analyze lessons learned in the delivery of the Mental Health Promotion projects and apply them to the delivery of projects in the Achieving Healthier Weights priority area.

Factors contributing to program inefficiencies were also observed. For instance, key informants (program staff, Agency staff) suggested that, since other programs within the HPCDP Branch and the Health Portfolio (CIHR) work in topic areas similar to the Innovation Strategy priority areas, there are opportunities for increased collaboration to promote complementarity and coordination of efforts and learn from one another. For example, outside of the Innovation Strategy, the Centre for Health Promotion also delivers other mental health promotion activities, while the Centre for Chronic Disease Prevention (Healthy Living) is the policy lead for the Public Health Agency of Canada in the area of Healthy Weights. Additionally, within each Centre, there are several programs (i.e., Community Action Program for Children (CAPC), Canada Prenatal Nutrition Program (CPNP), Aboriginal Head Start in Urban and Northern Communities (AHSUNC), and the Integrated Strategy on Healthy Living and Chronic Disease) that may address complementary topic areas or include intervention research-based elements (discussed in section 4.3), for which Innovation Strategy learnings may be applicable or useful.

According to key informants, while there have been informal connections made between these groups and the Innovation Strategy program with respect to knowledge exchange, no formal mechanisms (i.e., joint work planning) currently exist between these groups to share best practices or coordinate activities. As a result, many of the knowledge products developed by the Innovation Strategy do not appear to be reaching Agency staff that may benefit from them. Conversely, best practices and/or knowledge products developed by other programs that could inform or be used by Innovation Strategy projects may not be reaching the program. For example, there may be an opportunity to improve the coordination of Innovation Strategy work in the Achieving Healthier Weights policy area with the Centre for Chronic Disease Prevention (CCDP), which is the policy lead within the Public Health Agency of Canada for promotion of Healthy Weights. This might involve improved knowledge sharing between the two Centres (CHP and CCDP) with respect to policy, priorities and funding opportunities related to healthy weights. Formalizing this connection would support improved linkages between the Innovation Strategy and CCDP. Another example might include increased information sharing with other CHP programs, including CAPC, CPNP and AHSUNC. Given that Innovation Strategy interventions focus on children, youth and families, there may be opportunities to exchange lessons learned with CAPC, CPNP and AHSUNC programs, which are focused on healthy childhood development. The Innovation Strategy has not met with CAPC, CPNP or AHSUNC staff on a consistent basis and so there is opportunity for more deliberate processes and defined ways to share these lessons learned. This could include the rich knowledge of the population and communities in which CAPC, CPNP and AHSUNC serve and the emerging outcomes and evidence from Innovation Strategy funded projects on knowledge exchange and partnerships.

Outside the Public Health Agency of Canada, other government departments (Health Canada and Employment and Social Development Canada) are also involved in delivering population health interventions (discussed in section 4.3). The CIHR (Institute for Population and Public Health) is also involved in the funding of population health intervention research. Other population health intervention research-related activities are currently implemented by CIHR (e.g., Pathways to Health Equity, described in section 4.3) in partnership with the Public Health Agency of Canada.
The Innovation Strategy has built a relationship with CIHR, and the two organizations participate in the Population Health Intervention Research Initiative for Canada. This initiative is supported by the Institute of Population and Public Health, and aims to advance population health intervention research within Canada, increase capacity to conduct population health intervention research and enhance the global knowledge base^{Endnote 42}. According to key informants however, additional opportunities exist for increased, or more formal, collaboration. For example, the knowledge exchange mechanisms for the Pathways to Health Equity Initiative are outlined in a memorandum of understanding signed by CIHR, the Agency and FNIHB and are operationalized through a formal governance structure within Pathways to Health Equity Initiative, the Health Portfolio Coordinating Committee. Similarly, the Applied Public Health Chairs program has involved joint priority setting between the Agency and CIHR's Institute for Population and Public Health. Formal mechanisms such as those outlined in the above examples (i.e., memorandum of understanding and joint priority setting) might represent potential means by which the Innovation Strategy and CIHR relationship could be more formalized.

Observations on the Adequacy and Use of Performance Measurement Data

The program has been successful in monitoring performance, using researched and standardized tools. Performance data is systematically analyzed and used to inform program delivery (e.g., lessons learned report). However, to gain a better understanding of the impact of Innovation Strategy knowledge products, additional performance measurement is needed. The program has recently begun efforts to this end, including a dissemination plan and a knowledge uptake plan (for collecting information on knowledge product use).

While projects have not yet begun Phase 3, it will also be important for the program to consider long term performance measurement. For example, it may be of benefit to carry out periodic follow up of interventions post Innovation Strategy-funding in order to monitor project sustainability.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

Complex population health issues such as mental illness and obesity remain a public health concern. To effectively respond to them, evidence-based population health interventions which act upon the social determinants of health are required. Currently, there is limited evidence in Canada related to effective population health interventions. As a result, there is a continued need for population health intervention research to generate knowledge about policy and program interventions that have the potential to act at a population level.

5.1.2 Alignment with Government Priorities

Reducing health inequalities is identified by the federal government as a priority. In the past five years, Canada has signed federal and international declarations on this issue. Government of Canada strategic policy and planning documents, including the Speech from the Throne, also reflect this as a priority area. Within the Public Health Agency of Canada, Strategic Horizons 2013-2018 identifies the population health approach, including reducing health inequalities, as the model that will guide the Agency's focus on its priorities and strategic directions for the next five years. The current Innovation Strategy priority areas and activities of mental health and healthy weights are also aligned with federal and Agency priorities as outlined in planning and strategic policy documents including the Speech from the Throne and the Agency's Strategic Horizons.

5.1.3 Alignment with Federal Roles and Responsibilities

The Agency's role in health promotion and protection, and disease prevention is outlined in the Department of Health Act and the Public Health Agency of Canada Act. There is a clear federal role to promote the overall health and well-being of Canadians, particularly for public health issues of national concern such as obesity and mental health. The objectives and activities of the Innovation Strategy, which include reducing health inequalities in the areas of mental health promotion and achieving healthy weights through population health intervention research, are aligned with this federal role. The Innovation Strategy does not appear to duplicate the role of other stakeholders.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

Through the implementation, delivery, monitoring and evaluation of population health interventions, the Innovation Strategy has generally achieved its intended immediate outcomes. Population health interventions have been developed and are supported by intersectoral partnerships. A range of knowledge products have been developed, disseminated and accessed at the project level, and while knowledge products have also been developed at the program level, the extent to which they have been disseminated and accessed is less clear.

Progress has also been made by the Innovation Strategy towards most of its intermediate outcomes. At the project level, there is some evidence that knowledge generated through the program is informing policy and program changes; however, knowledge uptake has not been systematically tracked at the project or program level. A more consistent and systematic approach to tracking this information would facilitate the assessment of the program's progress towards this outcome. There are indications that the program is contributing to improved health outcomes, particularly in the Innovation Strategy priority area of Mental Health Promotion, which is further along in program delivery. These outcomes include improved knowledge and skills in children and families, including those in Aboriginal settings. Additional progress towards improved health outcomes is expected as projects complete Phase 2 and move into Phase 3, where full scale up will occur. Currently, there are indications of Phase 2 interventions in both priority areas demonstrating readiness for scale up.

5.2.2 Demonstration of Economy and Efficiency

The program design of the Innovation Strategy, including the phased approach to project funding and the staggered delivery of each priority area, has contributed to program efficiencies. Through the phased approach, only those projects demonstrating promise or effectiveness are funded in subsequent phases, thereby limiting the funding of ineffective interventions. The staggered delivery of the priority areas has allowed for lessons learned from Mental Health Promotion projects to be applied to the delivery of Achieving Healthier Weights projects.

Additional efficiencies could be gained through increased collaboration and information sharing within the HPCDP Branch and the Health Portfolio, specifically the CIHR-IPPH. While there have been informal connections made between the Innovation Strategy and these areas, few formal mechanisms currently exist to facilitate joint work planning, or information sharing.

The Innovation Strategy program has been active in collecting and using performance information, however, additional performance measurement is needed to better understand the impact of Innovation Strategy knowledge products, and to support the periodic follow up of interventions post-Innovation Strategy funding.

6.0 Recommendations

The findings from this evaluation have resulted in the following three recommendations.

1. Identify and action potential opportunities for strategic coordination of efforts and increased collaboration to leverage expertise and maximize efficiencies in the area of population health intervention research and in the two policy priority areas (mental health and healthy weights):

• Within the HPCDP Branch of the Public Health Agency of Canada; and
• With the CIHR - IPPH, other areas of the Health Portfolio as appropriate.

Evaluation evidence (i.e., key informant interviews, document review) consistently revealed that while there have been informal connections made between the Innovation Strategy and relevant program areas, many of these connections have not been formalized. As a result, there may have been missed opportunities to enhance program efficiencies and effectiveness through collaboration related to joint work planning, and/or coordination of activities. The evaluation therefore recommends that, to the extent possible, these relationships be formalized, or that tools and processes to support collaboration are identified to support overall efficiencies and contribute towards advancing progress in population health intervention research and in the two policy priority areas.

2. Develop a formal strategy to guide the sharing of information and lessons learned through the Innovation Strategy program, related to the policy priority areas (currently mental health and healthy weights) and population health intervention research.

Overall, the evaluation identified that although the Innovation Strategy has resulted in the development of a number of knowledge products, there has been no formal strategy guiding the knowledge dissemination efforts of the program. To support the achievement of program outcomes related to stakeholders accessing and using Innovation Strategy-generated evidence, a formal strategy to guide information sharing is recommended.

3. Enhance performance measurement activities related to:

• Tracking of information uptake and use to better measure and understand program impact, particularly in the policy priority areas; and
• Follow-up of projects post-Innovation Strategy funding to determine if interventions were sustained.

In order to fully measure and understand the effectiveness of the Innovation Strategy, improvements to performance measurement activities are needed. Currently, knowledge product uptake is not systematically tracked at either the project or program level. As a result, it is not clear if or how Innovation Strategy knowledge products are being used. Additionally, to gain perspective on intervention sustainability post-funding, it is recommended that periodic follow up of projects post-Innovation Strategy funding take place.

Appendix 1 - Innovation Strategy Projects

 

Appendix 2 - Logic Model

Footnotes

1. Logic model outcome directly linked to PAA 1.2.2. results and indicators (See Section 2.2 and Annex A for additional information)
2. Intervention research: the use of research methods to produce knowledge about interventions that have the potential to impact health at the population level, including how, when and in what context the interventions work or do not work.
3. Scale up: a deliberate effort to increase the reach and impact of successfully tested population health promotion interventions to benefit more people and to foster sustainable policy/programme development across diverse populations and communities.

Appendix 3 - Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance:

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

 

Appendix 4 - International Comparison

Australia

Australia's National Health and Medical Research Council (NHMRC) is the largest single funder of health and medical research in Australia. Many of NHMRC's funding schemes address multiple priority actions contained in their 2013-2015 Strategic Plan including intervention research in the area of obesity^{Endnote 43}. Knowledge, development and exchange to support the health of Australians is also an important part of NHMRC's 2013-2015 Strategic Plan.

Additionally, the Centres of Clinical Research Excellence scheme provides funding for innovative, high quality clinical research to:

• support clinical research with potential to lead to improved health outcomes for the community;
• foster training of clinical researchers, particularly those with a capacity for independent research and future leadership roles; and
• ensure effective translation of research outcomes into clinical practice^{Endnote 44},^{Endnote 45}.

United Kingdom

Healthcare in the United Kingdom is devolved, meaning that there are four regions: England, Scotland, Wales and Northern Ireland, each with its own healthcare system. The Department of Health is the Ministerial department responsible for government policy on health and social matters in England, along with a few elements not otherwise devolved to the Scottish Government, Welsh Government or Northern Ireland Executive. Although devolved, Public Health England (an Executive Agency of the Department of Health), the Public Health Agency in Northern Ireland, the National Health Service in Wales and the National Health Service in Scotland all explicitly state that their role or mission is to improve health and address/reduce health inequalities^{Endnote 46},^{Endnote 47},^{Endnote 48},^{Endnote 49}.

The National Institute for Health Research is a UK government body that coordinates and funds research for the National Health Service in England. The National Institute for Health Research has four main work strands: research, infrastructure, faculty and systems that are managed by National Institute for Health Research Coordinating CentresEndnote 50. In 2006, the National Institute for Health Research was launched as a ‘virtual' organisation, meaning that it is not a corporation or a legal entity or a ‘bricks and mortar' enterprise in the traditional sense; it is an overarching entity which collectively represents all publicly-funded research in the National Health Service.Endnote 51

Also in 2006, the major funders of public health research in the UK came together under the UK Clinical Research Collaboration (UKCRC) to develop a coordinated approach to improving the UK public health research environment. The findings of the UKCRC Public Health Research Strategic Planning Group are documented in a report, Strengthening Public Health Research in the UK. The outcome of which was a commitment by a consortium of eight funding partners^{Footnote viii} to create five UKCRC Public Health Research Centres of Excellence in 2008^{Endnote 52},^{Endnote 53},^{Endnote 54}.

The Centre for Development and Evaluation of Complex Interventions for Public Health (DECIPHer), a strategic partnership between Cardiff, Bristol and Swansea Universities in Wales, is one of the five Centres for Excellence. DECIPHer develops, tests, evaluates and implements complex interventions and policies that achieve sustainable improvements in health and wellbeing, and address health inequalities. DECIPHer's research focuses on three broad priority areas relating to the health of children and young people: tobacco, alcohol and drugs; obesity, physical activity and diet; mental health and wellbeingEndnote 55.

United States

In the United States, the National Institute on Minority Health and Health Disparities is the leader within the US National Institutes of Health^{Footnote ix} for scientific research to improve minority health and eliminate health disparities. The NIMHD Community-based Participatory Research (CBPR) Initiative supports collaborative research efforts between scientific researchers and community members to address diseases and conditions disproportionately affecting health disparity populations. The community is involved in the NIMHD CBPR Initiative as an equal partner with the scientists which helps to ensure that interventions created are responsive to the community's needs.

The NIMHD CBPR Initiative has three phases:

• Phase I (the Planning Phase) provides three years of funding for the community and its scientific research partners to conduct the needs assessment, identify their priorities, and design the intervention
• Phase II (the Research Intervention Phase) provides grantees up to five years of funding to refine the intervention, develop methods to evaluate its effectiveness, and implement the full-scale intervention.
• Phase III (the Information Dissemination Phase) provides three years of funding for the research community partners to share their research findings and insights gained from the intervention with the targeted CBPR community as well as with other researchers and organizations

Appendix 5 - Evaluation Description

Evaluation Scope

The scope of the evaluation included an assessment of the relevance and performance of the Innovation Strategy from 2009-2010 to 2013-2014.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Sources of information used in this evaluation included the following.

Document and File Review

All documents provided by the Project Authority for the purposes of the evaluation were reviewed to provide a foundation for the evaluation and contribute a line of evidence that addresses most evaluation questions. The following types of documents were reviewed: program- and project (study)-level descriptive and administrative materials; Government and departmental level policy and planning documents; and evaluation and performance reports: to maximize the efficiency and utility of the document review, a document review template was developed to facilitate the systematic review of materials.

Key Informant Interviews

The main purpose of the interviews was to gather information to fill gaps identified in the document/file review and to provide evidence and detailed information to help contextualize evidence gathered from other sources. Tailored guides were developed to be suitable for administration with two groups of key informants: internal respondents from the Public Health Agency who were involved in the design and delivery of the Innovation Strategy; senior management and staff from the Centre for Chronic Disease and Prevention (CCDP); staff from other divisions within the Centre for Health Promotion; other Health Portfolio staff (Health Canada and CIHR); provincial and territorial representatives and an external stakeholder from the Partnership Against Cancer's Coalitions Linking Action and Science for Prevention. In total, eight interviews with program representatives were conducted; five interviews were conducted with CCDP; three interviews were conducted with other Centre for Health Promotion staff; three were conducted with representatives from the Health Portfolio; three were conducted with provincial and territorial representatives and one interview was conducted an external stakeholder. In total, twenty-three interviews were conducted for this evaluation.

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included: systematic compilation, review and summarization of documents, quantitative analysis of data; thematic analysis of qualitative data from key informant interviews; and comparative analysis of data from disparate sources to validate summary findings.

References

• Armstrong, R., Waters, E., Moore, L., Riggs, E., Cuervo, L.G., Lumbiganon, P. and Hawe, P. (2008). Improving the reporting of public health intervention research: advancing TREND and CONSORT. Journal of Public Health, 30 (1), 103-109.
• Australian Government. (2012). NHMRC Strategic Plan 2013-2015. Retrieved from https://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/nh160_nhmrc_strat_plan_201315.pdf
• Bambra, C.L., Hiller, F.C., Moore, H.J., and Summerbell, C.D. (2012). Tackling inequalities in obesity: a protocol for a systematic review of the effectiveness of public health interventions at reducing socioeconomic inequalities in obesity amongst children. Systematic Reviews, 1(16), 1-7.
• Boivin, A., Lehoux, P., Burgers, J., and Grol, R. (2014). What Are the Key Ingredients for Effective Public Involvement in Health Care Improvement and Policy Decisions? A Randomized Trial Process Evaluation. The Milbank Quarterly, 92(2), 319-350.
• Cameron, R., Riley, B, Campbell, H.S., Manske, S. and Lamers-Bellio, K. (2009). The Imperative of Strategic Alignment Across Organizations: The Experience of the Canadian Cancer Society's Centre for Behavioural Research and Program Evaluation. Canadian Journal of Public Health, 100 (1), I27-I30.
• Canadian Institutes of Health Research. (2015). Population Health Intervention Research Initiative for Canada. Retrieved from http://www.cihr-irsc.gc.ca/e/38731.html
• Canadian Institutes of Health Research. (2015). About Us. Retrieved from http://www.cihr-irsc.gc.ca/e/37792.html
• Canadian Institutes of Health Research. (2015). About the Institutes. Retrieved from http://www.cihr-irsc.gc.ca/e/7155.html
• Canadian Institutes of Health Research. (2015). About PHIRIC. Retrieved from http://www.cihr-irsc.gc.ca/e/38732.html
• Canadian Institutes of Health Research. (2015). Institute of Population and Public Health. Retrieved from http://www.cihr-irsc.gc.ca/e/13777.html
• Canadian Institutes of Health Research. (2015). Pathways to Health Equity for Aboriginal Peoples. Retrieved from http://www.cihr-irsc.gc.ca/e/43630.html
• Canadian Mental Health Association. (2015). Fast Facts About Mental Illness. Retrieved from http://www.cmha.ca/media/fast-facts-about-mental-illness/#.VLS4yfnF_Fk
• Davies, J.K., and Sherriff, N. (2010). The gradient in health inequalities among families and children: A review of evaluation frameworks. Health Policy, 101(2011), 1-10.
• DECIPHer: Development and Evaluation of Complex Interventions for Public Health Improvement. (2015). About DECIPHer. Retrieved from http://decipher.uk.net/about-decipher/
• Di Ruggiero, E., Rose, A., and Gaudreau, K. (2009). Canadian Institutes of Health Research Support for Population Health Intervention Research in Canada. Canadian Journal of Public Health, 100(1), I15-I19.
• Edwards, N., and Di Ruggiero, E. (2011). Exploring which context matters in the study of health inequities and their mitigation. Scandinavian Journal of Public Health, 39(6), 43-49.
• Edwards, N. (2009). Revisiting our Social Justice Roots in Population Health Intervention Research. Canadian Journal of Public Health, 100(6), 405-406.
• Government of Canada. (1996). Department of Health Act. Retrieved from http://laws-lois.justice.gc.ca/PDF/H-3.2.pdf
• Government of Canada. (2015). Economic Burden of Illness in Canada (EBIC). Retrieved from http://ebic-femc.phac-aspc.gc.ca/custom-personnalise/results-national-resultats.php
• Government of Canada. (2014-12-20). Minister's Message: Healthy Living. Retrieved from http://www.healthycanadians.gc.ca/video/healthy-living-vie-saine-eng.php
• Government of Canada. (2006). Public Health Agency of Canada Act. Retrieved from http://lois-laws.justice.gc.ca/PDF/P-29.5.pdf
• Government of Canada. (2008). The Chief Public Health Officer's Report on the State of Public Health in Canada: Addressing Health Inequalities. Retrieved from http://www.phac-aspc.gc.ca/cphorsphc-respcacsp/2008/fr-rc/pdf/CPHO-Report-e.pdf
• Government of the United Kingdom. (2015). Public Health England. Retrieved from https://www.gov.uk/government/organisations/public-health-england
• Hajizadeh, M., Campbell, M.K. and Sarma, S. (2014-03). Socioeconomic inequalities in adult obesity risk in Canada: trends and decomposition analyses. European Journal of Health Economics, 15(2), 203-221.
• Haman, F., Fontaine-Bisson, B., Batal, M., Imbeault, P., Blais, J.M., and Robidoux, M.A. (2010). Obesity and type 2 diabetes in Northern Canada's remote First Nations communities: the dietary dilemma. International Journal of Obesity, 34, S24-S31.
• Hanusaik, N., O'Louhlin, J., Kishchuk, N., Paradis, G., and Cameron, R. (2009). Organizational capacity for chronic disease prevention: A survey of Canadian public health organizations. European Journal of Public Health, 20(2), 195-201.
• Hawe, P., Di Ruggiero, E., and Cohen, E. (2012). Frequently Asked Questions About Population Health Intervention Research. Canadian Journal of Public Health, 103(6), e468-e471.
• Hawe, P., Samis, S., Di Ruggiero, E., and Shoveller, J.A. (2011). Population Health Intervention Research Initiative for Canada: progress and prospects. NSW Public Health Bulletin, 22 (1-2), 27-32.
• Hawe, P., and Potvin, L. (2009). What is Population Health Intervention Research? Canadian Journal of Public Health, 100 (1), I8-I14.
• Health and Social Care in Ireland. (2015). Public Health Agency. Retrieved from http://www.publichealth.hscni.net/
• Health Canada. (2015). First Nations & Inuit Health: Mental Health and Wellness. Retrieved from http://www.hc-sc.gc.ca/fniah-spnia/promotion/mental/index-eng.php
• Hobin, E.P., Hayward, S., Riley, B., Di Ruggiero, E., and Birdsell, J. (2012). Maximising the use of evidence: exploring the intersection between population health intervention research and knowledge translation from a Canadian perspective. Evidence and Policy, 8(1), 97-115.
• Kendall, P.R.W. (2009). Commentary on Population Health Intervention Research. Canadian Journal of Public Health, 100(1), I31-I32.
• Lalonde, M. (1974). A New Perspective on the Health of Canadians: a Working Document (Catalogue No. H31-1374). Retrieved from http://www.phac-aspc.gc.ca/ph-sp/pdf/perspect-eng.pdf
• Mangham, L.J., and Hanson, K. (2010). Scaling up in international health: what are the key issues? Health Policy and Planning, 25, 85-96.
• Mental Health Commission of Canada. (2014). Why Investing in Mental Health Will Contribute to Canada's Economic Prosperity and to the Sustainability of our Health Care System. Retrieved from  http://strategy.mentalhealthcommission.ca/pdf/case-for-investment-en.pdf
• Merritt, M.W., Taylor, H.A., and Mullany, L.C. (2010). Ancillary Care in Community-Based Public Health Intervention Research. American Journal of Public Health, 100(2), 211-216.
• Morabia, A., and Costanza, M.C. (2012). Population Health Intervention Research (PHIR): Should we fear the “P”. Preventive Medicine, 54, 291-292.
• National Health Service. (2015). Public Health Wales: Priorities for Action. Retrieved from http://www.wales.nhs.uk/sitesplus/888/page/70468
• National Health Service. (2015). Health Scotland. Retrieved from http://www.healthscotland.com/
• National Institute for Health Research. (2015). About NIHR. Retrieved from http://www.nihr.ac.uk/about/
• National Institute for Health Research. (2015). History of the NIHR. Retrieved from http://www.nihr.ac.uk/about/history-of-the-nihr.htm
• Naylor, D. (2003). Learning from SARS: Renewal of Public Health in Canada (Catalogue No. H21-220/2003E). Retrieved from http://www.phac-aspc.gc.ca/publicat/sars-sras/naylor/index-eng.php
• Nutbeam, D. (2001). Evidence-based public policy for health; Matching research to policy. IUHPE Promotion and Education, 15-19.
• Pettersson, B. (2011). Some bitter-sweet reflections on the Ottawa Charter commemorative cake: a personal discourse from an Ottawa rocker. Health Promotion International, 26(2), 173-179.
• Prince, S.A. (2009). A population health approach to obesity in Canada - Putting the problem back into context. Transdisciplinary Studies in Population Health Series, I (I), 22-33.
• Public Health Agency of Canada. (2015). What Determines Health? Retrieved from http://www.phac-aspc.gc.ca/ph-sp/determinants/index-eng.php
• Public Health Agency of Canada. (2015). What is the Population Health Approach? Retrieved from http://www.phac-aspc.gc.ca/ph-sp/approach-approche/index-eng.php
• Public Health Agency of Canada. (2015). World Conference on Social Determinants of Health. Retrieved from http://www.phac-aspc.gc.ca/ph-sp/determinants/wcshd-cmdss-eng.php
• Public Health Agency of Canada. (2014). Innovation Strategy Performance Measurement Strategy.
• Public Health Agency of Canada. (2012-03-22). Scan of Current State of Intervention Research Funding & Support: Chronic Disease Prevention and Health Promotion.
• Public Health Agency of Canada. (2011). Obesity in Canada. Retrieved from http://www.phac-aspc.gc.utting ca/hp-ps/hl-mvs/oic-oac/assets/pdf/oic-oac-eng.pdf
• Sport Matters. (2013). Federal, Provincial and Territorial Ministers Advance Actions to Support Sport, Physical Activity and Healthy Weights. Retrieved from http://www.sportmatters.ca/en/news-82-federal-provincial-and-territorial-ministers
• Raine, K.D., Plotnikoff, R., Schopflocher, D., Lytvyak, E., Nykiforuk, C.I., Storey, K., Ohinmaa, A., Purdy, L., Veugelers, P., and Wild, T.C. (2013). Healthy Alberta Communities: Impact of a three-year community-based obesity and chronic disease prevention intervention. Preventive Medicine, 57, 955-962.
• Raine, K.D., Plotnikoff, R., Nykiforuk, C., Deegan, H., Hemphill, E., Storey, K., Schopflocher, D., Veugelers, P., Wild, T.C., and Ohinmaa, A. (2010). Reflections on community-based population health intervention and evaluation for obesity and chronic disease prevention: the Healthy Alberta Communities project. International Journal of Public Health, 55, 679-686.
• Rychetnik, L., Frommer, M., Hawe, P., and Shiell, A. (2002). Criteria for evaluating evidence on public health interventions. Journal of Epidemiological Community Health, 56, 119-127.
• Senate Subcommittee on Population Health. (2009-06). A Healthy Productive Canada: A Determinant of Health Approach. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/402/popu/rep/rephealth1jun09-e.pdf
• Senate Subcommittee on Population Health. (2008). Population Health Policy: Issues and Options. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/392/soci/rep/rep10apr08-e.pdf
• Senate Subcommittee on Population Health. (2014). The Health of Canadians - The Federal Role Interim Report. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/371/soci/rep/repintsep01part5-e.htm
• Smith, B.T., Smith, P.M., Harper, S., Manuel, D.G., and Mustard, C.A. (2013-12-20). Reducing social inequalities in health: the role of simulation modelling in chronic disease epidemiology to evaluate the impact of population health interventions. Journal of Epidemiology & Community Health, 1-6. Retrieved from http://jech.bmj.com/content/early/2013/12/20/jech-2013-202756.full.pdf+html
• Statistics Canada. (2015). Health at a Glance: Select health indicators of First Nations People living off reserve, Métis and Inuit . Retrieved from http://www.statcan.gc.ca/pub/82-624-x/2013001/article/11763-eng.htm
• Statistics Canada. (2013-09-18). Health at a Glance: Mental and substance use disorders in Canada. Retrieved from http://www.statcan.gc.ca/pub/82-624-x/2013001/article/11855-eng.htm
• Sullivan, L. (2009). Introduction to the Population Health Intervention Research Initiative for Canada. Canadian Journal of Public Health, 100 (1), I5-I6.
• Twells, L.K., Gregory, D.M., Reddigan, J., and Midodzi, W.K. (2014). Current and predicted prevalence of obesity in Canada: a trend analysis. Canadian Medical Association Journal, 2(1), E18-E26.
• World Health Organization. (2015). Rio Political Declaration on Social Determinants of Health. Retrieved from http://www.who.int/sdhconference/declaration/en/
• World Health Organization. (2006). Constitution of the World Health Organization. Retrieved from http://www.who.int/governance/eb/who_constitution_en.pdf
• World Health Organization. (2015). The Ottawa Charter for Health Promotion. Retrieved from http://www.who.int/healthpromotion/conferences/previous/ottawa/en/
• World Health Organization. (2015). Trade, foreign policy, diplomacy and health. Retrieved from http://www.who.int/trade/glossary/story046/en/
• World Health Organization. (2015). World Conference on Social Determinants of Health. Retrieved from http://www.who.int/sdhconference/en/
• Wozniak, L., Rees, S., Soprovich, A., Al Sayah, F., Johnson, S.T., Majumdar, S.R., and Johnson, J.A. (2012). Applying the RE-AIM framework to Alberta's Caring for Diabetes Project: a protocol for a comprehensive evaluation of primary care quality improvement interventions. BMJ Open, 2, 1-9.

List of Acronyms

AHSUNC

Aboriginal Head Start in Urban and Northern Communities

CAPC

Community Action Program for Children

CBPR

Community-based Participatory Research

CCDP

Centre for Chronic Disease Prevention

CHP

Centre for Health Promotion

CIHR

Canadian Institutes of Health Research

CPNP

Canada Prenatal Nutrition Program

DECIPHer

Centre for Development and Evaluation of Complex Interventions for Public Health

FNIHB

First Nations and Inuit Health Branch

F/P/T

Federal/Provincial/Territorial

HPCDP

Health Promotion and Chronic Disease Prevention

IPPH

Institute of Population and Public Health

IS

Innovation Strategy

NHMRC

National Health and Medical Research Council

NIMHD

National Institute on Minority Health and Health Disparities

PAA

Program Alignment Architecture

PATHS

Promoting Alternative Thinking Strategies

PHF

Population Health Fund

PHIR

Population Health Intervention Research

RPP

Report on Plans and Priorities

SEAK

Socially-Emotionally Aware Kids

TBS

Treasury Board of Canada Secretariat

UKCRC

United Kingdom Clinical Research Collaboration

Footnotes

Endnotes