Evaluation of the Innovation Strategy 2009-2010 to 2013-2014

For readers interested in the PDF version, the document is available for downloading or viewing:

Evaluation of the Innovation Strategy 2009-2010 to 2013-2014 (PDF document - 349 KB - 67 pages)

Prepared by
Office of Evaluation
Health Canada and the Public Health Agency of Canada

March 2015

Table of Contents

List of Tables

Executive Summary

This evaluation covered the Public Health Agency of Canada's Innovation Strategy for the period from 2009-2010 to 2013-2014. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the Innovation Strategy. This was a scheduled evaluation for 2014-2015 on the Health Canada and the Public Health Agency of Canada Five-Year Evaluation Plan.

Program Description

The Public Health Agency of Canada's Innovation Strategy, delivered by the Centre for Health Promotion, is designed to test and deliver innovative population health interventions to reduce health inequalities and address priority complex public health problems and their underlying factors. Employing an intervention research approach, the Innovation Strategy supports the delivery of a set of population health interventions to improve health outcomes, while generating critical knowledge through monitoring and evaluation on (1) how the intervention brought about change and (2) the context in which the intervention worked best and for which populations. Currently, the program supports intervention research projects in the public health priority areas of mental health and healthy weights. Through intersectoral project partnerships and knowledge exchange activities, the Innovation Strategy disseminates and shares the knowledge generated from the projects in support of program and policy development nationally. Interventions that are determined to be appropriate for scale up (expanded, replicated, adapted and sustained) to reach a greater number of people and reduce health inequalities are identified.

Over the last five years (2009-2010 through 2013-2014), the Innovation Strategy had total Gs&Cs expenditures of approximately $54 million.

CONCLUSIONS - RELEVANCE

Continued Need

Complex population health issues such as mental illness and obesity remain a public health concern. To effectively respond to them, evidence-based population health interventions which act upon the social determinants of health are required. Currently, there is limited evidence in Canada related to effective population health interventions. As a result, there is a continued need for population health intervention research to generate knowledge about policy and program interventions that have the potential to act at a population level.

Alignment with Government Priorities

Reducing health inequalities is identified by the federal government as a priority. In the past five years, Canada has signed federal and international declarations on this issue. Government of Canada strategic policy and planning documents, including the Speech from the Throne, also reflect this as a priority area. Within the Public Health Agency of Canada, Strategic Horizons 2013-2018 identifies the population health approach, including reducing health inequalities, as the model that will guide the Agency's focus for its priorities and strategic directions for the next five years. The current Innovation Strategy priority areas of mental health and healthy weights are also aligned with federal and Agency priorities as outlined in planning and strategic policy documents including the Speech from the Throne and the Agency's Strategic Horizons.

Alignment with Federal Roles and Responsibilities

The Agency's role in health promotion and protection, and disease prevention is outlined in the Department of Health Act and the Public Health Agency of Canada Act. There is a clear federal role to promote the overall health and well-being of Canadians, particularly for public health issues of national concern such as obesity and mental health. The objectives and activities of the Innovation Strategy, which include reducing health inequalities through population health intervention research, are aligned with this federal role. The Innovation Strategy does not appear to duplicate the role of other stakeholders.

CONCLUSIONS - PERFORMANCE

Achievement of Expected Outcomes (Effectiveness)

Through the implementation, delivery, monitoring and evaluation of population health interventions, the Innovation Strategy has achieved its intended immediate outcomes. Population health interventions have been developed and are supported by intersectoral partnerships. A range of knowledge products have been developed, disseminated and accessed at the project level, and while knowledge products have also been developed at the program level, the extent to which they have been disseminated and accessed is less clear.

Progress has also been made by the Innovation Strategy towards most of its intermediate outcomes. At the project level, there is some evidence that knowledge generated through the program is informing policy and program changes; however, knowledge uptake has not been systematically tracked at the project or program level. A more consistent and systematic approach to tracking this information would facilitate the assessment of the program's progress towards this outcome. There are indications that the program is contributing to improved health outcomes, particularly in the Innovation Strategy priority area of mental health, which is further along in program delivery. These outcomes include improved knowledge and skills in children and families, including those in Aboriginal settings. Additional progress towards improved health outcomes is expected as projects complete Phase 2 and move into Phase 3, where full scale up will occur. Currently, there are indications of Phase 2 interventions in both priority areas demonstrating readiness for scale up.

Demonstration of Economy and Efficiency

The program design of the Innovation Strategy, including the phased approach to project funding and the staggered delivery of each priority area, has contributed to program efficiencies. Through the phased approach, only those projects demonstrating promise or effectiveness are funded in subsequent phases, thereby limiting the funding of less effective interventions. The staggered delivery of the priority areas has allowed for lessons learned from Mental Health Promotion projects to be applied to the delivery of Achieving Healthier Weights projects. In addition, the partnerships developed through the delivery of the Innovation Strategy have allowed projects to leverage approximately $5.7 million, and receive approximately $5.6 million of in-kind support since project delivery commenced (equal to 31% of project funding over that time frame).

Additional efficiencies could be gained through increased collaboration and information sharing within the Health Promotion and Chronic Disease Prevention (HPCDP) Branch and the Health Portfolio, specifically the Canadian Institutes of Health Research (CIHR) Institute for Population and Public Health (IPPH). While there have been informal connections made between the Innovation Strategy and these areas, few formal mechanisms currently exist to facilitate joint work planning, or information sharing.

The Innovation Strategy program has been active in collecting and using performance information, however, additional performance measurement is needed to better understand the impact of Innovation Strategy knowledge products, and to support the periodic follow up of interventions post-Innovation Strategy funding.

RECOMMENDATIONS

Recommendation 1:

Identify and action potential opportunities for strategic coordination of efforts and increased collaboration to leverage expertise and maximize efficiencies in the area of population health intervention research and in the two policy priority areas (mental health and healthy weights):

  • Within the HPCDP Branch of the Public Health Agency of Canada; and
  • With the CIHR - IPPH, and other areas of the Health Portfolio as appropriate.

Evaluation evidence (i.e., key informant interviews, document review) consistently revealed that while there have been informal connections made between the Innovation Strategy and relevant program areas, many of these connections have not been formalized. As a result, there may have been missed opportunities to enhance program efficiencies and effectiveness through collaboration related to joint work planning, and/or coordination of activities. The evaluation therefore recommends that, to the extent possible, these relationships be formalized, or that tools and processes to support collaboration are identified to support overall efficiencies and contribute towards advancing progress in population health intervention research and in the two policy priority areas.

Recommendation 2:

Develop a formal strategy to guide the sharing of information and lessons learned through the Innovation Strategy program, related to the policy priority areas (currently mental health and healthy weights) and population health intervention research.

Overall, the evaluation identified that although the Innovation Strategy has resulted in the development of a number of knowledge products, there has been no formal strategy guiding the knowledge dissemination efforts of the program. To support the achievement of program outcomes related to stakeholders accessing and using Innovation Strategy-generated evidence, a formal strategy to guide information sharing is recommended.

Recommendation 3:

Enhance performance measurement activities related to:

  • Tracking of information uptake and use to better measure and understand program impact, particularly in the policy priority areas; and
  • Follow-up of projects post-Innovation Strategy funding to determine if interventions were sustained.

In order to fully measure and understand the effectiveness of the Innovation Strategy, improvements to performance measurement activities are needed. Currently, knowledge product uptake is not systematically tracked at either the project or program level. As a result, it is not clear if or how Innovation Strategy knowledge products are being used. Additionally, to gain perspective on intervention sustainability post-funding, it is recommended that periodic follow up of projects post-Innovation Strategy funding take place.

Management Response and Action Plan
Innovation Strategy
Recommendations Response Action Plan Deliverables Expected Completion Date Accountability Resources

1. Identify and action potential opportunities for strategic coordination of efforts and increased collaboration to leverage expertise and maximize efficiencies in the area of population health intervention research and in the two policy priority areas (Mental Health Promotion and Achieving Healthier Weights):

  1. Within the Health Promotion and Chronic Disease Prevention Branch (HPCDP) of the Public Health Agency of Canada.
  2. With the Canadian Institutes of Health Research - Institute for Population and Public Health (CIHR - IPPH), other areas of the Health Portfolio as appropriate.

Agree

Establish mechanisms with areas in the HPCDP Branch for improved collaboration on the two policy priority areas and population health intervention research, and prioritize options for strategic coordination of activities:

  • Centre for Health Promotion (policy lead on mental health and maternal and child health);
  • Centre for Chronic Disease Prevention (policy lead on healthy weights); and
  • Social Determinants and Science Integration Directorate (health equity lens, population health intervention research).

Mechanisms established for improved coordination and collaboration

January 2016

Assistant Deputy Minister, Health Promotion and Chronic Disease Prevention

Additional human resources are required to complete each of the deliverables for this recommendation, and  will be identified/ reallocated from within the Centre for Health Promotion

Engage CIHR - IPPH* and other areas of the Health Portfolio as appropriate to identify areas for increased coordination and collaboration in the area of population health intervention research and in the two policy priority areas.
*This work will be in alignment with the higher level agreements between PHAC and CIHR more broadly.

Mechanisms identified for improved coordination and collaboration

November 2016

2. Develop a formal strategy to guide the sharing of information and lessons learned through the IS program, related to the policy priority areas (currently Mental Health Promotion and Achieving Healthier Weights) and population health intervention research.

Agree

Develop a plan to share information and lessons learned from the IS program. The plan will include an overview of activities, and an approach to measure uptake and use.

Plan for sharing lessons learned

December 2015

Assistant Deputy Minister, Health Promotion and Chronic Disease Prevention

Existing Resources

3. Enhance performance measurement activities related to:

a) Tracking of information on uptake and use to better measure and understand program impact, particularly in the policy priority areas.

Agree

To better measure and understand program impact:

  • Enhance data collection tools to improve collection of information on uptake and use:
  • at the project level; and

Revised project reporting surveys

June 2015

Assistant Deputy Minister, Health Promotion and Chronic Disease Prevention

Existing Resources

Enhanced use of program knowledge uptake surveys June 2015 Existing Resources

b) Follow-up of projects post-IS funding to determine if interventions were sustained.

  • at the program level.

Program level knowledge product tracking system (tracking dissemination and uptake)

December 2015

Existing Resources

To measure project sustainability post-IS funding:

Develop a plan to measure project sustainability post-IS funding.

Plan for project follow up

March 2016

Existing Resources

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Public Health Agency of Canada's Innovation Strategy for the period of April 2009 to August 2014.

This was a scheduled evaluation as per the Public Health Agency of Canada/Health Canada approved Five-Year Evaluation Plan 2013-2014 to 2017-2018, in fulfillment of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

2.0 Program Description

2.1 Program Context

The Innovation Strategy originated from the former Population Health Fund (PHF), a grant and contribution program that funded voluntary not-for-profit organizations and educational institutions to develop knowledge and program models to address the determinants of health. Established in 1997, the PHF program design was based on accumulating evidence in support of the theory that aspects of the economic, social and physical environments (determinants of health) affect the health status of populations. The goal of the PHF was to increase community capacity for action on or across the determinants of health by funding projects to develop community-based models for applying the population health approach, increase the knowledge base about population health, and increase partnerships and intersectoral collaboration in Canada. The PHF administered transfer payments of approximately $12.1 million per year.

A 2007 review of the Promotion of Population Health Grants and Contributions (including the PHF) identified several weaknesses in the PHF approach including inconsistent priorities across the regions, smaller one-off projects with insufficient time to generate evidence of what worked, limited project funding for evaluation, and little sharing of lessons learned. In addition, there was an increased recognition of the complexity of public health issues and of the knowledge gaps about how to act effectively, providing additional incentive for an updated approach. As a result of the review, the Agency adopted the recommendation that the PHF be transformed into the Innovation Strategy in order to address several of the weaknesses identified with the PHF design.

The Innovation Strategy was established in 2009, based on a population health intervention research approach to build the evidence base to support interventions that focus on innovation and reducing health inequalities in Canada. At the outset of the program, the majority of public health research, population health research in particular, focused on the description of public health problems rather than the identification of potential solutions. Consequently, there was little evidence available for public health policy-makers and practitioners to inform decision-making regarding effective interventionsEndnote 1. Key features of the Innovation Strategy design included: providing a longer term, phased, funding approach; funding a smaller number of larger projects that covered several regions; dedicating a larger percentage of project funding to evaluation; ensuring that projects had sufficient capacity to carry out comprehensive evaluations; and developing a program knowledge exchange strategy to ensure that lessons learned were shared.

Previously delivered by the Strategic Initiatives and Innovations Directorate, accountability for the delivery of the Innovation Strategy was moved within the Health Promotion and Chronic Disease Prevention (HPCDP) Branch to the Centre for Health Promotion (CHP) in February 2013. Over the past five years (2009-2010 to 2013-2014) the program has administered approximately $9.7 million per year in transfer payments.

2.2 Program Profile

The Public Health Agency of Canada's Innovation Strategy is designed to test and deliver innovative population health interventions to reduce health inequalities and address priority complex public health problems and their underlying factors. Employing an intervention research approach, the Innovation Strategy supports the delivery of a set of population health interventions to improve health outcomes, while generating critical knowledge through monitoring and evaluation on (1) how the intervention brought about change and (2) the context in which the intervention worked best and for which populations. Stated more simply, the Innovation Strategy studies and promotes 'what works' and ‘what doesn't work’ in public health. Through intersectoral project partnerships and knowledge exchange activities, the Innovation Strategy disseminates and shares the knowledge generated from the projects in support of program and policy development nationally. Interventions that are determined to be appropriate for scale up (i.e., expanded, replicated, adapted and sustained) to reach a greater number of people and reduce health inequalities are identified.

To support the development, implementation and evaluation of innovative population health interventions, the Innovation Strategy provides funding support to external organizations in a variety of sectors such as health and education. These organizations receive Innovation Strategy funding to study interventions in the current Innovation Strategy priority areas of mental health and healthy weights. Funded projects focus on actions to address underlying environmental, social, demographic and economic conditions related to these priority areas, including a focus on northern and remote communities, children and youth, and those with low incomes. Each project is required to involve local and community partners and is encouraged to collaborate with the research/academic community, the health sector,  non-governmental organizations, all levels of government, social services and, where possible, the private sector. The Innovation Strategy priority areas reflect complex public health problems (obesity and mental illness) and were determined through internal and external consultation processes, availability of existing evidence and alignment with Agency and Branch priorities.

Intervention research in each Innovation Strategy priority area is funded through a “phased” approach. In the first phase (innovation phase), funding is provided for a 12-18 month period to support early development and implementation of population health interventions. In the second phase (learning phase), funding is provided for up to four years towards the full implementation and evaluation of the population health interventions. The intervention research approach used by the Innovation Strategy requires projects to submit an extensive evaluation of the intervention and synthesis of results by the end of Phase 2. It is anticipated that, by the end of the second phase, interventions with the greatest potential for scale up would be identified and Phase 3 would focus on scale up the interventionsEndnote 2. Scale up refers to a deliberate effort to increase the reach and impact of successfully tested population health promotion interventions to benefit more people and foster sustainable policy/program development across diverse populations and communities. The delivery of Innovation Strategy projects has been staggered by priority area with Mental Health Promotion project delivery having started in 2009-2010 and Achieving Healthier Weights having commenced in 2012-2013.

At the outset of each phase of Innovation Strategy funding for both priority areas, calls for proposals are issued by the Innovation Strategy program. A formal review committee is assembled to assess proposals submitted by external organizations for relevance, innovation merit, readiness for full implementation and evaluation, and knowledge development and exchange plans. Committee members are external to the Agency and have expertise in the identified health priority, population health interventions and public policy. When recruiting committee members, the Innovation Strategy program attaches priority to the selection of researchers with population health intervention experience and expertiseEndnote 3. As each phase of funding ends, projects are invited to submit a letter of intent, signifying their desire to continue in the next phase. Those projects demonstrating the most merit or promise, as assessed by the review committee, are invited to submit proposals to participate in the subsequent phase. Table 1 provides an overview of the two priority areas in terms of start dates and number of projects funded per phase. Of note, between 2010-2011 and 2012-2013, the Innovation Strategy also funded five Phase 1 projects related to Managing Obesity across the lifecycle (separate from the projects in the Achieving Healthier Weights priority area); however, these interventions were not funded beyond Phase 1 and are not included within the scope of this evaluation. A summary of Phase 2 funded Innovation Strategy projects is included in Appendix 1.

Table 1: Overview of Innovation Strategy Funding Phases
Priority Area Duration # of Funded Projects Funding
Mental Health Promotion
Phase 1 2009-2010 to 2010-2011
(12-18 months)
15 $4.85M
Phase 2 2010-2011 to 2014-2015
(48 months)
9 $23.6M
Achieving Healthier Weights
Phase 1 2011-2012 to 2012-2013
(12-18 months)
37 $7.42M
Phase 2 2012-2013 to 2016-2017
(48 months)
11 $15.99M
Total $51.86M

Note: In addition to the above, $1.77M in one time funding was provided for Phase 1 of projects related to Managing Obesity across the lifecycle (not included in evaluation scope).

Source: Innovation Strategy program files

During Phase 1, projects typically received between $250,000 and $400,000 total for the implementation of their interventions. During Phase 2, where interventions expand to additional communities, funding increased to $1.5 to $2.5 million total per project.

Governance

Overall accountability for the achievement of Innovation Strategy program outcomes rests with the Centre for Health Promotion. Key functions for the Innovation Strategy include:

  • Ensuring effective delivery of contribution projects, including managing calls for proposals, assessing and recommending proposals to the Minister, monitoring recipient reporting, and overseeing funded projects.
  • Supporting knowledge development and exchange related to population health, intervention research, and priority areas (mental health and healthy weights);
  • Building and maintaining partnerships related to population health, intervention research and priority areas;
  • Managing linkages with regions; and
  • Managing program budgets and expenditures.

Regional Offices also support delivery of the Innovation Strategy program. Key activities or functions for regional staff include providing support for performance measurement, knowledge development and exchange, and regional linkages for the Innovation Strategy in accordance with the Agency Centre-Regional Operations Agreement and Business Management Model. Of note, within the Public Health Agency of Canada, the Centre for Chronic Disease Prevention is the policy lead for healthy weights; and the Centre for Health Promotion is the policy lead for mental health.

2.3 Program Logic Model and Narrative

The long term or ultimate outcome for the Innovation Strategy focuses on national adoption or uptake of population health interventions to contribute to improved health outcomes and reduced health inequalities for Canadians across the life course. The activity areas, outputs, immediate and intermediate outcomes to achieve this final outcome are described below.

Program Activities

The Innovation Strategy program carries out activities in two key areas to support the achievement of intended outcomes: 1) Implement and test innovative population health interventions and 2) Knowledge development and exchange. As part of the former, the program funds, supports and monitors organizations to design, develop, implement, adapt and evaluate community-based population health interventions that address complex public health issues facing children, youth and families. As part of the latter, the program collaborates with stakeholders to develop and disseminate evidence-based knowledge products and lessons learned about population health interventions.

Outputs and Immediate Outcomes

Through the implementation and testing of population health interventions, it is expected that population health interventions will be developed or adapted, and that tools, approaches and models to support their implementation will be identified. As a result, it is expected that promising population health interventions will be implemented and evaluated. Additionally, through these activities, it is expected that funded projects will participate in new and existing partnerships, and that methods and models to promote intersectoral partnerships will be identified. These partnerships are expected to support the delivery of the population health interventions. Knowledge development and exchange activities are expected to lead to the production of knowledge syntheses, research papers, training materials, presentations, webinars, case studies, and summary reports. Once produced, it is expected that stakeholders (health practitioners, researchers, and other policy makers within and outside of the health sector) will access knowledge products and synthesized learnings to advance population health policy and practice.

Intermediate Outcomes

Three intermediate outcomes have been articulated for the Innovation Strategy program. Progress towards these outcomes is expected to be measurable approximately four years following initial funding of projects. At this time, it is expected that population health interventions will contribute to improved protective factors, reduced risk behaviours, and improved health outcomes for individuals, families and communities. Consistent with the timing of the Innovation Strategy phases, it is also expected that population health interventions will demonstrate readiness for scale up. With respect to knowledge exchange, it is expected that stakeholders will use knowledge products, intervention research evidence and synthesized learnings from the Innovation Strategy to advance population health policy and practice.

The connection between the program activity areas and the expected outcomes is depicted in the program logic model (see Appendix 2). The evaluation assessed the degree to which the defined outputs and outcomes have been achieved.

2.4 Program Alignment and Resources

The program is part of the Agency's Program Alignment Architecture (PAA): Program 1.2 Health Promotion and Disease Prevention, Sub-Program 1.2.2 Conditions for Healthy Living.

The program's expenditures for the years 2009-2010 through 2013-2014 are presented below (Table 2). Overall, the program had a budget of $54 million over five years. A summary of the program's planned versus actual spending is reviewed in section 4.5.

Table 2: Financial Data (2009-2010 to 2013-2014)
Year Expenditures ($)
Gs&Cs O&M Salary TOTAL
2009-2010 6,392,877 670,375 235,548 7,298,800
2010-2011 8,532,337 512,797 535,918 9,581,052
2011-2012 14,277,033 721,287 696,904 15,695,224
2012-2013 8,886,651 552,241 656,795 10,095,687
2013-2014 10,188,923 399,759 688,226 11,276,909

Note: The Innovation Strategy managed several one-time Grants and Contributions throughout the five year evaluation period, that are outside of the scope of this evaluation, but for which financial data is included in the above table. Funding of these one-time Grants and Contributions totalled $6,952,572.

Source: Public Health Agency of Canada, Office of the Chief Financial Officer

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from 2009-2010 to 2013-2014, and included Phases 1 and 2 of the Mental Health Promotion and the Achieving Healthier Weights funding of the Innovation Strategy. The following activities were considered out of scope for this evaluation: mental health promotion activities carried out by the Centre for Health Promotion that are not part of the Innovation Strategy and will be reviewed as part of the Evaluation of Mental Health and Mental Illness activities scheduled for 2015-2016, and the promotion of healthy weights activities carried out by the Centre for Chronic Disease Prevention's Healthy Living Fund that were included in the current Evaluation of the Agency's Chronic Disease Prevention Activities. Even though these activities were out of scope, these programs were consulted as part of this evaluation. Also outside the scope of this evaluation are the one-time Grants and Contributions managed through the Innovation Strategy that were not part of the Mental Health Promotion or Achieving Healthier Weights components of the Strategy.

The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 3. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned.

The Policy on Evaluation (2009) guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the policy. A non-experimental design was used based on the evaluation matrix, which outlined the evaluation strategy for these activities.
Data collection activities were carried out between June and November 2014. Data for the evaluation was collected using various methods, including: literature review, document and file review, financial data review, international review and key informant interviews - both internal and external. More specific details on the data collection and analysis methods are provided in Appendix 2. Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. Table 3 outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Table 3: Limitations and Mitigation Strategies
Limitation Impact Mitigation Strategy

Key informant interviews are retrospective in nature

Interviews retrospective in nature, providing recent perspective on past events. Can impact validity of assessing activities or results relating to improvements in the program area.

Triangulation of other lines of evidence to substantiate or provide further information on data received in interviews.

Document review provides corporate knowledge.

Limited performance data for grants and contribution projects as insufficient time has elapsed for outcomes to occur.

Difficulty in measuring impact of the grants and contributions projects at the intermediate outcome level and beyond.

Reported achievement of immediate outcomes, and triangulated other lines of evidence (e.g., key informant interview data) to provide progress towards achieving intermediate outcomes.

4.0 Findings

This section provides a summary of the findings organized under two broad headings:

  • Relevance: the need, priorities, and federal public health role in preventing chronic diseases.
  • Performance: the effectiveness, efficiency and economy of the Agency's activities in this area.

4.1 Relevance: Issue #1 - Continued Need for the Program

The Public Health Agency of Canada's Innovation Strategy contributes to addressing a need for effective population health interventions to mitigate the health inequalities that contribute to complex public health issues such as mental health and obesity.

The World Health Organization defines good health as a state of complete physical, social and mental well-being, and not merely the absence of diseaseEndnote 4. This paradigm suggests that at every stage of life, health is determined by complex interactions between social and economic factors, the physical environment and individual behaviour. These factors are referred to as the determinants of healthEndnote 5. The determinants of health include a wide range of personal, social, economic and environmental factors such as education, employment, income, social status, housing, gender and cultureEndnote 6. It is suggested that differences in health status from the combination and interaction of health determinants result in health inequalities between individuals and among various population groupsEndnote 7. The resulting challenge lies in how to use knowledge of the determinants of health to understand how to improve the health of CanadiansEndnote 8. Population health refers to health outcomes and their distribution in the populationEndnote 9. The population health approach aims to improve the health of the entire population, and to reduce health inequalities among specific population groupsEndnote 10.

Population health interventions are used to address the complex interaction between the social determinants of health. A 2008 Senate Subcommittee report recognised that disease issues are complex, largely due to the diverse social determinants of healthEndnote 11; therefore, a population health approach is needed to create change at the population level. Population health intervention research uses scientific methods to produce knowledge about policy and program interventions that operate within or outside the health sector and have the potential to impact health at the population levelEndnote 12,Endnote 13. While the majority of public health research focuses on the prevention and causes of public health problems, intervention research focuses on solutions to learn more about what works, under what conditions, and howEndnote 14 thereby increasing the understanding about the impact of policies and programs to improve population health and reduce health inequalitiesEndnote 15. Population health intervention research generates relevant, contextually sensitive, credible and timely knowledge that enables decision makers to continually improve programs and policies. An assessment of population health intervention research supported through CIHR concluded that even though interest in evidence-informed decision making related to population health has been growing, there is a lack of population health intervention research being funded in CanadaEndnote 16. In addition, a 2009 paper highlighting the Canadian Cancer Society's Centre for Behavioural Research and Program Evaluation's experiences in advancing the goals of the Population Health Intervention Research Initiative for Canada (PHIRIC) noted that the majority of research to date has been descriptive in natureEndnote 17 and needs to identify effective solutionsEndnote 18.

In Canada, there are questions about which policy, program and/or intervention will improve the health of particular populations. However, evidence-based information is often lacking, and there is limited capacity for conducting the kind of research needed to provide answers relevant to the Canadian contextEndnote 19. As a result, there is little evidence available for public health policy-makers and practitioners to inform decision-making regarding effective interventions. In addition, there is little information available to demonstrate how a promising intervention can be scaled up, sustained, and eventually replicated by others. The Innovation Strategy seeks to address this gap in knowledge by funding population health intervention research to generate knowledge about policy and program interventions that have the potential to impact health at a population levelEndnote 20.

Obesity and mental health are complex health problems that are influenced by a range of biological, social, physical and economic factors. As such effective population health interventions are required to help address these ongoing public health problems.

The following statistics highlight the prevalence and economic burden of mental illness and obesity.

  • Every year, one in ten Canadians will experience a mental illness, and one in three will experience one sometime in their livesEndnote 21. In 2008, the direct cost of mental illness in Canada was estimated to be at least $7.4 billionEndnote 22. As for vulnerable populations, Aboriginal youth commit suicide five to six times more often than non-Aboriginal youthEndnote 23.
  • It is estimated that approximately two out of every three adults in Canada are overweight or obese, and almost one in three children and youth is overweight or obeseEndnote 24, costing the Canadian economy up to $7.1 billion each yearEndnote 25. In 2007-2010, Aboriginal adults had higher obesity rates than non-Aboriginal adults: First Nations (26%), Inuit (26%), Métis (22%) versus non-Aboriginal (16%)Endnote 26. A recent study of socioeconomic-related inequalities in obesity risk among Canadian adults found that obesity is more prevalent among economically disadvantaged womenEndnote 27.

4.2 Relevance: Issue #2 - Alignment with Government Priorities

While there is no specific link between federal priorities and population health intervention research per se, reducing health inequalities through the development of evidence-informed population health interventions aligns with federal priorities.

In October 2011, the World Conference on Social Determinants of Health was held with participants from 125 Member States including Canada. The goal of the conference was to gather support for action on the social determinants of health, with a focus on developing strategies for reducing health inequalitiesEndnote 28,Endnote 29. At the conference, the Rio Political Declaration on Social Determinants of Health was adopted, reaffirming that health inequalities within and between countries are politically, socially and economically unacceptable, as well as unfair and largely avoidableEndnote 30. The Rio declaration underscores the principles set out in the WHO Constitution and the 1986 Ottawa Charter that both promote a focus on health equityEndnote 31,Endnote 32.

In the January 2009 Speech from the Throne, the Government of Canada committed to protect vulnerable populations including the unemployed, lower-income Canadians, seniors and Aboriginal Canadians. In the 2010 Speech from the Throne, the government stated that protecting the health of Canadians is a priority. More recent Budgets and Speeches from the Throne committed to protecting the health of Canadians, in particular the health and well-being of Aboriginal peoples.

Within the Public Health Agency of Canada, Strategic Horizons 2013-2018 identifies the population health approach, including reducing health inequalities, as the model that will guide the Agency's focus for its priorities and strategic directions for the next five years. Strategic Horizons 2013-2018 also identifies knowledge development and exchange, enhanced information sharing, strengthened partnerships and collaboration as key priorities. Knowledge development and exchange is built in as a requirement of Innovation Strategy project funding. Partnerships and collaboration are key components of funded projects. As noted in the Health Promotion and Chronic Disease Prevention Branch strategic and operational plans, there is alignment between Agency priorities and current activities of the Innovation Strategy.

The Innovation Strategy priorities of mental health and healthy weights are also aligned with broader Government of Canada and Public Health Agency of Canada priorities. The Economic Action Plan (2012) and the October 2013 Speech from the Throne highlighted the government's commitment to work with the provinces and territories, private and not-for-profit sectors to encourage young Canadians to be more physically active. The plan also proposed funding for mental health research.

The Government of Canada, along with the provincial and territorial Ministers of Health, reinforced that the prevention of chronic diseases and related conditions such as healthy weights was a priority by collaborating with its provincial and territorial counterparts to develop national agreements. The 2005 Pan-Canadian Healthy Living Strategy was strengthened in 2010, and pointed to new areas of opportunity, including preventing obesity. To this end, two initiatives were endorsed between 2010 and 2011:

  • Creating a Healthier Canada: Making Prevention a Priority - A Declaration on Prevention and Promotion from Canada's Ministers of Health and Health Promotion and the Sport Recreation and Physical Activity Ministers, which was a public statement to work together, and with others, to make the promotion of health and prevention of disease, disability and injury a priority for action; and
  • Curbing Childhood Obesity: A Federal, Provincial and Territorial Framework for Action to Promote Healthy Weights, which focused on reducing prevalence of childhood obesity in Canada and advancing strategies for Federal, Provincial and Territorial collaboration.

Within the Agency, Strategic Horizons 2013-2018 identified healthy weights, obesity, mental health and intervention research as priorities. Intervention research was identified as a key priority to help the Agency to achieve its Strategic Outcome - Protecting Canadians and empowering them to improve their health. The Agency's Corporate Risk Profile also identified obesity and mental health as key priorities. In addition, the Public Health Network Council (of which the Agency's Chief Public Health Officer is the federal co-chair) has listed mental health and healthy weights as priorities in its current strategic priorities documents.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

There is a federal public health role to conduct research, provide leadership and promote overall health. Intervention research as defined by the Innovation Strategy aligns with this role.

The Agency's role in health promotion and protection, and disease prevention is outlined in the Department of Health Act and the Public Health Agency of Canada Act. There is a clear federal role to promote the overall health and well-being of Canadians, particularly for public health issues of national concern such as obesity and mental health. The Agency's more specific roles to conduct research, provide leadership and support knowledge dissemination are presented under various program authorities.

The Innovation Strategy grants and contributions fall under the Terms and Conditions for Promotion of Population Health and are intended to promote the health of Canadians, while reducing health inequalities. The Innovation Strategy focuses on dissemination of knowledge of the results of promising population health interventions, and sharing these practices across the country with the aim of reducing health inequalities in its two priority areas. The Innovation Strategy focuses on the following four interrelated pillars that align with federal role in public health:

  • Development of partnerships and collaboration;
  • Evaluation and performance measurement;
  • Development of tools and resources; and
  • Knowledge development, exchange and uptake.

Further, the Constitution Act, 1867 outlines the provincial and territorial responsibility for delivering health care, but also identifies the federal government's roles and responsibilities such as ensuring access to health care for specific populations (e.g., Aboriginal peoples), health promotion, disease prevention, knowledge sharing, and funding health research. As an agency of the federal government, the Public Health Agency of Canada fulfills these roles to fund research, and promote overall health. Similarly, the objectives of the Innovation Strategy align with the Agency's role in reducing health inequalitiesFootnote i, and promoting the physical, social and mental well-being of CanadiansEndnote 33,Endnote 34, in addition to sharing knowledge of effective population health interventionsEndnote 35,Endnote 36.

While the Public Health Agency of Canada funds intervention research through the Innovation Strategy, it is not the only program within the Agency that is funding the delivery of interventions with a particular focus on vulnerable populations, nor is it the only federal government department/agency supporting population health intervention research. A challenge to identifying those involved in intervention research however lies in the fact that intervention research is defined differently within and outside the Agency. For instance, within the Agency, there are programs that fund the delivery of population health interventions (e.g., Aboriginal Head Start in Urban and Northern Communities, Community Action Program for Children, Canada Prenatal Nutrition Program, Fetal Alcohol Spectrum Disorder Initiative, Integrated Strategy on Healthy Living and Chronic Disease, Community Associated Infections and the Federal Initiative to Address HIV/AIDS)Footnote ii. The extent to which the delivery of these interventions constitutes or includes intervention research however, may be subject to interpretation.

Outside the Agency, other government departments that appear to support the delivery of population health interventions include Health Canada's First Nations and Inuit Health Branch (FNIHB), the Department of Employment and Social Development Canada, and the Canadian Institutes of Health Research (CIHR)Footnote iii. It is not clear if the population health intervention work of Health Canada or Employment and Social Development Canada include an intervention research component. CIHR is the Government of Canada's agency responsible for funding health research. This includes funding intervention research. CIHR has a mandate to create new scientific knowledge and enable its translation into improved health, more effective health services and products, and a strengthened Canadian health care system. CIHR provides leadership and support to health researchers through its 13 institutes. Each institute is made up of a network of researchers brought together for a specific area of focus, such as the Institute of Population and Public Health that supports research into the determinants of health to improve the health of Canadians while promoting health equityEndnote 37,Endnote 38,Endnote 39.

Overall, the Innovation Strategy does not appear to duplicate the work of CIHR (discussed in section 4.5). For example, while the Innovation Strategy funds the development, implementation, evaluation, and scale up of interventions, CIHR's IPPH primarily funds research on population health interventions. The Pathways to Health Equity for Aboriginal Peoples is an initiative that is led by CIHR, in partnership with the Public Health Agency of Canada and Health Canada's FNIHB. This initiative aims to develop the evidence base in the design and implementation of programs and policies that promote health and health equity in the following four priority areas: suicide prevention, tuberculosis, diabetes/obesity and oral healthEndnote 40. Possible overlap with the Innovation Strategy may exist given that this initiative is involved in the development, implementation, evaluation and scale up of interventions; however, this project is solely focused on Aboriginal peoples and funding is only used for research. Unlike the Innovation Strategy, funding of the intervention is explicitly excluded in the case of the Pathways Initiative.

Internationally, Australia, the United Kingdom, the United States and Canada are similar in that they all have government organizations responsible for health research, with a particular focus on addressing/reducing health inequalities and improving health outcomes. Each country also appears to place importance on knowledge development and exchange. Most similar to the Innovation Strategy, the U.S.' National Institute on Minority Health and Health Disparities Community-based Participatory Research Initiative uses a phased approach to support the delivery and research of population health interventions. Both initiatives incorporate the use of community partnerships to improve health outcomes and reduce health inequalities. The programs differ in that the U.S. Initiative provides funding for up to 11 years and funding recipients are generally academic researchers. Comparable programs in Australia and the United Kingdom were not identified as part of this international web review; however, the Australian Centres of Research Excellence and the UK Clinical Research Collaboration have coordinated research partnershipsFootnote iv,Footnote v to bring together all relevant components of the health system (discussed in more detail in Appendix 4).

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the immediate outcomes been achieved?

Immediate outcome #1: Population health interventions are implemented and evaluated

Through the Innovation Strategy, population health interventions aimed at promoting mental health and reducing barriers to healthy weights have been implemented and evaluated as planned. Tools, approaches and methods to support the implementation of interventions have been developed.

As discussed in Section 2.2, the first phase of Innovation Strategy funding is intended to support projects through the early development and implementation of population health interventions. Overall (both priority areas combined), a total of 52 interventions received Phase 1 funding (15 Mental Health Promotion interventions, 37 Achieving Healthier Weights interventions). Of these, a total of 51 population health interventions were implemented. One intervention funded under the Mental Health Promotion priority area was unable to complete reporting for phase 1 and therefore is not included in this report. As per funding requirements, all of the population health interventions involved in Phase 1 were identified as being new and/or adapted interventions.

Interventions funded under the Innovation Strategy priority area of Mental Health Promotion were organized under three clusters: interventions that aimed to address parenting competencies and family cohesion, school-based interventions that aimed to influence risk and protective factors, and interventions that sought to increase community capacity to address child and youth mental health. Interventions funded under the Innovation Strategy priority area of Achieving Healthier Weights were organized under five clusters: interventions that aimed to address food security (access to food), family based initiatives that aimed to support early childhood and youth, interventions that aimed to promote healthy school programs and environments, northern community-based initiatives and interventions that aimed to promote support environments (encouraging healthy lifestyles).

Projects in both priority areas covered rural, remote, urban, and inner-city areas across the country and targeted populations of varying socio-economic status, literacy level, and cultural background.

As part of their reporting requirements for Phase 1, Innovation Strategy-funded projects were required to report on the reach of their interventions (Table 4). The 14 Mental Health Promotion interventions that were implemented during Phase 1 reached approximately 5,000 individuals between 2009-2010 and 2010-2011. Within the Achieving Healthier Weights priority area, the 37 interventions that were implemented in Phase 1 reached approximately 84,000 individuals between 2011-2012 and 2012-2013. In general, these interventions reached a range of program stakeholders including: individuals who face specific risk factors, public health practitioners, policy makers and the public. In both priority areas, the majority of individuals reached (68% for Mental Health Promotion and 70% for Achieving Healthier Weights) were individuals who face specific risk conditions or risk factors. In Phase 2, the Innovation Strategy projects increased the numbers of intervention sites (discussed in more detail in section 4.4.2) and are now reaching close to 550,000 individuals across the country.

Table 4: Stakeholder reach of Phase 1 interventions
Type of Individuals Reached Mental Health Promotion
(# of individuals reached by 14 interventions in Phase 1)
Achieving Healthier Weights
(# of individuals reached by 37 interventions in Phase 1)
Individuals or communities who face specific risk conditions or risk factors 3,358 59,081
Practitioners and/or other service providers 1,098 10,745
Policy-makers 150 1127
General Public 190 12,057
Total 4,911 83,730

Source: Innovation Strategy, Core Indicator Reports

The Innovation Strategy was developed to explore innovation and learning in population health interventions in order to reduce inequalities. As such, as a component of Phase 1, projects were asked to report on innovative tools, approaches, and processes that were used to support the implementation of population health interventions and models. A total of eight tools/approaches were identified and deemed innovative by the program in the Mental Health Promotion priority area, and a total of 23 tools/approaches were identified and deemed innovative in the Achieving Healthier Weights priority areas during Phase 1. Examples of innovative tools/approaches used in the Mental Health Promotion priority area include: the use of multiple strategies to develop intersectoral partnerships, adaptation of interventions (to local context), emphasis on community engagement, and implementation of interventions in multiple settings. Examples of innovative tools/approaches used in the Achieving Healthier Weights priority area include: multi-sectoral and multilevel partnership development, tailoring and targeting of communications, leveraging of existing partnerships, and a community development approach.

As intended, all population health interventions that participated in Phase 1 of the Innovation Strategy were evaluated. On an annual basis, all projects were required to submit a completed “Project Evaluation and Reporting Tool”. This tool is used to monitor progress towards objectives and collect key performance data. In addition, all Phase 1 projects in both priority areas were required to submit final evaluation reports. These reports provided a summary on the implementation of the project and the results achieved.

Immediate outcome #2: Partnerships support delivery of interventions

A range of intersectoral partnerships have been established by all projects that support the delivery of the population health interventions.

The development of intersectoral partnerships was common to the delivery of all Innovation Strategy projects, across each phase and in both priority areas. Indeed, 100% of the funded Innovation Strategy projects to date have engaged in a range of partnerships to support the delivery of their interventions. Looking at each priority area individually, the ratio of partners to projects has increased over time. For Mental Health Promotion projects, 148 partners were reported in Phase 1, and 289 partners were reported in Phase 2. Given that there were 15 projects in Phase 1 and nine in Phase 2, the ratio of partners to projects rose from approximately 10:1 in the first phase to 32:1 in the second phase. Within the Achieving Healthier Weights priority area, 599 partners were reported in Phase 1, and 262 partners were reported in Phase 2. Given that there were 37 projects in Phase 1 and 11 in Phase 2, the ratio of partners to projects rose from approximately 16:1 in the first phase to 24:1 in the second phaseFootnote vi.

Partners have offered various forms of support to the projects including financial and in-kind resources (loaned staff, volunteers) (discussed in section 4.5 on efficiency and economy), support to enhance community engagement, expertise, support to create change in policy and support to disseminate information. In the Mental Health Promotion priority area, a series of interviews (n=31) were conducted with project partners to classify the nature of their relationship with the project implementers. In the Achieving Healthier Weights priority area, project staff were asked to categorize the nature of their relationships (n=262) with their partners. The results for both are shown in Table 5. Of note, despite the various ways in which the data was collected, approximately 20% of partnerships in each priority area are classified as ‘collaborative', characterized by frequent communication, mutual trust, and consensus reached on all decisions. Further, the majority of partnerships were categorized by project staff and partners as being at least ‘cooperative', characterized by formal communication channels, independent decision making and somewhat defined roles.

Table 5: Reported classification of Innovation Strategy project partnerships
Classification of Partnership Mental Health Promotion
(classified by project partners)
(%)
Achieving Healthier Weights
(classified by project staff)
(%)
Networking
Parties are aware of each other, have loosely defined roles, little communication, and decisions are made independently
3% 11%
Cooperative
Parties provide information to each other, have somewhat defined roles, and practice formal communication, while making all decisions independently
10% 40%
Coordination
Parties share information and resources, have defined roles, frequent communication, and some shared decision-making
42% 26%
Coalitions
Parties share ideas, resources, have frequent and prioritized communication, and all members have a vote in decision-making
26% 5%
Collaborations
Members belong to one system, frequent communication is characterized by mutual trust, and consensus is reached on all decisions
19% 18%

Source: Innovation Strategy Core Indicator Report, Partnership Survey Report

As noted by key informants, the complex population health issues being addressed by the Innovation Strategy are influenced by a range of social determinants, many of which extend beyond the domain or influence of public health. For this reason, all Innovation Strategy projects have prioritized and engaged in intersectoral partnerships. According to program documentation, partnerships are intersectoral when the project partner is working with the project implementers to meet objectives and goals and where the partner is either a different type of organization (i.e., private, public, not-for-profit), operates at a different geographic level or scope (i.e., international, national/federal, provincial/territorial) and/or has a different primary area of focus (i.e., health, education, social services, academia/research) than that of the funded project. Consistent across both priority areas, the projects have predominately engaged with the public and not-for-profit sectors, and to a lesser extent with the private sector. With respect to geographic level, projects in both priority areas have worked fairly evenly across the local/municipal, regional, provincial/territorial, and national levels. The most common areas of focus for partners in both priority areas were health, education, social services, and Aboriginal issues. Examples of intersectoral partners include: school boards, provincial ministries, local municipalities, grocery stores, Inuit community governments, local community wellness groups, law enforcement, parent organizations and Aboriginal friendship centres.

Immediate outcome #3: Stakeholders access knowledge products

A range of knowledge products have been developed, disseminated and accessed at the project level, and while knowledge products have also been developed at the program level, the extent to which they have been disseminated and accessed is less clear.

Over the course of both phases in each priority area, Innovation Strategy projects have been active in developing and disseminating knowledge products, informed by learnings through the implementation and delivery of their interventions. A variety of knowledge products have been produced, including manuals/training kits, brochures/pamphlets/posters, web sites, newsletters and position papers/research summaries. These products have been disseminated to project stakeholders through project websites, targeted mail outs, and knowledge activities. Indeed, Innovation Strategy projects participated in a range of knowledge activities including workshops/presentations, activities to influence policy, and community events. To date, Innovation Strategy projects have produced approximately 730 knowledge products and participated in approximately 2,721 knowledge activities (Table 6). Further, in both priority areas, the ratio of knowledge products produced to number of projects has increased from Phase 1 to Phase 2.

Table 6: Summary of knowledge product and knowledge activity outputs
Priority Area Phase 1 Phase 2
Knowledge Products Knowledge Activities Knowledge Products Knowledge Activities
Mental Health Promotion 71 244 334 565
Achieving Healthier Weights 273 1,563 123 506
Total 344 1,807 385 914

Note: Given that each priority area is at a different stage of delivery, the Phase 2 data in Table 5 includes three years of data for Mental Health Promotion projects, but only one for those related to Achieving Healthier Weights.
Source: Innovation Strategy, Core Indicator Reports

Performance measurement data collected by the Innovation Strategy projects included estimates of the degree to which knowledge products and activities were accessed. Estimates were informed by attendance tracked at knowledge events, or through tracking of dissemination efforts including website hits, newsletter mail outs, and knowledge product downloads. In general, project level knowledge products appear to be well accessed by program stakeholders. In the Mental Health Promotion priority area, knowledge products and activities were made accessible to an estimated 22,395 stakeholders in the first phase, approximately 5,294,766Footnote vii stakeholders in the first two years of Phase 2, and to approximately 273,682 stakeholders in the second reporting year for Phase 2. In the Achieving Healthier Weights priority area, knowledge products and activities were made accessible to an estimated 557,070 stakeholders in Phase 1, and to approximately 107,779 stakeholders in Phase 2.

Innovation Strategy program staff have also developed and disseminated knowledge products. At the program level, a total of 50 knowledge products have been developed over the last five years. These include summaries of the Innovation Strategy program to inform stakeholders (5), project summaries including details on findings (5), compilations of project performance information (summative across program) (15), research summaries (evidence reviews) to support project implementation and enhance knowledge around Innovation Strategy priorities (25), and knowledge events (15). To date, the program has produced a limited number of rolled up analyses (informed by Innovation Strategy projects) related to implementation science in general. However, key informants noted that such products would be more informed if produced in Phase 3.

In general, program level knowledge products have been shared with all projects, and internal program stakeholders. However, the extent to which they have been disseminated and accessed is relatively unknown as there has been inconsistent tracking of knowledge product dissemination at the program level. Key informants (including program staff and management) noted that program-level knowledge products have not been accessed as much as they would like. The program has recently begun to develop a knowledge dissemination and uptake plan that will introduce more consistent tracking of knowledge exchange activities.

4.4.2 To what extent have the intermediate outcomes been achieved?

Intermediate outcome #1: Stakeholders use knowledge products

While there are examples of knowledge products being used by stakeholders, particularly at the project level, the precise extent to which use is occurring, and how, has generally not been well documented at the project or program level.

The extent to which Innovation Strategy knowledge products (project and program level) are used, and how, is not well understood because the overall uptake of Innovation Strategy knowledge products has not been systematically or consistently tracked across the Innovation Strategy program. For example, details of how the projects tracked this information or for what purposes stakeholders used each of the knowledge products they produced was not required as part of the reporting. In each priority area, project staff reported on whether knowledge products had been used. In Phase 1of the Mental Health Promotion priority area, 21% of projects reported that stakeholders had used their knowledge products. In Phase 2 (2012-2013 reporting year), this number rose to 78%. In the priority area of Achieving Healthier Weights, 73% of projects reported that their knowledge products had been used by stakeholders in Phase 1, and in the first year of Phase 2, this percentage has remained at 73%.

Despite the limitations on how this information was tracked, examples of knowledge product use were available in both priority areas (Table 7). In some cases, the knowledge products were discussed or cited in stakeholder practice and policy forums. Examples also existed of knowledge products having been used by stakeholders to inform practice and policy. In some cases, Innovation Strategy knowledge products have even led to the implementation or adaptation of an intervention, policy or program outside of Innovation Strategy funding.

Table 7: Examples of Knowledge Product Use
Mental Health Promotion Achieving Healthier Weights
Project Example of knowledge uptake Project Example of knowledge uptake

Child and Youth Mental Health Intervention, Research and Community Advocacy project

The Qikiqtani Inuit Association is using the foster care report and the Inunnguiniq materials generated through the project to develop strategies for programming related to child and family health in the Qikiqtaaluk Region.

Disseminating Community Food Centres in Ontario

The Good Food Bank at The Table Community Food Centre in Perth, Ontario developed a “core foods policy”. This policy has been communicated to community organizations, businesses and schools who want to donate food. A local grocer has used the core food list to educate their clients about healthy food choices and about The Table's programs. Healthy food donations have risen by 25% from 2012 to 2013.

The Fourth R project - Promoting Youth Well-being through Healthy Relationships

Lessons learned from the Fourth R program have been used to inform the development of the safe-schools/anti-bullying policies and legislation in the Northwest Territories.
The NT Department of Education has also included Fourth R programming
as part of their territory-wide strategy.

Food Matters Manitoba project

Developed a knowledge product entitled “Community Tables Curriculum”. This product was used to work with community organizations serving food to those in need to formulate food policies.  As a result, organizations have begun to take action and make key food policy changes (e.g., eliminating the use of sugary juices during their snack program).

Source: Innovation Strategy program files

Similarly, at the program level, the use of knowledge products has not been systematically tracked. However, the program has recently conducted a series of surveys with individuals (n=36) who have requested knowledge products from the program in 2013-2014. To date, these surveys reveal that approximately 86% of these individuals have used the Innovation Strategy knowledge products they requested. Of these, approximately one third have used the Innovation Strategy knowledge product to support decision-making, while approximately one fifth have used knowledge to inform their work or practice. The program is currently developing a knowledge dissemination and uptake plan to support a more systematic and consistent approach to tracking this information at both the project and program level.

Intermediate outcome #2: Improved health outcomes

There are early indications that the population health interventions funded by the Innovation Strategy are contributing to improved health outcomes. These include improved knowledge and skills in children and families in diverse populations including Aboriginal settings.

Measureable progress towards this outcome is not expected until four to six years after projects begin Phase 1. While projects in the Mental Health Promotion priority area have been delivered for this duration of time, projects in the Achieving Healthier Weights priority area have not. As a result, most evidence in this section relates to projects from the priority area of Mental Health Promotion.

Each year, as part of their annual reporting requirements, Innovation Strategy projects report on progress made towards this outcome. More specifically, projects report on whether their activities or interventions have contributed to: improved health practices, skills, and/or outcomes, improved protective factors, and/or reduced risk factors in their target populations. In general, changes in target populations were measured by project staff using a mix of methods, including: pre-post surveys, retrospective surveys, focus groups, and observational data.

In the Mental Health Promotion priority area, progress towards this outcome has been made over time. In Phase 1 reporting, data was not available on this outcome as many projects had not yet even attempted to measure it. In the first year of Phase 2 reporting (2012-2013), 44% of projects reported that they had contributed to improved health practices, skills and/or outcomes, while the remainder noted that it was still too early to make an assessment. In the most recent year of Phase 2 reporting (2013-2014), 78% of projects reported that they had contributed to positive changes in health outcomes. For projects in the Achieving Healthier Weights priority area, similar trends were observed. Data was not available in Phase 1, and in the first year of reporting for Phase 2, 36% of projects reported that they had contributed to improved health outcomes.

Overall, changes in health outcomes, protective factors and risk factors have been measured at both the individual, family and community levels. In the Mental Health Promotion priority area, changes at the individual (child and youth) level have related to factors such as resilience, self-esteem and self-image, coping and social skills, and communication and conflict or problem-solving skills. In Aboriginal settings, emphasis was placed on factors such as identity, and sense of connection with family, community and culture. Outcomes for families have included improved parental skills and positive family interactions and cultural connections. In the school environment, observations suggested changes in teachers' approach and a culture shift in school environment. Specific examples of improved health outcomes in both priority areas are provided in Table 8.

Table 8: Examples of Improved Health Outcomes
Mental Health Promotion

Centre for Addiction and Mental Health - The Fourth R Project, Promoting Youth Well‐being through Healthy Relationships

The Fourth R is a comprehensive school‐based mental health program for younger adolescents, as well as parents, teachers and teacher candidates. In Phase 2, the Fourth R program demonstrated: positive effects on knowledge, awareness and coping in grade 7 and 8 students participating in the program; positive impacts on relationships, confidence and school success among aboriginal youth; and decreasing violent delinquency in vulnerable youth (relative to peers in control schools) at the two year follow-up. In addition, the program benefited teacher's candidates by increasing knowledge and self-efficacy and attitudes relative to a comparison group.

Achieving Healthier Weights

Community Food Centres Canada project - Building Health and Equity through Food Programs in Low-income Communities

Community Food Centres Canada is testing and replicating a model to address food security tailored to meet local needs by improving access to healthy food (e.g., drop-in meals), improving food skills (e.g., community gardens and cooking groups, drop-in meals), and supporting education and engagement in low-income communities(e.g., peer-support programs). In their first year of Phase 2 funding the project has already reported higher rates of positive healthy eating behaviours and physical and mental health outcomes from Community Food Centres programming participation. Of the community members surveyed across all access, skills, education and engagement programs: 80% report making healthy changes to their diet, 69% report eating more fruits and vegetables, 80% report improvement in their mental health, and 55% said their physical health has improved.

Source: Innovation Strategy program files

Intermediate outcome #3: Readiness for scale up

There are early indications of interventions demonstrating readiness for scale up, which is Phase 3 of the Innovation Strategy.

As noted in section 2.2, the third phase of the Innovation Strategy will support the scale up of effective population health interventions. Phase 3 for projects in the Mental Health Promotion priority area is scheduled to begin in 2015-2016, and for projects in the Achieving Healthier Weights priority area, not until 2017-2018. In both priority areas, there are already early indications of interventions demonstrating readiness for scale up.

A recently produced Innovation Strategy program knowledge product entitled “Understanding Scale Up”Endnote 41 presents key factors for effective scale up of interventions, informed through the delivery of Phase 1 and 2 of the Innovation Strategy. Key factors for scale up include the following:

  • Overall Readiness: Is the intervention ready for scale up?
  • Capacity: Is there established organizational infrastructure, with the resources required for scale up? What is the best way to achieve scale up?
  • Partnership Development: Are there vested partners with sustainable networks who are engaged in the project?
  • System Readiness: What are the institutional impediments to scale up? What are the type and quality of resources required for scale up?
  • Community Context: Are the sites ready with adequate resources, knowledge, development and exchange capacity and evaluation? Is the cultural context within the community supportive of the intervention?
  • Evaluation and knowledge, development and exchange: How will the scale up process be evaluated? What are the knowledge, development and exchange mechanisms needed to share results?
  • Cost Factors: What are the costs associated with the interventions compared to the costs of no action? How will scale up be sustained?

To help inform which Innovation Strategy projects will move forward to Phase 3, the Innovation Strategy program will assess each project against these criteria. While the evaluation was not able to report on all of these key factors, evidence was available for some of them. In terms of overall readiness, feedback from a range of key informants (program management and staff, Agency staff) indicated that there are interventions in both priority areas demonstrating readiness for scale up. With respect to capacity, particularly organizational infrastructure, it's clear that projects have moved from implementation and design to intervention delivery as they have expanded to additional locations, helping to prepare them for scale up. There is at least one Innovation Strategy intervention being delivered in all provinces/territories in a range of communities (up to 47 communities for one intervention). There is at least one Innovation Strategy project being delivered in all provinces/territories in a range of communities (up to 47 communities for one project). In Phase 2, nine Mental Health Promotion projects are currently delivered in 240 communities across Canada. Similarly, the eleven Phase 2 Achieving Healthier Weights projects are currently delivered in 100 communities across Canada.

In terms of partnership development, the success of the projects in this outcome area has already been discussed in section 4.4.1. However, it is also noteworthy that in both priority areas, projects are demonstrating the formation of committed (sustainable) partnerships. Approximately 60% of the project partnerships in the Mental Health Promotion priority area have been sustained for over three years, and while projects in the Achieving Healthier Weights priority area have not been established as long, 33% of these partnerships have been sustained for at least three years.

With respect to system readiness, key informants noted that P/T bodies play an important role related to project scale up and sustainability. Project level partnership activities are currently self-directed; however, the program encourages partnerships with P/T stakeholders. To date, 65% of projects (overall) have partnered with P/T bodies. During data collection with P/T representatives (n = 2), the notion of P/T engagement to facilitate system readiness was reinforced. It was suggested by these key informants that P/T bodies should be engaged early in project implementation to further facilitate project delivery and scale up. Failure to engage P/T bodies at project outset was identified as a potential impediment to scale up.

An example of an Innovation Strategy intervention demonstrating readiness for scale up is the Mental Health Promotion project entitled “Socially-Emotionally Aware Kids” (SEAK), led by the Canadian Mental Health Association - Nova Scotia. The SEAK project teaches elementary school children aged 5-12 the Promoting Alternative Thinking Strategies (PATHS) program: How to deal with emotions, pro-social behaviours, how to succeed in school, ways to prevent bullying, what to do when being the target of physical, social, emotional, or psychological harm, how to recognize suicide warning signs, how to reach out for immediate help and professional treatment. Table 9 provides an overview of ways in which the SEAK project has demonstrated readiness for scale up, based on the key factors discussed above.

Table 9: Assessment of Innovation Strategy-funded SEAK project's readiness for scale up
Key Factor for Scale up Project StatusTable 9 - Footnote *
*as assessed by Innovation Strategy program staff
  • Implementation Readiness
  • Intervention Evaluation

There is evidence of intervention effectiveness: Children have improved their social-emotional skills and are better able to identify their feelings, manage their emotions, build positive relationships and perform better in school; teachers and parents demonstrate better understanding of children's social-emotional development and their teaching and parenting skills are enhanced; and schools experience an overall reduction in classroom/school environment aggressive behaviour (bullying), improved impulse control and self-awareness, collaborative problem solving, development of emotional language literacy, and academic performance.

  • Implementation Readiness

Demonstrated reach and uptake of the intervention: The project has reached thousands of children annually across three provinces.

  • Partnership Development
  • Cost Factors
  • System Readiness

Strong partnership support: SEAK has developed strong partnerships throughout Phase 2 and has support from the four Atlantic Provinces, including on the collaborative development of a Vision document that has been approved at the ministerial level, and by the departments of education within the 4 provinces.

  • System Readiness
  • Organizational Capacity

Advisory Committee on Scaling Up Social and Emotional Learning (SEL) in Atlantic Canada: This committee involves key champions from across all four Atlantic provinces. This committee has resulted in the four Atlantic Provinces agreeing to collaborate to scale up social and emotional learning (SEL) in Atlantic Canada.

  • System Readiness
  • Cost Factors

Integration into existing systems: The project is building on work currently underway such as the current provincial and federal investment in Comprehensive School Health, including its tools and approaches, and is also establishing a relationship with the Pan‐Canadian Joint Consortium for School Health.

  • Organizational Capacity
  • Community Context

Vertical and Horizontal Scale up has already begun: Vertical scale up of activities has already begun as demonstrated through the securing of government stakeholders (multi-sector). Phase 2 school sites have independently begun horizontal scale up activities by adding more PATHS schools throughout their school communities. One Nova Scotia site has begun implementing a plan to bring PATHS to all elementary schools within its school board.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

Innovation Strategy projects have successfully leveraged funding to facilitate more economical delivery of the program. Efficiencies could be gained through improved collaboration and knowledge exchange with other areas of the HPCDP Branch and the Health Portfolio, specifically CIHR-IPPH. In general, Innovation Strategy activities are well managed and work has begun to improve performance measurement related to tracking the impact of knowledge products.

The Treasury Board Policy on Evaluation (2009) and the guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.

The evaluation provided observations on economy and efficiency based on findings from the document review, key informant interviews and available relevant financial data. In addition, the findings below provide observations on the adequacy and use of performance measurement information to support economical and efficient program delivery and evaluation.

Observations on Economy

In general, over the past five years, the Innovation Strategy has spent 98% of their planned resources (not including leveraged resources). As illustrated below (Table 10), there were some variances between planned and actual spending during the period evaluated. The largest variance was a surplus of 13.4% that occurred in 2009-2010, attributed largely to delays in program start up and implementation, while the smallest variance was a deficit of 1.9% that occurred in 2013-2014.

Table 10: Financial Data (2009-2010 to 2013-2014)
Year Planned Spending ($) Expenditures ($) Variance ($) % planned budget spent
Gs&Cs O&M Salary TOTAL Gs&Cs O&M Salary TOTAL
2009-2010 6,643,593 1,004,981 781,000 8,429,574 6,392,877 670,375 235,548 7,298,800 1,130,774 86.6%
2010-2011 9,030,574 721,846 660,000 10,412,420 8,532,337 512,797 535,918 9,581,052 831,367 92%
2011-2012 14,303,908 559,425 485,536 15,348,869 14,277,033 721,287 696,904 15,695,224 -346,355 102.3%
2012-2013 8,886,651 269,000 543,111 9,698,762 8,886,651 552,241 656,795 10,095,687 -396,925 104.1%
2013-2014 10,206,569 416,100 440,000 11,062,669 10,188,923 399,759 688,226 11,276,909 -214,240 102%

Note: The Innovation Strategy managed several one-time Grants and Contributions throughout the five year evaluation period, that are outside of the scope of this evaluation, but for which financial data is included in the above table. Funding of these one-time Grants and Contributions totalled $6,952,572.

Source: Public Health Agency of Canada, Office of the Chief Financial Officer

As previously discussed under the outcome related to partnerships supporting program delivery (4.4.1), Innovation Strategy projects have leveraged funds from external sources and obtained in-kind resources. Overall, Innovation Strategy projects have leveraged approximately $5.7 million, and received approximately $5.6 million of in-kind support (Table 11). Combined, this represents approximately 31% of total IS funding to date. Leveraged funds have most often been received from other federal government departments, P/T governments, regional health authorities, not-for-profit organizations and universities.

Of note, not all Innovation Strategy projects have been successful in leveraging additional funds. In the Mental Health Promotion priority area, three of the fifteen projects were responsible for all leveraged funds in Phase 1, while four of the nine funded projects successfully leveraged funds in Phase 2. In the Achieving Healthier Weights priority area, funds were leveraged by 22 of the 37 projects in Phase 1 and by five of the eleven projects in Phase 2. A greater percentage of projects in both priority areas were involved in obtaining in-kind resources. Approximately 80% of all (both phases) Mental Health Promotion projects and 90% of all Achieving Healthier Weights projects contributed to obtaining the in-kind resources. Factors that contributed to projects' ability to leverage funds or in-kind resources included project design, including nature and number of stakeholders, as well as geographic characteristics of project sites, including specific location, community size and infrastructure.

Table 11: Summary of Innovation Strategy leveraged and in-kind resources
Resource
Type
Mental Health Promotion Achieving Healthier Weights Total
Phase 1
(15 projects)
Phase 2
(9 projects)
Phase 1
(37 projects)
Phase 2
(11 projects)
*includes summation of three reporting years in Phase 2
Leveraged funds ($) $343,000 $1,001,500Table 11 - Footnote * $3,200,000 $1,177,400 $5,721,900
In-kind resources ($) $326,000 $1,834,874Table 11 - Footnote * $2,600,000 $916,500 $5,677,374

Source: Innovation Strategy program files

In addition, Innovation Strategy projects have also benefitted from loaned staff and volunteers (Table 12). Overall, Innovation Strategy projects have benefitted from approximately 110,000 hours of loaned staff time and at least 28,700 hours of volunteer time. In both priority areas, the majority of projects have been successful in obtaining these additional supports. In the Mental Health Promotion priority area, approximately 75% of projects in both phases have been successful in obtaining these resources (both loaned staff and volunteers). In the Achieving Healthier Weights priority area, approximately 90% of projects in each phase have benefitted from loaned staff, while approximately 70% of projects in each phase have obtained volunteer support. Common contributions made by volunteers included: marketing and writing press articles, peer mentorship, event organizing, teaching workshops, steering committee participation, program implementation, the distribution of evaluation materials and involvement in data collection for evaluation reporting.

Table 12: Summary of loaned staff and volunteer hours for Innovation Strategy projects
Resource
Type
Mental Health Promotion Achieving Healthier Weights Total
Phase 1
(15 projects)
Phase 2
(9 projects)
Phase 1
(37 projects)
Phase 2
(11 projects)
*includes summation of two reporting years in Phase 2
Loaned Staff (hours) 6,100 42,560Table 12 - Footnote * 40,000 20,682 109,342
Volunteer (hours) 3,238 8,291Table 12 - Footnote * 17,215 Not reported 28,744

Source: Innovation Strategy program files

Observations on Efficiency

The phased approach to the delivery of Innovation Strategy projects, combined with the on-going monitoring and evaluation of projects, supports program efficiency. With this approach, only those projects that demonstrate effectiveness or promise through intervention evaluations are moved to subsequent phases of funding. This helps prevent the funding of less effective interventions, while fostering those with the most potential. The staggered delivery of the individual priority areas has also resulted in program efficiencies. This has allowed the program to collect and analyze lessons learned in the delivery of the Mental Health Promotion projects and apply them to the delivery of projects in the Achieving Healthier Weights priority area.

Factors contributing to program inefficiencies were also observed. For instance, key informants (program staff, Agency staff) suggested that, since other programs within the HPCDP Branch and the Health Portfolio (CIHR) work in topic areas similar to the Innovation Strategy priority areas, there are opportunities for increased collaboration to promote complementarity and coordination of efforts and learn from one another. For example, outside of the Innovation Strategy, the Centre for Health Promotion also delivers other mental health promotion activities, while the Centre for Chronic Disease Prevention (Healthy Living) is the policy lead for the Public Health Agency of Canada in the area of Healthy Weights. Additionally, within each Centre, there are several programs (i.e., Community Action Program for Children (CAPC), Canada Prenatal Nutrition Program (CPNP), Aboriginal Head Start in Urban and Northern Communities (AHSUNC), and the Integrated Strategy on Healthy Living and Chronic Disease) that may address complementary topic areas or include intervention research-based elements (discussed in section 4.3), for which Innovation Strategy learnings may be applicable or useful.

According to key informants, while there have been informal connections made between these groups and the Innovation Strategy program with respect to knowledge exchange, no formal mechanisms (i.e., joint work planning) currently exist between these groups to share best practices or coordinate activities. As a result, many of the knowledge products developed by the Innovation Strategy do not appear to be reaching Agency staff that may benefit from them. Conversely, best practices and/or knowledge products developed by other programs that could inform or be used by Innovation Strategy projects may not be reaching the program. For example, there may be an opportunity to improve the coordination of Innovation Strategy work in the Achieving Healthier Weights policy area with the Centre for Chronic Disease Prevention (CCDP), which is the policy lead within the Public Health Agency of Canada for promotion of Healthy Weights. This might involve improved knowledge sharing between the two Centres (CHP and CCDP) with respect to policy, priorities and funding opportunities related to healthy weights. Formalizing this connection would support improved linkages between the Innovation Strategy and CCDP. Another example might include increased information sharing with other CHP programs, including CAPC, CPNP and AHSUNC. Given that Innovation Strategy interventions focus on children, youth and families, there may be opportunities to exchange lessons learned with CAPC, CPNP and AHSUNC programs, which are focused on healthy childhood development. The Innovation Strategy has not met with CAPC, CPNP or AHSUNC staff on a consistent basis and so there is opportunity for more deliberate processes and defined ways to share these lessons learned. This could include the rich knowledge of the population and communities in which CAPC, CPNP and AHSUNC serve and the emerging outcomes and evidence from Innovation Strategy funded projects on knowledge exchange and partnerships.

Outside the Public Health Agency of Canada, other government departments (Health Canada and Employment and Social Development Canada) are also involved in delivering population health interventions (discussed in section 4.3). The CIHR (Institute for Population and Public Health) is also involved in the funding of population health intervention research. Other population health intervention research-related activities are currently implemented by CIHR (e.g., Pathways to Health Equity, described in section 4.3) in partnership with the Public Health Agency of Canada.
The Innovation Strategy has built a relationship with CIHR, and the two organizations participate in the Population Health Intervention Research Initiative for Canada. This initiative is supported by the Institute of Population and Public Health, and aims to advance population health intervention research within Canada, increase capacity to conduct population health intervention research and enhance the global knowledge baseEndnote 42. According to key informants however, additional opportunities exist for increased, or more formal, collaboration. For example, the knowledge exchange mechanisms for the Pathways to Health Equity Initiative are outlined in a memorandum of understanding signed by CIHR, the Agency and FNIHB and are operationalized through a formal governance structure within Pathways to Health Equity Initiative, the Health Portfolio Coordinating Committee. Similarly, the Applied Public Health Chairs program has involved joint priority setting between the Agency and CIHR's Institute for Population and Public Health. Formal mechanisms such as those outlined in the above examples (i.e., memorandum of understanding and joint priority setting) might represent potential means by which the Innovation Strategy and CIHR relationship could be more formalized.

Observations on the Adequacy and Use of Performance Measurement Data

The program has been successful in monitoring performance, using researched and standardized tools. Performance data is systematically analyzed and used to inform program delivery (e.g., lessons learned report). However, to gain a better understanding of the impact of Innovation Strategy knowledge products, additional performance measurement is needed. The program has recently begun efforts to this end, including a dissemination plan and a knowledge uptake plan (for collecting information on knowledge product use).

While projects have not yet begun Phase 3, it will also be important for the program to consider long term performance measurement. For example, it may be of benefit to carry out periodic follow up of interventions post Innovation Strategy-funding in order to monitor project sustainability.

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

Complex population health issues such as mental illness and obesity remain a public health concern. To effectively respond to them, evidence-based population health interventions which act upon the social determinants of health are required. Currently, there is limited evidence in Canada related to effective population health interventions. As a result, there is a continued need for population health intervention research to generate knowledge about policy and program interventions that have the potential to act at a population level.

5.1.2 Alignment with Government Priorities

Reducing health inequalities is identified by the federal government as a priority. In the past five years, Canada has signed federal and international declarations on this issue. Government of Canada strategic policy and planning documents, including the Speech from the Throne, also reflect this as a priority area. Within the Public Health Agency of Canada, Strategic Horizons 2013-2018 identifies the population health approach, including reducing health inequalities, as the model that will guide the Agency's focus on its priorities and strategic directions for the next five years. The current Innovation Strategy priority areas and activities of mental health and healthy weights are also aligned with federal and Agency priorities as outlined in planning and strategic policy documents including the Speech from the Throne and the Agency's Strategic Horizons.

5.1.3 Alignment with Federal Roles and Responsibilities

The Agency's role in health promotion and protection, and disease prevention is outlined in the Department of Health Act and the Public Health Agency of Canada Act. There is a clear federal role to promote the overall health and well-being of Canadians, particularly for public health issues of national concern such as obesity and mental health. The objectives and activities of the Innovation Strategy, which include reducing health inequalities in the areas of mental health promotion and achieving healthy weights through population health intervention research, are aligned with this federal role. The Innovation Strategy does not appear to duplicate the role of other stakeholders.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

Through the implementation, delivery, monitoring and evaluation of population health interventions, the Innovation Strategy has generally achieved its intended immediate outcomes. Population health interventions have been developed and are supported by intersectoral partnerships. A range of knowledge products have been developed, disseminated and accessed at the project level, and while knowledge products have also been developed at the program level, the extent to which they have been disseminated and accessed is less clear.

Progress has also been made by the Innovation Strategy towards most of its intermediate outcomes. At the project level, there is some evidence that knowledge generated through the program is informing policy and program changes; however, knowledge uptake has not been systematically tracked at the project or program level. A more consistent and systematic approach to tracking this information would facilitate the assessment of the program's progress towards this outcome. There are indications that the program is contributing to improved health outcomes, particularly in the Innovation Strategy priority area of Mental Health Promotion, which is further along in program delivery. These outcomes include improved knowledge and skills in children and families, including those in Aboriginal settings. Additional progress towards improved health outcomes is expected as projects complete Phase 2 and move into Phase 3, where full scale up will occur. Currently, there are indications of Phase 2 interventions in both priority areas demonstrating readiness for scale up.

5.2.2 Demonstration of Economy and Efficiency

The program design of the Innovation Strategy, including the phased approach to project funding and the staggered delivery of each priority area, has contributed to program efficiencies. Through the phased approach, only those projects demonstrating promise or effectiveness are funded in subsequent phases, thereby limiting the funding of ineffective interventions. The staggered delivery of the priority areas has allowed for lessons learned from Mental Health Promotion projects to be applied to the delivery of Achieving Healthier Weights projects.

Additional efficiencies could be gained through increased collaboration and information sharing within the HPCDP Branch and the Health Portfolio, specifically the CIHR-IPPH. While there have been informal connections made between the Innovation Strategy and these areas, few formal mechanisms currently exist to facilitate joint work planning, or information sharing.

The Innovation Strategy program has been active in collecting and using performance information, however, additional performance measurement is needed to better understand the impact of Innovation Strategy knowledge products, and to support the periodic follow up of interventions post-Innovation Strategy funding.

6.0 Recommendations

The findings from this evaluation have resulted in the following three recommendations.

1. Identify and action potential opportunities for strategic coordination of efforts and increased collaboration to leverage expertise and maximize efficiencies in the area of population health intervention research and in the two policy priority areas (mental health and healthy weights):

  • Within the HPCDP Branch of the Public Health Agency of Canada; and
  • With the CIHR - IPPH, other areas of the Health Portfolio as appropriate.

Evaluation evidence (i.e., key informant interviews, document review) consistently revealed that while there have been informal connections made between the Innovation Strategy and relevant program areas, many of these connections have not been formalized. As a result, there may have been missed opportunities to enhance program efficiencies and effectiveness through collaboration related to joint work planning, and/or coordination of activities. The evaluation therefore recommends that, to the extent possible, these relationships be formalized, or that tools and processes to support collaboration are identified to support overall efficiencies and contribute towards advancing progress in population health intervention research and in the two policy priority areas.

2. Develop a formal strategy to guide the sharing of information and lessons learned through the Innovation Strategy program, related to the policy priority areas (currently mental health and healthy weights) and population health intervention research.

Overall, the evaluation identified that although the Innovation Strategy has resulted in the development of a number of knowledge products, there has been no formal strategy guiding the knowledge dissemination efforts of the program. To support the achievement of program outcomes related to stakeholders accessing and using Innovation Strategy-generated evidence, a formal strategy to guide information sharing is recommended.

3. Enhance performance measurement activities related to:

  • Tracking of information uptake and use to better measure and understand program impact, particularly in the policy priority areas; and
  • Follow-up of projects post-Innovation Strategy funding to determine if interventions were sustained.

In order to fully measure and understand the effectiveness of the Innovation Strategy, improvements to performance measurement activities are needed. Currently, knowledge product uptake is not systematically tracked at either the project or program level. As a result, it is not clear if or how Innovation Strategy knowledge products are being used. Additionally, to gain perspective on intervention sustainability post-funding, it is recommended that periodic follow up of projects post-Innovation Strategy funding take place.

Appendix 1 - Innovation Strategy Projects

Mental Health Promotion Phase II - Equipping Canadians - Mental Health Throughout Life
Project Title
(Funding Recipient)
Funding Detailed Description Key Objectives Target Populations Project Location(s)

Child and Youth Mental Health Intervention, Research and Community Advocacy Project in Nunavut (Qaujigiartiit Health Research Centre)

$2.4M

The purpose of this project is to develop, implement, and evaluate, child and youth mental health and wellness interventions in Nunavut. These interventions focus on northern and community-based ways of understanding and knowing about healthy children and youth.

  • Build capacity and knowledge for addressing child and youth mental health and wellness issues in Nunavut
  • Develop, implement and evaluate community-based mental health and wellness interventions for children and youth in Nunavut
  • To inform the development of short-term and long-term action plans for implementing culturally relevant child and youth mental health and wellness programs in Nunavut based on the findings from the research projects and evaluations of the interventions
  • Interventions
    Created by Nunavummiut for
    Nunavummiut
    (Nunavut community)
  • Children and youth in intervention communities
  • Nunavut Frontline mental health workers; social workers; youth workers; youth advocates; parenting support workers; human service workers; program developers and service providers
  • Policy and program developers
  • General public - parents, youth, elders

Nunavut

Connecting the Dots: A Community-led Mental Health Promotion Project
(Canadian Mental Health Association BC Division)

$2.6M

The purpose of this project is to promote the mental health of urban Aboriginal youth and families in British Columbia.

The project is led provincially by the Canadian Mental Health Association (CMHA) BC Division and the BC Association of Aboriginal Friendship Centres (BCAAFC) and implemented locally through Friendship Centres and CMHA branches in three urban Aboriginal communities: Kelowna, Port Alberni and Quesnel.

  • Mobilizing communities to address risk and protective factors influencing mental health.
  • First Nations youth, Elders and families living in 3 communities in BC
  • Service Providers that work with Aboriginal youth and families in the identified communities
  • Policy makers - Ministry of Child and Family Development, Ministry of Social Development, regional city council members, Health Authorities
  • Native and non-native individuals living and working in the communities who are interested in mental health and well-being of 1st Nations youth and families

British Columbia
(MN, YK)

Creating Responsive Communities to Promote Healthy Relationships in Young Children
(University of Victoria)

$2.6M

The purpose of the WITS (Walk away, Ignore, Talk it out, Seek help) Program is to bring together schools, families and communities to help elementary school children deal with bullying and peer victimization. 

  • Demonstrating the effectiveness of the WITS Programs in preventing peer victimization and increasing pro-social behaviours among elementary school children in a large scale study and to support the wide-spread implementation of the WITS Programs in elementary and middle schools across Canada.
  • elementary school children (kindergarten - grade 6)
  • school officials (i.e., school board representatives, administrators, teachers, teaching assistants and counsellors) and community leaders (i.e., RCMP officers)
  • policy makers - government officials at the PT and Federal levels who are involved in health promotion and education, as well as policy makers at the School District Level who review and approve programs for elementary schools
  • parents and families of children in participating schools

Canada-wide
(BC, AB, ON, NB)

Culturally-based, Family Centred Mental Health Promotion for Aboriginal Youth
(McGill)

$2.6M

The purpose of this project is to develop a culturally based approach to improving the mental health and well-being of youth living in Aboriginal communities across Canada.

  • Enhancing psychological, social, and emotional well-being among Aboriginal youth, their families and communities.
  • Bringing together community partners from across Canada with mental health researchers at several universities.
  • Aboriginal communities with high rates of suicide and other wellbeing challenges
  • local facilitators - professional and skilled local leaders

Manitoba
British Columbia
Ontario
Quebec

The Fourth R: Promoting Youth Well-Being through Healthy Relationships
(Centre for Addiction and Mental Health)

$2.5M

The purpose of this project is to extend the empirically-validated Fourth R comprehensive school-based violence prevention program to younger adolescents, as well as reach parents, teachers and teacher candidates. The intervention also has a particular focus on culturally appropriate programming for Aboriginal youth to develop healthy relationship skills.

The target populations involved are youth ages 12-20 with a particular focus on Aboriginal youth, teachers,
parents, and future educators/teacher candidates. Over 240,000 individuals are engaged in this project.

  • Increasing youths' well-being and connectedness and decreasing violence.
  • Increasing the well-being and connectedness of at-risk urban, rural and northern Aboriginal youth.
  • Increasing parents' awareness, self-efficacy and skills in taking a more active role in countering the negative socialization of media violence issues with children.
  • Increasing the competency and self-efficacy of teacher candidates.
  • FNMI students and parents
  • Policy makers (advisory committee members)
  • General public

Ontario, Saskatchewan, Northwest Territories Alberta, and
Manitoba.

Getting Prepared for Life: Development, Implementation and Evaluation of a Mental Health Promotion Program focusing on Adjustment Mechanisms for Children in Grades 3 - 6
(Université de Québec à Montréal)

$2.5M

Project dealing with friendship, communication, loneliness, bullying, change, loss and making a new start

  • Showing children how to deal with day-to-day problems, identifying and naming their feelings and exploring different ways of coping with them.
  • School-based mental health promotion program to develop adjustment mechanisms for Quebec children aged 8 to 12 (grades 3 to 6)

Quebec (urban and Aboriginal settings)

Handle with Care in At-Risk Communities: A Program for Parents and Early Childhood Educators to Promote Young Children's Mental Health
(Hincks Dell)

$2.4M

Canadian-based training program for parents and caregivers

  • Fostering the social and emotional health of children through links with families and the community
  • Parents and caregivers who may be experiencing risk factors such as social isolation, new to the community, low income.

Ontario, PEI, Manitoba, Yukon

Improving the Emotional and Social Health of Children in their Community: Implications for Population Health
(Canadian Mental Health Association Nova Scotia Division)

$2.8M

School-based mental health promotion

  • Increasing the social and emotional competence of children in a manner that is sustainable, enhancing existing supports and resources and building on community strengths.
  • Children
  • Teachers, school staff
  • Professional and service providers
  • Nova Scotia government
  • General public

Alberta, Manitoba, Nova Scotia

Towards Flourishing: Improving Mental Health Among Families in the Manitoba Families First Home Visiting Program
(University of Manitoba)

$2.6M

Demonstration project designed to enhance the mental well-being of parents and children

  • Introducing a mental health strategy for families, public health and mental health staff.
  • Families in urban, rural and northern Manitoba who are receiving the FF Home Visiting program and who have at least one child under 1 year old
  • Public health nurses and home visitors who deliver the Towards Flourishing Program, mental health staff who collaborate with public health, managers who oversee public health and mental health programs
  • Policy makers
  • General public

Manitoba
(urban, rural, northern, First Nations and francophone communities)

 
Healthy Weights Phase II - Achieving Healthier Weights in Canada's Communities
Project Title Funding Detailed Description Key Objectives Target Populations Project Location(s)

Atii! Let's do it! A comprehensive health-living intervention for children, youth and families in Inuit communities in Nunavut (Qaujigiartiit Health Research Centre)

$0.5M

The purpose of this project is to have a youth-led culturally -developed program that aims to improve the ability of Inuit families to make healthy choices about food and physical activity; improve health literacy in Inuktitut; engage children, parents and guardians in a fun, culturally relevant health promotion activity in school settings; and explore an avenue to help address the loss of traditional harvesting skills among children and youth.

This project will take place in the Inuit communities of Iqaluit, Arviat and schools in several other northern communities.

  • Increasing the ability of Inuit families to make healthy choices about food and physical activity
  • Improving health literacy in the Inuktitut language
  • Engaging children, parents and guardians in a fun, culturally relevant health promotion activity in school settings
  • Increasing opportunities for children and youth to learn vital traditional harvesting skills
  • Expanding the capacity of youth leadership in developing and implementing the project
  • Aboriginal populations
  • Children
  • Community members
  • Parents/caregivers
  • Youth

Nunavut

Engaging communities: Achieving healthier weights through community food security in remote Inuit populations (Food Security Network of Newfoundland and Labrador)

$1.6M

The purpose of this project will be to implement and evaluate an innovative community-led food assessment model for engaging rural, remote, northern and underserved communities (children, youth and their families, adults and seniors) in designing community interventions to address obesity and access to healthy food on both a community and regional level.

This project will take place in four Inuit communities: three in Nunatsiavut and one in another Inuit region in Canada.

  • Addressing obesity and access to healthy food on both a community and regional level.
  • Aboriginal populations
  • Children
  • Community members
  • Parents/caregivers
  • Youth

Newfoundland

May be implemented in NWT and Nunavut

Health promoting schools program (Saskatoon Regional Health Authority)

$1.4M

Public health and schools will collaborate to implement and evaluate this school-based initiative in disadvantaged and underserved neighbourhoods. This project will aim to enhance physical and social environments and increase coordination of policies and services across the school and community for physical activity, healthy eating and positive mental health among children and their families.

This project will take place in urban, small town and rural areas covered by the Saskatoon Health Region and Vancouver Island Health Authority.

  • Increasing coordination of policies and services across the school and community
  • Supporting healthy eating, physical activity and mental health in urban and rural complex needs schools
  • Supporting welcoming, caring, inclusive and improved relationships among students and their families
  • Increasing understanding by children and their families of the importance of physical activity, healthy eating, mental health, while increasing opportunities for healthy choices and actions
  • Improving the health and well-being and overall learning outcomes by enhancing education resources and curriculum for classrooms
  • Aboriginal populations
  • Children
  • Parents/caregivers

Saskatchewan,
British Columbia

Healthy Start / Départ santé: A multi-level intervention to increase physical activity and healthy eating among young children (age 3 - 5) attending early learning programs (Réseau Santé en Français de la Saskatchewan)

$1.6M

This project will implement and evaluate an inclusive evidence-based provincial strategy to promote physical activity and healthy eating for children ages 0 to 5 in Francophone communities through resources, support and education for parents and childcare centre staff engaging schools and preschools. This project will take place in anglophone and francophone early learning centres in Saskatchewan and New Brunswick.

  • Providing outreach to early learning centres and pre-kindergarten programs focusing on children from diverse cultural and socio-economic backgrounds
  • Identifying the impact of physical activity and healthier eating habits in early learning centres and at home
  • Increasing key stakeholders' involvement in the project
  • Identifying how the Saskatchewan provincial strategy will impact childcare centres located within urban and rural settings
  • Identifying necessary actions to enable the successful implementation of the Saskatchewan provincial strategy
  • Aboriginal populations
  • Children

Saskatchewan, New Brunswick

Healthy weights for children (The Bridge Youth and Family Services Society)

$1.6M

This project will promote the achievement and maintenance of healthy weights for disadvantaged and underserved children and their families through the implementation and evaluation of innovative asset-based education modules engaging children and youth under 19, their parents, foster-parents and/or caregivers and their communities. This project will take place in ten communities: Yellowknife NWT, Bonavista NL, Moncton NB, Brantford ON, Temiskaming ON, Saskatoon SK, Calgary AB, Dease Lake, BC, Castlegar BC and Vancouver BC.

  • Implementing and evaluating innovative family education modules engaging disadvantaged and underserved children and youth, their parents, foster-parents and/or caregivers and their communities
  • Evaluating the impact of this model for children in care and their families
  • Aboriginal populations
  • Children
  • Parents/caregivers

British Columbia, NWT, Newfoundland, New Brunswick, Ontario, Saskatchewan, Alberta

Healthy Weights Connections (Previously Healthy Weights Forum: Working together to promote the health of First Nations and Métis children in our communities)
(The University of Western Ontario)

$1.6M

This project will promote physical activity and healthy eating among urban and rural First Nations (on and off-reserve) and Métis children and youth to prevent obesity and create supportive environments in both on and off-reserve settings. This project will take place in three Ontario sites: the London area, the Midland-Penetanguishene area, and First Nations communities and townships on Manitoulin Island.

  • Reduce the risk of obesity among Aboriginal children and youth by improving how local health and wellness organizations serve Aboriginal children and families.
  • Increase the culturally-appropriate programming available for Aboriginal children and their families.
  • Improve the relationships and collaboration among all components of the health system serving Aboriginal peoples.
  • Aboriginal populations
  • Service providers

Ontario

Launching community food centres in Canada: Building health and equity through food programs in low-income communities (Community Food Centres Canada)

$1.5M

This project will support a range of programs (e.g., community gardens, drop-in meals) engaging disadvantaged and underserved low income communities to build on community-led initiatives that will increase healthy food behaviour, access to healthy food, physical activity and social well-being. This project will take place in a variety of CFCs across Canada including Calgary, Winnipeg, Toronto and Halifax/Dartmouth.

  • Identifying and engaging 15 low income communities
  • Developing comprehensive, locally-driven program plans in each community that will support healthy food access (e.g., drop-in meals), food skills (e.g., community gardens and cooking groups), education and engagement (e.g., peer-support groups)
  • Children
  • Community members

Ontario, Alberta, Manitoba, Nova Scotia

Our food: Achieving healthier weights by reconnecting food and community (Ecology Action Centre)

$1.5M

This project will implement and evaluate a series of interventions (e.g., building food preparation/preservation and gardening skills, building community gardens and policy initiatives) that increase fruit and vegetable consumption, bolster food security and promote healthy behaviours in individuals living in higher-risk communities and considering gender roles among low income families, on- and off-reserve Aboriginal and recent immigrant children, youth, adults and seniors. This project will take place in Nova Scotia and New Brunswick.

  • Increasing fruit and vegetable consumption, food access, physical activity, knowledge of food security issues and overall health.
  • Improving community members' health.
  • Establishing local infrastructure and resources to enhance access to nutritious food.
  • Participating in the development of health promotion policies.
  • Building a network for sustaining efforts to promote the access and availability of healthier foods.
  • Community members
  • Service providers

Nova Scotia, New Brunswick

Our Food Our Health Our Culture: Achieving healthier weights in Manitoba and Saskatchewan (Food Matters Manitoba)

$1.5M

This project builds on traditional food skills considering culture and gender roles to promote food access and healthy behaviours through retail initiatives in urban, rural/ remote and on-reserve higher risk communities among Aboriginal and new immigrant children, youth and their families, adults and seniors. This project will take place within in Manitoba: Fox Lake Cree Nation, Winnipeg's North End, and Gillam; and Saskatchewan: La Ronge, Lac La Ronge Indian Band, and Air Ronge.

  • Enabling communities to achieve conditions for healthier weights by providing skills to harvest and eat traditional Aboriginal foods
  • Improving access to healthier food for low income populations in urban, rural, remote and on-reserve higher risk communities
  • Developing and implementing food policies with community agencies that provide food for vulnerable populations
  • Aboriginal populations
  • Children
  • Community members
  • Parents/caregivers
  • Youth
  • Service providers

Manitoba, Saskatchewan

Toward a network of green, active, healthy neighbourhoods / Vers un réseau de Quartiers verts, actifs et en santé
(Société de développement communautaire de Montréal)

$1.6M

This project will use a community-based approach to increase physical activity by promoting active modes of transport for children and youth and families with an emphasis on low income, new immigrants and Francophone minority neighbourhoods in urban and rural areas. This project will take place in 12 disadvantaged local communities in Quebec, Ontario and Alberta.

  • Equipping and mobilizing children and youth in 12 disadvantaged communities
  • Designing an active transportation plan
  • Organizing a community of practice to share knowledge and tools
  • Encouraging communities to adopt policies to support active transportation
  • Children and youth and families (adults and seniors) with an emphasis on low income, new immigrants and Francophone minority neighbourhoods in urban and rural areas

Quebec, Ontario, Alberta

Working together to achieve healthier lifestyles in Yukon and Northwest Territories communities (Arctic Institute of Community-based Research)

$1.6M

This project will develop healthy weights activities based on cultural values and traditional healing and food security practices and will target children, youth, families, adults and seniors through workplace and school initiatives across all communities. The project will engage youth in each community in the design of the activities, as well as conduct training programs for the delivery of these initiatives. The project will take place in disadvantaged and underserved on- and off-reserve communities in the Yukon and the Northwest Territories.

  • Its key objective will be to create sustainable programs and partnerships that act in the interest of improving the health and well-being of northerners and take into account their unique health inequalities.
  • Disadvantaged and underserved on- and off-reserve Aboriginal and non-Aboriginal children, youth and their families (adults and seniors).

Yukon, NWT

Appendix 2 - Logic Model

Logic Model
Text Equivalent

A description of the logic model for the Innovation Strategy:

The Innovation Strategy uses grants and contributions funding and O&M budget to deliver its activities, produce outputs and accomplish its outcomes.

The Innovation Strategy consists of two main activities:

  • Test and implement innovative population health interventions
    • Fund, support and monitor organizations to design, develop, implement, adapt and evaluate community-based population health interventions that address complex public health issues facing children, youth and families (e.g., mental health promotion and achieving healthier weights)
  • Generate and disseminate knowledge
    • Collaborate with stakeholders to develop and disseminate evidence-based knowledge products and lessons learned about population health interventions

As a result of each activity, the Innovation Strategy generates a number of outputs, namely:

  • New and existing partnerships from different sectors
  • Methods and models to promote intersectoral partnerships
  • Tools, approaches and models to support the implementation of population health interventions
  • Development and/or adaptation of population health interventions
  • Knowledge syntheses, research papers, training materials, presentations, webinars, case studies, summary reports

Through these outputs, program activities correspond with specific immediate, intermediate and ultimate outcomes. In the immediate term, program activities are expected to lead to partnerships that support the delivery of population health interventions; promising population health interventions that are implemented and evaluated; and stakeholders (health practitioners, researchers, P/Ts and other policy makers) that have access to knowledge products and synthesized learnings to advance population health policy and practice.

The achievement of these immediate outcomes is expected to lead to the following intermediate outcomes: population health interventions that contribute to improved protective factors, reduced risk behaviours, and improved health outcomes for individuals, families and communities; population health interventions that demonstrate readiness for scale up; and stakeholders that use knowledge products, intervention research evidence and synthesized learning to advance population health policy and practice. In the longer-term the Innovation Strategy expects to have population health interventions that contribute to improved health outcomes and reduced health inequalities for Canadians across the life course.

Footnotes

  1. Logic model outcome directly linked to PAA 1.2.2. results and indicators (See Section 2.2 and Annex A for additional information)
  2. Intervention research: the use of research methods to produce knowledge about interventions that have the potential to impact health at the population level, including how, when and in what context the interventions work or do not work.
  3. Scale up: a deliberate effort to increase the reach and impact of successfully tested population health promotion interventions to benefit more people and to foster sustainable policy/programme development across diverse populations and communities.

Appendix 3 - Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance:

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Table 1: Relevance Rating Symbols and Significance
Questions Indicators Overall Rating Summary
Continued Need for the Program
Legend - Relevance Rating Symbols and Significance:

There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear.
There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear.
There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.

What are the health/societal needs contributing to the need for this program?

Description of needs the Program aims to address, including:

  • Evidence of health inequities across Canada (and associated current & projected burden if applicable indicators exist)
  • Evidence of need for effective population health interventions aimed at reducing health inequities
  • Evidence of need for improved knowledge, development and exchange (dissemination) around effective population health interventions.

High

Reducing health inequalities is one of the most significant challenges facing public health today. For example, obesity and mental health are complex health problems that are related to biological, social, physical and economic factors. Most senior public health policy-makers agree that specific interventions are needed to reduce health inequalities; however, which interventions will be most effective in reducing inequalities are not well understood. Population health intervention research generates relevant, contextually sensitive, credible and timely knowledge that enables decision makers to continually improve programs and policies. Even though the interest in evidence-informed decision making related to population health has been growing in Canada and internationally, there is a lack of population health intervention research being funded in Canada. Further, the majority of research to date has been descriptive in nature and needs to move towards the identification of effective solutions.

What is the current and projected burden of mental illness and obesity in Canada? Who is at greatest risk of experiencing mental illness and obesity? What is the current state of positive mental health and healthy weights in Canada?

  • Evidence of:
    • Current burden
    • Projected burden
    • New environment/trends
  • Evidence of burden linked to health inequities

High

Every year, one in ten Canadians will experience a mental illness, and one in three will experience one sometime in their lives. In 2008, the direct cost of mental illness in Canada was estimated to be at least $7.4 billion. As for vulnerable populations, Aboriginal youth commit suicide five to six times more often than non-Aboriginal youth.

As for vulnerable populations, Aboriginal youth commit suicide five to six times more often than non-Aboriginal youth.

It is estimated that over 1 in 4 Canadian adults are obese, with 8.6% of children and youth aged 6 to 17 obese, costing the Canadian economy up to $7.1 billion each year.

In 2007-2010, Aboriginal adults had higher obesity rates than non-Aboriginal adults: First Nations (26%), Inuit (26%), Métis (22%) versus non-Aboriginal (16%). A recent study of socioeconomic-related inequalities in obesity risk among Canadian adults found that obesity is more prevalent among economically disadvantaged women.

Alignment with Government Priorities

What are the federal priorities related to addressing health inequalities? Are PHAC's current activities aligned with federal priorities?

  • Evidence of federal priorities related to addressing health inequalities
  • Program objectives correspond to recent/current federal priorities
  • Current program activities correspond to federal priorities

High

Canada has signed the Rio Political Declaration on Social Determinants of Health (2011) which reaffirms that health inequities within and between countries are politically, socially and economically unacceptable, as well as unfair and largely avoidable. The Rio declaration underscores the principles set out in the WHO Constitution and the 1986 Ottawa Charter that both promote a focus on health equity.
In the 2010 Speech from the Throne, the government reiterated that protecting the health of Canadians is a priority.

What are the PHAC priorities related to addressing health inequalities? Are current activities aligned with PHAC priorities?

  • Evidence of PHAC priorities related to addressing health inequalities
  • Program objectives aligned with and contribute to departmental strategic outcomes
  • Current program activities correspond to PHAC priorities

High

As detailed in Strategic Horizons 2013-2018, a population health approach will guide the Agency's focus for its priorities and strategic directions for the next five years.

Do program priorities (mental health promotion and healthy weights) align with federal and PHAC priorities?

  • Program priorities correspond to federal and PHAC priorities

High

The Economic Action Plan (2012) highlights the government's commitment to work with the provinces and territories (P/Ts), private and not-for-profit sectors to encourage young Canadians to be more physically active. The plan also proposed funding for mental health research.

Canada collaborated with its P/T counterparts in developing key national agreements, including the Integrated Pan-Canadian Healthy Living Strategy (2005 and reinforced in 2010), the Declaration on Prevention and Promotion from Canada's Ministers of Health and Health Promotion/Healthy Living (2010), and Curbing Childhood Obesity: A Federal, Provincial and Territorial Framework for Action to Promote Healthy Weights (2010).

Within PHAC, Strategic Horizons 2013-2018 identifies enhanced information sharing, partnerships, collaboration, healthy weights, obesity, mental health and intervention research as priorities. The Agency's Corporate Risk Profile also identifies obesity and mental health as key priorities.

Alignment with Federal Roles and Responsibilities

What is the federal public health role related to reducing health inequalities? Is intervention research an appropriate means by which to fulfil this role?

  • Evidence of federal public health role related to reducing health inequalities
  • Program objectives align with federal jurisdiction
  • Program objectives fit with departmental mandate and roles

High

The Agency's broad role in health promotion is outlined in the Department of Health Act and the Public Health Agency of Canada Act. The Agency's more specific roles in this area are presented under the Implementation of Population Health Programming as the basis for the Department of Health (Health Canada) Promotion of Population Health Business Line (1998), the Sustaining the Federal Health Protection Capacity (2000), the Renewal of Terms and Conditions for Promotion of Population Health Grants and Contributions (2003) and the Amendment to the Terms and Conditions for Promotion of Population Health (2013) program authorities.
Under the Constitution, the P/Ts are generally responsible for delivering health care, but the federal government has a number of roles and responsibilities such as ensuring access to health care for specific populations (e.g., First Nations), health promotion, disease prevention and knowledge sharing, and funding health research.

Is the federal public health role aligned with the current environment?

  • Federal public health role aligns with current program activities
  • Federal public health role aligns with the current environment

High

The Agency has a role to conduct research, provide leadership and promote overall health. The objectives of the Innovation Strategy align with the Agency's role in reducing health inequalities, providing leadership to promote the physical, social and mental well-being of Canadians, in addition to sharing knowledge of effective population health interventions.

What is the role of stakeholders (i.e., other government departments, provincial/territorial government, non-governmental organizations (private sector), related to health equity? Related to population health intervention research?

  • Evidence of OGD, P/T, NGO and private sector role related to health equity
  • Evidence of OGD, P/T, NGO and private sector role related to population health intervention research

High

Outside the Agency, other government departments that appear to support the delivery of population health interventions include Health Canada's First Nations and Inuit Health Branch, the Department of Employment and Social Development Canada, and the Canadian Institutes of Health Research (CIHR). It is not clear if the population health intervention work of Health Canada or Employment and Social Development Canada include an intervention research component.

Does the federal public health role and current activities duplicate the role of stakeholders? Are there any gaps or overlaps?

  • Presence or absence of duplication/overlap/ complementarity of role between federal public health role and stakeholders role
  • Views/perceptions of gaps between federal public health role and stakeholders role

Partial

Overall, the Innovation Strategy does not appear to duplicate the work of CIHR (discussed in section 4.5). For example, while the Innovation Strategy funds the development, implementation, evaluation, and scale up of interventions, CIHR's IPPH primarily funds research on population health interventions. The Pathways to Health Equity for Aboriginal Peoples is an initiative that is led by CIHR, in partnership with the Public Health Agency of Canada and Health Canada's First Nations and Inuit Health Branch (FNIHB). This initiative aims to develop the evidence base in the design and implementation of programs and policies that promote health and health equity in the following four priority areas: suicide prevention, tuberculosis, diabetes/obesity and oral health. Possible overlap with the Innovation Strategy may exist given that this initiative is involved in the development, implementation, evaluation and scale up of interventions; however, this project is solely focused on Aboriginal peoples and funding is only used for research. Unlike the Innovation Strategy, funding of the intervention is explicitly excluded in the case of the Pathways Initiative.

Given that intervention research is defined differently by various organizations, the extent to which overlap may exist is not well understood.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Table 2: Performance Rating Symbols and Significance
Questions Indicators Overall Rating Summary
Achievement of Expected Outcomes (Effectiveness)

N/A - Not applicable

Legend - Relevance Rating Symbols and Significance:

The intended outcomes or goals have been achieved or met.
Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed.


Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.


To what extent have promising population health interventions been implemented and evaluated?

  • Evidence of promising projects from phase 1 being implemented and expanded in phase 2
  • Evidence of evaluations of phase 2 projects
  • Views on achievement of outputs and outcomes

Achieved

Population health interventions have been implemented, along with tools, approaches and methods, and evaluated as planned.

To facilitate implementation, evaluation and scale up of the interventions, projects have developed, used and disseminated tools, approaches and methods. Examples include: cultural adaptation tool-kits, community engagement strategies, training materials and activities, evaluation frameworks and intervention models.

To what extent have program activities contributed to partnerships that support the delivery of population health interventions?

  • Evidence of expertise and resources leveraged through IS partnerships (e.g., financial aid, in-kind resources, support to enhance community engagement, support to create change at a policy level, support to disseminate information)
  • Views on achievement of outputs and outcomes

Achieved

A range of intersectoral partnerships have been established by all projects that support the delivery of the population health interventions.

Based on 2013-2014 performance data, MHP projects (n = 9) currently involve 289 partnerships, whereas AHW projects (n = 11) involve 262 partnerships. Examples of intersectoral partners include: school boards, provincial ministries, local municipalities, grocery stores, Inuit community governments, local community wellness groups, law enforcement, parent organizations and Aboriginal friendship centres.

The most common areas of focus for partners were health, education, social services, and aboriginal issues.

To what extent have program stakeholders accessed program knowledge products and synthesized learnings (reach)?

  • Evidence of knowledge development and exchange plans developed by projects
  • Evidence of knowledge dissemination (with respect to project learnings) among projects
  • Evidence of knowledge dissemination to key stakeholders (researchers, health workers, policy makers)
  • Evidence of knowledge product access among key stakeholders

Achieved (at the Project Level)

Progress Made; Further Work Warranted (at the Program Level)

At the project level, knowledge products and activities have been generated and accessed by program stakeholders and the public. Knowledge synthesis and dissemination has occurred to a lesser extent at the program level.

IS projects have developed a number of knowledge products (e.g., manuals/training kits, brochures/ pamphlets/posters, web sites, newsletters, position papers/research summaries) and delivered a number of knowledge activities (e.g. workshops/presentations, activities to influence policy, and community events).

Project-generated knowledge products and activities have been accessed by program stakeholders as reported:

MHP Phase 2 knowledge products and activities were accessed by 273,682 stakeholders.

AHW Phase 2 knowledge products and activities were accessed by 107,779 stakeholders.

At the program level, 50 knowledge products have been developed over the last five years. These include summaries of the IS program to inform stakeholders (5), project summaries including details on findings (5), analysis of project information (summative across program) (15), research to support implementation and enhance knowledge around IS priorities (25), and knowledge events (15).

There has been limited tracking of knowledge product dissemination at the program level, however the program has recently begun efforts to measure this.

To what extent have program stakeholders used program knowledge products and synthesized learnings to advance population healthy policy and practice?

  • Evidence of knowledge product use among key stakeholders (e.g., to advance population health policy and practice)
  • Views on achievement of outputs and outcomes

Progress Made; Further Work Warranted

While there are examples of knowledge products being used by stakeholders, particularly at the project level, the precise extent to which use is occurring, and how, has generally not been well documented at the project or program level.

While not systematically or consistently tracked across the projects, the following metrics were available on use of IS knowledge products (by individuals outside the funded projects):
MHP

  • Phase 1 - not reported
  • Phase 2 (2012-2013) - 78% of projects reported that knowledge products were being used.

AHW

  • Phase 1 - 39% of knowledge products reported to have been used by stakeholders.

Phase 2 - 40% of knowledge products reported to have been used by stakeholders.

Use of program-generated knowledge products has not been systematically tracked. However, surveys of knowledge users (n=36) from the IS Network (IS program and project staff and program partners) who requested knowledge development and exchange products in 2013‐2014 reveals that the majority of respondents had used IS knowledge products.

The program has recently begun efforts to track and understand uptake and use of IS knowledge products.

To what extent do funded population health interventions contribute to improved protective factors, reduced risk behaviours, and improved health outcomes for individuals, families and communities?

  • Evidence of (performance data on) funded projects contributing to improved protective factors, reduced risk behaviours and improved health outcomes for individuals, families and communities.
  • Views on achievement of outputs and outcomes

N/A
(too soon to assess) for Healthy Weights)

Progress Made; Further Work Warranted
(for Mental Health)

There are early indications that the population health interventions funded by the Innovation Strategy are contributing to improved health outcomes. These include improved knowledge and skills in children and families, including in Aboriginal settings.

Measureable progress towards this outcome is not expected until after 4-6 years. While MHP projects have been delivered for this length of time, it is still early to fully expect these outcomes for AHW projects.

In their annual reports, IS projects report on progress towards this outcome. Changes in health outcomes (improved protective factors and reduced risk factors) were measured using mixed methods including pre/post surveys, retrospective surveys, and observational data. Overall changes in health outcomes were self-reported through surveys.

Are there early indications that program population health interventions demonstrate readiness for scale up?

  • Perceptions/views on whether funded projects appear to be on track towards readiness for scale up

Progress Made; Further Work Warranted

There are early indications of interventions demonstrating readiness for scale up, which is Phase 3 of the Innovation Strategy.

A recently produced IS program knowledge product entitled “Understanding Scale Up” presents lessons learned from the delivery of Phase 1 and 2 of the IS. Key factors for effective scale up of interventions are highlighted in this paper (see appendices). Among these are the following for which the evaluation was able to collect data on: Partnership development and system readiness.

Demonstration of Economy and Efficiency

Has the program undertaken its activities in the most efficient manner?

  • Are there alternate, more efficient ways to deliver these activities?
  • How could efficiency of activities be improved?

Has the program undertaken its activities in the most economical manner?

  • Variance between planned and actual expenditures, and implications
  • Views on if funds are appropriately targeted
  • Views on whether costs of producing outputs is as low as possible and value is being obtained, incl. administrative demands of program participation for funding recipients
  • Degree of leveraged resources
  • Comparison of cost per output between similar programs/ projects (where available)
  • Appropriateness of administrative overhead % (where relevant)
  • Evidence of and views on alternative program models that would achieve outcomes at lower cost (where available)

Progress Made; Further Work Warranted

Innovation Strategy activities are well managed and work has begun to improve performance measurement related to tracking the impact of knowledge products. Further efficiencies could be gained through improved collaboration and knowledge exchange with other areas of the Agency and the Health Portfolio.

The phased approach of the IS supports program efficiency, given that only those projects demonstrating promise are moved to the subsequent phases to receive continued funding.

Within the Health Portfolio, other population health intervention research-related activities are currently delivered by the Agency and CIHR (e.g. Pathways to Health Equity), however it is not clear what mechanisms exist to foster information exchange among these activities. The IS has built a relationship with CIHR, however, additional opportunities exist for improved collaboration. For example, the Agency has signed a memorandum of understanding with CIHR and Health Canada to support the delivery of the Pathways to Health Equity Initiative. Similarly, the Applied Public Health Chairs program has involved joint priority setting between the Agency and CIHR.

Innovation Strategy projects have successfully leveraged funding to support program delivery.

IS projects have been successful in leveraging funds and receiving in-kind resources, particularly in the form of loaned staff and volunteers. In 2013-2014 alone, IS projects have leveraged approximately $1.8M, and received approximately $1.4M of in-kind support.

Is there appropriate performance measurement in place? If so, is the information being used to inform senior management decision-makers?

  • Existence of performance measurement framework or strategy
  • Adequate collection of performance information
  • Evidence of use of performance measurement information in decision-making
  • Perception of appropriateness of performance measurement data collected (supports decision making? Supports evaluation?)

Progress Made; Further Work Warranted

The program has been successful in monitoring performance, using researched and standardized tools. Performance data is systematically analyzed and used to inform program delivery (e.g., lessons learned report). However, to gain a better understanding of the impact of IS knowledge products, additional performance measurement is needed. The program has recently begun efforts to this end, including a dissemination plan and a knowledge uptake plan (for collecting information on knowledge product use).

 
Table 3: Summary of Relevance Ratings
Evaluation Issue High Partial Low
Legend - Relevance Rating Symbols:

There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear.
There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear.
There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.
Issue 1: Continued need for the program

What are the health/societal needs contributing to the need for this program?

High

N/A

N/A

What is the current and projected burden of mental illness and obesity in Canada? Who is at greatest risk of experiencing mental illness and obesity? What is the current state of positive mental health and healthy weights in Canada?

High

N/A

N/A

Issue 2: Aligned to federal government priorities

What are the federal priorities related to addressing health inequalities? Are PHAC's current activities aligned with federal priorities?

High

N/A

N/A

What are the PHAC priorities related to addressing health inequalities? Are current activities aligned with PHAC priorities?

High

N/A

N/A

Do program priorities (mental health promotion and healthy weights) align with federal and PHAC priorities?

High

N/A

N/A

Issue 3: Program consistent with federal roles and responsibilities

What is the federal public health role related to reducing health inequalities? Is intervention research an appropriate means by which to fulfill this role?

High

N/A

N/A

Is the federal public health role aligned with the current environment?

High

N/A

N/A

What is the role of stakeholders (i.e., other government departments, provincial/territorial government, non-governmental organizations (private sector), related to health equity? Related to population health intervention research?

High

N/A

N/A

Does the federal public health role and current activities duplicate the role of stakeholders? Are there any gaps or overlaps?

N/A

Partial

N/A

Table 4: Summary of Performance Ratings
Evaluation Issue Achieved Progress Made; Further Work Warranted Little Progress; Priority for Attention

Legend -  Performance Rating Symbols:

The intended outcomes or goals have been achieved or met.
Considerable progress has been made to meet the intended outcomes or goals, but attention is still needed.


Little progress has been made to meet the intended outcomes or goals and attention is needed on a priority basis.

Issue 4: Achievement of intended outcomes (effectiveness)

To what extent have promising population health interventions been implemented and evaluated?

Achieved

N/A

N/A

To what extent have program activities contributed to partnerships that support the delivery of population health interventions?

Achieved

N/A

N/A

To what extent have program stakeholders accessed program knowledge products and synthesized learnings (reach)?

Achieved

Progress Made;
Further Work
Warranted

(at the Program Level)

N/A

To what extent have program stakeholders used program knowledge products and synthesized learnings to advance population healthy policy and practice?

N/A

Progress Made;
Further Work
Warranted

N/A

To what extent do funded population health interventions contribute to improved protective factors, reduced risk behaviours, and improved health outcomes for individuals, families and communities?

N/A

Progress Made;
Further Work
Warranted

(for Mental Health - too soon to assess for Healthy Weights)

N/A

Are there early indications that program population health interventions demonstrate readiness for scale up?

N/A

Progress Made;
Further Work
Warranted

N/A

Issue 5: Demonstrated economy and efficiency

Has the program undertaken its activities in the most efficient manner?

  • Are there alternate, more efficient ways to deliver these activities?
  • How could efficiency of activities be improved?

Has the program undertaken its activities in the most economical manner?

N/A

Progress Made;
Further Work
Warranted

N/A

Is there appropriate performance measurement in place? If so, is the information being used to inform senior management decision-makers?

N/A

Progress Made;
Further Work
Warranted

N/A

Appendix 4 - International Comparison

Australia

Australia's National Health and Medical Research Council (NHMRC) is the largest single funder of health and medical research in Australia. Many of NHMRC's funding schemes address multiple priority actions contained in their 2013-2015 Strategic Plan including intervention research in the area of obesityEndnote 43. Knowledge, development and exchange to support the health of Australians is also an important part of NHMRC's 2013-2015 Strategic Plan.

Additionally, the Centres of Clinical Research Excellence scheme provides funding for innovative, high quality clinical research to:

  • support clinical research with potential to lead to improved health outcomes for the community;
  • foster training of clinical researchers, particularly those with a capacity for independent research and future leadership roles; and
  • ensure effective translation of research outcomes into clinical practiceEndnote 44,Endnote 45.

United Kingdom

Healthcare in the United Kingdom is devolved, meaning that there are four regions: England, Scotland, Wales and Northern Ireland, each with its own healthcare system. The Department of Health is the Ministerial department responsible for government policy on health and social matters in England, along with a few elements not otherwise devolved to the Scottish Government, Welsh Government or Northern Ireland Executive. Although devolved, Public Health England (an Executive Agency of the Department of Health), the Public Health Agency in Northern Ireland, the National Health Service in Wales and the National Health Service in Scotland all explicitly state that their role or mission is to improve health and address/reduce health inequalitiesEndnote 46,Endnote 47,Endnote 48,Endnote 49.

The National Institute for Health Research is a UK government body that coordinates and funds research for the National Health Service in England. The National Institute for Health Research has four main work strands: research, infrastructure, faculty and systems that are managed by National Institute for Health Research Coordinating CentresEndnote 50. In 2006, the National Institute for Health Research was launched as a ‘virtual' organisation, meaning that it is not a corporation or a legal entity or a ‘bricks and mortar' enterprise in the traditional sense; it is an overarching entity which collectively represents all publicly-funded research in the National Health Service.Endnote 51

Also in 2006, the major funders of public health research in the UK came together under the UK Clinical Research Collaboration (UKCRC) to develop a coordinated approach to improving the UK public health research environment. The findings of the UKCRC Public Health Research Strategic Planning Group are documented in a report, Strengthening Public Health Research in the UK. The outcome of which was a commitment by a consortium of eight funding partnersFootnote viii to create five UKCRC Public Health Research Centres of Excellence in 2008Endnote 52,Endnote 53,Endnote 54.

The Centre for Development and Evaluation of Complex Interventions for Public Health (DECIPHer), a strategic partnership between Cardiff, Bristol and Swansea Universities in Wales, is one of the five Centres for Excellence. DECIPHer develops, tests, evaluates and implements complex interventions and policies that achieve sustainable improvements in health and wellbeing, and address health inequalities. DECIPHer's research focuses on three broad priority areas relating to the health of children and young people: tobacco, alcohol and drugs; obesity, physical activity and diet; mental health and wellbeingEndnote 55.

United States

In the United States, the National Institute on Minority Health and Health Disparities is the leader within the US National Institutes of HealthFootnote ix for scientific research to improve minority health and eliminate health disparities. The NIMHD Community-based Participatory Research (CBPR) Initiative supports collaborative research efforts between scientific researchers and community members to address diseases and conditions disproportionately affecting health disparity populations. The community is involved in the NIMHD CBPR Initiative as an equal partner with the scientists which helps to ensure that interventions created are responsive to the community's needs.

The NIMHD CBPR Initiative has three phases:

  • Phase I (the Planning Phase) provides three years of funding for the community and its scientific research partners to conduct the needs assessment, identify their priorities, and design the intervention
  • Phase II (the Research Intervention Phase) provides grantees up to five years of funding to refine the intervention, develop methods to evaluate its effectiveness, and implement the full-scale intervention.
  • Phase III (the Information Dissemination Phase) provides three years of funding for the research community partners to share their research findings and insights gained from the intervention with the targeted CBPR community as well as with other researchers and organizations

Appendix 5 - Evaluation Description

Evaluation Scope

The scope of the evaluation included an assessment of the relevance and performance of the Innovation Strategy from 2009-2010 to 2013-2014.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Table 1: Core Evaluation Issues and Questions
Core Issues Evaluation Questions
Relevance

Issue #1: Continued Need for Program

Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians

  • What are the health/societal needs contributing to the need for this program?
  • What is the current and projected burden of mental illness and obesity in Canada? Who is at greatest risk of experiencing mental illness and obesity? What is the current state of positive mental health and healthy weights in Canada?

Issue #2: Alignment with Government Priorities

Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes

  • What are the federal priorities related to addressing health inequalities? Are PHAC's current activities aligned with federal priorities?
  • What are the PHAC priorities related to addressing health inequalities? Are current activities aligned with PHAC priorities?
  • Do program priorities (mental health promotion and healthy weights) align with federal and PHAC priorities?

Issue #3: Alignment with Federal Roles and Responsibilities

Assessment of the role and responsibilities for the federal government in delivering the program

  • What is the federal public health role related to reducing health inequalities? Is intervention research an appropriate means by which to fulfil this role?
  • Is the federal public health role aligned with the current environment?
  • What is the role of stakeholders (i.e., other government departments, provincial/territorial government, non-governmental organizations (private sector), related to health equity? Related to population health intervention research?
  • Does the federal public health role and current activities duplicate the role of stakeholders? Are there any gaps or overlaps?
Performance (effectiveness, economy and efficiency)

Issue #4: Achievement of Expected Outcomes (Effectiveness)

Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes

  • To what extent have promising population health interventions been implemented and evaluated?
  • To what extent have program activities contributed to partnerships that support the delivery of population health interventions?
  • To what extent have program stakeholders accessed program knowledge products and synthesized learnings (reach)?
  • To what extent have program stakeholders used program knowledge products and synthesized learnings to advance population healthy policy and practice?
  • To what extent do funded population health interventions contribute to improved protective factors, reduced risk behaviours, and improved health outcomes for individuals, families and communities?
  • Are there early indications that program population health interventions demonstrate readiness for scale up?

Issue #5: Demonstration of Economy and Efficiency

Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes

  • Has the program undertaken its activities in the most efficient manner?
    • Are there alternate, more efficient ways to deliver these activities?
    • How could efficiency of activities be improved?
  • Has the program undertaken its activities in the most economical manner?
  • Is there appropriate performance measurement in place? If so, is the information being used to inform senior management decision-makers?

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Sources of information used in this evaluation included the following.

Document and File Review

All documents provided by the Project Authority for the purposes of the evaluation were reviewed to provide a foundation for the evaluation and contribute a line of evidence that addresses most evaluation questions. The following types of documents were reviewed: program- and project (study)-level descriptive and administrative materials; Government and departmental level policy and planning documents; and evaluation and performance reports: to maximize the efficiency and utility of the document review, a document review template was developed to facilitate the systematic review of materials.

Key Informant Interviews

The main purpose of the interviews was to gather information to fill gaps identified in the document/file review and to provide evidence and detailed information to help contextualize evidence gathered from other sources. Tailored guides were developed to be suitable for administration with two groups of key informants: internal respondents from the Public Health Agency who were involved in the design and delivery of the Innovation Strategy; senior management and staff from the Centre for Chronic Disease and Prevention (CCDP); staff from other divisions within the Centre for Health Promotion; other Health Portfolio staff (Health Canada and CIHR); provincial and territorial representatives and an external stakeholder from the Partnership Against Cancer's Coalitions Linking Action and Science for Prevention. In total, eight interviews with program representatives were conducted; five interviews were conducted with CCDP; three interviews were conducted with other Centre for Health Promotion staff; three were conducted with representatives from the Health Portfolio; three were conducted with provincial and territorial representatives and one interview was conducted an external stakeholder. In total, twenty-three interviews were conducted for this evaluation.

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included: systematic compilation, review and summarization of documents, quantitative analysis of data; thematic analysis of qualitative data from key informant interviews; and comparative analysis of data from disparate sources to validate summary findings.

References

  • Armstrong, R., Waters, E., Moore, L., Riggs, E., Cuervo, L.G., Lumbiganon, P. and Hawe, P. (2008). Improving the reporting of public health intervention research: advancing TREND and CONSORT. Journal of Public Health, 30 (1), 103-109.
  • Australian Government. (2012). NHMRC Strategic Plan 2013-2015. Retrieved from https://www.nhmrc.gov.au/_files_nhmrc/publications/attachments/nh160_nhmrc_strat_plan_201315.pdf
  • Bambra, C.L., Hiller, F.C., Moore, H.J., and Summerbell, C.D. (2012). Tackling inequalities in obesity: a protocol for a systematic review of the effectiveness of public health interventions at reducing socioeconomic inequalities in obesity amongst children. Systematic Reviews, 1(16), 1-7.
  • Boivin, A., Lehoux, P., Burgers, J., and Grol, R. (2014). What Are the Key Ingredients for Effective Public Involvement in Health Care Improvement and Policy Decisions? A Randomized Trial Process Evaluation. The Milbank Quarterly, 92(2), 319-350.
  • Cameron, R., Riley, B, Campbell, H.S., Manske, S. and Lamers-Bellio, K. (2009). The Imperative of Strategic Alignment Across Organizations: The Experience of the Canadian Cancer Society's Centre for Behavioural Research and Program Evaluation. Canadian Journal of Public Health, 100 (1), I27-I30.
  • Canadian Institutes of Health Research. (2015). Population Health Intervention Research Initiative for Canada. Retrieved from http://www.cihr-irsc.gc.ca/e/38731.html
  • Canadian Institutes of Health Research. (2015). About Us. Retrieved from http://www.cihr-irsc.gc.ca/e/37792.html
  • Canadian Institutes of Health Research. (2015). About the Institutes. Retrieved from http://www.cihr-irsc.gc.ca/e/7155.html
  • Canadian Institutes of Health Research. (2015). About PHIRIC. Retrieved from http://www.cihr-irsc.gc.ca/e/38732.html
  • Canadian Institutes of Health Research. (2015). Institute of Population and Public Health. Retrieved from http://www.cihr-irsc.gc.ca/e/13777.html
  • Canadian Institutes of Health Research. (2015). Pathways to Health Equity for Aboriginal Peoples. Retrieved from http://www.cihr-irsc.gc.ca/e/43630.html
  • Canadian Mental Health Association. (2015). Fast Facts About Mental Illness. Retrieved from http://www.cmha.ca/media/fast-facts-about-mental-illness/#.VLS4yfnF_Fk
  • Davies, J.K., and Sherriff, N. (2010). The gradient in health inequalities among families and children: A review of evaluation frameworks. Health Policy, 101(2011), 1-10.
  • DECIPHer: Development and Evaluation of Complex Interventions for Public Health Improvement. (2015). About DECIPHer. Retrieved from http://decipher.uk.net/about-decipher/
  • Di Ruggiero, E., Rose, A., and Gaudreau, K. (2009). Canadian Institutes of Health Research Support for Population Health Intervention Research in Canada. Canadian Journal of Public Health, 100(1), I15-I19.
  • Edwards, N., and Di Ruggiero, E. (2011). Exploring which context matters in the study of health inequities and their mitigation. Scandinavian Journal of Public Health, 39(6), 43-49.
  • Edwards, N. (2009). Revisiting our Social Justice Roots in Population Health Intervention Research. Canadian Journal of Public Health, 100(6), 405-406.
  • Government of Canada. (1996). Department of Health Act. Retrieved from http://laws-lois.justice.gc.ca/PDF/H-3.2.pdf
  • Government of Canada. (2015). Economic Burden of Illness in Canada (EBIC). Retrieved from http://ebic-femc.phac-aspc.gc.ca/custom-personnalise/results-national-resultats.php
  • Government of Canada. (2014-12-20). Minister's Message: Healthy Living. Retrieved from http://www.healthycanadians.gc.ca/video/healthy-living-vie-saine-eng.php
  • Government of Canada. (2006). Public Health Agency of Canada Act. Retrieved from http://lois-laws.justice.gc.ca/PDF/P-29.5.pdf
  • Government of Canada. (2008). The Chief Public Health Officer's Report on the State of Public Health in Canada: Addressing Health Inequalities. Retrieved from http://www.phac-aspc.gc.ca/cphorsphc-respcacsp/2008/fr-rc/pdf/CPHO-Report-e.pdf
  • Government of the United Kingdom. (2015). Public Health England. Retrieved from https://www.gov.uk/government/organisations/public-health-england
  • Hajizadeh, M., Campbell, M.K. and Sarma, S. (2014-03). Socioeconomic inequalities in adult obesity risk in Canada: trends and decomposition analyses. European Journal of Health Economics, 15(2), 203-221.
  • Haman, F., Fontaine-Bisson, B., Batal, M., Imbeault, P., Blais, J.M., and Robidoux, M.A. (2010). Obesity and type 2 diabetes in Northern Canada's remote First Nations communities: the dietary dilemma. International Journal of Obesity, 34, S24-S31.
  • Hanusaik, N., O'Louhlin, J., Kishchuk, N., Paradis, G., and Cameron, R. (2009). Organizational capacity for chronic disease prevention: A survey of Canadian public health organizations. European Journal of Public Health, 20(2), 195-201.
  • Hawe, P., Di Ruggiero, E., and Cohen, E. (2012). Frequently Asked Questions About Population Health Intervention Research. Canadian Journal of Public Health, 103(6), e468-e471.
  • Hawe, P., Samis, S., Di Ruggiero, E., and Shoveller, J.A. (2011). Population Health Intervention Research Initiative for Canada: progress and prospects. NSW Public Health Bulletin, 22 (1-2), 27-32.
  • Hawe, P., and Potvin, L. (2009). What is Population Health Intervention Research? Canadian Journal of Public Health, 100 (1), I8-I14.
  • Health and Social Care in Ireland. (2015). Public Health Agency. Retrieved from http://www.publichealth.hscni.net/
  • Health Canada. (2015). First Nations & Inuit Health: Mental Health and Wellness. Retrieved from http://www.hc-sc.gc.ca/fniah-spnia/promotion/mental/index-eng.php
  • Hobin, E.P., Hayward, S., Riley, B., Di Ruggiero, E., and Birdsell, J. (2012). Maximising the use of evidence: exploring the intersection between population health intervention research and knowledge translation from a Canadian perspective. Evidence and Policy, 8(1), 97-115.
  • Kendall, P.R.W. (2009). Commentary on Population Health Intervention Research. Canadian Journal of Public Health, 100(1), I31-I32.
  • Lalonde, M. (1974). A New Perspective on the Health of Canadians: a Working Document (Catalogue No. H31-1374). Retrieved from http://www.phac-aspc.gc.ca/ph-sp/pdf/perspect-eng.pdf
  • Mangham, L.J., and Hanson, K. (2010). Scaling up in international health: what are the key issues? Health Policy and Planning, 25, 85-96.
  • Mental Health Commission of Canada. (2014). Why Investing in Mental Health Will Contribute to Canada's Economic Prosperity and to the Sustainability of our Health Care System. Retrieved from  http://strategy.mentalhealthcommission.ca/pdf/case-for-investment-en.pdf
  • Merritt, M.W., Taylor, H.A., and Mullany, L.C. (2010). Ancillary Care in Community-Based Public Health Intervention Research. American Journal of Public Health, 100(2), 211-216.
  • Morabia, A., and Costanza, M.C. (2012). Population Health Intervention Research (PHIR): Should we fear the “P”. Preventive Medicine, 54, 291-292.
  • National Health Service. (2015). Public Health Wales: Priorities for Action. Retrieved from http://www.wales.nhs.uk/sitesplus/888/page/70468
  • National Health Service. (2015). Health Scotland. Retrieved from http://www.healthscotland.com/
  • National Institute for Health Research. (2015). About NIHR. Retrieved from http://www.nihr.ac.uk/about/
  • National Institute for Health Research. (2015). History of the NIHR. Retrieved from http://www.nihr.ac.uk/about/history-of-the-nihr.htm
  • Naylor, D. (2003). Learning from SARS: Renewal of Public Health in Canada (Catalogue No. H21-220/2003E). Retrieved from http://www.phac-aspc.gc.ca/publicat/sars-sras/naylor/index-eng.php
  • Nutbeam, D. (2001). Evidence-based public policy for health; Matching research to policy. IUHPE Promotion and Education, 15-19.
  • Pettersson, B. (2011). Some bitter-sweet reflections on the Ottawa Charter commemorative cake: a personal discourse from an Ottawa rocker. Health Promotion International, 26(2), 173-179.
  • Prince, S.A. (2009). A population health approach to obesity in Canada - Putting the problem back into context. Transdisciplinary Studies in Population Health Series, I (I), 22-33.
  • Public Health Agency of Canada. (2015). What Determines Health? Retrieved from http://www.phac-aspc.gc.ca/ph-sp/determinants/index-eng.php
  • Public Health Agency of Canada. (2015). What is the Population Health Approach? Retrieved from http://www.phac-aspc.gc.ca/ph-sp/approach-approche/index-eng.php
  • Public Health Agency of Canada. (2015). World Conference on Social Determinants of Health. Retrieved from http://www.phac-aspc.gc.ca/ph-sp/determinants/wcshd-cmdss-eng.php
  • Public Health Agency of Canada. (2014). Innovation Strategy Performance Measurement Strategy.
  • Public Health Agency of Canada. (2012-03-22). Scan of Current State of Intervention Research Funding & Support: Chronic Disease Prevention and Health Promotion.
  • Public Health Agency of Canada. (2011). Obesity in Canada. Retrieved from http://www.phac-aspc.gc.utting ca/hp-ps/hl-mvs/oic-oac/assets/pdf/oic-oac-eng.pdf
  • Sport Matters. (2013). Federal, Provincial and Territorial Ministers Advance Actions to Support Sport, Physical Activity and Healthy Weights. Retrieved from http://www.sportmatters.ca/en/news-82-federal-provincial-and-territorial-ministers
  • Raine, K.D., Plotnikoff, R., Schopflocher, D., Lytvyak, E., Nykiforuk, C.I., Storey, K., Ohinmaa, A., Purdy, L., Veugelers, P., and Wild, T.C. (2013). Healthy Alberta Communities: Impact of a three-year community-based obesity and chronic disease prevention intervention. Preventive Medicine, 57, 955-962.
  • Raine, K.D., Plotnikoff, R., Nykiforuk, C., Deegan, H., Hemphill, E., Storey, K., Schopflocher, D., Veugelers, P., Wild, T.C., and Ohinmaa, A. (2010). Reflections on community-based population health intervention and evaluation for obesity and chronic disease prevention: the Healthy Alberta Communities project. International Journal of Public Health, 55, 679-686.
  • Rychetnik, L., Frommer, M., Hawe, P., and Shiell, A. (2002). Criteria for evaluating evidence on public health interventions. Journal of Epidemiological Community Health, 56, 119-127.
  • Senate Subcommittee on Population Health. (2009-06). A Healthy Productive Canada: A Determinant of Health Approach. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/402/popu/rep/rephealth1jun09-e.pdf
  • Senate Subcommittee on Population Health. (2008). Population Health Policy: Issues and Options. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/392/soci/rep/rep10apr08-e.pdf
  • Senate Subcommittee on Population Health. (2014). The Health of Canadians - The Federal Role Interim Report. Retrieved from http://www.parl.gc.ca/Content/SEN/Committee/371/soci/rep/repintsep01part5-e.htm
  • Smith, B.T., Smith, P.M., Harper, S., Manuel, D.G., and Mustard, C.A. (2013-12-20). Reducing social inequalities in health: the role of simulation modelling in chronic disease epidemiology to evaluate the impact of population health interventions. Journal of Epidemiology & Community Health, 1-6. Retrieved from http://jech.bmj.com/content/early/2013/12/20/jech-2013-202756.full.pdf+html
  • Statistics Canada. (2015). Health at a Glance: Select health indicators of First Nations People living off reserve, Métis and Inuit . Retrieved from http://www.statcan.gc.ca/pub/82-624-x/2013001/article/11763-eng.htm
  • Statistics Canada. (2013-09-18). Health at a Glance: Mental and substance use disorders in Canada. Retrieved from http://www.statcan.gc.ca/pub/82-624-x/2013001/article/11855-eng.htm
  • Sullivan, L. (2009). Introduction to the Population Health Intervention Research Initiative for Canada. Canadian Journal of Public Health, 100 (1), I5-I6.
  • Twells, L.K., Gregory, D.M., Reddigan, J., and Midodzi, W.K. (2014). Current and predicted prevalence of obesity in Canada: a trend analysis. Canadian Medical Association Journal, 2(1), E18-E26.
  • World Health Organization. (2015). Rio Political Declaration on Social Determinants of Health. Retrieved from http://www.who.int/sdhconference/declaration/en/
  • World Health Organization. (2006). Constitution of the World Health Organization. Retrieved from http://www.who.int/governance/eb/who_constitution_en.pdf
  • World Health Organization. (2015). The Ottawa Charter for Health Promotion. Retrieved from http://www.who.int/healthpromotion/conferences/previous/ottawa/en/
  • World Health Organization. (2015). Trade, foreign policy, diplomacy and health. Retrieved from http://www.who.int/trade/glossary/story046/en/
  • World Health Organization. (2015). World Conference on Social Determinants of Health. Retrieved from http://www.who.int/sdhconference/en/
  • Wozniak, L., Rees, S., Soprovich, A., Al Sayah, F., Johnson, S.T., Majumdar, S.R., and Johnson, J.A. (2012). Applying the RE-AIM framework to Alberta's Caring for Diabetes Project: a protocol for a comprehensive evaluation of primary care quality improvement interventions. BMJ Open, 2, 1-9.

List of Acronyms

AHSUNC
Aboriginal Head Start in Urban and Northern Communities
CAPC
Community Action Program for Children
CBPR
Community-based Participatory Research
CCDP
Centre for Chronic Disease Prevention
CHP
Centre for Health Promotion
CIHR
Canadian Institutes of Health Research
CPNP
Canada Prenatal Nutrition Program
DECIPHer
Centre for Development and Evaluation of Complex Interventions for Public Health
FNIHB
First Nations and Inuit Health Branch
F/P/T
Federal/Provincial/Territorial
HPCDP
Health Promotion and Chronic Disease Prevention
IPPH
Institute of Population and Public Health
IS
Innovation Strategy
NHMRC
National Health and Medical Research Council
NIMHD
National Institute on Minority Health and Health Disparities
PAA
Program Alignment Architecture
PATHS
Promoting Alternative Thinking Strategies
PHF
Population Health Fund
PHIR
Population Health Intervention Research
RPP
Report on Plans and Priorities
SEAK
Socially-Emotionally Aware Kids
TBS
Treasury Board of Canada Secretariat
UKCRC
United Kingdom Clinical Research Collaboration

Endnotes

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: