Archived Evaluation of the Healthy Living and Chronic Disease Prevention – Multi-Sectoral Partnerships (MSP) Program 2014-15 to 2018-19

Prepared by
the Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

March 2020

Table of contents

• Executive summary
• Management Response and Action Plan
• 1.0 Evaluation Scope
• 2.0 Program Profile
• 3.0 Ongoing Importance of Supporting Health Promotion in Priority Areas
• 4.0 Evidence of Program Success
  • 4.1 Participation in the MSP Projects
  • 4.2 Improved Social or Physical Environments to Support Healthy Behaviours
  • 4.3 Improved Knowledge, Capacity or Skills on Healthy Behaviours
  • 4.4 Participants have Improved Health Behaviours
  • 4.5 Innovative Intervention and New Models of Public Health are Identified and Shared
• 5.0 Elements of an Innovative Program
• 6.0 Program Efficiency
  • 6.1 Program Spending
  • 6.2 Collection and Use of Performance Measurement Data
  • 6.3 Sex and Gender-Based Analysis Plus (SGBA+)
• 7.0 Conclusions
• 8.0 Recommendations
• Appendix 1 – Evaluation Scope and Approach
• Appendix 2 – Data Collection and Analysis Methods
• Appendix 3 – Logic Model

List of Figures and Tables

• Figure 1: Timeline of Significant Events which have Affected Program Activities
• Table 1: MSP Program Planned Budget, 2014-15 to 2018-19
• Table 2: Differences by Sex for Common Modifiable Risk Factors for Chronic Disease
• Table 3: Differences between Lowest and Highest income Levels by Sex for Selected Health Risk Behaviours
• Table 4: Differences between Indigenous and Non-Indigenous Populations by Sex for Selected Health Risk Behaviours
• Table 5: Number of MSP Projects by Type of Risk Factor Addressed
• Table 6: Number of MSP Projects According to the Primary Target Population
• Table 7: Demographic Breakdown of MSP Projects Based on Available Data
• Table 8: Planned and Actual Program Expenditures (2014-15 to 2018-19)

List of Acronyms

ISHLCD

Integrated Strategy on Healthy Living and Chronic Diseases

MSP

Multi-Sectoral Partnership

PHAC

Public Health Agency of Canada

SGBA+

Sex and Gender-Based Analysis Plus

TBS

Treasury Board of Canada Secretariat

WHO

World Health Organization

Executive Summary

The Healthy Living and Chronic Disease Prevention – Multi-Sectoral Partnerships (MSP) Program launched in 2013, with the aim of advancing innovative solutions to public health challenges. The Program is premised on multi-sectoral partnerships and provides co-funding to recipients in order to test and scale up the most promising primary prevention interventions that address common modifiable risk factors for chronic disease, particularly a lack of physical activity, unhealthy eating, and smoking.^{Footnote 1} By engaging with multiple sectors of society, partners can leverage knowledge, expertise, and resources to work towards the shared goal of producing better health outcomes for Canadians.

The objective of this evaluation was to review the relevance and performance of the MSP Program from April 2014 to December 2018. This evaluation also looked for innovative transfer payment models that are currently being used across the Government of Canada to explore best practices and lessons learned from their implementation.

The evaluation focused on MSP Program projects funded under the following contribution funding terms and conditions:

• Men's Health;
• Canadian Breast Cancer Initiative;
• Canadian Diabetes Strategy;
• Cancer;
• Cardiovascular Disease Program; and
• Healthy Living Fund.

It also examined projects, both approved and under development, that were created under the new Treasury Board of Canada Secretariat pilot on Generic Terms and Conditions, which was officially launched on April 1st, 2017. The evaluation did not review the relevance and performance of interventions tied to the Federal Tobacco Control Strategy, as these were examined as part of the Evaluation of the Federal Tobacco Control Strategy 2012-13 to 2015-16, that was approved in January 2017 and will be evaluated again in 2022-23.

This evaluation was conducted jointly with an audit of the MSP Program, as scheduled in both the Departmental Evaluation Plan and the Risk-Based Audit Plan. A separate report on audit findings has been developed following the normal internal audit process.

What we found

The MSP Program responds to an ongoing need to address common modifiable risk factors for chronic disease. This need is well recognized by PHAC and available statistics further demonstrate that there are significant differences in the prevalence of major chronic disease risk factors among certain segments of the Canadian population. As such, chronic disease prevention efforts continue to be aligned with federal priorities and PHAC's mission to promote and protect the health of Canadians through leadership, partnership, innovation, and action in public health.

To date, there are early indications of success stemming from program activities and outputs, in terms of number of individuals participating in funded projects, knowledge development relating to healthy behaviours, and evidence of progress towards behaviour change and improved health. At this early stage in the Program's life cycle, it has not yet identified and shared what works or does not work in terms of innovative interventions and new models.

The MSP Program is perceived as a leader across those within the Government of Canada using experimental program design and applying innovative funding models. In fact, the Program has been able to leverage $92 million in matched funding from other organizations, such as provincial, territorial, and local governments, health service organizations, school boards, and universities.

The overall delivery model of the Program includes a focus on establishing multi-sectoral partnerships, a continuous intake process, a requirement for matched funding from non-taxpayer sources, as well as use of innovative financing mechanisms, which has demonstrated the viability of the multi-sectoral approach. Additionally, a range of partners have shown themselves to be willing to contribute their own resources in pursuit of shared goals that respond to and address common modifiable risk factors for chronic disease. Most interviewed project applicants, funding recipients, and partners noted the need for a clearer indication of the types of projects the MSP Program is looking to fund, as well as the priority at-risk populations targeted by the Program.

The Program has conducted Sex- and Gender-Based Analysis Plus (SGBA+) analyses in critical areas, and is collecting information on populations targeted in individual projects. That said, the Program has encountered challenges in applying SGBA+ due to its broad scope of chronic disease risk factors and its current program design. There may be opportunities to see if there are gaps in populations targeted and potential mechanisms which can be used to bridge those gaps. More could be done to share SGBA+ information gathered to date from projects with internal and external stakeholders. At the same time, the Program could more clearly determine how it should respond to the needs of vulnerable populations in addressing common chronic disease risk factors in conjunction with similar programs in PHAC's Health Promotion and Chronic Disease Prevention Branch.

The MSP Program collects performance data through regular project monitoring of each individual funding agreement, as well as final reports and evaluations of individual projects to capture longer-term results of program investments. However, the program design makes the collection of data inherently difficult. Continuous intake means that the project portfolio includes a mix of projects that started at different points in time, and thus were established under different policy contexts and requirements (e.g., SGBA+). Further compounding the difficulty of putting together an overarching performance story is the fact that the Program targets multiple risk factors and intervention settings. This diversity in project focus makes it challenging to aggregate consistent project performance information over time. As such, there is a lack of reliable performance information, including performance targets and baseline data, to serve the Program's need for strategic information. Therefore, the MSP Program is unable to effectively present an overall performance story on its contribution to reducing common modifiable risk factors of chronic disease.

The systematic process of identifying, integrating, and sharing lessons learned on the Program's innovative approach and individual projects (i.e., best practices, areas of improvement) to present what is or isn't working, for whom, and in what context, would also benefit its efficiency and effectiveness, as well as its future development.

Recommendations

The evaluation findings discussed in this report has led to the following recommendations:

Recommendation 1:

Considering the complexities of its current design, the Program should determine how to integrate SGBA+ findings into the design of projects, in order to attain the strategic objectives of the Program.

Recommendation 2:

Revise current performance measurement practices to ensure that performance data is being collected consistently across all funded projects, to effectively measure program impact, and to ensure that expected program-level results are being appropriately tracked and communicated in order to guide future decision making on project selection.

Recommendation 3:

Introduce a systematic process to compile lessons learned on what is or isn't working, for whom, and in what context, and share these lessons learned with internal and external partners and stakeholders.

Management Response and Action Plan

Evaluation of the Healthy Living and Chronic Disease Prevention Multi-Sectoral Partnerships Program

Recommendations	Response	Action Plan	Deliverables	Expected Completion Date	Accountability	Resources
Recommendation as stated in the evaluation report	Identify whether program management agrees, agrees with conditions, or disagrees with the recommendation, and why	Identify what action(s) program management will take to address the recommendation	Identify key deliverables	Identify timeline for implementation of each deliverable	Identify Senior Management and Executive (DG and ADM level) accountable for the implementation of each deliverable	Describe the human and/or financial resources required to complete recommendation, including the source of resources (additional vs. existing budget)
Recommendation 1: Considering the complexities of the Program's current design, moving forward, the program should determine how to integrate SGBA+ findings into the design of projects, to enhance the strategic objectives of the program.	Agree	1.1 In consultation with the Centre for Grants and Contributions, we will develop and implement a Program Charter and Business Management Model for the Healthy Living and Chronic Disease Prevention - Multi-Sectoral Partnerships (MSP) Program, including a dedicated Appendix on Implementation and Monitoring of SGBA+.	1.1 SGBA+ findings integrated into relevant stages of MSP business processes, and incorporated into the Program Charter and Business Management Model.	December 2020	Vice President, Health Promotion and Chronic Disease Prevention	Existing resources
Recommendation 2: Revise current performance measurement practices to ensure that performance data is being collected consistently across all funded projects, to effectively measure program impact, and to ensure that expected program-level results are being appropriately tracked and communicated to guide future decision making on project selection.	Agree	2.1 In consultation with the Centre for Grants and Contributions, we will develop and implement a Program Charter and Business Management Model for the MSP Program that sets procedures for consistent collection of data and the effective measurement of program results and impacts.	2.1 Procedures for data collection and results measurement implemented and incorporated in Program Charter and Business Management Model.	December 2020	Vice President, Health Promotion and Chronic Disease Prevention	Existing resources
		2.2 In consultation with the Centre for Grants and Contributions, we will develop and implement a Program Guide for Applicants for the MSP Program that communicates program impacts and expectations for funded projects.	2.2 Project requirements for data, results and measurable impacts reflected in the Program Guide for Applicants.	December 2020
Recommendation 3: Introduce a systematic process to compile lessons learned on what is working and what is not working, for whom and in what context, when it comes to the Program’s current design and funded projects, and share these lessons learned with internal and external partners and stakeholders.	Agree	3.1 We will develop a knowledge transfer strategy that clarifies the process and expectations for capturing MSP program-level findings, strengthens third party evaluation and reporting requirements, expands existing knowledge transfer mechanisms, includes an approach for capturing and sharing project-level learnings at appropriate intervals, and clarifies governance and oversight expectations related to knowledge transfer. This knowledge transfer strategy will be incorporated and implemented through the Program Charter and Business Management Model for the MSP Program.	3.1 Knowledge transfer strategy implemented and incorporated into the Program Charter and Business Management Model.	December 2020	Vice President, Health Promotion and Chronic Disease Prevention	Existing resources
		3.2 We will develop and implement the relevant knowledge transfer expectations as identified under 3.1 in an MSP Program Guide for Applicants.	3.2 Expectations and approach to knowledge transfer explained in MSP Program Guide for applicants.	December 2020

1.0 Evaluation Scope

The objective of this evaluation was to review the relevance and performance of the Healthy Living and Chronic Disease Prevention – Multi-Sectoral Partnerships (MSP) Program from April 2014 to December 2018. This evaluation also looked for innovative transfer payment models currently being used across the Government of Canada, in order to explore best practices and lessons learned from their implementation. Information was collected through several means, including a literature review, a review of program documents and files, and 44 interviews with senior management, project managers and staff, federal partners, funding partners, funding recipients, and applicants. See Appendix 1 for more detail on how data was collected and analyzed, and Appendix 2 for more detail on the evaluation scope and approach.

The evaluation focused on MSP Program projects funded under the following contribution funding terms and conditions:

• Men's Health;
• Canadian Breast Cancer Initiative;
• Canadian Diabetes Strategy;
• Cancer;
• Cardiovascular Disease Program; and
• Healthy Living Fund.

It also examined projects, both approved and in development, created under the Treasury Board of Canada Secretariat pilot on Generic Terms and Conditions, which was officially launched on April 1^st, 2017. The evaluation did not review interventions tied to the Federal Tobacco Control Strategy, as these activities were examined as part of the Evaluation of the Federal Tobacco Control Strategy 2012-13 to 2015-16, which was approved in January 2017 and will be evaluated again in 2022-23.

2.0 Program Profile

Program Context

According to the World Health Organization (WHO), non-communicable diseases, commonly known as “chronic diseases” in Canada, are increasingly complex health challenges. In Canada, more than two-fifths of the population over the age of twenty is living with at least one major chronic disease, such as cancer, diabetes, cardiovascular disease, or chronic respiratory disease. Physical inactivity and sedentary behaviour, unhealthy eating, tobacco use, the problematic use of alcohol, as well as exposure to unhealthy built environments, are all common modifiable risk factors for chronic disease. Obesity is also considered to be a key driver of chronic disease in Canada, where one in three children are overweight or obese, and over one in four adults are obese.

Over 150,000 Canadians die annually from diseases that are preventable. Chronic diseases are also responsible for a high rate of morbidity, associated reductions in quality of life, and negative impacts on communities and the economy. Chronic diseases and other illnesses cost the Canadian economy $190 billion annually, including $122 billion in indirect income and productivity losses, and $68 billion in direct health care costs.^{Footnote 2}

In May 2012, the World Health Assembly endorsed the first global target to reduce, by 2025, premature mortality from non-communicable disease by 25 percent. Member states also came to an agreement on a comprehensive global monitoring framework. The 2030 Agenda for Sustainable Development increased this global target to one-third (33 per cent) by 2030.

In 2005, the Public Health Agency of Canada (PHAC) received $255 million for the first five years, and $67.3 million for every following year in ongoing funding, for the Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD), which represents PHAC's foundation in the areas of healthy living and chronic disease prevention. The aim of the ISHLCD framework was to ensure that Canada had an integrated approach to addressing major chronic diseases by focusing on common modifiable risk factors for chronic disease, as well as through complementary disease-specific work. The framework consisted of three pillars: promoting health, preventing chronic diseases by minimizing risk, and early detection and management of chronic diseases. The Centre for Chronic Disease Prevention and Health Equity (CCDPHE), within the Health Promotion and Chronic Disease Prevention (HPCDP) Branch, leads PHAC's chronic disease prevention activities. Within the CCDPHE, the Partnerships and Strategies Division (PSD) is directly responsible for administering the MSP Program.

Of the $67.3M in ongoing funding for the ISHLCD framework, approximately $20M per year is used to fund MSP Program activities. PHAC's MSP Program was launched in 2013, with the aim of advancing innovative solutions to public health challenges. The Program provides co-investment to partners with the goal of preventing chronic diseases, while reducing health inequalities among population groups where necessary.

Overall Program Approach

Traditional approaches to chronic disease prevention have not led to desired results at the population level, and often struggle to engage individuals in behaviour change interventions. Therefore, the Program sought new solutions for the complex challenge of preventing chronic diseases in Canada. It provided funding through PHAC contribution agreements to test and scale up the most promising primary prevention interventions (i.e., those that enable and change behaviour in ways that will positively affect health). Typically, these interventions emphasize increased physical activity, healthy eating, smoke-free living, and the creation of social and physical environments that support healthy behaviours. Project leaders have recognized that it is not enough to assume 'if they build it, people will come' or assume that if a target population is reached, the intervention will be adopted. Instead, the MSP Program projects tested theories of behaviour change rooted in research on behavioural insights, user-centred design, and choice architecture in order to influence, attract, inspire, or motivate populations to actively use or participate in the intervention. Additionally, by delivering interventions in ways that are culturally or socially appropriate for specific populations, they were expected to be more likely to succeed in promoting healthy behaviours, not only in the short term, but also in the long term after the project has finished.

The MSP Program worked across sectors to leverage investments, expertise, and ingenuity in order to develop and test integrated upstream approaches that help reduce common modifiable risk factors of chronic disease. Each project must include partnerships from both the private and not-for profit sectors, such as governmental organizations, academia, workplaces, industry, and communities. By engaging multiple sectors of society, partners can leverage knowledge, expertise, and resources to work towards the common goal of producing better health outcomes for Canadians. Project findings for both successes and failures were to be shared with internal partners and external stakeholders to help build the evidence base of what interventions work, for whom, and in what context. Ultimately the Program aimed to contribute to increasing healthier behaviours and preventing chronic diseases among Canadians by having multi-sectoral partners^{Footnote 3} adopt, scale up, and sustain effective healthy living policies and interventions. Appendix 3 provides a graphic that shows the program logic.

Key elements of the Program's use of contribution agreements to fund projects were as follows:

• Primary prevention projects must go beyond raising awareness to include a coordinated set of activities that enable and change behaviour in ways that will reduce the common risk factors for chronic disease (physical inactivity and sedentary behaviour, unhealthy eating, and smoking);
• Projects must test innovative ideas and approaches using a strong theoretical basis, or must scale up evidence-based approaches that have been shown to produce measurable behaviour change;
• A matched funding ratio of 1:1 from non-tax payer funded sources or private sector partners is required, although exceptions do exist as the Program focuses more on meeting a 1:1 ratio at a portfolio level;
• Projects must have “pay-for-performance” agreements, where payments are tied to accomplishing health outcomes and outputs that are specified in advance, jointly negotiated, and measureable; and
• As appropriate, some projects are structured to advance other “pay-for-performance” funding models, including social impact bonds and other social finance approaches.

Funding recipients were also required to develop and implement a rigorous evaluation of their intervention, and report on results on an annual basis according to their risk profile.
The Program used a continuous intake approach for project submissions. Rather than using a specific proposal deadline, the Program used an open solicitation process and a two-step review process for incoming proposals. The review process included Letters of Intent (LOI), followed by a full proposal. This approach generally involved a high degree of co-creation between program officials and project proponents.

As mentioned above, the Program generally required that project partners secure matched funding (identified as a ratio of 1:1) of financial (cash), and in-kind contributions from non-taxpayer funded sources and private sector partners to be eligible for program funding, which is consistent with the multi-sectoral approach. No other Government of Canada program exists that has a combined focus on continuous intake with a two-step review process, and a multi-sectoral approach specifically focusing on private sector matched funding (this applies only to the federal sector).

The MSP Program also made use of outcome-based funding models in its project design, as set out under the Treasury Board of Canada Secretariat's (TBS) new Generic Terms and Conditions for Innovative Uses of Transfer Payments Pilot. Such funding models focus on innovative financing tools, including prize/challenge, base plus premium, and micro-funding models, which transition from funding based on tasks and activities, to funding based on the achievement of concrete results and goals. Some risk is assumed, given that it is unknown if an intervention will 'work' with its intended population (in some cases, projects may be effective in promoting healthy behaviours, but may reach a different population than intended). As a complement to innovative financing, the Program also identified a series of physiological and psychological measures related to common modifiable risk factors for chronic disease (e.g., blood pressure, body mass index), which were used as performance indicators to tie payments to recipients' successful achievement of project results.

The MSP Program used innovative financial models before the formal approval of TBS' Generic Terms and Conditions, including signing a Social Impact Bond (SIB) with the Heart and Stroke Foundation in 2016 for the ACTIVATE project (Community Hypertension Prevention Initiative), and contributing to a prize/challenge approach implemented by the LIFT Philanthropy Partners in 2014 for The Play Exchange. Since TBS' Generic Terms and Conditions were launched in April 2017, the MSP Program has also approved three new projects using the Base Plus Premium Payment model.

Program timeline

A timeline of the Program's history is presented below (Figure 1), showing the sequence of significant events which have had an impact on program activities since 2013, notably the introduction of the Treasury Board Policy on Results in 2016. Other events include a requirement to incorporate Sex and Gender-Based Analysis Plus (SGBA+) in performance reporting, and a Directive on Experimentation in December 2016, as well as the introduction of the Generic Terms and Conditions in April 2017. The figure also presents examples of projects approved by the Program and their start dates. The diagram shows that the Program has had to respond to an evolving set of policies over the period of this evaluation. This has led to changes, for example, in project reporting requirements, which has affected program performance measurement data (discussed in more detail in Section 6.2).

Figure 1 - Text description

The Multi-Sectoral Partnerships (MSP) Program launched in 2013, followed by the start of a number of projects and activities including: the AirMiles Reward Program in 2013, The Play Exchange prize challenge initiative in 2014, the social impact bond signed with the Heart and Stroke Foundation in 2016 for the ACTIVATE project as well as the KidFood Nation project in the same year, the launch of the Social Enterprise Incubator and three projects funded under the Base Plus Premium Payment model in 2018. Lastly the Office of Audit and Evaluation undertook data collection for a Joint Engagement audit and evaluation of the Program in 2018 and 2019.

A series of significant events have had an impact on program activities, starting with the introduction of the 'plus' in Gender-Based Analysis Plus by Status of Women Canada in 2012 and then the introduction of the Treasury Board Policy on Results in 2016 which included a requirement to incorporate sex and gender-based analysis plus in performance reporting. Furthermore, the Treasury Board issued a Directive on Experimentation in December 2016 and issued the Generic Terms and Conditions in April 2017. The Federal, Provincial and Territorial Declaration on Public Sector Innovation was announced in November 2017. The Pan-Canadian Health Inequalities Reporting Initiative was launched in 2018. On May 31st of the same year, the Federal Provincial and Territorial document A Common Vision for Increasing Physical Activity and Reducing Sedentary Living in Canada was released. Lastly, in September 2018, Canada adopted the United Nations Political declaration of the third high-level meeting to control and prevent non-communicable diseases.

Since 2013, the MSP Program has invested approximately $112 million in support of 49 projects, which together have leveraged an additional $92 million in investment from private organizations and involved 400+ partners from across a range of sectors and industries.

Program Resources

The PSD's planned budget information, which includes MSP program funding, is presented below in Table 1 for the period of 2014-15 to 2018-19. Overall, the PSD had a budget of $107.7 million over the five-year period.

Table 1: PSD Planned Budget, 2014-15 to 2018-19

Year	Grants & Contributions (G&Cs)	Operation & Maintenance (O&M)	SalaryTable 1 - Footnote a	Total
Data Source: Estimated planned budget provided by Health Promotion and Chronic Disease Prevention Branch; Table 1 - Footnote a Salary includes Employee Benefit Program Return to Table 1 - Footnote a referrer Table 1 - Footnote b This figure includes approximately $5M provided through ParticipAction funding which is not administered through the MSP Program. Return to Table 1 - Footnote b referrer
2014-15	17,307,120	735,765	2,850,220	20,893,105
2015-16	14,067,871	690,954	2,847,469	17,606,294
2016-17	15,341,876	550,471	2,847,469	18,739,816
2017-18	17,707,053	666,279	3,089,798	21,463,130
2018-19	24,775,243Table 1 - Footnote b	1,015,035	3,204,822	28,995,100
Total	89,199,163	3,658,504	14,839,778	107,697,445

3.0 Ongoing Importance of Supporting Health Promotion in Priority Areas

The MSP Program responds to an ongoing need to address common modifiable risk factors for chronic disease, and it continues to be clearly aligned with federal and departmental priorities, roles, and responsibilities.

Continued Need for the Program

Evidence from both the document review and interviews all supported the need for the MSP Program. Several common risk factors (e.g., physical inactivity and sedentary behaviour, unhealthy eating, tobacco use, problematic alcohol use) responsible for increased risks of chronic disease morbidity and mortality later in life can be mitigated, and chronic disease prevented or its onset delayed.

According to 2016 data from the Canadian Chronic Disease Surveillance System (CCDSS), chronic diseases, such as cancer, diabetes, and cardiovascular disease, are the leading causes of death and reduced quality of life. Forty-four percent of adults aged 20+ have at least one in ten common chronic conditions. Furthermore, according to the Canadian Community Health Survey, 82.5 percent of Canadian adults 18 and over do not meet the Canadian physical activity guidelines, while only one-third of Canadians aged 12 or older report eating at least five servings of fruits and vegetables a day, an indicator of healthy eating. Finally, in 2015, there were 4.6 million tobacco users in Canada aged 15 and over (15% of the population), of which 3.9 million were cigarette smokers (Canadian Tobacco, Alcohol and Drugs Survey).

While problematic alcohol use is considered one of the top four risk factors when it comes to chronic disease, the MSP Program focused its project selection on physical inactivity, sedentary behaviour, unhealthy eating, and tobacco use. At the time of the evaluation, a scoping paper was being developed to help mitigate the risks and harms associated with the harmful use of alcohol in Canada.

Alignment with Government Priorities and Appropriateness of Roles and Responsibilities

Chronic disease prevention efforts continued to be aligned with federal priorities domestically and internationally, as well as with PHAC's mission to promote and protect the health of Canadians through leadership, partnership, innovation, and action in public health. Domestically, the 2015 Mandate Letter to the Minister of Health outlined the importance of building partnerships with provincial, territorial, and municipal governments in an effort to collectively improve health outcomes for Canadians. More recently, A Common Vision for Increasing Physical Activity and Reducing Sedentary Living in Canada: Let's Get Moving, a policy framework developed by federal, provincial, and territorial governments, was released on May 31, 2018. Also, the Government of Canada's five-year renewal of Canada's Tobacco Strategy in Budget 2018 provides funding for interventions that target smoking as a common risk factor for chronic diseases. These policy frameworks are also consistent with the Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD) focus on partnerships and multi-sectoral collaboration.

Furthermore, in December 2015, the Centre for Chronic Disease Prevention and Health Equity (CCDPHE) launched its new strategic plan, entitled “Improving Health Outcomes: A Paradigm Shift.” The main goal of the plan was to continue focusing on better health outcomes for all Canadians, mobilizing multi-sectoral and evidence-based action to promote healthy living and prevent chronic disease and injuries, with a vision for Canadians to live healthier and more productive lives. PHAC was therefore adapting its role to meet the changing environment by focusing on common modifiable risk factors for chronic disease using innovative solutions, and providing the co-investment needed to test and scale up the most promising interventions.

The Federal, Provincial and Territorial Declaration on Public Sector Innovation (November 2017) identified that innovation, experimentation, and openness require constant effort, even if they present risks. The Declaration also identified that the federal, provincial, and territorial Clerks and Cabinet Secretaries should commit to taking action, including experimenting and measuring results by identifying what works and what doesn't.

The MSP is also aligned with Canada's international commitments. At the Third United Nations General Assembly High Level Meeting on Non-Communicable Diseases in September 2018, global health leaders, including Canada's Minister of Health, adopted a political declaration, which included commitments to scale up efforts and further implement actions to control and prevent non-communicable diseases, including through the development of adequate national multi-sectoral responses.

Likewise, some internal and external key informants felt that a multi-sectoral approach was necessary to address chronic diseases in Canada, and should involve all levels of government, as well as other sectors (e.g., the private sector). Furthermore, by shifting their approach to focus on common modifiable risk factors for chronic disease, interviewees felt that the Program would have a more positive impact on reducing chronic disease morbidity and mortality.

As part of its strategy, the Program is also advancing a variety of partnership arrangements and funding models to promote more effective use of its grants and contributions investments, focused on achieving measurable results. The MSP Program is developing partnerships with the private and not-for-profit sectors, organizations within and outside the health sector, and other levels of government that support behavioural changes that aim to positively affect health.

Targeting At-Risk Population Groups

Different population groups show inequalities in rates of health risk behaviours. This suggests that people face different challenges in improving their health depending on various social, economic and demographic factors such as sex and gender, income, education, race and ethnicity, and where they live. Therefore, interventions to reduce chronic disease need to account for these different determinants of health and health inequalities in order to effectively meet the needs of diverse population groups.

Public health authorities, including PHAC and the WHO, have recognized that different population groups have unequal rates of health risk behaviours and exposure to unhealthy environments, leading to unequal rates of chronic diseases. These inequalities can be seen across populations as distinguished by sex and gender, age, cultural and racial backgrounds, First Nations, Inuit or Métis identity, income, education, as well as rural versus urban residency. These inequalities mean that strategies for promoting healthy behaviours and living environments among different population groups need to be tailored to address the diverse ways in which risk factors are manifested and experienced; there is no ‘one size fits all' model. This kind of thinking reflects the Government of Canada's commitment to apply ‘Sex- and Gender-Based Analysis Plus' (SGBA+) to policy and program development.^{Footnote 4}

The Pan-Canadian Health Inequalities Data Tool^{Footnote 5} displays information on health risk behaviours between population groups, as derived from Statistics Canada's Canadian Community Health Survey (2010-13).^{Footnote 6} The following table, drawn from the Data Tool, shows differences between females and males for health risk behaviours prioritized by the MSP Program.

Table 2: Differences by Sex for Common Modifiable Risk Factors for Chronic Disease

-	Females	Males
Source: Statistics Canada, Canadian Community Health Survey - Annual Component (2010-2013) as reported by the Pan-Canadian Health Inequalities Data Tool, 2017
Physical activity, active or moderately active (aged 18+)	49.8%	54.3%
Fruit and vegetable consumption, 5 or more times per day (aged 18+)	47.9%	33.5%
Smoking, daily or occasionally (aged 18+)	18.2%	24.5%

In the table presented above, males were shown to be more physically active than females, while females ate more fruits and vegetables and smoked less than males. These differences between the sexes were generally consistent when other population characteristics were added to the analysis, such as income levels, as shown in the table below.

Table 3: Differences between Lowest and Highest Income Levels by Sex for Selected Health Risk Behaviours

-	Lowest income level (quintile)		Highest income level (quintile)
Source: Statistics Canada, Canadian Community Health Survey - Annual Component (2010-2013) as reported by the Pan-Canadian Health Inequalities Data Tool, 2017
-	Females	Males	Females	Males
Physical activity, active or moderately active (aged 18+)	40.2%	46.4%	62.5%	64.2%
Fruit/vegetable consumption, 5 or more times per day (aged 18+)	40.1%	30.4%	54.8%	35.3%
Smoking, daily or occasionally (aged 18+)	23.6%	32.8%	13.6%	18.6%

Table 3 shows that a clear difference exists between the poorest and richest income groups regarding health risk behaviours and health promotion behaviours. Rates for each of these health risk behaviours improved progressively when comparing each income level to the next higher one. Furthermore, health promotion behaviours increased with each increase in income level. Similar differences in rates for these behaviours were observed when comparing population groups according to the level of educational achievement, wherein health risk behaviours were higher in populations with lower levels of education. When considering other population subgroups that matter for effectively engaging people on improving health, crucial differences existed in health risk behaviours. For instance, the table below displays figures for three Indigenous peoples in Canada, as compared to non-Indigenous populations.

Table 4: Differences between Indigenous and Non-Indigenous Populations by Sex for Selected Health Risk Behaviours

-	First Nations off reserve		Métis		Inuit		Non-Indigenous
-	Females	Males	Females	Males	Females	Males	Females	Males
Source: Statistics Canada, Canadian Community Health Survey - Annual Component (2010-2013) as reported by the Pan-Canadian Health Inequalities Data Tool, 2017 Table 4 - Footnote a To be interpreted with caution as the low numbers of people in this group mean that the percentage reported here may not be accurate. Return to Table 4 - Footnote a referrer
Physical activity, active or moderately active (aged 18+)	47.9%	57.5%	52.7%	56.2%	41.8%	42.5%	49.9%	54.3%
Fruit/vegetable consumption, 5 or more times per day (aged 18+)	35.6%	26.0%	39.8%	32.3%	26.9%	17.8%Table 4 - Footnote a	48.2%	33.7%
Smoking, daily or occasionally (aged 18+)	42.4%	41.8%	35.0%	39.9%	55.4%	56.7%	17.5%	23.9%

The rates for different Indigenous populations show differences in health risk behaviours among them, as well as when compared to non-Indigenous people; all Indigenous groups had a lower rate of fruit and vegetable consumption, as well as a higher rate of smoking, particularly for Inuit people, whereas Métis women and First Nations men showed the highest rates of regular physical activity across all the population groups. The historical, social, political, and economic contexts specific to Indigenous people in Canada are key to understanding these differences in health risk behaviours. The colonial legacy of residential schools, displacement of communities into remote settlements and reserves, and systemic discrimination are some factors leading to health inequities between First Nations, Métis, Inuit, and non-Indigenous people.^{Footnote 7}

All three of the preceding tables illustrate that there are varying relationships between population characteristics and health risk behaviours. This suggests that people face different challenges in improving their health, depending on education, income levels, and ethnicity. Examples of these challenges include access to information and resources, as well as supportive physical and social environments. As such, a variety of strategies are needed to engage and empower different population groups in reducing the risks of chronic disease.

4.0 Evidence of Program Success

Since 2014-15, the MSP Program has provided $112 million in funding to 49 projects that address one or more risk factors for chronic diseases. Of the 49 projects, 19 were completed by March 31st, 2019. Of the other 30 projects, nine are scheduled to be completed in 2019-20, ten in 2020-21, four each in 2021-22 and 2022-23, and the remaining three are scheduled to be completed in 2023-24. The largest groups of projects, by risk factor, focused on physical activity (14), such as the Right To Play's Promoting Life-skills in Aboriginal Youth (PLAY) program, or combined physical activity and healthy eating (14) (e.g., Community Food Centres Canada - Food Fit: Promoting Healthy Eating and Fitness in Low-Income Communities). The next largest group of projects (11) addressed multiple risk factors, in some combination of the modifiable risk factors. These projects include the Girls Action Foundation - Girls' Health and Wellness Project: Promoting Healthy Living, Healthy Weights and Tobacco Reduction among Girls. Table 5 below shows the number of projects by type of risk factor addressed.

Table 5: Number of MSP Projects by Type of Risk Factor Addressed

Type	Number	%
Source: Program documents Table 5 - Footnote a Although outside the scope of this evaluation, we have included tobacco-focused projects in Table 5 to show the full range of projects funded under the MSP Program to date. Return to Table 5 - Footnote a referrer
Physical Activity and Healthy Eating	14	29%
Physical Activity	14	29%
Multiple Risk Factors	11	22%
TobaccoFootnote a	6	12%
Built Environment	3	6%
Injury Prevention	1	2%
Total	49	100%

The evaluation found early indications of success in terms of who participated in the projects (numbers of participants and engagement of vulnerable populations), improved social and physical environments, knowledge development relating to healthy behaviours, progress towards behavioural change, and the identification and sharing of models for reducing risk factors for chronic diseases. While performance information is available, reporting varied according to the scope of the project and its stage of implementation, as well as project recipients' interpretation of information requirements. Therefore, considering that not all projects collected the same information, the statistics in the following sections vary according to the number of projects that collected specific information on the goals of the project, and MSP in general.

4.1 Participation in the MSP Projects

Program documents showed that 31 of the 49 funded projects had recorded a cumulative total of almost 1.9 million participants up to the end of 2017-18. Projects varied widely in scope, with the smallest reaching 105 participants (Girls Action Foundation) and the largest over 850,000 (Carrot Rewards) in 2017-18. This reflects the variety of activities across the projects, from intensive live workshops in a few communities to online smartphone apps available nationwide. Eight funding recipient interviewees mentioned receiving positive feedback from participants on the quality of different project activities, specifically that participants enjoyed using a variety of tools and that the projects had benefits for a wide variety of people (i.e., children, youth, adult workers and seniors, individuals and families, parents and caregivers).

As shown in Table 6, 69% (34 out of 49) of the projects specifically targeted a vulnerable population known to have a higher exposure to risk factors for chronic diseases (e.g., adult smokers, low income earners, Indigenous). Twelve of these projects specifically targeted children and youth, based on the concept that interventions at a young age can lower risk factors for developing chronic diseases later in life. Table 6 shows the breakdown of the number of projects according to the primary target population.

Table 6: Number of MSP Projects According to the Primary Target Population

Target population	# of projects (total=49)
Source: Program documents
Vulnerable populations	34
Adult smokers	2
Adults: inactivity or chronic diseases	5
Children / Youth	12
Children / Youth: intellectual disability	1
Children / Youth: obesity	2
Girls / Women	4
Indigenous	3
Low income	2
Newcomers to Canada	1
Seniors	2
Adults/general population	10
Health professionals	5

Some funding recipient key informants mentioned that different vulnerable groups benefited from projects that had intentionally targeted these specific populations, such as Indigenous children and youth, adults who are more at risk of developing chronic diseases, newcomers to Canada, and young athletes facing potential injuries. Some funding recipients also identified groups who were not well-served, specifically newcomers who have multiple jobs, and people with Type 2 diabetes in Indigenous communities.

It should be noted that the classification of projects by primary target population did not take into account additional factors noted by the projects (e.g., targeting adults in their workplaces or places of residence, environments that may expose people to risk factors like sedentary behaviour).

A demographic breakdown from a limited number of projects that collected SGBA+ performance data (between 3 and 12 depending on the specific demographic indicator, as per Table 7 below) and reported on some of the groups reached in 2016-17 and 2017-18, shows that females formed a clear majority of participants. Furthermore, almost one-quarter lived in a rural location (averaged over both years), approximately two-thirds of participants had at least some post-secondary education, and there was very little Indigenous representation.

Table 7: Demographic Breakdown of MSP Projects Based on Available Data

Demographic	2016/17	2017/18
Source: Program documents Table 7 - Footnote a n = denotes the number of projects that reported data on this item Return to Table 7 - Footnote a referrer Projects that reported reaching Indigenous participants were Boks, Flat Bay, and Right to Play Return to Table 7 - Footnote b referrer
Sex - Male	34.4% (n=12Table 7 - Footnote a)	36.7% (n=12)
Sex - Female	65.6% (n=12)	63.3% (n=12)
Location - Urban	64.6% (n=11)	82.3% (n=10)
Location - Rural	35.4% (n=11)	17.7% (n=10)
Education - No high school	3.4% (n=3)	2.0% (n=3)
Education - High school	29.2% (n=3)	28.3% (n=3)
Education - Post-secondary	67.4% (n=3)	69.7% (n=3)
IndigenousTable 7 - Footnote b	0.5% (n=3)	0.3% (n=3)
Non-Indigenous	99.5% (n=3)	99.7% (n=3)

4.2 Improved Social or Physical Environments to Support Healthy Behaviours

Performance data shows some early progress in making improvements in social or physical environments, which is a factor in sustaining healthy behaviours. Three projects in 2017-18 reported that approximately 72% of project participants indicated improved social environments, thanks to opportunities to connect with peers through activities that promoted physical activity and healthy eating. The Girls Action Foundation Learning Lab is a good example, where 105 girls across the country had the opportunity to bolster skills related to making healthy choices about food, drug use, and physical activity. Another was the Get BUSY project, where Boys and Girls clubs supported teens in becoming peer mentors to younger children in healthy eating and physical movement activities.

4.3 Improved Knowledge, Capacity, or Skills on Healthy Behaviours

According to program data, 71% of project participants from nine projects surveyed in 2017-18 demonstrated improved knowledge of chronic disease or risk, and related protective factors. For example, the Canadian Cardiovascular Harmonized National Guideline Endeavour (C-CHANGE) has produced an online tool for primary care practitioners treating patients who suffer from cardiovascular disease to become more knowledgeable and apply up to nine related clinical guidelines. It has also developed workshops intended to increase knowledge, influence attitudes, and change clinical practice with the uptake of best practices for cardiovascular disease prevention and management. Participants felt that the overall initiative should lead to improved cardiovascular management and outcomes by promoting greater awareness and adherence to the C-CHANGE guidelines in primary care. In another example, nearly 20,000 women used the Canadian Breast Cancer Foundation's online breast health planning tool that promotes awareness of cancer risk factors. Preliminary results of the Food Fit program indicate that 96% of participants (n=3,000) have learned something new about healthier eating and cooking. Though it does not directly demonstrate improved knowledge, one external key informant reported that 70,000 people have visited and used CHEO Research's online interactive portal called “Build our Best Day” and also mentioned that many are downloading the Canadian 24-Hour Movement Guidelines for Children and Youth (ages 5-17 years) from the Canadian Society for Exercise Physiology Web site.

4.4 Participants have Improved Health Behaviours

Ninety-six percent of project participants in 11 projects reported improvements in their health behaviours related to common risk and protective factors, due to their involvement in these projects. Key informants for two projects indicated that participants had maintained or increased healthy behaviours, such as increased vegetable consumption or walking to school, even after the project came to an end. This is illustrated by the AirMiles-YMCA Physical Activity Project, which found an increase (between 1.37 and 3.84 times) in weekly physical exercise between those participating in the Project, compared to those who did not. As well, the Special Olympic Canada project reported a decrease in social isolation and increased likelihood of lifelong participation in physical activity for individuals with intellectual disabilities. Another example where participants improved their health behaviours was the Food Fit project, which was launched at 30 community sites, with a total of 3,000 participants. Project documentation indicates that almost all participants learned something new about healthier eating and cooking. Furthermore, 72% of participants increased their daily number of servings of fruits and vegetables and their average daily numbers of steps taken. This led to most participants reporting an improvement in their physical (78%) and mental health (82%). It also led to partnerships with Six Nations Health Services (Ohsweken, Ontario) to support a Food Fit curriculum adaptation to co-create and co-brand a program for an Indigenous community.

4.5 Innovative Intervention and New Models of Public Health are Identified and Shared

One of the MSP Program's main objectives is to provide evidence of what does or does not work, for whom and in what contexts, through the chronic disease prevention initiatives funded by the MSP Program. It is expected that the Program will also provide information on the factors that facilitate or impede the effectiveness of a specific intervention. However, at this early stage in the Program's life cycle, there is no systematic evidence available on the identification and sharing of innovations and models across the 49 funded projects.

It should be noted that current thinking about chronic disease prevention, based on insights on human behaviour such as nudge theory, recognizes that it is not enough to offer or encourage people to choose “healthier options”, but that tangible and perceived barriers to change must be also recognized and addressed. Such barriers will differ between groups and individuals, given the differences in lived realities and social inequities. In keeping with the 2016 Directive on Experimentation, the Program has accepted that some ideas will fail while others may succeed, yet both outcomes are important for learning. It is understood that the evidence from funded projects would be used to inform the design of current and future projects, as well as to make program adjustments. In addition, evidence-based knowledge products (e.g., articles, infographics, webinars) may be developed for internal and external audiences.

A scan of 13 project evaluation reports revealed a number of lessons learned from the project experiences. For example, projects that engaged people with online tools found that individualized goal-setting, short-term rewards, and fresh content were important factors for recruiting and keeping users. Projects that emphasized community-level engagement highlighted the need for a variety of stakeholders to take ownership and agree to pool resources to collaborate on commonly-identified goals. Knowledgeable and adaptable facilitators were also seen as crucial for keeping participants engaged and helping them benefit from programming. Furthermore, there were some examples where activities had been sustained or expanded by recipient organizations for projects whose MSP funding has ended, such as:

• The Girl's Action Foundation published a girl's health and wellness facilitation guide, developed through the learning lab experience and covering topics such as group capacity building, healthy eating, physical activity, and health body image (addressing reduced tobacco and alcohol use);^{Footnote 8}
• Trottibus, the Canadian Cancer Society's walking school bus project, received financing from the Government of Québec to sustain and promote the project. This project has also started to expand into several communities in Ontario;
• Right To Play's Promoting Life-skills in Aboriginal Youth (PLAY) program has continued to expand across Canada, with a model of using community mentoring in sport as a positive influence for increasing physical activity, as well as promoting mental health, and school attendance and retention;
• The Canadian Cardiovascular Harmonized National Guideline Endeavour (C-CHANGE) continues to educate primary health practitioners on the use of nine clinical practice guidelines for preventing and managing cardiovascular diseases; and
• The ParticipACTION Learn to Play program has brought hundreds of youth sport stakeholder organizations together at the local level to build capacity for increasing physical literacy, as well as in forming and sustaining partner relationships, with indications that this will continue to have an effect past the end of the funding period.

5.0 Elements of an Innovative Program

The overall delivery model of the Program, which includes a focus on establishing multi-sectoral partnerships, a continuous intake process, a requirement for matched funding from non-taxpayer sources, as well as use of innovative financing mechanisms, has demonstrated the viability of the multi-sectoral approach and the willingness of partners to contribute their own resources in pursuit of shared goals.

According to a majority of federal key informants interviewed, the Program is considered to be a leader in experimenting with innovative G&Cs approaches within the Government of Canada, having developed a unique delivery approach and incorporated a continuous intake model for submissions, as well as a two-step review approach for proposals, including a co-creation component, and matched funding from non-taxpayer sources. At this time, no other programs within PHAC or other federal departments were found to have the same approach as the MSP Program.

The MSP Program has been experimenting with innovative financial models since 2014, well before the formal approval of the TBS Generic Terms and Conditions. To date, the Program has signed five agreements using this new TBS model, including signing a social impact bond with the Heart and Stroke Foundation in 2016 (the Activate Project, formerly known as the Community Hypertension Prevention Initiative), and using a prize/challenge approach in 2014 (the Play Exchange). Since TBS' Generic Terms and Conditions were launched in April 2017, the MSP Program has also approved three new projects using the Base Plus Premium Model. These projects also support the Government of Canada's overall approach to social innovation and social financing, as well as realizing the Agency's objective of encouraging innovation and effectiveness in public health programming.

As previously mentioned in Section 2.0: Program Profile, the Program also identified a series of physiological and psychological measures to be used to tie payments to the achievement of precise health-related measures. In the case of the Activate Project, reduced blood pressure levels for people at risk of developing high blood pressure (hypertension) is being used as a performance measure to tie payments to the successful achievement of precise health results aimed at reducing heart disease and stroke, as high blood pressure is an important risk factor for heart disease and stroke.

The approach to preventing chronic diseases has also shifted upstream and now focuses on common modifiable risk factors for chronic disease. By focusing on common risk factors, interviewees felt that the Program can better influence obesity rates and healthy lifestyle choices among children, which will then have an impact on reducing chronic disease morbidity and mortality later in adulthood. This corresponds strongly with positions taken by PHAC and the WHO that an integrated approach to address common chronic disease risk factors is a cost-effective way of improving health and lowering health care costs at a population level.^{Footnote 9}

The multi-sectoral partnerships approach allowed PHAC to gain some efficiency (e.g., through partnership requirements and leveraging matched funding), which is expected to translate into improved results. These measures have enabled the Program to participate in initiatives that have a greater scope and reach than would be the case if the Program was the sole funding partner. By securing other partners, it also increases the likelihood that a successful initiative will be able to sustain itself past the end of the contribution agreement.

The Program invested $112 million between April 2013 and December 2018 to promote multi-sectoral collaboration across various sectors to address chronic diseases through a requirement for matched funding. Projects were required to obtain matched funding in the form of cash or in-kind contributions from non-tax payer funded sources in the private sector, or from non-governmental organizations. Through this approach, the Program has expanded the number of partners involved in the 49 MSP-funded projects to over 400 corporations, small and medium-sized enterprises, charitable foundations, individual donors, as well as non-governmental organizations representing the industry, health, and community recreational sectors. Furthermore, by implementing the matched funding model, the MSP Program has leveraged $92 million (a 1:0.8 ratio) from private organizations up to now. It has also leveraged additional funding from other sources, such as provincial, territorial, and local governments, health service organizations, school boards, and universities.

These partnerships have also allowed national, regional, and local organizations striving to reach similar goals of chronic disease prevention to leverage each other's strengths, in order to achieve outcomes more efficiently. This includes extending project reach to new participants (e.g., Right to Play linking with First Nation community-based leaders, Sharing Dance collaboration with the national Revera chain of long-term care facilities), broadcasting key messages on healthy living (e.g., simultaneous launch of 24-hour movement guidelines for children in Canada and Australia), and attracting additional funders and sources of expertise (e.g., APPLE school project in Ft. McMurray attracting business investment).

The MSP Program approach also included a continuous intake element, with no call for proposal, or closing date. This was seen as a viable approach, with trade-offs between flexibility and operational efficiency, and financial management and planning. This model received positive feedback from all interviewees, especially applicants and recipients, as it provided more flexibility for the applicant to work with the Program in developing the submission, without it being tied to a specific proposal deadline. Although they generally took longer to develop, and thus likely affected the Program's ability to meet its previously established service standards of 45 days for the review of LOIs, the proposals were perceived to be of better quality and better aligned to MSP Program objectives.

Recipients also felt that the Program was successful in providing support and guidance through regular phone calls, emails, and teleconferences. They felt supported, encouraged, and well-resourced throughout their project.

On the other hand, most interviewed project applicants, funding recipients, and partners were aware of the Program's Web site, but they had difficulty finding relevant information on priorities, strategic directions, and target populations. They would have liked a clearer indication of the types of projects the MSP Program is looking to fund, as well as the priority at-risk populations that the Program aims to serve.

Some internal staff felt that a continuous intake approach had created an unpredictable workload, as proposals could be submitted at any time of the year, rather than working with a set schedule and clear deadlines. Staff also mentioned that the open solicitation process made budgeting more difficult, because project funding could be approved throughout the year, instead of at a specific period.

6.0 Program Efficiency

6.1 Program Spending

Over the five-year course of the evaluation, the Program spent most (approximately 94%) of its budget, despite individual variances in spending for each year.

Table 8 presents planned and actual program expenditures for the period between 2014-15 and 2018-19. During the evaluation period, the Program lapsed approximately $6 million, largely attributable to fiscal year 2015-16 and the Program's inability to commit new funding or negotiate new agreements during the election period. Lapsed funding in 2016-17 and 2018-19 were due to other Agency pressures, such as the need for immediate resources to support various efforts to address the opioid crisis, which delayed the signing of new agreements until the fall of 2018-19.

Table 8: Planned and Actual PSD Expenditures (2014-15 to 2018-19)

Year	Budget ($)				Expenditures ($)				Variance ($)	% Budget Spent
	G&Cs	O&M	Salary	TOTAL	G&Cs	O&M	Salary	TOTAL
Source: Financial data provided by the Office of the Chief Financial Officer.
2014-15	$17,307,120	$735,765	$2,850,220	$20,893,105	$16,927,504	$948,780	$2,971,816	$20,848,100	$45,005	99.8%
2015-16	$14,067,871	$690,954	$2,847,469	$17,606,294	$12,342,318	$594,947	$3,049,493	$15,986,758	$1,619,536	90.8%
2016-17	$15,341,876	$550,471	$2,847,469	$18,739,816	$13,570,585	$461,863	$2,596,955	$16,629,403	$2,110,413	88.7%
2017-18	$17,707,053	$666,279	$3,089,798	$21,463,130	$17,385,341	$510,273	$2,733,388	$20,629,002	$834,128	96.1%
2018-19	$24,775,243	$1,015,035	$3,204,822	$28,995,100	$23,661,225	$942,741	$2,903,664	$27,507,630	$3,374,461	88.4%
Total	$89,199,163	$3,658,504	$14,839,778	$107,697,443	$83,886,973	$3,458,604	$14,255,316	$101,600,893	$6,096,550	94.3%

6.2 Collection and Use of Performance Measurement Data

The MSP Program collected performance measurement information on each project for the period examined by the evaluation. There are, however, opportunities to improve the Program's performance measurement practices, including setting appropriate targets for each performance indicator and ensuring that data is collected consistently at the project level.

Furthermore, although the Program has collected information at the end of each project, the lessons learned about what is and what isn't working, for whom and in what context, is not being systematically compiled and shared with program stakeholders.

The Program has collected performance data on individual projects through regular monitoring of each individual project funding agreement, as well as final reports and evaluations of recipients to capture longer-term results of program investments. Many recipient key informants reported that they were monitoring the delivery of planned activities under their funded projects, including the contributions of their partners, a finding corroborated by internal interviewees. However, both groups felt that improvements were needed to make this data collection more relevant to the projects and to serve the Program's need for strategic information. Moreover, a few internal key informants stated that the project information received to date was based on different types of required reports that have been developed over time, but have been reviewed in isolation. This has resulted in fragmented information that does not clearly support program and senior management needs, or that could be communicated to external partners interested in proposing new projects. For example, there have not been any analyses of project gaps or overlaps that could inform program priorities for specific health risks. In addition, the collected data has not been able to:

• provide direction on how the Program can best reach high-risk populations (see sidebar);
• determine which types of projects and tools would be most appropriate to invest in; and
• identify lessons learned or best approaches for community-based projects.

The MSP Program has also collected 13 third-party evaluations for various projects, and has had articles published in journals on its multi-sectoral approach and many of its funded interventions, including Alliance Wellness, Carrot Rewards, SmartMoms, and UpNgo with ParticipACTION. A number of recipient key informants said that they had robust project evaluation processes in place, with some being led by third-party consultants, and mentioned holding regular meetings with partners to update them on the status of the projects. However, this evaluative feedback is not being systematically compiled or shared with others, including applicants, recipients, and contribution programs within PHAC and across the Government of Canada. Internal and external stakeholders, including applicants and recipients, expressed interest in lessons learned from the implementation of new funding models and intervention types. The Program is currently exploring options to develop a peer-to-peer network for program recipients, as well as funding partners and their networks.

6.3 Sex- and Gender-Based Analysis Plus (SGBA+)

The Program has conducted SGBA+ analyses in critical areas, and is collecting information on populations targeted in individual projects. Considering how SGBA+ could be incorporated into the design of the Program could better balance innovative and experimental interventions with the need to address common modifiable risk factors for chronic disease in the most vulnerable populations.

The Government of Canada is committed to using gender-based analysis plus in the development of policies, programs, and legislation in order to be more responsive to specific needs and circumstances. Strengthening the ability of PHAC's programs to address sex, gender, and diversity issues has the potential to improve performance and enhance impact, particularly by optimizing reach and the manner in which activities are delivered.

In an evaluation context, it is asked if the health issue addressed by a program differs systematically across population groups, if a program has been designed to address these differences, and if any impact has been demonstrated as a result. As noted in section 3.0, there is evidence that different population groups show inequalities in rates of health behaviours. The Program is aware of these differences as they have conducted SGBA+ analyses for healthy eating (December 2017), smoking (October 2018), and sedentary behaviour and physical inactivity (November 2018). More recently, applicants were asked to identify their project's target populations in their proposals. However, the current program design (e.g., the broad scope of projects funded, open intake process, matched funding requirement) makes it challenging to intentionally target specific populations that have been identified to be most at risk, or disproportionally impacted. For example, as mentioned previously, the Program is focused on addressing three common modifiable risk factors for chronic disease, each with their own identified set of vulnerable population groups. Moreover, with an open intake process that leads to a constant influx of applications throughout the year, it has been a challenge for the Program to take a step back to assess coverage and identify any gaps for vulnerable populations among the funded projects. Finally, up until now, the Program has taken a conscious approach not to target specific populations in order not to limit the potential breadth of proposals, and therefore did not direct any proposal requests, which is in line with the current program design. However, with the SGBA+ analyses completed, and with the information learned from the various projects completed and underway, there may be opportunities to see if there are gaps in target populations and what potential mechanisms could be used to bridge them.

7.0 Conclusions

The MSP Program responds to an ongoing need to address common modifiable risk factors for chronic disease, which is well-recognized by PHAC. Available statistics further portray that there are significant differences in the prevalence of major chronic disease risk factors among certain segments of the Canadian population. As such, chronic disease prevention efforts continue to be aligned with federal priorities and PHAC's mission to promote and protect the health of Canadians through leadership, partnership, innovation, and action in public health.

To date, there are early indications of success stemming from program activities and outputs, in terms of number of individuals participating in funded projects, knowledge development relating to healthy behaviours, and evidence of progress towards behaviour change and improved health. At this early stage in the Program's life cycle, it has not yet identified and shared what works or does not in terms of innovative interventions and new models.

This Program is perceived as a leader across the Government of Canada in using experimental program design and applying innovative funding models. In fact, the MSP Program has been able to leverage $92 million in matched funding from other organizations, such as provincial, territorial, and local governments, health service organizations, school boards, and universities.

The overall delivery model of the Program includes a focus on establishing multi-sectoral partnerships, a continuous intake process, a requirement for matched funding from non-taxpayer sources, as well as use of innovative financing mechanisms, which has demonstrated the viability of the multi-sectoral approach. Additionally, a range of partners have shown themselves to be willing to contribute their own resources in pursuit of shared goals that respond to and address common modifiable risk factors for chronic disease. Most interviewed project applicants, funding recipients, and partners noted the need for a clearer indication of the types of projects the MSP Program is looking to fund, as well as the priority at-risk populations it is targeting.

The Program has conducted Sex- and Gender-Based Analysis Plus (SGBA+) analyses in critical areas, and is collecting information on populations targeted in individual projects. That said, the Program has encountered challenges in applying SGBA+, due to the broad scope of chronic disease risk factors and its current program design. There may be opportunities to see if there are gaps in targeted populations and to determine the potential mechanisms that can bridge those gaps. More could be done to share SGBA+ information gathered to date from projects with internal and external stakeholders. At the same time, the Program could determine more clearly how it should respond to the needs of vulnerable populations in addressing common chronic disease risk factors, in conjunction with similar programs in PHAC's Health Promotion and Chronic Disease Prevention Branch.

The MSP Program collects performance data through regular project monitoring of each individual funding agreement, and through final reports and evaluations of individual projects to capture longer-term results of program investments. However, the program design makes the collection of data inherently difficult. Continuous intake means that the project portfolio includes a mix of projects that started at different points in time and were thus established under different policy contexts and requirements (e.g., SGBA+). Further compounding the difficulty of putting together an overarching performance story is the fact that the Program targets multiple risk factors and intervention settings. This diversity in project focus makes it challenging to aggregate consistent project performance information over time. Given this, there is a lack of reliable performance information, including performance targets and baseline data, to serve the Program's need for strategic information. As a result, the MSP Program is unable to effectively present an overall performance story on its contribution to reducing common modifiable risk factors of chronic disease.

The systematic process of identifying, integrating, and sharing lessons learned on the Program's innovative approach and individual projects (i.e., best practices, areas of improvement) to present what is and isn't working, for whom, and in what context, would also benefit its efficiency and effectiveness, as well as its future development.

8.0 Recommendations

The evaluation evidence discussed in this report led to the identification of the following recommendations.

Recommendation 1: Considering the complexities of the Program's current design, the Program should determine how to integrate SGBA+ findings into the design of projects, to attain the strategic objectives of the Program.

Balancing the complexity of an innovative approach to grants and contributions funding has its challenges, including the ability to direct the objectives of individual projects. The Program has conducted SGBA+ analyses in critical areas (healthy eating, smoking, and sedentary behaviours). Coupled with information gathered on populations that are targeted by current projects, the SGBA+ analyses would provide some guidance into any missing populations that should be targeted by the Program. Furthermore, the more individual projects advance, the more the Program will be able to discern the most effective approaches for different populations. This could also provide guidance to potential applicants of the types of projects the MSP Program is looking to fund, as well as any priority at-risk populations.

Recommendation 2: Revise current performance measurement practices to ensure that performance data is being collected consistently across all funded projects, to effectively measure program impact, and to ensure that expected program-level results are being appropriately tracked and communicated to guide future decision making on project selection.

The evaluation found performance measurement data collection requirements changed over time, considering the various policies that were implemented over the lifetime of this project. Therefore, there were inconsistencies at the project level in the collection of performance data, as well as gaps in collecting SGBA+ data. These then affect the ability of the Program to identify project impacts on targeted populations. There are opportunities to improve the MSP Program's performance measurement, which would allow it to more effectively present its overall performance story, including program impacts on reducing common modifiable risk factors of chronic disease. Improvements would also help to provide clearer information to support decisions, such as identifying priorities and opportunities for scaling up MSP Program-funded projects.

Recommendation 3: Introduce a systematic process to compile lessons learned on what is and isn't working, for whom, and in what context, and share these lessons learned with internal and external partners and stakeholders.

Internal and external stakeholders, including applicants and recipients, valued and expressed interest in identifying lessons learned and best approaches for community-based projects to learn from the implementation of new funding models and intervention types. By introducing a systematic process to compile and share lessons learned and best practices, the Program would be better able to provide evidence of what does or does not work, for whom, and in what contexts, as well as the factors that facilitate or impede the effectiveness of a specific intervention.

Appendix 1 – Evaluation Scope and Approach

The evaluation was scheduled in the Departmental Evaluation Plan 2018-19 to 2022-23, and is required under the Financial Administration Act. The objective of this evaluation was to review the relevance and performance of the Multi-Sectoral Partnerships Program from April 2014 to December 2018. This evaluation also looked for innovative transfer payment models that are currently being used across the Government of Canada to explore best practices and lessons learned from their implementation.

The evaluation focused on MSP Program projects funded under the following programs:

• Men's Health;
• Canadian Breast Cancer Initiative;
• Canadian Diabetes Strategy;
• Cancer;
• Cardiovascular Disease Program; and
• Healthy Living Fund.

It also examined projects, both approved and under development, that were created under the new Treasury Board of Canada Secretariat pilot on Generic Terms and Conditions, which was officially launched on April 1st, 2017. The evaluation did not review the relevance and performance of interventions tied to the Federal Tobacco Control Strategy, as these activities were examined as part of the Evaluation of the Federal Tobacco Control Strategy 2012-13 to 2015-16, that was approved in January 2017 and will be evaluated again in 2022-23. The following table shows the issues and questions addressed by the evaluation.

Evaluation Issues and Questions

Issue Area	Evaluation Questions
Relevance and Performance
1: Continued Need	1.1 What is the current and projected burden of chronic disease and how have PHAC activities adapted to changing needs related to chronic disease in Canada? 1.2 To what extent do chronic diseases differ systematically across population groups? Are there certain population groups that should be targeted?
2: Program alignment with federal and PHAC priorities and responsibilities	2.1 Since the previous evaluation in 2015, have there been any significant changes to Government of Canada and PHAC priorities and roles related to chronic disease prevention?
3: Performance – Effectiveness	3.1 To what extent has the MSP achieved its immediate, intermediate, and long-term expected outcomes? 3.2 Have the expected outcomes of the MSP had a different impact on different population groups? If yes, to what extent, and in what ways? 3.3 Have any unintended (beneficial or adverse) outcomes been produced?
4: Performance – Efficiency and Economy	4.1 To what extent is the MSP efficient and cost-effective? How have new funding mechanisms improved the efficiency and cost-effectiveness of the MSP to date? 4.2 Are there alternative and innovative ways to promote the health of Canadians and reduce the impact of chronic disease in Canada? 4.3 Does the MSP respond to the needs of its target population groups (e.g., outcomes in the Program's logic model)? If yes, how? 4.3 Does the MSP respond to the needs of its target population groups (e.g., outcomes in the Program's logic model)? If yes, how?

Appendix 2 – Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Data collection started in September 2018 and ended in March 2019. Data was analyzed by triangulating information gathered using the different methods listed below. The use of multiple lines of evidence and triangulation was intended to increase the reliability and credibility of the evaluation findings and conclusions.

Literature review:

A search of literature on chronic disease prevention interventions, including material from other innovative transfer payment models that are currently being used across the Government of Canada, was conducted to explore best practices and lessons learned from their implementation.

Program document and file review:

The evaluation reviewed a series of documents to inform findings related to relevance, effectiveness, and efficiency of the Program. Approximately 300 documents were reviewed.

Key informant interviews:

Key informant interviews were conducted to gather in-depth information related to the relevance, effectiveness, and efficiency of the Program. Interviews were conducted in a semi-structured manner, based on a predetermined questionnaire. A total of 44 interviews were conducted with senior management (n=2), project managers and staff (n=10), as well as federal partners (n=7), funding partners (n=2), funding recipients (n=19), and applicants (n=4). The list of interviewees was developed in consultation with Partnerships and Strategies Division (PSD). Once the list was developed, PSD and the Office of Audit and Evaluation (OAE) communicated with all individuals on the list, informing them of the review and the request for an interview.

Limitations and Mitigation Strategies:

Most evaluations face constraints that may have implications on the validity and reliability of evaluation findings and conclusions. The table below outlines the limitations encountered during the implementation of the methods selected for this evaluation. Also noted are the mitigation strategies put into place to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Limitations and Mitigation Strategies

Limitation	Impact	Mitigation Strategy
Limited primary data was collected from direct beneficiaries of funded activities.	Direct beneficiaries of funded community-based activities were not consulted as part of primary data collection. More interviews with these stakeholders could have provided greater insight into the performance of funded activities.	Although interviews with direct beneficiaries were not conducted, triangulation methods were used to corroborate key findings (literature and document reviews, as well as key informant interviews with other stakeholders).
Key informant interviews are retrospective in nature.	As interviews are retrospective in nature, this may lead to the provision of recent perspectives on past events. This can affect the validity of assessing activities or results relating to improvements in the program area.	Triangulation of other lines of evidence was used to substantiate or provide further information on data received from interviews.
Performance measurement data was limited and heavily reliant on participants' self-reported data. Furthermore, as data collection requirements changed over time, not all projects collected the same information.	A significant portion of project data presented is captured through self-reported data that participants provide. The total number of projects reporting on given statistics varied according to how many collected and reported on that information, which makes it much more difficult to rely on aggregated performance data to reach a conclusion on the achievement of program expected outcomes.	Triangulation of other lines of evidence was used to substantiate or provide further information on performance measurement data received from the Program.

Appendix 3 – MSP Program Logic Model

Innovation Strategy Logic Model - Text description

The Multi-Sectoral Partnerships (MSP) Program contributes to the Health Promotion and Chronic Disease Prevention Branch's logic model on Chronic Disease Prevention.

There are three activities under the logic model: develop partnerships and multi-sectoral collaborations to design and test innovative approaches and experiment with newer models for public health; use tested suites of financial tools to leverage resources, funds and expertise for interventions; provide leadership and engagement with national and international stakeholders.

This leads to the output of theory and/or evidence-based healthy living and chronic disease prevention interventions. Indicators are the ratio of funds leveraged for each intervention and the number of multisectoral partnerships brokered identified in contribution agreements.

The output of healthy living and chronic disease prevention interventions leads to the short-term outcome of target populations participating in the interventions. Indicators are the number of individuals participating in each intervention, identified by demographic characteristics such as gender, age, geographic location, etc.; and the percent of each target population participating.

It is anticipated that this leads to two medium-term outcomes. The first is that social and/or physical environments are improved to support ongoing healthy behaviours. Indictors are the percent of project participants who agree that they have improved access to environments that support healthy living, e.g. facilities, programs, parks, playgrounds, etc.; and the percent of project participants who report that their social environments are improved to support ongoing healthy behaviour. The second medium-term outcome is that project participants have the knowledge, capacity or skills to support ongoing healthy behaviours. Indictors are the percent of project participants demonstrating knowledge of chronic disease or risk and protective factors; and the percent of project participants demonstrating skills, ability or self-confidence to support healthy behaviours.

It is anticipated that the medium-term outcomes will lead to the long-term outcome that project participants have improved health behaviours, with the indicator of the percent of project participants that improve health behaviour related to common and protective risk factors, e.g. increased physical activity, decreased sedentary behaviour, increased fruit and vegetable consumption and smoking cessation. It is also anticipated that the preceding outcomes will contribute to the identification and sharing of innovative interventions and new models of public health. Indicators of this outcome are the percent of interventions demonstrating positive results; initiatives demonstrating or forecasting positive social return on investment (using a social return on investment ratio); and documented learnings on facilitators and barriers related to intervention success, e.g. from project evaluation results.

It is further anticipated that these two long-term outcomes will contribute to intervention evidence being incorporated across sectors into policies, programs and practices. The indicator is the nature of incorporation of evidence into policies, programs and practices. For the program to contribute to increased physical activity, decreased obesity and the prevention of chronic disease, policies and interventions for healthy living and chronic disease must be adapted, scaled up and sustained by multisectoral partners across Canada. Note: these three long-term outcomes link to a complimentary Evidence logic model applied by the Health Promotion and Chronic Disease Prevention Branch.

The long-term outcomes are anticipated to contribute to the departmental-level impacts: that Canadians have improved health behaviours, and that chronic diseases are prevented. Indicators for improved health behaviours are the percent increase in average minutes per day of physical activity among adults and among children and youth. Indicators for the prevention of chronic diseases are the percent increase in years lived in good health by seniors, the percent of adults who are obese, the percent of children and youth who are obese, and the rate of new diabetes cases among Canadians..