ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Healthy Living Program Component

 

Key Findings

This section discusses the relevance of the Healthy Living Program. As this is a formative evaluation, the issue of relevance was assessed in a limited fashion, focusing on the stakeholders’ identification of need and continued alignment with PHAC and the Government of Canada’s priorities. In addition, the issue of overlap and duplication across the Healthy Living components was examined. A more robust evaluation of relevance is recommended for the next stage of evaluation. Findings are derived from the document reviews, survey and the staff and stakeholder interviews.

3.1 Relevance

3.1.1 Continued Need

Evaluation Issue: 
Is there a continued need for the Healthy Living Components?

Healthy Living Program

The Healthy Living Program (HLP) is part of the Government of Canada’s contribution to the Integrated Pan-Canadian Healthy Living Strategy, an intersectoral framework for sustained collaborative action to reduce non-communicable disease in Canada. The Program is part of the Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD), the federal framework for public health activities, in keeping with the federal role, across the spectrum of health and chronic-disease and through disease-specific strategies on diabetes, cancer and cardiovascular disease.Footnote 2 As part of the ISHLCD, the HLP spans the six core public health functions, including:

  • surveillance;
  • knowledge development, exchange and dissemination;
  • community-based programming and community capacity-building;
  • public information;
  • leadership, coordination and strategic policy development; and
  • monitoring and evaluation.

The Healthy Living Program is comprised of four components, which are well aligned with PHAC’s Strategic Plan 2007-12. The four components are: 1) Healthy Living Fund; 2) Intersectoral Health Living Network; 3) Healthy Living Social Marketing; and 4) Healthy Living Knowledge Development and Exchange. The Healthy Living Roadmap describes how the Healthy Living Program will respond to and advance two of PHAC’s objectives as follows:

Table 5: PHAC’s Strategic Plan Linkage to the Healthy Living Program
PHAC Strategic Plan Objective Healthy Living Program Response
To anticipate and respond to the health needs of Canadians The Program serves as a vehicle for delivering a range of policy and programming priorities, including the promotion of physical activity, healthy eating, and their relationship to healthy weights (obesity). The Program assists in achieving goals and targets established by Federal-Provincial/Territorial Ministers of Health. The Program will remain flexible and will continue to look for opportunities to enhance the health of Canadians.
To ensure actions are supported by integrated information and knowledge functions The Healthy Living Program is committed to surveillance, research, knowledge translation and partnership components that directly support the development, dissemination and uptake of knowledge associated with PHAC priorities, including obesity, as it relates to physical activity, healthy eating and their relationship to healthy weights. It is supported by a strong performance measurement framework that will be used to inform policy and program development.
Healthy Living Fund
  • The Healthy Living Fund contributes directly to two of the six Functional Components of the ISHLCD: Community-based Programming and Monitoring and Evaluation.
  • National Stream:
    • Funding recipients agreed that there is a continued need for the National Stream of the Healthy Living Fund because the funding mechanism is instrumental in implementing and fostering national community-based projects.
    • This funding mechanism was deemed to be useful in developing community-based projects that lead to national programs.
    • It was reported that the idea of a national program that fosters community-based projects was deemed unique and that statistics demonstrate that there is a need to focus these projects on health promotion and in particular to target high-risk populations.
    • Unsuccessful funding applicants strongly believed that there is a need for the HLF in the future.
    • Overall demand for HLF, as assessed through the 50 funding submissions, was approximately $12.1 million per year or $24.0 million over 2 years; whereas, funding available within that stream was limited to $4.359 million over 18 months.
    • The continued need for the funding was strongly supported; however, the approval process and the lengthy delays associated with it negatively affected respondents’ answers about the continued need for the fund.
  • Regional Stream:
    • Interviews revealed support for the regional stream with concerns over the timeliness and complexity of establishing the Bilateral Agreements and associated contribution agreements.
    • It was noted by two interviewees that smaller provinces would likely benefit more from the “relatively small” amounts available in comparison to their larger counterparts.
    • From a program staff and management perspective, it was noted that the bilateral process was a new way of looking at issues of joint priority and interest. On the other hand, the bilateral process was seen as having “created an additional layer of complexity and politics to the file”.
    • One of the key issues is the timeliness of PHAC and the provinces and territories.  It was observed that the bilateral mechanism was not the most efficient way to obtain commitments.
    • Formal agreements created a greater comfort level from a federal perspective, creating a joint environmentFootnote 3 through the agreements of the provinces/territories.
    • The bilateral process has had a positive effect in some provinces of bringing P/T physical activity and health promotion ministries together to develop joint initiatives.
    • Interviewees perceived a need for the regional fund and have experienced frustration with the bilateral development process. Signature delays pertained in some instances to negotiation, language, matching of funds and the need for projects to be new ones.  Further delays were attributable to the sign-off of project proposals by PHAC and project agreements by some of the provinces and territories.

KD&E
  • KD&E contributes directly to three of the six Functional Components of the ISHLCD: Surveillance; Knowledge Development, Exchange and Dissemination; and Monitoring and Evaluation.
  • Internationally, Canada’s leadership and contribution to surveillance activities and guidelines in physical activity is both recognized and valued. This is evidenced by the recent evaluation of the Physical Activity Monitoring program (PAM) which found that it is “unique in its longitudinal focus on physical activity and sport in Canada and has a depth of information on the topic that has been recognized as among the best in the world by groups including the Centres for Disease Control.”Footnote 4 Leadership on physical activity guidelines through the hosting of the international consensus conference could lead to new international standards according to stakeholder interviews.
  • Stakeholders indicated a strong continued need for the KD&E component of the Healthy Living program. The component is seen as essential in its provision of the foundation knowledge about current status of healthy living and of research and best practices that must be considered when designing and implementing strategies to engage Canadians in healthy living behaviours.


IHLN
  • The IHLN is designed to contribute to two of the six functional components of the ISHLCD. These are Leadership, Coordination and Strategic Policy Development; and Monitoring and Evaluation.
  • After a lack of progress in the first 18 months of the Network, as described by stakeholders, and because a new structure was subsequently put in place, the Healthy Living Issue Group (HLIG) was formed and given responsibility for most of the activities originally to be assumed by the Network. The HLIG is the administrative arm of the IHLN with an aim to fostering two-way communication and to keep the IHLN informed on healthy living efforts taking place across the country. Its activities are seen as relevant and contributing to the functional areas described above. However, the larger Network, (i.e., the approximately 700 members who are not part of the HLIG), seems to be unaware or confused over the roles of the HLIG and the IHLN. In the Integrated Pan-Canadian Healthy Living Strategy (PCHLS) Blue Book, shared responsibility for the implementation of the PCHLS is a principle of the Strategy. As such, the responsibility for advancing the Strategy falls to all healthy living partners (federal government, P/Ts, NGOs, private sector and Aboriginal organizations). This was originally mandated to the IHLN. The HLIG now fosters collaboration in support of this responsibility taken on by all partners.
  • Only 54% (n=76) of IHLN membership survey respondents believe there is a continued need for the IHLN and 42% are neutral (see diagram 1). This survey finding is indicative of both the full survey results and interviews with stakeholders. Survey responses were highly ambivalent when it came to the IHLN, with many indicating a lack of knowledge about it, and many indicating that they were not sure why or if they were a member of the Network.

Figure 1: Survey Question 8 Response – IHLN Continued Need

Survey Question 8: There is a continued need for the IHLN (n=76)
Text Equivalent - Figure 1

Figure 1 depicts the response from Question 8 of the online-survey with the Intersectoral Healthy Living Network (IHLN) e-bulletin subscribers regarding a continued need for the IHLN. 54% agree there is a continued need for the IHLN, 42% were neutral and 4% disagree.

  • It should be noted, however, that just over 62% of P/T respondents indicated a continued need for the Network. This seems consistent with interview findings that report a ’better fit‘ for government (F-P/T) members in the IHLN because of specific accountabilities and goals put in place by F-P/T members toward the Integrated Pan-Canadian Healthy Living Strategy.
  • Results of interviews with those in leadership of the IHLN were mixed with some strongly agreeing that there is a continued need, while others indicated that the anticipated outcomes for the IHLN could be achieved in other ways which might be more effective. There was consistent input that while there may be a need for a network of some sort, the IHLN does not currently have the resources or structure to effectively carry out a full network function.
Social Marketing
  • The Social Marketing Program Component is designed to contribute to two of the six functional components of the ISHLCD. These include Public Information Leadership and Monitoring and Evaluation.
  • A review of both internal and external documents reveals that social marketing activities are valued internationally as making an important contribution to the overall strategy of increasing healthy living behaviours in populations in order to:
    • create a sense of movement or action regarding healthy living in Canada;
    • raise awareness of the importance of healthy living behaviours;
    • provide information, tools and suggestions to Canadians on how to adopt healthy living behaviours; and
    • to motivate Canadians to take steps to improve specific behaviours, i.e., healthy eating and physical activity.
  • Interviews with stakeholders provided consistent feedback that a well-designed and aligned social marketing component is needed and is an essential component of a multi-faceted approach to increasing healthy living behaviours in the population.
  • Interviews also revealed that there is a continued need for awareness and education initiatives that provide background messages to Canadians and that compliment other components of the HLP, as well as strategies being implemented by all sectors.
 

Conclusions:

The Healthy Living Program, as part of the Integrated Strategy on Healthy Living and Chronic Disease, is well-aligned with PHAC objectives.

Generally, stakeholders expressed strong support and identified a continued need for the Healthy Living Fund, KD&E and the Social Marketing components. Stakeholder opinion also indicated that relevance for the IHLN was somewhat mixed. This could be attributed to a lack of awareness of the Network.

 

3.1.2 Duplication within the Healthy Living Components

Evaluation Issue: 
Is there overlap and duplication across the Healthy Living Components?

  • The following planning documents indicate there is no overlap and duplication across the Healthy Living Components:
    • the detailed planning documents for the ISHLCD clearly describe the budgets and distinct activities of the four program components of the Healthy Living Program; and
    • the Healthy Living Roadmap (Strategic Plan) clearly identifies four core components and their relationship to the six core public health functions. Each functional component is clearly described and the Roadmap guides their implementation. Details of the implementation are summarized across three of the four components in the summary workplan.
  • While there is no current evidence of overlap, the Social Marketing component as a whole is absent from the Healthy Living workplans. The reason for this seems to be because it is managed by a separate communications unit within PHAC. This may limit the opportunity for alignment of components and creates a risk of duplication; therefore, it is recommended that workplans across all four components of the Healthy Living Program be developed collaboratively to ensure maximum impact and to avoid any future risk of duplication.
  • Interviewees strongly indicated that there was no duplication between the Healthy Living Program components but cited the value of coordination and alignment. More synergy and coordination between the four healthy living components is desired by stakeholders. For example, stakeholders would like to see knowledge and learnings from the KD&E component being considered in the development of social marketing campaigns, as well as being used to help determine priority interventions and contribute to best practices that might be implemented through the HLF.
 

Conclusions:

No evidence of duplication across the four Healthy Living Program components was found; however, a need for increased integration across the components has been identified.

 

3.1.3 Governance across Functional Components

Evaluation Issue:
Are there effective governance structures in place to ensure coordination and manage potential for duplication between the Healthy Living component and the Functional Components?

The Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD) coordination mechanism is composed of three main mechanisms, with ultimate accountability resting with the Branch Head. The governance structure depicted in the diagram below (diagram 2) was created in response to a recommendation from the first ISHLCD Implementation Review to adopt and fully implement a revised ISHLCD coordination mechanism.

The managers of each component of the Healthy Living Program, report to their directors who are part of the ISHLCD Coordinating Committee. These three in turn report to the Program Authority consisting of DGs.

In terms of coordination, Secretariat staff and the new ISHLCD Co-ordination Structure have been put in place, as recommended in the first ISHLCD Implementation Review.

The ISHLCD is coordinated and managed in a matrix model that runs across the 14 Program Components and six Functional Components.

Figure 2: ISHLCD Governance Structure

ISHLCD Working Groups
Text Equivalent - Figure 2

Figure 2 depicts the Integrated Strategy on Healthy Living and Chronic Disease (ISHLCD) governance structure. ISHLCD working groups are accountable to the ISHLCD Coordinators Network, who in turn are accountable to the ISHLCD Coordinators Committee, who report to the Program Authority who are ultimately accountable to the Branch Head. Reciprocal information sharing occurs at all levels.

Accountabilities are described in the ISHLCD Second Implementation Review and documented revisions based on various circumstances are noted, showing efforts to continue to improve governance functions.

Clear lines of reporting exist for the IHLN, KD&E, and the Healthy Living Fund. Structurally, these same clear lines exist for Social Marketing; however, this Component sits outside of the overall management of the Healthy Living Program, falling under Communications leadership. Issues of alignment emerge in this evaluation, and this appears to be a direct result of this governance gap.

 

Conclusions:

Effective governance structures are in place for the Healthy Living Program within the ISHLCD. These generally ensure coordination and manage the potential for duplication between the Healthy Living components and the Functional components.

Social Marketing is managed separately from the other components of the Healthy Living Program which has created concerns about alignment with the other components.

 

3.2 Success

This section provides a review of the success of the Program in terms of achievement of its activities, outputs and early outcomes. Findings are derived primarily from the document reviews the PERT reports, web information, and internal and external interviews.

3.2.1 Activities and Outputs

Evaluation Issue:
Have the key activities and outputs been implemented as planned for each of the four components?

Healthy Living Fund
  • National Stream:
    • Key activities and outputs are on track and being delivered through the national contribution agreements, for example: Canada's Physical Activity Report Card for Children and Youth;  Physical Activity and Healthy Eating — A Project for Children and Youth with Disabilities; and Enhancing the Effectiveness of PAR-Q and PARmed-X Screening for Physical Activity Participation.
  • Regional Stream:
    • From a bilateral agreement process, all Provinces and Territories have signed agreements in place, with the last addition being Québec in early 2009. As such, some projects are underway while others have yet to be initiated, delaying the process close to a year.
    • The time to obtain approval on the Bilateral Agreements varied widely from one province/territory to another. These were said to be attributable to some delays provincially and at the PHAC level.
    • Some Bilateral Agreement projects are in the early stages of inception. No indications were received or provided to the effect that planned activities and outcomes, once approval has/had been granted, were not or would not be on target.
  • The following table outlines the activities and outputs identified in the HLF logic model and the extent to which they were met for both the national and the regional streams. In general, most of the activities are occurring and the anticipated outputs appear to have been broadly delivered.
Table 6: Healthy Living Fund – Activities, Outputs and Key Implementation Results
Activity Outputs Key Implementation Results
Provision of funding to national and regional/community level projects
  • Funding solicitations (National and Regional)
  • Bilateral Agreements with P/Ts
  • Contribution Agreements with NGOs
National
  • National solicitation process was created including guide. Sent to over 500 potential respondents/ interested parties.
  • 50 national funding proposals were received and screened with a total combined funding request of $24.2 million.
  • 12 national projects selected  and 11 funded – $4.359 million for an 18 month period.
Regional
  • Solicitation process was created. Targeted approach.
  • 13 bilateral agreements in place.
  • Total of 38 projects have been funded to date (Alb 16, BC 2, Man 6, NB 1, Nfld 2, NWT 1, NS 4, Nun 1, On 1,  PEI 1, Yuk 3) and 5 in Québec (TBC), Sask. approval of 1 project is pending.
Engagement of national/regional stakeholders and provincial/territorial governments
  • Capacity building, addressing the determinants of healthy eating and physical activity
National
  • Capacity building activities reported in PERT and being monitored.
  • Projects were asked (through PERT) to indicate whether they used approaches to increase community capacity to influence policy over the Q1-Q2 projecting period.
  • The approach used most frequently – by 5 of the 6 projects that intended to influence policy – was the development of resources and tools for use by communities in policy analysis.
  • 3 projects developed a new group to work on policy.
  • 5 projects also identified other approaches including forming a committee to broaden communications, conducting research, holding workshops to bring together service providers and Aboriginal community members, holding consultations about what to include in a healthy eating supplement that provides guidance on influencing municipal recreation policies, and developing education modules that provide policy ideas for schools and communities.
Regional
  • Capacity building activities ongoing within the regional component, progress being tracked by regional staff and province. Progress will be tracked in PERT and PDCAS (results not available). In Québec, it has not been determined yet which questions of PERT will be used. However, the “Société de Gestion” plans to use their own tool. PHAC has sent them the PERT and they will explore how to integrate the questions.
Analysis, synthesis and dissemination of research evaluation results and lessons learned
  • Reports
  • Research
  • Database input (BP Portal)
National
  • Reports and research products where applicable are being prepared and progress is being monitored by program consultants and reported in PERT.
  • Funded recipients reported being on track with reports and research (slight delay on one project).
  • No mention of input to the BP Portal but input is planned based on project completion.
Regional
  • Report and research progress being tracked by regional staff and province and captured in the PERT.
  • No indication of database input on BP Portal.
KD&E
  • There appears to be strong alignment between activities and originally documented plans, workplans, activities, and outputs in the KD&E component of the HLP.
  • The KD&E component has built and nurtured key partnerships with the Canadian Fitness and Lifestyle Research Institute (CFLRI) and the Canadian Society for Exercise Physiology (CSEP) and managed them through directed contribution agreements. These NGO partners are delivering expected activities in surveillance, and in the scientific review of physical activity guidelines, in an effective manner that generally achieves stated objectives. Effective partnerships are also underway with international and domestic partners in both surveillance and physical activity guideline review initiatives.
  • Significant improvements in funding cycle timeliness of contributions to the CFLRI have resulted in marked enhancements in its ability to deliver expected results and plan future activities. The organization is recognized nationally and internationally for its CAN Play and PAM surveillance activities. Funding delays to CSEP have contributed to difficulties in delivering in a timely manner; however, activities and outputs are being achieved as planned.
  • Key activity reports for CFLRI and CSEP KD&E projects are available, and activities and outputs are documented. Most key activities and outputs are underway, continuing or completed. These include:
Table 7: Knowledge Development and Exchange – Activities, Outputs and Key Implementation Results
Activity Outputs Key Implementation Results
Provision of funding to support the scientific review of physical activity guidelines and surveillance monitoring systems
  • Exchange and disseminate key findings through academic and other published media, international and domestic consensus conferences
  • Directed funding to CFLRI for surveillance and monitoring related to F-P/T targets; and
  • Directed funding to CSEP for the scientific review of the PA Guidelines and Measurement.
Collaborate with domestic and international organizations to enhance knowledge and surveillance products
  • Maintenance of domestic and international collaborative partnerships
  • Facilitated international and domestic collaboration through exchange at international consensus conferences and domestic conferences; and
  • Contribution to the development of Statistics Canada surveys, research and other research and surveillance activities.
  • Enhanced knowledge and surveillance products
  • CFLRI delivery of PAM and CAN PLAY surveillance systems.
Identify, analyze, produce and disseminate best practices in physical activity and healthy living
  • Intervention research related to healthy living
  • Coordinated collaboration meetings with the Canadian Institute for Nutrition, Metabolism and Diabetes (CIHR) on intervention research relating to healthy living initiatives.
  • Enhanced best practice portal to provide physical activity and healthy living content
  • Identified, analyzed, produced and disseminated best practices in physical activity and healthy living through the best practices portal and through academic and other publications.
  • Technical papers, journal articles, policy briefs, popular articles
  • Exchange and dissemination of key findings to internal and external partners, including academic and other published media,  the PHAC website and other Agency publications.
IHLN
  • On its current course, the IHLN will achieve many of its promised activities and outputs, despite facing early challenges to implementation which delayed progress on the activities and outputs. The lack of a clear mandate for the IHLN, lack of resources for the secretariat and membership issues all contributed to early challenges such that work on most of the activities was stalled for up to 18 months.
  • Achievement of activities and outputs under the IHLN have improved dramatically in the past 18 months with significant credit given to PHAC staff by the Healthy Living Issue Group (HLIG) and Working Groups. The activities have been largely carried out within the past 12 months, with few results prior to that period of time. Activities include:
Table 8: Integrated Healthy Living Network – Activities, Outputs and Key Implementation Results
Activity Outputs Key Implementation Results
Facilitate communication among members and reinforce consistent messages on key issues Develop network communication mechanisms Production of 4 Healthy Living E-Bulletins and distribution to the IHLN membership
  • Health Disparities;
  • Children and Youth;
  • Populations in Isolated, Remote and Rural Areas; and
  • Aboriginal Communities.
Produce and distribute the Annual Report Report annually to Conferences of F-P/T Deputy Ministers and F-P/T Ministers of Health to track progress in reaching the targets outlined in the Strategy and to highlight key efforts Production of the first Annual Report on the Pan-Canadian Healthy Living Strategy was completed in 2007, with a second annual report underway.
Monitor and evaluate the work of the Healthy Living Issue Group (HLIG) Produce a Monitoring and Evaluation Plan for the HLIG and implementation strategies Three-year Monitoring and Evaluation Plan for the Healthy Living Issue Group completed (April 2007 to March 2010).
Facilitate joint work on health disparities Recommend a common set of health disparity indicators and a feasible approach to their implementation in the Canadian context Indicators Joint Working Group (HLIG and PHPEG) established in July 2008. Resources pooled between PHPEG and HLIG (November 2008). Background Report for Developing Health Inequalities
Disparities Indicators in Canada (August 2008) produced. Indicators of Health Inequalities Workshop was held on Jan. 13, 2009 in Ottawa.
Foster partnerships and opportunities for collaboration among sectors and across jurisdictions Define and develop the IHLN The gap in the work of the IHLN is to “define and develop the IHLN” itself. This is identified in the 2007-08 workplan for the Communications working group of the IHLN. The workplan lists an August/September 2007 timeline for a revised IHLN Terms of Reference, but it was determined that the best approach would be to focus the TOR on the administrative arm of the IHLN, the HLIG.
Identify joint project opportunities annually with multi-sectoral and multi-lateral collaboration In late fiscal 07/08, the IHLN was engaged and recommended that a collaborative project focused on the built environment be undertaken in fiscal 08/09. The project titled, ‘Planning the Built Environment to Promote Health: A Profile of Promising Practices in Canada and Abroad’ is currently being developed. A final report is due March 31, 2009.
Social Marketing
  • Social marketing activities to date have included the following:
    • funding of the ParticipACTION campaign;
    • one-time funding for the warehousing and distribution of Canada’s Physical Activity Guides;
    • Healthy Pregnancy Campaign (which included a physical activity component); and
    • the Concerned Children’s Advertiser initiative.
  • ParticipACTION was re-launched in Canada in 2007-08, and $1.6 million was allocated (has been reduced since) from the social marketing strategy budget, as part of a partnership with Sport Canada.
    • It was understood by PACTION that project funding would be reduced over time, on the assumption that they would be receiving dollars else where.
  • A new overall campaign or strategic plan for the Social Marketing Component of the Healthy Living Program has not yet been developed and implemented. It is reported by staff that this plan is in development within the Communications Unit; however, the lack of an overall plan during this two-year period has been identified as a major gap by stakeholders.
  • Planned activities and outputs are listed below:
Table 9: Social Marketing – Activities, Outputs and Key Implementation Results
Activity Outputs Key Implementation Results

Public education awareness campaigns on physical activity and healthy eating

  • Educational awareness campaigns

 

*Healthy Pregnancy Campaign which included a physical activity component  – A campaign media buy and web site.

  • Public information materials, e.g., posters, fact sheets, and website

*Concerned Children’s Advertisers – News Canada Articles were produced and distributed.

Coordinate the promotion of community opportunities designed to motivate and encourage Canadians to become involved in physical activity

  • Contribution Agreement with a social marketing agency (ParticiPACTION)

ParticipACTION funded and developed initial and secondary campaigns. Its awareness targets and funding targets were met or exceeded.

Provide Canadians with useful information about the importance of maintaining healthy behaviours

  • One-time printing and warehousing of Canada’s Physical Activity Guides

Concerned Children’s Advertisers – News Canada Articles were produced and distributed.

Healthy Pregnancy Campaign which included a physical activity component  – A campaign media buy and web site.

  • Data is available for the results of the ParticipACTION campaigns.
    • Two separate evaluations of the ParticipACTION campaigns were conducted, one by Angus Reid and one by the ParticipACTION Monitoring Unit of the CFLRI. Researchers conducting the evaluations of these campaigns have confirmed that they achieved awareness among Canadians.
 

Conclusions:

The outputs and activities for the Healthy Living Fund, KD&E and Social Marketing have been broadly achieved as planned.

On its current course, the IHLN will achieve many of its outputs but may not see progress in some of its defined key result areas.

 

3.2.2 Reach Targets

Evaluation Issue:
To what extent have the key reach targets been met for all the four components?
  • what extent have the following vulnerable populations been reached?
    • children and youth;
    • Aboriginal groups; and
    • isolated rural communities.
  • To what extent have partnerships been formed with P/Ts, private sector, NGOs?
3.2.2.1  Vulnerable Populations
Healthy Living Fund
  • National Stream
    • From a national perspective, the request for proposals called for projects in the following target groups, which differ from those identified in the evaluation issue:
      • children and youth;
      • older adults; and
      • at risk (isolated, remote and rural areas and Aboriginal communities).
    • A review of the project assessment review form indicated that projects were screened for compliance based on the aforementioned target groups.
    • Interviews with project management staff and funding recipients indicate target groups are generally being reached.
    • A review of the PERT data currently being collected suggests that target group information is not being collected in a manner that is consistent with the Request For Proposal’s definition of target population or the evaluation definition of target population (see above). The PERT groups are defined as:
      • individuals and communities facing specific risks;
      • practitioners, professionals and other service providers;
      • policy makers; and
      • general public.
    • Individual projects are also asked to describe the target population that they are trying to reach.
    • According to the PERT groupings, the following results are being achieved:
    • The total number of individuals reached in the first six months for all four of the target populations was over 40,000. By far the largest numbers reached were in the category of practitioners, professionals, and other service providers (32,415), which is consistent with the types of target populations of the projects, followed by individuals and communities facing specific risks (8,312).
    • It should further be noted that the PERT will be collecting information as to populations targeted by the project in a manner consistent with the HLF RFP. This information was not yet available in the PERT. This question was not identified to funded recipients from the outset of the project to enable NGOs to establish baseline information and to collect the data. The manner in which the questions will be asked will not likely provide the answers to the evaluation framework indicators created by PHAC as to the:
      • number/percentage of children and youth reached;
      • number/percentage of Aboriginal groups reached; and
      • number/percentage of isolated rural communities reached.
    • Project targets have not been set to guide interpretation of results.
  • Regional Stream
    • From a regional perspective, the notion of target populations was defined specific to each Bilateral Agreement, specifically in the Program Guidelines.
    • Based upon the understanding of the regional portion of the HLF and the review and oversight mechanisms in place at the federal and provincial level, it is expected that target populations, as identified within the respective proposals, should be reached when projects are initiated. This will need to be confirmed based on a review of the pertinent project files during the next evaluation steps for the ISHLCD. The information used for this review does not provide information on the extent these populations are being reached or whether they are consistent with the national HLF categories. Interviewees indicated that target populations, according to the HLF categories, are being reached and that in certain projects, other categories such as women and single parents, recent immigrants are also benefitting from the planned projects.
Table 10: Healthy Living Fund PERT Target Populations
Target population Number (percent of projects) HLF(n=11)
Individuals and communities facing specific risks 7 (64%)
Practitioners, professionals, and other service providers 11(100%)
Policy makers 6 (55%)
General public 7 (64%)
KD&E
  • KD&E activities target intermediaries, including researchers, policy makers, practitioners and others in the public that work with a variety of populations.
  • The development of current knowledge activities does provide a significant amount of surveillance and scientific data related to physical activity levels among the target population of children and youth through CAN PLAY and Canada’s Physical Activity Guides for Children and Youth.
  • This foundation work of scientifically reviewing guidelines will hopefully allow greater reach to vulnerable populations.
IHLN
  • The IHLN target audience is comprised of intermediaries described as federal, provincial and territorial governments, non-government organizations (NGOs), Aboriginal organizations and the private sector.
  • The HLIG has produced E-Bulletins on Health Disparities; Children and Youth; Populations in Isolated, Remote and Rural Areas; Aboriginal Communities; and Health Disparities. In addition it has facilitated the identification of a common set of health disparities indicators for Canada, in collaboration with PHPEG, through the Joint Task Group on Indicators
  • While not designed to specifically target vulnerable populations, the above activities seem to be a reasonable approach by the HLIG to increase the knowledge base of the membership and to enable their efforts with these populations.
Social Marketing
  • The Social Marketing Component of the HLP defines its target audience as “community organizations, and the general public.” Data found on reach to these targets is limited to that collected on the ParticipACTION campaign.
  • It will be extremely important going forward for social marketing activities to be aligned with target populations and to be measured for awareness among those populations.
  • The ParticipACTION campaign is a successful example of a campaign designed to identify and meet targets. It targeted parents of children aged seven to twelve, with a clear logic path to affect the behaviours of those most influencing children’s physical activity behaviours. The campaign achieved a high rate of recall Footnote 5 in two separate evaluations ranging from 21% to 48%, depending on the method of measurement used by the researchers.
 

Conclusions:

Strategies to reach target populations are not well integrated into the design of the four components; however, some specific initiatives are focused on the identified populations.

 


3.2.2.2  Partnerships
Healthy Living Fund
  • National Stream
    • Partnerships are being maintained and created as a result of the Healthy Living Fund. Projects have reported in the PERT that the 11 projects had 56 existing partnerships and over the course of the six-month period had created 45 new ones for a total of 101 (118 including KD&E).
    • In response to the evaluation framework question, it was difficult to assess to what extent partnerships have been formed with P/Ts, the private sector and NGOs because the PERT question does not ask for this information.  However, was possible to find results for not-for-profit and/or voluntary (46 partnerships), and one private sector partnership. The results also highlighted Aboriginal (6), and education partners (18).
  • Regional Stream
    • One of the funding conditions, under the regional HLF, was the ability of the organization to demonstrate current capacity for partnership and the ability to demonstrate intersectoral collaboration. The signing of the Bilateral Agreements with the Provinces and Territories is an indication of a partnership. Interviewees noted that partnerships had been enhanced at the P/T level but that the lengthy processes, in certain instances, created some damage to the relationship. Interviewees noted that partnerships with the private sector had not been enhanced by the bilateral process but that relationships with NGOs had somewhat increased.
KD&E
  • KD&E activities have led to a small number of partnerships, but they tend to be significant partnerships.
  • P/T partnerships have been enhanced through surveillance work of CFLRI. Both the PAM and CAN PLAY reports are produced nationally and by province/territory, and there is a high degree of satisfaction with both the process of collaboration and the outcomes among provincial/territorial stakeholders. Recently, the P/T Ministers responsible for physical activity endorsed and approved physical activity targets for children and youth.  This may not have been possible without the CAN PLAY surveillance work that produced the baseline data and made it possible to monitor progress against that baseline.
  • NGO partnerships have been enhanced nationally through the availability and ease of use of surveillance data. For example, the availability of surveillance data from CFLRI has provided the foundation for partnership activities with other NGOs who use the data for their initiatives, e.g. Active Healthy Kids Report Card.
  • International partnerships have been enhanced through the consensus conference activities and current work to recommend international guidelines for physical activity. Discussions have taken place on current Canadian physical activity guidelines, guideline gaps, (i.e., preschool, teens 15-19 years, Aboriginal peoples, persons with a disability, and pregnant women) and on international physical activity guideline initiatives in England, Australia, the U.S., and the WHO.
IHLN
  • The membership of the IHLN (i.e., current subscribers (700) to the Healthy Living E-) has all of its intermediary target audiences represented. There is no data on the numbers for each membership category; however, based on 124 survey responses, membership of each category is listed below in estimated order of numbers: federal government; NGO; individual; P/T government; municipal government; private sector; and Aboriginal organization.
  • The membership survey of the IHLN revealed that 35% (n=124) of NGO members disagreed that joint work with partners had been enhanced through the IHLN and that only 27% of respondents felt that the IHLN had enhanced joint work with partners. Furthermore, 67% of P/T respondents were neutral about whether joint work with partners had been facilitated and 78% were neutral about whether partnerships across jurisdictions had been fostered. Given that P/T partnerships are key to the success of the HLS, these findings merit attention.
  • Stakeholder interviews identified several key themes:
    • The IHLN is not resourced or structured to function as a network as originally planned. It may be positioned for effective communications, but it is too early to tell. As such, it is not likely to support the development of significant partnerships.
    • The HLIG component of the IHLN is beginning to develop strong partnerships through working group activities and through the Co-Chairs, but this does not hold true throughout the network and is limited in scope to a small number of members of the network, namely those represented on the HLIG and its working groups.
    • The HLIG component is more suited to F-P/T partnerships than to NGO partnerships since F-P/T partnerships have mechanisms for representation and NGOs do not. Individual NGOs are usually not accountable to the Coalitions or Networks they participate in, whereas there are two specific P/T mechanisms, i.e., sport/physical activity and the Pan-Canadian Public Network, responsible for physical activity, and there are direct federal accountabilities for the Healthy Living Program.
    • Early private sector representation was evident with the IHLN, but these partnerships were not maintained. Interviewees felt that this was primarily due to the slow rate of progress by the IHLN in its early days and an unclear mandate and role for the sector.
  • The HLIG currently has 20 active members, with federal, P/T, and NGO Co-Chairs, and this is currently the primary avenue for partnership in the IHLN.
Social Marketing
  • Stakeholders at the P/T level indicate a need for collaborative efforts to increase awareness and provide education to Canadians about healthy living and its benefits. However, there is no evidence of consultation being undertaken with these potential partners to align efforts and plan together to achieve integrated social marketing activities.
  • A contribution agreement between ParticipACTION and the Social Marketing component does exist although it is not considered a partnership. Given that ParticipACTION is the single largest recipient of contribution funds from the HLP, a formal partnership and a more strategic approach may prove to maximize the alignment with PHAC priorities for the healthy living social marketing activities. Partnerships have been developed through the Healthy Pregnancy Campaign and the Concerned Children’s Advertiser initiative.
 

Conclusions:

Partnerships are being formed as a result of the activities of the Healthy Living Fund, KD&E and some Social Marketing activities such as the Healthy Pregnancy Campaign and the Concerned Children’s Advertiser initiative. While the IHLN is currently not effectively developing partnerships, the HLIG is increasingly successful developing partners in the F-P/T arena.

 

3.2.3 Immediate Outcomes

Evaluation Issue:
To what extent has progress been made to date on the immediate outcomes?
Healthy Living Fund
  • National Stream
    • Interviewees noted that it was too early in the lifecycle to assess outcomes but that mechanisms are in place to track progress.
    • Information pertaining to outcomes is collected in a very limited fashion through PERT.
      • From an increased inter-organizational, inter- and intra-sectoral, multi-jurisdictional engagement, alignment and collaboration, PERT does collect information pertaining to the roles of partners and contributions to the project; however, the number and types of organizations reporting better levels of engagement and alignment are not collected. The number of partnerships and the results achieved through these partnerships is being tracked.
      • PERT does not directly collect information on the number of community groups reporting strengthened capacities to support healthy eating and physical activity. PERT does, however, seek to answer: “the number of individuals reached through the Healthy Living-funded project who reported increased levels of knowledge about Healthy Eating and Physical Activity.” The PERT does collect information on several other dimensions of capacity: target population involvement, partnership development, policy action, staff and volunteer training.
      • PERT should collect information pertaining to the number of individuals reached through “your Healthy Living-funded project who reported changes in behaviour related to Healthy Eating and Physical Activity.” It does not appear that direction was given to project staff as to the requirements to collect this information prior to launching their project as the following caveat pertains to a number of outcomes questions: “Please indicate the number by population, if this information was collected by project staff.” Collecting this type of information may not have been difficult if funding recipients had been informed of the need to collect this type of information from the outset.
      • The number and types of organizations reporting improved capacities for knowledge development and dissemination should be collected in PERT Q11b on the types of knowledge products developed and disseminated at project end.
      • Information on the increased reach to target populations to enhance healthy eating and physical activity should be available at project end. PERT Q16f states “Please indicate the number of individuals reached through your Healthy Living-funded project who reported increased levels of knowledge about Healthy Eating and Physical Activity. (Please indicate the number by population, if this information was collected by project staff).”
      • Information on increased knowledge dissemination, including reports and research findings should be collected through PERT Q11 and Q20 at the end of the project.  These questions will capture information pertaining to intended audience (including specific target population), number produced, number distributed, and whether or not you know of any other organizations that have used the product.
    • Interviewees were able to summarily identify and articulate to which of the immediate outcomes they thought their respective project was contributing.
  • Regional Stream
    • Information as to what extent has progress been made to date on the HLF (regional) immediate outcomes was not available for review.
    • Interviews with program management staff indicated that projects were underway and that both PHAC and the Provinces and Territories would be monitoring progress.
KD&E
KD&E activities have resulted in progress toward stated outcomes. This includes:
  • Updated Status on Canadian Physical Activity Levels  - Physical Activity Monitoring (PAM) and CAN PLAY
    • The Canadian Physical Activity Levels Among Youth (CAN PLAY) Survey provides an effective and objective means of studying current fitness and physical activity patterns of Canada's young people. CAN PLAY is the first nation-wide study of its kind. The study will be conducted annually until 2010. Data will be critical to developing long-term solutions and programs for a more active and healthier younger generation.
    • The Physical Activity Monitoring (PAM) program is a comprehensive system building on other large-scale national data collection to increase information about physical activity in Canada. It takes a comprehensive population health approach to understanding and changing physical activity levels for the population in the context of their everyday lives and is part of a bi-annual progress reporting to ministers. The program provides different data annually on a five-year cycle. Under the HLP, the PAM is renewed to 2010, with current plans extending to 2012.
  • Physical Activity Levels of Children and Youth and Approved F-P/T Targets
    • Enhanced objective data on physical activity levels for children and youth has led to approved F-P/T physical activity targets for children and youth. On May 22, 2008, federal, provincial and territorial Ministers responsible for Sport, Physical Activity, and Recreation agreed on a number of areas at their annual meeting in Victoria, BC. Ministers received an update from the CFLRI on current levels of physical activity among children, youth and adults in Canada. Having established a baseline level for physical activity of children and youth, using the CAN PLAY, Ministers set Canada’s first-ever national physical activity targets for children and youth aged five to 19.
  • Updated Scientific Evidence for Physical Activity Guidelines
    • With the financial support of the PHAC KD&E Component, the CSEP initiated a scientific review to assess whether the information in the physical activity guides might be out of date as a result of major advances in physical activity sciences since their release.
    • The International Consensus Conference, held to review the science behind Canada’s physical activity guidelines, was a major step in that process.
    • There was discussion on current Canadian physical activity guidelines, guideline gaps, i.e., preschool, teens 15-19 years, Aboriginal peoples, persons with a disability, and pregnant women, and on international physical activity guideline initiatives in England, Australia, the U.S., and the WHO. An independent international scientific panel was tasked with assessing the evidence from systematic reviews and developing consensus recommendations.
  • Best Practices
    • Enhanced physical activity content within the Best Practices Portal is demonstrated by 102 physical activity interventions listed and specific enhancements for Active Transportation.
    • Ninety-six healthy eating interventions and ten resources are also listed in the portal, indicating a good balance in emphasis.
IHLN
  • The IHLN had identified three immediate result areas as outcomes for the component. They are:
    • increased number of IHLN members;
    • increased sharing of information, knowledge and experiences among IHLN members across sectors and jurisdictions; and
    • increased awareness of the IHLN and its membership.
  • While there has been significant progress on activities and outputs in the past 12 to 18 months, these have not been matched with the desired outcomes, as noted in section 3.1. The reason for this appears to be the shift in mandate/direction that changed the focus from the IHLN doing work itself to work being done on behalf of the IHLN by the HLIG.
  • The IHLN membership has increased, going from zero to 700 in the last two years. This membership number is based on the number of subscribers to the E-Bulletin. There is currently no other indication of membership available.  The virtual Network membership represents all sectors with good federal, NGO and P/T representation; however, private sector involvement has been minimal.
  • The IHLN is currently described as a network of networks. The IHLN leadership describe how the member organizations will each promote the IHLN within their own memberships and networks. Since many of the members of the network have extensive memberships in the hundreds or thousands of stakeholders, it would be logical to conclude that if even one or two of the member organizations were well engaged in the IHLN, the membership would reflect this, and be significantly larger.
  • Evidence from interviews suggests that stakeholders (organizations with a primary mandate related to Healthy Living) of the IHLN are not always aware of the Network. For example, several NGO interviewees were not aware of the Network.
  • Survey results of members (see diagram 3) also indicated a lack of awareness of the IHLN and its activities. Only 27% (n=84) of respondents agreed that their awareness of the IHLN and its activities had increased over the past two years. Forty percent disagreed and 33% were neutral. Nearly half of respondents did not or could not complete the survey, often citing lack of knowledge about the IHLN. Twenty-two of twenty-five comments related to a lack of awareness about the IHLN.
Figure 3: Survey Question 8 Response – Awareness of the IHLN
Survey Question2: My awareness of the IHLN and its members has increased over the past two years (n=84)
Text Equivalent - Figure 3

Figure 3 depicts the response from Question 2 of the online-survey with the Intersectoral Healthy Living Network (IHLN) e-bulletin subscribers regarding awareness of the IHLN. 27% of IHLN e-bulletin subscribers agree that their awareness of the IHLN and its membership has increased over the past two years, 40% disagree and 33% are neutral.

  • Members are mostly unclear as to whether the IHLN has increased communication among members (only 32% agree it has) and, while internal stakeholders indicated that there may be a small amount of increased sharing of knowledge, information and experiences, these seem to be limited to F-P/T government relationships and not NGOs. Only one NGO survey indicated past or ongoing collaboration with the IHLN.
  • Overall, the benefits of increased sharing of information appear to extend only to members of the HLIG to date.
Social Marketing
  • The Social Marketing Component of the HLP has two immediate results areas (see Annex A):
    • increasing awareness about the importance of healthy living and maintenance of healthy weights among Canadians; and
    • increasing awareness about Canada’s Physical Activity Guides.
  • The ParticipACTION campaign supported by PHAC and Sport Canada had a high level of recall among target audience, i.e., 48%. Of those, 94% could recall the physical activity message according to an Angus Reid report. In a separate study by the ParticipACTION Monitoring Unit (PMU), housed within the CFLRI, about three-quarters of Canadians report having heard of ParticipACTION when prompted and just over half of Canadians report having heard a physical activity media message in the month prior to the interview. While it is outside the purview of the HLP formative evaluation to evaluate the outcomes of one of its funded projects, the indication is that the campaign has contributed to the Social Marketing desired outcome of “increasing awareness about the importance of healthy living...among Canadians.”
  • Data is collected by the CFLRI on Canadians awareness of Canada’s Physical Activity Guides. In a report about to be released, awareness of Canada’s Physical Activity Guides is at 27%, which is down significantly from a high of 37% in 2003.
  • There is no evidence to demonstrate increased awareness of healthy eating or healthy weights messages in Social Marketing.
  • It will be important to monitor all campaigns to allow proper assessment of their impact on Canadians. It is not currently possible to assess the extent to which other healthy living Social Marketing campaigns have been successful.
 

Conclusions:

While there is very limited information to date regarding the achievement of outcomes related to the Healthy Living Fund, interviewees report activities and outputs are aligned and likely to contribute to the identified outcomes.

Progress has been made on KD&E outcomes.

It appears that while care has been taken to achieve the outputs and activities identified for the IHLN, this has not resulted in the desired outcomes due to a change in direction and the enhanced role of the HLIG.

In Social Marketing, evidence exists of a successful ParticipACTION campaign, with areas of concern related to the drop in awareness of Canada’s Physical Activity Guides and no social marketing efforts related to healthy weights.

 

3.2.4 Unintended Outcomes

Evaluation Issue:
What have been the unintended outcomes as a result of the Healthy Living Program activities or each of the components, if any?
Healthy Living Fund
  • National Stream
    • Overall, interviewees felt it was too early to definitively identify unintended outcomes.
    • Some HLF stakeholders reported that their projects garnered greater media reach than anticipated.
    • One project highlighted that the adoption of Canada-wide guidelines was better than anticipated; whereas, another cited the creation of certain new partnerships as being unintended but welcome.
    • A significant portion of the interviewees reported that relationships with NGOs suffered as a result of PHAC processes, i.e., approval delays put the NGOs programs at risk and re-profiling challenges. The lengthy delays in screening the projects and ultimately obtaining approval were deemed to be unreasonable.
  • Regional Stream
    • From a regional perspective, it was also noted that it was too early to point to specific unintended outcomes.
    • It was highlighted that the bilateral process created stronger linkages between provincial/territorial sport/physical activity and health promotion ministries.
    • Some bilateral stakeholders also noted that more partnerships/linkages than expected were created to date.
    • A significant portion of the interviewees reported that relationships with provincial and territorial governments suffered as a result of PHAC’s and the Provinces and Territories lengthy process for sign-off on the Bilateral Agreements and project approvals.
KD&E

Progress on the activities of KD&E have led to some unintended outcomes which should be considered in order to enhance the component in future planning.

The success of the upcoming international consensus conference and subsequent exploration of new international guidelines for physical activity that are based on the outcomes of the conference could have significant future resource implications for KD&E.

The uptake of surveillance data generated by the CAN PLAY and PAM initiatives has created a large demand for additional data. The CFLRI is being impacted by the volume of requests and does not have the current capacity to meet them all. This capacity is an important part of the Exchange part of KD&E, but may require future adjustments to meet demand.

The CAN PLAY initiative has demonstrated both the efficacy and the desirability of objective data on children’s physical activity levels. Currently comparable data is not available for adults, but the need for this data is increasingly recognized. Stakeholders indicate that this may directly lead to new work to obtain this objective data.

Directed funding mechanisms are extremely helpful to carry out strategic or essential activities for the HLP. However, under the current system, both directed and contribution funds appear to be delivered through the same funding process, i.e., RFP contribution process. Stakeholders indicate that this creates the perception that organizations with directed funding are in a competitive position with contribution recipients, and they indicate that this can sometimes limit partnership opportunities.

NGO dependence on volunteers to carry out projects has at times been underestimated by some organizations. In these cases, it is doubtful that they will be able to take similar implementation approaches in the future. Volunteer effort may have to be replaced with paid resources.

IHLN
With a view to implementing the planned activities of the IHLN, the HLIG was created in 2006 to support, promote and advance the vision, goals and guiding principles of the Integrated Pan-Canadian Healthy Living Strategy (PCHLS). This approach has been successful in terms of the implementation of activities and their resulting outputs. Although this change has allowed for the expansion of the IHLN's membership, it has not created a fully engaged membership, one of the IHLN’s key outcomes.
Social Marketing

While the funding of ParticipACTION appears to have been successful in contributing to the Social Marketing outcomes, the manner in which the funding is delivered has led to some unintended outcomes.

Providing directed funds through the same contribution process as those projects in the Healthy Living Fund, creates the perception that organizations like ParticipACTION are in a competitive position with funding recipients, and stakeholders indicate that this can sometimes limit opportunities.

The Communications Unit does not normally manage contribution agreements, however, they are accountable for the Social Marketing budget. This had led to uncertainty around roles of management and accountability between the Communications Unit and the Healthy Living Unit, but they have currently been resolved.

 

Conclusions:

One unintended outcome was generated as a result of the Healthy Living Program’s activities. Lengthy approval processes, within PHAC and the Provinces and Territories generated a negative impact on both project delivery and relations between PHAC, the Provinces and Territories and funding recipients within the Healthy Living Fund (national and regional).

 

3.3 Design and Delivery

This section examines the management and decision-making structures, gaps, the use of lessons learned and the monitoring systems.

3.3.1 Gaps

Evaluation Issue:
Are there any overall key gaps in the Healthy Living Program or each of the components?
Overall
  • Interviewees noted that healthy eating and healthy weights are not well integrated into the design of the Healthy Living Program or into its components. Despite the fact that Health Canada is responsible for Healthy Eating and the Strategic Initiatives and Innovations Directorate for obesity, the relationship with those groups does not appear to ensure that healthy eating and healthy weights are fully integrated in program design.
Healthy Living Fund
  • National Stream
    • One of the main gaps pertains to the perception of the lack of continuity of the HLF. The future funding levels associated with the Healthy Living Fund appear unclear and subject to change in the future, thus creating some program uncertainty. While funding is considered to be ongoing, the ability to build upon promising project successes created through the Healthy Living Fund is hampered by funding uncertainty, as well as a lack of an overarching strategy/direction. Stability and direction would contribute to effecting long-term change or durable outcomes. There appears to be a mismatch between funding timeframes and longer term projects required to effect change in target populations.
    • A review of funded projects revealed that certain projects may have focused on certain target populations more than others and that a more deliberate funding approach may have generated a more strategic distribution and focus. The request for proposals or the project selection process did not appear to emphasize the need for projects in all target populations or components. It was noted by interviewees that the healthy eating and healthy weights components were under-represented, as well as projects focusing on Aboriginals. As such, there may be a requirement to be more strategic in focusing projects towards target populations.
  • HLF proposals sought a certain amount of funding but received less; requestors had to adjust their plans accordingly in terms of overall budget, as well as time remaining to complete the project (i.e., revised from 24 to 18 months).
  • Regional Stream
    • In terms of the development and signing process, the level of signature of the Bilateral Agreements was left to the P/Ts discretion. This approach was well received but interviewees felt that maybe the level of signature should be lower as it slowed the process.  A lot of time was spent on obtaining sign-off on the agreements; some Provinces and Territories were not a problem whereas some had to go to Cabinet. From a PHAC and Provinces and Territories perspective, it took a long time to achieve sign-off. “We had an agreement in principle in July and took until January to get sign-off.” The province has the power to sign-off on the Bilateral projects and PHAC can only be committed by the Minister. This lead one interviewee to note that “we are negotiating above our control and that the federal process does not guarantee approval.” A majority of interviewees expressed concern over the fact that the federal approval process exposed federal gaps. A number of directed comments were heard to this effect.
    • An interviewee noted that the Bilateral Agreements did not place emphasis on official language communities.
    • In terms of implementation of the associated projects, it is too early to identify implementation gaps, however, a few “pre-implementation” gaps were observed. The signing-off of projects took much longer than anticipated. Some interviewees were unable to identify what the hold-up had been, but indicated that it was more than one issue. The following reasons were cited by an interviewee: a federal election while negotiating the projects, federal approval; a change in Ministers; and provincial and territorial elections. It was also mentioned that PHAC had a new process for routing projects for approval and that this had in fact slowed down the process. The effects of this process were summarized by one interviewer who indicated that the Provinces and Territories could not get their projects underway because of the federal delays and that the projects were never protected by the Bilateral Agreements.
    • An interviewee highlighted that all guidelines, tools and templates should have been available from the beginning. This includes requirements/guidelines for communications related to the signing of Bilateral Agreements and the implementation of associated projects.
    • Another interviewee noted that there is a need to explore “the potential of engaging First Nations/Inuit directly given existing partnership processes in all three territories, self-government in Yukon, and program focus of vulnerable populations (Aboriginal people, rural/remote, children and youth).”
KD&E
  • The KD&E component of the Healthy Living Program is operating quite effectively within its planned activities, outputs and result areas.
  • The component seems well connected to other HL program components and ISHLCD functional components, through its surveillance partnership with the CFLRI PAM and CAN PLAY programs. However, under its broader objective to “increase the base of knowledge about physical activity, healthy eating and healthy weights that will inform stakeholders and support effective healthy living policies and Programs. KD&E and surveillance inform all that we do in policy development and program delivery” there appears to be a need to link learnings from the HLF and from best practices from within the healthy living community to provide strategic directions for both policy and program.
  • Stakeholders have indicated that they would benefit from leadership that shares knowledge about:
    • What have we learned from the HL funded projects that we can build on in Canada?
    • What interventions are most effective in changing behaviours in vulnerable populations?
    • What new and innovative strategies are being undertaken internationally and how could they be applied here in Canada?
    • What policies are leading to substantive change for HL?
  • As the KD&E component moves forward it may choose to examine how it can contribute to filling this gap and move from the provision of knowledge to shaping strategy based on knowledge.
IHLN
The IHLN has faced challenges already noted in this report. Three major gaps have been identified in this evaluation, and these should be addressed to ensure progress against the components’ outcomes:
  • The network function of the IHLN – particularly the NGO and private sector engagement – mandated for the IHLN has not taken place among the full membership. The HLIG was created as the administrative arm of the IHLN as a result of efforts related to the 2005 creation of the Pan-Canadian Public Health Network. This group has been very successful recently in carrying out the work identified for this component. However, there was a great deal of confusion exhibited in stakeholder interviews between the HLIG and its role; and the IHLN and its role. It will be difficult to engage members in the Network without a clear purpose to encourage and clarify their involvement.
  • The IHLN was not assigned resources to implement its original mandate. The mandate was redefined in 2006 under the creation of the HLIG and a number of activities have been implemented through the HLIG.
  • The membership is not engaged in the IHLN. Engagement is limited to the HLIG which is approximately 3% of the current membership. Many are not aware they are members and most are ambivalent about the value of the IHLN.
Social Marketing
There are two significant gaps that are apparent after reviewing data and stakeholder input about the social marketing component:
  • Currently the Communications Unit of PHAC does not have the staff capacity to manage a contribution agreement process. Although the HL Unit has the expertise in managing contribution agreements, they do not have authority over the budget. Discussions will need to take place between both groups to ensure an effective process for managing the future social marketing contribution agreements is maintained.
  • The Communications Unit (which has the lead) and the Healthy Living Unit are not carrying out joint strategic planning to ensure full and seamless linkage of social marketing with healthy living objectives.
 

Conclusions:

The key gap is the exclusion of healthy eating and healthy weights in the overall design of the Healthy Living Program.

The Fund gaps include: the lack of perceived funding continuity; the need to focus on a specific target population; and a more efficient process to enhance timing for approvals and sign-offs.

KD&E’s lack of knowledge exchange and transfer process, IHLN’s lack of clarification between the HLIG and the IHLN and Social Marketing’s lack of joint planning with the Healthy Living Unit are key gaps identified.

 

3.3.2 Lessons Learned

Evaluation Issue:
What have been the overall key lessons learned to date from the Healthy Living Program and each of the components?
Overall
  • No significant lessons learned were identified to date for the Program overall.
Healthy Living Fund
  • National Stream
    • Interviewees noted that funding approval timing is critical, especially for school programs or weather/seasonal dependant programs. As such the solicitation and approval lead times need to be carefully thought out. One interviewee noted that if it takes a year to obtain approval, then they should launch the process that much earlier. Some projects noted that it is a challenge to get in-kind support from volunteers. One project reported that it was able to take advantage of new technologies to reach its audience, as these technologies matured, e.g., electronic communities and webinars.
    • Projects were asked within the PERT to identify and describe any lessons learned about reaching their target populations from which other projects could benefit. Six of the 11 projects identified some lessons learned. Each of the projects tended to report on different lessons learned for reaching target populations. The types of lessons learned identified by these six projects dealt with the following:
      • allocating enough time;
      • consulting with and working with partners;
      • using various methods to reach youth;
      • identifying individuals and/or groups who know their communities well; and,
      • communicating materials in the appropriate format.
    • Other evidence includes lessons learned about attracting media coverage.
  • Regional Stream
    • In terms of the development and signing process, the level of commitment and engagement applied to the negotiations was very high and purposeful and was carried out in person and allowed for face-to-face discussions with health and recreation ministries at the P/T level. Flexibility in allowing for the use of provincial funds to be redeployed from existing budgets to support Bilateral Agreements was a good initiative and was supportive of the realities in the provinces. Provinces/territories were allowed to decide the level of signature for their agreements (director and above). This approach was well received but the level of signature should be lower as it slowed the process.
    • In terms of the implementation of the associated projects, it is too early in the process to draw lessons learned. Projects/regions should be able to report in due course. It was however noted that quicker turn around for approvals from PHAC, matching rate of P/Ts, would have been helpful. Interviewees noted that getting projects approved in a timely fashion, at the federal level, has been a significant challenge. The following comments articulate some of the comments received from interviewees: “Approve them when they are to start.” “You better be able to deliver on what you said.” “Province was ready to walk away!!!!”
KD&E
  • The funding model for the CFLRI should be enhanced to ensure the success of the program. The Healthy Living Program has had difficulty dispersing funds for contribution agreements in a timely manner, with one exception. In 2008, the CFLRI received confirmation of its funding allocation at the beginning of the fiscal year in April. The organization has praised PHAC for this improvement in process. The timely confirmation has allowed the organization to carry out its activities more efficiently and has enabled it to look forward and plan effective means by which it can leverage its current work to further benefit KD&E activities, both domestically and internationally.
  • This improved funding process and the manner in which it was directed through the CFLRI is a best practice that should be implemented wherever possible and wherever consistent with Healthy Living Program objectives.
IHLN
  • None identified.
Social Marketing
  • Investments in initiatives like ParticipACTION can achieve results for the healthy living message in a relatively short period of time. These types of contribution agreements can help achieve HL social marketing outcomes but must be managed strategically and effectively.
 

Conclusions:

Investments can be leveraged effectively and the timely release of funding is critical to ensuring the success of projects.

 

3.3.3 Monitoring System

Evaluation Issue:
Is the current monitoring system for the Healthy Living Program components effective?
Healthy Living Fund
  • National Stream
    • All 11 projects are currently reporting through the PERT. The data provided is reviewed by the HLF project officer and further reviewed by the HL evaluation manager. Funding recipients are also required to submit evaluation plans describing how they will collect PERT data and to expand on additional evaluation plans for their projects.   Discrepancies, questions or observations found in PERT are brought forward to the Project by the project officer. The process appears to be working well with good responsiveness from the funding recipients as well as the funder.
    • The relevance and usefulnessFootnote 6 of a number of the PERT questions is not clear to funding recipients. The following questions highlight this point:
      • Over the last six months, please indicate how many members of your target population have contributed to the management and/or delivery of this project?
      • Over the past year, has your project monitored changes in your target population’s access to health services?
      • Please provide examples from the past six months of your attempts to get media attention for your project?
    • From an overarching perspective, outputs and outcomes within the PERT have not been measured yet. These outputs are linked to the goals and objectives of the project and are reviewed for consistency by the PHAC project officer. These however do not appear to be reported or tracked within PERT against project plans or targets.
    • Interviewees noted that the level of effort to complete project reporting, e.g., PERT, to be about right, i.e., medium to high.
  • Regional Stream
    • Project monitoring for the regional HLF will be completed through PERT. Project results will be reviewed and monitored at the regional level as well as by the province. Québec is not planning on using the PERT as the “Société de Gestion” plans to use its own tool. Québec will explore how to integrate PERT questions but has not yet determined which PERT questions it will use. A crosswalk will be developed to facilitate a PERT roll-up of the Bilateral Agreement project data.


KD&E
  • The major components of the KD&E Component, i.e., the Physical Activity Guidelines, CAN PLAY and PAM, are delivered via NGO partners and monitored through the PERT reporting system as discussed above
IHLN
  • The IHLN has developed a draft Three-Year Monitoring and Evaluation Plan for the HLIG. There is no specific evaluation plan for the IHLN as a Network, but many of the indicators in the plan relate to the Network. The draft plan appears to be well formulated with a logic based results framework which articulates expected results, performance measurements, and assumptions and risk indicators.
  • The workplan calls for the implementation of the monitoring plan in 2008-09 and its effectiveness will be determined at that time.
Social Marketing
  • The Social Marketing component does not appear to have a consistent monitoring plan in place for its campaigns and activities except for ParticipACTION.
 

Conclusions:

The PERT has been implemented to monitor the Healthy Living Fund and KD&E projects and appears relatively effective with regular monitoring.  Relevancy issues exist with some of the PERT questions.

IHLN does not have a monitoring plan in place; however, the HLIG does and will be implementing it this year.

Social marketing is being monitored in an ad hoc manner with effective monitoring of ParticiPACTION.

 

3.3.4 Program Design and Operating Processes

Evaluation Issue:
Are the program design and operating process efficient and effective?
Process Review
This section describes the steps in the management of the HLF. Below, we outline the key tasks, the strengths and weaknesses of each step.
  • National Stream
Table 11: Healthy Living Fund - Funding Process Strengths and Weaknesses
Step Key Tasks Strengths Weaknesses
1. Establish Fund Level
  • Notional indication of budget
  • Allocation of notional funding across the four components
  • Staff prepare materials based on notional amount
  • Management  finalizes funding level
  • Secured money through a submission process for fixed period
  • Perception that the HLF is an initiative rather than a program
2. Generate Guidelines
  • Review guidelines
  • Draft new guidelines
  • Seek Management approval
  • Internal consultations
  • Revised and adapted guidelines to meet the new HLF requirements
  • Guidelines do not portray critical approval path or timelines
3. Call for Proposals
  • Draft Call Letter/RFP
  • Management approval
  • Distribute call letter/RFP
  • Used a broad band approach of  distributing call letters using the attendee list from prior events, the Healthy Living Unit website and Email
  • Most reported that the RFP was clear; there were some  expectations that the consultants would provide guidance
  • Mechanisms for clarification were effective: email and telephone contacts with consultants
  • Interviewees expressed concerns with some elements of submission process. In particular, the RFP released mid-December but should have been much earlier  to allow adequate time for proposal development
  • “If they are not going to change the dates, how do we make the process shorter and more effective?”
  • The mid-December release made it difficult for project applicants to obtain letters of support
  • Timing and response requirements were difficult to meet
  • Level of effort to complete the RFP was significant and an interviewee observed that the process did not meet recommendation eight of the Independent Blue Ribbon Panel, i.e. streamlined application process
4. Evaluate and Select
  • Receive proposals
  • Screen and assign reviewers
  • Develop rating guides
  • Conduct independent reviews
  • Consolidate reviews
  • Conduct review panel
  • Assign priorities
  • Allocate funds
  • Notify organizations
  • Reviewers included a subject matter expert
  • Confidentiality and control for conflict of interest was instituted
  • Review process was viewed as transparent
  • Overall process from submission to receipt of approval/rejection letter not timely
  • Time associated with assessment of proposals and decision-making by PHAC
  • Very transparent process: “So transparent that it takes forever”
  • Timeliness: not timely
  • Communication of status: “ongoing but no end in sight”
  • Evaluation process logical but bureaucratic
  • It was noted that a Letter of Intent Process could be considered that would leverage consulting support while assisting with the quality of the project as well as partnership creation
5. Distribute Funds
  • Create Contribution Agreement (CA)
  • Transfer first payment
  • Timely release of funds once all formal documentation received
  •  Signing of the contribution agreement: “No problems.”
  • Appropriateness and thoroughness of the CA requirements: standard was fine
  • Timelines and communication: “The agreement is usually quick, but you need to amend the proposal.”
  • CA is dependent on receipt of revised budget and project plan which needed to be amended.  This process, in some instances took time and several iterations
6. Monitor Projects
  • Request and receive status reports
  • Review status
  • Influence project
  • Procedures followed
  • PERT documentation completed by projects
  • Payments transferred in a timely fashion once all documentation received
  • Monitoring and feedback to organizations provided by project consultants
  • Some challenges by funded organizations to collect Project Evaluation Reporting Tool information as the questions arrived too late or had changed
  • Outcome questions not provided to funded organizations from outset of project.
  •  “Please indicate the number by population, if this information was collected by project staff” statement at the end of the project could have generated information by population if it had been requested from outset
  • Quality of quarterly reporting appears inconsistent and not mapped against targets
  • Limited ability to conduct outcomes measurement and evaluation due to short-term nature of funding

Regional Stream
Step Key Tasks Strengths Weaknesses
1. Establish Fund Level
  • Notional indication of budget
  • Allocation of  notional funding across the four components
  • Staff prepare materials based on notional amount
  • Management  finalizes funding level
  • Creation of a regional component was a good idea and generated positive linkages within the P/Ts as well as a joint approach with federal counterparts
  • Amount of money available to be matched was limited in some Agreements
2. Generate Guidelines
  • Review former guidelines
  • Draft new guidelines
  • Seek Management approval
  • Revised and adapted guidelines to meet the new HLF requirements
  • Creation of the joint coordinating committee process
  • Both levels of government set priorities jointly, determined with provincial and territorial governments, specific to each P/T and highlighted within the Bilateral Agreements
  • Challenge with approval of Bilateral Agreements
  • PHAC and Provinces and Territories approval challenges
3. Call for Proposals
  • Issue of call
  • Targeted and focused on organizations with demonstrated capacity to deliver projects
  • Level of effort to complete the proposal
4. Evaluate and Select
  • Receive proposals
  • Conduct review panel
  • Assign priorities
  • Allocate funds
  • Notify organizations
  • Joint Coordinating Committee review process was viewed as transparent
  • Evaluation process logical
  • PHAC project/RAF approval challenges
  • Communication of status: unclear
5. Distribute Funds
  • Create Contribution Agreement
  • Transfer first payment
  • Timely release of funds once all formal documentation received
  • Delays in approval
6. Monitor Projects
  • Request and receive status reports
  • Review status
  • Influence project
  • PERT documentation to be completed by projects
  • PERT results not yet available
  • Final project results are not yet available
KD&E
  • The KD&E component is managed through carefully nurtured relationships with key stakeholders. These include service providers, e.g., CFLRI and CSEP, and F-P/T mechanisms, e.g., the Interprovincial Sport and Recreation Committee (ISRC) and the Physical Activity and Recreation Committee (PARC), other domestic surveillance partners, stakeholders in healthy eating, and international collaborators. These well-managed relationships have been leveraged to ensure activities and outputs are effectively carried out.
  • KD&E appears to identify the needed expertise relative to its outcomes and then enter in to strategic partnerships and alliances with groups that can help them achieve those outcomes as effectively as possible.
  • Activities and strategic partnerships have been undertaken within the existing funding mechanisms of the HL contribution agreements and have been monitored using the consistent PERT reporting required by other funded projects. The partnerships have been enhanced by taking a directed funding approach, which first identifies the appropriate partner with the skills to complete the deliverable and then establishes the funding relationship. In the case of the long-standing relationship with the CFLRI, funding has been more stable and predictable, allowing the organization to add substantial value to KD&E efforts.
  • These types of focused and targeted approaches may be a model for other initiatives.
  • Looking forward, it will be important for KD&E to mine the knowledge gained from the other components of the HL program and share and examine information that can contribute to further advances in Healthy Living.
IHLN
As previously noted, the IHLN has faced challenges that have impeded its progress:
  • the Network was originally governed by a coordinating committee which was terminated; and
  • It was determined, through discussion with all healthy living partners that the Network did not have the capacity to 'direct and advance the development and implementation of the Integrated Pan-Canadian Healthy Living Strategy'. As a result, the original mandate given to the Network was redefined in 2006, under the HLIG and was endorsed by the Population Health Promotion Expert Group.
Despite significant recent progress, some operational challenges continue, including:
  • limited financial and human resources;
  • the original mandate outlined in planning documents has changed significantly, and its new mandate is unclear and has no terms of reference in place; and
  • the IHLN is lacking governance mechanisms to guide strategy, direction and operation.

The IHLN is currently described as a ‘virtual network by staff and does not have the ‘implementing‘ role for the Healthy Living Strategy, as described in the original plans. Stakeholders are unclear as to the mandate of the IHLN and the roles of members. They describe difficulties integrating NGO stakeholders within the IHLN, especially the HLIG because it functions like an F-P/T mechanism and is accountable both federally and provincially.

While the HLIG supports work of the IHLN, there is no accountability mechanism to the IHLN. Key activities and outputs under the current workplan are for the HLIG rather than the IHLN as a whole. While the HLIG represents the administrative arm of the IHLN and supports the IHLN by fostering appropriate linkages and lines of communication with relevant stakeholder groups, it has its own workplan and body of work, and these are distinct from any specific activities by the membership of the IHLN. The HLIG Communications Working Group has responsibility for defining the IHLN and has undertaken activities to examine possible roles.

While current resource levels for the IHLN are inadequate to carry out its intended activities, resources have been provided to support the activities of the HLIG, the administrative arm of the IHLN. The HLIG delivers excellent value by carrying out its activities and outputs.

Social Marketing

Clear lines of reporting exist for Social Marketing; however, this Component sits outside of the overall management of the Healthy Living program and is under Communications leadership rather than the leadership of the Healthy Living Director. Issues of accountability, planning, and alignment have emerged in this evaluation, and these appear to be a direct result of this governance gap.

To date, the governance of the Social Marketing Component has caused some confusion around the management of the ParticipACTION Contribution Agreement. The Communications Unit has the responsibility for the annual budget of ParticipACTION of over $1.6 million but has no expertise to manage the Contribution Agreement. Since ParticipACTION is a major contributor to social marketing efforts, it is essential that effective mechanisms are in place to strategically manage its activities and the delivery of other social marketing campaigns.

Planning for the Social Marketing Component is a cause for concern as well since after two years of operation, processes to establish stronger linkages with the Healthy Living Program have not been established.

 

Conclusions:

The funding processes for the Healthy Living Fund are well documented and operate in a transparent manner. Process improvements are indicated primarily in terms of improving timeliness and efficiency.

The KD&E component is well managed through carefully nurtured relationships with key stakeholders.

The IHLN through the activities of the HLIG is well managed and operating effectively.

The Social Marketing activities are managed separately from the other components, and planning and reporting mechanisms display weaknesses.

 
Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: