ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Healthy Living Program Component

 

Annex B: Evaluation Framework

Annex B: Evaluation Framework
Questions Indicators Data Sources

A:  Relevance – does the Component continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need?

A.1 Is there a continued need for the Healthy Living Component

  • Stakeholder identified need
  • External and Internal interviews

A.2 Is there overlap and duplication across the Healthy Living Components - Healthy Living Fund, Knowledge Development and Exchange, Intersectoral Healthy Living Network, and Social Marketing?

  • Distinguishing characteristics including areas of focus for each individual Healthy Living Program components
  • Reduced overlap and duplication through integrated Healthy Living Program work plans
  • Healthy Living Strategic Plan (Roadmap)
  • Healthy Living Program component work plans
  • ISHLCD First and Second Implementation Review
  •  MC & TB Submission
  • Internal & external interviews

A.3 Are there effective governance structures in place to ensure coordination and manage potential for duplication between the Healthy Living component and the Functional Components?

  • Existence of governance structures
  • Evidence of coordination and management of components
  • Terms of Reference
  • Meeting Minutes
  • ISHLCD First and Second Implementation Review

B: Success/Progress – is the Component effective, within budget and without unwanted outcomes?

B.1 Have the key activities and outputs been implemented as planned for each of the four components?

B.1.1 Have the key activities and outputs of the Healthy Living Fund been implemented and achieved as planned?
  • # of proposals received
  • # of projects funded, $ value
  •  # of contribution agreements
  • # of Bilateral Agreements
  • # of knowledge products developed and disseminated
  • Workplan
  • PERT Q7, 10, 15d
  • External interviews
  • Project proposals
  • Signed agreements with P/T

B.1.2 Have the key activities and outputs for the IHLN been as planned?

  • Number of communication tools/vehicles developed and disseminated under the IHLN
  • Types of mechanisms used to promote  communication tools
  • # and types of annual report content requested and received
  • # of annual reports produced and distributed
  • Monitoring and Evaluation plan developed and implemented
  • # and types of joint projects on health disparities
  • Workplan
  • PHAC website
  • Internal and external interviews
  • External survey
  • Annual reports
  • Monitoring and Evaluation plan
  • Health  Disparities Task Force
  • Health disparities reports

B.1.3 Have the key activities and outputs for Social Marketing been achieved as planned?

  • # and type of public education campaigns
  • # and type of community-based initiatives coordinated
  • # and list of social marketing partners
  • # of Physical Activity (PA) Guidelines distributed
  • # and type of communication tools developed and used

(e.g. websites, toolkits, publications)

  • Workplan
  • Social Marketing reports
  • ParticipACTION evaluation report
  • Internal and external interviews
  • Warehouse statistics

B.1.4 Have the key activities and outputs of KD&E been achieved as planned?

  • Number of projects funded
  • Description of funded project activities and objectives
  • # of domestic and international partners
  • # and types of knowledge and surveillance products enhanced
  • Types of Best Practices Portal populated with Physical Activity content
  • # of research papers, reports developed  and dissemination of publications on healthy living
  • Project proposals
  • PERT project planning questions
  • CANPLAY PA survey
  •  Publications for PA Monitor
  •  Publication of advancing physical activity measurement and guidelines
  • Internal and external interviews
  • PERT Q3, Q6, Q7, Q8
  • Reports on domestic and international knowledge and surveillance meetings
  • Physical Activity content in Best Practices Portal

B.2 To what extent have the key reach targets been met for all the four components?

B.2.1 To what extent have the following vulnerable populations been reached?

  • Children and youth
  • Aboriginal groups
  • Isolated rural communities
  • # / % of children, and youth reached
  • Project reports
  • PERT Q9a-c
  • PERT Q10
  • ECIP Case studies
  • External interviews
  • # / % of Aboriginal groups reached
  •  Funded project evaluation reports
  •  PERT Q9a-c
  •  PERT Q10
  • External interviews
  • #/% of isolated rural communities reached
  •  Funded project evaluation reports
  •  PERT Q9a-c
  •  PERT Q10
  • Internal interviews

B.2.2 To what extent have partnerships been formed with P/Ts, private sector, NGOs?

  • # of  new and existing partners by sector/type 
  • PERT Q7
  • ECIP Case studies
  • External interviews

B.3 To what extent has progress been made to date on the immediate outcomes?

B.3.1 To what extent has progress been made to date on the Healthy Living Fund immediate outcomes?

  • Increased inter-organizational, inter and intra-sectoral, multi-jurisdictional engagement, alignment and collaboration
  • Increased community capacity to support the relationship between healthy eating and physical activity
  • Increased reach to target populations to enhance healthy eating and physical activity
  • Increased knowledge dissemination including reports and research findings
  • # and types of organizations reporting better levels of engagement and alignment
  • # of partners
  • # of community groups reporting strengthened capacities to support healthy eating and physical activity
  • # and types of organizations reporting improved capacities for knowledge development and dissemination
  • Types of knowledge products developed and disseminated
  • ECIP Case studies
  • External interviews
  • PERT Q8, Q15d
  • PERT Q7
  • PERT Q 3a & b
  • PERT A6a & b
  • PERT Q18C
  • PERT Q9
  • PERT Q8
  • PERT 10

B.3.2 To what extent has progress been made to date on the IHLN immediate outcomes?

  • Increased number of IHLN members
  • Increased sharing of information, knowledge and experiences among IHLN members across sectors and jurisdictions
  • Increased awareness of the IHLN and its membership
  • # of IHLN members by sectors and jurisdictions  
  • Types of information, knowledge and experiences shared among IHLN members by sectors and jurisdictions
  • % of IHLN members reporting an increase in their awareness about the IHLN
  • IHLN reports
  • Internal and external interviews
  • External survey

B.3.3 To what extent has progress been made to date on the following KD&E immediate outcomes?

  • Updated scientific evidence to protect integrity of physical activity guidelines and for development of clinical practice guidelines
  • Updated status on Canadian physical activity levels
  • Increased healthy living focus within domestic surveillance and research activities
  • Enhanced physical activity content within the Best Practices Portal
  • Objective data on physical activity levels of children and youth.
  • Approved F-P/T physical activity targets for children and youth.
  • # and types of scientific reviews and updates on Physical activity guidelines and clinical practice guidelines
  • # and types of updates on the Canadian physical activity levels
  • # and types of domestic surveillance and research activities on healthy living
  • Funded project evaluation reports
  • Internal and external interviews
  • PERT Q7
  • PERT Q 3a & b
  • PERT A6a & b

B.3.4 To what extent has progress been made to date on the following Social Marketing immediate outcomes?

  • Increasing awareness about the importance of healthy living and maintenance of healthy weights among Canadians
  • Increased awareness about Canada’s Physical Activity Guides
  • % of Canadians reporting increased awareness about the importance of healthy living and maintenance of healthy weights
  • % of Canadians reporting increased levels of awareness about Canada’s Physical Activity Guides
  • # of Guides distributed
  • ParticipAction evaluation report
  • SAWA Evaluation report
  • Guide Warehouse statistics

B.4 What have been the unintended outcomes as a result of the Healthy Living Program activities or each of the components, if any?

  • Types of positive & negative effects as a result of program activities by component
  • ISHLCD First and Second  Implementation Reviews
  • Funded project evaluation reports
  • PERT Analysis report
  • ECIP Report
  • Social Marketing reports
  • Internal and external interviews

C: Design and Delivery – are the most appropriate and effective means being used to achieve objectives, relative to alternative design and delivery of approaches? 

C.1 Are there any overall key gaps in the Healthy Living Program or each of the components?

  • Types of gaps identified through the overall Healthy Living Program planning and implementation
  • Types of gaps identified through the HLF, IHLN, Social Marketing, KD&E component planning and implementation
  • Extent  to which expectations are in line with resources provided
  • Healthy Living Strategic Plan
  • Healthy Living Program component work plans
  • MC & TB submission
  • Internal and external interviews

C.2 What have been the overall key lessons learned to date from the Healthy Living Program and each of the components?

  • Types of lessons in terms of the overall planning and implementation and achievement of  results and outcomes
  • Internal and external interviews
  • ECIP case studies

C.3 Is the current monitoring system for the Healthy Living Program components effective?

  • Extent to which funded organizations comply with the PERT monitoring and reporting requirements
  • Other monitoring systems in place
  • Funded project evaluation plans
  • PERT and PDCAS processes
  • Internal and external interviews

C.4 Are the program design and operating process efficient and effective?

  • Review of program design and operating processes:
    • HLF contribution agreement management
    • IHLN structures and processes
    • KD&E processes
    • Social marketing program design and processes
  • Identified strengths and weaknesses of each program component
  • Identified internal and external factors that have contributed or restricted program design
  • Process review
  • Interviews
  • Program files

Page details

Date modified: