ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Knowledge Development and Exchange Functional Component

 

Annex 3: Monitoring and Evaluation Plan

KD&E Functional Component Monitoring and Evaluation Plan (Formative - 2008/09)

This Annex contains the Version 7 (Feb. 4th, 2009) of the PHAC Evaluation Plan for Process (2008) and Outcome (2009) Evaluations. This plan was prepared by PHAC.

Issues/Questions Indicators Data Sources

A) Relevance – does the component continue to be consistent with departmental and government-wide priorities and does it realistically address a real need?

A.1 Is there a continued need for:

  • KD&E Functional Component?
  • Specific elements?
  • Canadian Best Practices Initiative
  • Task Force
  • Demonstration projects
  • CVD, Diabetes and Cancer-specific KD&E activities
  • Overall KD&E planning/coordination
  • evidence of pan-Canadian nature and magnitude of problems/issues related to knowledge development and exchange and the uptake of evidence

Document review: (Doc. Review protocol Q 8 a.)

  • KD&E environmental scans
  • Portal evaluation report
  • stakeholder identified need for knowledge development and exchange initiatives

Document review (Doc. Review protocol Q.8a.)

  • KD&E environmental scans
  • Portal evaluation report
  • Internal Interviews: (Q. 10)
  • External stakeholder structured interview (Q. 3 )

A.2 Is there overlap and duplication within KD&E Functional Component or across ISHLCD Functional Components?

  • evidence of overlap and duplication, if any
  • Second Implementation Review
  • Document review (Doc. Review protocol Q.8b)
  • Internal interviews protocol (Q. 16. b)
  • evidence of joint planning and integration
  • Second Implementation Review
  • Internal interviews protocol  (Q. 5&6)

Are there any elements of the KD&E Functional Component that should be transferred in whole or in part to the Provinces/Territories or other external organizations?

  • role of the Agency is clearly defined and supported by evidence
  • Portal evaluation report

Internal interview protocol (Q.16.b)

  • evidence of overlap and duplication with external organization work

Document review:  (Q.8b)

  • Internal interviews protocol (Q. 16. b)

B) Success/progress – Is the component effective, within budget and without unwanted outcomes?

B.1 What have been the key activities and outputs achieved?

B.1.1 To what extent have priorities been jointly set and activities been planned and implemented with internal and external stakeholder to:

a) Increase system support and b) coordinate KD&E activities overall?

  • # of KD&E meetings and presentations to external stakeholder groups/organization
  • Document review protocol: (Q.3)
  • Internal interviews protocol (Q.5, &6)
  • External stakeholders interviews protocol (Q.4.2. A)
  • Stakeholder linkages and involvement with PHAC KD&E activities
  • # and type of stakeholders engaged
  • # of joint planning sessions for KD&E
    1. between PHAC divisions/centres
    2. with external organizations/
      networks
  • # and type of needs assessments undertaken including key identified needs, gaps
  • # and description of priority proposals/plans
  • # and description of KD&E communication products developed and # of external organization/networks reached
  • Document review protocol (Q.3)
  • Internal interviews protocol (Q. 5 & 6)
  • External stakeholder interviews protocol (Q. 4.2.A, B; Q.4.3 A.; Q. 4.4.A)

B.1.2 To what extent have new knowledge products been developed in consultation with intended users and partners?

  • # and type of knowledge products developed (risk analyses, evidence syntheses, economic analyses, guidelines, tools, reports)
  • Document review protocol (Q.3)
  • Internal interview protocol (Q. 2 & 7)
  • External stakeholders interview protocol (Q. 4.2 & 4.3)
  • Description of intended purpose/evidence of knowledge products developed
  • Document review protocol (Q.3)
  • Internal interview protocol (Q.2 & 7)

B.1.3 To what extent have knowledge exchange and capacity building activities been:

  1. planned; and
  2. implemented

In consultation with key partners and target audiences?

  • # and type of activities undertaken (forums, events, presentations, publications, workshops, training, electronic/web activities) and dissemination plans
  • Document review (Q.3)
  • Internal interview (Q. 2, & 7)
  • External stakeholder interviews protocol (Q. 4.4)
  • Description of purpose and specific audiences targeted
  • Documents review protocol (Q.3)
  • Internal interview protocol (Q.10)
  • # of nominations of Best Practices from field for the Portal and nomination promotion efforts
  • # of new Portal content from practitioners
  • Best Practices Portal Evaluation
  • Document review protocol (Q.10)
  • #/type of activities to facilitate sharing of practice reflections and practice-based evidence
  • Internal interview protocol (Q4, 5, 6 & 9)
  • Document review protocol (Q.3 & 10)
  • External stakeholder protocol (Q.4.5 A & B)

B.2 To what extent have key reach target groups been met with; and key target groups identified?

  • # of knowledge products distributed
  • #/type of recipient organizations and individuals
  • Canadian Best Practices Portal evaluation report
  • Document review (Q. 4; 4.a; 5; 6)
  • Internal interview protocol (Q.7 7 8)
  • External stakeholder protocol (Q.4.4 A.)
  • #/type of participants in workshops/events
  • Document review (Q.4.)

B.3 To what extent has progress been to date made on direct immediate or intermediate outcomes?

   
  • Improved sectoral, jurisdictional and  organizational engagement
  • Increased networking and strategic linkages among key stakeholders and intended users
  • Span of membership on working groups, advisory groups
  • Change in number/type of organizations involved in program activities
  • Internal interview protocol (Q.5, 6, 7, & 8)
  • External stakeholder interview protocol (Q.5.1)
  • Improved awareness and climate among organizations and decision-makers
  • Enhanced capacity for evidence-informed decision-making within organizations
  • Applications for Demonstration projects from P/T governments & partners
  • Practitioner and stakeholder organization ratings and perceptions
  • Canadian Best Practices Portal evaluation report
  • Document review  protocol (Q. 12)
  • External stakeholders (Q. 5.2, 5.8)
  • Alignment of resources, priorities, etc, evaluations
  • Logic models consistent across component, plans aligned with partner organizations
  • KD&E plans –internal & external
  • Documents review
  • Internal interview protocol (Q. 12)
  • External stakeholder interview protocol (Q.5.3, 5.4)
  • Increased sharing of information and knowledge with and between external organizations
  • More targeted knowledge translation to address gaps in knowledge and capacity?
  • Practitioner and stakeholder organization ratings and perceptions
  • Internal interviews (Q.12)
  • External stakeholder interview protocol (Q. 5.5, 5.6, 5.7)
  • Larger and accessible inventories of evidence to support practice and policy decisions
  • Increased use and adoption of evidence and tools from evidence inventories
  • Number of interventions on Portal
  • Reported use of Portal interventions
  • Practitioner and stakeholder organization ratings and perceptions
  • Portal Evaluation Report
  • Internal interviews (Q.12)
  • External stakeholder interview protocol (Q. 5.9, 5.10)

B.4 What have been the unintended positive or negative impacts, if any, of the KD&E Functional Component to date?

  • Evidence of unintended positive and negative consequences are monitored

Document review (Q. 13)

  • Internal interviews (Q. 19)
  • External stakeholders interview protocol(Q. 6.6)
  • Implementation review II

C) Design and Delivery – are the most appropriate and effective means being used to achieve objectives, relative to alternative design and delivery approaches?

C. 1 Are roles and responsibilities in supporting KD& E component clearly defined and understood?

  • Clear statement of roles/responsibilities
  • Document review
  • Internal interviews protocol (Q.18)
  • Internal awareness and understanding of roles/responsibilities
  • Document review
  • Internal interviews protocol (Q.18)

C.2 Are there any key gaps in the KD&E Functional Component?

  • Gaps identified
  • Document Review (Q. 12. A)
  • Internal interviews (Q. 16. a)
  • External stakeholder interview protocol (6.4.1)

C.3 What have been the key lessons learned to date in the KD& E component and specific KD& E initiatives?

  • Key lessons are identified, shared, and applied, as appropriate
  • Document review (Q. 12)
  • Internal interviews (Q. 17)
  • External stakeholder interview protocol (Q6.4)

C.4 Is the current monitoring system for the KD&E Functional Component effective?

  • Key effective indicators are developed and monitored for component and its initiatives
  • Identification of concerns or suggestions re: monitoring system
  • Document review (Q. 12)
  • Internal interviews (Q. 16.c)
  • External stakeholders interview protocol (Q.6.4)

C.5 As the KD&E Functional Component continues, how could its design, delivery, and impact be improved?

  • Identification of design and delivery concerns and suggestions
  • Document review (Q. 14.C)
  • Internal interviews (Q. 20)
  • External stakeholders interview protocol (Q.6.4)

Page details

Date modified: