Issues/Questions |
Indicators |
Data Sources |
A) Relevance – does the component continue to be consistent with departmental and government-wide priorities and does it realistically address a real need? |
A.1 Is there a continued need for:
- KD&E Functional Component?
- Specific elements?
- Canadian Best Practices Initiative
- Task Force
- Demonstration projects
- CVD, Diabetes and Cancer-specific KD&E activities
- Overall KD&E planning/coordination
|
- evidence of pan-Canadian nature and magnitude of problems/issues related to knowledge development and exchange and the uptake of evidence
|
Document review: (Doc. Review protocol Q 8 a.)
- KD&E environmental scans
- Portal evaluation report
|
- stakeholder identified need for knowledge development and exchange initiatives
|
Document review (Doc. Review protocol Q.8a.)
- KD&E environmental scans
- Portal evaluation report
- Internal Interviews: (Q. 10)
- External stakeholder structured interview (Q. 3 )
|
A.2 Is there overlap and duplication within KD&E Functional Component or across ISHLCD Functional Components? |
- evidence of overlap and duplication, if any
|
- Second Implementation Review
- Document review (Doc. Review protocol Q.8b)
- Internal interviews protocol (Q. 16. b)
|
- evidence of joint planning and integration
|
- Second Implementation Review
- Internal interviews protocol (Q. 5&6)
|
Are there any elements of the KD&E Functional Component that should be transferred in whole or in part to the Provinces/Territories or other external organizations? |
- role of the Agency is clearly defined and supported by evidence
|
Internal interview protocol (Q.16.b) |
- evidence of overlap and duplication with external organization work
|
Document review: (Q.8b)
- Internal interviews protocol (Q. 16. b)
|
B) Success/progress – Is the component effective, within budget and without unwanted outcomes? |
B.1 What have been the key activities and outputs achieved? |
B.1.1 To what extent have priorities been jointly set and activities been planned and implemented with internal and external stakeholder to: a) Increase system support and b) coordinate KD&E activities overall? |
- # of KD&E meetings and presentations to external stakeholder groups/organization
|
- Document review protocol: (Q.3)
- Internal interviews protocol (Q.5, &6)
- External stakeholders interviews protocol (Q.4.2. A)
|
- Stakeholder linkages and involvement with PHAC KD&E activities
- # and type of stakeholders engaged
- # of joint planning sessions for KD&E
- between PHAC divisions/centres
- with external organizations/
networks
- # and type of needs assessments undertaken including key identified needs, gaps
- # and description of priority proposals/plans
- # and description of KD&E communication products developed and # of external organization/networks reached
|
- Document review protocol (Q.3)
- Internal interviews protocol (Q. 5 & 6)
- External stakeholder interviews protocol (Q. 4.2.A, B; Q.4.3 A.; Q. 4.4.A)
|
B.1.2 To what extent have new knowledge products been developed in consultation with intended users and partners? |
- # and type of knowledge products developed (risk analyses, evidence syntheses, economic analyses, guidelines, tools, reports)
|
- Document review protocol (Q.3)
- Internal interview protocol (Q. 2 & 7)
- External stakeholders interview protocol (Q. 4.2 & 4.3)
|
- Description of intended purpose/evidence of knowledge products developed
|
- Document review protocol (Q.3)
- Internal interview protocol (Q.2 & 7)
|
B.1.3 To what extent have knowledge exchange and capacity building activities been:
- planned; and
- implemented
In consultation with key partners and target audiences? |
- # and type of activities undertaken (forums, events, presentations, publications, workshops, training, electronic/web activities) and dissemination plans
|
- Document review (Q.3)
- Internal interview (Q. 2, & 7)
- External stakeholder interviews protocol (Q. 4.4)
|
- Description of purpose and specific audiences targeted
|
- Documents review protocol (Q.3)
- Internal interview protocol (Q.10)
|
- # of nominations of Best Practices from field for the Portal and nomination promotion efforts
- # of new Portal content from practitioners
|
- Best Practices Portal Evaluation
- Document review protocol (Q.10)
|
- #/type of activities to facilitate sharing of practice reflections and practice-based evidence
|
- Internal interview protocol (Q4, 5, 6 & 9)
- Document review protocol (Q.3 & 10)
- External stakeholder protocol (Q.4.5 A & B)
|
B.2 To what extent have key reach target groups been met with; and key target groups identified? |
- # of knowledge products distributed
- #/type of recipient organizations and individuals
|
- Canadian Best Practices Portal evaluation report
- Document review (Q. 4; 4.a; 5; 6)
- Internal interview protocol (Q.7 7 8)
- External stakeholder protocol (Q.4.4 A.)
|
- #/type of participants in workshops/events
|
|
B.3 To what extent has progress been to date made on direct immediate or intermediate outcomes? |
|
|
- Improved sectoral, jurisdictional and organizational engagement
- Increased networking and strategic linkages among key stakeholders and intended users
|
- Span of membership on working groups, advisory groups
- Change in number/type of organizations involved in program activities
|
- Internal interview protocol (Q.5, 6, 7, & 8)
- External stakeholder interview protocol (Q.5.1)
|
- Improved awareness and climate among organizations and decision-makers
- Enhanced capacity for evidence-informed decision-making within organizations
|
- Applications for Demonstration projects from P/T governments & partners
- Practitioner and stakeholder organization ratings and perceptions
|
- Canadian Best Practices Portal evaluation report
- Document review protocol (Q. 12)
- External stakeholders (Q. 5.2, 5.8)
|
- Alignment of resources, priorities, etc, evaluations
|
- Logic models consistent across component, plans aligned with partner organizations
- KD&E plans –internal & external
|
- Documents review
- Internal interview protocol (Q. 12)
- External stakeholder interview protocol (Q.5.3, 5.4)
|
- Increased sharing of information and knowledge with and between external organizations
- More targeted knowledge translation to address gaps in knowledge and capacity?
|
- Practitioner and stakeholder organization ratings and perceptions
|
- Internal interviews (Q.12)
- External stakeholder interview protocol (Q. 5.5, 5.6, 5.7)
|
- Larger and accessible inventories of evidence to support practice and policy decisions
- Increased use and adoption of evidence and tools from evidence inventories
|
- Number of interventions on Portal
- Reported use of Portal interventions
- Practitioner and stakeholder organization ratings and perceptions
|
- Portal Evaluation Report
- Internal interviews (Q.12)
- External stakeholder interview protocol (Q. 5.9, 5.10)
|
B.4 What have been the unintended positive or negative impacts, if any, of the KD&E Functional Component to date? |
- Evidence of unintended positive and negative consequences are monitored
|
Document review (Q. 13)
- Internal interviews (Q. 19)
- External stakeholders interview protocol(Q. 6.6)
- Implementation review II
|
C) Design and Delivery – are the most appropriate and effective means being used to achieve objectives, relative to alternative design and delivery approaches? |
C. 1 Are roles and responsibilities in supporting KD& E component clearly defined and understood? |
- Clear statement of roles/responsibilities
|
- Document review
- Internal interviews protocol (Q.18)
|
- Internal awareness and understanding of roles/responsibilities
|
- Document review
- Internal interviews protocol (Q.18)
|
C.2 Are there any key gaps in the KD&E Functional Component? |
|
- Document Review (Q. 12. A)
- Internal interviews (Q. 16. a)
- External stakeholder interview protocol (6.4.1)
|
C.3 What have been the key lessons learned to date in the KD& E component and specific KD& E initiatives? |
- Key lessons are identified, shared, and applied, as appropriate
|
- Document review (Q. 12)
- Internal interviews (Q. 17)
- External stakeholder interview protocol (Q6.4)
|
C.4 Is the current monitoring system for the KD&E Functional Component effective? |
- Key effective indicators are developed and monitored for component and its initiatives
- Identification of concerns or suggestions re: monitoring system
|
- Document review (Q. 12)
- Internal interviews (Q. 16.c)
- External stakeholders interview protocol (Q.6.4)
|
C.5 As the KD&E Functional Component continues, how could its design, delivery, and impact be improved? |
- Identification of design and delivery concerns and suggestions
|
- Document review (Q. 14.C)
- Internal interviews (Q. 20)
- External stakeholders interview protocol (Q.6.4)
|