ARCHIVED - Annex C: Evaluation Tools

 

Surveillance Component Evaluation –
Managers/Leader/Committee Interview GuideFootnote 3

Thank you for agreeing to discuss with me the surveillance component of the Integrated Strategy on Health Living and Chronic Disease (ISHLCD)!

We will be interviewing policy leaders, program managers and stakeholders involved in the surveillance functional component of the strategy. We will focus on your role as a surveillance system manager/ policy leader /committee member, using the surveillance component logic model and the strategy “matrix” as a means to organize our discussion.

The interview should take less than an hour and will be recorded for note taking purposes. Any direct quotes that are important for the analysis will be grouped into theme categories, to minimize the identification of individual respondents. This being said we cannot guarantee anonymity given the relatively small number of people being interviewed.  Are you comfortable with proceeding with the interview?

This evaluation study intends to guide and inform stakeholders regarding “how far and how fast” program implementation has proceeded and what early outcomes have been generated for the action areas of:

  • Enhancing the use of existing data,
  • Expanding data sources, and
  • Coordinating, planning and evaluating surveillance activities.

Guiding these action areas is a shared vision of integrated chronic disease surveillance as a blend of centralized and decentralized surveillance systems that serve as a platform for the early detection and management of chronic disease. Acting on this vision would ensure that the “right” internal and external stakeholders are engaged at the “right” time with the “right” intensity.


1)

In regard to the Strategy “matrix”

  1. Looking at the “columns” of the matrix, are you involved in functionalcomponents other than surveillance? PROBE: If “YES”, what other functional components are you involved with & in what capacity?
  2. Turning to the “rows” of the matrix, which program components are you most involved with?
  3. Now that we have “situated” you in relation to the matrix, please describe your role relating to the implementation of the surveillance component of the integrated strategy.

2)

So we can get a better understanding of the context in which the Strategy has unfolded, can you identify policy initiatives, priority shifts or directives which have impacted on the surveillance component since the implementation of the strategy in 2007?

PROBE: For any identified – Was there any specific impact on surveillance?
(These might include OAG, strategic plans …)

PROBE: Overall, how would you describe the federal, P/T and NGO roles during the implementation of the strategy?

3)

  1. From a general perspective, if your program were reviewed, what would be the basis of your rationale for the continuance of the program?
  2. Specifically, for your activities within the surveillance functional component of the strategy, can you identify which Agency Strategic Objective or Objectives1 links to your work? NOTE: If “NO” – see footnote to remind respondent of the possible options.

4)

In your opinion, how well do the surveillance functional component action areas, depicted in the logic model, “align” with the current priorities and directions of the:

  • Public Health Agency of Canada,
  • the broader Health Portfolio (which includes Health Canada, PMRA …),
  • the broader federal government including departments with an interest in health such as the Department of National Defence, Correctional Services Canada, the RCMP…and others);
  • P/T Governments, and
  • Health NGOs

For rating alignment we will use “1” to indicate “very well aligned”, “2” for “somewhat aligned”, 3 for “not well aligned” and “4” for “Not at all aligned”.

 So the surveillance action areas align with the current priorities of the …

Very well Somewhat Not well Not at all PROBE: Any particular rationale for this ...

a.

The Public Health Agency of Canada as a whole. (Including documents such as the Strategic plan …)

1

2

3

4

 

b.

Health Portfolio (Health Canada, PMRA, PHAC, Patent Med Review Board..

1

2

3

4

 

c.

Broader Federal Gov’t as a whole

1

2

3

4

 

d.

P/T Gov’t

1

2

3

4

 

e.

Health NGOs

1

2

3

4

 

5)

Now with the matrix in mind, (make sure this is in front of the respondent) are you aware of any important linkages or relationships between the surveillance component and any of the other functional components identified in the strategy matrix?
(Yes (follow-up), No, Don’t know)

If “YES”:

  1. What are they?
  2. What is the most important linkage from the perspective of the surveillance component?
  3. What do you think is required to maintain or further develop this (or these)  relationship(s)?

6)

Are there any important linkages or relationships between the surveillance component and any other healthy living and chronic disease initiatives outside of the integrated strategy? If “YES” please obtain a description from the respondent.

7)

With the logic model in mind, what new resources, both in terms of staff and money have been dedicated to the surveillance component? NOTE: Please have respondents group these by action area.

PROBE: Would you agree that your activities are now fully supported by the resources designated under the strategy?

PROBE: If “NOT”, why not? & when will this be accomplished?

8)

When you think about building chronic disease surveillance capacity on a national scale, is there anything missing when you consider the action areas of enhancing the use of existing data, expanding data sources, and coordinating, planning and evaluating surveillance activities? If “YES” Please have respondent elaborate.

9)

Now turning to the activities described in the logic model (Give respondent a few moments to review the activities)

  1. What activities are you directly involved with? (Have respondent describe by action area)
  2. Are there any errors or omissions in the current descriptions of activities?
    If “yes” please describe…
  3. Where has there been progress? PROBE: Which activities are complete or nearing completion this fiscal year?
  4. Are there activities which have not been implemented at all?
    If “YES”, which ones, and when will they likely be implemented (2009/2010, 2010/2011)?

10)

Keeping the logic model in mind,

  1. How would you describe your target group for surveillance information?
  2. How does your program communicate with this target group? PROBE: How often? By what means? and
  3. Has the implementation of the surveillance component had an impact on how you communicate with your target group? PROBE: If not now, do you think it will in the future?

11)

As part of implementing the surveillance component “who” have you chosen to partner with and why?

PROBE: Have these partnerships been effective in contributing to your program so far?

PROBE: Are there any additional partnership relationships you hope to establish in the future?

12)

Keeping the logic model in mind:

a.

Please rate how effective the surveillance component has been with the engagement of organizations, sectors and jurisdictions.

For rating effectiveness we will use “1” to indicate “not effective”, “2” for somewhat effective” and 3 for “very effective”.

So how effective has the surveillance component been with engaging organizations, sectors and jurisdictions in increasing (the) …

Not effective Somewhat effective Very Effective Don’t know Not Applicable

Sharing of information (e.g., on planning, surveillance, products …)
 

1

2

3

DK

NA

Sharing of resources (e.g., FTEs or time)

1

2

3

DK

NA

Clarity of roles with partners

1

2

3

DK

NA

Consistency of funding

1

2

3

DK

NA

Level of trust among partners 

1

2

3

DK

NA

Information on the tracking of chronic disease in Canada

 

 

 

 

 

Quality of data

1

2

3

DK

NA

Comparability of data

1

2

3

DK

NA

Access to data and metadata

1

2

3

DK

NA

Knowledge, skills and resources to do surveillance

1

2

3

DK

NA

Capacity of data users to better interpret surveillance products

1

2

3

DK

NA

Knowledge about trends in chronic disease, risk factors and disease outcomes

1

2

3

DK

NA

Use of surveillance data to guide & evaluate decisions about chronic disease research, policies, programs & services.

1

2

3

DK

NA

Effectiveness and efficiency of programs, policies and services.

1

2

3

DK

NA

b.

What would your overall assessment be of the effectiveness of the surveillance functional component in increasing the engagement of organizations, sectors and jurisdictions.  (where “1” is “not effective”, “2” is somewhat effective” and 3 is “very effective”)

PROBE: How do you think engagement should be handled in the future?

13)

Associated with the implementation the of surveillance functional component, did anything unanticipated or unintended (either positive or negative) occur:

a.

across action areas?

b.

across activities?

c.

in the generation of outputs or early outcomes?

PROBE: (If any identified) Would you say that what happened was merely unforeseen or was it unforeseeable?

14)

What “lessons learned” have been identified either formally or informally during implementation?

d.

With respect to how the program was designed?

e.

With respect to what has been accomplished so far?

f.

With respect to the achievements of outputs & early outcomes?

15)

When you think of the surveillance component overall, are you aware of any major design, resourcing, or delivery “gaps” within the component? PROBE: (If any identified) Have these impacted on the achievement of surveillance component outputs or early outcomes?

16)

What formal or informal practices and tools do you use to monitor the implementation of the surveillance component to ensure that it is “on-track” and aligned with the strategy objectives (e.g., coordinator committee meetings, tables, Gantt charts, etc.)?

17)

We would like to identify any gaps in relevant documentation, by getting your feedback on the reference list we have shared with you.

g.

Are there any key documents missing?

h.

Are there any additional documents you would recommend for us to improve our knowledge of the surveillance component?

18)

To learn a little more about you, please tell me:

i.

What was the area of your training & what qualifications have you obtained?

j.

How many years have you been involved with chronic disease surveillance & in what capacities?

k.

What is your current job title?

19)

Finally, what would be the most useful piece of information you would like to gain from this evaluation on the implementation of the surveillance functional component of the integrated strategy?

Thank you very much for your participation!

Supplementary Endnotes:

1 (Q#3) Five Year Priorities for Action  - Strategic Objective 1: Anticipate and respond to the health needs of Canadians – Delivering on policy and program priorities

 

1.1

Meeting major Public Health challenges;

 

1.2

Addressing determinants and disparities

 

1.3

Building Public Health Capacity

 

1.4

Fulfilling our mandate

2

Strategic Objective 2: Ensure actions are supported by integrated information and knowledge functions –Aligning programs and research to support priorities

3

Strategic Objective 3: Further develop PHAC’s dedicated, professional workforce by providing it with the tools and leadership it needs and by ensuring a supportive culture – Enhancing Agency capabilities and accountabilities.

REFERENCE : The Public Health Agency of Canada - Strategic Plan: 2007 – 2012, Information, Knowledge, Action


Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: