ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Healthy Living Program Component

 

Annex D: Interview Guides

Healthy Living Program Formative Evaluation: Interview Guide (Funding Recipients)

Introduction:

TDV Global, a management consulting firm, has been engaged by the Public Health Agency of Canada (PHAC) to conduct a formative evaluation of the Healthy Living Program.  The purpose of the Formative Evaluation of the Healthy Living Program is to:

  • Assess program design and its implementation;
  • Assess success/progress and continuous improvement;
  • Assess performance measurement systems being used, including the Program Evaluation Reporting Tool (PERT), and the Effectiveness of Community Interventions Program (ECIP);
  • Identify lessons learned; and
  • Make recommendations on key issues.

The formative evaluation of the Healthy Living programs also fulfils requirements in the Results Based Management and Accountability Framework (RMAF) developed for the Integrated Strategy on Healthy Living and Chronic Disease at the request of Treasury Board. In addition to fulfilling requirements articulated in the RMAF, the evaluation will provide valuable information about progress being made toward the Goals, Objectives and Outcomes which have been established for: The Healthy Living Fund, the Intersectoral Healthy Living Network, Knowledge Development and Exchange, and Social Marketing; which comprise the core components of the Healthy Living Program. Formative evaluations early in the life of a program provide information about what is working well, but also facilitate any necessary corrections to the program which affect the ability of a program to achieve its desired outcomes. The report and recommendations resulting from the evaluation will be used to improve the Healthy Living Program.

Discussion Guide:

Based on our evaluation framework, we will be interviewing key national/regional program personnel, selected provincial/territorial partners and stakeholders, funding recipients as well as a number of funding requestors that have been declined.  We would like to discuss this program with you, based on the questions on the next page. We request approximately 30 minutes of your time. Your opinions are a very important element in evaluating the Healthy Living Program’s progress to date as well as charting the course for future actions. We do appreciate your time. All individual responses will be kept in strict confidence, and results will only reported in tabulated form.

If you have any specific documents that would help to answer some of the questions, we would like to obtain a copy of these documents after the interview.

Relevance – Consistency of component with departmental and government-wide priorities and whether it realistically address an actual need

1.

(a) There is a continued need for the Healthy Living Program. What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly.

Success/Progress – Success to date in achieving outputs, outcomes and reaching key target groups

2.

Please provide an overview of how successful you have been to date in implementing your activities.

3.

What enablers or constraints have affected the success of your project to date?

4.

(a) Who are your target populations?

(b) Which of your activities target the following vulnerable populations?

  • Children & youth
  • aboriginal groups
  • isolated rural communities
  • others

5.

(a) What are the results of your activities to date?

(b) How are you tracking the results?

6.

Have you established partnerships/collaborations to date with other organizations and /or levels of government? For example:

  • provincial/territorial government
  • private sector
  • others

7.

(a) To which of the following immediate results is your project contributing to and how?

  • Increased inter-organizational, inter and intra sectoral, multi-jurisdictional engagement, alignment and collaboration
  • Increased community capacity to support the relationship between healthy eating and physical activity
  • Increased reach to target populations to enhance healthy eating and physical activity
  • Increased knowledge dissemination including reports and research findings

(b) What is your criteria for determining the progress made?

8.

Have there been any unintended outcomes to date as a result of your activities?

Design and Delivery – Appropriateness and effectiveness of means being used to achieve objectives, relative to alternate design and delivery approaches

9.

What is working well and what could be improved in each of the following steps in the funding process:

  • Solicitation process
    • Distribution and reach
    • Clarity of RFP
    • Mechanisms to seek clarification
    • Timing and response requirements
    • Level of effort required
  • Approval and decision making by PHAC
    • Transparency of process
    • Timeliness
    • Opportunities for clarification
    • Communication of status
  • Establishment of contribution agreement and fund dispersal
    • Timelines and communication
    • Contribution agreement
    • Appropriateness and thoroughness of requirements
    • Timely dispersal
  • Monitoring and reporting
    • Procedures in place
    • Level of effort required
    • PHAC feedback
    • Opportunity for program adjustment

10.

How satisfied are you with the submission and granting process?  For each of the following areas, please rate your level of satisfaction using the following scale:

  1
Very satisfied
2
Satisfied
3
Neither satisfied nor dissatisfied
4
Dissatisfied
5
Very dissatisfied
No
opinion
Solicitation process            
Approval and decision making by PHAC            
Establishment of contribution agreement and fund dispersal            
Monitoring and reporting            

11.

What internal or external factors contributed or hindered operating processes?

12.

What, if any lessons can be learned at this point?

13.

What, if any, gaps are there in the HLF? If so can you explain?

Healthy Living Program Formative Evaluation: Interview Guide (Unsuccessful Applicants)

Introduction:

TDV Global, a management consulting firm, has been engaged by the Public Health Agency of Canada (PHAC) to conduct a formative evaluation of the Healthy Living Program.  The purpose of the Formative Evaluation of the Healthy Living Program is to:

  • Assess program design and its implementation;
  • Assess success/progress and continuous improvement;
  • Assess performance measurement systems being used, including the Program Evaluation Reporting Tool (PERT), and the Effectiveness of Community Interventions Program (ECIP);
  • Identify lessons learned; and
  • Make recommendations on key issues.

The formative evaluation of the Healthy Living programs also fulfils requirements in the Results Based Management and Accountability Framework (RMAF) developed for the Integrated Strategy on Healthy Living and Chronic Disease at the request of Treasury Board. In addition to fulfilling requirements articulated in the RMAF, the evaluation will provide valuable information about progress being made toward the Goals, Objectives and Outcomes which have been established for: The Healthy Living Fund, the Intersectoral Healthy Living Network, Knowledge Development and Exchange, and Social Marketing; which comprise the core components of the Healthy Living Program. Formative evaluations early in the life of a program provide information about what is working well, but also facilitate any necessary corrections to the program which affect the ability of a program to achieve its desired outcomes. The report and recommendations resulting from the evaluation will be used to improve the Healthy Living Program.

Discussion Guide:

Based on our evaluation framework, we will be interviewing key national/regional program personnel, selected provincial/territorial partners and stakeholders, funding recipients as well as a number of funding requestors that have been declined.  We would like to discuss this program with you, based on the questions on the next page. We request approximately 30 minutes of your time. Your opinions are a very important element in evaluating the Healthy Living Program’s progress to date as well as charting the course for future actions. We do appreciate your time. All individual responses will be kept in strict confidence, and results will only reported in tabulated form.

If you have any specific documents that would help to answer some of the questions, we would like to obtain a copy of these documents after the interview.

Relevance – Consistency of component with departmental and government-wide priorities and whether it realistically address an actual need

1.

There is a continued need for the Healthy Living Program.  What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly.

Design and Delivery – Appropriateness and effectiveness of means being used to achieve objectives, relative to alternate design and delivery approaches

2.

What is working well and what could be improved in each of the following steps in the funding process:

  • Solicitation process
    1. Distribution and reach
    2. Clarity of RFP
    3. Mechanisms to seek clarification
    4. Timing and response requirements
    5. Level of effort required
  • Evaluation and decision making by PHAC
    1. Transparency of process
    2. Timeliness
    3. Opportunities for clarification
    4. Communication of status

3.

How satisfied are you with the submission and granting process?  For each of the following areas, please rate your level of satisfaction using the following scale:

  1
Very satisfied
2
Satisfied
3
Neither satisfied nor dissatisfied
4
Dissatisfied
5
Very dissatisfied
No
opinion
Solicitation process            
Approval and decision making by PHAC            

4.

What, if any, lessons can be learned at this point?

Healthy Living Program Formative Evaluation: Interview Guide (Program Staff and Management - IHLN)

Introduction:

TDV Global, a management consulting firm, has been engaged by the Public Health Agency of Canada (PHAC) to conduct a formative evaluation of the Healthy Living Program. The purpose of the Formative Evaluation of the Healthy Living Program is to:

  • assess program design and its implementation;
  • assess early successes and progress;
  • assess performance measurement tools being used, including the Project Evaluation and Reporting Tool (PERT), and the Effectiveness of Community Interventions Project (ECIP);
  • identify lessons learned; and
  • make recommendations on key issues.

Formative evaluations early in the life of a program provides information about what is working well, but also facilitates any necessary corrections to the program which affect the ability of a program to achieve its desired outcomes.  The evaluation will provide valuable information about progress being made toward the Goals, Objectives and Outcomes which have been established for: the Healthy Living Fund; the Intersectoral Healthy Living Network; Knowledge Development and Exchange; and Social Marketing – which comprise the core components of the Healthy Living Program.

You have been selected to participate in this interview because of your involvement with the Intersectoral Healthy Living Network.  The report and recommendations resulting from the evaluation will be used to improve the Network and will be shared with you when finalized.

Discussion Guide:

Based on our evaluation framework, we will be interviewing key management and program personnel.  The interview should take approximately 45 minutes of your time. Any direct quotes that are important for the analysis will be grouped into theme categories, to minimize the potential for identification of individual respondents. This being said, we cannot guarantee complete anonymity given the relatively small number of people being interviewed. Are you comfortable with proceeding with the interview?  We would like to discuss the Intersectoral Healthy Living Network with you based on the questions on the next page. Your opinions are a very important element in evaluating the Network’s progress to date as well as charting the course for future actions.  Please take some time to review and prepare your answers before the interview.

Should you feel that that there are important documents that would help you to answer the questions, we would appreciate it if you would provide us with a copy of the document.  We will also share with you the Intersectoral Healthy Living Network Logic Model to assist you in answering some of the questions.

Thank you.

Relevance

1.

(a)  There is a continued need for the Intersectoral Healthy Living Network?  What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly…

2.

(a) In your opinion, does the Healthy Intersectoral Healthy Living Network “align“ with the current priorities and directions of the following:

I. Public Health Agency of Canada/Health Canada/Health Portfolio
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly…

II. P/T Governments
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain your reasons briefly…

3.

(a)  There is no overlap/duplication across the four Healthy Living Program components: Healthy Living Fund; Knowledge Development and Exchange; Intersectoral Healthy Living Network; and Social Marketing. What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

If you feel there is overlap, please provide examples.

4.

(a)  There are mechanisms in place to ensure that duplication does not occur between the Intersectoral Healthy Living Network and the Integrated Strategy’s Functional Components. What is your opinion with regards to this statement?  (interviewer shows Functional Component Place Mat)

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) if you feel there are existing mechanisms please provide examples…

Design and Delivery

5.

(a)  The overall design and the operating processes within the Intersectoral Healthy Living Network is satisfactory?  What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain briefly…

6.

(a) Specifically, what is working well and what could be improved in the design and delivery of the Intersectoral Healthy Living Network including:

  • Development process for the Annual Report.
  • E-Bulletin development and distribution.
  • Working groups processes.
  • Secretariat coordination.
 

(b)  Please explain briefly what could be improved in the design and delivery of the Intersectoral Healthy Living Network.

7.

(a) What, if any early lessons learned in the overall design and delivery of the Intersectoral Healthy Living Network can be identified?

(b) Please explain briefly…

8.

(a) What, if any gaps in the overall design and delivery of the Intersectoral Healthy Living Network can be identified?

(b) Please explain briefly…

Success/Progress

9.

(a)  Please provide an overview of the progress to date in the development of the Intersectoral Healthy Living Network in regards to the following: (interviewer shows logic model)

  • Definition and development of the IHLN.
  • Reporting annually to Conference of FPT Deputy Ministers of Health to track progress in reaching the targets outlined in the Strategy, and highlight key efforts.
  • Development of a Monitoring and Evaluation Plan for the IHLN.
  • Recommendations for a common set of health disparities indicators and a feasible approach to their implementation in the Canadian Context.
  • Identified joint project opportunities annually with multi-sectoral and multi-lateral collaboration.

(b) How are you measuring your progress?

(c) What has positively affected the progress of the Intersectoral Healthy Living Network to date?

(d) What barriers have affected the progress of the Intersectoral Healthy Living Network to date?

10.

Which of the following vulnerable populations does the Intersectoral Healthy Living Network  reach:

  • children & youth;
  • Aboriginal Peoples;
  • isolated, rural and remote communities; and/or
  • others?

11.

Has the process for establishing the Intersectoral Healthy Living Network enhanced partnerships/collaborations to date with the following:

  • provincial/territorial governments;
  • private sector;
  • CIHR; and/or
  • others (NGOs & alliances/coalitions)?

12.

(a)  Which of the following immediate outcomes/results is the Intersectoral Healthy Living Network contributing to:   (interviewer shows logic model)

  • Increased number of IHLN members.
  • Increased sharing of information, knowledge and experience among the members across sectors and jurisdictions.
  • Increased awareness of the Network and its membership.

(b) Please explain briefly… (PROMPT - how is the Network contributing to the identified outcome/result?)

13.

(a) Have there been any unanticipated positive outcomes to date as a result of your activities?  (PROMPT – these are outcomes that are surprising, unforeseen or unplanned)

(b) Have there been any unanticipated negative outcomes to date as a result of your activities?  (PROMPT – these are outcomes that are surprising, unforeseen or unplanned)

            

Healthy Living Program Formative Evaluation: Interview Guide (Program Staff and Management – KD&E)

Introduction:

TDV Global, a management consulting firm, has been engaged by the Public Health Agency of Canada (PHAC) to conduct a formative evaluation of the Healthy Living Program.  The purpose of the Formative Evaluation of the Healthy Living Program is to:

  • assess program design and its implementation;
  • assess early successes and progress;
  • assess performance measurement tools being used, including the Project Evaluation and Reporting Tool (PERT), and the Effectiveness of Community Interventions Project (ECIP);
  • identify lessons learned; and
  • make recommendations on key issues.

Formative evaluations early in the life of a program provides information about what is working well, but also facilitates any necessary corrections to the program which affect the ability of a program to achieve its desired outcomes.  The evaluation will provide valuable information about progress being made toward the Goals, Objectives and Outcomes which have been established for: the Healthy Living Fund; the Intersectoral Healthy Living Network; Knowledge Development and Exchange; and Social Marketing – which comprise the core components of the Healthy Living Program.

You have been selected to participate in this interview because of your involvement as program staff and/or management for the Knowledge Development and Exchange component of the Healthy Living Program.  The report and recommendations resulting from the evaluation will be used to improve the Knowledge Development and Exchange component and will be shared with you when finalized.

Discussion Guide:

Based on our evaluation framework, we will be interviewing key management and program personnel.  The interview should take approximately 45 minutes of your time.  Any direct quotes that are important for the analysis will be grouped into theme categories, to minimize the potential for identification of individual respondents. This being said, we cannot guarantee complete anonymity given the relatively small number of people being interviewed. Are you comfortable with proceeding with the interview?  We would like to discuss the Knowledge Development and Exchange component with you based on the questions on the next page. Your opinions are a very important element in evaluating the Knowledge Development and Exchange components’ progress to date as well as charting the course for future actions.  Please take some time to review and prepare your answers before the interview.

Should you feel that that there are important documents that would help you to answer the questions, we would appreciate it if you would provide us with a copy of the document.  We will also share with you the Knowledge Development and Exchange Logic Model to assist you in answering some of the questions.

Thank you.

Relevance

1.

(a)  There is a continued need for the Knowledge Development and Exchange component within the Healthy Living Program?  What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly…

2.

(a)  In your opinion, does the Knowledge Development and Exchange component “align” with the current priorities and directions of the following:

I. Public Health Agency of Canada/Health Canada/Health Portfolio
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly…

II. P/T Governments
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain your reasons briefly…

3.

(a)  There is no overlap/duplication across the four Healthy Living Program components: Healthy Living Fund; Knowledge Development and Exchange; Intersectoral Healthy Living Network; and Social Marketing? What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

(b) If you feel there is overlap/duplication, please provide examples…

4.

(a)  There are mechanisms in place to ensure that duplication does not occur between the Healthy Living Fund (Bilateral Agreements) and the Integrated Strategy on Healthy Living and Chronic Disease’s Functional Components. What is your opinion with regards to this statement?  (interviewer shows Functional Component Place Mat)

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  If you feel there are existing mechanisms please provide examples…

Design and Delivery

5.

(a) The overall design and the operating processes within the Knowledge Development and Exchange component is satisfactory. What is your opinion with regard to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain briefly what can be improved in the design and operating processes with the Knowledge Development and Exchange component

6.

Specifically, what is working well and what could be improved in the design and delivery of the Knowledge Development and Exchange component?

7.

(a)  What, if any early lessons learned in the overall design and delivery of the Knowledge Development and Exchange component can be identified?

(b) Please explain briefly…

8.

(a)  What, if any gaps in the overall design and delivery of the Knowledge Development and Exchange component can be identified?

(b) Please explain briefly…

Success/Progress

9.

(a)  Please provide an overview of the progress to date in the development of the Knowledge Development and Exchange outputs in regards to the following: (interviewer shows logic model)

  • Exchange and disseminate key findings through academic and other published media, international and domestic consensus conferences;
  • Maintenance of domestic and international collaborative partnerships;
  • Enhanced knowledge and surveillance products;
  • Intervention research related to healthy living;
  • Enhanced best practice portal to provide physical activity and healthy living content;
  • Technical papers, Journal articles, policy briefs, popular articles.

(b)  How are you measuring your progress?

10.

What has positively affected the progress of the Knowledge Development and Exchange component to date?

11.

What barriers have affected the progress of the Knowledge Development and Exchange component       process to date?

12.

Which of the following vulnerable populations are reached through the Knowledge Development and Exchange component:

  • children & youth;
  • Aboriginal Peoples;
  • isolated, rural and remote communities; and/or
  • others (please specify)?

13.

Has the developmental process for the Knowledge Development and Exchange component enhanced partnerships/collaborations to date with the following:

  • provincial/territorial governments;
  • private sector;
  • international groups;
  • CIHR; and/or
  • others (NGOs & alliances/coalitions)?

14.

Which of the following immediate results/outcomes is the Knowledge Development and Exchange component contributing to? (interviewer shows logic model)

  • Updated scientific evidence to protect integrity of physical activity guidelines and for development of clinical practice guidelines;
  • Updated status on Canadian physical activity levels;
  • Increased healthy living focus within domestic surveillance and research activities;
  • Objective data on physical activity levels of children and youth; and/or
  • Approved FPT physical activity targets for children and youth.

15.

(a) Have there been any unanticipated positive outcomes to date as a result of your activities?  (PROMPT – these are outcomes that are surprising, unforeseen or unplanned)

(b) Have there been any unanticipated negative outcomes to date as a result of your activities?  (PROMPT – these are outcomes that are surprising, unforeseen or unplanned)

Healthy Living Program Formative Evaluation: Interview Guide (PHAC Program Staff and Management – Social Marketing)

Introduction:

TDV Global, a management consulting firm, has been engaged by the Public Health Agency of Canada (PHAC) to conduct a formative evaluation of the Healthy Living Program. The purpose of the Formative Evaluation of the Healthy Living Program is to:

  • assess program design and its implementation;
  • assess early successes and progress;
  • assess performance measurement tools being used, including the Project Evaluation and Reporting Tool (PERT);
  • identify lessons learned; and
  • make recommendations on key issues.

Formative evaluations early in the life of a program provides information about what is working well, but also facilitates any necessary corrections to the program which affect the ability of a program to achieve its desired outcomes.  The evaluation will provide valuable information about progress being made toward the Goals, Objectives and Outcomes which have been established for: the Healthy Living Fund, the Intersectoral Healthy Living Network, Knowledge Development and Exchange, and Social Marketing – which comprise the core components of the Healthy Living Program.

You have been selected to participate in this interview because of your involvement with the Social Marketing component of the Healthy Living Program.  The report and recommendations resulting from the evaluation will be used to improve the Social Marketing component and will be shared with you when finalized.

Discussion Guide:

Based on our evaluation framework, we will be interviewing key management and program personnel.  The interview should take approximately 45 minutes of your time. Any direct quotes that are important for the analysis will be grouped into theme categories, to minimize the potential for identification of individual respondents. This being said, we cannot guarantee complete anonymity given the relatively small number of people being interviewed. Are you comfortable with proceeding with the interview?  We would like to discuss the Social Marketing component with you based on the questions on the next page. Your opinions are a very important element in evaluating the Social Marketing progress to date as well as charting the course for future actions.  Please take some time to review and prepare your answers before the interview.

Should you feel that that there are important documents that would help you to answer the questions, we would appreciate it if you would provide us with a copy of the document.  We will also share with you the Social Marketing Logic Model to assist you in answering some of the questions.

Thank you.

Relevance

1.

(a)  There is a continued need for the Social Marketing component within the Healthy Living Program?  What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain your reasons briefly…

2.

(a)  In your opinion, does the Social Marketing component “align” with the current priorities and directions of the following:

I. Public Health Agency of Canada/Health Canada/Health Portfolio
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly…

II. P/T Governments
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain your reasons briefly…

3.

(a)  There is no overlap/duplication across the four Healthy Living Program components: Healthy Living Fund; Knowledge Development and Exchange; Intersectoral Healthy Living Network; and Social Marketing. What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(c) If you feel there is overlap, please provide examples.

4.

(a) There are mechanisms in place to ensure that duplication does not occur between the Social Marketing component and the Integrated Strategy on Healthy Living and Chronic Disease’s Functional Components. What is your opinion with regards to this statement?  (interviewer shows Functional Component Place Mat)

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

(b) If you feel there are existing mechanisms please provide examples.

Design and Delivery

5.

(a) The overall design and the operating processes within the Social Marketing component is satisfactory. What is your opinion with regard to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain briefly what could be improved within the Social Marketing design and operating processes…

6.

Specifically, what is working well and what could be improved in the design and delivery of the Social Marketing component?

7.

(a)  What, if any early lessons learned in the overall design and delivery of the Social Marketing component can be identified?

(c) Please explain briefly…

8.

(a) What, if any gaps in the overall design and delivery of the Social Marketing component can be identified?

(c) Please explain briefly…

Success/Progress

13.

(a)  Please provide an overview of the progress to date in the development of the Social Marketing component activities/outputs in regards to the following: (interviewer shows logic model)

  • Public education awareness campaigns on physical activity and healthy eating;
  • Coordinating the promotion of physical activity community opportunities;
  • Physical Activity Guide promotion; and
  • Providing Canadians with useful information on healthy behaviours.

(e) How are you measuring your progress?

(f) What has positively affected the progress of the Social Marketing component to date?

(g) What barriers have affected the progress of the Social Marketing component to date?

14.

Which of the following vulnerable populations does the Social Marketing component reach:

  1. children & youth;
  2. Aboriginal Peoples;
  3. isolated, rural and remote communities; and/or
  4. others?

15.

Has the process for establishing the Social Marketing component enhanced partnerships/collaborations to date with the following:

  • provincial/territorial governments;
  • private sector;
  • CIHR; and/or
  • others (NGOs & alliances/coalitions)?

16.

(a)  Which of the following immediate outcomes/results is the Social Marketing component contributing to:   (interviewer shows logic model)

  • Increased awareness among Canadians about the importance of healthy living in the maintenance of healthy weights, and the attainment of overall good health; and
  • Increased awareness of Canada’s Physical Activity Guides.

(b) Please explain briefly… (PROMPT - how is the Social Marketing component contributing to the identified outcome/result?)

17.

(a) Have there been any unanticipated positive outcomes to date as a result of your activities?  (PROMPT – these are outcomes that are surprising, unforeseen or unplanned)

(b) Have there been any unanticipated negative outcomes to date as a result of your activities?  (PROMPT – these are outcomes that are surprising, unforeseen or unplanned)

            

Healthy Living Program Formative Evaluation: Interview Guide Healthy Living Fund (Bilateral Agreements - Management)

Introduction:

TDV Global, a management consulting firm, has been engaged by the Public Health Agency of Canada (PHAC) to conduct a formative evaluation of the Healthy Living Program.  The purpose of the Formative Evaluation of the Healthy Living Program is to:

  • assess program design and its implementation;
  • assess early successes and progress;
  • assess performance measurement tools being used, including the Project Evaluation and Reporting Tool (PERT);
  • identify lessons learned; and
  • make recommendations on key issues.

Formative evaluations early in the life of a program provides information about what is working well, but also facilitates any necessary corrections to the program which affect the ability of a program to achieve its desired outcomes.  The evaluation will provide valuable information about progress being made toward the Goals, Objectives and Outcomes which have been established for: the Healthy Living Fund, the Intersectoral Healthy Living Network, Knowledge Development and Exchange, and Social Marketing – which comprise the core components of the Healthy Living Program.

You have been selected to participate in this interview because of your involvement with the Bilateral Agreements of the Healthy Living Fund.  The report and recommendations resulting from the evaluation will be used to improve the Bilateral Agreement process and will be shared with you when finalized.

Discussion Guide:

Based on our evaluation framework, we will be interviewing key NCR, regional management and program personnel, as well as selected provincial/territorial partners.  The interview should take approximately 45 minutes of your time. Any direct quotes that are important for the analysis will be grouped into theme categories, to minimize the potential for identification of individual respondents. This being said, we cannot guarantee complete anonymity given the relatively small number of people being interviewed. Are you comfortable with proceeding with the interview?  We would like to discuss the Bilateral Agreements with you based on the questions on the next page. Your opinions are a very important element in evaluating the Bilateral Agreement progress to date as well as charting the course for future actions.  Please take some time to review and prepare your answers before the interview.

Should you feel that that there are important documents that would help you to answer the questions, we would appreciate it if you would provide us with a copy of the document.  We will also share with you the Healthy Living Fund Logic Model to assist you in answering some of the questions.

Thank you.

* Please Note – The “Bilateral Agreement process” includes both the negotiation and signature of the Bilateral Agreement and the identification and funding of projects associated with the Agreement.

Relevance

1.

(a)  There is a continued need for the Bilateral Agreements within the Healthy Living Fund?  What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain your reasons briefly…

2.

(a)  In your opinion, do the Bilateral Agreements “align” with the current priorities and directions of the following:

I.  Public Health Agency of Canada/Health Canada
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b) Please explain your reasons briefly…

II. P/T Governments
1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain your reasons briefly…

3.

(a)  There is no overlap/duplication across the four Healthy Living Program components: Healthy Living Fund; Knowledge Development and Exchange; Intersectoral Healthy Living Network; and Social Marketing? What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  If you feel there is overlap, please provide examples.

4.

(a)  There are mechanisms in place to ensure that duplication does not occur between the Healthy Living Fund  and the Integrated Strategy on Healthy Living and Chronic Disease’s Functional Components. What is your opinion with regards to this statement?  (interviewer shows Functional Component Place Mat)

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

(b)  If you feel there are existing mechanisms please provide examples…

Design and Delivery

5.

(a)  The overall process involved in developing and signing the Bilateral Agreements is satisfactory. What is your opinion with regards to this statement?

1
Strongly disagree
2
Disagree
3
Neither agree nor disagree
4
Agree
5
Strongly Agree
No
opinion
           

 

(b)  Please explain briefly what could be improved in the development and signing of the Bilateral Agreements?

6.

Specifically, what is working well and what could be improved in the projects associated with the Bilateral Agreements in regards to the following categories:

  1. Solicitation process:
    • Distribution and reach of the solicitation call.
    • Clarity of the Solicitation Guide.
    • Level of effort required to complete the Solicitation Guide.
    • Mechanisms to seek clarification (e.g., phone calls to regional PHAC staff, emails).
    • Timing and response requirements.
  2. Assessment of proposals and decision making by the Joint Coordinating Committees (i.e., PHAC and P/T government representatives):
    • Transparency of process.
    • Timeliness.
    • Opportunities for clarification.
    • Communication of proposal status.
  3. Establishment of contribution agreements and fund dispersal:
    • Timelines and communication.
    • Signing of the Contribution Agreement(s).
    • Appropriateness and thoroughness of the Contribution Agreement requirements.
    • Timely funding dispersal.
  4. Project monitoring and reporting requirements:
    • Procedures in place (PERT).
    • Level of effort required.
    • Opportunity for project adjustment.

7.

We would now like to discuss the overall process, inclusive of both the Bilateral Agreement signing process and the associated projects.

(a) What, if any early lessons learned can be determined from the Bilateral Agreement process including:

  • the development and signing process; and
  • implementation of the associated projects?

Please explain briefly…

8.

(a)  What, if any gaps can be identified from the Bilateral Agreement process including:

  • the development and signing process; and
  • implementation of the associated projects?

(b)  Please explain briefly…

Success/Progress

9.

(a)  Please provide an overview of the progress to date in the creation of the Bilateral Agreement outputs. (interviewer shows HLF logic model)

(b)  How are you measuring your progress?

(c)  What has positively affected the progress of the Bilateral Agreement process to date?

(d)  What barriers have affected the progress of the Bilateral Agreement process to date?

10.

Which of the following vulnerable populations do your Bilateral Agreements reach:

  • children & youth;
  • Aboriginal Peoples;
  • isolated, rural and remote communities; and/or
  • others (please specify)?

11.

Has the process for establishing the Bilateral Agreements and associated projects enhanced partnerships/collaborations to date with the following:

  • provincial/territorial governments;
  • private sector;
  • CIHR; and/or
  • others (NGOs & alliances/coalitions)?
      

12.

Which of the following immediate outcomes is the Bilateral Agreement contributing to? (interviewer shows HLF logic model)

  • increased inter-organizational, inter and intra sectoral, multi-jurisdictional engagement, alignment and collaboration;
  • increased community capacity to support the relationship between healthy eating and physical activity;
  • increased reach to target populations to enhance healthy eating and physical activity; or
  • increased knowledge dissemination including reports and research findings.

(b)  Please explain briefly… (How are the Bilateral Agreements contributing to the identified outcome/result?)

13.

(a) Have there been any unanticipated positive outcomes to date as a result of your activities?

(b) Have there been any unanticipated negative outcomes to date as a result of your activities?


Page details

Date modified: