ARCHIVED - Formative Evaluation of the Canadian HIV Vaccine Initiative

Evaluation Methodology

This section will outline the rationale for the evaluation, the evaluation objectives, questions and methodology. It concludes by identifying the limitations to this evaluation.

PHAC programs, such as the CHVI, are subject to evaluation to demonstrate their successes and show results in accounting for (health) returns on the investments. The “Integrated Result-Based Management and Accountability Framework and Risk-based Audit Framework” (RMAF/RBAF) called for a mid-term (formative) evaluation prior to the completion of the third year of the Initiative (i.e. before March 2010). The purpose of this evaluation was “to assess how effectively the initiative is being implemented and whether adjustments are necessary.”¹ The formative evaluation results were also expected to serve “as baseline data for the final summative evaluation.”²

The Request for Proposals (RFP) specified that the results of the evaluation would serve three major purposes:

• Provide a preliminary assessment of the progress made in the first two years of implementation – how well the CHVI is contributing to/advancing discovery and social research; clinical trial capacity building and networks; policy, regulatory and community and social dimensions, partnerships, committees, and knowledge sharing will be examined;
• Contribute to better decision-making around how best to deliver the CHVI and provide strategies for continuous improvement;
• Assess the continued relevance of the CHVI and the projects it supports, providing objective information on whether the projects funded and the program as a whole make sense in terms of the needs and priorities the program is intended to address. This examination will assist with decisions on the program’s future (key strategic directions, scope and allocation of resources, program priorities) and help ensure adequacy of response to public policy objectives.³

3.1 Evaluation Objectives

The objective of this formative evaluation was to assess the progress of the CHVI in the first two years of implementation and determine if changes are required to the design, delivery and direction of the activities.

The specific evaluation questions identified in the RFP for this evaluation were:

Implementation Progress

1. Is funding used fully, effectively and in keeping with plans and authorities?

2. What activities have been funded?

3. What client groups and organizations have received funding?

4. What research areas and what research priorities have received funding?

5. What delivery mechanisms have been used?

6. Is the program being delivered/ implemented as it was designed?

7. Is the governance structure effective?

8. Are collaborative arrangements between departments/ agencies/ Gates foundation effective?

9. Are there areas of duplication?

Achievement of Early Program Results

10. What progress has been made toward achievement of results?

11. What effect is the program having on beneficiaries?

12. Any unexpected results, whether positive or negative?

13. Is there a performance measurement system in place that involves all participating departments?

Role of CHVI and Ongoing Relevancy

14. What is required in the current HIV vaccine environment?

15. Consistency with Branch and partner organizations (other Government of Canada) mandate, objectives and priorities

16. Consistency with needs and priorities of primary beneficiaries and constituencies

Promising Practices and Lessons Learned

17. Would the CHVI benefit from a reallocation of funding and a change of emphasis between activities?

18. What worked well in the delivery of the research and community initiatives programs?

19. What were some of the challenges in the community initiatives program?

20. What worked well in the pilot scale manufacturing facility component?

21. What were some of the challenges in the pilot scale manufacturing facility component?

22. Is CHVI linking well with key stakeholders involved in the HIV vaccine field?

23. Is CHVI linking well with key stakeholders involved in other areas of the HIV prevention technology field?

24. Are there ways to improve program delivery from either an effectiveness or efficiency perspective?

However, at the outset of the evaluation, it was clear that it would not be possible to address some proposed questions. Given the delays in the implementation of some CHVI components, it was suggested that these questions be excluded from this formative evaluation, but be addressed in a later evaluation. This was agreed to by the CHVI Secretariat and reflected in the revised evaluation matrix. As a result, the following questions were excluded from the evaluation:

11. What effect is the program having on beneficiaries?

12. Any unexpected results, whether positive or negative?

18. What worked well in the delivery of the research and community initiatives programs?

21. What were some of the challenges in the pilot scale manufacturing facility component?

A table in Appendix B shows the linkages between the evaluation findings within the report and the evaluation questions.

3.2 Evaluation Methodology

The methodology for this evaluation was primarily qualitative and included two sources of information – documents and key informants.

Document Review

The team conducted a review of key program documents, including among others:

• CHVI RMAF/RBAF;
• CHVI presentations, the CHVI web site, annual reports and other program background documents;
• CHVI financial information;
• Selected funding documents (announcements, Letters of Invitation, etc.); and
• Interdepartmental Steering Committee minutes.

The complete list of documents reviewed is provided in Appendix C.

A preliminary review was conducted to match the documentation available to the evaluation questions. Then selected documents were reviewed in detail to address specific evaluation questions. As additional documentation became available during the evaluation, it was reviewed by the evaluation team.

Key Informant Interviews

There were two groups of key informants – internal GoC stakeholders and external stakeholders. Interviews, using a structured interview guide, were conducted with both groups:

• In-person or telephone interviews with 20 internal stakeholders from the PHAC CHVI Secretariat and all participating GoC departments (CIDA, IC, CIHR, PHAC, HC). These interviews were conducted in June and July 2009; and
• Telephone interviews with 20 international and domestic external stakeholders with expertise relevant to specific CHVI domains. Interviewees included representatives from international organizations with an interest in HIV vaccine development as well as knowledge in specific issues related to the development of HIV vaccines in and for LMICs; representatives from domestic HIV/AIDS non-governmental organizations (NGOs); domestic HIV/AIDS researchers; experts on pilot scale vaccine manufacturing facilities; and representatives from the private sector (biomedical industry). These interviews were conducted in July and early August 2009.

Details of the key informant interview methodology are found in Appendices D to F:

• Appendix D: a summary of the key informants interviewed;
• Appendix E: internal stakeholder interview guide; and
• Appendix F: external stakeholder interview guide.

3.3 Limitations of the Evaluation

As with any evaluation, there are limitations on the interpretation of the evaluation findings that arise from the evaluation methodologies. Given the stage of CHVI implementation, the heavy reliance on qualitative information and the varied nature of the key informant respondents, many findings of this evaluation are based on a narrow field of interview responses. This limitation was somewhat mitigated by the use of documents to substantiate the findings from the key informant interviews.

Reliance on Qualitative Information

The original design of this formative evaluation included a survey of beneficiaries to examine early impacts of the Initiative. However, it became clear, given the delays in implementation across all components, that a survey of beneficiaries to assess early impacts would be premature. As a result, the evaluation focused on CHVI program activities and processes, rather than the measurement of results. Given that there are different external stakeholders for each CHVI component, it was decided that the most effective way to explore process issues with a diverse group of stakeholders was through key informant interviews. Although these were conducted using structured interview guides, there were no predetermined indicators for certain key concepts in the evaluation issues (e.g. “collaboration arrangements”) and key informants defined their own expectations with respect to these concepts. As a result, it is possible that respondents had different ways of interpreting the concept and that the information collected is not consistent across all respondents.

Further, as a result of implementation delays, selection processes for some grants were still ongoing at the time of the evaluation. As a result, the evaluation team was not able to address the evaluation question related to the application processes. The evaluation findings thus rely, in some cases, on information from respondents with only general, second-hand knowledge of the application processes for specific grants.

In addition, since it was not possible to assess early impacts, this formative evaluation will not be able to provide the baseline data required for the next CHVI evaluation.

Limited Knowledge of Respondents

The evaluation team interviewed a variety of respondents with diverse expert knowledge of the CHVI components. Given the nature of the Initiative, with its diverse components, it was difficult for the evaluation team to interview stakeholders, other than a few internal stakeholders, who had sufficient knowledge of the Initiative as a whole. Since it was difficult for the evaluation team to find a large group of respondents who had experience with each aspect of the CHVI components, the team often had to rely on a limited number of responses to specific evaluation questions to develop the evaluation findings. This limited the team’s analysis of specific components and results in somewhat fragmented views of the Initiative, as a whole. The team attempted to confirm the different perceptions of CHVI stakeholders by relying on documents, where available, or confirm early findings with later interviews (also where this was feasible).

Limited Access to Key Informant Interview Respondents

Key informants for the evaluation were identified by the CHVI Secretariat on the basis of those who had the most knowledge of the Initiative. The evaluation team was given a list of potential participants to contact rather than creating this list in cooperation with CHVI and key informants. This was an issue when key informants identified other external stakeholders with whom they felt the evaluation team should speak but were not on the list chosen by CHVI Secretariat. In addition, some potential participants declined to be interviewed.

This may have affected the detail reflected in evaluation results since, as noted above, a limited number of respondents could comment on aspects of each component. As follow-up interviews were not conducted, when suggested by other respondents, this may have limited the range of perspectives identified to the evaluation team but, more likely, limited the level of detail available to substantiate the views on some issues.

1. Request for Proposal, Statement of Work
2. Ibid.
3. Ibid.