Evaluation of the Canadian HIV Vaccine Initiative 2009-2010 to 2014-2015
For readers interested in the PDF version, the document is available for downloading or viewing:
Office of Evaluation
Health Canada and the Public Health Agency of Canada
Table of Contents
- Executive Summary
- Management Response and Action Plan
- 1.0 Evaluation Purpose
- 2.0 Program Description
- 3.0 Evaluation Description
- 4.0 Findings
- 4.1 Relevance: Issue #1 - Continued Need for the Initiative
- 4.2 Relevance: Issue #2 - Alignment with Government Priorities
- 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
- 4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)
- 4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency
- 5.0; Conclusions
- 6.0 Recommendations
- Appendix 1 - References
- Appendix 2 - Logic Model
- Appendix 3 - Summary of Findings
- Appendix 4 - Evaluation Description
List of Tables
- Table 1: Planned Budget for 2007-08 to 2016-17 by Key Area of Focus ($M)
- Table 2: Limitations and Mitigation Strategies
- Table 3: Variance Between Planned vs Actual Spending 2010-11 to 2014-15 ($M)
This evaluation covered the Canadian HIV Vaccine Initiative (CHVI) for the period from 2009-10 to 2014-15. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation.
Evaluation Purpose and Scope
The purpose of the evaluation was to assess the relevance and performance of the Canadian HIV Vaccine Initiative. The evaluation was also designed to inform strategic renewal decisions prior to the Initiative's expiry in March 2017.
Established in 2007, the Canadian HIV Vaccine Initiative (CHVI) ($139M) is a collaboration between five Government of Canada departments/agencies ($111M) and the Bill & Melinda Gates Foundation (BMGF, $28M) to further strengthen global efforts to accelerate the development of HIV vaccines, and to reduce the spread of HIV/AIDS.
The five CHVI government partner departments/agencies are: the Public Health Agency of Canada (lead), Health Canada, the Canadian Institutes of Health Research (CIHR), Industry Canada, and the Department of Foreign Affairs, Trade and Development Canada (DFATD).
During the first three years (2007 - 2010), several changes occurred in the HIV vaccine development environment, including a renewed Scientific Strategic Plan developed by the Global HIV Vaccine EnterpriseEndnote i, and greater availability of global pilot scale manufacturing capacity (which had been a key focus of the 2007 CHVI). These changes led to an early renewal (in 2010) and reorganization of CHVI's activities. The overall objective to develop a safe and effective vaccine remained in the renewed initiative, and another objective to reduce the spread of HIV/AIDS (via a new component focused on Preventing Mother-to-Child Transmission of HIV (PMTCT)) was added.
While the global incidence of HIV has decreased over the last five years, HIV still affects a significant number of people, particularly in sub-Saharan Africa. In this region, where access to quality health services can be limited, mother-to-child transmission of HIV remains an issue. In Canada, the incidence of HIV has remained steady, and the prevalence is more concentrated among specific populations.
Despite recent advances in HIV prevention and treatment technologies, an HIV vaccine is still considered the most efficient and cost-effective means of eradicating the disease. While many of the challenges facing HIV vaccine development are unique to HIV (e.g., global variability of the virus, and lack of validated animal models), some of the broader challenges (both scientific and regulatory) are common across a number of different diseases. There has been a recent trend to take a broadened perspective on vaccinology, immunology, and regulatory issues so that discoveries can contribute to the development of vaccines for a broader range of diseases.
Alignment with Government Priorities
CHVI continues to broadly align with the Government of Canada's priorities, as identified in federal announcements and international agreements, as well as in partner departments'/agencies' strategic plans (e.g., research and innovation, maternal and child health).
Alignment with Federal Roles and Responsibilities
Partner departments/agencies have clear legislated and policy mandates for the work undertaken in CHVI. Moreover, the activities to address the Initiative's goals are consistent with those outlined by Cabinet authorities across all government departments involved in CHVI.
Achievement of Expected Outcomes (Effectiveness)
CHVI was successful in supporting collaborations among HIV vaccine researchers, both in Canada and in low- and middle-income countries (LMICs). Collaborations between the industry sector and academics were established, and there is a desire among some to further strengthen these ties. There was evidence to suggest that the capacity of researchers, health workers, and regulators to conduct and monitor HIV research was improved over the course of the Initiative. However, performance data was not consistently collected during the Initiative on the aspect of ‘capacity-building' and there was no clear target for what success would look like in the achievement of program objectives.
CHVI demonstrated progress towards longer term outcomes, such as improved policy frameworks among regulators in LMICs. While the prevention of mother-to-child transmission of HIV (PMTCT) projects were still early in implementation, they did demonstrate some progress in generating information that could inform improvements to PMTCT services in LMICs.
Overall, the activities of the initiative were well-coordinated across partners. A few areas were identified for improving strategic decision-making processes that are mainly related to the role of the Advisory Board and clarifying the specific priority areas that CHVI was expected to address (i.e., preventative and/or therapeutic vaccines, other HIV-related technologies, regulatory issues concerning vaccines for other diseases, etc.).
Demonstration of Economy and Efficiency
Since the 2010 renewal, budgets have been generally spent as planned for most key areas of focus. CHVI produced its outputs and achieved progress towards outcomes in an economical manner, often leveraging additional funds and utilizing cost minimization measures (e.g., utilization of pre-existing funding mechanisms) to ensure optimization of resource use.
Overall, the coordination of the initiative was effective; however, some areas of overlap were identified between the CHVI Secretariat and the Alliance Coordinating Office (ACO). The collaboration between the GoC and the Bill & Melinda Gates Foundation (BMGF) was positive, and in certain cases, facilitated the initiation of subsequent collaborations on other priority areas outside HIV (e.g., Ebola and Hepatitis C). The performance measurement strategy was not fully implemented and since no baseline data were collected, it was difficult to assess the magnitude of any ‘increases' or ‘improvements' in the program outcomes.
Revisit objectives and goals of CHVI within the current context.
In the current context, there is no HIV vaccine and no vaccine is expected to be licensed and available for use within the next several years. However, there are new preventative technologies that have been used to extend the lives of HIV-infected persons, and vaccine research has advanced. Future funding considerations should build on the work conducted through CHVI and consider shifts in approaches for vaccine research, including the broader federal approach to vaccine research and development.
Greater clarity of priorities would also inform revisions to the performance measurement strategy, and identify clear definitions and targets for the expected outcomes. Throughout the course of the evaluation, it was difficult to demonstrate progress as there were no baseline measures to determine changes in outcomes, goals or objectives. The CHVI Secretariat should be responsible for rolling up performance information and providing a whole-of-initiative perspective on progress towards outcomes.
Enhance efficiencies concerning governance.
The governance approach within CHVI has improved since the 2009 evaluation, and the coordination within the Initiative was perceived to be effective by both internal and external key informants. However, there were still areas of overlap identified and opportunities to improve the current governance include:
- streamlining decision-making (i.e., in terms of approving CHVI proposals) and re-examining the mandate of the Advisory Board in order to maximize the members' expertise (i.e., advising on the scope or direction of CHVI's future activities and funding opportunities); and
- clarifying the roles and mandates of the CHVI Secretariat and ACO to minimize overlapping activities.
Management Response and Action Plan
|Recommendations||Response||Action Plan||Deliverables||Expected Completion Date||Accountability||Resources|
1. Revisit objectives and goals of CHVI within the current context.
Agree with conditions.
Individual departments may review their participation in this initiative at the end of the current MOU. Departments that decide to continue this partnership, as well as any new department that may join from now on, agree to review goals and objectives as part of the renegotiation of an MOU with the Bill and Melinda Gates Foundation.
GoC departments/ agencies will review current goals and objectives and propose potential common objectives for a possible renewed MOU with the Bill & Melinda Gates Foundation. The Gates Foundation will be approached to determine their interest in potential common objectives in a renewed collaboration.
GoC partners would collaborate and revise the logic model to reflect revised goals and objectives and identify common outcomes.
1.1 Potential common objectives for a renewed MOU.
ADM - IDPCB
2.5 FTEs from existing budget (PHAC)
1.2 Advice to Ministers of Health, Foreign Affairs, Trade and Development Canada, and Industry Canada (and Cabinet) on the possible renewal of the MOU.
DG - BGTD, HPFB (HC)
Director, Strategic Initiatives Branch
DG - GID
Director - PSD
1.0 FTE from existing budget (CIHR)
As a partner in this horizontal initiative, BGTD will support PHAC until the end of March 2016 as it addresses this recommendation. Support for activities beyond this date would be contingent on a renewed MOU and new funding (HC)
1.3 A Revised Logic Model
0.25 FTE from existing budget (DFATD)
1.1 FTE per year for the evaluation and implementing the MRAP (IC, NRC-IRAP)
GoC partners would collaboratively revise the Performance Measurement Strategy (PMS), including program profile, performance measurement framework and associated baselines, targets, where applicable, and data collection strategies.
1.4 A Revised Performance Measurement Strategy
2. Enhance efficiencies concerning governance.
Building on the existing CHVI governance model, GoC partners will revisit the governance structure as part of the possible renewal of the CHVI:
2.1 An overall governance strategy for CHVI, including a schematic of the new governance structure.
2.2 Revised Terms of Reference for all internal and external bodies.
ADM - IDPCB
DG - BGTD, HPFB (HC)
Director, Strategic Initiatives Branch
DG - GID
Director - PSD
Same FTEs as above:
2.5 FTEs from existing
1.0 FTE from existing budget (CIHR)
As a partner in this horizontal initiative, BGTD will support PHAC until the end of March 2016 as it addresses this recommendation. Support for activities beyond this date would be contingent on a renewed MOU and new funding (HC).
0.25 FTE from existing budget (DFATD)
1.1 FTE per year for the evaluation and implementing the MRAP (IC, IC-IRAP)
1.0 Evaluation Purpose
The purpose of the evaluation was to assess the relevance and performance of the Canadian HIV Vaccine Initiative for the period of 2009-10 to 2014-15.
The evaluation was required by the Financial Administration Act (for Grants and Contributions) and the Treasury Board of Canada's Policy on Evaluation.The evaluation was also designed to inform strategic renewal decisions since the Initiative is set to expire in March 2017.
2.0 Program Description
2.1 Program Context
In 2007, the Canadian HIV Vaccine Initiative (CHVI) ($139M) was established to implement a Memorandum of Understanding (MOU) signed between the Government of Canada (GoC) ($111M) and the Bill & Melinda Gates Foundation (BMGF) ($28M) to contribute to the development and delivery of HIV vaccines. Aligned with the Global HIV Vaccine EnterpriseEndnote ii (GHVE), CHVI is part of Canada's global commitment to accelerate the development of safe, effective, affordable and globally accessible HIV vaccines especially in low- and middle-income countries (LMICs) and in Canada.
By 2010, several changes had occurred in the HIV vaccine development environment. There was a renewed GHVE Strategic Scientific Plan developed to guide global HIV vaccine research, and greater global capacity in pilot scale manufacturing. This led to an early renewal and reorganization of CHVI's activities. The overall objective to develop a safe and effective vaccine remained in the renewed initiative, and another objective to reduce the spread of HIV/AIDS (via a new component focused on Preventing Mother-to-Child Transmission of HIV (PMTCT)) was added. A new governance structure was established (including a CHVI Advisory Board and the CHVI Alliance Coordinating Office (ACO)). CHVI was extended until 2016/17 but no new budget funds were allocated.
2.2 Program Profile
CHVI is a horizontal initiative comprising five federal government departments/agencies in joint collaboration with the Bill & Melinda Gates Foundation.
- Public Health Agency of Canada (the Agency) ($18M) is the lead agency and is responsible for supporting domestic and international efforts related to the research and development of an HIV vaccine, the development of the HIV Vaccine Translational Support fund, and the CHVI ACO.
- Health Canada (HC) ($5M) is responsible for supporting regulatory readiness and strengthening the capacity of regulatory authorities in LMICs in the area of vaccine research and clinical trial evaluations (through training and mentorship) and exchange of best practices, policies, and protocols related to regulation of vaccines, with a focus on HIV vaccines.
- Canadian Institutes of Health Research (CIHR) ($15M) is responsible for advancing the basic science of and building capacity for HIV vaccine discovery and social research in Canada and LMICs. This was accomplished through funding research grants to HIV vaccine researchers in Canada and LMICs.
- Industry Canada transferred funds ($13M) to the National Research Council (NRC) which is responsible for supporting new and innovative technologies for prevention, treatment and diagnosis of HIV in pre-commercial development via the Canadian HIV Technology Development (CHTD) program (a component of NRC's Industrial Research Assistance Program (IRAP)).
- Foreign Affairs, Trade and Development Canada (DFATD) ($60M) is responsible for increasing capacity to conduct high-quality clinical trials of HIV vaccine in LMICs through new teams of Canadian and LMIC researchers and institutions, increasing access to high quality PMTCT services, and improving capacity, promoting involvement and collaboration in HIV vaccine discovery in Canada and in LMICs.
The activities of the renewed CHVI were organized under the following key areas of focus:
- Advancing Basic Science (ABS) of HIV vaccine discovery and social research in Canada and LMICs;
- Translating Basic Science (TBS) discoveries into clinical research with a focus on accelerating clinical trials in humans;
- Addressing Enabling Conditions (AEC) to facilitate regulatory approval and community preparedness;
- Preventing Mother-to-Child Transmission of HIV (PMTCT) by enhancing the accessibility, quality, and uptake of services in LMICs; and
- Supporting Coordinated Efforts (SCE) to enable horizontal collaboration within CHVI and with domestic and international stakeholders.
The CHVI Secretariat (housed at the Agency) is responsible for ensuring horizontal coordination across partner departments/agencies and with the BMGF; providing coordinating policy advice on CHVI-related issues to partner departments/agencies and the BMGF; and overseeing the contribution agreement with the International Center for Infectious Diseases (ICID) to operate the CHVI ACO (described below). The CHVI Secretariat also led the establishment of the Advisory Board and the CHVI ACO, in collaboration with GoC CHVI partner departments/agencies and the BMGF.
The CHVI Advisory Board was composed of three external experts, appointed by the GoC CHVI Ministers; three representatives from the BMGF; the Director of the CHVI ACO; and one (non-voting) representative from each of the GoC CHVI partner departments/agencies. The external experts were drawn from the business, scientific, and international development sectors to provide advice to CHVI Ministers and the BMGF. The role of the CHVI Advisory Board is to provide strategic direction and recommendations on projects to be funded by CHVI; to oversee the implementation of the MOU; and to ensure that activities funded were aligned with the Key Areas of Focus identified in the MOU.
The CHVI ACO was established in December 2011 via a contribution agreement with the ICID. It was created to provide administrative support for the CHVI Advisory Board, as well as to create and promote a network of HIV researchers via the CHVI Research and Development (R&D) Alliance (the Alliance) to help ensure that Canada is a leading contributor to the global HIV vaccine research effort.
2.3 Program Logic Model and Narrative
One or more GoC partner departments/agencies are responsible for activities to address each of the five key areas of focus noted above. Through these activities, CHVI seeks to achieve a number of expected outcomes, namely:
- Improved collaboration, networking, and knowledge-sharing;
- Increased capacity among researchers and regulators to conduct HIV vaccine research, and among health workers to deliver Prevention of Mother-to-Child Transmission of HIV (PMTCT) services;
- Improved policy and regulatory processes/ frameworks;
- Improved implementation of quality PMTCT services;
- Increased demand for and use of PMTCT services by women and their families;
- Increased strategic decision-making; and
- Increased R&D in Canada and in Low and Middle Income Countries (LMICs).
It was expected that these outcomes would contribute to global efforts to accelerate the development of safe, effective, affordable and globally accessible HIV vaccines, as well as to help reduce the spread of HIV (particularly in LMICs).
The connections between CHVI activities and the expected outcomes are illustrated in the logic model (Appendix 2). The evaluation assessed the degree to which the defined outputs and outcomes had been achieved over the timeframe covered by this evaluation.
2.4 Program Alignment and Resources
Considering that the Agency is the federal lead of the Initiative, CHVI is part of the Public Health Agency's 2014-15 Program Alignment Architecture (PAA): program 1.2 Health Promotion and Disease Prevention, sub-program 1.2.2 Infectious Disease Prevention and Control, and sub-sub program 188.8.131.52 Infectious and Communicable Diseases. This program contributes to the prevention and control of infectious diseases by monitoring emerging and re-emerging infectious diseases that are identified by the Agency as leading causes of hospitalization and death in Canada, and by developing strategic approaches to reduce the likelihood of infection. While not explicitly mentioned, CHVI activities are also covered in the PAAs of partner departments.
The breakdown of the budget for fiscal years 2007-08 through 2016-17 is presented below (Table 1). The total GoC portion of CHVI's budget was $111M over 10 years.
|Areas of Focus||PHAC||HC||CIHR||IC||DFATD||Total|
3.0 Evaluation Description
3.1 Evaluation Scope, Approach and Design
The scope of the evaluation covered the period between April 2009 to December 2014, and included activities under the five Key Areas of Focus (i.e., Advancing Basic Science, Translating Basic Science, Addressing Enabling Conditions, Supporting Coordinated Efforts, and Preventing Mother-to-Child Transmission of HIV). The evaluation did not assess activities solely funded by the Bill & Melinda Gates Foundation, or other HIV-related activities funded under separate programs (e.g., Federal Initiative to Address HIV/AIDS in Canada).
The evaluation issues were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 3. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.
The Treasury Board's Policy on Evaluation (2009) guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the policy. A non-experimental design was used based on the Evaluation Framework document, which detailed the evaluation strategy for this Initiative and provided consistency in the collection of data to support the evaluation.
Data for the evaluation was collected using various methods: a literature review, a document review, key informant interviews (internal and external stakeholders), case studies of five Large Teams Grants, and a web survey of funding recipients. More details on the data collection and analysis methods can be found in Appendix 4. In addition, data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.
3.2 Limitations and Mitigation Strategies
Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings are reliable and can be used with confidence to guide program planning and decision-making.
|Limited baseline and outcome data, and variable stages of implementation (e.g., PMTCT projects were mid-implementation)||Data gaps, particularly for longer term outcomes||Conducted a survey of funding recipients and case studies to augment evidence for assessing outcomes and incremental changes.|
|Limited ability to survey Mentorship Program participants - limited internet availability in low and middle income countries||Potential negative impact on survey response rates||Conducted an in-person group interview with three participants attending Health Canada's International Regulatory Forum in October 2014.|
|Complex program with multiple partners each with their own expected outcomes||Difficult to collect and report data in a coherent manner||Developed set of shared outcomes based on thematic areas in logic model.|
This section will present the findings for the relevance and performance of CHVI, including its progress towards expected outcomes under each of the five Key Areas of Focus. Where appropriate, findings for each partner department/agency are reported separately.
4.1 Relevance: Issue #1 - Continued Need for the Initiative
In Canada, the incidence of HIV has remained relatively steady over the last five years (5.9 new cases per 100,000 people) and tends to be concentrated among specific populations. While the global incidence of HIV has decreased over the last five years, HIV still affects a significant number of people, particularly in Africa. While no effective HIV vaccine currently exists, it is still considered to be the most efficient way to eradicate the disease.
In Canada, HIV is considered to be a concentrated, low-level epidemic. The rate of new HIV infections (incidence) has remained steady. The Public Health Agency of Canada reported that in 2011 there were an estimated 3,175 new HIV infections in Canada and approximately 71,300 Canadians living with HIVEndnote 1. On the other hand, between 2008 and 2011, the number of people living with HIV (prevalence) rose 11.4 percentEndnote 2. The increased prevalence is in part attributable to improvements in treatments resulting in reductions in the number of AIDS-related deathsEndnote 3.
The populations most affected by HIV in Canada include: gay and other men who have sex with men, people who inject drugs, Aboriginal people, and people from countries where HIV is endemic, people in prison, women, and at-risk youth. In 2011, gay and other men who have sex with men accounted for most of the new infections in Canada, representing 49 percent of all positive HIV tests with a known exposure categoryEndnote 4.
Internationally, the burden of HIV/AIDS continues to be significant: 35 million people live with HIV globally, 24.7 million (70 percent) in Sub-Saharan AfricaEndnote 5. Nearly one in every 20 adults in Sub-Saharan Africa is living with the virusEndnote 6. Most of the countries in that region are low- and middle- income countries (LMICs), where access to quality health services (including HIV treatment services) is usually low and mother-to-child transmission of HIV remains an issue. Without prophylactic treatment, approximately 15 to 30 percent of infants born to HIV-positive mothers will become infected with HIV during pregnancy, labour and delivery, and 5 to 15 percent will become infected during breastfeedingEndnote 7. A significant proportion of mothers and children in sub-Saharan Africa are unable to access needed interventions.
In recent years, the number of people newly infected with HIV has declined in most parts of the world. In 2013, there were 2.1 million new HIV infections, a decline of 38 percent from 2001 (3.4 million new infections)Endnote 8. This decline is attributed to the rapid increase in the number of people on antiretroviral therapy and thus, less likely to transmit the virus. Despite such success, 22 million people living with HIV (60 percent) are still not accessing antiretroviral therapyEndnote 9. Hence, while there have been advances in treatment and prevention strategies, an HIV vaccine is still viewed as the most efficient and cost-effective method to eradicate HIVEndnote 10.
Challenges in HIV Vaccine Development
Vaccine development is a long, complex process that utilizes the clinical trial format, requiring several phases, each of which can take up to several years to complete. Vaccines must undergo several stages of rigorous testing before they are approved for use, ensuring its quality, safety and efficacyEndnote 11. For example, a vaccine for the Human Papillomavirus (HPV) was initiated in the mid-1980s and the US Food and Drug Administration approved the first preventative HPV vaccine in 2006. Vaccine development also needs to be supported by research at the front but also production and distribution at the final stages to ensure mass vaccination, all of which require additional time.
Developing an HIV vaccine has been more challenging compared to other vaccines for a variety of scientific reasons, including the global variability of the virusEndnote 12, HIV's unique ability to infect and compromise the immune systemEndnote 13, the lack of a natural protective immune response to HIV (although some populations seem to be naturally immune)Endnote 14, and the lack of valid animal models that accurately predict the human immune responseEndnote 15. All this has made advancing clinical trials difficult. As a result, of the more than 200 Phase I trials of HIV vaccine trials conducted to date (globally), only six have progressed to Phase IIb/ IIIEndnote 16.
Social and institutional barriers also present challenges for HIV vaccine development, such as difficulties recruiting volunteers to participate in HIV vaccine clinical trialsEndnote 17 Endnote 18, and insufficient capacity among African National Regulatory Authorities to review and monitor clinical trials taking place in Africa.
Recent Developments in HIV Vaccine Research
While no HIV vaccine has been developed to date, there have been developments in the HIV vaccine field since CHVI was launched in 2007. In one Phase III trial (RV144), the HIV vaccine candidate showed modest efficacy (31 percent), which was more promising than the results indicated by any other candidate. Building on the success of the RV144 trial that took place in Thailand, the Pox-Protein Public-Private Partnership (P5) was established in 2010, in order to pursue an RV144-like regimen (Pox-Protein) and ultimately license an HIV vaccineEndnote 19. The P5 stakeholders chose South Africa as the setting for clinical research, since the region has a high incidence of HIV/AIDS (is highly impacted by the HIV epidemic)Endnote 20. A new trial is scheduled to begin in 2016 in South Africa.
There are several other vaccine candidates using a variety of approaches (e.g., viral vector vaccines, DNA vaccines, protein vaccines) currently in the HIV vaccine pipeline which may prove promising as well. For instance, a Phase I clinical trial SAV001-H is based on a genetically modified killed whole-virus vaccine. This trial was completed in December 2013 and the complete data became available in March 2014. There were no serious adverse effects in 33 participantsEndnote 21 and significant antibody titres to two viral antigens were detected over a period of several months in the group immunized with killed virus plus adjuvant. The firm is currently conducting additional pre-clinical studies and is developing a Phase II protocol.
Beyond HIV vaccines, a number of other advances have been made in the prevention of HIV. New HIV prevention technologies (NPTs) have been developed including oral and topical regimens used for PrEP (Pre-exposure prophylaxis) that might increase use to prevent infection for those at risk. New microbicides are also being developed, for instance CAPRISA 004, which is the very first gel-based microbicide whose protective efficacy has been established. While NPTs are being developed and used, HIV vaccine trials are becoming more complex and expensive as these NPTs need to be incorporated in the study designs.
All of the above factors have contributed to the recent trend of considering integrated aspects of HIV prevention research. In 2014, the world's first global scientific meeting on biomedical HIV prevention research (Research for Prevention (HIV R4P) conference) brought together researchers working in all areas of HIV prevention research (both vaccines and NPTs), rather than having separate conferences as in previous years. The conference's broader focus on HIV prevention could potentially help researchers better understand HIV immunology and encourage researchers to find solutions for cross-cutting issues.
Another emerging trend is the increasing attention being paid to HIV cure research. In 2012, a group of researchers from the International AIDS Society launched a Global Scientific Strategy: Towards an HIV Cure. A combination of new HIV treatments and therapeutic vaccines could become key components in the development of an HIV cure. Since then, there has been an increase in resources for HIV cure research. The 2014 HIV Vaccines and Microbicides Resource Tracking Working Group report estimated that there had been a 16 percent increase in global funding for HIV cure research from 2012 to 2013 to a total of $102.7 million (USD)Endnote 22. During the same time period, funding for preventive HIV vaccine research and development declined by three percent to $818 million (USD), however, Canada's contribution, largely through CHVI, actually increased from $15M (USD) in 2012 to $16.3M (USD) in 2013.
4.2 Relevance: Issue #2 - Alignment with Government Priorities
CHVI broadly aligns with current Government of Canada priorities as identified in federal announcements and international agreements, as well as partner departments'/agencies' priorities described in their strategic plans. The priorities focus on reducing the international HIV/AIDS burden, improving maternal health, and supporting research and development.
Alignment with Government of Canada Priorities
CHVI continues to align with the federal government's priorities to help alleviate international HIV/AIDS burden, to improve maternal health, and support research and development. In 2002, Canada committed to international assistance to contribute to the global effort to reach the Millennium Development Goals. This commitment was reiterated in 2005 at the G8 summit. The fifth goal of the Millennium Development Goals is to “improve maternal health” and the sixth goal is to “combat HIV/AIDS, malaria and other diseases”, both of which align with CHVI's key areas of focus.
The 2011 Speech from the Throne stated that the “Government will continue to make targeted investments to promote and encourage research and development”. Budget 2014 reconfirmed the Government of Canada's commitment to “supporting advanced research and innovation” and its plan to support “leading-edge research though the granting councils” in institutions such as the CIHR. Hence, with support for research and development as one of its main components, CHVI's activities were aligned with these federal priorities.
Moreover, Budget 2014 underlined the Government of Canada's continued commitment to contribute to the fight against AIDS and other infectious diseases, and to improve maternal and child health through two key commitments: the Global Fund to Fight AIDS, Tuberculosis and Malaria (2013), and the Muskoka Initiative for maternal and child health. Although these initiatives are not part of CHVI, they help illustrate the importance that the GoC continues to place on issues of HIV/AIDS and maternal health.
Alignment with Partner Departments'/Agencies' Priorities
PHAC and partner departments'/agencies' priorities were aligned with those of CHVI as demonstrated in partner departments'/agencies' annual Report on Plans and Priorities (RPP) and strategic plans over the period being evaluated. These priorities included those related to HIV vaccine development, regulatory harmonization of vaccine processes, HIV research, advancements in life science technologies, and maternal and child health abroad.
4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
CHVI's mandate aligns with the mandate of each of the federal government partners. It is an appropriate role for the federal government to help reduce the international HIV/AIDS burden, to improve maternal health, and to support research and development.
Public Health Agency of Canada
The Public Health Agency of Canada (the Agency) has the Cabinet authority to accelerate the development of a safe and effective HIV vaccine by building on Canada's scientific excellence for the benefit of those most in need (particularly in Africa but also in Canada). The Agency's role in CHVI is appropriate and its activities are adequately aligned with the authorities.
According to the Public Health Agency of Canada Act, the Agency was established for the purpose of assisting the Minister in exercising or performing the Minister's powers, duties and functions in relation to public health. It also explicitly supports international health engagement with foreign governments and international organizations. The Department of Health Act identifies, among other responsibilities, the following areas of federal public health responsibility:
- the promotion and preservation of the physical, mental and social well-being of the people of Canada;
- the protection of the people of Canada against risks to health and spreading of diseases; and
- investigation and research into public health, including the monitoring of diseases.
These align with the Agency's role in CHVI in supporting domestic and international efforts related to the research and development of an HIV vaccine, the HIV Vaccine Translational Support Fund, and the CHVI ACO.
The mandate of Health Canada's Office of Policy and International Collaboration (OPIC) within the Biologics and Genetic Therapies Directorate (Health Products and Food Branch) is to develop new and update existing policies, standards, guidelines, directives and other legislative or regulatory instruments regarding biologics and radiopharmaceuticals. The Office also coordinates international collaboration and activities for the Directorate. This mandate aligns directly with HC's role in the CHVI to:
- support regulatory readiness and strengthen the capacity of regulatory authorities in LMICs in vaccine production and clinical trial evaluations (through training and mentorship); and
- exchange best practices, policies, and protocols related to the regulation of vaccines, with a focus on HIV vaccines.
Canadian Institutes of Health Research
The Canadian Institutes of Health Research Act states that the objective of CIHR is to “excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products”. This mandate aligns with CIHR's responsibilities under CHVI which are to:
- advance the basic science of and build capacity for HIV vaccine discovery and social research in Canada and LMICs by funding research grants to HIV vaccine researchers in Canada and LMICs, including Large Teams (in collaboration with DFATD);
- increase the number of young Canadian and LMIC vaccine researchers; and
- enhance linkages among researchers via networking and information sharing.
The mandate of Industry Canada is to help make Canadian industry more productive and competitive in the global economy. This mandate aligns with IC's responsibility under CHVI to:
- support new and innovative technologies for the prevention, treatment, and diagnosis of HIV in pre-commercial development through the National Research Council's (NRC) Industrial Research Assistance Program (IRAP) Canadian HIV Technology Development (CHTD) program.
Foreign Affairs, Trade and Development Canada (formerly the Canadian International Development Agency)
The Canadian International Development Agency's (CIDA) mandate was to manage Canada's international assistance effectively and accountably to achieve meaningful, sustainable development results, and to engage in policy development in Canada and internationally, enabling Canada to realize its development objectives. CIDA was merged with DFATD in 2013. One of DFATD's current mandates is to lead Canada's international development and humanitarian assistance. This mandate aligns with DFATD's responsibilities in CHVI, which are to:
- increase the capacity to conduct high-quality clinical trials of HIV vaccine in LMICs through new teams of Canadian and LMIC researchers and institutions;
- increase access to high quality PMTCT services; and
- promote capacity, involvement, and collaboration in HIV vaccine discovery in Canada and in LMICs.
4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)
CHVI activities are making progress towards their immediate and some intermediate outcomes under each of the five Key Areas of Focus.
The Initiative is progressing as planned and early achievements have been identified. A few areas were identified for improving strategic decision-making processes that are mainly related to the role of the Advisory Board and clarifying the specific priority areas that CHVI was expected to address (i.e., preventative and/or therapeutic vaccines, other HIV-related technologies, regulatory issues concerning vaccines for other diseases).
There was some evidence that the capacity of researchers, health workers, and regulators to conduct or monitor HIV research improved over the course of the Initiative. However, it was difficult to demonstrate progress as there were no baseline measures to determine outcome achievement.
4.4.1 To what extent have the outcomes for Advancing Basic Science of HIV vaccines (ABS) been achieved?
The aim of the Advancing the Basic Science of HIV vaccines (ABS) area was to support basic discovery and social research activities in HIV vaccine research. Activities funded under ABS supported collaboration among Canadian and LMIC researchers allowing them to contribute new knowledge to global HIV vaccine development efforts. CIHR and DFATD were the primary departments/agencies responsible for activities in this area.
Collaboration and Networking
There was evidence of collaborative activities between researchers in Canada as well as between researchers in Canada and in LMICs. Since the Initiative was launched in 2007, a total of 51 grants were funded, including 15 catalyst grants (one year in duration), 17 operational grants (three to five years in duration), and five large team grants (four to five years in duration) for a total of $25.73M. Some of these grant fund mechanisms had a stronger emphasis on collaboration than others, for example, the team grants, travel grants, and some of the catalyst and operating grants that included teams of co-investigators. The large team grants, in particular, were required to have a collaborative team that comprised both Canadian and LMIC researchers. The travel grants allowed Canadian researchers to participate in vaccine conferences (including a Partnership Development Forum), enabling them to network with LMIC researchers. Half of these travel grants led to the development of successful large team grant applications.
Topics examined by researchers included biomedical and social aspects of the discovery and development of preventative and therapeutic HIV vaccines, as well as other treatment strategies. Canadian and international researchers worked collaboratively on projects originating from different disciplines. A total of 17 investigators from Africa, three from India, 10 from the US, one from New Zealand and one from Switzerland were formally included on CHVI-funded research projects. Results from a survey of CHVI-funded Primary Investigators indicated that 15 of 22 respondents had collaborated with international researchers and experts. Large team grants were good examples of collaborative relationships that resulted from CHVI funding, both within and between teams. Within teams, multidisciplinary researchers worked together on large team grant projects. Between teams, researchers shared knowledge with one another, particularly through the Afri-Can forum.
Canadian researchers interviewed as part of the CIHR case studies identified that working with LMIC researchers was critical to the success of the projects because LMIC researchers and partners provide local insight into the project, as well as complementary skills and expertise from researchers working at the study sites. Conversely, LMIC research teams benefit from collaborating with Canadian researchers because they are able to access additional resources and expertise that may not be available in the countries where the study sites are located. While the different skill sets and expertise present in each country have provided learning and training opportunities for most research teams, a common theme across the case studies that was noted by both Canadian and LMIC research team members was the importance of conducting the projects within a culturally appropriate context. Having researchers from LMICs on the team was helpful to ensure the cultural context was taken into consideration.
Outside of the team grants, other collaborative relationships were developed between members of the teams. Survey results indicated that Principal Investigators worked together with co-investigators and other members of the team that were usually also from a biomedical science background. Also, 19 of 22 and 15 of 22 respondents reported working with other researchers and experts in HIV vaccine research. A majority (20 of 22) believed that collaborations built by the project would continue after funding ended. In spite of the positive response, challenges to collaboration were cited by the grant recipients. According to the survey, the three most commonly reported barriers to collaborations were human resources constraints (10 of 22), financial controls (7 of 22), and time constraints (6 of 22), respectively.
Studies conducted in LMICs by experienced and knowledgeable LMIC researchers provided insight into the cultural context and barriers, especially in areas most impacted by HIV/AIDS. This insight facilitated access to target populations through relationships with local health organizations (e.g., community centres). The knowledge of these local community-based organizations was leveraged, as the studies ensured that values and interests of key stakeholders were taken into account in the research plan. For example, staff in a project at a trial site in Cape Town, South Africa, ensured that sensitivities in a post-apartheid context were taken into account in the research plan. Local partners were leveraged to help create intervention materials that were accessible to the local population.
While it is evident that collaborative activities took place during the period assessed by the evaluation, it was difficult to assess the impact CHVI funding had on collaboration since there were no established baselines and indicators to assess and measure increased collaboration.
Improved Knowledge Sharing
In terms of advancing the field of biomedical HIV vaccine research, CHVI-funded projects investigated novel ideas, methods, and approaches that generated new information and knowledge. Specifically, some projects challenged previously established vaccine models, and proposed new diagnostic approaches and tools for HIV vaccine research (e.g., research assays, analysis tools, and measures of mucosal antibody production). A recent CIHR case study indicated that a large team grant project succeeded in developing new tools and approaches to be used in HIV vaccine research. In terms of social science HIV vaccine research, one project focused on advancing key concepts of informed consent and determining priorities and concerns of the end-users of a HIV vaccine.
Research results were published, presented and disseminated through various academic channels. Across all the projects included in the survey, respondents reported producing various outputs -mainly journal articles and presentations. A CIHR case study showed that one large team project was able to present its findings to other researchers working on CHVI projects through meetings hosted by the ACO, the Canadian Association for HIV Research (CAHR) and a local community workshop in Botswana, respectively. Such dissemination mechanisms helped share discovery and innovative knowledge with stakeholders in the HIV vaccine field. Overall, CHVI funding allowed researchers to conduct research, disseminate the results, and contribute to global knowledge of HIV vaccine development.
While there was evidence that research results were created and disseminated, the impact of these activities could not be ascertained in the evaluation. However, there is some evidence to suggest that findings are beginning to have influence in the field. For example, a CIHR case study showed that one team published an article in an open access journal that received hundreds of hits. The Nominated Principal Investigator from the same team has also been invited to give a presentation of early findings to the National Institutes of Health. Bibliometric information could have helped determine what impact CHVI-funded studies had in HIV vaccine research. However, since many of the research projects were still mid-implementation, it was too early to complete a bibliometric analysis for this evaluation. In addition, despite evidence to support the advancement of novel ideas and approaches, the basic science of HIV vaccine research has not advanced sufficiently to bring vaccine candidates to clinical trial in Canada within the past five years. A paper produced about the translational fund noted that the CHVI ACO found that 11 Canadian researchers had HIV vaccine candidates under development. Three of these 11 researchers have been funded by CIHR under the auspices of the CHVI for projects examining their HIV vaccine candidates.
Other examples of the project achievements revealed by the CIHR case studies thus far include: successful cohort recruitment; development of publications, presentations and manuscripts; progress towards the development of new research assays, tools, approaches and methods; leveraging resources; as well as training and mentoring of young researchers both within Canada and in LMICs.
Capacity Among Researchers to Conduct HIV Vaccine Research
There was evidence that individuals involved in the projects increased their capacity to conduct HIV vaccine research. All survey respondents agreed that their own capacity to conduct HIV vaccine research had increased as a result of the initiative. Most of these researchers were not new to the HIV vaccine field (only one researcher reported being new to the field). However, for some researchers, CHVI allowed them to increase their focus on HIV, rather than other related research areas.
Survey respondents reported an increase in students and trainees' capacity to conduct HIV vaccine research. Students were able to further their education by participating in the funded projects and gaining new knowledge and skills in conducting vaccine research. From the 22 projects included in the survey, researchers reported that they engaged a total of 37 PhD students, 29 Masters students and 18 undergraduate students, and specifically supported 23 Masters theses or doctoral dissertations in the HIV vaccine field. The CIHR case study on large team grants indicated that the funding resulted in having trained scientists in Botswana in methods of biogenetics and supported trainees, junior researchers and students in completing advanced degrees, including their Masters or PhD degrees. All interviewees from the CIHR case studies agreed that their project had positive impacts on careers including allowing trainees and early-career researchers to pursue higher education, including graduate degrees or postdoctoral fellowships. The projects enabled senior researchers to become mentors and leaders as well as increased opportunities for publications and presentations, including presenting their research internationally.
Capacity building opportunities that took place in LMICs, in the form of training and courses delivered, while conducting research projects, increased the capacity and skills of local LMIC staff and researchers. For example, in one large team project, staff located in South Africa were hired and trained to create, translate and deliver interviews and focus groups, which helped to complete the research project. In the same project, community representatives, site staff, students and junior investigators in India received capacity-building research training and informed consent training. Findings from the CIHR case study revealed that social scientists from one of the large team grants had trained local staff in Nigeria to educate mothers and nurses on how to complete consent forms which were necessary to support the project. These training initiatives and efforts improved the knowledge and awareness of individuals in LMICs and built their capacity to conduct HIV vaccine research.
Overall, the ABS area of focus resulted in advancing novel ideas, tools, and approaches while increasing the capacity of researchers and their teams in conducting HIV vaccine research. However, as there is still no effective HIV vaccine available, more research needs to be done to advance knowledge in this area.
4.4.2 To what extent have the outcomes for Translating Basic Science (TBS) been achieved?
The aims of Translating Basic Science were to:
- encourage private sector participation that would contribute to the development of an HIV vaccine;
- strengthen the capacity of researchers and research institutions to conduct high-quality clinical trials; and
- build site capacity in LMICs to conduct HIV vaccine clinical trials.
There was evidence of increased numbers and capacity in Canadian firms working on HIV vaccine technologies (and other HIV-related technologies); increased capacity among LMIC researchers to conduct HIV prevention trials; and efforts towards increased collaboration among the researchers. PHAC, IC/NRC and DFATD were responsible for the activities to achieve these outcomes.
Translating Basic Science Through Private Sector Technology Development
The Canadian HIV Technology Development (CHTD) was established as a component of the NRC's Industrial Research Assistance Program (IRAP) and was delivered using IRAP's existing structures, such as its Industrial Technology Advisors and networks of small- and medium-sized enterprises (SMEs). The objective of the CHTD Program was to encourage and support Canadian SMEs to develop an HIV vaccine and other technologies related to the prevention, diagnosis and treatment of HIV.
Projects that focused on HIV vaccine technologies were prioritized to receive funding; however, in order to advance HIV technologies in Canada, other non-vaccine HIV technologies were also funded. To date, 28 projects were funded by the CHTD: 12 projects ($6.0M) in vaccine technologies, 11 projects ($3.2M) in diagnostics and five projects ($2.2M) in therapeutics. Supporting a cluster of HIV-related research and development projects was deemed essential in ensuring that there are science and technology receptor companies developed in academia and research institutions that would help move technologies towards commercialization.
Collaboration was identified as an important priority for industry partners. NRC's evaluation of the IRAP program detailed the important role that Industrial Technology Advisors (ITAs) play in connecting the business community members. ITAs had networks in the regions and in industrial sectors that they were responsible for, and were often considered as “the main entry point for NRC-IRAP clients to develop networks and linkages in the business community”Endnote 23. To support collaboration, in 2013, the ACO hosted a symposium in an effort to better understand the needs of SMEs and build collaborations between funding recipients, departmental staff, funders, as well as Canadian and international researchers. Five SMEs funded by the CHTD program were invited to present on their research, including the challenges they faced. Some of the challenges identified by the CHTD recipients included the need for increased collaboration with larger commercial entities and lack of access to suitable manufacturing facilities to support continued development. Companies identified a lack of mechanisms to engage with potential collaborators for clinical trials, assembling trial materials, recruitment of subjects and trial participants, and data analysis.
The CHTD projects helped increase capacity among SMEs to conduct HIV vaccine and/or other HIV technologies-related research. The companies were able to hire new employees and conduct research in areas that would not have been possible otherwise. Survey respondents representing the funded SMEs (n=15) agreed that the CHTD program was a catalyst for the firms to enter the field of HIV vaccine research (or other HIV-related technologies). The majority (73 percent) of respondents indicated that they hired full- and part-time staff for the CHTD project. There was also evidence that CHTD funding allowed the firms to advance their technology toward commercialization. Most of the survey respondents (93 percent) indicated that the funding enabled them to move their technology towards commercialization, a perspective that was also supported by key informants.
The Industrial Technology Advisors (ITAs) provided advice to the SMEs, which contributed to the strengthening of firms' capacity to conduct research, as outlined in the NRC-IRAP evaluation and supplemented by the key informant interviews. During project implementation, the ITAs reviewed progress and regularly provided business and technical advice to the funding recipients. ITAs usually have relevant experience since these individuals are frequently former senior managers or previous proprietors of enterprises. ITA advice enabled the recipients to problem-solve more effectively, assisting them in advancing their technology projects toward commercialization.
Overall, the CHTD had success in increasing R&D activity in HIV-related technologies in Canada, tripling the number of firms conducting research in the field. The survey, document review, and interviews all confirm that progress has been made. Most survey respondents indicated that funding has led to new research findings and indicated that they were able to develop new technologies through CHTD. For instance, one firm utilized a novel strategy to develop an HIV vaccine. Instead of using recombinant HIV proteins, either alone or in combination, as previous HIV vaccine attempts have done, the firm developed an inert whole-virus method. This method could elicit broadly neutralizing antibodies that could possibly prohibit both the initial acute infection and establishment of a latent reservoir of HIV virus in cells. Recently, a Phase I clinical trial was completed on the vaccine candidate using this method. Another CHTD-funded firm investigated the feasibility of an orally administered drug to work in conjunction with High Active Antiretroviral Therapy (HAART) to help manage HIV. The drug could potentially improve the effectiveness of HAART treatment by activating and flushing latent HIV reservoirs. The firm was unable to provide more detailed information at this time due to intellectual property issues.
Translating Basic Science Through Capacity Building
DFATD collaborated with the International Development Research Centre's (IDRC) Global Health Research Initiative (GHRI) to fund the HIV Prevention Trials Capacity Building Grants (Phase 2) Program ($16M; 2009 - 2015). This program funded nine research projects (as well as five complementary grants and 10 pilot awards to young researchers) designed to build capacity of LMIC researchers (as well as institutional, regulatory and lab capacity) to conduct HIV prevention trials in Africa.
There was evidence of efforts towards increased collaboration through GHRI activities. One key collaborative event was the Afri-Can Forum, organized by representatives from each GHRI project, CIHR, and the ACO. The Afri-Can Forum aimed to further develop synergies and complementarities within the GHRI program, with other CHVI-funded components, and with other global efforts to build capacity for HIV prevention trials. The event attracted over 100 participants.
Some examples of other collaborations supported by the GHRI projects include:
- A partnership between Canadian and LMIC researchers who were co-investigators on a project, which was identified as being instrumental in the team's success in a subsequent CIHR funding opportunity (a five-year research programmatic grant from CIHR's Health Equity program).
- A partnership between the Canada-Africa Prevention Trials (CAPT) Network, Oxford University, and the European & Developing Countries Clinical Trials Partnership (EDCTP) on a medical male circumcision project. The project helped to secure a partnership with Harvard School of Public Health on a mixed tuberculosis strain.
- A partnership between one GHRI project team and researchers from McGill University for the creation of an online professional Masters program in primary health care planning and management. The program is currently in development.
There was evidence of efforts to improve researchers' and students' capacity to conduct high-quality clinical trials, including training activities to improve qualitative and quantitative research skills, publication writing skills, project management, and laboratory skills. Researchers and students further improved their capacity by presenting their work at international conferences. In 2013-14, grantees published nine articles in peer-reviewed publications and gave 19 scientific presentations.
One GHRI project provided training to health workers, which improved their skills related to the implementation and evaluation of workplace-based HIV programs, and equipped two hospitals in South Africa with an HIV and TB module for an occupational safety database to assist them to properly report and monitor occupational infections. Focusing on improving health workers' capacity was seen as an important link to support the translation of research results into practice. Such capacity building efforts are important as HIV prevention and/or vaccine trials require careful planning and coordination with health workers to ensure successful implementation.
Other projects aimed at improving institutional, regulatory, and laboratory capacity were implemented through infrastructure-building activities such as the construction of laboratory facilities (e.g., cold-room structure), the establishment of a local research ethics committee, and training of Institutional Review Board (IRB) members.
Overall, the Translating Basic Science (TBS) projects supported research and advancements in developing and testing new ideas, tools and approaches. CHTD and GHRI projects investigated and developed new ideas, tools, and approaches and may advance the development of an HIV vaccine (or other HIV technologies). However, without baseline measures, it was difficult to assess the degree to which research and development in Canada and in LMICs increased as a result of CHVI funding.
The Agency identified translational research barriers in 2011. However, the Agency was not able to spend all of its planned budget ($4.5M) in this activity area. This was largely due to the lack of promising vaccine candidates that would benefit from the Translational Support Fund. In 2013-14 and 2014-15, CHVI partners agreed that the Agency would transfer a total of $1.2M to CIHR to support basic science projects. The remaining $3.3M had not been spent on CHVI activities; instead, it was used for other Agency HIV priorities, such as financing a WHO initiative, jointly with the Federal Initiative to Address HIV/AIDS in Canada, in support of common objectives.
4.4.3 To what extent have the outcomes for Addressing Enabling Conditions (AEC) been achieved?
The aim of the Addressing Enabling Conditions was to focus on policy and regulatory issues related to vaccine clinical trials, in order to better prepare both LMICs and Canadian communities for future HIV vaccine clinical trials. Health Canada, DFATD, and PHAC each funded activities under this area of focus, resulting in achievement of outcomes. For example, since 2009, the number of AVAREF-member countries able to conduct vaccine clinical trials with internationally accepted ethics and regulatory approvals and oversight had doubled.
Health Canada collaborated with National Regulatory Authorities (NRA) in LMICs, particularly in Africa where several HIV vaccine clinical trials are being planned, and also with LMICs involved with the Pan American Network for Drug Regulatory Harmonization (PANDRH). Health Canada developed a Mentorship Program to increase the regulatory capacity among regulators in Malawi (2012) and Nigeria (2014). An action plan was developed for each of the two countries, outlining the outcomes expected to be achieved. The Mentorship Program delivered by Health Canada (delivered either in Canada or the host country) included a series of training sessions delivered by Health Canada staff and focused on issues relevant to respective countries. The program helped NRAs in Malawi build capacity in the use of databases, the development of standard operating procedures, and the use of benefit/risk assessments for making regulatory decisions. The training sessions in Nigeria focused on submission-processing, screening and review of common technical documents. Though too early to assess, it is expected that the knowledge gained via the Mentorship Program helped Nigerian NRAs become better equipped to navigate some of the policy and regulatory processes related to the clinical trials for the current Ebola vaccine. (At the time of the evaluation, Malawi had not experienced any cases of Ebola, and no Ebola vaccine trials were being planned there).
Health Canada also worked actively to create knowledge exchange opportunities via the International Regulatory Forum (IRF). This forum was an annual conference, beginning in 2009, which brought NRAs from LMICs together to share knowledge about systems, tools, and approaches; to network with regional and international participants; and to attend satellite sessions on specific subjects such as biologics and quality review; and build relationships between ethics boards and regulatory authorities.
The IRF, identified by regional and international partners as an important regulatory event, had a positive reputation internationally. Health Canada sponsored several delegates from NRAs (ranging from 22 to 40 individuals from over 16 LMIC countries) to attend IRF each year. A majority of participants in attendance at the 2012 IRF reported that, while the focus of CHVI training was on vaccine and clinical trial regulation in general, they would be able to apply the knowledge and skills to the HIV-vaccine clinical trials planned to take place in their respective countries. They also stated that the annual IRF meetings increased the capacity of regulators over time as there were opportunities to share knowledge. While Health Canada consulted attendees regarding topics for upcoming fora, some participants expressed a desire to provide additional input to make the IRF even more pertinent to their needs (e.g., post-market surveillance).Endnote iii
The DFATD provided a $2M grant to the World Health Organization (WHO) to formalize the African Vaccine Regulatory Forum (AVAREF), a network of NRAs and national Ethics Committees (EC) in 19 African countries. The network seeks to strengthen regulatory capacity in Africa for vaccines to provide required reviews, approvals, and oversight of vaccine trials in a more rapid and coordinated manner. As of 2012, more than 50 percent of the participating countries had endorsed the set of Terms of References created for the formalization of AVAREF.
The WHO delivered training courses (including inspection, evaluation of clinical data, legislation, Good Clinical Practices) to AVAREF country members. There was a coordinated Government of Canada effort in support of the annual AVAREF meetings; DFATD funded the development of guidelines and setting norms and standards for vaccine regulation. Health Canada played a role in building capacity in AVAREF, through the provision of advice and technical and regulatory expertise, which aided the development of a strategic plan, guidelines, norms and standards for vaccine regulation, and AVAREF's virtual collaborative platform for the participants to exchange information and collaborate online.
The contribution to AVAREF supported participating LMICs in increasing their capacity to conduct clinical trials and meet international standards. The document review indicated that there is a willingness and strong enthusiasm to work together within the AVAREF network. Countries have either adopted or adapted the developed guidelines within the network, such as Good Clinical Practices inspection guidelines.
These efforts have had an impact on the ability to conduct clinical trials in Africa. Since 2009, the number of AVAREF-member countries able to conduct vaccine clinical trials with internationally accepted ethics and regulatory approvals and oversight had doubled - a total of 16 countries had conducted fourteen vaccine clinical trials for various diseases with internationally acceptable ethics and national regulatory authorities' approvals. These trials led to at least seven successful vaccines that have been prequalified by WHO, licensed and are now in use in the African region (e.g., conjugate meningitis A, rotavirus, human papillomavirus). In addition, several AVAREF-member countries now review and approve clinical trial applications in a common technical format comparable to those of more advanced NRAs.
Improvements in international standards were documented, including improvements in both quality of reviews and decrease in the time it takes for clinical trial applications review and approval processes. Overall, CHVI funding for AVAREF assisted participating LMICs to adopt internationally approved standards for reviewing and monitoring clinical vaccine trials. While training is typically a good method of building capacity, it was often followed by attrition as newly-trained staff went on to accept positions elsewhere (e.g., international positions). This may indicate the need for continual training in order to maintain and sustain the level of capacity built within NRAs.
Through the work of AVAREF, the Pan-African Clinical Trial Alliance (PACTA) was developed to support the integration of three pillars of clinical trials: ethical review, regulation and registration of clinical trials in Africa. In terms of the registration pillar, a clinical trial registry was created and endorsed by all 19 AVAREF countries that started enforcing pre-registration before accepting clinical trial applications for review and approval. The number of clinical trials registered grew exponentially - from 25 trials in September 2010 to over 80 in October 2013 (though these were not all related to HIV vaccine clinical trials, they also included clinical trials on other diseases).
Regulators used networks that CHVI helped establish, including the Alliance Virtual Committee (Alliance VC) developed by the ACO to facilitate online collaboration between its members. Specifically, members of AVAREF used the Alliance VC to connect with one another. Health Canada posted its 2012 training online in English, French and Spanish. This allowed organizations with limited resources to connect with one another and avoid travel costs. This online community allowed individuals to access training materials in their own time and at their own pace.
While there was evidence that capacity in support of regulatory and ethical processes in clinical trials was enhanced, these enhancements benefited practices related to clinical trials for vaccines in general, rather than focussing on HIV vaccine clinical trials exclusively. Regulatory capacity was also built in LMICs, but this expertise has yet to be applied to any HIV vaccine candidates, due to the current lack of HIV vaccines. This may demonstrate a further need to investigate how the improvement of regulatory and ethical processes for vaccines in general can indirectly help HIV vaccine candidates in the long run, which could then inform future funding priorities.
Improved Knowledge Sharing
PHAC funded several projects in the AEC area of focus that supported domestic knowledge sharing activities (including annual HIV vaccine research conferences), and other capacity-building activities in Canadian communities. The engagement activities with organizations and communities helped to raise the level of understanding of the role that the communities play in supporting potential future vaccine trials (e.g., as volunteers for trials) and/or new prevention technologies. For example, PHAC funded the Canadian AIDS Society to develop an HIV vaccine preparedness toolkit for service providers in AIDS services organizations, community-based organizations, and in community health centers to increase the awareness of the impact of vaccine development and prevention research. Progress reports from PHAC-funded projects demonstrated the immediate impact of raised awareness in target communities with regard to vaccine preparedness, engagement for trials and vaccine development. Similar results were found for a project that worked with partners in Africa and Canada to develop a series of training tools, workshops and outreach activities that strengthened their knowledge of HIV vaccines and new prevention technologies.
PHAC funded projects that helped to make international standards more accessible to help other countries prepare for HIV clinical trials. Specifically, PHAC funded a project to translate two guides, namely the Good Participatory Practice Guidelines, that helped to ensure globally accessible guides for use by a wide range of stakeholders in HIV prevention trials. These provide guidance on how to effectively engage stakeholders in designing and conducting biomedical HIV prevention trials. The guides and the translations are available publicly online and are supplemented with online tools and training tools for learners. According to a final report, one workshop on the guide took place in Cambodia and groups in Thailand have begun planning a consultation meeting, where the Thai version of the guide will be disseminated.
Overall, PHAC contributed to both domestic and international HIV vaccine trial readiness, including awareness building efforts with community level partners on broader HIV prevention topics and for funding annual conferences for knowledge sharing.
4.4.4. To what extent have the outcomes for Preventing Mother-to-Child Transmission of HIV (PMTCT) been achieved?
Progress towards collaboration and knowledge sharing in this area, as well as capacity building among health workers to deliver PMTCT services was demonstrated. Given that the projects were still mid-implementation, it was too early to assess longer-term outcomes. However, DFATD projects were generating information that would help inform improvements to the quality, demand for, and use of PMTCT services. Overall, CHVI's PMTCT activities were largely separate from the rest of CHVI's vaccine-focused activities. DFATD was the primary departments responsible for activities in this area.
The objective of the PMTCT area of focus was to increase the quality, access and uptake of PMTCT services, to help reduce the spread of HIV in the absence of a safe and effective vaccine. DFATD supported a $20M grant to WHO (2011-12 to 2015-16) to conduct six research projects, called the “INtegration and Scaling Up PMTCT through Implementation Research” (INSPIRE) projects, in three African countries, namely Malawi, Nigeria, and Zimbabwe. Each of the country's two projects examined how to enhance services in health facilities or communities in order to improve retention-in-care of HIV-infected women and mothers.
The INSPIRE projects have already strengthened collaborations between researchers, health workers, and community leaders through an Implementation Research Platform (a virtual learning network), and investigator meetings to share lessons on protocol and intervention issues across all INSPIRE projects. Knowledge sharing (e.g., four abstracts presented at conferences, and a draft manuscript) led to an increase in researchers and health workers' awareness and understanding of some of the barriers women and families face in accessing PMTCT services.
DFATD provided a $10M grant to the Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) for a project titled “Advancing Community Action for Improving Maternal, Newborn and Child Health/PMTCT Outcomes” (ACCLAIM) that sought to increase access, demand and retention in maternal and child health services and prevention of mother-to-child transmission of HIV (PMTCT) services. It targeted pregnant women, their partners and children with a social and behavioural approach that integrated both maternal, newborn and child health (MNCH) as well as HIV interventions at the community-level in three African countries, namely Zimbabwe, Swaziland, and Uganda. A curriculum for the community intervention was developed and contextualized for use in Zimbabwe and Swaziland in order to make it culturally relevant. Activities in this project demonstrated, amongst other elements, an increase in awareness and understanding of MNCH issues, such as the importance of antenatal care, and the transmission of HIV to infants. Results from the ACCLAIM project's Knowledge, Attitudes, Practices and Beliefs (KAPB) Survey, while intended to inform the design of the project's PMTCT interventions, could potentially also inform future HIV vaccine trial designs, other disease (e.g., Ebola) vaccine trial designs, or the implementation of HIV vaccination programs that would gain better knowledge of socio-cultural factors (e.g., issues around compliance, retention, informed consent, barriers to access care).
The ACCLAIM project strengthened collaboration among researchers and health care workers. The project was conducted in three countries in a coordinated fashion, including joint annual meetings with representatives from each participating country. This allowed the projects to be designed and implemented in a standardized way, including the recruitment of community leaders by collaborating with political stakeholders within local councils and over 50 chiefdoms. Progress has been slower in Uganda due to delays in obtaining ethical clearance (which was received in April 2014). Community leaders, through peer group sessions and workshops, have contributed to the project's secondary objective of sharing knowledge at the community level in order to improve birth preparedness and related outcomes for mothers and newborns, while promoting uptake and adherence to interventions related to PMTCT (e.g., early antenatal care attendance, facility delivery, etc.). Presentations delivered at the annual global meeting of EGPAF's Technical Directors Forum helped project coordinators share early lessons learned from the projects and meetings with USAID in Swaziland helped reduce duplication between the groups' similar interventions. A publications committee was established to facilitate dissemination and publication of findings in academic journals.
The PMTCT projects increased capacity among health care workers to deliver PMTCT services, and among decision makers, such as parliamentarians, to better understand and interpret research results. Since the INSPIRE projects began, over 1,800 health care workers have been trained in areas such as Good Clinical Practices (GCP), ethics clearance, data collection and management, research techniques, PMTCT principles and service delivery, and in the roll-out of a new PMTCT treatment approach (i.e., Option B+).
Data monitoring plans have been established and quality assurance measures were implemented to help in the regular collection of complete and accurate patient data that will eventually feed into EGPAF's Global AIDS System for Results (GLASER) database used for managing data and indicators across all EGPAF projects worldwide. For instance, while not attributable to the ACCLAIM project specifically, EGPAF monitored the recent decline in rates of PMTCT in Zimbabwe. When EGPAF began working in Zimbabwe in 2001, the rate of PMTCT was nearly 30 percent. By 2011, about the time when the project began, it was down to 8.8 percent. EGPAF reports that by May 2014, the rate was even lower, though no exact figure was reported. Similar progress can be observed in other parts of Africa, with a rapid decline of new HIV infections among children by 50 percent or more between 2009-2012 in Botswana, Ethiopia, Ghana, Malawi, South Africa, and Zambia.
INSPIRE project investigators in Nigeria, in collaboration with WHO, were involved in developing the 2013 Harmonized National Guidelines for HIV Prevention, Treatment and Care, which incorporates revisions to its PMTCT interventions based on WHO recommendations. The intent is to help ensure that women have access to quality PMTCT services aligned with current WHO guidelines. This highlights the need for collaboration and knowledge sharing among researchers and policy makers to ensure that new guidelines or recommendations are incorporated into future study protocols.
The importance of sharing lessons learned from the PMTCT projects with the rest of CHVI's projects was highlighted at the Advisory Board level, but to date, this has not been fully explored. For instance, PMTCT project researchers have not been included in other CHVI-led collaborations or consultations, which have largely been vaccine-focussed. Therefore, there could be opportunities in the future to learn from knowledge gained through PMTCT activities, and apply them to the broader CHVI and vice versa.
4.4.5 To what extent have the outcomes for Supporting Coordinated Efforts (SCE) been achieved?
Efforts to strengthen the coordination of Canadian HIV vaccine-related activities with other global efforts were largely achieved through PHAC's CHVI Secretariat and the ACO. The CHVI enhanced collaboration with key global partners and non-government organizations, while the ACO established networks, and enhanced collaboration and knowledge sharing among researchers, industry representatives, government, and Canadian and international non-governmental organizations. PHAC was the primary agency responsible for activities in this area.
Immediate outcomes associated with supporting coordinated efforts were central to ensuring collaboration and networking within the field of HIV vaccines among vaccine researchers in Canada. The CHVI Secretariat (housed at PHAC) provided key support to ensure that the work of CHVI partners (including the BMGF) was coordinated effectively. This included facilitating communications, reporting responsibilities and liaison between the GoC and stakeholders. It also provided the lead on policy and program-related issues for the overall initiative (including horizontal reporting requirements) and managed the contribution agreement with the CHVI ACO ($3.2M, 2011-2015). The CHVI Secretariat also participated in Advisory Board meetings and represented the Government of Canada at key global and international HIV vaccine and related conferences and workshops. The CHVI Secretariat was responsible for coordinating some CHVI activities, such as the monthly/bi-monthly teleconference calls with CHVI partners (including the ACO and BMGF). These calls were useful in providing partner updates on activities and for coordinating attendance and activities (e.g., satellite sessions) at upcoming conferences and meetings.
The main objectives for the ACO were to provide support to the CHVI Advisory Board, provide advice on Canada's strengths in the field of HIV vaccines, identify gaps within Canada and research results to better focus funding and activities and serve as a platform to increase collaboration and networking among stakeholders.
The ACO's objectives were largely achieved and included activities such as:
- sharing information about CHVI and other HIV-vaccine related information via its website and e-bulletins, which provided updates on funding opportunities, recaps of meetings and conferences, and other ACO updates;
- hosting webinars on current and emerging vaccine topics where respondents agreed that the webinars improved their understanding of the content and achieved the stated learning outcomes; and
- partnering with the Canadian Association for HIV Research (CAHR) to develop a database of HIV researchers.
The CHVI Advisory Board was established to provide strategic advice on the direction of CHVI activities and to make recommendations to Government of Canada Ministers and the BMGF on projects to be funded.
Overall, the ACO had a positive reputation among internal and external stakeholders. During the last four years, the ACO created the Alliance and its membership grew to over 300 members (including 35 percent international members). Moreover, CIHR case studies also described the ACO as a good platform for the facilitation of collaborations and communication both within and between research teams, by facilitating conference calls, as well as across the five CHVI research teams through organizing meetings, conferences, and webinars. However, it was not clear from the documentation reviewed who these members were or what goals the Alliance hoped to achieve. In addition, there was no evidence to describe what impact this collaboration had on the progress of CHVI, other than to say stakeholders were engaged. The ACO provides coordination and administrative services to some researchers (e.g., hosting the Virtual Network Event with large team grantees) but the scope and criteria for these services were not clear (i.e., why some researchers benefit from these services and not others). So while these coordination services were likely useful for researchers and allowed them to network and collaborate, CHVI would need to articulate the purpose and goals of these collaborations in order to determine who should participate in them, and how to monitor the outcomes achieved.
One of the key goals for SCE was to increase strategic decision-making. This was implemented largely through the CHVI Advisory Board, whose key role was to review proposals to ensure alignment with the CHVI MOU (though GoC CHVI Ministers had ultimate decision-making authority on GoC funds). However, given that the objective in the MOU was broad, any project related to HIV vaccines could notionally be recommended for approval. It was emphasized that the Board's role was not to comment on the science of the proposals, and some key informants felt that CHVI was not benefiting from the high level of the Board's expertise as much as it could have. Part of this may be a timing issue (Board review occurs after the sponsoring department/agencies' internal peer-review and advisory committees have already approved proposed projects). In certain instances, the Board did not review CHVI-funded projects, for example the PMTCT project and the Health Canada Mentorship Program, although it was not clear why this was the case. Advisory Board members' contribution to strategic decision-making could have been enhanced if they had been involved in the proposal design stage and/or in broader strategic planning discussions. This would have allowed Advisory Board members to contribute their knowledge about the HIV vaccine landscape and emerging issues, to help inform future activities, priorities, and/or countries that would benefit from future funding opportunities and/or to help identify solutions to barriers and challenges encountered by the project teams.
There was some evidence of strategic planning discussions that were happening outside the Board (e.g., discussion at CAHR 2014 on a new strategic plan for 2015-2020 of CIHR's HIV/AIDS Research Initiative , including CHVI, and other HIV programs, but no strategic planning discussions were held regarding CHVI overall.
Document reviews and key informant interviews identified that the specific priorities that CHVI was intended to fund were not clear among partners and the Advisory Board members, resulting in inconsistent administration of the Initiative which has since been resolved. The original 2007 policy documents focused on preventive vaccines. Since then, it had become more evident that results from therapeutic HIV vaccine research could make important contributions to the discovery and development of preventive HIV vaccines. The 2010 CHVI renewal documents refer to “HIV vaccines” in general, which could arguably include both therapeutic and preventive vaccines. This may have led to some inconsistencies at the Advisory Board level as to whether or not therapeutic proposals were aligned with overall CHVI objectives (although this has been clarified with CIHR projects). In fact, three CHTD proposals were not recommended for approval due to the lack of linkages to preventive vaccines; whereas eight proposals from CIHR focusing on therapeutic vaccines research were recommended for approval. The three proposals from NRC that were previously rejected were re-submitted and were recommended. In this case, it was agreed that while therapeutic vaccine proposals could be supported under CHVI, HIV vaccine and diagnostic projects should be prioritized.
The other area that was unclear was related to the extent to which proposals were required to include Canadian content. This was particularly evident with the proposals coming from the BMGF. Ensuring adequate Canadian content in some of the BMGF proposals was an issue raised by certain Board members (e.g., ensuring some Canadian linkages with the South African immunology lab). It was unclear in the MOU how the goal of “strengthening efforts… by building upon Canadian expertise” should be operationalized.
As described earlier in the Continued Need section (Section 4.1), there have been a number of recent changes in the HIV vaccine environment, which could potentially impact future considerations for the Initiative, including:
- advances in new prevention technologies (NPTs);
- move to combine the sharing of all biomedical HIV prevention research - from vaccines, to microbicides, to treatment as prevention - as per the latest HIV Research for Prevention (R4P) international conference;
- emergence of HIV Cure research; and
- the Canadian Immunization Research Network (CIRN) that focusses broadly on all areas of vaccine, immunization, and infectious diseases.
These recent shifts in the HIV vaccine environment highlight the need to revisit the priorities and scope of activities that CHVI may fund in the future.
4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency
Opportunities to improve efficiencies in decision making and reduce overlap among governance structures were identified.
The Treasury Board of Canada's Policy on Evaluation (2009) and guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.
The data structure of the detailed financial information provided for the Initiative did not facilitate the assessment of whether program outputs were produced efficiently, or whether expected outcomes were produced economically. Specifically, the lack of output/outcome-specific costing data limited the ability to use cost-comparative approaches. In terms of assessing economy, challenges in tracking funding within the broader program envelopes limited the assessment. Considering these issues, the evaluation provided observations on economy and efficiency based on findings from the literature review, survey, key informant interviews and available relevant financial data.
Observations on Economy
Since renewal in 2010, budgets have been generally spent as planned in most departments/agencies. As noted in Table 3 below, exceptions were PHAC's activities under TBS ($3.3M uncommitted, $1.2M transferred to support CIHR's ABS), also BMGF activities ($18M uncommitted at the time of the evaluation, though a proposal for $16.1M was presented to the CHVI Advisory Board in November 2014). However, overall underspending was common for CHVI as documented in the reported spending in the Departmental Performance Reports (2010-11 to 2013-14). Prior to and during renewal, there was significant underspending (i.e., >$15M) as the program readjusted to new changes, such as the decision to cancel certain program components and to create new ones. However, recent 2014 audit findings on PHAC spending on CHVI suggest that underspending is still an issue. The audit posits that management reports indicate vaccine efforts have not advanced as expected to be able to disburse funds as originally intended. Since renewal, several common explanations for the variances found in the DPR include lack of proposals for projects, delays in launch for proposals and set-up/administrative delays.
|Fiscal Years $M|
% of planned total spentTable 3 - Footnote ***
|PlannedTable 3 - Footnote *||Actual||Planned||Actual||Planned||Actual||Planned||Actual||PlannedTable 3 - Footnote *|
Source: PHAC Supplemental tables for Horizontal Initiatives, appendix of PHAC DPRs
Note:( ) is underspending,
|IC/NRC||0.5Table 3 - Footnote **||0.0||3.0||1.3||2.5||2.5||4.2Table 3 - Footnote **||3.2||3.2||(1.7)
CHVI produced its outputs and achieved its outcomes in an economical manner, as demonstrated by leveraging additional funds and utilizing cost minimization measures.
Several CHVI-funded projects were successful in leveraging funds for their current or future projects. Based on survey results, eight research projects funded by CIHR were able to secure a total of $5.37M in additional funding for their current project from sources such as: CIHR (outside of CHVI funds), the National Institutes of Health, and provincial organizations. Four other CIHR research projects received a total of $1.2M for a future research project. Thus, for every one CHVI dollar funded, $0.37 was leveraged for the current CIHR research projects, and $0.08 leveraged for future projects.
Other examples of leveraging funds included:
- some GHRI project teams that were successful in obtaining funding for future projects through a CIHR funding opportunity, largely as a result of the collaboration developed through the GHRI project, and
- the requirement for SMEs to contribute 10 percent of the total financial requirement for the CHTD projects (the CHTD only funds up to 90 percent of eligible project costs).
Altogether, these examples demonstrate CHVI's ability to leverage additional funds for HIV vaccine research and development.
Researchers interviewed as part of the CIHR case studies highlighted that is difficult to determine how these HIV vaccine research projects can, or will, continue without CHVI funding. Researchers leading projects with a longitudinal aspect were particularly concerned over losing the cohort after the funding period ends, thus diminishing the potential impact the study could have in the long term. Although most agreed that collaborations between Canadian and LMIC researchers would continue, there would likely be some challenges in sustaining those working relationships without the required funding . While the large teams were active in HIV research prior to CHVI, the end of CHVI funding is expected to have a very dramatic impact on all five research teams, with all teams likely to seek funding outside of Canada.
CHVI partners used cost minimization measures such as using pre-existing mechanisms to ensure the program costs were kept to a minimum.
In the conduct of CHVI, efficiencies were gained by adapting funding mechanisms and processes by using pre-existing funding programs (e.g., NRC's IRAP, CIHR's catalyst and operating grant programs, and Health Canada's Mentorship Program), and thus the Initiatve was able to identify and/or launch funding opportunities efficiently. As such:
- ABS research projects were reviewed using pre-existing review protocols and review boards that are used for other CIHR grant projects;
- the CHTD program was adapted to the HIV vaccine development using pre-existing models of NRC's IRAP program and similar systems of review were leveraged for the review and distribution of TBS funds to support SMEs;
- in developing regulatory capacity, Health Canada used a pre-piloted model of the mentorship program from experiences in India in developing regulatory capacity in Malawi and Nigeria; and
- UNAIDS Guides, funded by PHAC, were translated into various languages to make them more accessible, used pre-existing guides for ethical regulatory trials, and avoided undue costs of creating new guides.
Overall, using pre-existing, pre-tested models for CHVI program implementation processes and delivery methods helped to avoid costs associated with creating new structures and processes. Although there was no information on accelerated timelines, the evidence highlights that no additional costs were incurred by CHVI.
Cost minimization efforts were also evident at the project level, such as leveraging existing processes beyond the target population. For example, Health Canada provided their mentorship training not only to the target groups but also to other countries nearby with similar levels of capacity in order to maximize the reach of training and capacity building efforts. As well, the use of online technology was documented as the avenue for various CHVI-funded activities, including the ACO Virtual Community and the AVAREF Virtual Community. The use of online technology reduced the costs of working with LMICs, including travel costs, while promoting international collaboration between CHVI stakeholders.
Domestic administrative barriers caused significant delays in obtaining approvals for international travel (upwards of six months). A couple of departments/agencies were required to provide extensive rationales for each travel request, even when the type of travel occurred regularly, and often despite having activities already approved in workplans.
Observations on Efficiency
This section examined the efficiency of CHVI in terms of governance structures, the collaboration with the BMGF, and the implementation of the performance measurement strategy.
Overall, the coordination of the horizontal initiative was effective, and regular communication was supported by the CHVI Secretariat through monthly or bi-monthly partners' group meetings. This allowed partner departments/agencies to provide updates on activities, funding opportunities, and to coordinate attendance at upcoming events (e.g., international conferences). However, there was little evidence of coordinated initiative-wide planning, such as an overall CHVI strategic plan. As noted previously, some planning and priority setting exercises related to HIV vaccine development were occurring, but these did not include PMTCT aspects.
The CHVI Secretariat was responsible for managing the contribution agreement to ICID (to operate and manage the CHVI ACO). Quarterly reports and other deliverables were received in a timely manner. The various roles and responsibilities of the CHVI ACO were articulated in a variety of documents, including: MOU (2010), Contribution Agreement (2011), the CHVI ACO's outline of roles and responsibilities developed in December 2011, Terms of Reference (2012), and the White Paper (2012). A few areas of potential overlap were identified between the CHVI ACO and the CHVI Secretariat, including: monitoring the progress of the CHVI, funding two similar landscape papers identifying priority areas for HIV vaccine research in Canada (CHVI Secretariat funded one in 2011, then the CHVI ACO funded another in 2012, with largely similar findings), facilitating communication between CHVI partners, reviewing proposals prior to the Advisory Board to ensure alignment with the MOU, and funding or sponsoring new and early career HIV vaccine scientists (this was an area that the CHVI Secretariat and CIHR were already involved in supporting). These were not major overlaps, but would be important to revisit the responsibilities in these areas to ensure any overlaps are minimized.
Collaboration with the BMGF
All key informants noted that the collaboration with the BMGF was positive, particularly providing their insights and perspectives on CHVI's activities while on the Advisory Board, as well as participating in CHVI-related satellite sessions and consultations. While some departments already had strong ties with the BMGF outside of CHVI, for others, this collaboration facilitated the initiation of subsequent collaborations with the BMGF on other priority areas, such as Ebola and Hepatitis C. In the area of HIV vaccines, CIHR established a direct collaboration during 2013-14 with BMGF to jointly support research in a priority area identified by BMGF - mucosal immunology. For CIHR, this collaboration would not have been possible outside the framework of CHVI. Some respondents expressed the need for the Government of Canada to take a whole-of-government approach in examining how to best maximize the collaboration with the BMGF.
Observations on the Adequacy and Use of Performance Measurement Data
While a performance measurement strategy was developed, it was not fully implemented until recently (October 2014). Performance information was not regularly collected beyond the annual parliamentary reporting processes (i.e., the Departmental Performance Reports that provide high-level activity-based information on progress towards expected results across all CHVI departments/agencies). Further, since no baseline data had been collected, it was difficult to assess the magnitude of ‘increases' or ‘improvements' in the program outcomes. The performance measurement strategy did not clearly establish success milestones or determine what success would look like. For instance, it is not clear at what level of ‘increased strategic decision-making' would signal success. Indicating a starting point and then determining what constitutes an achievement of an outcome would allow for focussed planning for future activities.
5.1 Relevance Conclusions
While the global incidence of HIV has decreased over the last five years, HIV still affects a significant number of people, particularly in sub-Saharan Africa. In Canada, HIV is concentrated in specific populations. Recently, there has been an increase in uptake of antiretroviral therapy, yet many infected persons still do not have access to the therapy. An HIV vaccine is considered the most efficient and cost-effective means for eradicating the disease. Developing an HIV vaccine has been more challenging compared to other vaccines for a variety of scientific, social, and institutional reasons. While no HIV vaccine has been developed to date, there have been developments in the HIV vaccine field and in the fields of HIV prevention and cure research since CHVI was launched.
There has been a recent trend to take a broad perspective on vaccinology, immunology, and regulatory issues so that research discoveries can contribute to the development of vaccines for a wide range of diseases. Some of the challenges facing HIV vaccine development are common to a number of different diseases - and ultimately, learning more about how the human immune system works and responds to viruses in general could have implications for the development of vaccines for a wide range of diseases, including HIV.
CHVI continues to broadly align with Government of Canada's priorities, as identified in federal announcements and international agreements, as well as in partner departments'/agencies' priorities described in their strategic plans. In addition, partner organizations have clear legislated and policy mandates for the work undertaken in CHVI.
5.2 Performance Conclusions
5.2.1 Achievement of Expected Outcomes (Effectiveness)
Vaccine development is a long, complex process that can take up several years or decades to complete. The development of an HIV vaccine has proven to be more challenging and complex than for other infectious diseases. HIV vaccine discovery research requires a global, collaborative effort across multiple disciplines and stakeholders working together towards a common goal. CHVI was successful in supporting collaborations among HIV vaccine researchers, both in Canada and in LMICs. Collaborations between the industry sector and academics were established, and there is a desire among some to further strengthen these ties - particularly since clinical trials are expensive, and are likely to require resources from private sector partners.
There was some evidence to suggest that the capacity of researchers, health workers, and regulators to contribute to HIV research improved as a result of CHVI funding. This included generating knowledge on HIV vaccine discovery and social research. However, performance data was not consistently collected during the Initiative on the aspect of ‘capacity-building' and there was no clear target for what success would look like (i.e., what the increased capacity would enable these stakeholders to do).
CHVI demonstrated progress towards longer term outcomes, such as improved policy frameworks among regulators in LMICs. And while the PMTCT projects were still early in their implementation, they did demonstrate some progress in generating information that will inform improvements to PMTCT services in LMICs.
Information sharing within CHVI was timely and effective in ensuring that all partners' activities were shared across the Initiative. The network of internal and external stakeholders has grown. The roles of the current governance structures did not fully enable strategic decision-making, and the specific priority areas that CHVI was expected to address were not always clear (i.e., preventative and/or therapeutic vaccines, other HIV-related technologies, regulatory issues for vaccines of other diseases, etc.).
5.2.2 Demonstration of Economy and Efficiency
Since the 2010 renewal, budgets have been generally spent as planned for most key areas of focus. CHVI produced its outputs and achieved progress towards outcomes in an economical manner, often leveraging additional funds and utilizing cost minimization measures (e.g., using pre-existing funding mechanisms) to ensure costs were kept to a minimum.
Overall, the coordination of the Initiative was effective; however, some areas of overlap were identified between the CHVI Secretariat and the ACO. The collaboration between the GoC and the BMGF was positive, and in certain cases, it facilitated the initiation of subsequent collaborations on other priority areas outside HIV. The performance measurement strategy was not fully implemented and since no baseline data were collected, it was difficult to assess the magnitude of any ‘increases' or ‘improvements' in the program outcomes.
Revisit objectives and goals of CHVI within the current context.
In the current context, there is no HIV vaccine and no vaccine is expected to be licensed and available for use within the next several years. However, there are new preventative technologies that have been used to extend the lives of HIV-infected persons, and vaccine research has advanced. Future funding considerations should build on the work conducted through CHVI and consider shifts in approaches for vaccine research, including the broader federal approach to vaccine research and development.
Greater clarity of priorities would also help revise the performance measurement strategy, and identify clear definitions and targets for the expected outcomes. Throughout the course of the evaluation, it was difficult to demonstrate progress as there were no baseline measures to determine changes in outcomes, goals or objectives. The CHVI Secretariat should be responsible for rolling up performance information and providing a whole-of-initiative perspective on progress towards outcomes.
Enhance efficiencies with regard to governance.
The governance approach within CHVI has improved since the 2009 evaluation, and the coordination within the Initiative was also perceived to be effective by both internal and external key informants. However, there were still areas of overlap identified and opportunities to improve the current governance include:
- streamlining decision-making (i.e., in terms of approving CHVI proposals) and re-examining the mandate of the Advisory Board in order to maximize the members' expertise (i.e., advising on the scope or direction of CHVI's future activities and funding opportunities); and
- clarifying the roles and mandates of the CHVI Secretariat and ACO to minimize overlapping activities.
Appendix 1 - References
- Alla, N.R. (2014) “Commentary: The pursuit of a HIV vaccine - Trials, challenges, and strategies” Journal of Aids Clinical Research 5(5).
- AVAC (2013) “Research Reality Checks”, AVAC Report 2013. Retrieved on August 8, 2014 from http://www.avac.org/sites/default/files/resource-files/AVAC%20Report%202013_AIDS% 20Vaccines_0.pdf
- Canadian Institutes of Health Research. (2008). HIV/AIDS Research Initiative
- Strategic Plan 2008-2013. Retrieved on January 9, 2015 from http://www.cihr-irsc.gc.ca/e/37801.html
- Canadian Institutes of Health Research. (2010). Canadian Institutes of Health Research 2010-2011 Estimates. Retrieved on January 9, 2015 from http://www.tbs-sct.gc.ca/rpp/2010-2011/inst/cri/cri-eng.pdf
- Canadian Institutes of Health Research. (2012). Canadian Institutes of Health Research 2012-2013 Report on Plans and Priorities. Retrieved on January 9, 2015 from http://www.tbs-sct.gc.ca/rpp/2012-2013/inst/cri/cri-eng.pdf
- Canadian Institutes of Health Research. (2013). 2013-14 Report on Plans and Priorities.
- Retrieved on January 9, 2015 from http://www.cihr-irsc.gc.ca/e/46303.html
- Clinical Trials (2014). “Safety and Immune Response Assessment Study of Killed-whole HIV-1 Vaccine (SAV001-H) in Chronic HIV-1 Infected Patients.” Retrieved on August 8, 2014 from http://clinicaltrials.gov/show/NCT01546818;
- Cunningham A. et al. (2010). “Manipulation of dendritic cell functions by viruses.” Current opinion in microbiology. 13(4).
- Department of Finance. (2014). The road to balance: Creating jobs and opportunities. Ottawa: Government of Canada. Retrieved on January 9, 2015 from http://www.budget.gc.ca/2014/docs/ plan/pdf/budget2014-eng.pdf
- Dhalla, S, K.E. Nelson, J. Singer, and G. Poole, (2009). “HIV vaccine preparedness studies in the non-organization for economic co-operation and development (non-OECD) countries” AIDS Care 21(3). 335.
- Dhalla, Shayesta, and Gary Poole, (2011). “Barriers of enrolment in HIV vaccine trials: A review of HIV vaccine preparedness studies,” Vaccine 29: 5850-5859.
- Esparza, J. (2013). “What has 30 years of HIV vaccine research taught us?” Vaccines.
- Fast, P.E. and P. Kaleebu. (2010). “HIV Vaccines: current status worldwide and in Africa,” AIDS 24 (suppl 4).
- Foreign Affairs, Trade and Development Canada. (2013). 2013-14 Report on Plans and Priorities. Retrieved from Retrieved on January 9, 2015 from http://www.acdi-cida.gc.ca/rpp-e
- Foreign Affairs, Trade and Development Canada. (2014). Report on Plans and Priorities 2014-15. Retrieved from Retrieved on January 9, 2015 from http://www.international.gc.ca/department-ministere/plans/rpp/rpp_1415.aspx?lang=eng
- Government of Canada. (1996). Department of Health Act. Retrieved on January 9, 2015 from: http://laws-lois.justice.gc.ca/eng/acts/H-3.2/FullText.html.
- Government of Canada. (2010). Canada's contribution to the global effort to reach the millennium development goals. Retrieved on January 9, 2015 from http://www.acdi-cida.gc.ca/inet/images.nsf/vLUImages/MDGpdf/$file/10-206%20MDG%20Report-E.pdf
- Government of Canada. (2011). Speech from the throne to open the First session of the Forty-First Parliament of Canada. Ottawa: Government of Canada. Retrieved on January 9, 2015 from http://www.speech.gc.ca/local_grfx/docs/sft-ddt-2011_e.pdf
- Government of Canada. Canadian Institutes of Health Research Act. Retrieved on January 9, 2015 from http://laws.justice.gc.ca/eng/acts/c-18.1/index.html
- Government of Canada. Public Health Agency of Canada Act. Retrieved on January 9, 2015 from http://www.laws-lois.justice.gc.ca/eng/AnnualStatutes/2006_5/page-1.html
- Health Canada. (2013). Health Canada 2013-14 Report on Plans and Priorities. Retrieved on January 9, 2015 from http://www.hc-sc.gc.ca/ahc-asc/alt_formats/pdf/performance/estim-previs/plans-prior/2013-2014/report-rapport-eng.pdf
- HIV Resource Tracking (2014). “HIV Prevention Research & Development Investment in 2013: In a changing global development, economic, and human rights landscape”. Retrieved on September 8, 2014 from http://www.hivresourcetracking.org http://www.international.gc.ca/development-developpement/ priorities-priorites/mdg-omd.aspx?lang=eng.
- HVTN. (2014). “HVTN in Southern Africa: A Journey of Hope.” HVTNews. Retrieved on September 28, 2014 from http://hvtnews.wordpress.com/2014/02/21/hvtn-in-southern-africa-a-journey-of-hope/
- Industry Canada. (2013). 2013-14 Estimates — Report on Plans and Priorities. Retrieved on January 9, 2015 from http://www.ic.gc.ca/eic/site/017.nsf/eng/07292.html
- International AIDS Society. (2012). “Global Scientific Strategy: Towards an HIV Cure”. Nature Reviews Immunology, 12: 604-614.
- Koff, W.C. (2010) “Accelerating HIV vaccine development.” Nature Vol 464.; CNW. (2012) “Canada's contribution to global pursuit of an HIV vaccine backed by public.” November 1, 2012. Retrieved on September 8, 2014 from http://www.newswire.ca/en/story/1063165/canada-s-contribution-to-global-pursuit-of-an-hiv-vaccine-backed-by-public
- Koff, W. C. et al. (2013). “Accelerating the development of a safe and effective HIV vaccine: HIV vaccine case study for the Decade of Vaccines.” Vaccine 31S.
- Kumar, V. (2012). Robbins Basic Pathology (9th ed.). p. 147.
- Kwong, Peter D., John R. Mascola and Gary J Nabel. (2012) “The changing face of HIV vaccine research” Journal of International AIDS Society 15:17407.
- McKellar-Stewart, N. (2014). “Much Accomplished, More to Achieve.” VAX. 12(5) .
- National Research Council Canada. (2014). Reports on Plans and Priorities 2014-15. Retrieved on January 9, 2015 from http://www.nrc-cnrc.gc.ca/eng/reports/2014_2015/rpp.html
- National Research Council Canada. (2012). Evaluation of the NRC Industrial Research Assistance Program (NRC-IRAP). Retrieved on February 3, 2015 from http://www.nrc-cnrc.gc.ca/eng/about/ planning_reporting/evaluation/2012_2013/irap.html
- NIAID. (2012). “Post-RV144/ Thai Research Update” Research Highlights Retrieved on September 28, 2014 from http://www.niaid.nih.gov/topics/hivaids/research/vaccines/clinical/Pages/highlights.aspx
- Public Health Agency of Canada. (2009). Reports on Plans and Priorities 2009-10 - Table 5: Horizontal Initiatives. Retrieved on January 9, 2015 from http://www.tbs-sct.gc.ca/rpp/2009-2010/inst/ahs/st-ts05-eng.asp.
- Public Health Agency of Canada. (2014). Evaluation of the Federal Initiative to Address HIV/AIDS in Canada 2008-9 to 2012-13.
- Public Health Agency of Canada. (2011). Reports on Plans and Priorities 2011-12 - Supplementary Tables. Retrieved on January 9, 2015 from http://www.tbs-sct.gc.ca/rpp/2011-2012/inst/ahs/st-ts03-eng.asp
- Public Health Agency of Canada. (2012). At a Glance - HIV and AIDS in Canada: Surveillance Report to December 31st, 2011. Retrieved on August 1, 2013, from http://www.phac-aspc.gc.ca/aids-sida/publication/survreport/2011/dec/index-eng.php
- Public Health Agency of Canada. (2012). Supplementary Information Tables: 2013-14 Report on Plans and Priorities. Retrieved on January 9, 2015 from http://www.phac-aspc.gc.ca/rpp/2013-2014/hi-ih-eng.php
- Public Health Agency of Canada. (2013). Supplementary Information Tables: 2014-15 Report on Plans and Priorities. Retrieved on January 9, 2015 from http://www.phac-aspc.gc.ca/rpp/2014-2015/hi-ih-eng.php
- UNAIDS. (2014). 2014 Gap Report. Retrieved on August 8, 2014 from http://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2014/UNAIDS_Gap_report_en.pdf
- University of Western Ontario. “UWO researchers on the verge of a vaccine for HIV”. Ontario Business Report. Retrieved from http://www.mri.gov.on.ca/obr/?p=995
- University of Western Ontario. (2013). “HIV vaccine produces no adverse effects in trials”. Western News. Retrieved from http://news.westernu.ca/2013/09/hiv-vaccine-produces-no-adverse-effects-in-trials/ on March 11, 2015.
- WHO. (2014). Global Guidance on Criteria and Processes for Validation: Elimination of Mother-to-Child Transmission of HIV and Syphilis. Retrieved on August 8, 2014 from http://apps.who.int/iris/bitstream/10665/112858/1/9789241505888_eng.pdf?ua=1&ua=1
- World Health Organization (2014). Global summary of the AIDS epidemic 2013. Retrieved from http://www.who.int/hiv/data/epi_core_dec2014.png?ua=1 on Jan 26, 2015.
Appendix 2 - Logic Model
|Key Activity Areas||Supporting Coordinated Efforts||Advancing Basic Science of HIV Vaccines||Translating Basic Science of HIV Vaccines into Clinical Trials||Addressing the Enabling Conditions||Preventing Mother to Child Transmission|
Increased strategic decision-making
Increased R&D in Canada and in LMICs
Strengthened contribution to global efforts to accelerate the development of safe, effective, affordable and globally accessible HIV vaccines
Increased contribution to the global efforts to reduce the spread of HIV/AIDS, particularly in LMICs
Appendix 3 - Summary of Findings
Rating of Findings
Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.
Relevance Rating Symbols and Significance:
A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.
|1. Continued Need for the Program|
|Legend - Relevance Rating Symbols and Significance::
|What is the current and projected burden of HIV/AIDS in Canada and internationally?||
||High||In Canada, HIV is considered to be a concentrated, low-level epidemic. The rate of new HIV infections (incidence) has remained steady. Evidence showed that an increased prevalence, in part attributable to improvements in treatments resulting in reductions in the number of AIDS-related deaths. Internationally, the burden of HIV/AIDS continues to be significant; 35 million people are living with HIV in the world, 24.7 million (70 percent) of whom live in Sub-Saharan Africa.|
|How has the environment/context changed over the last five years? What is the current stage of HIV Vaccine development and is there a continued need to contribute to global efforts in developing an HIV vaccine?||
||High||Developing an HIV vaccine has been more challenging compared to other vaccines for a variety of scientific reasons, including the global variability of the virus, HIV's unique ability to infect and compromise the immune system, the lack of natural protective immune response to HIV (although some populations seem to be naturally immune), and the lack of valid animal models that accurately predicts the human immune response. This has made advancing clinical trials difficult. In recent years, the number of people newly infected with HIV declined in most parts of the world, largely attributable to the rapid increase in the number of people on antiretroviral therapy (and thus less likely to transmit the virus). Despite such success, 22 million (60 percent) people living with HIV do not have access to antiretroviral therapy. Hence, while there have been advances in treatment and prevention strategies, an HIV vaccine is still viewed as the most efficient and cost-effective ways to eradicate the disease.|
|2. Alignment with Government Priorities|
|What are the federal priorities related to HIV vaccine development and other CHVI activities? Are CHVI's current activities aligned with federal priorities?||
||High||In 2002, Canada committed to international assistance to contribute to the global effort to reach the Millennium Development Goals and was reiterated in 2005 at the G8 summit, to “improve maternal health” and “combat HIV/AIDS, malaria and other diseases”. The 2011 Speech from the Throne stated that the “Government will continue to make targeted investments to promote and encourage research and development”, aligning with research and development components of CHVI. Budget 2014 underlined the Government of Canada's continued commitment to contribute to the fight against AIDS and other infectious diseases, and to improve maternal and child health through two key commitments: the Global Fund to Fight AIDS, Tuberculosis and Malaria (2013), and the Muskoka Initiative for maternal and child health. Although these initiatives are not part of CHVI, they help illustrate the importance that the GoC continues to place on issues of HIV/AIDS and maternal health.|
|What are PHAC and partner departments' priorities related to HIV vaccine development and other CHVI activities? Are CHVI's current activities aligned with these priorities?||
||High||PHAC and partner departments'/agencies' priorities were aligned with CHVI. These priorities included those related to HIV vaccine development, regulatory harmonization of vaccine processes, HIV research, advancements in life science technologies, and maternal and child health abroad, as demonstrated in partner departments'/agencies' annual Report on Plans and Priorities (RPP) and strategic plans over the period being evaluated.|
|3. Alignment with Federal Roles and Responsibilities|
|What are the federal roles related to contributing to the development of a safe, effective, affordable, globally accessible HIV vaccine and other CHVI activities? Are CHVI's current activities aligned with these federal roles?||
||Partial||Various legislative, policy and program authorities identify that each federal government partner has is an appropriate role for the federal government to help reduce the international HIV/AIDS burden, to improve maternal health, and to support research and development. CHVI aligns well with the mandate of each of the federal government partners.|
Performance Rating Symbols and Significance:
A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.
|4. Achievement of Expected Outcomes (Effectiveness)|
Legend - Relevance Rating Symbols and Significance:
|Advancing the Basic Science||
||Progress Made; Further Work Warranted||The Advancing the Basic Science (ABS) area of focus resulted in advancing novel ideas, tools, and approaches while increasing the capacity of researchers and their teams in conducting HIV vaccine research. Collaborative relationships were built between Canadian and international researchers. However, there was no baseline to determine changes and measurement has been limited.|
|Translating the Basic Science||
||Progress Made; Further Work Warranted||The Translating Basic Science (TBS) projects supported research and advancements in developing and testing of new ideas, tools and approaches. CHTD and GHRI funded projects investigated and developed new ideas, tools, and approaches that may advance the development of an HIV vaccine (or other HIV technologies). However, without baseline measures, it was difficult to assess the degree to which research and development in Canada and in LMICs increased.|
|Addressing the Enabling Conditions||
||Progress Made; Further Work Warranted||There was some early anecdotal evidence to suggest that the knowledge gained via the Mentorship Program helped Nigerian NRAs become better equipped to navigate some of the policy and regulatory processes related to the clinical trials for the current Ebola vaccine. Improvements in international standards were documented, including improvements in both the quality of reviews and decrease in time taken for clinical trial applications review/approval. Progress reports from PHAC-funded projects demonstrated the immediate impact of raised awareness in target communities with regard to vaccine preparedness, engagement for trials and vaccine development|
|Preventing Mother-To-Child Transmission||
||Progress Made; Further Work Warranted||Progress towards collaboration and knowledge sharing in this area, as well as building capacity among health workers to deliver PMTCT services was demonstrated. Given the projects were still mid-implementation, it was too early to assess longer term outcomes. However, projects were generating information that would help inform improvements to the implementation of, and demand and use of, quality PMTCT services. Overall, CHVI's PMTCT activities were largely separate from the rest of CHVI's vaccine-focused activities.|
|Supporting Coordinated Efforts||
||Progress Made; Further Work Warranted||Overall, the ACO had a positive reputation among internal and external stakeholders. During the last four years, the ACO created the CHVI R&D Alliance and membership grew to over 300 members. There was some evidence of strategic planning discussions that were happening outside the Board, but no strategic planning discussions were held for CHVI as a whole.|
|5. Demonstration of Economy and Efficiency|
|Has CHVI produced its outputs and achieved its outcomes in the most economical manner?||
||Progress Made; Further Work Warranted||Budgets have been generally spent as planned in most departments/agencies, since renewal. Exceptions were PHAC's activities under Translating Basic Science ($3.8M uncommitted, $1.2M transferred to support CIHR's Advancing Basic Science), also BMGF activities ($18M uncommitted to date). Several CHVI-funded projects were successful in leveraging funds for their current or future projects and CHVI partners used costs minimization measures such as using pre-existing mechanisms to ensure the program costs were kept to a minimum.|
|Is there a performance measurement culture and practice in place? How is the information being used to inform senior management decisions?||
||Progress Made; Further Work Warranted||While a performance measurement strategy was developed, it was not fully implemented until recently (October 2014). Performance information was not regularly collected beyond the annual parliamentary reporting processes (i.e. the Departmental Progress Reports which provide high level/activity-based information on progress towards expected results across all CHVI departments/agencies). Further, since no baseline data were collected, assessments of whether or not the initiative achieved ‘increases' or ‘improvements' for the various outcomes were difficult to determine.|
|Are the CHVI governance structures efficient? Are there opportunities for improvement or best practices to share?||
||Progress Made; Further Work Warranted||Overall, the coordination of the horizontal initiative was effective, and regular communication was supported by the CHVI Secretariat through monthly/bi-monthly partners' group meetings.|
|How effectively was the CHVI leveraged to support basic science, researchers and Canadian industry? Did the partnership with the Bill & Melinda Gates Foundation allow the federal government to carry out activities that would not have been done otherwise?||
||Progress Made; Further Work Warranted||All key informants noted that the collaboration with the BMGF was positive, particularly providing their insights and perspectives on CHVI's activities while on the Advisory Board, as well as participating in CHVI-related satellite sessions and consultations|
|Issue 1: Continued need for the program|
|What is the current and projected burden of HIV/AIDS in Canada and internationally? How has the environment/context changed over the last five years? What is the current stage of HIV Vaccine development and is there a continued need to contribute to global efforts in developing an HIV vaccine?||High||N/A||N/A|
|Issue 2: Aligned to federal government priorities|
|What are the federal priorities related to HIV vaccine development and other CHVI activities? Are CHVI's current activities aligned with federal priorities?||High||N/A||N/A|
|What are PHAC and partner departments' priorities related to HIV vaccine development and other CHVI activities? Are CHVI's current activities aligned with these priorities?||High||N/A||N/A|
|Issue 3: Program consistent with federal roles and responsibilities|
|What are the federal roles related to contributing to the development of a safe, effective, affordable, globally accessible HIV vaccine and other CHVI activities? Are CHVI's current activities aligned with these federal roles?||High||N/A||N/A|
|Evaluation Issue||Achieved||Progress Made; Further Work Warranted||Little Progress; Priority for Attention|
|Issue 4: Achievement of intended outcomes (effectiveness)|
|To what extent has progress towards outcomes within CHVI's five key activity areas been achieved?||N/A||Progress Made;
Further Work Warranted
|Issue 5: Demonstrated economy and efficiency|
|Has CHVI produced its outputs and achieved its outcomes in the most economical manner?||N/A||Progress Made;
Further Work Warranted
|Is there a performance measurement culture and practice in place? How is the information being used to inform senior management decisions?||N/A||Progress Made;
Further Work Warranted
|Are the CHVI governance structures efficient? Are there opportunities for improvement or best practices to share?||N/A||Progress Made;
Further Work Warranted
|How effectively was the CHVI leveraged to support basic science, researchers and Canadian industry? Did the partnership with the Bill & Melinda Gates Foundation allow the federal government to carry out activities that would not have been done otherwise?||N/A||Progress Made;
Further Work Warranted
Appendix 4 - Evaluation Description
The scope of the evaluation included an assessment of the relevance and performance of the Canadian HIV Vaccine Initiative from 2010-11 to 2014-15.
The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation. These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.
|Core Issues||Evaluation Questions|
|Issue #1: Continued Need for Program||Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians
|Issue #2: Alignment with Government Priorities||Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes
|Issue #3: Alignment with Federal Roles and Responsibilities||Assessment of the role and responsibilities for the federal government in delivering the program
|Performance (effectiveness, economy and efficiency)|
|Issue #4: Achievement of Expected Outcomes (Effectiveness)||Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes
|Issue #5: Demonstration of Economy and Efficiency||Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes
Data Collection and Analysis Methods
Evaluators collected and analyzed data from multiple sources.
Sources of information used in this evaluation included literature review, document review, surveys, and key informant interviews.
Literature review: A review of academic and grey literature was conducted to examine the current HIV vaccine research context and assess the continued need for the initiatives. Specifically, the literature review helped to highlight the current stage of HIV vaccine in Canada and internationally and whether this has changed over the evaluation period. Keywords for the search included “HIV vaccine”, “challenges”, and “lessons learned”. The literature was retrieved using Health Canada-Public Health Agency of Canada Library Services and contained primarily English peer-reviewed journal articles.
Document review: Documents and files associated with the Canadian HIV Vaccine Initiative were reviewed to provide a foundation for the evaluation and to assess relevance and performance. Documents were provided by each partner department. The following types of documents were reviewed: program- and project-level descriptions and administrative materials, Government and departmental level policy and planning documents, performance and monitoring documents, financial information and studies including case studies on Large Team grants (conducted by CIHR). A document review template was developed to facilitate the systematic review of materials.
Surveys: Two online surveys were conducted to determine the impact of CHVI on recipients, initiative delivery and accomplishments, and solicit views related to the performance of CHVI (e.g., perceived impact, program alternatives). The survey was conducted with CIHR recipients (n=32, response rate of 69%) and NRC-IRAP CHTD recipients (n=20, response rate 75%).
Interviews: Key informant interviews (N=39) were conducted to fill gaps identified in the document/file review and to provide evidence and detailed information to help contextualize evidence gathered from other sources. Tailored guides were developed to be suitable for administration with groups of key informants: internal program staff/senior management (n=32), external advisory board members (n=3), and other external stakeholders (n=4).
Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included systematic compilation, review and summarization of data to illustrate key findings; analysis of financial data; thematic analysis of qualitative data from interviews and survey; and comparative analysis of data from disparate sources to validate summary findings.
List of Acronyms
- Advancing the Basic Science
- CHVI Alliance Coordinating Office
- Assistant Deputy Minister
- Addressing Enabling Conditions
- Acquired Immune Deficiency Syndrome
- Access to Information and Privacy
- Bill & Melinda Gates Foundation
- Canadian International Development Agency
- Canadian Institutes of Health Research
- Canadian HIV Technology Development
- Canadian HIV Vaccine Initiative
- Department of Foreign Affairs, Trade and Development Canada
- Departmental Performance Report
- Global HIV Vaccine Enterprise
- Government of Canada
- Health Canada
- Human Immunodeficiency Virus
- Human Papillomavirus
- Industry Canada
- Low- and Middle-Income Countries
- Memorandum of Understanding
- Management Response and Action Plan
- Non-government Organizations
- New Prevention Technologies
- National Regulatory Authorities
- National Research Council
- National Research Council – Industrial Research Assistance Program
- Office of Evaluation (Public Health Agency of Canada)
- Program Alignment Architecture
- Public Health Agency of Canada
- Prevention of Mother-to-Child Transmission of HIV
- Results-based Management and Accountability Framework
- Research and Development
- Supporting Coordinated Efforts
- Small to Medium-sized Enterprises
- Translating Basic Science
Report a problem or mistake on this page
- Date modified: