Section 4: Evaluation of the International Health Grants Program (2008–09 to 2012–13) – Findings — performance
Although a performance measurement strategy was developed, it has not yet been fully implemented. As a result there was a lack of routinely collected performance data which limited the ability of the evaluation team to fully assess the performance of the Program.
To assess Canada’s engagement in PAHO, the evaluation team examined the extent to which Canada has achieved its objectives for its membership in PAHO.
The approach taken to assess the performance of the project component of the Program was guided by a logic model, endorsed by senior management, and some basic performance data existed.
While Health Canada made an effort to have a whole-of-government approach for engagement with PAHO through interdepartmental consultations, evidence suggested that information-sharing and collaboration could be improved.
A whole-of-government approach requires taking into consideration the respective mandates and accountabilities of the departments and agencies involved. While Health Canada had developed the Framework for Cooperation between the Pan American Health Organization and the Department of Health of CanadaEndnote 71 , it did not incorporate expectations for engagement from other federal partners. Therefore, there is still a need for a strategic document to guide the whole-of-government engagement with PAHO that defines respective roles and responsibilities of the federal partners, identifies federal priorities, defines Canada’s objectives for its membership and includes a performance measurement strategy.
The issue of a lack of a Canadian strategy and vision for Canada to enable PAHO to better meet our needs was raised during the interviews and was seen as a gap to the achievement of more focussed results and more meaningful reporting to Canadians. Some partners indicated that they were not involved in the development of the Framework for Cooperation although their institutions were mentioned in the document.
External partners, including the Department of Foreign Affairs and Trade, the Canadian International Development Agency and the Organization of American States expressed a high level of satisfaction with Health Canada’s consultation approach for Canada-PAHO relations. One key informant went so far as to say that the process should serve “as a model”.
However, there are opportunities for Canada to have greater influence by improving internal consultations and coordination of input to PAHO. For example, some internal stakeholders expressed the need to be more involved in the technical content of the official documents developed by PAHO at an earlier stage. They indicated that earlier involvement would enable them to provide a more comprehensive response, and that an earlier contribution to the scientific content of documents would provide a greater opportunity for Canada to influence the policies and programs to reflect domestic priorities.
4.1.1.Advancing Canada’s global health priorities and influence
Evidence showed that the work of the Health Portfolio contributed to the objectives of advancing Canada’s global health priorities and to strengthening Canada’s influence on PAHO.
In terms of advancing Canada’s global health priorities through strategic approaches, evidence indicates that this is done systematically through Canada’s at governance meetings. Position papers developed by Health Canada, in consultation with the Department of Foreign Affairs and International Trade, the Public Health Agency of Canada, the Canadian International Development Agency and Canada’s Permanent Mission to the Organization of American States, as required, reflected Canada’s global priorities and were approved by the Minister of Health. Also, several initiatives implemented in concert with PAHO reflect the efforts made to advance Canada’s own global health priorities.
PAHO is a strategic multilateral partner in the Americas. Interviews and the document review indicated that PAHO is an important health “broker” for Canadian interests in the Region of the Americas. Canada leveraged relationships developed through its membership in PAHO to benefit Canada. PAHO is represented in 27 country offices in the Americas and Canada often used contacts from these country offices as vital sources of information on disease outbreaks, civil strife, and natural disasters which affect Canadian interests. PAHO also plays an important role as a facilitator to gain consensus on global health issues of national and regional importance including: international health regulations, pandemic preparedness, tobacco control, food-borne illnesses, health and the environment, and chronic diseases.
The Pan American Network on Drug Regulatory Harmonisation initiative was one example of where Canada successfully advanced its priorities to benefit Canadians. This initiative directly contributes to fulfilling Canada’s commitment to support PAHO’s Directing Council ResolutionEndnote 72 which aimed to strengthen national regulatory authorities for medicines and biologics in the Americas. The initiative supported the process of pharmaceutical regulatory harmonization in the Americas and corresponded to the Health Canada priority of technical support regulatory/policy development. The development of common standards, so that each country does not need to do primary scientific evaluations/assessments of drugs but instead share results, is intended to decrease the assessment burden and increase timely access to drugs. Regulatory harmonization will ensure that drugs and medical devices entering Canada meet common national standards, thus protecting Canadian consumers from adverse health effects and indirectly costing our health system and economy less.
Studies have demonstrated that the advocacy of Canadian interests has had an impact on the multilateral stage. A series of results were identified in the Americas Strategy relating to advocacy issues, including demonstrating leadership in the Americas, influencing decision making and fostering dialogue. The evaluation of the Americas Strategy indicated that these aspects are at the core of the overall Americas Strategy, and are important to be captured and reported on.Endnote 73 This observation also applied to Canada’s engagement with PAHO.
Canada has achieved results from its membership in PAHO. The 2011 Evaluation of the Americas Strategy carried out by the Evaluation Division of the Department of Foreign Affairs and International Trade examined a range of issues around the Americas Strategy and found that Canada has achieved tangible results through increased engagement with multilateral and regional organizations by leveraging its influence. The Americas Strategy indicated that amongst its priorities, efforts would be made to assist the strengthening of key groups delivering benefits to citizens in the region.
"One of the most important groups in this area is PAHO. A range of support has been provided to meet this objective. Health Canada has been supporting improvements in the governance structure of PAHO. The focus of the work has been on areas such as improved transparency of operations and better tracking and reporting of results. PAHO has been making substantial progress in these areas as a result. The Public Health Agency of Canada is also currently working with Caribbean agencies and PAHO to develop a Caribbean Public Health Agency".Endnote 74
A number of activities implemented by both Health Canada and the Public Health Agency contributed to “promoting the common values of PAHO. Canada played an influential role in strategy/policy development with PAHO over the past years. Attendance at meetings of the governing bodies and at expert consultations provided a forum for the wider dissemination of Canadian-based values related to health and the provision of health-care services and public health priorities.
Canada, for example, placed great importance on technical cooperation with Latin America and the Caribbean, based on the principles of equity and Pan-Americanism and there was evidence that Canada was highly regarded as a result. The following are specific examples that demonstrate that Canada was instrumental in promoting common values of equity:
- In 2012, an initiative from Health Canada aimed at capacity building in mental health among indigenous people was implemented. The purpose was to strengthen intercultural approaches to Mental Health in Canada, Chile, Guatemala and other Latin American and Caribbean countries. A process to exchange experiences and best practices was implemented.
- In 2011, Health Canada presented, co-managed, co-resourced and co-chaired with PAHO in Panama a regional workshop on Health Human Resources and Indigenous Peoples.
- In 2009, Health Canada played a leadership role by co-managing, co-resourcing and co-chairing with PAHO a regional workshop in Ecuador on the Determinants of Health and Indigenous peoples.
To strengthen its influence on the programming, resources and policies of PAHO, Canada has sought opportunities to actively participate in key governance committees and in strategic initiatives. For example:
- Canada actively sought a seat on the Executive Committee of PAHO for the period of 2012 to 2017.
- In 2010–2011, Health Canada worked closely with PAHO to influence the early policy development and subsequent refinement of regional strategies and plans of action, including:
- strengthening National Regulatory AuthoritiesEndnote 75
- refinement of a regional strategy and plan of action for substance use reductionEndnote 76
- development of a regional e-health strategy and plan of action to strengthen health systemsEndnote 77
- development of a regional strategy and plan of action to reduce the harmful use of alcohol Endnote 78
- development of a regional strategy and plan of action on climate changeEndnote 79
- promoting improved Indigenous health including human resources for health and mental health - in response to Resolution CD47.R18 Health of the Indigenous Peoples in the Americas,Endnote 80 and CSP27.R7 Regional Goals for Human Resources for Health 2007-2011Endnote 81.
- supporting health human resources capacity-building in needs-based planning in the Caribbean in response to CSP27.R7 Regional Goals for Human Resources for Health 2007-2011.Endnote 82
The file review also revealed instances where Canada influenced decisions of PAHO. At the 28th Pan American Sanitary Conference (2012)Endnote 83 a resolution to strengthen the coordination of international humanitarian assistance in the health sector during disasters was adopted. Canada was supportive of the spirit of the resolution, which was consistent with Canada’s position of strengthening the international humanitarian assistance system to ensure the effective and efficient delivery of assistance to affected populations. However, Canada voiced several concerns during the meeting pertaining to the need to ensure that any new mechanism needed to be implemented in accordance with specific actions. Canada was able to gain support on these points from several Member States and changes to the proposed resolution were adopted.
4.1.2.Management of Canada’s contribution to the Pan American Health Organization
One objective of Health Canada’s engagement with PAHO is to ensure effective management of Canada’s contribution to PAHO.
Evidence of PAHO’s commitment to achieving effective management was demonstrated by its adoption of the recommendations arising from a broad-reaching review by the Working Group on PAHO in the 21st Century of PAHO/WHO’s technical cooperation programs, its organizational and managerial models, and its relations with Member States and other stakeholders.Endnote 84 As reported in The Pan American Health Organization in the 21st CenturyEndnote 85 the review led to a major reorganization effort that produced changes in PAHO’s program structure, resource allocation, human resources management, and interactions with Member States and other partners. Further, recommendations of the Working Group on Streamlining Governance Mechanisms of PAHO,Endnote 86 in 2006 have led to new governance rules. Among them are criteria for use by the Member States in selecting their nominees, a timeline and procedures for nominations and for holding the election, and the establishment of a new Candidates’ Forum, timed to coincide with meetings of the Executive Committee, to allow candidates to present their platforms and answer questions from Member States.
In addition, PAHO has received positive endorsement of its fiscal and administrative practices from Member Governments, including Canada. It has been open to ways to strengthen its managerial practices and has put in place a number of innovative mechanisms designed to improve the effectiveness and accountability of the Organization. Its independent external auditor undertakes a thorough fiscal analysis of the financial health of the Organization and reports each year to the Directing Council. The Director of PAHO, through an Annual Report, also reports on regional programmatic and financial achievements in health and development, governance, best practices and accountability issues of interest to Canada.
Canada’s participation in the meetings of governing bodies was intended to achieve ongoing monitoring and influence over the broad policy and program issues in the Americas and to ensure transparency and accountability of the Organization’s governance structures. Through participation and influence in the governing committees of PAHO, Canadian representatives aimed to ensure that Canada’s policy and financial contributions to PAHO, including Canada’s portion of PAHO’s program budget, were effectively managed. One of Canada’s principal functions at meetings of the governing bodies is to closely scrutinize program delivery and to ensure that programs advance strategic priorities endorsed by Canada and other member states.
The document review indicated that Canada's interventions to strengthen the governance of PAHO and its oversight mechanisms (audit and evaluation) were constant, convincing and contributed to increase the transparency of the Organization in reporting on results achieved. Through input into the discussions pertaining to budget, planning and administration, Canada has gained a high respect in the hemisphere for its emphasis on good governance, results-based management, and accountability. Interviews conducted in the context of other evaluations confirmed that PAHO considered the Canadian contribution on issues of governance as very useful, as “a model” to quote one respondent.
On the subject of core funding, it was not clear if Canada had in place an approach to advocate for its membership fees to be directed to activities that align with Government of Canada’s objectives. A few external interviewees indicated that some member states were exploring ways to achieve more control over the use of their membership fees; for example, by negotiating that a larger portion of its funding be allocated to the regular programming (projects) of the Organization that aligns with its own governmental objectives and priorities. There may be opportunities to achieve greater benefits by opening up such a dialogue with PAHO. However, currently, all the Health Portfolio funding is allocated to core funding which supports the regular operations of the Organization and membership fees are not earmarked for particular projects (only voluntary contributions can be specified).
In managing the relations between Canada and PAHO, a file review of the activities of the International Affairs Directorate of Health Canada indicated that staff took an active role in leading the development of position papers in consultation with Canadian partners as required (Department of Foreign Affairs and International Trade, the Canadian International Development Agency, the Public Health Agency of Canada, and Canada’s Permanent Mission to the Organization of American States).
There was also evidence that Health Canada had a process in place to formally instruct the Canadian delegation to advocate positions on policy and management issues within the Organization that were consistent with Canada’s policy frameworks, Treasury Board policies and procedures as well as other Government and modern management practices, which were adapted, as appropriate for the international environment.
However, there was no evidence that Health Canada had assessed the outcomes of its approach, and the effectiveness of the dialogue with PAHO. As the lead department, Health Canada did not have a mechanism in place to ensure regular feedback from partner departments on the effectiveness of Canada's dialogue with PAHO and suggestions on how it could be improved.
Despite evidence that Canada’s consultation process with Canadian stakeholders was viewed as “a model”, there are opportunities for improvement. The information produced as a result of attendance at official meetings was not necessarily made available to all interested internal parties who would benefit from being informed. Some internal partners indicated that they received their information directly from their own contacts at PAHO in Washington, instead of through Health Canada.
4.1.3.Communicating the achievements of the Pan American Health Organization
Another objective of Canada’s engagement with PAHO is communicating to Canadians the achievements of PAHO.
Canada has been actively engaged in promoting health in the Americas since 1971, when it officially joined PAHO/World Health Organization. To ensure transparency and accountability, it is important that the Canadian public be informed on Canada’s engagement with PAHO.
In an effort to communicate and report to the Canadian public on the results of Canada’s engagement with PAHO, the PAHO Canada Portal was established in 2009 as a collaborative user-friendly space for discussion of health issues in the Americas. This initiative, led by PAHO in collaboration with Government of Canada institutions who work regularly with the organization, collected information important to health professionals and researchers in Canada and the whole of the Americas. However, the PAHO Canada Portal has not been updated regularly and there was no evidence that its effectiveness had been assessed since it was established. Most interviewees were unaware of the existence of the portal for communicating information on the engagement of Canada with PAHO. The Departmental Performance Reports available to Canadians briefly covered Canada’s relations with PAHO, but were not sufficient to adequately inform the Canadian public.
There was evidence from the file review that the International Affairs Directorate was exploring options over the last few years to increase public information, knowledge sharing and consultation with respect to Canada’s engagement with PAHO. The files revealed that discussions took place with departmental experts to initiate scoping options to enhance the current consultation process. There was also reference to pursuing options to enhance its web presence, including providing web updates on Canadian engagement with PAHO. However, the evaluation team found no evidence that this work was completed and that actual progress took place.
During the period covered by the evaluation, the project component facilitated the Health Portfolio’s participation in international activities that had goals and objectives that complemented national objectives. The implementation of the project component was guided by a logic model and a performance measurement strategy, included in the 2008 program authorities.
The current evaluation drew on all relevant findings and conclusions flowing from a Mid-Point Review of the ProgramEndnote 87 which was conducted in 2010. The analysis was complemented by additional data.
4.2.1.Design and Delivery:
The complicated delivery model involving two branches of Health Canada (Strategic Policy Branch and Regions and Programs Branch) in the disbursement of an irregular, small unstable budget did not contribute to effectiveness. Among the implementation issues identified during interviews and from the document review were the following:
- lack of stable funding due to the fluctuation in the rates of the two membership fees (US dollar and Euro)
- delays related to the approval process for projects
- management decision that projects had to be completed in the same year as funding
- delays in selecting and processing projects due to need to pay memberships first
- lack of clarity in the roles and responsibilities of the two branches involved in the implementation of the component.
Significant changes occurred to the design and delivery of the project component during the five years covered by this evaluation, as described in the program governance section (3.1).
Over the last five years, it was the Regions and Programs Branch that mainly managed all elements of program-based projects, including assessing requests for proposals, negotiating grant agreements, issuing payments, measuring performance and contributing s to program evaluation, as well as paying membership fees on behalf of Canada.
The Strategic Policy Branch provided policy advice and analysis of international issues affecting health, including the identification of priorities for projects. The Strategic Policy Branch was also responsible for providing guidance to the Minister regarding Health Canada’s participation with PAHO, IARC and the Global Health Research Initiative.
The shared responsibility for the program between the Strategic Policy Branch and the Regions and Programs Branch of Health Canada led to a lack of clarity about roles and expectations. The Regions and Programs Branch became accountable for implementing the Program and although the Strategic Policy Branch was involved in project selection, the final decision for funding was the responsibility of the Regions and Programs Branch.
Interviews and the 2010 Mid-Point Review indicated that organizational changes contributed to a lack of clarity of roles and responsibilities between the two branches involved in the implementation of the project component of the program. However, there is evidence that efforts were made to discuss this issue; a document delineating the roles and responsibilities of all involved was drafted but never finalized.
Between 2008-2009 and 2009-2010, project component funds were allocated to three streams: HIV AIDS, Tobacco Control and a General Stream. This approach further complicated the implementation of the project component of the Program as the staff from the respective technical branches in Health Canada had to be consulted and were also involved in the project selection process.
Several additional factors affected the implementation of the project component. Budget decisions led to a reduction to the International Health Grants Program by $895,000 starting in 2009-2010. Within the next year, a new direction was also taken with the Public Health Agency assuming the lead for the HIV vaccine component and the permanent transfer of funds to the Strategic Policy Branch. These budgetary decisions and obligations to PAHO and other international memberships, left limited resources for individual projects, depending on remaining funds after currency conversion. As a consequence, focussed work through a HIV stream stopped, although HIV projects were still eligible for funding through the general stream.
In 2010, a dedicated co-ordinator position was established by the Regions and Programs Branch to manage and administer the call for proposals which contributed to a more effective interface between the two Branches involved.
The interviews confirmed that the transfer of the Program to the Office of International Affairs for the Health Portfolio will provide an opportunity to review the design and delivery model of this component. However, the call for proposals for 2012-2013 was cancelled due to the organization’s transition to a shared service for the Health Portfolio and the renewal of the program authorities. Although an improved delivery model is being discussed, it has yet to be decided and tested.
4.2.2.Achievement of objectives and outcomes
The project-based component is meant to respond to current federal priorities and emerging global health issues, commitments and obligations. The document review observed that program priorities for the project component are informed by departmental priorities, internal planning and priority setting for global health, analysis of international reports and consultation with stakeholders. Specific parameters were established through program authorities for the grant agreements in terms of eligibility, maximum amount payable, application requirements, process and accountability. A detailed process was introduced to manage the projects.
During the last five years, a total of 46 projects (including commitments such as IARC, the Office of Economic Cooperation and Development Health Committee, and the Global Health Research Initiative) were funded. The list of projects funded is available in Appendix 2. Also, observations flowing from two project case studies on the benefits to Canada from grants to two organizations will be presented in the next section of the report.
While project outputs generally contributed to the achievement of short-term outcomes, it is unclear whether the project funding assisted the program in achieving some level of expected longer-term outcomes. Considering that only a small fraction of the Program budget is available for projects, the expected impact is mostly commensurate with the level of investments. Routinely collected performance data on outcomes was unavailable although outputs are reported in project final reports and in Health Canada’s Reports on Plans and Priorities. The file review and some interviewees mentioned that there were some discussions by the Regions and Programs Branch about the introduction of performance measurement to report on the results of these projects but the lack of resources was cited by interviewees as the main impediment to do so.
While some projects’ final reports included a dissemination plan to ensure knowledge transfer, the evaluation team found that no systematic approach was in place to ensure that the information produced by these projects was disseminated and/or informed the development of national health policy. However, there was evidence of the sharing of the results of some funded projects (bag lunch discussions/presentations etc.) and the file review revealed that several project applications provided a dissemination plan. The interviews confirmed that this uneven approach to dissemination was related to the fact that during most of the five year period of this evaluation, there was no continuity due to staff turnover in the Program.
Some projects identified in the case study and document review did inform Canadian policies and priorities and provided demonstrable benefits to Canada. The case studies (described in the highlighted text that follows) illustrate the following tangible benefits to Canada of two projects funded through the Program:
- A grant to the Centre de recherche interdisciplinaire sur la biologie, la santé, l’environnement et la société, Université du Québec. This project addressed two strategic health priorities for Canada: health and the environment and food, and health and consumer product safety. The grant contributed to the objectives of the Program in that it:
- provided relevant information for regulatory agencies in Canada, Costa Rica and elsewhere in the world
- increased knowledge on a current and emerging global health issue that will inform policy and program development in Canada and elsewhere
- strengthened Canada’s leadership role in a global health issue
- enhanced global health capacity in an area of priority for the Government of Canada and Health Canada priority for 2010-2011: to “reduce health and environmental risks from products and substances, and healthy, sustainable living and working environments.” And the sub-activity on pesticide regulation
- contributed to improving collaboration on international responses to emerging and current priority global health issues
- generated knowledge and information for communities about the effects of pesticide exposure and safer ways to manage and use pesticides
- A grant to PAHO for the establishment of collaborative relationships between the Province of Tierra del Fuego, Argentina, and Nunavut, Canada (2010-2011). The project presented an opportunity for a technical cooperation among countries, and set the stage for a model of Pan American cooperation and solidarity between the most northern and most southern geographic areas of the hemisphere. Collaboration and cooperation among the countries of the hemisphere, supported by international organizations, was seen as an essential tool for establishing and developing strategic areas, as well as addressing common issues, with the purpose of reducing inequities in the hemisphere and within the countries. The project built on the existing relationship between Canada and PAHO and forged a new link on a sustainable basis with Argentina. It aligns with PAHO’s technical cooperation strategy as well as with Canada’s domestic agenda. The project complemented the following Program project component priority objectives:
- inform health decisions with international evidence through exchange of experience that advance health innovation and strengthen health system sustainability, and by fostering information exchanges on indigenous health and issues of mutual interest
- identification, assessment and promotion of approaches, models and best practices responding to Canada’s global health priorities (i.e., health systems and capacity building; health promotion and disease prevention , health and the environment)
- promote better health linkages through increased collaboration and strengthened relationships with key partners and stakeholders (i.e., Argentina and other member countries of PAHO)
- strengthen Canada’s leadership on global health issues, such as indigenous health.
Case Study #1:
Grant to the Centre de recherche interdisciplinaire sur la biologie, la santé, l’environnement et la société, Université du Québec.
A grant of $68,874 facilitated and leveraged the work already done by the Regional Institute for Studies of Toxic Substances of the National University of Costa Rica on prenatal manganese exposure from mancozeb. Mancozeb is a pesticide used extensively in Canada and worldwide.
The objectives of the study were to expand analysis to include maternal and child biomarkers of manganese exposure and to examine the relationship between these biomarkers, maternal biomarkers of effect, birth outcomes and children’s neurodevelopment and health.
The deliverables were peer reviewed scientific publications, as well as accessible brochures for the communities involved with the objective of reducing exposures. Results were disseminated to governmental agencies in Costa Rica and Canada, to international authorities and to the scientific community through publications.
Case Study #2
Establishment of collaborative relationships between the Province of Tierra del Fuego, Argentina, and Nunavut, Canada (2010-2011).
A grant of $44,000 to PAHO facilitated technical cooperation between Argentina and Nunavut, Canada. Nunavut and the Province of Tierra del Fuego, Argentina share similar demographic trends and health challenges impacting the health of their populations, including:
- operating within the context of similar health systems
- large youth populations
- youth share similar health risks, including high rates of sexually transmitted infections and drug and alcohol abuse
The overall objective of this project was to continue to improve the health of Canadians through an informal comparative policy dialogue and international collaboration on common issues in health and health care delivery. Specifically, the project was intended to:
- build on the existing relationship between Canada and PAHO and strengthen capacity among indigenous and Inuit health leaders in the hemisphere
- foster information exchanges on indigenous health and issues of mutual interest
- highlight best practices and lessons learned
- address the principal health problems of common concern (i.e. culturally appropriate resources for health, innovative approaches to the delivery of primary care services in remote locations, including the use of telehealth, prevention and control of communicable diseases such as tuberculosis, and health promotion including nutrition, diet and physical activity and mental well-being) and their determinants;
- provide training for appropriate health care services for indigenous populations in both territories
- organize preventive and promotion primary health care services
The key deliverable was a full report documenting the highlights of the exchange and plans for a sustainable relationship.
Further examples of 2011-2012 projects that informed Canadian policies and priorities include:
- funding to the Izaak Walton Killam Hospital for Children to enhance capacity to effectively address child and youth mental health across Central America in Central America allowed Canada to provide leadership on an issue of increasing priority. Youth mental health (including depression, anxiety, suicide etc.) has frequently been cited in the Canadian media and among the general public as a priority issue.
- funding to a University of British Columbia project involving the promotion of public health engagement and education through live forums and electronic media for chronic disease management, led to a better understanding of how health technologies can support chronic disease management. Both the file review and interviews confirmed that the projects contributed to the Program’s objectives; however, they may not be yielding significant impact in view of the amount and duration of funding.
The funding of individual projects gave the Program an opportunity to support national and international initiatives that contributed to the achievement of the Program’s objectives. The interviews confirmed that the project component provided flexibility to fund interventions of priority and allowed the Program to contribute to, and benefit from, the work of partners.
The transfer of the Program to the Office of International Affairs of the Health Portfolio within the Public Health Agency provides an opportunity to test alternative methods of delivery/implementation of the project component of the Program. Internal interviewees were asked to provide suggestions on how to improve the implementation of this component. The following points were raised:
- Consider assigning a stable budget to the project-based component in order to ensure a more strategic selection of projects and a multi-year funding approach. While projects’ duration could not exceed one year, in practice most of the projects had to be implemented within a four to six month period as it was not known before the Fall whether remaining funds would be available for individual projects after membership payments were made. Providing greater certainty as to the amount of available funding year over year would allow the project based component to be more pro-active and less reactive in the selection of projects.
- Considering the fact that the Strategy for the Americas has been renewed, Canada will remain a member of PAHO for many years to come. Therefore, prepaying this membership or a portion of it for pursuant year, would allow the Department to know more exactly and sooner the amount of funds available for project funding.
- Conducting an anticipatory call for proposals would ensure that qualified projects are available when funds are made available.
- Funding fewer and larger individual projects with international organizations or national partners involved in priority areas for the Portfolio would be more efficient. However, it should be noted that others cautioned that funding only large, multi-year projects would reduce the ability to react to emerging issues and that some very small projects have had significant and long-lasting impacts.
- Review and simplify the reporting requirements for the project component. Collect performance data for a few indicators.
4.2.3.Effectiveness of Canada's membership in the International Agency for Research on Cancer
Canada had clear objectives for working with IARC (refer to section 1.2.1). Consistent with these objectives, Canada had advanced some specific priorities with tangible results. IARC’s publication program is one of the mechanisms that IARC uses to fulfill its mission to disseminate information on the epidemiology of cancer, on cancer research and on the causation and prevention of cancer throughout the world. IARC’s monogram publications are considered “state-of-the-art” and an authoritative review of research results on important substances, products and processes that lead to cancer. The file review revealed that Canada contributed to efforts to reduce the backlog of publications which led to improved access to publications. Canada promoted a faster production of the monographs and an increase in the frequency of publications. IARC also directed its publication program to facilitate access to new publications by making available several online electronic databases. New access policies of IARC aligned with the Canadian Institutes of Health Research’s Open Access Policy.
IARC played an important role in building research capacity. Canada provided funding to support senior scientist awards. The education and training program expanded and there was great interest in the scientific community to apply.
In addition, Canadian fellows and scientists were invited to partner on specific areas of research. Health Canada, the Canadian Institutes of Health Research, and the Public Health Agency of Canada contributed to discussions at the Governing Council and contributed their extensive experience with cancer research and governance of multilateral international health agencies.
Through its membership, Canada leverages the scientific information, research, publications and studies on cancer to improve domestic policies and regulations. The information from IARC is used by different parties in Canada. For example:
- The Canadian Cancer Registry (maintained by Statistics Canada) uses IARC reporting rules to ensure data from different provincial and territorial jurisdictions is comparable.
- Health Canada, the Public Health Agency of Canada, the Canadian Institutes of Health Research, federal regulatory agencies responsible for the safety of food, water, air and consumer products, universities, research establishments, non-governmental organizations, and the private sector (pharmaceutical companies, health product manufactures, etc.) make extensive use of IARC publications to help make decisions related to the health of Canadians. Of particular importance are the regulatory responsibilities of the federal government and the obligation to ensure the safety of Canadians.
- Results of many studies led by IARC have impacted on some government policy decisions. For example, the vaccination of adolescent girls against human papilloma virus to protect against the development of cervical cancer. Another example provided during an interview with staff is the on-going international studies on nutrition and cancer and on the potential relationship between cell phones and brain cancer and how this will have far reaching consequences and influence governments recommendations.
In addition, membership in IARC provided research fellowship and training opportunities to Canadians each year. Interviews and the file review demonstrate that Canadian researchers have been the recipients of many of these opportunities.
The document review indicated that Health Canada actively participated in IARC through the Governing Council and the Scientific Council. Through the Governing Council, Health Canada influenced how resources were spent, what research was conducted and how information was disseminated. Canada also sat on the Scientific Council, which has the primary role to ensure the scientific integrity of IARC and to provide impartial scientific advice to the Director and the Governing Council of IARC.Endnote 88
The Canadian delegation that attended the meetings of IARC always included a scientific representative from the Canadian Institutes of Health Research. The continuity of this representation from the Canadian Institute of Health Research linked Canada to an international network of researchers and up-to-date research results and procedures that would not otherwise have been available to Canada. Canada also benefited from IARC’s role to direct and analyse the results of cancer research and guide countries in their efforts to institute best prevention practices.Endnote 89
The Health Portfolio, academia, and non-profit organizations were the main beneficiaries of the work conducted by IARC. The significant experience that Canadian cancer researchers bring to the work of IARC is acknowledged in that the results of Canadian research are often used in the production of its monographs.
The Treasury Board Policy on Evaluation (2009) defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment of economy and efficiency is based on the assumption that departments have standardized performance measurement systems and that financial systems use object costing.
The evaluation could not conduct an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes for the Program for the following reasons:
- An assessment of how PAHO uses Canada’s assessed and voluntary contributions to achieve expected outcomes was not possible. Member States have agreed to not conduct their own evaluations of PAHO since PAHO has its own review processes.
- The short-term nature of many of the individual projects funded by the Program made it difficult to assess progress toward expected outcomes.
- Performance data that is linked to the quality and types of outputs was not routinely collected by the Program.
Considering these issues, the evaluation provided observations on efficiency and economy based on findings from the document review as well as perspectives from key informants. More specifically, observations were provided on program delivery efficiency and overall expenditures.
4.3.1.Observations on efficiency
This section provides a review of the factors that impacted program delivery, as well as examples of areas where there are opportunities for greater program delivery efficiencies.
External interviewees identified opportunities for greater efficiency in terms of the Government of Canada’s engagement with PAHO. In particular, it was observed that there was no whole-of-government approach, or coordinated process, governing the relationship of federal departments and agencies with PAHO. For example, the Framework for Cooperation between the Pan-American Health Organization and the Department of Health CanadaEndnote 90 only governed Health Canada’s relationship with PAHO. A coordinated, whole-of-government approach would enable Canada to be more strategic when establishing Government of Canada priorities for its engagement with PAHO, as well as provide the opportunity to leverage the work of federal partners.
The document review and interviews also identified opportunities for greater efficiencies in the administration of the individual projects, particularly in the project selection process. A rigorous selection process, involving expert staff from across the organization in the review and selection of projects, was implemented by Health Canada’s Regions and Programs Branch to ensure that individual projects being selected aligned well with the objectives of the project component. The document review and interviews suggested that the investments of time from expert staff from across the organization in the review and selection process contributed to the implementation of projects that were relevant and produced expected outputs and short-term outcomes. While there is a benefit to having a diverse range of expertise, particularly to assess highly technical or complex proposals, it was suggested that the proposal review and decision-making process was more onerous than necessary and that it exceeded the accountability requirements for small investments of short duration and low risk, as well as general requirements related to grants as a funding instrument.
4.3.2.Observations on economy
This section provides observations on how Program inputs have been optimized to achieve outcomes. More specifically, it presents observations on the funding of PAHO and the stability of funding for individual projects.
The Program pays Canada’s assessed contribution to PAHO. The assessed contribution is used to cover the operating costs and the core work of the Organization. Moreover, as described in section 1.4, federal departments and agencies, in particular the Canadian International Development Agency, also provide sizable voluntary contributions to PAHO. In interviews with the Canadian International Development Agency, it was observed that a portion of their voluntary contributions also support the operating costs and the core work of PAHO. The potential for, and the extent of, overlap and duplication of funding to support operating and core activities of PAHO is an area to explore further in the context of a whole-of-government approach to Canada’s engagement with PAHO.
Key informant interviews and the file review confirmed that the projects provided good return for investment. The relatively modest budget invested in 46 projects and initiatives enabled the Government of Canada to develop or maintain relations with several national and international partners and to leverage knowledge and have access to networks and platforms that have had, in some instances, long-lasting effects on priority research and policies.
Overall, the program used minimal resources to implement and deliver the project component. The resources required to administer the project component were absorbed within the existing resources in the Branch. In practice, several employees were involved in the administration of the project component for a portion of their time.
In terms of funding to individual projects, the availability and timeliness of funding were two factors that affected the optimization of inputs.
Funding of individual projects occurred yearly only if there were remaining funds available after other priorities were paid for (e.g., PAHO and IARC memberships and other international commitments). For example, 97% of program funds in fiscal year 2008-2009 and 92% of funds in 2009-2010 were used for memberships.
As membership fees are paid in foreign currencies (US Dollar and Euro), the fluctuations in the currency exchange rate also directly impacted on the funds available for individual projects. As a consequence, there were no calls for proposals issued in 2009-2010 or 2012-2013. The Program was seen by some program staff as a program whose primary objective was payment for two memberships (PAHO and IARC) and three initiatives (World Health Organization — Framework for Control on Tobacco, Organization for Economic Co-operation and Development Health Committee Working Group, and the Global Health Research Initiative lead by International Development Research Centre (cancelled since 2011).
The Mid-Point Review already brought the issue of irregular funding of the individual projects to the attention of the Program senior management as follows:
“To ensure that the Program can be responsive to current federal priorities and emerging global health issues, through the issuance of grants, the Department will need to find a solution to address the increasing costs of memberships and their impact on the Program attaining its stated program objectives”.
While efforts were made before the mid-point review to find a solution to stabilize the project funding situation, there was no evidence that efforts to find an in-house solution were sustained. Interview respondents stated that discussions on possible options (including implementing an anticipatory call for proposals, the idea of multi-year funding, funding only a few large projects) will be further explored before the next round of project funding in 2013-2014.
The timeliness of available funding to allow for sufficient time for the call for proposal and project decision-making process was also a factor that impacted on the achievement of outcomes. The file review indicated that in most instances, it was not possible to fund projects before the third quarter of the fiscal year. However, since projects had to be completed during the same fiscal year as funded, this impacted the selection of projects, and limited the ability to be strategic in project selection and, therefore, on the ability to achieve outcomes. Furthermore, it was difficult to demonstrate intermediate or long-term outcomes for projects that were small and of short duration.
Approval of program authorities is currently being sought for 2013-2014 to 2018-2019. During the latest discussions with the Treasury Board of Canada Secretariat, senior management was informed that, going forward, the payment of the memberships will be due in the last quarter of the fiscal year. A revised payment schedule for Canada’s assessed contributions is being incorporated as a result of the renewal of the authorities for the program. Typically, PAHO requires member countries to pay their annual assessed contribution by the first of January each year; however, it has been long standing practice between Health Canada and the Organization to delay payment until April in order to align with the Government of Canada’s fiscal year. Movement to a fourth quarter payment schedule will pose challenges for allocating residual funding for individual projects as the precise amount of available funding will only become known near the end of the fiscal year after all assessed contributions have been paid.
Anticipatory calls for proposals could be established to address this; however the program will be much less flexible in terms of responding to strategic opportunities that may arise over the course of the fiscal year. Unless a stable amount is dedicated to the selection of projects at the beginning of the fiscal year, it will be impossible to implement a project during a given fiscal year (i.e., within the last 3 months of a fiscal year), given the timelines associated with the project selection process, including: call for proposal, project selection, negotiation of agreements, approval process and implementation of the project.
Report a problem or mistake on this page
- Date modified: