Evaluation of the Office of International Affairs for the Health Portfolio 2012-13 to 2017-18

List of Acronyms

AMR

Antimicrobial resistance

APEC

Asia-Pacific Economic Cooperation

BESTD

Bilateral Engagement, Summits and Trade Division

CARPHA

Caribbean Public Health Agency

CFIA

Canadian Food Inspection Agency

CIHR

Canadian Institutes of Health Research

DGO

Director General’s Office

FCTC

Framework Convention on Tobacco Control

GAC

Global Affairs Canada

GHSA

Global Health Security Agenda

GHSI

Global Health Security Initiative

G&Cs

Grants and Contributions

IANPHI

International Association of Public Health Institutes

IARC

International Agency for Research on Cancer

IHGP

International Health Grants Program

ISF

International Strategic Framework

MRD

Multilateral Relations Division

ODSID

Official Delegations and Strategic Integration Division

OECD

Organisation for Economic Co-operation and Development

OIA

Office of International Affairs for the Health Portfolio

O&M

Operations and Maintenance

PAHO

Pan American Health Organization

PEC-IAWC

Partnership Executive Committee International Affairs Working Committee

PHAC

Public Health Agency of Canada

PIE-WG

Portfolio International Engagement Working Group

PMPRB

Patented Medicine Prices Review Board

UN

United Nations

WHO

World Health Organization

Executive Summary

The purpose of this evaluation was to assess the design and delivery of the Office of International Affairs (OIA) for the Health Portfolio as a shared service. This assessment focussed on OIA activities conducted from April 2012 to September 2017.

Program Description

In June 2012, as part of the Health Canada and the Public Health Agency's (PHAC) Shared Services Partnership Framework Agreement, Health Canada's International Affairs Directorate and PHAC's International Public Health Division were merged to create the Office of International Affairs for the Health Portfolio under PHAC's Strategic Policy, Planning and International Affairs Branch.

Since 2012, OIA has undergone numerous internal organizational changes (e.g., three Directors General) and four internal reorganizations to align files with division mandates. In the fall of 2016, the Strategic Policy, Planning and International Affairs Branch separated into two separate offices: OIA and the Office of Strategic Policy and Planning, which gave the Director General of OIA the authorities of a branch head and a direct reporting relationship to both the Health Canada and PHAC deputy heads.

OIA activities aim to strategically advance the Health Portfolio's international priorities in an effort to support Canada's domestic health policy and the Government of Canada's foreign policy objectives, as well as ensure that Canada's interests and values are reflected in the global health agenda. Overall, the activities carried out by OIA can be categorized into three broad groups: global (international) health policy analysis, advice and support; international relations and engagement; and priority setting and international performance reporting (strategic planning). The current total average annual budget allocation for OIA is approximately $7.5M, provided by both Health Canada and PHAC, with $5.9M in operating funding and $1.6M in grants and contributions (G&Cs).

Conclusions

Office of International Affairs Design and Delivery

The mandate of the OIA is quite broad; however, it reflects the complexity of its work and the expanding global health context within which this work is conducted. Many internal and partner/client key informants and survey respondents agreed that OIA's mandate is appropriate and clear. OIA's activities are in line with its mandate. However, with the breadth of activities conducted and finite resources available, OIA must prioritize its activities; therefore, on occasion, other activities, such as trend analyses and training for the Health Portfolio, occurred less frequently than originally planned.

The evaluation found that, while governance mechanisms were in place, the Partnership Executive Committee International Affairs Working Committee (PEC-IAWC) could have been functioning more effectively. This Assistant Deputy Minister (ADM)-level committee, which was often delegated to other staff, appeared to be more of an information-sharing body than one where strategic discussions took place. Furthermore, during the time of this evaluation, OIA's Director General was a member of PHAC's Executive Committee, but did not sit on Health Canada's Executive Committee. Some key informants (mostly from OIA and one from Health Canada) saw the value in having the Director General at this table to make stronger linkages across OIA and the programs' international activities; however, a few other Health Canada key informants did not agree, as other forums exist for conversations on international work.

There were mixed reviews of OIA's products and services. Key informants were generally positive; however, survey respondents were less positive, especially in regards to OIA's Weekly Newsletter, its global health policy analysis and advice, and its approach to information sharing and intelligence gathering.

Overall, there was agreement among key informants that the shared services model has been effective for international activities. This model has many perceived advantages and it has demonstrated that OIA is seen as providing added value to the coordination of Health Canada and PHAC international activities. Key informants identified a few challenges, such as the need to coordinate across many branches; however, it was felt that these challenges did not interfere with OIA's ability to carry out its activities.

Generally, OIA's relationship with the branches was seen as working well. Most internal and partner/client key informants indicated that the roles and responsibilities were clear when the Minister or the deputy heads attend an event. Roles and responsibilities were less clear when the issue was of a more technical nature. Some internal and Portfolio partner key informants noted that OIA roles and responsibilities were not formally documented in a way that could be shared across the Portfolio. Another challenge noted was that there is uneven capacity across branches, with only some having well-established international groups, which can create tension when it comes to roles and responsibilities in terms of the perception of OIA stepping into the program's area of expertise.

Impact

OIA has engaged in numerous multilateral and bilateral relationships, in conjunction with other Portfolio partners. While it is difficult to determine the impact of these joint activities, there is evidence of impact related to the higher-profile activities.

This impact includes supporting advancement of Canada's domestic priorities, leading the Portfolio's engagement in the development of Canadian policy positions on international issues, and identifying ongoing priorities and strategic opportunities for Canadian leadership, which is achieved through activities such as:

• managing and administering the International Health Grants Program (IHGP);
• sitting as a member on the World Health Organization (WHO) Executive Board (which ended in May 2018) and the Pan American Health Organization's Executive Committee (2017-18);
• acting as the international secretariat for the Global Health Security Initiative (which was transferred to the Health Security Infrastructure Branch in July 2018);
• playing a role in helping to influence the negotiation and implementation of international trade agreements; and
• providing Health Portfolio coordination and policy support through its liaison with Global Affairs Canada (GAC).

While not addressing impact, it is worth noting that key informants from international organizations were complementary in discussing their relationships with OIA. In addition, pluralities of survey respondents were satisfied with OIA's overall approach to advancing global and domestic health priorities through its international engagements.

Demonstration of Economy and Efficiency

There is evidence that, since 2014, OIA has been running an annual deficit. As OIA activities have increased, so too has the demand on resources. For example, there were no resources assigned to G7 and G20 activities as part of the Shared Services Partnership Agreement; however, there is now a team of OIA staff dedicated to these activities. In addition to these new activities, developing internal capacity to support existing priority files such as health and trade has contributed to increased funding pressures. A number of internal and partner/client key informants felt that the organization was under resourced, especially with an increased profile internationally for the Government of Canada, but others felt that OIA had sufficient resources to fulfill its perceived mandate.

Even though OIA has been able to secure funds by putting forward special requests, there is a risk that this is not sustainable; however, it should be noted that for 2017-18, OIA received a significant and permanent increase to its budget (approximately $1.3M).

Recommendations

Recommendation 1

Clearly articulate roles and responsibilities regarding international files, and communicate those within the Portfolio

Overall, partner/client and external key informants indicated having good working relationships with OIA. While many key informants and survey respondents stated that roles and responsibilities are well established when the Minister or deputy heads attend an event, others noted a lack of clarity, especially as it relates to matters of a technical nature. Both OIA and the branches have international roles to play; however, there are some areas where it is less clear who is leading on a particular file, or who should be leading or supporting the international work on that file. Since OIA roles and responsibilities are not formally documented, and understanding that these technical files and events require additional conversations, it may be helpful if the roles and responsibilities were more clearly articulated to share within the Health Portfolio. Furthermore, existing governance tables could be used to initiate discussions regarding roles and responsibilities.

Recommendation 2

Further explore opportunities to improve products and services

The Office of International Affairs for the Health Portfolio provides a number of products and services to internal clients and key stakeholders. Generally, key informants reported that OIA is proactive, provides well-thought-out advice, is forward thinking, produces useful materials that are appropriately tailored to the audience, is very responsive, and they generally feel well served by OIA. However, there is dissatisfaction with some OIA products and services, especially among survey respondents (e.g., usefulness of OIA weekly Newsletter, OIA’s global health policy analysis and advice, its approach to information sharing and intelligence gathering from international engagements). It is unknown why this dissatisfaction exists; therefore, OIA must first determine the reasons behind it.

Recommendation 3

Explore ways to measure the impact of OIA activities

OIA engages in numerous multilateral and bilateral relationships, and supports these international engagements in conjunction with other Portfolio and Government of Canada partners. While it is difficult to determine the impact of all these joint activities, there is evidence of impact related to the higher-profile activities, such as managing and administering the IHGP, being involved with key multilateral organizations (e.g., the United Nations, WHO and Pan American Health Organization), acting as the international secretariat for the Global Health Security Initiative, and supporting negotiations and implementation related to international trade. Impacts from these activities include supporting the advancement of Canada’s domestic priorities, leading the Portfolio’s engagement in the development of Canadian policy positions, identifying ongoing priorities and strategic opportunities for Canadian leadership, and advancing policy positions that are consistent with existing Canadian efforts. In order to better demonstrate OIA’s value, it may be worth examining ways to measure the impact of all of its activities as much as possible.

Management Response and Action Plan

Evaluation of the Office of International Affairs for the Health Portfolio (OIA)
2012-13 to 2017-18

Recommendations	Response	Action Plan	Deliverables	Expected Completion Date	Accountability	Resources
Clearly articulate roles and responsibilities regarding international files, and communicate those within the Portfolio.	1. Management supports this recommendation. Will build on engagement with Health Portfolio (HP) branches and governance committees to improve collective understanding of roles and responsibilities, communications and planning.	1.1. Strengthen OIA engagement with HP branches and governance committees to facilitate discussions on roles and responsibilities as well as priorities for HP international engagement.	Strengthened engagement supported by OIA participation in branch-level and governance committee meetings, as required.	June 2019	OIA-DG OIA-DG	Within existing OIA resources, assisted as requested by PHAC Office of Strategic Policy and Planning (OSPP), HC Strategic Policy Branch (SPB) and Chief Finance Officer Branch (CFOB).
			Implementation of an approach to facilitate HP international priority-setting and reporting that is aligned with HC/PHAC processes.	September 2020
		1.2. Update communications products and tools to clearly articulate our collective roles and responsibilities and OIA services.	Updated communications products and tools.	December 2019	OIA-DG	Within existing OIA resources, assisted by Communications and Public Affairs Branch (CPAB) staff.
Further explore opportunities to improve services and products.	2. Management supports this recommendation. Will identify opportunities for continuous improvement of OIA products and services and will strengthen HP awareness and capacity.	2.1. Improve feedback mechanisms to identify gaps and opportunities to improve products and services.	Implementation of regular feedback mechanisms.	June 2019	OIA-DG	Within existing OIA resources.
		2.2. Provide learning opportunities for HP program area counterparts to increase knowledge of international affairs and build capacity in international policy development, global health governance, and multilateral negotiations, thereby improving ultimately the products developed.	Learning opportunities for HP program area counterparts.	April 2020	OIA-DG	Within existing OIA resources.
Explore ways to measure the impact of OIA activities.	3. Management supports this recommendation. If feasible, will further inform our understanding of roles and responsibilities regarding HP international files.	3.1. Explore the feasibility of measuring the impact of OIA’s activities as a shared service.	Completed environmental scan of impact assessment scope and options.	March 2019	OIA-DG OIA-DG	Within existing OIA resources, assisted as required by Office of Audit and Evaluation (OAE) and OSPP.
			OIA Management Table to consider capacity/ability to measure the impact of OIA services.	June 2019

1.0 Evaluation Purpose

The purpose of this evaluation was to examine the design and delivery of the Office of International Affairs (OIA) for the Health Portfolio as a shared service, and focused on OIA activities conducted from April 2012 to September 2017.

2.0 Program Description

2.1 Program Context

OIA has gone through several organizational changes since it became a shared service. In June 2012, as part of the Health Canada/PHAC Shared Services Partnership Framework Agreement, Health Canada’s International Affairs Directorate and PHAC’s International Public Health Division were merged to create the Office of International Affairs for the Health Portfolio, under PHAC’s Strategic Policy, Planning and International Affairs Branch.^{Footnote A} There have been three Directors General for OIA between 2012 and the present day.

Between 2014 and 2017, OIA underwent four different changes to Health Portfolio international governance to increase engagement from within the Portfolio and improve the effectiveness of its governance committees. At the same time, there have been a number of changes to division mandates within OIA, which have been carried out to reflect shifts in approaches and roles on the international stage. For example, in December 2015, the health and trade file was transferred from what is now called the Official Delegations and Strategic Integration Division to the Bilateral Engagement, Summits and Trade Division (BESTD), in order to take advantage of its overall understanding of bilateral relationships with key countries.

In 2016, OIA and the Office of Strategic Policy and Planning became stand-alone entities to bring an enhanced profile and direction to their respective mandates. This gave the Director General of the OIA branch head authorities and a direct reporting relationship to both the Health Canada and PHAC deputy heads.

While OIA is administratively situated within PHAC, it serves both PHAC and Health Canada. In addition to providing centralized services for both organizations, OIA plays a horizontal policy and liaison role for other Health Portfolio organizations, such as the Canadian Food Inspection Agency (CFIA), the Canadian Institutes of Health Research (CIHR), and the Patented Medicines Prices Review Board (PMPRB).^{Footnote 1Footnote B}

Not only has OIA undergone organizational changes, but the broader global health context has also evolved over the last few years. There have been recent infectious disease outbreaks (e.g., Ebola in 2014 and Zika in 2016) that have highlighted the need for improved global capacity to prevent, detect, and respond to health threats. In addition, health issues have become increasingly complex, requiring greater collaboration, as many of them cut across several Portfolio program areas and branches, as well as other government departments (e.g., antimicrobial resistance). Furthermore, the number of global health bodies has increased, there are more international foundations, think tanks, and universities with larger global health centres, and health is discussed more frequently in non-traditional arenas, such as the World Economic Forum, the World Bank, and the G7/G20. Finally, a new federal government was elected in 2015 and it signalled a renewed interest in collaborating internationally and in supporting multilateralism. All of these have led to an increase in OIA’s activities.

2.2 Program Profile

The OIA aims to strategically advance the Health Portfolio's international priorities in an effort to support Canada's domestic health policy and foreign policy objectives, and to ensure that Canada's interests and values are reflected in the global health agenda. To that end, OIA conducts a range of activities that can be categorized into three broad groups: global (international) health policy analysis, advice, and support; international relations and engagement; and priority setting and international performance reporting (strategic planning). These activities are carried out by the following three OIA divisions: BESTD, the Multilateral Relations Division, and the Official Delegations and Strategic Integration Division (see appendix 1 for more information on each of the three divisions and their activities).

BESTD was created through the merger of the Bilateral Engagement Division and the Horizontal Policy and Integration Division, and the addition of the health and trade file in December 2015. This division leads and manages the Portfolio's engagement with key bilateral and regional partners, and supports the development, coordination, and monitoring of the Portfolio’s positions on international trade negotiations. It serves as the primary liaison with the G7/G20, the Organisation for Economic Co-operation and Development (OECD) and the Asia-Pacific Economic Cooperation (APEC).

The Multilateral Relations Division (MRD) provides expertise in multilateral negotiations and multilateral global health governance. MRD manages the Health Portfolio’s engagement with the following multilateral organizations and initiatives: the World Health Organization (WHO), the Pan American Health Organization (PAHO), the United Nations (UN), the International Association of National Public Health Institutes (IANPHI), the Commonwealth, the WHO’s International Agency for Research on Cancer (IARC) and the Framework Convention on Tobacco Control (FCTC), the Global Health Security Initiative (GHSI) secretariat, the Global Health Security Agenda (GHSA),^{Footnote C} as well as a variety of health and human rights initiatives.

The Official Delegations and Strategic Integration Division (ODSID) leads internal stakeholder engagement and communications activities for the Health Portfolio and the Government of Canada, and serves as the Portfolio’s focal point for incoming and outgoing delegations, including logistical support for the Minister and senior officials’ participation in events. It also supports strategic discussions on international priority setting and alignment with domestic priorities, including planning and reporting on international engagement activities, as well as supporting the Portfolio’s international affairs governance committees. The ODSID also supports delivery of the International Health Grants Program (IHGP) to advance strategic partnerships.

2.3 Program Resources

As shown in Table 1 below, expenditures for OIA activities totalled $96.7M over the years 2012-13 to 2017-18.

Table 1: Program Expenditures ($M)^{Footnote a}

Year	G&Cs	O&M	SalaryFootnote b	TotalFootnote c
2012-13	$13.3	$0.7	$3.3	$17.4
2013-14	$15.6	$0.9	$4.1	$20.6
2014-15	$16.5	$0.6	$3.9	$21.0
2015-16	$17.4Footnote D	$0.6	$4.1	$22.1
2016-17	$2.1	$0.6	$5.0	$7.7
2017-18	$1.5	$0.7	$5.7	$8.0
TotalFootnote c	$66.5	$4.1	$26.2	$96.8

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The evaluation focused on OIA activities conducted from April 2012 to September 2017. As this was the first evaluation of all OIA activities, it was determined that this evaluation would examine the delivery of OIA as a shared service (also referred to as a process evaluation). In addition, as part of establishing the scope of the evaluation, it was deemed important to examine the impact of OIA’s activities to the fullest extent possible.

Although the evaluation examined the range of international activities conducted by PHAC and Health Canada branches, an assessment of branch-led international activities was outside the scope of this evaluation.

The evaluation is consistent with the Treasury Board’s Policy on Results (2016) and examined questions pertaining to the design and delivery of OIA activities. Data for the evaluation were collected using multiple lines of evidence, including a review of the literature on select issues related to global health and shared services models, program documents, files, and financial data; a Health Portfolio client survey (n=33); key informant interviews (n=43); and a comparative analysis (NRCan Audit of the Management of International Activities). See appendix 2 for more details on the evaluation approach and methodology.

Data were analyzed by triangulating information gathered from the different lines of evidence listed above, with the intention of increasing the reliability and credibility of evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

The following table outlines the limitations encountered during the implementation of the data collection methods selected for this evaluation. Also noted are the mitigation strategies put in place to ensure that evaluation findings could be used with confidence in guiding program planning and decision making.

Table 2: Limitations and Mitigation Strategies

Limitation	Impact	Mitigation Strategy
Retrospective nature of key informant interviews	As interviews were retrospective in nature, this led to the provision of recent perspectives on past events. This can affect the validity of assessing activities or results relating to improvements in the program area.	Triangulation of other lines of evidence to substantiate or provide further information on data received in interviews.
Health Portfolio client survey – lower response rate and limited number of open-ended questions	Lack of reliable evidence or information on survey questions (e.g., inability to determine reasons for dissatisfaction).	Triangulation of patterns of results with key informant interviews and document review, and inclusion of open-ended questions asking for suggested improvements in survey questionnaire.
Lack of performance data	No overall program logic model articulating short-, medium-, and long-term outcomes. Performance data was limited to outputs, which made it difficult to assess the program’s subsequent impacts.	Other lines of evidence, such as file and document review and key informant interviews, were used to help provide as clear of a picture as possible as to impact of activities.

4.0 Findings

4.1 Design and delivery

4.1.1 Mandate

The mandate of the Office of International Affairs is quite broad; however, it reflects the complexity of its work and the expanding global health context within which its work is conducted. Many key informants and survey respondents agreed that OIA's mandate was appropriate and clear.

The mandate of the Office of International Affairs for the Health Portfolio is “to strategically advance the Health Portfolio’s international priorities in support of Canada’s domestic health policy and foreign policy objectives, and to ensure that Canada’s interest and values are reflected in the global health agenda”. This mandate, while broad, reflects the complexity of the work that OIA undertakes as a shared service in connecting the domestic and branch-specific international work of Health Canada and the Public Health Agency of Canada with the broader global health landscape, in order to assist the Minister in representing Canada’s best interests internationally.

Many key informants, including internal program staff^{Footnote E}, Portfolio partners, and members of senior management, agreed that the OIA mandate was appropriate. The majority of survey respondents also agreed that the mandate was appropriate (64%) and clear (55%). Of the remainder of survey respondents, 12% did not know if the mandate was appropriate, 12% said it was neither appropriate nor inappropriate, and 12% said it was inappropriate. Of those few survey respondents who reported that the mandate was inappropriate, reasons given for this included that the mandate was too broad or they provided comments about the need for sufficient collaboration with the program areas.

Corporate documents supported the importance of engaging internationally in order for the Health Portfolio to conduct its work adequately. Health Canada’s 2017-18 Departmental Plan explained that many of the challenges it faces as it works to deliver results to Canadians are beyond the sole control of Health Canada, thereby requiring the department to work with a wide range of partners, including international governments, regulatory partners, and stakeholders. Similarly, the Global Health Framework for the Public Health Agency of Canada’s International Activities (2012-17) (developed before OIA was created) highlighted that, in an increasingly interconnected world, the health of Canadians is closely linked to the health of people around the globe, creating a need for PHAC to work internationally to fulfill its mission to promote and protect the health of Canadians and, at the same time, contribute to better global health outcomes.

In addition, the global health landscape has grown and changed over the last few years. In 2011, a report from an expert panel on Canada’s Strategic Role in Global Health indicated that complex global health issues were likely to continue to increase in scope and complexity.^{Footnote 2} In May 2017, the first Meeting of Health Ministers of the Group of 20 (G20) was held in Berlin. This marked the first time that global health was addressed broadly at a Ministerial-level meeting within the G20. Another Health Ministers meeting of the G20 will be held in Argentina in October 2018,^{Footnote 3} signalling the prominence and importance of engaging internationally on issues related to health.

Internal key informants described how major global health issues, such as Ebola, antimicrobial resistance (AMR), and Zika, have resulted in an increased workload for OIA. One internal key informant indicated that OIA has evolved and grown to be able to understand the horizon and the landscape of what is happening around the world, particularly because the number of global health players has dramatically increased in recent years. Where there used to be mainly multilateral organizations (with member states), there are now more and more private sector organizations, such as the Bill and Melinda Gates Foundation, as well as think tanks and universities that have very robust global health centres. It was also noted that economic bodies such as the World Bank now include robust and complex global health items on their agendas. The increase in issues and players has required greater integration in the work that OIA is doing (i.e., they have expanded their number of partners, stakeholders, and engagements to address issues such as trade and the social determinants of health).

A shift occurred with the election of a new federal government in October 2015. Many internal key informants pointed to the drastic shift they felt in their work following a speech from the Prime Minister stating, “Canada is back” when referring to Canada’s voice in the international arena.^{Footnote 4} Internal key informants described how their “phones were ringing off the hook” and how countries from all over the world began reaching out to them with requests to collaborate. This shift has also been seen in the interest of recent federal Ministers of Health to engage internationally, and to develop a set of global health priorities.

4.1.2 Activities

The Office of International Affairs’ activities are in line with its mandate. However, with its wide breadth of activities and finite resources, OIA must prioritize certain activities; therefore, on occasion, other activities such as trend analyses and training on trade occurred less frequently than originally planned.

To fulfill its mandate, OIA conducts a wide range of activities including, but not limited to:

• analyzing, supporting, and negotiating global health policy, including coordinating Health Portfolio input into trade negotiations;
• acting as international secretariat for the Global Health Security Initiative (until July 2018);
• contributing to government to government bilateral (and regional) relations, and Portfolio engagement with the G7 and G20;
• leading on governing body multilateral relations (e.g., UN, WHO, PAHO);
• providing logistical advice and support on protocol, international events and official outgoing delegations led by Portfolio senior management;
• managing and administering the International Health Grants Program, including budget transfers from the Health Portfolio; and
• acting as Portfolio secretariat for international working groups (see appendix 1 for more information regarding activities).

An audit conducted by Natural Resources Canada (2018) reviewed the international activities of other Government of Canada departments, including OIA’s. It was found that, while there were similarities in the types of international activities conducted by various departments, such as bilateral and multilateral engagements (including ministerial missions), differences included some departments having a larger trade component, some having greater enforcement and operational mandates, and some having greater or lesser international transfer payment mechanisms.

The expanding global health landscape and Canada’s increasing willingness to engage internationally have been reflected in an increase in OIA’s activities over the last several years. For example, internal documents showed an increase in the Minister of Health’s attendance at international events from 10 in 2015 to 33 in 2016. This trend continued into 2017, where the Minister attended 33 events from January to the beginning of November. Similarly, in 2016 OIA completed 316 senior-level requests, which is up from 280 in 2015 and 178 in 2014. Several internal key informants pointed to an increase in activity over the last few years. Internal key informants also indicated that they are required to manage additional global health files that do not easily fit within the mandates of other branches or federal departments (e.g., biological diversity, refugee health, immigration health, and the Sustainable Development Goals). OIA will also lead external liaison on the UN Sustainable Development Goals with other government departments and provide the Government of Canada an interface with the UN, while supporting the relevant areas of the Health Portfolio (namely the Office of Strategic Policy and Planning and the Office of Audit and Evaluation) in domestic implementation and reporting.

OIA produced numerous tracking documents to strategically organize, plan, support, and report on international events and official delegations, as well as to keep track of engagements. It had tools in place to support senior management’s participation in bilateral and multilateral meetings. These tools have been well received, as they support logistics, provide background information on issues of interest, and include meeting objectives and potential talking points for Canadian delegation members, as well as debriefs to program areas on the outcomes of international meetings, lessons learned, and meeting reports.

As previously mentioned, OIA became a stand-alone office in 2016, which led to the need for it to take on the additional corporate responsibilities of a branch, including producing dockets, conducting quality assurance, responding to access to information and privacy requests, providing cabinet document attestations, as well as having an increased representation in PHAC governance committees.

Keeping in mind the breadth of OIA’s mandate, the activities it conducted were aligned with this mandate; however, with such a wide range of activities and finite resources, prioritization becomes essential. It was clear that OIA is working toward prioritizing international activities overall, with documents such as the International Strategic Framework for the Health Portfolio, as well as the development of ministerial priorities to help determine the domestic priorities to champion internationally.

While work has been prioritized in certain ways, it was not clear how the International Strategic Framework and ministerial priorities were being used to triage the work of OIA. A presentation on the 2017-20 PHAC Global Health Strategy contained a table that demonstrated how priorities may be set by checking off the priority areas where activities fell, however it was unclear if and when, or how often, this strategy was used.

Prioritization in OIA’s bilateral engagements was evident through its more in-depth engagement with certain countries and regions, particularly the United States, the Caribbean, Mexico, Brazil and China. OIA also developed engagement strategies for Canada to guide engagement with multilateral partners, such as the WHO and PAHO during Canada’s terms on their Executive Board and Executive Committee, respectively. A few internal key informants spoke to activities, particularly training related to trade and conducting trend analyses, which could be useful, but were not undertaken due to a lack of resources or other higher-profile priorities.

4.1.3 Governance

Governance mechanisms are in place; however, the Partnership Executive Committee International Affairs Working Committee (PEC-IAWC) could be functioning more effectively.

As a shared service, the Office of International Affairs for the Health Portfolio has a number of governance committees it uses to engage with program areas, share information, and seek approval of items. There are two formal governance bodies for which OIA is the secretariat, namely the PEC-IAWC and the Portfolio International Engagement Working Group (PIE-WG), while several focussed and time-limited working groups have been created when needed to address specific files and issues. These governance bodies are used to promote and address issues of international concern across the Health Portfolio, such as strategic input for major international events, the overall policy direction of OIA, and new and emerging global health issues of concern to Canada.

The Partnership Executive Committee International Affairs Working Committee

The PEC-IAWC was created under the Shared Services Partnership model when OIA was created in 2012. Its Terms of Reference were most recently updated in 2016. According to the Terms of Reference, the mandate of the PEC-IAWC is to serve as a forum for identifying and addressing operational, program, and policy issues related to the international affairs of the Health Portfolio. The most recent Terms of Reference indicate that it meets up to four times per year and is chaired by the Director General of OIA, with membership made up of Assistant Deputy Ministers (ADM) or branch heads from relevant program branches of both Health Canada and the Public Health Agency of Canada, as well as representatives from the Canadian Food Inspection Agency, the Canadian Institutes of Health Research and the Patented Medicine Prices Review Board. Members are encouraged to present branch-specific issues of relevance to the Health Portfolio at meetings and to provide debriefs to their respective Executive Committees, as required.

Similar to previous studies of the Partnership Executive Committee governance, including the 2015 Office of the Auditor General Horizontal Internal Audit of Shared Accountability in Interdepartmental Service Agreements, this evaluation found that, while governance mechanisms are in place, the PEC-IAWC could be functioning more effectively. Key informants noted that key issues with this committee were related to poor attendance at the ADM/Vice-President level and a lack of strategic discussion. A review of records of decisions also confirmed poor attendance and a lack of discussion.

To accomplish the stated mandate of the PEC-IAWC, effective, two-way information sharing and strategic discussion is required, but many key informants, internal program staff, and Portfolio partners, as well as senior management, agreed that the committee is currently an information-sharing body with information typically being shared by OIA with members, and very little information sharing from other groups. Furthermore, some of these key informants thought PEC-IAWC should be a forum for strategic discussion.

A review of 10 records of decisions from 2014-17 was consistent with key informant perceptions. It was found that the majority of agenda items (58%) were for information, with discussion^{Footnote F} occurring 12 times over 36 agenda items. Of the 36 items that were presented, 23 (64%) were from OIA, one was from a committee member outside of OIA (CFIA), one was provided jointly between OIA and an ADM-level committee member, and three were provided by guests. It was unclear who presented the other eight agenda items.

A couple of internal key informants thought that, because PEC-IAWC was seen as a forum for OIA to share information with members, then perhaps members were delegating their attendance at the meetings. Other internal key informants hypothesized that the lack of attendance was the reason that strategic discussions were not taking place.

Regardless of why members were not attending, it was noted by several internal key informants that lack of attendance by members, specifically at the ADM-VP level, was an issue. The review of records of decisions supported the claim that some ADMs were frequently absent from meetings, often, but not always, sending a delegate. From a review of 10 records of decision from 2014-17, it was found that there were 27 members invited to attend; however, on average 12 members would send regrets, eight of whom were ADM-level members. It could be that ADM attendance was lower, due in part to the fact that the PEC-IAWC was not seen as a decision-making body. Decisions were made at other committees, such as MinDM, PHAC or HC Executive Committee meetings, and the topics discussed at PEC-IAWC were not seen as relevant for all committee members’ day-to-day activities.

On the other hand, several key informants, including internal, Portfolio partners, and senior management, pointed out that PEC-IAWC was a good mechanism for gathering and communicating with representatives from across Health Canada and PHAC when needed. They also noted that, through the efforts of OIA, PEC-IAWC has continued meeting at regular intervals, unlike other Partnership Executive Committee bodies.

The Office of International Affairs has continued efforts to improve the efficacy of the PEC-IAWC. In November 2017, meeting frequency increased from quarterly to monthly, and presentations on global health issues by OIA at the Portfolio Executive Committee and Tier 2-level meetings also increased. Some internal key informants and a Portfolio partner were hopeful that having more frequent meetings would provide the space and time for two-way information sharing and strategic discussion. While outside the period of this evaluation, it is worth noting that in February 2018, a special meeting on the Health Portfolio’s Priorities for International Engagement for 2018 was held to discuss the Minister’s international priorities. A few internal key informants remarked that they saw increased attendance and engagement from members at that particular meeting. OIA has also been encouraging other areas of the Health Portfolio to lead presentations. These approaches are currently being assessed to determine if they have increased the efficacy of meetings.

The Portfolio International Engagement Working Group (PIE-WG)

The PIE-WG is an information-sharing body consisting of membership at the working level (i.e., senior advisor/analyst, manager, and director levels). Records of decisions indicate that OIA presents most items, but that other program areas participate in discussions. Among those key informants that commented on the PIE-WG, they tended to be generally satisfied.

The PIE-WG has a mandate to serve as a forum for identifying and addressing horizontal issues related to international affairs of the Health Portfolio, including supporting the work of the PEC-IAWC and operationalizing the Health Portfolio’s International Strategic Framework. A review of four records of decisions from 2015 highlighted discussion among members on most topics on the agenda, including updates from PEC-IAWC meetings, the International Strategic Framework, and a presentation from Health Canada's Safe Environment Directorate.

Similar to the review of records of decisions, internal and Portfolio partner key informants indicated that the PIE-WG functions well as a two-way information-sharing body, with discussions taking place during meetings. It was clear to key informants that the purpose of the meetings was to prepare for PEC-IAWC meetings.

Other Governance Issues

Some internal key informants, as well as one senior management representative and one Portfolio partner, raised the issue that OIA was not always fully aware of the direction of key files at Health Canada, because the Director General of OIA is not a member of their Executive Committee, as she is with PHAC. These key informants indicated that not being at the Executive Committee table created an information gap, causing OIA to be more reactive than proactive on Health Canada files. It was also noted that while Health Canada has a great deal of international capacity within its branches, OIA can provide the overarching perspective that allows connections to be made with work being done across the Health Portfolio. However, a few other Health Canada key informants did not see the value in the Director General (DG) being a member of this Executive Committee. Another suggested that OIA’s DG could attend the Health Canada Executive Committee meetings two to three times a year to discuss upcoming international events.

While the Director General of OIA is a member of the DG Policy table at Health Canada, she is not a member of the Health Canada Executive Committee, similar to most other heads of shared services (i.e., Corporate Services, Communications and Public Affairs, and Audit and Evaluation).

The Natural Resources Canada audit found that governance structures varied for the different science-based departments it examined. Half of the departments consulted had an executive committee for strategic direction on international activities, which would be similar to OIA’s PEC-IAWC. Some departments described having director-level committees, such as the PIE-WG, while others had science-based management committee structures or no cross-organizational governance.

4.1.4 Products and services

There were mixed reviews of Office of International Affairs’ products and services. Key informants were generally positive, however survey respondents were less so.

As previously mentioned, OIA provides a number of products and services to the branches of Health Canada and the Public Health Agency of Canada, as well as to the Minister of Health and deputy heads. Services include, but are not limited to:

• global health policy analysis, advice, support and negotiations;
• coordinating Health Portfolio input into trade negotiations;
• priority setting and strategic planning;
• stakeholder engagement and support;
• trends analysis and environmental scanning;
• secretariat support for Portfolio committees, such as the PEC-IAWC and the PIE-WG, and secretariat support for international committees, such as the Global Health Security Initiative (GHSI) (which has now transferred to the Health Security Infrastructure Branch);
• managing the International Health Grants Program;
• supporting health emergency operations;
• information sharing, monitoring, and reporting on Health Portfolio international activities; and
• providing logistical advice and support on protocol, international events, and official outgoing delegations.    

As part of these services, a number of products are developed and shared, including policy papers, research papers, Canadian positions on various topics, briefing materials with recommendations and options, and presentations. The Office of International Affairs also produces a weekly newsletter and maintains a GCPedia page and an Intranet Portal on MySource.

Overall, a majority of senior management, Portfolio partners, other government departments, and international organization key informants provided positive comments on OIA’s products and services. Key informants were largely pleased with the timeliness for products, recognizing that the environment in which OIA operates is ever-changing, presenting an extra challenge to being timely. Generally, key informants reported that OIA is proactive, provides well-thought-out advice, is forward thinking, produces useful materials tailored appropriately to the audience, is very responsive, and that they generally feel well served by OIA.

Some key informants (several Portfolio partners, one from senior management and a representative from another government department) reported areas for improvement. Suggested areas for improvement included OIA increasing its capacity to deal with trade issues, making briefing products less dense, and sharing final products with those who helped contribute to their development.

Survey respondents were more mixed in their views on a variety of products and services provided by OIA. In certain areas, proportions of positive responses were noticeably higher than negative responses; however, it is worth noting that the proportions that were positive were still low:

• “I am satisfied with the information provided in OIA’s Weekly Newsletter” – 50% agreed, 10% disagreed.
• “I am satisfied with the level of support and advice OIA provides for outgoing international visits for high-level senior management engagement” – 40% agreed, 23% disagreed.
• “I am satisfied with the level of support and advice OIA provides for incoming delegations of senior level officials from other countries” – 37% agreed, 17% disagreed.

Significant proportions of respondents both agreed and disagreed:

• “I am satisfied with the quality of global health policy analysis and advice OIA provides” - 43% agreed and 40% disagreed.
• “I am satisfied with OIA’s approach to info sharing and intelligence gathering from international engagements” - 40% agreed and 37% disagreed.

A larger proportion of respondents neither agreed nor disagreed:

• “OIA’s Weekly Newsletter is useful in my work” – 50% neither agreed nor disagreed, 20% agreed, 25% disagreed.
• “I am satisfied with the quality of briefing materials and products produced by OIA” – 37% neither agreed nor disagreed, 37% agreed, 17% disagreed.

With respect to the levels of dissatisfaction, due to the design of the survey (i.e., primarily closed-ended questions with only a few open-ended questions), the reasons for this dissatisfaction are not known.

Some key informants and survey respondents suggested some areas for improvement. Specifically, it was indicated that it would be helpful for OIA to prioritize and better coordinate the number of requests going to branches. A few partner/client key informants suggested that a focal point or single window within OIA could be useful for tasking branches with requests, in order to reduce confusion and increase efficiency. At the same time, many internal key informants and some Portfolio partners highlighted the importance of developing relationships through multiple contacts to make sharing information and completing tasks easier.

Additionally, a few survey respondents and a key informant indicated that OIA could increase its capacity with respect to trade files by identifying strategic directions and developing departmental positions in this regard. In addition, comments on increasing trade capacity were echoed by a small number of internal and Portfolio partner key informants.

4.1.5 Shared Services Model

Overall, there is agreement that the shared services model has been effective for international activities. This model has many perceived advantages and OIA is seen as adding value to the coordination of Health Canada and PHAC international activities.

A literature review revealed that the concept of shared services is most commonly used by large organizations that wish to combine common functions (e.g., administrative operations) to reduce business process duplication and deliver specialized value-added services across the entire organization.^{Footnote 5} In general, it is a collaborative strategy meant to increase efficiency, improve service quality, and reduce costs.^{Footnote 6}

There were numerous benefits and advantages to a shared service model identified in the literature review. The most common of these were improved cost-effectiveness and improved service quality and delivery.^{Footnote 7} Improved cost-effectiveness is generally derived from consolidating administrative and business services among several units, thus reducing duplication.^{Footnote 8} Additional benefits cited that are relevant to this evaluation include effective use of and access to resources,^{Footnote 9} efficiency gains from increased flexibility and staffing resources,^{Footnote 10} a single point of contact for customers (e.g., the Minister and international stakeholders),^{Footnote 11} and consolidation of core business processes.^{Footnote 12}

There was general agreement across the majority of key informants (internal and partners/clients) that the shared services model is effective for international activities. Similar to the literature review that noted that having a single point of contact and reducing duplication as advantages of this model, key informants’ most commonly cited advantages included providing one point of contact for the Minister and senior management, ensuring linkages across Health Canada and PHAC, and reducing duplication. Before the transition in 2012, there appeared to be a lot of confusion as to which department would lead or who would speak first internationally. Since the merger, the process has been streamlined, reducing silos and the need for two delegations at each international meeting, and also allowing for the integration of other Portfolio agencies and organizations, such as CFIA and CIHR, and other departments where appropriate. Not only has there been a reduction in duplication, but now OIA is able to take a holistic perspective, providing input from across the Health Portfolio, which helps the Minister and senior management speak with one voice.

Few key informants pointed out challenges with the shared services model. Of those that did, the most frequently cited challenges were the fact that OIA has had to coordinate across many branches, gathering input from many different areas of the Portfolio (all of whom have varying needs and capacities), a lack of regular interaction with OIA, and the location of OIA, which has given it more of a presence at PHAC. However, it was not felt that these challenges significantly affected OIA’s ability to carry out its activities. The evaluation found evidence that OIA attempted to address these challenges. Specifically, with respect to the location issues and loss of branding, OIA made sure to identify itself as serving the Health Portfolio in its signature block, and by identifying themselves as representatives of the Portfolio when attending meetings.

The advantages of the model were thought to outweigh the challenges. Several internal and partner/client key informants noted that OIA brings added value to the coordination of international activities by providing the bigger global picture and presenting a coordinated approach for the Minister, senior management and external international organizations. This coordinated approach is important as several issues cross the Portfolio (e.g., mental health, AMR).

Furthermore, according to internal documents and a few internal and partner/client key informants, since becoming a shared service, OIA has worked to brand itself as a centre for expertise in multilateral and bilateral relations, which includes being a conduit into Global Affairs Canada (GAC), making connections between the Portfolio’s domestic work and initiatives underway at GAC.

Similar to key informants noting that the shared service model was effective for international activities, a majority of survey respondents (64%) agreed that OIA’s role as a shared service for the Health Portfolio was clear, whereas 18% neither agreed nor disagreed and 15% disagreed. Of the few who disagreed, the most common reasons cited were that OIA should increase its trade capacity, programs could play a greater role, and that there should be increased collaboration with programs.

According to an internal key informant, there are other government departments, such as CFIA, Agriculture and Agri-Food Canada and Indigenous Services Canada/Crown-Indigenous Relations and Northern Affairs, that are in the process of implementing a shared services model and have had discussions with OIA to learn from its experiences.

4.1.6 Relationship with branches and external stakeholders

Generally, OIA’s relationship with the branches is seen as working well; however, some issues remain, such as the lack of clarity regarding roles and responsibilities.

Several partner/client and external key informants indicated having good working relationships with OIA, describing these as collegial, collaborative, supportive, responsive, and helpful. Likewise, internal key informants noted that OIA has worked to establish good relationships with programs, making sure to engage with them through teleconferences and the sharing of trip reports, and the weekly newsletter. OIA has also worked to create relationships with those in the branches who have a vested interest in particular files by establishing working groups on trade, China, APEC, Brazil, and the United States. For example, the United States working group recently held a meeting attended by approximately 60 Health Canada and PHAC employees. Most internal key informants noted that OIA has good working relationships with the branches and, while OIA cannot always be aware of the ongoing, day-to-day international activities within the branches and other Portfolio members, OIA has worked to increase communication channels and information sharing to improve those relationships.

Many internal and partner/client key informants noted that the process of having a single window in each of the branches has been working well. However, OIA has sent numerous requests, and from the branches’ perspective, these were coming from multiple people within OIA, leading to an increased burden on the programs. Also, branches did not always know who they could contact in OIA (i.e., there was not always a single window into OIA). For more formal dockets, OIA’s DGO functioned as a single window, but for more technical issues such as AMR, a few partner/client key informants noted that having a single window into OIA would have been helpful, especially in terms of having a single point of contact. Having one person within OIA assigning requests could have helped prioritize them and avoid duplication.  

Most internal and partner/client key informants noted that OIA has established good working relationships with branches, working collaboratively across Health Canada and PHAC. OIA provides expertise with regards to understanding how multilateral engagements work, as well as different international organizations and players. However, some partner/client key informants felt that OIA, on occasion, may have overstepped its mandate and touched on the program’s area of expertise; although it should be noted that key informants felt that OIA had overstepped less frequently recently than it did when OIA was first created as a shared service. In addition, it is worth noting that OIA sees itself as playing a horizontal policy role, but this is not the case for all Portfolio partners and senior management, as some of them see OIA’s role as primarily one of organizing logistics and coordination.

Some internal and Portfolio partner key informants noted that OIA roles and responsibilities are not formally written in a document that could be shared across the Portfolio; however, it should be noted that information on some high-level roles and responsibilities are available on the Health Portfolio Intranet. That having been said, most internal and partner/client key informants felt that roles and responsibilities were clear when the Minister or the deputy heads attended an event. This was less clear when the Minister and deputy heads were not in attendance and when the issue was of a more technical nature. In addition, it was not always clear which files OIA should lead, which were the program’s to lead, and where there was overlap between the two. For example, a number of branches within PHAC worked in collaboration with PAHO, as did OIA, and sometimes it was not clear who was leading certain initiatives. Similarly, for the GHSI, OIA was the secretariat for this initiative but other areas of PHAC were involved in global health security, and the roles of OIA and the program were not always clear. Furthermore, a Portfolio partner key informant noted that mental health is currently one of the Minister’s international priorities; however, it was unclear how this was determined, given that their Branch had made other suggestions.

At the same time, more than half of survey respondents (55%) agreed that the difference between OIA’s role and the role of their branch or program area in international affairs was clear; however 27% of respondents disagreed with that statement. When given an opportunity to explain, survey respondents tended to make general statements that there was a lack of clarity regarding roles and responsibilities or a need for the programs (as subject matter experts) to play a greater role.

While key informants were generally positive in regards to working relationships with OIA, survey respondents were mixed in their views on their interactions with OIA.

In certain areas, proportions of positive responses were noticeably higher than negative responses:

• “OIA’s briefing material and products take into account input from my organization, branch and/or program area” – 53% agreed, 7% disagreed.
• “I am satisfied with OIA’s approach to consulting and engaging my program area in the development of global health policy analysis and advice” – 40% agreed, 27% disagreed.
• “OIA provides added value to the work of my organization, branch and/or program area” – 45% agreed, 27% disagreed.
• “My expectations are generally met when I work or engage with OIA” – 42% agreed, 27% disagreed.
• “I receive a timely response” – 77% agreed, 4% disagreed.
• “I am satisfied with the information provided” – 65% agreed, 4% disagreed.

Significant proportions of respondents both agreed and disagreed:

• “I am satisfied with how OIA works with my organization, branch and/or program area” – 42% agreed, 30% disagreed, 24% neither agreed nor disagreed.
• “OIA should play a larger role in supporting my organization in the coordination of international visits or delegations” – 43% neither agreed nor disagreed, 37% agreed, 13% disagreed. 

A larger proportion of respondents neither agreed nor disagreed:

• “Having contacted these sources [OIA direct contacts], I better understand OIA and the work that they do” – 50% neither agreed nor disagreed, 42% agreed, 8% disagreed.

There was a wide range of resources dedicated to branch-led international activities^{Footnote G} (ranging from 0 to 16 full-time equivalents and 0 to 18 part-time equivalents), with rough estimates for operations and maintenance (O&M) ranging from $88,000 to $1.5 million annually for program-specific international activities. Some of these activities included, but were not limited to:

• engagement;
• networking through committees, conferences, working groups and task forces;
• subject matter expertise;
• technical negotiations;
• international disease monitoring; and
• strategic communications.

One challenge of the relationship between OIA and the branches was that there was uneven capacity across branches, with some having well-established international groups, and this may have created tension when it came to roles and responsibilities in terms of the perception that OIA was stepping into the program’s area of expertise. At the same time, OIA filled policy gaps when it was unclear where the expertise lay, or cutting across files on which there was no clear lead with other government departments (e.g., the Sustainable Development Goals).

Several key informants reported that OIA’s timelines were challenging; however, for the most part, it was recognized that these timelines were often externally driven and by the nature of the work, meaning that they were unlikely to change. Similar to the key informants, 55% of survey respondents felt that there was not enough time given to provide input to OIA-generated action requests. OIA attempted to address this challenge by pre-populating templates with the most recently provided information to help programs respond to the request in a timelier manner. A few partner/client key informants noted that OIA could pre-populate more templates, and given that there are many high-level meetings in succession, it was suggested that OIA could first look at the last template filled out to determine if there was a need to send another request. A representative from another government department suggested that OIA could re-examine the level of approvals required for templates as a way to help with timelines.

It is worth noting that key informants from international organizations were complimentary in discussing their relationships with OIA, describing their engagement with OIA as a “dedicated, very reliable” group. Also, all international partners interviewed during the evaluation commented on the value that OIA provided as an entry point to gain a multisectoral view from within the Government of Canada, not just from PHAC and Health Canada.

4.2 Impact

OIA engages in numerous multilateral and bilateral relationships. It supports these international engagements in conjunction with other Portfolio partners. While it is difficult to determine the impact of all these joint activities, there is evidence of impact related to higher-profile activities.

OIA has actively engaged with a number of priority partners: at the bilateral level with the United States, China, and Mexico, and multilaterally, namely with the WHO, PAHO, the OECD, the G7/G8 and G20, among others. The evaluation did not examine the impact of all of OIA’s activities, it focused specifically on international engagements. Due to the nature of these activities, it was difficult to determine their impact. However, there is evidence of impact related to its higher-profile activities. This impact includes supporting advancement of Canada’s domestic priorities, leading the Portfolio’s engagement in the development of Canadian policy positions, sharing information, building a stronger scientific evidence base to inform national strategies/policies, strengthening engagement with the Caribbean Public Health Agency (CARPHA), and identifying ongoing priorities and strategic opportunities for Canadian leadership, which is achieved through activities such as:

• managing and administering the IHGP;
• involvement with key multilateral organizations: the United Nations, WHO and PAHO;
• acting as the international secretariat for the Global Health Security Initiative; and
• negotiations and implementation related to international trade.

International Health Grants Program (IHGP)

OIA manages and administers the IHGP, which is a mechanism available to the Health Portfolio to support the advancement of Canada’s domestic health priorities and the fulfillment of Canada’s international commitments by supporting national and international projects and initiatives. There are three streams of IHGP funding: grants from the base envelope, grants using funds transferred from other Health Canada or PHAC branches, and Canada’s assessed contribution to the FCTC. Funding recipients include the WHO, CARPHA, PAHO, the OECD, Canadian universities, and other organizations engaged in global health issues.

Prior to June 2016, OIA made large payments towards Canada’s contribution to IARC, PAHO, and FCTC (on average $15.7 million per year). In 2016, the payments for IARC and PAHO were transferred to GAC.

OIA has facilitated and coordinated Canada’s membership and participation in the WHO’s Framework Convention on Tobacco Control. It has also provided strategic advice to advance Canada’s engagement on international tobacco control through bilateral engagement with key countries and in other multilateral forums, such as the WHO and PAHO. OIA ensured that Canada’s FCTC commitments were aligned with its domestic Federal Tobacco Control Strategy. For example, OIA supported the advancement of the Government of Canada’s negotiations at the Fifth Session of the Conference of Parties to the FCTC in 2012. The negotiations led to the adoption of the Protocol to Eliminate Illicit Trade in Tobacco Products.

Since the transfer, IHGP funding has ranged from $35,000 to $341,000 for projects lasting from six months to three years. These projects have focussed on enhancing laboratory capacity for the Zika virus, building knowledge and action for health equity in the Americas, and creating the WHO’s Global Dementia Observatory, among others.

Involvement with key multilateral organizations, including the United Nations (UN) bodies, WHO and PAHO

OIA leads and manages the Health Portfolio’s engagement with key multilateral organizations, especially related to UN bodies such as PAHO, the WHO, and the UN General Assembly. OIA’s involvement primarily centres on providing global health policy analysis and advice, coordinating program and policy input for governing body meetings of multilateral organizations, and on high-level international engagements. It coordinates the development of Canada’s positions in negotiations of international binding and non-binding instruments in multilateral forums. As detailed in appendix 3, there are a variety of ministerial and senior management-level engagements involving the G7/G20, GHSA/GHSI, OECD, PAHO, and the World Health Assembly on topics including AMR, dementia, healthy aging, health security, mental health, and opioids, among others.

In consultation with focal points from across the Health Portfolio and Government of Canada, OIA developed a strategy to secure a seat on the WHO Executive Board for a period of three years (May 2015 – May 2018), which included identifying key areas of focus for Canada during its term and strategic opportunities for Canadian leadership. For example, OIA led a resolution on the role of the health sector in the sound management of chemicals, which was endorsed by the World Health Assembly in May 2016.

OIA has also had a long-standing active role within PAHO, completing a three-year term as a member of the PAHO Executive Committee (2012-15). In September 2017, Canada was elected to PAHO’s Executive Council for a three-year term (2017-20), with OIA’s Director General acting as Canada’s representative. According to internal documents, OIA ensures alignment between the international and domestic fronts by introducing language that is amenable to the Canadian context (consideration of federal, provincial, and territorial levels of responsibility on some of the issues).

OIA engages in preparations for the annual UN General Assembly High Level Week by providing input into the Canadian statement and/or informing the development of Canadian-hosted high-profile events at the UN General Assembly, participating at side events, and supporting Ministerial participation, including developing the Minister’s remarks and briefing book. In particular, OIA provides Health Portfolio coordination and policy support and liaises with Global Affairs Canada on behalf of the Portfolio, as well as coordinates policy input and leads the development of negotiation language to inform health-related resolutions that are tabled at the UN General Assembly.

In consultation with program areas across the Health Portfolio and within PHAC and Health Canada, OIA has supported the identification of key domestic priorities for the Minister to champion internationally. OIA has coordinated and supported numerous incoming and outgoing international visits and, as previously mentioned, OIA produced numerous tracking documents and had tools in place to support engagements.

Global Health Security Initiative^{Footnote H} (GHSI) International Secretariat

Between 2012 and July 2018, OIA served as International Secretariat for the GHSI. In that capacity, OIA delivered, organized and managed all ministerial and senior officials’ meetings by assisting senior officials in setting strategic directions and objectives for the GHSI, ensuring that work plans were aligned with ministerial commitments and senior officials’ direction, managing records for the network, and developing and operating a secure website to facilitate information exchange and collaboration between officials and experts from GHSI member countries.

Some of the benefits that OIA has facilitated through the GHSI, with other Portfolio partners, include sharing of Zika virus samples to inform risk assessments and diagnostics when other channels proved slow, sharing real-time information during H1N1 on vaccines to inform a Canadian approach, building a stronger scientific evidence base to inform the development of national strategies and policies (for example, informing the draft version of Canada’s current vaccine strategy).

Negotiations and implementation related to international trade

According to internal documents, OIA led Health Portfolio efforts to support the Canada-European Union Comprehensive Economic and Trade Agreement, the Trans-Pacific Partnership Agreement, the North American Free Trade Agreement, and other trade negotiations, working closely with other Portfolio officials, as well as Industry Canada and Global Affairs Canada (the former Department of Foreign Affairs and International Trade). OIA facilitated and contributed to the development of Canada’s positions on health-related issues.

OIA led the Health Portfolio’s engagement in GAC-led negotiations, leading up to Canada signing onto the UN Sustainable Development Goals (Agenda 2030) in the fall of 2015. OIA’s role later shifted to leading intense consultations across the Portfolio on draft health indicators to support the targets and advising the Portfolio on all questions related to this policy agenda.

Similarly, pluralities of survey respondents were satisfied with OIA’s overall approach to advancing global and domestic health priorities through its international engagements, with satisfaction somewhat higher for advancing global health priorities than domestic health priorities:

• “I am satisfied with OIA’s overall approach to advancing Canada’s global health priorities in multilateral engagements” – 50% agreed, 30% neither agreed nor disagreed, 13% disagreed.
• “I am satisfied with OIA’s overall approach to advancing Canada’s domestic health priorities in multilateral engagements” – 40% agreed, 30% neither agreed nor disagreed, 20% disagreed.
• “I am satisfied with OIA’s overall approach to advancing Canada’s global health priorities in bilateral and regional engagements” – 40% agreed, 37% neither agreed nor disagreed, 17% disagreed.
• “I am satisfied with OIA’s overall approach to advancing Canada’s domestic health priorities in bilateral and regional engagements” – 37% agreed, 27% neither agreed nor disagreed, 23% disagreed.

4.3 Demonstration of Economy and Efficiency

OIA activities are currently allocated approximately $7.8 million per year. There is evidence that since 2014, OIA has been running a deficit ranging from $44K to $2.2M. OIA ran a deficit in G&Cs in FYs 2014-15, 2015-16 and 2016-17, which was attributable to foreign exchange impacts on the payment of Canada’s assessed contribution to IARC, PAHO, and WHO’s Framework Convention on Tobacco Control. This will not be a challenge moving forward, as the payments of Canada’s assessed contributions for IARC and PAHO were moved to GAC in June 2016. According to a few key informants (both internal and partners/clients), the interest in IHGP funding far exceeds what is available.

Table 3: Planned Spending and Expenditures^{Footnote a}
2012-13 and 2016-17 ($M)

Year	Planned Spending ($)				Expenditures ($)				Variance ($)	% planned budget spent
	G&Cs	O&M	Salary	TOTAL	G&Cs	O&M	Salary	TOTALFootnote b
2012-13	$15.1	$1.1	$3.8	$20.1	$13.3	$0.7	$3.3	$17.4	-$2.7	86%
2013-14	$15.6	$0.8	$4.2	$20.6	$15.6	$0.9	$4.1	$20.6	$0.003	100%
2014-15	$15.0	$0.4	$4.2	$19.6	$16.5	$0.6	$3.9	$21.0	$1.4	107%
2015-16	$15.3	$0.4	$4.2	$19.8	$17.4Footnote I	$0.6	$4.1	$22.1	$2.2	111%
2016-17	$1.6	$0.4	$4.2	$6.1	$2.1	$0.6	$5.0	$7.7	$1.6	126%
2017-18	$1.6	$0.6	$5.7	$7.9	$1.5	$0.7	$5.7	$8.0	$0.04	101%
Footnote a Financial data provided by the Office of the Chief Financial Officer. Return to footnote a referrer Footnote b Numbers may not add up to the total column due to rounding. Return to footnote b referrer

Over the last six years, OIA has generally overspent on O&M, except for FY 2012-13, where it underspent by $388,284, and OIA has generally underspent on salaries except for FY 2016-17 and 2017-18, where it overspent by $863,848 and $361,716 respectively. A summary of budgets and expenditures is presented in Table 3 above.

Internal documents showed that the largest portion of OIA O&M spending was on travel. For example, in 2012-13, OIA spent $193,178 on travel, whereas by 2014-15, it was spending $339,122 on travel, which increased to $389,575 and $471,550 through 2016-17 and 2017-18, respectively. This could be partly due to the fact that OIA is more active internationally, but it could also be a part of succession planning. A few internal key informants stated that “you cannot enter into a negotiation having never seen a negotiation”. However, a couple of partner/client key informants argued that personnel in the program areas that provide a great deal of support for these negotiations should be given the opportunity to attend some high-level meetings.

As OIA activities have increased, so too has the demand for resources. For example, previously there were no resources assigned to the G7 and G20 activities in the Shared Services Partnership Agreement; however, there is now a team of OIA staff dedicated to these activities. In addition to these new activities, developing internal capacity to support existing priority files, such as trade, has contributed to increased funding pressures. A number of internal and partner/client key informants felt that the organization was under-resourced, especially with an increased international profile for the government, but others felt that OIA had sufficient resources to fulfill its perceived mandate.

Also impacting OIA’s ability to meet this increased demand is its high staff turnover rate, which OIA has tried to address by relying on casuals, terms and students. This approach can be costly and is a short-term fix which lacks stability. Internal key informants attribute the turnover to a high workload, the fast pace of work, and the higher visibility for OIA staff, which has led to increased employment opportunities outside of OIA. According to internal documents, there were 65 staffing actions between March 2014 and August 2016. During this time period, OIA FTE’s increased from 30 to 48, with approximately 38 full-time equivalents per year.

OIA’s percentage of budget spent ranged from 87% in 2012-13 to a high of 124% in 2016-17. Even though OIA has been able to secure funds by putting forward special requests^{Footnote J}, there is a risk that this is not sustainable. If OIA is not able to secure additional funding, it may be unable to complete all of its activities given its current budget. However, it should be noted that OIA’s budget was permanently increased by $1.3 million in 2017-18. With this increase, OIA ran a slight deficit of only $44,254. This should be monitored going forward to ensure that OIA does not return to deficits of the previous three years.

5.0 Conclusions

OIA Design and Delivery

The mandate of the OIA is quite broad; however, it reflects the complexity of its work and the expanding global health context within which this work is conducted. Many internal and partner/client key informants and survey respondents agreed that OIA’s mandate is appropriate and clear. OIA’s activities are in line with its mandate. However, with the breadth of activities conducted and finite resources available, OIA must prioritize its activities; therefore, on occasion, other activities such as trend analyses and training for the Health Portfolio occurred less frequently than originally planned.

The evaluation found that, while governance mechanisms are in place, the Partnership Executive Committee International Affairs Working Committee (PEC-IAWC) could have been functioning more effectively. This Assistant Deputy Minister (ADM)-level committee, which was often delegated to other staff, appeared to be more of an information-sharing body than one where strategic discussions took place. Furthermore, during the time of this evaluation, OIA’s Director General was a member of PHAC’s Executive Committee, but did not sit on Health Canada’s Executive Committee. Some key informants (mostly from OIA and one from Health Canada) saw the value in having the Director General at this table to make stronger linkages across OIA and the programs’ international activities; however, a few other Health Canada key informants did not agree, as other forums existed for conversations on international work.

There were mixed reviews of OIA’s products and services. Key informants were generally positive; however, survey respondents were less positive, especially in regards to OIA’s Weekly Newsletter, its global health policy analysis and advice, and its approach to information sharing and intelligence gathering.

Overall, there was agreement among key informants that the shared services model has been effective for international activities. This model has many perceived advantages and it has demonstrated that OIA is seen as providing added value to the coordination of Health Canada and PHAC international activities. Key informants identified a few challenges, such as the need to coordinate across many branches; however, it was felt that these challenges did not interfere with OIA’s ability to carry out its activities.

Generally, OIA’s relationship with the branches was seen as working well. Most internal and partner/client key informants felt that the roles and responsibilities were clear when the Minister or the deputy heads attended an event. Roles and responsibilities were less clear when the issue was of a more technical nature. Some internal and Portfolio partner key informants noted that OIA roles and responsibilities were not formally documented in a way that could be shared across the Portfolio. Another challenge was that there was uneven capacity across branches, with some having well-established international groups and others not, which could havecreated tension when it came to roles and responsibilities, in terms of the perception of OIA stepping into the program’s area of expertise.

Impact

OIA engages in numerous multilateral and bilateral relationships, in conjunction with other Portfolio partners. While it is difficult to determine the impact of these joint activities, there is evidence of impact related to higher-profile activities.

This impact includes supporting advancement of Canada’s domestic priorities, leading the Portfolio’s engagement in the development of Canadian policy positions on international issues, and identifying ongoing priorities and strategic opportunities for Canadian leadership, which is achieved through activities such as:

• managing and administering the International Health Grants Program (IHGP);
• sitting as a member on the World Health Organization (WHO) Executive Board  (which ended in May 2018) and the Pan American Health Organization’s Executive Committee (2017-18);
• acting as the international secretariat for the Global Health Security Initiative (which was transferred to the Health Security Infrastructure Branch in July 2018);
• playing a role in helping to influence the negotiation and implementation of international trade agreements; and
• providing Health Portfolio coordination and policy support through its liaison with Global Affairs Canada (GAC).

While not addressing impact, it is worth noting that key informants from international organizations were complementary in discussing their relationships with OIA. In addition, pluralities of survey respondents were satisfied with OIA’s overall approach to advancing global and domestic health priorities through its international engagements.

Demonstration of Economy and Efficiency

There is evidence that, since 2014, OIA has been running an annual deficit. As OIA’s activities have increased, so too has its demand on resources. For example, there were no resources assigned to G7 and G20 activities as part of the Shared Services Partnership Agreement; however, there is now a team of OIA staff dedicated to these activities. In addition to these new activities, developing internal capacity to support existing priority files such as health and trade has contributed to increased funding pressures. A number of internal and partner/client key informants felt that the organization is under-resourced, especially with an increased profile internationally for the Government of Canada, but others felt that OIA had sufficient resources to fulfill its perceived mandate.

Even though OIA has been able to secure funds by putting forward special requests, there is a risk that this is not sustainable; however, it should be noted that for 2017-18, OIA received a significant and permanent increase to its budget (approximately $1.3M).

6.0 Recommendations

Recommendation 1

Clearly articulate roles and responsibilities regarding international files, and communicate those within the Portfolio

Overall, partner/client and external key informants indicated having good working relationships with OIA. While many key informants and survey respondents stated that roles and responsibilities are well established when the Minister or deputy heads attend an event, others noted a lack of clarity, especially as it relates to matters of a technical nature. Both OIA and the branches have international roles to play; however, there are some areas where it is less clear who is leading on a particular file, or who should be leading or supporting the international work on that file. Since OIA roles and responsibilities are not formally documented, and understanding that these technical files and events require additional conversations, it may be helpful if the roles and responsibilities were more clearly articulated to share within the Health Portfolio. Furthermore, existing governance tables could be used to initiate discussions regarding roles and responsibilities.

Recommendation 2

Further explore opportunities to improve products and services

The Office of International Affairs for the Health Portfolio provides a number of products and services to internal clients and key stakeholders. Generally, key informants reported that OIA is proactive, provides well-thought-out advice, is forward thinking, produces useful materials that are appropriately tailored to the audience, is very responsive, and they generally feel well served by OIA. However, there is dissatisfaction with some OIA products and services, especially among survey respondents (e.g., usefulness of OIA’s weekly Newsletter, OIA’s global health policy analysis and advice, its approach to information sharing and intelligence gathering from international engagements). It is unknown why this dissatisfaction exists; therefore, OIA must first determine the reasons behind it.

Recommendation 3

Explore ways to measure the impact of OIA activities

OIA engages in numerous multilateral and bilateral relationships and it supports these international engagements in conjunction with other Portfolio and Government of Canada partners. While it is difficult to determine the impact of all of these joint activities, there is evidence of impact related to higher-profile activities, such as managing and administering the IHGP, being involved with key multilateral organizations (e.g., the United Nations, WHO, and PAHO), acting as the international secretariat for the Global Health Security Initiative, and supporting negotiations and implementation related to international trade. Impacts from these activities include supporting the advancement of Canada’s domestic priorities, leading the Portfolio’s engagement in the development of Canadian policy positions, identifying ongoing priorities and strategic opportunities for Canadian leadership, and advancing policy positions that are consistent with existing Canadian efforts. In order to better demonstrate OIA’s value, it may be worth examining ways to measure the impact of all of its activities as much as possible.

Appendix 1 – Description of Activities

Bilateral Engagement, Summits and Trade Division (BESTD)

This division undertakes activities to lead country-to-country negotiations and coordinate Health Portfolio input in trade negotiations, analysis, and advice. These are made possible through the following services:

• Connections with foreign policy and trade agendas
  • Input into international negotiations, documents, and positions;
  • Input and review of policy and cabinet documents; and
  • Development of briefs and policy positions.
• Government-to-government bilateral (and regional) relations
  • Development, implementation, and tracking of progress on international bilateral Memoranda of Understanding;
  • Hosting and leading the development of ministerial or senior-level policy dialogues, and bilateral or trilateral meetings;
  • Development and contribution to international agreements and interdepartmental letters of agreement;
  • Development of briefs and policy research papers;
  • E-scanning of country health systems, priorities, and emerging issues; and
  • Primary liaison with G7/G20, Organisation for Economic Co-operation and Development (OECD), and APEC secretaries.

For example, OIA developed proposals to include health in Canada’s G7 presidency and led the Health Portfolio and the Government of Canada’s global health input into the preparations for the 2017 G7 Leaders’ Summit and the Meeting of the Health Ministers of the G20.^{Footnote 13}

Multilateral Relations Division (MRD)

Through the Multilateral Relations Division, OIA:

• Acted as the international secretariat for the Global Health Security Initiative (GHSI) by providing the following:
  • Secretariat services for an international body, including meeting logistics, advice on forward planning, and governance reviews;
  • Development of policy papers, briefs, and notes for Committee chairs and Canadian delegations;
  • Primary contact point and liaison function for international emergency management and response during Emergency Operations Centre activation; and
  • Management of a secure website on behalf of the GHSI secretariat.
• Acts as lead on governing body multilateral relations (e.g., United Nations, World Health Organization, Pan American Health Organization) through the following key activities:
  • Development of policy papers, options, recommendations, and advice;
  • Development of Canadian positions and awareness of Canada’s role globally;
  • Input into resolutions, international commitments, and leader’s declarations; and
  • Development of briefs and policy research papers.

Official Delegations and Strategic Integration Division (ODSID)

The Official Delegations and Strategic Integration Division supports protocol, international events, official visits and delegations, international performance reporting and strategic planning, as well as Health Portfolio international governance structures and grants through the following key activities:

• Supporting visits and outgoing delegations led by Portfolio senior management
  • Development of Minister’s and deputy ministers’ programs (placemats, scenario notes, etc.);
  • Support for logistics, protocol, and meeting and event planning for the Minister and deputy ministers;
  • Coordinating the international events calendar, and tracking and reporting or e-scanning of events; and
  • Provision of advice on logistics and protocol to the Health Portfolio.
• Managing the International Health Grants Program (IHGP)
  • Development of the program priorities and management of grant program solicitation and proposals, supporting budget transfers from Health Canada, funding agreements with recipients, funding approval forms and coordination of payments to recipients, in coordination with PHAC’s Centre for Grants and Contributions.
• Priority setting and performance reporting
  • Development and implementation of the International Strategic Framework; and
  • Coordination of the Health Portfolio’s international priority setting, and performance and corporate reporting.
• Acting as Portfolio secretariat for international working groups
  • Provision of secretariat services for the Health Portfolio’s international committees, including meeting logistics, advice on forward planning, and governance reviews.
• Strategic integration initiatives
  • Development of communication strategies, deliverables, and tools (e.g., weekly Newsletter, online presence on GCPedia, GCConnex, and mySource Intranet Portal) to raise awareness of OIA, as well as Canada’s role internationally;
  • Social media monitoring and use (e.g., OIA-DG Twitter account) for global health trends; and
  • Development and implementation of OIA communication strategies.

There are several other activities carried out by all three divisions, each identified with an office of primary interest (OPIs):

• Global health policy development, analysis, and negotiations (BESTD and MRD are OPIs)
  • Leading development of an integrated global health agenda for the Health Portfolio and Minister (priority setting for the Health Portfolio) and Canada’s “branding” profile (ODSI is the OPI);
  • Development of policy papers, options, recommendations, and advice;
  • Development of Canadian positions;
  • Input into resolutions, international commitments, and leader’s declarations; and
  • Development of briefs and policy research papers.
• Global health issues and trends analysis (ODSID and DGO are OPIs)
  • Development of medium-term planning documents, trends analysis, and e-scanning; and
  • Input into transition briefing materials, presentations, and policy briefs.
• Stakeholder engagement on global health issues
  • Development and implementation of stakeholder engagement strategies (MRD and ODSI are OPIs);
  • Focal point for Health Portfolio outreach (ISF implementation) on key global health issues (ODSI is the OPI);
  • Coordinate key stakeholder meetings and consultations (MRD and BESTD are OPIs); and
  • Ensure that provincial and territorial, youth organizations, and other civil societies (non-governmental organizations) participate in international delegations, where appropriate (MRD is the OPI).
• Coordinating and contributing to international priority setting (ODSID and DGO are OPIs)
  • Leading development of an integrated global health agenda for the Health Portfolio and Minister (priority setting for the Health Portfolio) and Canada’s “branding” profile;
  • Development of medium-term planning documents, trends analysis and e-scanning; and
  • Input into transition briefing materials, presentations, and policy briefs.
• Planning, monitoring, and reporting on deliverables and commitments (ODSID and DGO are OPIs)
  • Provision of operational, strategic planning, and reporting advice and input to corporate and Government of Canada documents and approaches;
  • Development and implementation of strategies and approaches for international priority setting (e.g., ISF implementation and framework development) and documentation (including evaluations and reviews);
  • Focal point for the collection of streamlined data;
  • Development of indicators, performance frameworks, and implementation of the online compendium and other digital tools and information resources for OIA; and
  • Lead Official Development Assistance reporting for the Health Portfolio.
• Interacting with other government departments on global health (carried out by all divisions)
  • Consultations, meetings, and interdepartmental meetings on global health issues to find common approaches, share lessons learned, and develop a community of practice; and
  • Provide advice and input into reports, policy positions, and negotiations.

Appendix 2 – Evaluation Description

Evaluation scope

The evaluation covered the period from April 1, 2012 to September 30, 2017, and included all OIA activities. It did not assess international activities carried out by other branches within the Health Portfolio.

The evaluation examined questions in relation to the relevance and value of OIA as a shared service, as shown in the Table below.

Core Evaluation Issues and Questions

Core Issues	Evaluation Questions
Design and delivery
Issue #1: Design and delivery	How appropriate are OIA’s mandate and activities? Is OIA being governed as planned? How well is the shared services model’s governance functioning? To what extent have OIA’s products and services reached intended audiences, including audiences from other Health Portfolio agencies (i.e., CIHR, CFIA, PMPRB)? Is the Health Portfolio shared services model an efficient way to address international issues for PHAC and Health Canada? What value added does OIA provide as a shared service that wasn’t present before? What is the relationship between branch-led international activities and OIA’s? How does OIA compare to other similar Government of Canada Portfolio structures?
Performance (effectiveness, economy and efficiency)
Issue #2: Effectiveness/Impact	How effective has OIA been in supporting the advancement of the Health Portfolio’s international priorities? How effective has OIA’s international engagement been in supporting Canada’s domestic priorities?
Issue #3: Demonstration of Economy and Efficiency	What is the level of resources (human and financial) allocated for OIA? To what extent have OIA’s resources been used or spent as planned?

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Data collection started in November 2017 and ended in July 2018. Data for the evaluation were collected using the following methods:

• Literature review – A search of literature from the past few years using search terms such as “lean management in government”, “pros and cons of shared services”, “best management practices”, “shared services” and “global health”. After examining documents to ensure relevance, approximately 35 articles were reviewed.
• Program document and file review – Approximately 560 documents, held by the divisions responsible for OIA activities, were reviewed to obtain information regarding all aspects of the activities related to international affairs. This includes a review of the Records of Decisions from 10 PEC-IAWC meetings.
• Financial data review – A review of financial data from 2012-13 to 2017-18 was conducted, including budgeted and actual expenditures.
• Key informant interviews – Interviews were conducted with 43 stakeholders: internal current and former program staff (n=15); partners and clients (n=23), including senior management (7), finance (2), Portfolio partners (11), and other government departments (3); and external stakeholders (n=5), including representatives from international organizations (4), and provinces and territories (1). Key informants were selected for their knowledge of, and experience with, OIA activities.
• Client survey – Survey questions were developed based on the Evaluation Framework to ensure that the data collected would support the questions being asked during the evaluation. The survey was initially sent to 128 recipients and 22 additional recipients were added throughout the duration of the survey as the initial recipients provided e-mail addresses of colleagues who fit the respondent criteria, for a total of 150 recipients. The survey was sent to respondents across the Health Portfolio and to a mix of senior management and working level stakeholders, many of whom were members of the PEC-IAWC, PIE-WG, and branch single windows. The evaluation received 33 completed surveys for a response rate of 22%.

Data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

Appendix 3 – Senior Management International Engagements

Office of International Affairs: Senior Management Engagements

Year	Total engagements	Event type (examples)	Topics discussed (examples)
2017Footnote K	33 involved the Minister 5 involved the Chief Public Health Officer (CPHO)	G7 Health Ministers' Meeting Visit to Washington, D.C. PAHO Reception G20 Health Ministers' Meeting OECD Policy Forum and Health Ministers Meeting Annual meeting of the International Association of Public Health Institutes (IANPHI)	Vaccination, tobacco control, healthy eating, Canada’s 2018 G7 Presidency, mental health, climate change and health, AMR, International Health Regulations (2005), emergency preparedness and response, health security, GHSI, GHSA, regulatory cooperation, opioids, etc.
2016	34 involved the Minister 8 involved the  CPHO or President of PHAC   9 involved senior management (DM, ADM)	Annual Meeting of the IANPHI Global Fund Replenishment Conference G7 Health Ministers Meeting Visit from Public Health England World Health Assembly UNGASS on the World Drug Problem GHSI and Ministerial bilateral program WHO Executive Board	Vector-borne diseases, trilateral initiative on childhood obesity, International Health Regulations, Canada-China High-Level Dialogue, healthy aging, dementia, HIV/AIDS, tuberculosis, International Health Regulations Focal Point capacity, AMR, etc.
2015	10 involved the Minister 55 involved CPHO, Deputy CPHO or President 46 involved senior management (DM, ADM)	GHSA Steering Group Meeting GHSAG Senior Officials teleconference Canada-U.S. Health Summit IANPHI Annual Meeting G7 Health Ministers Meeting WHO Executive Board 137 World Health Assembly 68 CPHO Visit to West Africa PAHO Deputy Director Visit DM visit to Washington, D.C. 14th World Congress on Public Health WHO EB 136	Bilateral engagement, Syrian refugees, International Health Regulations, public health capacity building, AMR, dementia, aging, innovation, Ebola, non-communicable diseases, Millennium Development Goals - Post 2015, etc.
2014	45 involved the Minister 17 involved the  CPHO or Deputy CPHO   25 involved senior management (Associate DM, DM, ADM)	GHSI MIN 2014 GHSAG Senior Officials Meeting GHSA White House Event Visit to Canada (China) GHSAG Senior Officials teleconference DM visit to Washington, D.C. AIDS 2014 World Health Assembly 67 Canada-China Policy Dialogue 2014	Healthy aging, obesity, Ebola, AMR, WHO governance - Executive Board, dementia, G7 Presidency, bilateral engagement, GHSAG/GHSI governance, laboratory capacity building, etc.
2013	18 involved the Minister 7 involved the Deputy CPHO 28 involved senior management (Associate DM, DM, ADM)	GHSI MIN 2013 G8 Dementia Summit PAHO Director visit Minister’s visit to Washington, D.C. Visit to Canada (CARPHA) IANPHI 2013 Executive Board 8th Global Conference on Health Promotion World Health Assembly 66 GHSAG Senior Officials teleconference	MERS, influenza, healthy living/obesity, dementia, violence prevention, bilateral collaboration, travel and tourism health, chronic disease prevention, mental health and suicide prevention, vaccination, tuberculosis, Millennium Development Goals, etc.
2012	24 involved the Minister 10 required CPHO participation 30 involved senior management (Associate DM, DM, ADM)	Minister’s visit to Sweden GHSI MIN 2012 GHSAG Senior Officials Meeting 2012 CARPHA TAC 13th World Congress on Public Health AIDS 2012 2012 Pan American Sanitary Conference World Health Assembly 65	Bilateral relations, mental health, northern health, Beyond the Border, healthy living/obesity, International Health Regulations, global health security, health care, non-communicable diseases, WHO reform, etc.

Source: OIA internal document