Evaluation of Defence Policy and Diplomacy

November 2013

1258-198 (CRS)

Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.

Acronyms and Abbreviations

Executive Summary

This report presents the findings and recommendations of the evaluation of Defence Policy and Diplomacy (DPD) program activities within the Department of National Defence (DND). The evaluation was conducted by Chief Review Services (CRS) between June of 2012 and June of 2013, as a component of the DND/CAF Five-Year Evaluation Plan (fiscal year (FY) 2012/13 to FY 2016/17), and in compliance with the Treasury Board Secretariat (TBS) Policy on Evaluation (2009). As per the TBS policy, the evaluation examines the relevance and the performance of the Program over a five-year period (FY 2008/09 to FY 2012/13). A grant and contribution program—The Military Training Cooperation Program (MTCP)—was also included in the scope of the evaluation, as it is administered by the defence policy function. Those findings are presented separately in Annex E.

Program Description

The DPD program is, for the purposes of this review, an amalgamation of the activities that support the ability of both the DND/Canadian Armed Forces (CAF) and the Government of Canada to make informed decisions with respect to international and domestic military affairs, as well as those activities that support the planning and conduct of military operations. This includes those activities involving defence issues and policy, and the many functions that the DND and the CAF undertake as part of their overall defence diplomacy effort, including foreign military training, state visits and exchanges by senior military/DND personnel, and numerous cooperation programs with foreign partner countries. For the purposes of this review the focus has been on the activities of policy analysts, defence attachés and foreign liaison officers.

Relevance and Performance

The evaluation determined that the need to conduct both defence policy and diplomacy is of continuing relevance and is aligned with federal government and departmental roles, responsibilities and priorities. The Program plays a key function in providing overall guidance and direction for the DND, strongly supports international diplomacy, and is critical to the delivery of international military operations.

With respect to performance, during the period of the evaluation the Program met all stakeholder expectations, including the timely provision of valued information and non-partisan advice to both military and civilian decision-makers. It was seen to be effective in establishing foreign-military relationships and engagements, and in supporting military operations such as those conducted in Afghanistan, Haiti and Libya. During those operations, the Program successfully provided vital support in matters such as the coordination of local resources and infrastructure, access to airspace rights, and the use of foreign bases by Canadian assets.

Concerning the MTCP grant and contribution program, the evaluation found that it is in high demand, well-managed, and operates in an efficient manner. It demonstrated that it was a useful tool to assist with diplomatic efforts, although there is only anecdotal evidence of its ability to ultimately achieve its stated objectives of building foreign capacity for peace operations and promoting Canadian values among participating nations. This Program has also begun to take steps to re-orient its focus towards capacity building through the establishment of training centres in key participating countries, unlike its past concentration on providing training within Canada to participants from a broad range of countries.

In summary, the DPD program has demonstrated sound management and results, with only relatively minor issues identified as improvement opportunities. Going forward, the Program has taken steps to be more strategic. The Global Engagement Strategy (GES), introduced in 2010, is a broad-reaching initiative that appears to be more comprehensive than past military diplomatic approaches. There is a concern, however, that as the implementation of the strategy comes into force, budget constraints may impact the Program’s ability to be delivered in the most effective manner. While the GES is being designed on the premise that no additional funding should be required for implementation, the reductions undertaken by the various attaché offices during the evaluation period have created challenges for these offices.

The findings and recommendations of the evaluation are presented as follows:

Findings and Recommendations

Key Findings 1 and 2. There is an ongoing demonstrable need for the defence policy and diplomacy function within DND.

Key Finding 3. The DPD program activities align with federal roles and responsibilities.

Key Finding 4. DPD program objectives align with current DND priorities.

Key Finding 5. The Program is effective in providing timely, evidence-based advice and support on defence policy matters.

Key Finding 6. While, overall, satisfaction of clients with the policy function is quite high, recent budget reductions and workload increases may impact the ability to sustain this level on an ongoing basis.

Key Finding 7. The outcome “Relationships and Partnerships are established and strengthened” is being met; however, there are some concerns regarding the implementation of the GES.

Key Finding 8. Key program-recipient stakeholders indicated that the Program strongly supports the Government of Canada’s ability to make well-informed decisions involving defence policies, issues and military operations.

Key Finding 9. Case study evidence demonstrates that the defence policy and diplomacy  areas play critical roles in the ability of the forces to undertake military operations.

Key Finding 10. Based upon trend analyses and benchmark comparisons, the DND Policy Group is operating efficiently.

Key Finding 11. The efficiency of the Canadian Defence Attachés (CDA) and Liaison Officers (LO) program areas has increased. There are concerns, however, that the reductions in travel and hospitality are beginning to impact the ability of the Program to operate effectively.

Recommendations

1.         Develop and implement a performance measurement strategy to measure ongoing client satisfaction.

OPI: ADM(Pol)

2.         Acknowledging staffing pressures, develop a strategy to ensure that succession plans are in place to ensure continuity of the Program, in particular through the development of new policy officers.

OPI: ADM(Pol)

3.         As per the Policy Group’s business plans, address the issue of ensuring that staff have proper clearances as well as access to secure infrastructure networks.

OPI: ADM(Pol) and ADM(IE)

4.         It is recommended that the aims and priorities of the GES need to be further articulated, possibly on a region-by-region basis.

OPI: ADM(Pol)

5.         In support of the GES, there is a need to coordinate high-level visits and other engagement activities centrally, and to share the results of each activity among all stakeholders.

OPI: SJS and ADM(Pol)

6.         The current distribution of resources (office locations and numbers, staffing, travel and hospitality budgets) should be examined and re-allocated as necessary to ensure that there is a realistic alignment and balance of resources to identified priorities and needs.

OPI: VCDS

Note: Please refer to Annex A—Management Action Plan for the management response to the CRS recommendations.

1.0 Introduction

1.1 Profile of Defence Policy and Defence Diplomacy

1.1.1 Background

This report presents the findings and recommendations of the evaluation of DPD sub-sub-program activities at DND. The evaluation was conducted by CRS, within DND, as a component of the Department’s Five-Year Evaluation Plan (FY 2012/13 to FY 2016/17), in compliance with the TBS Policy on Evaluation (2009). As per this TBS policy, the evaluation examines the Program’s relevance and performance.

Previously, CRS conducted evaluations of some of the sub-elements covered by this evaluation, e.g.: Evaluation of NATO Contribution Program (2011); Evaluation of Security and Defence Forum (SDF) Class Grant Program (2010);Summative Evaluation of the Contribution Agreement with the Military Training Assistance Program (MTAP) (2009); and Evaluation of CF Outside Canada (OUTCAN) Exchange and Liaison Programs (2006).¹

In the conduct of the evaluation, CRS was supported by an advisory committee composed of representatives from Assistant Deputy Minister (Policy) (ADM(Pol)) and from the Vice Chief of the Defence Staff (VCDS) Directorate of Foreign Liaison (DFL) and VCDS OUTCAN Coordination (OUTCAN Coord). This advisory committee was consulted on many occasions, including regarding the project scope, approach, and preliminary findings.

The evaluation was conducted from October 2012 to July 2013. In terms of data and collection of evidence, it covered the period from FY 2008/09 to FY 2012/13. Annex B, Evaluation Methodology and Limitations, provides a summary of the methods used by the evaluation team to gather information.

1.1.2 Program Objectives

A logic model was developed in order to identify the activities and the outputs of the Program, as well as to determine its immediate, intermediate and ultimate outcomes (see Annex C). The logic model illustrates the two principal, interrelated areas of this sub-program: defence policy and diplomacy.

The objective of defence policy is to provide to senior DND officials, the Minister of National Defence (MND), Cabinet and Parliament, timely, evidence-based and forward-looking advice and support on issues concerning the CAF and the role of National Defence.

“Defence diplomacy” is defined as the use of the CAF and DND assets as a tool of wider foreign and security policy.² The objectives of defence diplomacy are to build cooperative relationships through engagement with foreign militaries in order to support operational needs, military decision-making, and the advancement of Canadian goals and agendas. Most of the activities of this area are performed under the guidance of the GES.

Overall, these interrelated objectives lead towards support of the departmental strategic outcomes of the Defence of Canada, the Defence of North America, and participation in Domestic and International Peace, Stability, and Security.

1.1.3 Program Description

The DPD program includes the activities within the Department that share the common objective of supporting the ability of both the DND/CAF and the Government of Canada to make informed decisions with respect to international and domestic military affairs, including those involving defence issues, policy, international military relations, and support to the planning and conduct of military operations. There are, in fact, many activities that the DND and the CAF undertake that can be considered part of its overall defence diplomacy efforts, including for example foreign military training, state visits and exchanges by senior military/DND personnel, ship visits to foreign ports, as well as numerous cooperation programs with foreign partner countries. For the purposes of this review, the focus has been on the activities of policy analysts, defence attachés and foreign liaison officers.

Concerning defence policy, the evaluation takes into account activities primarily conducted under ADM(Pol). These include those related to the Director General Policy Coordination, particularly with respect to advice provided in support to Ministers and DND/CAF senior officials. The evaluation also addressed the development of policies, the formulation of advice and the writing of strategic reports by the Director General Policy Planning and by the Director General International Security (DGIS). This includes domestic and international defence policy initiatives, whether these are “global,” such as the Canada First Defence Strategy (CFDS) and the GES, or sectored, such as arms control issues.

Defence diplomacy includes three areas of activity. The first, which falls under the authority of the VCDS, represents the contribution of the CDA, the Canadian Defence Liaison Staff London (CDLS(L)), Canadian Defence Liaison Staff Washington (CDLS(W)), and the various LO. The information gathered in the field by CDA and LO help the Policy Group, the Strategic Joint Staff (SJS) and the Environmental Commands (Royal Canadian Navy, Canadian Army, Royal Canadian Air Force) to develop and address defence policy and operational issues, including those which pertain to Assistant Deputy Minister (Materiel) (ADM(Mat)) and Chief Defence Intelligence (CDI).

The second area of activity, within the jurisdiction of the Deputy Minister (DM) and delegated to ADM(Pol) for implementation, comprises Strategic Engagements and high-level visits made by high-ranking officials (i.e., ministers, senior officers and public servants) to other countries, in support of bilateral and multilateral relations and information exchange.

The third activity area is the MTCP, directed by the Policy Group’s DGIS. This grant and contribution program provides language training, professional development and staff courses, and peace support operations training, to a selection of foreign partner countries. Due to the interrelationships of this program with DPD, an analysis of this program was included as part of this evaluation. The findings of the MTCP evaluation are presented in Annex E.

1.1.4 Stakeholders

The main stakeholders involved in this evaluation are the program area managers and the program service recipients. Program managers include staff members from ADM(Pol) and the VCDS, including the Canadian Defence Attachés Offices (CDAO) and the LO posted abroad, who are directly responsible for managing the elements reviewed. In particular, from ADM(Pol), they are: the Director General Policy Coordination; the Director General Policy Planning; and the DGIS. From the VCDS, they are: the DFL 2; OUTCAN Coord; the CDLS(W); and the CDLS(L). From the CDS, they are the SJS and the CJOC.

Program recipient stakeholders include: the MND, the DM and the CDS, DND/CAF senior officers and officials (L1), the Prime Minister’s Office, Privy Council Office (PCO), Department of Foreign Affairs, Trade and Development (DFATD), Canadian International Development Agency (CIDA, now part of DFATD), Public Safety, Citizenship and Immigration Canada, and Canadian defence industry contractors. As users of the outputs of defence policy and diplomacy, these stakeholders rely upon its effectiveness in generating knowledge and establishing international relationships. Stakeholders also include foreign partner countries who participate in engagement activities (e.g., MTCP students).

1.2 Evaluation Scope

1.2.1 Coverage and Responsibilities

This evaluation supports several areas detailed in the DND/CAF Program Activity Architecture (PAA) effective 1 April 2013. It is linked to the strategic outcome for “Defence operations [to] improve peace, stability and security wherever deployed,” including portions of the sub-program activity Military Diplomacy (3.4.2.0). The evaluation also covers elements included under Management and Oversight (5.1.1.1). One grants and contributions program, under the Policy Group authority, is covered by this evaluation: the MTCP.

This evaluation excludes the North Atlantic Treaty Organization (NATO) Contribution Program already assessed in 2011, as well as the Exchange Positions and other OUTCAN assignments (e.g., Canadian militaries posted outside Canada: exchange with NATO, North American Aerospace Defence Command, United States, and other key allied forces), and does not cover the CDI program. The grants and contributions program Defence Engagement Program is also excluded, having been introduced only recently.

1.2.2 Resources

A breakdown of applicable resources utilised by the Program is shown in Table 1 and Table 2. The expenditures vary from $99 million in FY 2008/09 to $95.7 million in FY 2011/12, for an annual average of $96.7 million.

Table 1. Overall Expenditures for Defence Policy and Diplomacy, FY 2008/09 to FY 2011/12

The number of full time employees (FTE) allocated to the delivery of the program is shown in Table 2. Overall, there is not a lot of variation over the years covered by the evaluation.

Table 2. Full-time Employees for Defence Policy and Diplomacy, FY 2008/09 to FY 2011/12

1.2.3 Issues and Questions

As per the TBS Directive on the Evaluation Function, the evaluation addressed the following core issues (see Annex D for the evaluation matrix, which contains the evaluation questions, the indicators and the data sources).

2.0 Findings and Recommendations

Evaluation findings and recommendations are outlined in Sections 2.1 through 2.5.

2.1 Relevance – Continued Need

To determine the ongoing need for defence policy and diplomacy activities, the following areas were assessed:

• analysis of empirical data to determine if there was a demonstrable demand for the program; and
• assessment of qualitative evidence provided by the opinions of program managers and Program recipients (stakeholders) regarding the need for the program.

Evaluation Question. Is there an ongoing need for defence policy and diplomacy?

Key Finding 1. There is an ongoing demonstrable need for the defence policy function within DND.

Based upon empirical and interview data gathered from key stakeholders, the evaluation determined that over the evaluation period there was and remains an ongoing need for the defence policy function within DND.

The Business Plans (FY 2007/08 to FY 2012/13) for ADM(Pol) demonstrate that this Program continues to be the main source of defence policy advice for the MND, the DM, the CDS, the ADM(Pol), and to Level 1 (senior managers of the 21 organizational divisions within DND). According to senior decision-makers, the Program provides critical policy support, challenge and analytical insight into defence-related operations, issues and priorities, as well as for Cabinet liaison and as support to parliamentary affairs. In addition, it acts as a liaison between the defence and military groups via the policy advisors posted in the operational commands. These advisors facilitate policy alignment among the Commands, National Defence Headquarters, and the broader government community.

Since 2009, the demands and requests for information, policy advice and support to advance key government priorities appear to have grown. For example, there appeared to be an increase in the amount of requests to the Policy Group from the Government, the opposition parties and House of Commons committees. Over the last few years, the Government has undertaken a whole-of-government approach in conducting operations. In order to achieve that, the DND Policy Group was tasked with assuring coherence among multiple departments for various issues, such as overseas operations, defence intelligence, and the Canadian Arctic. The Policy Group was also consistently tasked with informing the PCO on issues related to national defence and international and domestic security (e.g., on Syria, Mali, Libya, or emerging issues such as cyber-space and biometrics). Table 3 presents statistics on deliverables prepared in 2012 alone.

Table 3. Deliverables Produced by the Policy Group in 2012

The interviews conducted during the evaluation corroborate the empirical evidence. All interviews with senior leaders, including those within ADM(Pol) and program stakeholders (Strategic Joint Staff, Canadian Joint Operations Command (CJOC), ADM(Mat), PCO Liaison, London and Washington Liaison Offices) confirmed that the Policy Group is an essential link between DND/CAF, PCO and the Cabinet. Where defence issues arise and are applicable, it promotes the coordination of a whole-of-government policy approach among multiple departments, such as DFATD, Industry Canada, and DND.

The program stakeholders stated through interviews that the Policy Group provides the necessary context to the CAF, explaining how actions are aligned with government orientations. According to these interviewees, the Group provides the over-arching policy framework for military activity, such as the GES for military diplomacy. As indicated by many interviewees, “the Policy Group provides the glue to the two components of defence: DND and the CAF.” Essentially, the Policy Group provides advice to the DM and MND through the lens of national and defence interests.

Key Finding 2. There is an ongoing, demonstrable need for the defence diplomacy function within DND.

Similarly, empirical and interview data gathered from the program managers (ADM(Pol) and VCDS), other stakeholders (SJS, CJOC, ADM(Mat), PCO Liaison, CDLS(W) and CDLS(L)) and the DM, led to the evaluation’s determination that there is an ongoing need for DND/CAF to conduct defence diplomacy-related activities, including the use of CDA and LO posted abroad. These resources are required in order to support the GES by establishing international partnerships and by assisting in military operations.

The vast majority of the interviews done with civilian and military senior leaders from both the program managers and the recipients affirm that the CDA act as an essential component of the GES. The CDA are a source of information for decision making regarding international issues. They act as “eyes and ears” for DND within their respective regions and provide context that helps the Policy Group to formulate advice. CAF military organizations, such as the SJS, CJOC and VCDS, consult CDA on a regular basis. The Policy Group also regularly advises the PCO on matters concerning the political and social climate of other countries, which is often derived from information obtained by the CDA.

Empirical and interview research with PCO officers indicated that the CDA’s presence is essential for the CAF to conduct international operations. The CDA establish relationships and partnerships with foreign forces. These contacts allow them to gather situational awareness, and to arrange in-country support. This includes obtaining landing authorizations overflight approval, and at times the use of allies’ military facilities. The CDA also provide advice and support to any Head of Mission (HOM) regarding military affairs.

Furthermore, it is evident³ that over the five-year period of the evaluation, the demand for CDA activities has grown. There has been a significantly increased need for foreign engagement strategy, both from the civilian and military senior leaders, to keep up with an increased overseas operational tempo. As demonstrated by VCDS business plans, there has been a significant increase in scope and level of activity in the foreign engagement strategy from both the MND and CDS offices.

The increase in the number of countries covered by the CDA program also illustrates the growing demand. In 1994, there were 28 CDAOs covering 55 countries. As of August 2013, the number of locations increased to 30 CDA offices, but the number of countries those offices are responsible for increased to 138.⁴ Their ongoing importance is also highlighted by the fact that Global Engagement became one of the key objectives in the 2013 CDS Directive for Force Posture and Readiness.⁵

2.2 Relevance – Alignment with Federal Roles and Responsibilities

Evaluation Question. Is it the role and/or responsibility of the federal government (and DND specifically) to deliver defence policy and diplomacy?

Key Finding 3. The defence policy and diplomacy activities align with federal roles and responsibilities.

A review of applicable federal legislation, department mandates, objectives and reports was conducted in order to determine whether the Program areas align with the roles and responsibilities of the Government of Canada and the DND in particular.

While the evaluation found that the review of federal legislation did not explicitly reference defence policy or diplomacy, these activities are seen to be an appropriate role for the Government of Canada to deliver on its Whole-of-Government Framework and support of the stated strategic outcome of “A safe and secure world through international engagement.” As per the outcome description:

The involvement of DND in this role is also appropriate. As per the FY 2012/13 Departmental Plans and Priorities, the activity areas of DPD align with the three key roles of DND: the defence of Canada; partnership in the defence of North America, and the contribution to international peace and security, by projecting leadership abroad.⁷

Although DND works with various other departments in the conduct of defence diplomacy, including DFATD, the evaluation found no evidence of duplication of roles. In the Government of Canada, DND/CAF is responsible for all defence-related issues and the involvement of military personnel,⁸ while other departments are responsible for security and international affairs. According to the documents and files consulted, the increasing complexity and horizontal nature of defence policy and military diplomacy requires cooperation and coordination among government departments, external leaders and international partners. The ADM(Pol) organization consults regularly with DFATD and other departments to ensure activities are aligned and complementary to each other. For example, there are cross-departmental meetings on initiatives such as regarding cyberspace, biometrics, to counter piracy, Canada-US defence relations, relations with the Americas, and Canada's participation in related international forums, including those with the United Nations and NATO. These represent just a few of the many activities of ADM(Pol).

2.3 Relevance – Alignment with Government Priorities

Evaluation Question. Are the objectives of the DPD program consistent with current Government of Canada and DND priorities?

Key Finding 4. The DPD program objectives align with current DND priorities.

As stated previously, the objective of defence policy is to provide to senior DND officials, the MND, Cabinet and Parliament, timely, evidence-based and forward-looking advice and support on issues concerning the CAF and the role of national defence.

The objectives of defence diplomacy are to build cooperative relationships through engagement with foreign militaries in order to support operational needs, military decision-making, and the advancement of Canadian goals and agendas.

The Canadian economy has been a well-voiced priority by the Government of Canada under the Harper administration. The DPD program contributes to it by promoting and developing cooperative defence engagements that can also open the door to enhanced economic relationships with partner countries. Recent examples of such engagements include:

• In FY 2012/13, senior officials, such as the CDS, DM, and Defence Minister, made official visits to such places as China, India, and Russia, with the aim of establishing stronger ties.⁹
• New agreements and arrangements aimed at strengthening military partnerships have been signed with a variety of countries as diverse as Mongolia, Peru and Israel,¹⁰ at the same time as Canada continues to play a key role in its membership in all hemispheric defence organizations. Further, Operational Support Hub agreements have recently been signed with Jamaica, Germany and Kuwait.

A literature review demonstrated that these program objectives also align with current (2013-2017) DND priorities.¹¹ These priorities include the following:

Ensuring Sustainable Operational Excellence both at Home and Abroad. The Program supports this priority through the provision of expert analysis, policy advice and a challenge function on a wide range of defence and security issues, both domestically and internationally; and through the management of Canada’s defence relationships and partnerships with allies.

Reconstituting and Aligning the CAF Post-Afghanistan. The Program supports this priority through: Focusing on DND’s post-2011 policy orientation to ensure the Department is prepared to respond to new challenges and to take advantage of emerging opportunities, including the provision of expert policy advice, and options to ensure the continued success of the transitioning to a training mission in Afghanistan.

Furthermore, the Departmental Plans and Priorities for 2013 have specifically highlighted defence policy and diplomacy activities:

• Supporting its Military Diplomacy activity, Defence has established a Global Engagement Strategy which forms the foundation for the development of comprehensive engagement plans for advancing strategic collaboration with key global partners. As part of this, Defence will initially focus effort on enhancing military defence and security relationships with key nations in the Western Hemisphere.

• In addition to participation in multilateral forums, Defence will continue to strive to build on current bilateral and multinational peace and security relationships and/or in establishing new ones in order to improve collaborative work in support of combined joint force employment and generation initiatives leading to increased interoperability.

2.4 Performance – Achievement of Expected Outcomes (Effectiveness)

To determine the overall effectiveness of the DPD program, a logic model of the Program was developed. The model grouped the program activities by common outputs, and then linked outputs to desired outcomes.

Effectiveness was then assessed by applying appropriate performance measures and/or key performance indicators against each outcome. Data for the performance measures was obtained from the ADM(Pol) and VCDS business plans, activity reports, a survey of defence attachés and LO, as well as conducted numerous interviews with program managers, attachés and recipient stakeholders. Based upon the results of this analysis, the overall effectiveness of the Program was determined.

Accordingly, an assessment was made of the following outcome areas:

• Immediate Outcomes
  • Timeliness and quality of advice and information provided; and
  • Establishment and maintenance of relationships with international military partners.
• Intermediate Outcomes
  • Ability to support defence issue decision-making; and
  • Support to planning and conduct of military operations.
• Ultimate Outcomes
  • Contribution to overall department strategic direction and successful military operations.

Immediate Outcome 1. To what extent is timely, quality advice and information on defence issues provided to senior stakeholders?

Key Finding 5. The Program is effective in providing timely, evidence-based advice and support on defence policy matters.

Based upon the recipient stakeholder interviews, the vast majority of interviewees said that the Policy Group meets all expectations with respect to providing timely, evidence-based and forward-looking advice. Examples of such advice and information included:

• formulation and articulation of domestic and international defence policy initiatives;
• management of the Department's bilateral and multilateral defence and international security relations;
• advice and support in dealing with Cabinet and Parliamentary issues; and
• strategic policy support on major corporate initiatives such as Strategic Review, the implementation of the Canada First Defence Strategy and Canadian Forces Transformation.

There was seen to be a high level of satisfaction from stakeholders with respect to the timeliness and quality of work within the policy domain. Interviews with both recent and current senior civilian and military decision-makers, including those with experience at the highest levels within the federal government, indicated that the policy and information they received met all their expectations and that they were completely satisfied with the speed, content, and quality of response.

While, overall, there was strong satisfaction with the delivery of policy advice and information, stakeholder and program manager interviews, documents and file reviews revealed some issues and concerns.

The first issue identified concerns about workload. Since 2007, the number of requests and demands appear to have increased considerably, partly due to changes in the domestic political context and in part due to an increase in the amount of participation by DND in international engagement and crisis responses. Further, the increased use of social media has expanded the sheer volume of information that needs to be addressed, making defence and security issues more complex and dynamic.

While the number of FTE involved in these activities has remained stable¹² over the last five years, there has been an increase in the number of stakeholders and a corresponding increase in requests for information, particularly from the opposition parties and parliamentary committees. As a result, there is a need to establish priorities. This, however, is difficult, given the nature of the work where essentially everything is deemed a priority, particularly when supporting the Minister and Parliament.

A second issue that raised concerns was the impact of the recent Strategic Review and the Deficit Reduction Action Plan (DRAP). Increased workload demands have increased operating costs. For example between FY 2008/09 and FY 2011/12, translation expenses have increased by 64.2 percent (from $534,643 to $877,788), and mandatory increases in salaries have created an inflationary pressure of 16 percent during the period of the evaluation.¹³ At the same time, the budget of the Policy Group has remained flat. As a consequence, there are fewer FTE and the operations and maintenance (O&M) envelope has been significantly reduced. As a result, the Program has reduced expenditures in various items, particularly the travel budget. For some areas of the Program, concerns were raised during the interviews that this is reducing the ability to gather and exchange information on the international stage.

Another potential consequence of stagnant funding is the associated human resource risks. Budget restrictions have prevented the group from bringing in new employees (especially new analysts who can be groomed for future roles), and limiting internal promotion opportunities. While the Program has not seen a higher than normal turnover, this may be a consequence of the current lack of movement in the federal government at large, but it could become an issue. Further, any future financial reductions and the ongoing impact of inflationary pressures may erode the ongoing capacity of an already lean program. Although the evaluation found no evidence of these cuts impacting the ability of the Policy Group to meet expectations to date, it may become an issue over time.

A third issue is concerning the lack of any secure infrastructure. New and emerging areas of concern, like cyberspace, biometrics and intelligence, as well as increased engagement with the intelligence community, require infrastructure for secure communications and document production and handling. The Policy Group does not have direct access to top secret electronic systems, such as Spartan and Mandrake, so employees with a top secret security clearance can only access these networks from a CDI facility. The business plans¹⁴ of ADM(Pol) have repeatedly highlighted the need to improve this area. Another issue in the security realm is the long delay to obtain higher security clearances, such as Secret (Level II) and Top Secret (Level III) for employees. It takes upwards of a year for an employee to gain a Secret (Level II) security clearance, and up to two years for the Top Secret (Level III) clearance. As a result, this translates into delays and an increased workload for those who need these security clearances.

Taken together, these issues may begin to have an effect on overall effectiveness with regard to the timeliness and even the quality of advice and information provided by the Program. It will be critical to ensure that these impacts of increased workload and flat/reduced budgets do not lead to diminished performance. There are at present no performance measures for the area that could monitor benchmarks such as stakeholder satisfaction.

Key Finding 6. While the overall satisfaction of clients with the policy function is quite high, recent budget reductions and workload increases may impact the ability to sustain this level on an ongoing basis.

Based upon these observations, the evaluation recommends the following:

Immediate Outcome 2. How effective are the DPD program’s approaches with respect to establishing/strengthening relationships and partnerships?

Key Finding 7. The outcome “Relationships and Partnerships are established and strengthened” is being met; however, there are some concerns regarding the implementation of the GES.

The establishment of relationships and partnerships with other militaries is a multi-faceted function that can involve a range of activities.¹⁵ To determine the effectiveness of defence diplomacy in achieving this outcome, the evaluation conducted in-depth interviews with client stakeholders (force generators and force employers), as well as a survey of all CDA and LO. Furthermore, a review of CDA reports was performed to assess their activity levels and to determine if indeed they were successful in the establishment of MOU and other formal agreements with other militaries or governments. It is of note that CDAs facilitate immensely the negotiation of MOUs, but they are actually developed and negotiated by capital. Lastly, the geographic location of activities was assessed against the strategic direction of the Program. Although this outcome is mainly related to the activities funded by military resources, it should be noted that the Policy Group does play a strategic role in providing a departmental orientation through the GES.

Based upon the results of the survey, document reviews, and the stakeholder interviews, the evaluation concludes that the expectations for this area are generally being met. The military diplomacy activities were seen to lead to more open and frequent exchanges of information, and furthered establishing and strengthening relationships and partnerships. Stakeholders, the CDA and the LO all agreed that it is important to participate in networking activities (formal and informal) in order to obtain a comprehensive understanding of other countries’ perspectives and their defence organizations. There was also significant evidence of the establishment of formal agreements and exchanges that can be attributed to the efforts of the CDA and other defence diplomacy activities.

However, some areas of concern were raised.

According to the survey data, the primary concerns among the CDA cadre appears to be a sense of lack of direction with respect to what type of strategic outcomes they are supposed to achieve, a lack of coordination of activities among all players towards achieving those outcomes, and the availability of resources to meet objectives.

Concerning strategic direction, the framework is outlined in the 2010 GES. The objective of the strategy is to “provide a framework for the conduct of international defence relations by the Department of National Defence and the Canadian Forces, to promote coherence in allocation of defence resources by setting clear priorities for engagement and partnerships.”¹⁶

A significant portion of the CDA and LO surveyed agreed that the GES is a great improvement over past directions, in that it is a formal approach delivered from the center, as opposed to the more organic “free for all” that existed prior to the launch of the strategy. However, all indicated concerns over the actual implementation of the strategy. Comments from the CDA included that the GES is “unclear with respect to its aims and priorities.” It was also seen to be too far-reaching and that it does not seem to provide guidance on what to concentrate upon within specific regions. For example, there appeared to be uncertainty among the CDAs with respect to what they were supposed to be “engaging in”—i.e., should they be focussed on military cooperation, economic activities such promotion of Canadian industries, or providing perspectives on the local civil-political-military climate?

Other CDAs surveyed raised concerns that the Attachés are often chosen at the end of their careers and, therefore, often retire after their posting. As a result, the Department does not benefit from their diplomatic knowledge and networks. In other countries, such as the United Kingdom and Australia, being an attaché is part of a career path: as experience is gained in different fields related to an attaché’s job, more important posts are offered.

With respect to effort and geographic location, a significant portion of the CDA and LO surveyed agreed that the defence policy and diplomacy activities are focused on the right countries to support the GES. The evaluation examined the location of CDA offices and countries visited/engaged to determine if any changes had occurred as a re-alignment with the objectives of GES. Since 2011, there have been movements of CDA offices; however, the evaluation found that these moves were minor and, generally, that the offices stayed within the same regions. For example, the CDA in the Ivory Coast CDAO moved to Senegal, the Hungary office moved to Serbia, Argentina’s was moved to Chile, and the office in Syria moved to Lebanon. As a result, it would be difficult to conclude that the GES has resulted in a change of focus of CDA activity as of 2013.

Concerning other military diplomacy types of engagement, a review of activities demonstrated that the CAF and senior DND officials are concentrating less on traditional regions (NATO and the United States) and more on other areas such as the Middle East, the Caribbean, South America and Asia, thereby complying with the global approach of the new strategy.

Issues with coordination of effort were also noted. Initially, CJOC was responsible for developing a Global Engagement Plan (GEP) for implementation of the GES. However, as the GEP followed the strategy, which only came into effect late in 2010, the GEP has not yet been fully incorporated. Further, authority for managing the GEP in FY 2012/13 has been subsequently transferred to the SJS. The revised GES framework will align Department’s Global Engagement activities more concisely in the future. The GES system will include a framework guidance document substantiated by a number of regional annexes. L1s will be expected to follow the GES guidance in planning and allocating resources to Global Engagement activities. There will no longer be a GEP as of December 2013. Rather, L1s will be responsible to report their Global Engagement activities to SJS, which will coordinate and deconflict engagement activities in accordance with the guidance in the GES framework document and associated annexes.

Lastly, there are concerns with the ability of the Program to fully engage all areas highlighted by the GES, due to a lack of resources. Many CDA and LO felt that some key areas/countries were not serviced to the extent required, simply due to a lack of resources. For example, face-to-face dialogues were not occurring due to budget restraints on travel. In addition, there was an overriding consensus from all Program managers that the new policy on hospitality is very time consuming and difficult to manage. The CDA, the CDLS(L) and CDLS(W) have to request a minimum of ten weeks in advance for any hospitality request going to the MND. Lower levels have authorization to approve hospitality, but for lower amounts and not for certain situations. For example, DFL can approve a dinner at a restaurant with no alcohol and no spousal accompaniment. However, if alcohol is served or the CDA has their spouse there, the MND must sign. Sometimes, they have to cancel a military diplomatic event because they didn’t receive the approval in time. The hospitality restrictions were seen to have negative effects on networking activities, especially when LO from other countries are not subject to the same funding restrictions.

The majority of CDA and LO also stated that their workload grew over the five-year period of the evaluation, mainly due to an increase in volume and responsibilities with no additional resources to execute the work. Budget restrictions have caused a reduction in support staff, impacting the ability of CDA and LO to conduct “diplomatic” activities as opposed to administrative ones. Related to this are the increased demands placed upon the CDA and LO for geographic coverage. In 2008, CDAs were responsible for an average of two countries; in 2012, they were responsible for an average of four.

Based upon these observations, the evaluation recommends the following:

Outcome 3. To what extent does the Program support the Government of Canada and DND with regard to decision-making concerning issues of national defence?

Outcome 4. To what extent does the Program support the CAF as regards decision-making concerning military operations?

Key Finding 8. Key program-recipient stakeholders indicated that the Program strongly supports the Government of Canada’s ability to make well-informed decisions involving defence policies, issues and military operations.

Based upon the opinions of both current and past senior Program-recipient stakeholders (DM, VCDS, SJS, CJOC, PCO), the Program was found to have met all expectations and strongly contributed both to the decision-making process and the implementation of defence policy, as well as to support to operations. In discussions with these key stakeholders, the evaluation found that defence policy provided critical contextual information, guidance, options analysis and coordination concerning multi-departmental issues. In particular, the Program was deemed to provide coherent, independent, non-partisan strategic policy advice to the MND and the PCO Secretariat on Defence and Foreign Relations. PCO liaison interviewees stated, for example, that the Policy Group is essential for identifying solutions to defence issues.

Program managers interviewed also felt that the information and advice they provided supported the decision-making process. Although verifiable evidence was not available to definitively demonstrate the impact of their input to results, program managers certainly felt that they were an important and valued component in the process. For example, significant numbers of the CDA and the LO surveyed stated that their reports, briefs and advice provided critical information that influenced defence decision-makers. Furthermore, they facilitate the exchange of key information with international partners.

Outcome 5. To what extent does the DPD program contribute to overall department strategic direction and successful military operations?

Key Finding 9. Case study evidence demonstrates that the defence policy and diplomacy areas play critical roles in the ability of the forces to undertake military operations.

To assess the effectiveness of the Program in supporting military operations, the evaluation examined the role of military diplomacy and defence policy in the following operations conducted by the Department during 2008-13:

Arctic Sovereignty Operations. National Defence plays a key supporting role in Arctic Foreign Policy by conducting Canada’s international defence relations. Going forward, the Policy Group will continue to provide the Department, senior leadership and the government with overarching policy advice regarding the DND/CAF contribution to these key government policies related to the Arctic.¹⁷

Afghanistan Mission. Abroad, the CAF continued to make a significant contribution to security in Afghanistan. In October 2011, Canada completed the stand-up of the new training mission in Kabul, supporting Afghan National Security Forces’ training through a contribution of up to 950 CAF personnel—the second largest contribution to this NATO training mission after the United States.¹⁸ The Policy Group has been heavily involved in Canada’s mission in Afghanistan, providing strategic advice on issues such as biometrics, detainees, whole-of-government support, or any other aspects of direct relevance to the in-theatre mission. The CDA posted in that country provided help in different situations, including: assisting in the Afghan mission’s transition from a combat role to a training role; processing various authorizations and associated documentation in the detainee transfer; the whole-of-government efforts; and the visit coordinations for VIP/VVIP and bilateral contents.¹⁹

Humanitarian Assistance Mission Following the Earthquake in Haiti. Defence is furthering Canada’s strategic aims and ability to effectively take a leadership role in support of the global community through enhanced responsiveness and readiness to participate in disaster and humanitarian relief, non-combatant evacuation, peace support and combat operations. Operationally, DND continues to support the UN Mission in Haiti by providing personnel for key staff positions, including the Chief of Staff (COS).²⁰ The MTCP capacity building in the Caribbean, which led to the recent implementation of the Jamaica regional centre of excellence for air force and maritime training, contributed to having Jamaica deploy troops in Haiti to assist with humanitarian assistance and disaster response operations. Canada was invited to assist by moving personnel and supplies through several flights.²¹

Libya Mission. The CAF also demonstrated a high level of readiness to deploy elsewhere around the world and conduct a range of operations at the request of the Government of Canada, as evidenced by our participation in the NATO-led response to the crisis in Libya. The rapid and effective deployment of the CAF in support of the NATO-led effort to impose a UN authorized arms embargo and no-fly zone in Libya exemplified Canada’s leadership role in the global security environment.²² CDA supports to operations, including the role of the primary CAF advisor to the incident commander, required many tasks: organization and coordination of activities; communications, both secure and non-secure; moving to different locations by following CDI information and HOM directions, as well as the planning of the VIP/VVIP visits (Minister, etc.).²³

The examination of the above cases, as well as of the CDA surveys, demonstrated that the Program played a critical role in the actual conduct of operations. The majority of the CDA surveyed responded that it contributed to the conduct of major CFDS international operations and in assisting the management of crisis responses abroad. During the past five years, 70 percent of CDA reported that they were directly or indirectly involved in the support of UN/NATO operations and/or multinational exercises. This included supporting evacuation procedures of personnel, anti-smuggling, countering illegal migrations, securing overflight requests, facilitating foreign deployment logistics and lodging, operational supply chain management and sustainment, and partnership assistance. They assessed their contribution as being critical, whether it was simply in the role of exchanging information for planning purposes, or in enabling the actual operation through local logistic/administrative and other types of support.

More than half of the LO surveyed (54 percent) also indicated that they contributed to well-informed decision-making and to the ability of the CAF to conduct international operations and/or to assist in the management of crisis responses abroad. This included supporting the initial deployment of troops, redeployment planning; coordinating of support (life support, material support) and security clearances for personnel; ensuring the flow of information; performing in-country reconnaissance and liaison; and assisting in program identification and communications.

2.5 Performance – Demonstration of Efficiency and Economy

In order to analyse the efficiency of the Program, the evaluation team examined the resources utilised to produce outputs, including:

• the number of FTE within the Policy Group, in comparison with similar departments;
• the cost related to CDA and LO operations; and
• the use of best practices and demonstration of efficiency measures.

The following data sources were used:

• DND financial systems;
• opinion of program managers and stakeholders through interviews; and
• literature review and media scan.

Evaluation Question. Are defence policy and diplomacy activities being delivered efficiently?

Key Finding 10. Based upon trend analyses and benchmark comparisons, the DND Policy Group is operating efficiently.

To assess the efficiency of the Program, the evaluation focussed on expenditure trends and, where possible, benchmarks to other organizations.

With respect to the policy areas, it was noted that during the study period overall expenditures remained relatively flat, despite inflationary pressures. Cost per FTE rose by 8 percent over the four-year period (FY 2008/09 to FY 2011/12) (from $229,000 to $247,000). Yet overall expenditures remained stable at $43.5 million, reflecting a 7 percent decrease in the number of FTE. Essentially, as labour cost increased due to salary adjustments, the Program has reduced OM expenditures. As a result, the efficiency of the Program was improving throughout the evaluation period, despite the noted increases in workload.

A key concern going forward is whether the organization can continue to function effectively with 14 fewer FTE and a smaller O&M budget. This is a particular concern in that there do not appear to have been any new measures incorporated to gain cost-effectiveness; rather, employees are simply doing more with less. Previous sections of this evaluation noted that there are challenges with respect to limitations in funds for travel, translation, and security equipment.

Table 4. Policy Group Expenditures, FY 2008/09 to FY 2011/12

Compounding this observation is that the initial baseline from which efficiency gains were made appears to have already been quite lean. A direct comparison to other policy organizations is difficult, as all departments have unique needs and operating environments. However, the evaluation conducted a general comparison of FTE numbers within ADM(Pol) to those in other departments whose policy requirements should be most similar to those of DND. The evaluation found that DND has a much smaller policy group compared to departments such as DFATD and DFO. This is based upon actual baseline comparisons, as well as using metrics involving the percentage of policy FTE or policy budgets versus overall departmental numbers. Whether absolute values were used, or percentages of overall numbers compared, DND was consistently seen to have a smaller policy group.

Table 5. Comparison of Number of FTE per Policy Groups in Four Departments

Key Finding 11. The efficiency of the CDA and LO program areas has increased. There are concerns, however, that the reductions in travel and hospitality are beginning to impact the ability of the Program to operate effectively.

Similar results were found concerning military diplomacy. Overall expenditures for CDA offices remained flat despite inflationary pressures and the addition of two offices (a 6.7 percent growth). Over the past decade, the amount of countries each CDA was responsible for increased from an average of two countries in FY 1998/99 to four countries in FY 2011/12.

Table 6. CDA Offices Expenditures, FY 2008/09 to FY 2011/12

To address these additional pressures without a corresponding budget increase, the CDA program has reduced hospitality expenses by approximately 60 percent and kept flat the overall O&M expenses, including travel, despite the increase in coverage and inflation.

Table 7. Expenditures in O&M and Hospitality for the CDAO, FY 2010/11 to FY 2012/13

Even more dramatic reductions have occurred with respect to the major liaison offices in both Washington and London. Both CDLS(L) and CDLS(W) operated with significantly reduced budgets in 2011/12 compared to 2008/09, while maintaining the same staffing levels. The expenditure cuts represent 26.4 percent for CDLS(L) and 25.1 percent for CDLS(W). The cuts affected mostly hospitality and travel funds.

Table 8. Expenditures for the CDLS(L), FY 2008/09 to FY 2011/12

Table 9. Expenditures for the CDLS(W), FY 2008/09 to FY 2011/12

Efficiency and economy measures were implemented over the last two years due to financial restraint; namely, the new policy on hospitality reduced travel budgets for hospitality and travels (Figure 1). The DND/CAF policy on hospitality is aligned with TBS, but not with hospitality governed by the Official Hospitality Outside Canada policy, which was established by DFATD and the Heads of Post and Foreign Service Official Hospitality directives. According to this latter, only an HOM can host official diplomatic events, not the CDA. DND/CAF thinks differently, because the CDA has to organize military diplomatic events with high-level people, including senior officers. These events are deemed essential for defence diplomacy and for exchange of information.

Figure 1. Total Hospitality Expenditures for the VCDS L1s and L2s

Table 10. Total Hospitality Expenditures for the VCDS L1s and L2s, FY 2008/09 to FY 2012/13

Evaluation Question. Are the most appropriate means being used to achieve objectives, relative to alternative design and delivery approaches?

An assessment was performed by the evaluation team to determine if there are more efficient ways to deliver the DPD program through alternative approaches or delivery options. The evaluation observed that there has been no significant change in the organizational structure, tasks or delivery approach in this Program since the post-Cold War. That being said, the “in-house” approach appears to be the model utilised by those countries that work in the military diplomacy and defence policy domain.

Based upon discussions with the LO in London and Washington, a comparison was made with the policy groups of various defence organizations within Canada, the United States and the United Kingdom. Among all three countries, the development of defence policy and the formulation of advice all follows the same model of utilising information gathered through military diplomacy activities. In all these countries, there is a mix of civilian public servants and military employees working in policy groups to gather information to formulate advice and provide options to their minister. In the United States, closer links were seen to exist with external think tank groups and private sector consultants. However, no alternative approaches of significance were utilised.

The DND program stakeholders interviewed did not indicate that any alternative approaches, e.g., outsourcing the work to academia, relying on allies or other departments (DFATD for example), or creating a stand-alone organization or agency would result in improving cost-effectiveness. Defence diplomacy is considered a more appropriate lens to use, as it includes the broader defence community in its approach.

Program managers did suggest some potential reorganizational changes that may improve cost-effectiveness. This included more cooperation between the Policy Group activities, capability-based planning, and the various diplomatic and planning functions within the VCDS.

The evaluation also gave consideration to the notion of merging the activities of the policy, intelligence and military diplomacy functions into a single organization. This could potentially improve the coordination of diplomatic activities, the sharing of information and resources (particularly secure environments), and the provision of a unified message and advice to senior leaders. There were some concerns expressed with this option—in particular the fundamental belief that defence policy issues and decisions remain the responsibility of the DM, and, therefore, must not be unduly influenced by the military. As such, this alternative was not pursued.

Annex A—Management Action Plan

Performance – Achievement of Expected Outcomes (Effectiveness)

Management Action

ADM(Pol) will undertake a collective, whole-of-policy-team approach to a review of the organization that will take place during the period of October 2013 to March 2015. The review will see the Policy Group closely examine itself as an organization while reaching out to its stakeholders in a deliberate and systematic manner to identify and implement optimization and change that:

• encompasses and values its people; and
• enhances its ability to continue to deliver excellence to the Government of  Canada, defence team leadership, the Canadian public, allies and partners.

A critical component of the review will include engagement with both internal and external Policy Group stakeholders. Stakeholder engagement will be conducted early in the review to provide vital context to early mandate review and optimization analysis, with a view to gaining a richer understanding of the stakeholder dynamics within and surrounding the Group.

Stakeholder engagement for this review will occur as a series of interviews conducted by the Policy Group with its key stakeholders during the period of 12 November until end of December 2013.

This engagement will inform the review and also serve as a pilot exercise to a longer-term performance measurement strategy that incorporates stakeholder feedback into the regular conduct of operations of the Policy Group.

OPI: ADM(Pol)

Target Date: March 2015

Management Action

ADM(Policy) will develop a comprehensive talent management strategy to include focused training and assignment opportunities designed to attract and retain a stable pool of defence policy officers and to ensure that those who are ready for opportunities at the executive level are appropriately supported to transition to the next stage in their careers.

OPI: ADM(Pol)

Target Date: April 2015

ADM(Pol) will undertake a comprehensive review of the Policy Officer Recruitment Program. This process will include a full analysis of all aspects of the Program, including but not limited to, its size, training, evaluation process and point of graduation. The intent of this review will be to refresh and revitalize the program to ensure it continues to produce exceptional defence policy officers who fulfill a policy challenge function both within the Policy Group and elsewhere in the Department.

As part of a succession planning strategy, ADM(Policy) will develop a process, complementary to the Policy Officer Recruitment Programme, to attract and hire defence policy officers at all levels with a view to maintaining a steady FTE count within the group. In developing this strategy, other existing competitive and developmental processes will be examined and considered in our goal to achieve an optimal mix of competencies and experiences amongst ADM(Policy) defence policy officers.

OPI: ADM(Pol)

Target Date: April 2014

Management Action

The Policy Group will improve business processes that enable necessary staff access to Secret and Top Secret materials. This is already underway, and will align succession planning and security clearance staffing to ensure that clearances are processed in advance of internal postings. Concurrently, the Group will work with CDI to streamline the delivery of and access to daily Top Secret products for the ADM(Pol) and division leaders.

OPI: ADM(Pol) and ADM(IE)

Target Date: April 2014

Longer-term objectives related to improving infrastructure will be pursued through the Headquarters Transformation and Accommodations Committee. Beyond upgrading temporary Top Secret document-holding facilities (now complete), no further investment is envisaged until decisions on relocation of the Policy Group have been made.

OPI: ADM(Pol) and ADM(IE)

Target Date: April 2016

Management Action

ADM(Pol) concurs with this recommendation and is already well advanced in implementing such an approach. In March 2013, ADM(Pol) launched a review of the 2011 GES. A GES Working Group led by ADM(Pol) will define the strategic interests that drive global engagement priorities, and articulate the broad aims and objectives of global engagement activity. This work will culminate in the development of a GES Strategic Guidance document, authored by ADM(Pol), that will be completed in 2014. The GES and, in particular, the regional annexes will be subject to annual reviews to ensure that they remain relevant and consistent with broader Government of Canada policy and priorities, and to provide up-to-date guidance to internal stakeholders.

OPI: ADM(Pol)

Target Date: March 2014

Management Action

SJS. Coordination, synchronization and de-confliction are already occurring for FY 2014/15 planning. Further, this recommendation is presently being addressed as part of the GES refresh exercise. The following are extracts from the draft document:

OPI: SJS and ADM(Pol)

Target Date: December 2013

Performance – Demonstration of Efficiency and Economy

Management Action

DFL will continue to host two yearly meetings to discuss CDAO locations, staffing levels and other strategic items. The CDA Program Management Committee, a decision-level meeting typically held in the fall, is chaired by COS VCDS, and includes director general-level representation from the major stakeholders (SJS, ADM(Pol), DFATD, ADM(Mat), CJOC, CF Int Grp). The Strategic Operational Planning Group, a director-level meeting chaired by DFL, will continue to be convened each summer to ensure that direction from the mentioned Committee is followed, and to prepare information for its forthcoming meeting. To ensure relevance, the Committee’s Terms of Reference will be updated during the Committee’s fall 2013 meeting. DFL is in constant communication with all major stakeholders and works to ensure that short-term priorities from all stakeholders are balanced against long-term strategic goals, including the GES; it also locates CDAOs where needed.

DFL continues to work closely with COS VCDS, and the COS VCDS Comptroller, to ensure that sufficient funds are available to properly support CDAOs. Additional financial reporting has been instituted at all posts this fiscal year, allowing an in-depth review of the budgetary requirement for each post. This reporting will be used to compile budgets for coming years, which will be balanced against all stakeholder priorities, including both those that are short-term and strategic.

OPI: VCDS

Target Date: June 2014

Annex B—Evaluation Methodology and Limitations

1.0 Methodology

1.1 Overview of Data Collection Methods

The evaluation used multiple lines of evidence and complementary research methods as a means to ensure the reliability of information and data collected. The following research methods were used to gather qualitative and quantitative data for the evaluation:

• document and file reviews;
• key informant interviews;
• survey questionnaires to CDA and LO;
• review of financial and administrative data; and
• literature review.

1.2 Documents and Files Review

A preliminary document review was conducted as part of the planning phase of the evaluation to help gain an understanding of the DPD program’s activities and their context. A comprehensive document review was undertaken as part of the conduct phase of the evaluation, focusing on the relevance and the performance of the Program’s activities.

The following documents were reviewed during the conduct phase of the evaluation:

• Program documents: Treasury Board submission for the MTCP, business plans (Policy Group, VCDS, DFL, CDLS(L), CDLS(W), OUTCAN Coord), MTCP annual assessments, previous evaluation reports.
• Accountability documents: Reports on Plans and Priorities, Departmental Performance Reports, CFDS.
• Literature and research: Since the main scope of the project was the evaluation of the Policy Group, a literature review was undertaken on the themes of policy evaluation and on the impacts of research on policy development. Another literature review was conducted on military diplomacy in the United States, Britain, Canada and Australia.

The document review was conducted using a customized template organized according to the evaluation questions and indicators.

1.3 Key Informant Interviews

Interviews (n=31) were held with key senior stakeholders from the Policy Group (directors general and directors) and the VCDS (senior representatives of the CDLS(L), CDLS(W), and DFL). Evaluators also met with senior officials who are stakeholders in close relation with these two groups: DND, SJS, CJOC and ADM(Mat), as well as representatives from PCO. The following table provides the details.

Table 11. Number of Interviewees by Organizations

The interviewees were given an interview guide in advance. All the interviews were conducted in person with the exception of representatives of CDLS(W) and CDLS(L), which were done by phone. Interviews were recorded when authorized by the interviewees. The evaluators transcribed the notes taken during the interviews and paired them with the recorded documents.

The evaluation used the following qualifiers in the findings to give the reader a sense of the “weight” of the interview responses:

• “some or a few” refers to fewer than five respondents;
• “many” refers to less than half of the respondent group, but more than “some”;
• “most” refers to the majority of respondents; and
• the “vast majority” of respondents refers to more than 80 percent of the respondent group.

1.4 Survey Questionnaires to CDA and LO

A survey questionnaire was administered to the CDA and LO in order to get their perspective on the achievements of the Program between May 2013 and June 2013. The questionnaire was sent by secure email to all the 30 CDA and to 30 LOs posted abroad who had a direct link with DPD activities. We received 41 completed questionnaires (28 CDA and 13 LO) out of 60, for a 68 percent answer rate.

The questionnaire topics covered were: support received, the GES, the MTCP outcomes, the performance and efficiency of the Program, as well as alternative delivery options and improvements that could be implemented.

Respondents were asked to note their level of agreement with several statements using a five points scale ranging from 1, “Strongly disagree,” to 4, “Strongly agree,” five being “Not Applicable.” Respondents were also asked to provide their opinions and qualitative statements by writing them in open boxes.

The completed questionnaires were sent back to the evaluators by secure email, and then compiled manually on an Excel document.

1.5 Review of Financial and Administrative Data

The DPD program’s administrative and financial data was reviewed in order to determine the degree of efficiency and economy of the activities and outputs. The data covered four fiscal years, from FY 2008/09 to FY 2011/12. The data for FY 2012/13 was not finalized in July 2013, when this report was written. The information reviewed was derived from multiple sources in the DND financial reporting data base.

The administrative data reviewed was: Business and Human Resources plans (Policy Group, VCDS, CDLS(L), CDLS(W), DFL and OUTCAN Coord); historical reports from both CDLS(L) and CDLS(W), program activity reports related to high-level visits and hospitality, and MTCP reports.

The financial data for the Policy Group, the CDLS(L), CDLS(W) and the MTCP were extracted from the Defence Resource Management Information System; more specifically, from the Financial Management Accountability System, the HRMS and the Cost Factor Manual.

The data on the number of regular force personnel was also added to the costing, taken from either the Program PAA attribution or established from sources such as organizational charts with positions and ranks per year for organizations and divisions. Also, the regular force salary and the benefits related to Military Foreign Service Regulations were added to the costing for each group included in the evaluation.

The attributed PAA funds for the DPD program activities came mainly from defence diplomacy under PAA sub-sub activity 3.4.2. The Policy Group’s attributed funds came from several other elements of the PAA, mainly Civilian Recruitment – 1.2.1.3; Support to Other Organizations – 4.4.2.0; and Management and Oversight – 5.1.1.1.

Since many elements of the PAA 3.4.2 were excluded from the evaluation, the evaluators kept only 20 percent of the total expenditures attributed to this sub-sub activity. This required discussion with many L1s’ Comptroller in order to establish the exact expenditures related to the DPD program activities.

1.6 Literature Review

A literature review was undertaken on the following themes: evaluation of policy; evaluation of the impacts of research on policy development; defence policy and strategic studies; civilian-military relations; and military cooperation and diplomacy. Canada was the focus of this review; however, some benchmark research was also done on the United States, Great Britain and Australia.

2.0 Limitations

The following are limitations of the evaluation:

• the challenge of assessing effectiveness and attribution of “advice” in decision-making;
• the survey questionnaires were compiled manually by an evaluator, which can introduce error in the data entry;
• the possibility that the interviewees provided biased information and only positive stories about their program;
• determining exactly which CDA and LO positions related to the scope of the evaluation, while considering the exclusions identified in PAA 3.4.2.0; and
• financial information that could be unrelated to the scope of the evaluation.

The following table shows the limitations related to the sources, and the mitigation strategies applied to them.

Table 12. Evaluation Limitations and Mitigation Strategies

Annex C—Logic Model

Figure 2. Logic Model for the Defence Policy and Diplomacy Program

Annex D—Evaluation Matrix

Table 13. Evaluation Matrix—Relevance

Table 14. Evaluation Matrix—Performance (Effectiveness)

Table 15. Evaluation Matrix—Performance (Efficiency and Economy)

Annex E—Evaluation of the Military Training and Cooperation Program

1.0 Introduction

1.1 Profile of the Military Training and Cooperation Program

1.1.1 Background

The MTCP is a grants and contributions program of DND. The Program was evaluated as an element of a broader evaluation—the Evaluation of DPD program activities—and was conducted by the DND CRS as a component of the Department’s Five-Year Evaluation Plan (2012-13 to 2016-17), in compliance with the TBS Policy on Evaluation (2009). As per TBS policy, the evaluation examines the relevance and performance of the Program.

CRS conducted two previous evaluations of the MTCP: the Summative Evaluation of the Contribution Agreement with the Military Training Assistance Program (MTAP), in 2009; and the Military Training Assistance Program (MTAP) Formative Evaluation, in 2008.²⁴

In the conduct of this evaluation, CRS was supported by an advisory committee composed of representatives from the ADM(Pol) and the VCDS – DFL, and VCDS –OUTCAN Coord. This advisory committee was regularly consulted throughout the evaluation process and provided input regarding the project’s scope, the approach, and the preliminary findings.

The evaluation was conducted from October 2012 to July 2013. In terms of data and evidence collection, it covered the period from FY 2008/09 to FY 2012/13. A summary of the methods used by the evaluation team to gather information can be found in Annex B.

1.1.2 Program Objectives

Defence diplomacy is defined as the use of CAF or DND assets as a tool of wider foreign and security policy.²⁵ The objective of defence diplomacy is to build cooperative relationships through engagement with foreign militaries in order to support operational needs, military decision-making, and the advancement of Canadian goals and agendas. Most of the activities of the MTCP are performed in support of the DND GES through capacity-building and training of foreign military personnel. Overall, these interrelated objectives help to support this DND strategic outcome: “Defence operations will improve domestic and international Peace, Stability, and Security.”

A logic model of the DND DPD program, which included the MTCP component, was developed in order to identify the activities and outputs of the MTCP, and to determine its immediate, intermediate and ultimate outcomes (see Annex C).

1.1.3 Program Description

The MTCP is a grants and contributions program that provides language training, professional development and staff courses, and peace support operations training to a selection of non-NATO partner countries. It is delivered by the CAF personnel, as well as DND civilians, and in limited cases, by Defence contractors both in Canada and abroad. Training outside of Canada is sponsored by the MTCP in partnership with implementing partner member countries. The Program is managed by the DMTC, within ADM(Pol).

1.1.4 Stakeholders

The main stakeholders involved in this evaluation were program area managers and program service recipients.

Program area managers included those DND personnel who were directly responsible for management of the elements reviewed, including the MTCP directorate and the DGIS.

The program recipient stakeholders included: all DND/CAF Organizational leaders (L1); CDA; the Prime Minister’s Office, PCO; DFATD; CIDA, now part of DFATD; Public Safety; Citizenship and Immigration Canada; and selected developing non-NATO countries. Other stakeholders also included are: MTCP students, embassies and implementing partner countries (e.g., United States, United Kingdom, etc.). As users of the outputs of defence diplomacy, these stakeholders utilize the Program to help support the establishment and maintenance of international relationships.

1.2 Evaluation Scope

1.2.1 Coverage and Responsibilities

This evaluation supports activity 3.4.20 under the DND/CAF PAA. It is linked to strategic outcome: “Defence operations improve peace, stability and security wherever deployed.”

1.2.2 Resources

The grant and contribution program has an annual expenditure of approximately $20.6 million. Details are as follows:

Table 16. Overall Costs for Delivering the MTCP, FY 2008/09 to FY 2011/12

The number of FTE allocated to the delivery of the Program is described in Table 17. The groups of employees were affected by the implementation of DRAP. Starting in FY 2012/13, the Reserve Force positions were cut from 32 to 14, and will be terminated in 2013/14. Civilian language positions were also terminated in FY 2012/13. There are still Canadian Forces Language School (CFLS) instructors supporting DMTC programs; however, they are not part of the DMTC organization (FTE count).

Table 17. Full-time Employees for MTCP, FY 2008/09 to FY 2012/13

1.2.3 Issues and Questions

As per the Treasury Board Directive on the Evaluation Function, this evaluation addressed the following core issues: relevance, effectiveness and efficiency. The evaluation questions directly related to the MTCP are described below and the evaluation matrix for activities related to the DPD program can be found in Annex D.

2.0 Findings and Recommendations

Evaluation findings and recommendations are outlined in Sections 2.1 through 2.5.

2.1 Relevance – Continued Need

To determine the ongoing need for the MTCP, the following areas of performance measurement were utilised:

• Empirical data demonstrating a demand for the Program; and
• Qualitative evidence based on the opinions of program area managers and program recipient stakeholders (“clients”).

Evaluation question. Is there a continuous need for the MTCP?

Key Finding 1. There is an ongoing demand for the MTCP.

The MTCP has offered training to members of foreign armed services for nearly 50 years. It began with an initial cohort of 75 students in 1965 and has steadily increased in size over time. Approximately 992 students from 62 different countries were enrolled in the MTCP in 2011/12. According to Program data, Program growth has been limited due to issues with capacity rather than a lack of foreign interest. Limits have had to be placed on the number of MTCP participants every year since its founding due to high international demand.

Interview and survey data indicated that both program area managers and program recipient stakeholders felt that the MTCP was an important program.²⁶ CDA stated that the MTCP allowed them to gain access to senior defence officials from various countries and helped to establish relationships and good will.²⁷ In fact, the MTCP is a strategic bilateral program that offers cooperative capacity-building training to 59 non-NATO partner countries across the world. It is a key instrument in defence diplomacy for Canada's contribution to international peace and security.

Program managers indicated that the three main training programs offered by the MTCP, namely language, professional development, and peace support operations training, fill a critical need internationally.²⁸ Military personnel of non-NATO foreign partner countries must acquire the necessary English or French skills in order to meet their international peace support operations commitments. Program managers also felt that the skills taught through the Program enhanced the ability of participating nations to conduct peacekeeping operations and to be more interoperable with their allies. Only qualitative performance data was available to substantiate this opinion. As a key bilateral program utilised to support defence diplomacy activities, the MTCP also indirectly supports the DND role of “Contributing to international peace and security by projecting leadership abroad.”

2.2 Relevance – Alignment with Federal Roles and Responsibilities

Evaluation Question. Is the delivery of the MTCP consistent with the roles and responsibilities of the federal government and, more specifically, the roles and responsibilities of DND/CAF?

Key Finding 2. MTCP activities align with Federal Roles and Responsibilities.

To determine whether program activities align with the roles and responsibilities of the Government of Canada, and of DND in particular, a review of applicable federal legislation, departmental mandates, objectives and reports was conducted.

While the evaluation found that the review of federal legislation did not explicitly reference defence policy or diplomacy, these activities are seen to be an appropriate role for the Government of Canada to deliver its Whole-of-Government Framework and achieve its stated strategic outcome of “A safe and secure world through international engagement.”²⁹ The MTCP activities are aligned with the following definition of the above strategic outcome:

The involvement of DND in this role is also appropriate. As per the FY 2012/13 Departmental Plans and Priorities, the activity areas of MTCP align with one of the three key roles of DND:

2.3 Relevance – Alignment with Government Priorities 

Evaluation Question. Are the objectives of the MTCP consistent with current Government of Canada and DND/CAF priorities? 

Key Finding 3. The objectives of the MTCP align with current Government of Canada and DND Priorities.

As per the FY 2012/13 Departmental Plans and Priorities, the MTCP supports the Departmental strategic outcome: Defence operations improve peace, stability and security wherever deployed³¹ by enhancing defence diplomacy activity. As stated in the FY 2012/13 priorities:

The MTCP is the main tool used by the GES to develop and maintain relationships with developing, non-NATO countries. One of the objectives of MTCP is to build cooperative relationships through engagement with foreign militaries in order to support operational needs, military decision-making, and the advancement of Canadian agendas abroad by promoting the Canadian interests and values. As per the Departmental Performance Report FY 2010/11, Defence’s operations support the government’s priorities that contribute to nation building through assistance to select states through the provision of specialized military advice, training and assistance.

The MTCP also responds to the whole-of-government approach with its Military Assistance Steering Committee Working Group, formed by directors of DND, DFATD, Public Safety and CIDA. This Steering Committee, chaired by DGIS, defines the policy priorities for MTCP engagement.

2.4 Performance – Achievement of Expected Outcomes (Effectiveness)

A logic model was developed to determine the overall effectiveness of the DPD Program. The model grouped program activities by common outputs, and then linked those outputs to desired outcomes.

Program effectiveness was assessed by applying appropriate performance measures and/or key performance indicators against each outcome. Data for the performance measures was obtained from ADM(Pol) business plans, activity reports, a survey of CDA and LO, and numerous interviews with program managers and recipient stakeholders.

Accordingly, an assessment was made of the following outcome areas:

• establishment and maintenance of relationships with international military partners;
• contribution to capacity building of the non-NATO foreign forces program recipients; and
• support to the advancement of Canada’s international agenda.

Immediate Outcome 1. How effective is the MTCP approach with respect to establishing/strengthening relationships and partnerships?

Key Finding 4. The outcome “Relationships and Partnerships are established and strengthened” is being met.

To determine the effectiveness of MTCP in achieving this outcome, the evaluation conducted in-depth interviews with program managers and stakeholders (force-generators and force employers), and surveyed all CDA and pertinent LO. Furthermore, a review of MTCP annual assessments was performed to assess their activity levels and to determine if, indeed, they were successful in the establishment of partnerships with other militaries or governments. Although this outcome is mainly related to the activities found by MTCP resources, it should be noted that the Policy Group does play a strategic role in providing a departmental orientation, through the GES and the MASC.

Based upon these reviews, the evaluation concludes that the expectations for this area are generally being met. The activities under MTCP were seen to lead to establishing and strengthening relationships and partnerships. There also was significant evidence of building capacity of foreign military partners that can be attributed to the efforts of the MTCP personnel and training activities.

For example, the training exchanges and courses that are delivered through the MTCP were seen to build relationships with partner countries and create opportunities for Canada to partner and coordinate defence diplomacy activities. Specific examples of establishing and strengthening of relationships and partnerships are highlighted below.

Chile. As part of a partnership commitment agreed to during FY 2012/13, Canada, through the MTCP, has formed a partnership with both Jamaica and Chile to train Guatemalan pilots at the Caribbean Military Aviation School in Jamaica. Chilean instructors are currently assigned to the Caribbean Military Aviation School in Jamaica to provide pilot training to Guatemalan, Spanish-speaking students. As for the program managers, this trilateral partnership was made possible in large part due to the strong ties formed through sustained military cooperation engagement with Chile and Jamaica through the MTCP.

Brazil. In addition to participating in MTCP training, Brazil partnered with MTCP in the delivery of a Senior Officer (Colonel to General) Peace Support Operations course at the Escola Superior de Guerra [Brazilian Staff College] in Rio de Janeiro in 2011 and 2012. Over this two-year partnership, Brazil hosted nearly 50 senior-level (Lieutenant Colonel – Brigadier General) officers from across the Americas region on two consecutive serials of delivering this course in collaboration with the Canadian Defence Academy and the MTCP. Brazilian officers are currently working with this Academy’s curriculum development staff to develop an advanced English Language Training program for MTCP students.

Kenya. In FY 2006/07, MTCP enhanced its Africa programming with the posting of two MTCP-sponsored CAF officers to Kenya. The officers are integrated within the International Peace Support Training Centre in Nairobi, Kenya. The MTCP officers posted in Kenya directly contribute to the development of this Training Centre, and influence the enhancement of regional peace support operations capacity-building in Africa. In partnership with the Training Centre, MTCP officers deliver regional Peace Support Operations logistics training each year to approximately 40 students.

Intermediate Outcome. To what extent does the MTCP contribute to foreign military capacity building?

Key Finding 5. A review of program activities would suggest that they are increasing the ability of nations to participate in international operations, although there is no direct evidence that this has yet led to an actual increase in participation.

DMTC annual assessments,³³ stakeholder interviews,³⁴ and survey data³⁵ all would suggest that the MTCP is contributing to international peace-keeping capacity-building in other countries. This is believed to be due to a combination of the following three activities: language training, professional development and staff courses, and peace support operations training Whether this has actually translated into enhanced participation in international operations was not directly evident.

As per the annual assessments, the language training courses enhanced the ability of participants to communicate in English and/or French and, therefore, facilitated communication and interoperability among international forces. The professional development and staff courses improved participants’ understanding of democratic control over armed forces, enabled participants to govern their armed forces democratically, and should enhance their ability to function as efficient and professional staff officers within a strategic or operational environment. The peace support operations training also strives to develop the capacity of participating countries to undertake peace support operations.

It is difficult to tie these activities to increased international participation, as such participation is dependent upon numerous factors, most particularly political will and, for many participants, financial capacity. The program managers believe that the Program is very effective at building capacity and, as such, may have influenced various missions. Examples of this include:

Golan Heights. MTCP training in the areas of language and PSO training were considered critical in enabling Ukrainian soldiers to successfully replace the Canadian contingent deployed as part of the UN observation mission on the Golan Heights in 2006.

United Nations Mission in Darfur. The MTCP-sponsored Tactical Operations Staff Course was designed in FY 2006/07 to prepare African Union troops to deploy as part of the United Nations Mission in Darfur, an objective that was successfully achieved.

The Caribbean. The establishment of the Caribbean Military Aviation School, the Caribbean Military Maritime Training Centre, the Caribbean Counter Terrorism Training Centre, and the Caribbean Junior Command and Staff Course, in partnership with the Jamaican Defence Force in Jamaica are some examples of MTCP capacity building. Thanks to MTCP-sponsored special projects, Jamaica has become a regional centre of excellence in the areas of air force and maritime training, thus contributing to strengthening regional security and enhancing interoperability among Caribbean militaries. Jamaica’s longstanding partnership with Canada, through the MTCP, has consistently produced a direct strategic effect for Canadians. This was evident in 2009, when CanJet flight 918 was hijacked in Jamaica with more than 170 Canadians onboard. The Canadian-trained Jamaican Defence Force team successfully ended the hijacking without firing a shot.

In addition, as part of the international community’s response to the earthquake in Haiti, Jamaica deployed 150 JDF troops to the country to assist in humanitarian assistance and disaster response operations. On the invitation of the Government of Jamaica, Canada established an air bridge from Kingston, Jamaica, to Port-au-Prince, Haiti, moving personnel and supplies into the region via 28 dedicated flights. Jamaica also contributed two Canadian-trained Jamaican Defence Force military engineers to the Canadian mission in Afghanistan in 2009 and 2010. That Force’s engineers deployed with Canada under Op Athena in Kandahar. This was an excellent manifestation of the force multiplying effect that MTCP programming has for DND/CAF.

Ukraine. The transition of the Ukrainian Armed Forces to a fully professional military is directly supported through the capacity-building training sponsored by the MTCP. Canada supports many of the short-term objectives and ongoing efforts of the NATO-Ukraine Joint Working Group on Defence Reform. The language training (English Language and English Teacher Trainer) provided under the auspices of the MTCP contributed to the implementation of NATO standards in the Ukrainian Armed Forces. DMTC’s key role in the coordination of Partnership for Peace exercises such as MAPLE ARCH and RAPID TRIDENT enhances the operational capabilities of the Ukrainian Armed Forces and their ability to contribute to NATO-led operations.

Indonesia. In 2012, DMTC hosted the Canadian Defence Academy Peace Support Operations Seminar in Sentul, Indonesia. The 5-day seminar was conducted in partnership with the Indonesian Armed Forces, and had participation from Asia-Pacific MTCP member nations, including 15 Indonesians. The seminar, developed by the Canadian Defence Academy, allows DND/CAF to develop contacts with peace partners and enhance interoperability, all with a view to developing stronger collaboration with our peace allies.

No data, however, is being tracked to demonstrate that participant countries have increased their contributions to international military operations, or changed their military doctrines to support a more democratic approach. It is difficult to measure the long-term impact of the Program based solely on annual assessments because they only provide a snapshot of short-term results obtained upon course completion. To prove the effectiveness of the MTCP, mid-term and long-term impacts should be measured by surveying program participants five and ten years after course completion.

Intermediate Outcome 1. To what extent does the MTCP help to advance Canada’s global agenda?

Key Finding 6. Stakeholders and program participants feel that the MTCP helps to advance Canada’s global agenda by promoting its interests and values abroad.

Feedback from course participants, as reported in MTCP Annual Assessments, indicated that they were very satisfied with their exposure to Canada’s culture, values and system of governance.³⁶ They also agreed that the training improved their understanding of civil-democratic control of the CAF. In these annual assessments, the CDA strongly agreed that the MTCP projected a recognizable and independent Canadian profile within their region. They see the Program as an essential tool with which to promote Canada’s agenda abroad. They agreed that responses to MTCP offerings are always prompt and enthusiastic and believe that they are highly regarded by their counterparts. Foreign Relations Officers strongly agreed that MTCP graduates apply the knowledge that they have gained to their current jobs. A significant portion of the CDA and LO surveyed agreed that the MTCP did help advance Canada’s global agenda by promoting Canadian values such as democracy, the rule of law and human rights.³⁷

2.5 Performance – Demonstration of Efficiency and Economy

In order to analyse the efficiency of the Program, the evaluation team examined the resources used to produce outputs, including:

• the number of FTE within the MTCP;
• the cost per MTCP language training participant; and
• use of best practices and demonstration of efficiency measures.

The following data sources were used:

• DND financial systems;
• opinion of program managers and stakeholders through interviews; and
• literature review and media scan.

Evaluation Question. Are MTCP activities being delivered efficiently?

Key Finding 7. Over the period of the evaluation, the MTCP program has applied cost-saving measures while continuing to offer a high quality of services and level of training.

In lieu of reductions to its notional allocations (a reduction of 7.1 percent ($828,124) incurred over the period of the evaluation), the DMTC has reorganized and transformed the focus of its programming to include  increasing the implementation of partner engagement in the delivery of MTCP capacity-building training overseas. This transformation will enable DMTC to optimize the effectiveness and efficiency of its training programming within the current fiscal constraints and continue to build strong cooperative relationships with member countries.

Over the period covered by the evaluation, the cost per MTCP language training participant increased only slightly (from $20,670 to $21,247, or 2.8 percent over a 5-year period), despite inflationary pressures. According to MTCP annual assessments for FY 2009/10 and FY 2010/11, the cost of delivering language training at the CFLS is 5.6 percent higher than the cost of delivering similar training in a private school.³⁸ However, CFLS training includes a language curriculum that exposes MTCP students to Canadian democratic governance principles, CAF ethics and leadership. The benefits of the tailored curriculum offered by CFLS to MTCP language students provide substantive justification for the variance in this cost comparison and represents good value for money.

Table 18. Costs Related to the Delivery of the Language Training Portion of the MTCP, FY 2008/09 to FY 2011/12

Evaluation Question. Are the most appropriate means being used to achieve objectives, relative to alternative design and delivery approaches?

The evaluation found that the Program has taken further measures to contain costs and improve its delivery model, primarily in response to the 2010 Strategic Review and the DRAP that followed.

For example, the closure of CFLS Borden in FY 2012/13 reduced MTCP language training in Canada by 50 percent. The FY 2012/13 ADM(Pol) Business Plan and the MTCP Transitional Planning Horizon will offset this loss by transforming the delivery model to focus more on out-of-Canada capacity building. Namely, the MTCP implemented a “Train the Trainer” approach in which people from allied countries are taught how to provide training to their own militaries.

In FY 2013/14, DMTC began to reduce its in-Canada programming and formed a mobile training team of regular force officers to provide regional capacity-building training overseas, in partnership with member countries. The focus of this transition from in-Canada to overseas capacity-building is intended to increase training effectiveness through the establishment of multi-year training partnerships with member countries, as opposed to short term in-Canada training billets.

In another example, the MTCP has terminated in-Canada staff training programs at CFB Aldershot, as a result of a complete reduction to its reservist cohort. This will result in the termination of the DMTC/MTCP Junior Command and Staff Course in 2014.

The DMTC staff training model is being rebuilt around a core cadre of DMTC regular force and civilian staff who will be able to provide this training capacity at overseas locations provided by key partner allies.

DMTC also saw a reduction of $411,000 Vote 10, or in its grants and contributions funding, following the 2010 Strategic Review. As a result, the DMTC model for capacity building will rely on a greater proportion of support from our implementing partners, including host nation member countries and allied partners. As of FY 2013/14, MTCP training of foreign military officers in Canada began to decrease as DMTC transitioned its programming focus to the development and strengthening of regional training capacities within select MTCP member countries. In accordance with this Program transition, select MTCP member countries—Guatemala, Colombia, Kenya, Malaysia and Indonesia—have been identified as areas of focus for DMTC Peace Support Operations programming over the next three fiscal years.

___________________________________________________________________________________________________________________________

Footnote 1 National Defence and the Canadian Forces, CRS, CRS Evaluation Reports, 1997-2013, http://www.crs-csex.forces.gc.ca/reports-rapports/rp-ev-eng.aspx. Last Consultation: July 15, 2013.

Footnote 2 For a complete definition, please see: Andrew Cottey and Anthony Forster, Reshaping Defence Diplomacy: New Roles for Military Cooperation and Assistance, Adelphi Paper 365, Oxford, Oxford University Press, 2004, pages 6-9.

Footnote 3 DND, VCDS, Programme Assessment, Business Plan and Civilian Human Resources Plan, for FY 2007/2008 to FY 2012/2013.

Footnote 4 Data provided by DFL-2, 27 August 2013.

Footnote 5 DND, CDS, CDS Directive on CAF Force Posture and Readiness 2013, 28 June 2013, pages 8-10. Unclassified document.

Footnote 6 TBS Website, Descriptors for Government of Canada Outcomes Areas, http://www.tbs-sct.gc.ca/ppg-cpr/descript-eng.aspx#bm03. Last consultation: August 27, 2013.

Footnote 7 DND, Report on Plans and Priorities FY 2012/13. Part III – Estimates, http://www.tbs-sct.gc.ca/rpp/2012-2013/inst/dnd/dnd00-eng.asp. Last consultation: August 27, 2013.

Footnote 8 Justice Laws Website, National Defence Act (R.S.C. 1985, c. N-5), http://laws-lois.justice.gc.ca/eng/acts/N-5/index.html. Last consultation: August 27, 2013.

Footnote 9 Lee Berthiaume, “Military taking on increasing diplomatic role”, in The Star Phoenix, Saturday, June 8, 2013, http://www2.canada.com/saskatoonstarphoenix/news/story.html?id=aa2336f7-8d0a-45b8-b73d-1890025addb5. Last consultation: August 27, 2013.

Footnote 10 DND, Memorandum of Understanding Database, Defence Cooperation and Exchanges (Ministry Of Defence of Mongolia), 2012. Defence Cooperation (Peru), 2012. Defence and Military Relations and Strategic Cooperation (Israel), 2010.

Footnote 11 DND, Report on Plans and Priorities FY 2012/13. Part III – Estimates, http://www.tbs-sct.gc.ca/rpp/2012-2013/inst/dnd/dnd00-eng.asp. Last consultation: August 26, 2013.

Footnote 12 See section 1.2.2 Resources – Table 2: Full-time employees for Defence Policy and Diplomacy, FY 2008/09 to FY 2011/12 of the current evaluation report.

Footnote 13 DND, ADM(Pol), Programme Assessment and Business Plan FY 2012/13, Financial data extracted from the HRMS.

Footnote 14 ADM(Pol), Program Assessment and Business Plan, FY 2012/13, page 12. ADM(Pol), Program Assessment and Business Plan FY 2011/12, pages 12 and 21.

Footnote 15 They include: senior official/military personnel visits and exchanges; Royal Canadian Navy ship visits; Royal Canadian Air Force air demonstration teams; joint exercises; joint training programs, establishment of various treaties or Memoranda of Understanding; and the opening of CDA offices, which is seen as the highest expression of a military relationship with another country.

Footnote 16 DND, Defence Diplomacy Global Engagement Strategy, 2010.

Footnote 17 DND, ADM(Pol), Programme Assessment and Integrated Business Plan 2011/12, page 16.

Footnote 18 DND, FY 2011/12 Departmental Performance Report, http://www.forces.gc.ca/en/about-reports-pubs-departmental-performance/2012-table-contents.page. Last consultation: August 28, 2013.

Footnote 19 ADM(Pol), Programme Assessment and Integrated Business Plan 2011/12, page 16. CDA Annual Workshop 2012, CDA Afghanistan Presentation.

Footnote 20 DND, 2010-11 Departmental Performance Report, page 33. http://www.tbs-sct.gc.ca/dpr-rmr/2010-2011/index-eng.asp?acr=1877. Last consultation: August 28, 2013.

Footnote 21 DND, Military Training and Cooperation Program, MTCP Strategic Effect Document, August 2013, 3 pages.

Footnote 22 DND, 2011/12 Departmental Performance Report, http://www.forces.gc.ca/en/about-reports-pubs-departmental-performance/2012-table-contents.page. Last consultation: August 28, 2013.

Footnote 23 DFL reporting from CDA Italy-Op Mobile February-October 2011.

Footnote 24 National Defence and the Canadian Forces, CRS, CRS Evaluation Reports, 1997-2013, http://www.crs-csex.forces.gc.ca/reports-rapports/rp-ev-eng.aspx. Last Consultation: August 22, 2013.

Footnote 25 The new GES will provide a DND/CAF definition of this activity.

Footnote 26 Interviews with MTCP director, ADM(Pol) directors, SJS, CJOC.

Footnote 27 Survey. MTCP Annual Assessments.

Footnote 28 MTCP director, ADM(Pol) directors.

Footnote 29 Canada, TBS Website, Descriptors for Government of Canada Outcomes Areas, http://www.tbs-sct.gc.ca/ppg-cpr/descript-eng.aspx#bm03. Last consultation: August 27, 2013.

Footnote 30 DND, Report on Plans and Priorities 2012-13. Part III – Estimates, page 7, http://www.tbs-sct.gc.ca/rpp/2012-2013/inst/dnd/dnd00-eng.asp. Last consultation: August 27, 2013.

Footnote 31 DND, Report on Plans and Priorities 2012-13. Part III – Estimates, page 11, http://www.tbs-sct.gc.ca/rpp/2012-2013/inst/dnd/dnd00-eng.asp. Last consultation: August 27, 2013.

Footnote 32 Ibid. Part III – Estimates, page 28.

Footnote 33 DND, DMTC, Annual Assessment/Performance Measurement of the Directorate of Military Training & Cooperation (DMTC), FY 2008/09 to FY 2011/12.

Footnote 34 Program mangers included MTCP director and ADM(Pol) directors and DGs. Program recipient stakeholders included CJOC, SJS and VCDS.

Footnote 35 Survey sent to CDA and LO, June 2013.

Footnote 36 DND, DMTC, Annual Assessment/Performance Measurement of The Directorate of Military Training & Cooperation (DMTC), FY 2008/09 to FY 2011/12.

Footnote 37 Survey, June 2013.

Footnote 38 DND, DMTC, Annual Assessment/Performance Measurement of The Directorate of Military Training & Cooperation (DMTC), FY 2009/10 to FY 2011/12.

Footnote 39 The evaluation team allocated 45 percent of the total expenditures for language training in order to establish to cost per participant, based on an internal document and prior evaluation done by CRS in 2009.