Evaluation of the Military Training and Cooperation Program

ADM(Fin)
Assistant Deputy Minister (Finance)
ADM(Pol)
Assistant Deputy Minister (Policy)
ADM(RS)
Assistant Deputy Minister (Review Services)
CAF
Canadian Armed Forces
CDA
Canadian Defence Attaché
DG
Director General
DGIS Pol
Director General International Security Policy
DMTC
Director Military Training and Cooperation
DND
Department of National Defence
DTEP
Defence Team Establishment Plan
DXPC
Defence Executive Policy Committee
FTE 
Full-time Equivalent
GAC
Global Affairs Canada
GC
Government of Canada
GES
Global Engagement Strategy
MASC
Military Assistance Steering Committee
MTCP
Military Training and Cooperation Program
NATO
North Atlantic Treaty Organization
PMEC
Performance Measurement and Evaluation Committee
PSO
Peace Support Operation
SECB 
Strategic Engagement and Capacity Building
SJS
Strategic Joint Staff
SOP
Standard Operating Procedure
SSE 
Canada’s defence policy: Strong, Secure, Engaged
TB 
Treasury Board
UN 
United Nations

Key Findings and Recommendations

Key Findings Recommendations

Relevance

1. There is a continuing need for the activities offered by the MTCP.

 

2. The objectives of the MTCP are aligned with governmental and departmental priorities.

 

Performance – Effectiveness

3. Anecdotal evidence suggests that the MTCP training contributes to enhancing the skills and abilities of course participants; however there is a lack of a consistent follow-up mechanism to track MTCP graduates.

See recommendation #4

4. Anecdotal evidence strongly suggests that the MTCP is effective in expanding and reinforcing Canada’s bilateral relations with member countries.

 

5. Available evidence suggests that MTCP training promotes Canadian democratic principles among its course participants, and raises their awareness of gender, peace and security issues.

 

6. There is anecdotal evidence that the MTCP achieves regional influence in areas of strategic interest to Canada by establishing relations with select countries in these regions.

 

Performance – Efficiency

7. Overall, the MTCP membership structure is appropriate and efficient, and the program’s activities and country prioritization are aligned with Canada’s strategic priorities.

 

8. The MTCP lacks a graduation mechanism beyond joining NATO, and the program’s membership criteria require updating.

1. Revise the MTCP’s terms and conditions to ensure that the program is implemented as intended. This includes reviewing and clarifying the membership criteria and establishing a graduation mechanism.

9. In general, the MTCP governance structure and processes are clearly defined and efficient, and there is effective communication between DMTC and key program stakeholders and partners; however, there is a need to clarify the roles and responsibilities related to capacity building efforts within DND.

2. Clarify how the MTCP aligns with the broader capacity building initiative under SSE, and ensure the roles and responsibilities of DMTC and the Strategic Joint Staff (SJS) with respect to capacity building efforts are aligned to avoid overlap.

10. The MTCP activities have increased despite decreasing resources and staffing shortages over the evaluation period; however, anecdotal evidence suggests the MTCP continues to achieve its program outcomes.

3. Develop a strategy to address DMTC staffing challenges.

11. While the MTCP collects some performance data as part of its Annual Assessment, the program lacks a comprehensive performance measurement strategy, and requires improvements to its collection and management processes of performance and financial data to ensure program performance is measured and monitored consistently.

4. Develop a comprehensive performance measurement strategy, including revising program performance indicators and program logic model, and establish a consistent performance and financial data collection and management process to ensure the MTCP tracks spending and monitors progress towards its intended outcomes in a systematic manner.

Table Details - Key Findings and Recommendations


Back to Table of Contents

1.0 Introduction

1.1 Context for the Evaluation

In accordance with the coverage requirements outlined in the TB of Canada’s Policy on Results and the Financial Administration Act applicable to contribution programs, ADM(RS) has conducted an evaluation of the MTCP. This evaluation is a component of the DND/CAF five-year evaluation plan presented to the Performance Measurement and Evaluation Committee in December 2017, and approved in March 2018, and examines the relevance, effectiveness and efficiency of the activities of the MTCP during a five-year period from fiscal year (FY) 2013/14 to FY 2017/18. This evaluation did not assess the following: compliance with the Policy on Transfer Payments, Financial Administration Act, internal controls, financial authorities, expense eligibility, contracting and procurement authorities related to program delivery.

An evaluation of the MTCP was previously conducted by ADM(RS) as part of the 2013 Evaluation of Defence Policy and Diplomacy. It was preceded by two other evaluations: the Summative Evaluation of the Contribution Agreement with the Military Training Assistance Program (2009); and the Military Training Assistance Program Formative Evaluation (2008).

1.2 Program Profile

1.2.1 Program Description

The MTCP is a Grants and Contributions program. Its mandate is to enhance defence relations through cooperative, capacity building projects with non-NATO, developing countries of strategic interest to Canada. The MTCP is detailed in the DND/CAF Departmental Results Framework (DRF), under Core Responsibility 1: Operations, Engagement and Outreach – Program 1.4 Global Engagement (corresponding Program Activity Architecture line: 1.3.4).

The MTCP’s main objectives are to:

The program is a key enabler of DND/CAF’s Global Engagement Strategy (GES), which provides guidance on international defence diplomacy activities and promotes prioritization and coherence in resource allocation.Footnote 1  The MTCP’s activities support the GES by building capacitiesFootnote 2  of member countries. The MTCP is administered by DMTC which reports through the Director General International Security Policy (DGIS Pol) to ADM(Pol). An interdepartmental committee, the Military Assistance Steering Committee (MASC), oversees the program. The MASC is composed of officials from DND and Global Affairs Canada (GAC) and is the decision-making body responsible for reviewing the MTCP’s membership, and recommending adjustments of funding priorities to ensure alignment with current needs and strategic interests. The other standing committee that makes decisions impacting aspects of the MTCP is the Professional Military Education International Allocation Board (chaired by Military Personnel Generation) that produces, among other things, the annual international invitation list for senior staff courses offered by Canadian Forces College Toronto.

Canadian Defence Attachés (CDA), the Canadian Forces' official representatives abroad, are instrumental to the implementation of MTCP activities. CDAs maintain close relationships with member countries, and are often the main point of contact for the MTCP. CDAs enable communication about training requests and course offers, and provide logistical support when necessary. The MTCP is delivered by CAF personnel, as well as DND civilians, and in limited cases, by defence contractors both in Canada and abroad. The MTCP runs in-Canada courses, out-of-Canada courses, exercises, and expert team and delegation visits. Training outside of Canada is sponsored by the MTCP but hosted and delivered in partnership with implementing and/or partner member countries.

The following three pillars of training are conducted in Canada and abroad:

In addition to its three pillars of training, the MTCP also undertakes special projects such as sponsoring expert team visits to provide Canadian expertise in another member country and delegation visits from foreign member countries to Canada to learn about the Canadian system.

The MTCP member countries are developing non-NATO countries that meet the program’s membership criteria as assessed by the Government of Canada (GC) through the MASC. The MTCP accepts new member countries that:

1.2.2 Program Resources

Table 1 illustrates the MTCP expenditures over the course of the evaluated period. In FY 2017/18, MTCP expenditures were reported as $14.58 million. The expenditures shown for the MTCP include spending that went towards activities related to the outcomes that were not included within the scope of this evaluation.

Table 1. Military Training and Cooperation Program expenditures, FY 2013/14 to FY 2017/18.

  FY 2013/14 FY 2014/15 FY 2015/16 FY 2016/17Footnote 4  FY 2017/18

Salaries and Operating Expenditures ($M)

5.1

3.7

3.8

3.9

4.1

Grants and Contributions Funds ($M)

10.1

9.7

10.4

10.8

10.5

Total ($M)

15.1

13.4

14.1

14.8

14.6

Table 1. Military Training and Cooperation Program expenditures, FY 2013/14 to FY 2017/18.Footnote 5  This table displays the salaries and operating expenditures, Grants and Contributions funds and total expenditures for the MTCP for FYs 2013/14 to 2017/18.

Table 1 Details - Military Training and Cooperation Program expenditures, FY 2013/14 to FY 2017/18.

1.2.3 Stakeholders

The MTCP works with a number of partners and stakeholders to deliver training and capacity building. A list of principal stakeholders is presented in the following table:

Table 2. List of stakeholders.

Organizations Stakeholders

Organizations internal to DND/CAF

  • ADM(Pol)
    • DGIS Pol
      • DMTC
      • DGIS Pol Geographic Desk
  • Vice Chief of the Defence Staff
    • Directorate of Foreign Liaison
      • Canada’s network of Defence Attachés
  • SJS
  • Canadian Defence Academy
    • Royal Military College
    • Canadian Forces Language School
  • Canadian Army Doctrine and Training Centre
    • Canadian Army Command and Staff College
    • Peace Support Training Centre
    • Combat Training Centre
  • Other organizations as required (provision of instructors and positions on activities)

Other government departments

  • GAC

Other organizations

  • CALIAN and other contractors
  • Foreign militaries
    • Training participants and instructors
  • Government representatives of member countries
  • Multilateral organizations (e.g., the United Nations (UN))

Table 2. List of stakeholders. This table lists some of the key internal and external stakeholders for the Military Training and Cooperation Program.

Table 2 Details - List of stakeholders.

1.3 Evaluation Coverage

1.3.1 Evaluation Scope

The evaluation covers the MTCP, which falls within program 1.4 - Global Engagement – in Core Responsibility 1 (Operations, Engagement and Outreach) of the Program Inventory in the Departmental Results Framework.

The evaluation assessed the relevance, effectiveness and efficiency of the MTCP, and addressed the extent to which the MTCP’s objectives have been achieved during the period of observation (FY 2013/14 to FY 2017/18). The evaluation’s scope focused on the extent that the MTCP contributes to the short-term and medium-term outcomes outlined in the program logic model ( Annex C).

The short-term outcome, “developing capacity of member countries to undertake peace support operations,” was excluded from the scope after consultation with program staff indicated it would be difficult to assess directly; however, aspects were assessed through other outcomes. The remaining short-term outcomes were assessed as a combined outcome, “contribution to enhancing the abilities of military trainees.” The long-term outcome, “contributing to international peace and security,” was also excluded as it is strategic in nature and it would be difficult to show the contribution made by the MTCP specifically, given the myriad global factors outside the program’s scope and control.

1.3.2 Issues and Questions

An evaluation matrix listing each of the evaluation questions, with associated indicators and data sources, is provided in Annex D. The methodology used to gather evidence in support of the evaluation questions can be found in Annex B. Refer to Annex B for information related to the analysis of interview data.


Back to Table of Contents

2.0 Findings and Recommendations

2.1 Relevance

In assessing relevance, the evaluation examined the extent to which the MTCP addressed a demonstrable and continued need, and was aligned with GC priorities and DND/CAF strategic outcomes. The discussion of relevance is based on evidence from document review and interviews.

2.1.1 Relevance - Continued Need

Key Finding 1: There is a continuing need for the activities offered by the MTCP.

The MTCP is a key component of Canada’s defence diplomacy that is designed to enable, improve and enhance defence relations with nations around the world. Interviewees indicated that the MTCP is one of the main instruments available to DND/CAF to establish relationships and goodwill with a wide range of non-NATO countries, providing opportunities for ongoing communication and opening doors to future engagement. Most CDAs stated that the MTCP is a highly visible defence diplomacy tool, and is frequently recognized at the highest levels of governments and militaries as an example of defence cooperation. This cooperation is intended to strengthen Canada’s strategic security relationships and address common security concerns.

All interviewees, both internal and external to the program, felt that there is a continued need for the activities offered by the MTCP. Without the MTCP, there would be a gap in Canada’s ability to expand and reinforce defence relations. All program managers noted that the MTCP is often the primary or in some cases the only military-to-military diplomacy tool available to the network of CDAs abroad, particularly in countries where there is no other regularized defence engagement. For example, it is the primary vehicle for military professionalization and capacity building in the Americas. The MTCP enables communication with militaries in member countries, and provides opportunities for open dialogue through the professional relationships developed with the MTCP graduates.

The evaluation found little to no duplication of effort with other federal government programs. Document review and interview data indicated that the MTCP is unique within DND. One CDA noted that other DND/CAF activities across the world, such as participation in joint exercises, are more sporadic in nature and can leave significant time gaps in interactions while the MTCP provides a continuing line of engagement. Interviewees reported that there are a few complementary programs offered by GAC, as well as international engagement funds administered by other federal government departments and organizations. These programs are complementary to the MTCP but have different mandates. For example, while the following three programs within GAC complement the work of the MTCP, their focus is different: the Anti-Crime Capacity Building Program works to improve the capacity of beneficiary states, government entities and international organizations to deal with transnational crime; the Counter-Terrorism Capacity Building Program provides assistance to foreign states to enable them to prevent and respond to terrorist activity; and the Peace and Stabilization Operations Program enables the GC to take rapid and coordinated action to prevent and respond to emerging and ongoing instances of violent conflict and state fragility.

Internationally, other like-minded nations conduct similar programs which may sometimes overlap with the activities offered by the MTCP. The United States, the United Kingdom, France and Australia offer similar activities, such as short and long-term courses to build military professionalism, language training, military-to-military exchanges, and courses on topics related to human rights, democratic values and civil control over the military.Footnote 6  Nonetheless, interviewees stated that, given a high demand for these training activities internationally, overlap or duplication between the MTCP and allies’ programming is minimal. Document review and interview data indicated that DMTC makes efforts to coordinate with allies providing similar training to non-NATO states to ensure minimum duplication of effort, and seeks to fill a critical need internationally by offering niche courses and training.

2.1.2 Relevance - Alignment with Federal Government Priorities

Key Finding 2: The objectives of the MTCP are aligned with governmental and departmental priorities.

The MTCP is aligned with GC and DND priorities. The MTCP directly supports several objectives outlined in SSE, specifically Mission Five: to “engage in capacity building to support the security of other nations and their ability to contribute to security abroad.” The SSE states that “cooperative defence relationships also support the advancement and promotion of broader government priorities.”Footnote 7  The MTCP is a defence diplomacy tool with the objective to expand and reinforce bilateral defence relations which contribute to broader diplomacy and relations. SSE also outlines the need for Defence to pursue relationships with countries other than our traditional allies “including those facing instability and those whose policies and perspectives are not aligned with Canada’s” in order to “build mutual understanding, enhance transparency, resolve differences, demonstrate leadership and promote Canadian values.”Footnote 8  The MTCP interacts with up to 67Footnote 9  diverse member countries throughout the world, including countries facing instability, emerging democracies, and countries with different styles of government with the goal of establishing bilateral relations and promoting Canadian values abroad, such as the rule of law, human rights and democratic principles.

The MTCP is also aligned with other priorities and commitments around international engagement and capacity building outlined in various government documents, including Ministerial mandate letters, federal budgets, the Whole-of-Government Framework, and other government initiatives, such as Canada’s National Action Plan on Women, Peace and Security 2017-2022. These priorities include:

2.2 Performance—Achievement of Expected Outcomes (Effectiveness)

2.2.1 Contributing to enhancing the abilities of foreign military trainees

Key Finding 3: Anecdotal evidence suggests that the MTCP training contributes to enhancing the skills and abilities of course participants; however there is a lack of a consistent follow-up mechanism to track MTCP graduates.

Analysis of DMTC Annual AssessmentsFootnote 10  and program data, along with stakeholder interviews, suggests that MTCP training contributes to enhancing the skills and abilities of course participants; however no consistent participant tracking data was available. The absence of an alumni tracking mechanism makes it difficult to assess whether MTCP graduates apply the knowledge they gain in their current job and their career progression. A continuing increase in participants, and demand for MTCP courses combined with positive feedback from participants and CDAs indicate that MTCP training is effective at enhancing the abilities of trainees.

During the five-year evaluation period (FY 2013/14 – FY 2017/18), the MTCP has seen an increase of 29 percent in course participants, and a 50 percent increase during the past 10 years (since FY 2007/08), with no corresponding increase in resources. CDAs interviewed for this evaluation were unanimous that there is high demand for MTCP training, and observed that the demand is much higher than the program’s supply of training opportunities. DMTC staff reported that they receive high volumes of training requests. In preparation for the 2018/19 MASC, DMTC compiled a list of over 500 unique training requests from 54 countries.

According to DMTC Annual Assessments, MTCP course participants are satisfied with the courses, and feel that MTCP training meets their needs and will improve their job performance. In addition, GAC Foreign Relations Officers believe that MTCP graduates apply the knowledge they have gained in their current jobs.Footnote 11  In interviews, manyFootnote 12  CDAs said that in most cases MTCP graduates use the skills they learned through participation in the program. The CDAs, document review and case studies provided several examples of instances where graduates were applying the skills acquired at MTCP training. For example:

Although the anecdotal evidence suggests graduates are applying the knowledge, interviews with various stakeholder groups found that it was virtually impossible to determine if graduates are, in fact, doing so without an alumni tracking process. While the MTCP collects data on course attendance, the program does not systematically track its participants and graduates and their progress over time, thereby rendering the assessment of the MTCP’s impact on its graduates’ careers difficult. Some interviewees noted that, to be able to measure its success in a meaningful way, the MTCP would need to develop a rigorous tracking mechanism to better trace program graduates. A few interviewees argued that an alumni tracking tool could also be used to share program updates and maintain interest for the program, and help the MTCP maintain relations with its graduates and leverage these contacts. However, according to DMTC staff, the substantial amount of staff effort and time that would be required to develop, maintain and leverage such a database is beyond the ability of DMTC to achieve given current human resources.

2.2.2 Expanding and reinforcing Canada’s bilateral defence relations

Key Finding 4: Anecdotal evidence strongly suggests that the MTCP is effective in expanding and reinforcing Canada’s bilateral relations with member countries.

Anecdotal evidence strongly suggests the MTCP expands and reinforces bilateral relations with member countries. There are no quantifiable performance measures to track the progress of the MTCP’s contribution to improving bilateral defence relations. However, document review, case studies and interviews strongly suggest the MTCP is effective at establishing and maintaining relations with member countries. In interviews, stakeholders reported that the MTCP is a highly effective tool in expanding and strengthening bilateral relations with member countries, and that this is the outcome of most value to DND/CAF and Canada. The program has high visibility and establishes dialogue and goodwill with foreign countries. According to annual assessments, CDAs believe the MTCP has improved Canada’s defence relationship with member countries. Many CDAs interviewed agreed that the MTCP is key to establishing individual professional relationships. CDAs cited several examples of senior officers in foreign militaries who refer to their past participation on MTCP training, and offer their support and act as points of contact. The MTCP also acts as an important tool to establish and strengthen relationships between CDAs and GAC personnel and their counterparts in member countries through the planning, coordinating and delivery of training activities.

Many interviewees, across stakeholder groups, stated that one of the most valuable aspects of the MTCP is its ability to “keep the door open and the lights on.” In other words, the MTCP establishes and maintains relations with countries where the GC has little to no other involvement, thus facilitating future engagement should the need or opportunity arise. For example, according to case studies, the MTCP engagement in Ukraine was seen to facilitate the launching of Operation UNIFIER, as the CAF already had a presence and had built strong relationships. In Mali, the MTCP was the only program operated by Canada that engaged the military and established goodwill and contacts which are perceived to have expanded Canada’s role. A few stakeholders stressed the point that the relationships built through the MTCP go beyond defence to include broader diplomacy and even trade. For example, case studies found that the foundation laid by the MTCP in Jordan enabled GAC to deliver the Build Partner Capacity Program there; and in Vietnam, the MTCP is the only program that engages the military community and provides access to the senior decision makers in government. Similarly, in Colombia, the MTCP is the only instrument Canada has to work with the military, which is a key institution in the country.

Another aspect of defence relations is interoperability, or the ability of two or more militaries to operate effectively together. It is a challenge to assess the contribution of the MTCP to improving interoperability and bilateral relations with member countries as no performance data exists, and the very nature of the outcome makes it difficult to quantify and track progress. Particularly with regard to interoperability, many interviewees felt the MTCP contributes to improvement; for example, language training is essential to interoperability, as it facilitates communication between militaries. Interviewees noted, however, that it is difficult to measure improvement and demonstrate the contribution made by the MTCP among other factors. Furthermore, they stated that the impact on overall capabilities is limited due to the fact that the MTCP trains a relatively small percentage of member countries’ militaries.

Case studies found some examples of improvement in interoperability:

2.2.3 Promoting Canadian democratic principles, the rule of law and human rights in the international arena

Key Finding 5: Available evidence suggests that MTCP training promotes Canadian democratic principles among its course participants, and raises their awareness of gender, peace and security issues.

Evidence from DMTC Annual Assessments, stakeholder interviews, document review and case studies suggests that the MTCP contributes to promoting Canadian democratic principles and values among course participants from member countries. There was little specific performance data available to establish whether or not the program’s activities have actually translated into increased awareness and appreciation of democratic principles, the rule of law and human rights in the international arena. In the absence of an alumni tracking mechanism (see Key Finding 3) and more specific performance indicators, it is difficult to track progress on this outcome and to demonstrate the MTCP’s contribution.

Document review and interview data indicated that Canadian democratic principles, values and ethics are embedded in the MTCP course curricula. CDAs reported that foreign military trainees are exposed to Canadian values and principles through their participation in MTCP training activities. According to the DMTC Annual Assessments, MTCP course participants are satisfied with the exposure the courses gave them to the Canadian culture, values and system of governance. Interviewees also noted that in-Canada training in particular has been useful in promoting these values, as participants are immersed in the Canadian culture and become more familiar with Canadian values and code of conduct. In this way, the MTCP positively influences senior officers within foreign militaries, and, in turn, corporate culture and senior level decision making, contributing to the efforts to promote democratic principles in the international arena. For example, case study evidence indicates that there has been a significant progress made in Jordan: the inclusion of more women from the Jordanian Armed Forces on MTCP courses and their exposure to Canadian values has promoted Jordan’s efforts to improve human rights.

Interviewees reported that gender integration, equality and military ethics training are examples of particularly valuable contributions of the MTCP. They noted that topics such as gender, peace and security, and empowerment of women are currently one of the key focus issues for the GC. Documentary evidence indicated that the MTCP has been placing emphasis on advancing the gender agenda, and gender-sensitive topics are formally integrated into some of MTCP’s activities. This directly aligns with the Implementation Plan for Canada’s National Action Plan on Women, Peace and Security 2017-2022, which outlines a whole-of-government approach to implementing gender agenda. Activities such as the MTCP Women, Peace and Security workshopFootnote 15  and symposium are specifically designed to address the women, peace, and security agenda and improve participants’ understanding of the impacts and benefits associated with the increased presence and role of women in military organizations.Footnote 16  This activity was developed in response to demands from MTCP member countries for professional development courses to address changes in the cultural environment of armed forces around the world, and to foster an expanded role for women in military organizations. Other MTCP training courses that incorporate gender perspectives include the Ethics in the Military Profession Workshop and Symposium, Chaplain Development Course, Joint Command and Staff Program, National Security Program, Canadian Security Studies Program, International Officer Training Program, UN Staff Officer Course, UN Logistics Officer Course, and Civil-Military Relations Courses.

In FY 2016/17, the MTCP delivered three iterations of its Women, Peace and Security workshop in Jordan, Colombia and Côte d’Ivoire, reaching an audience of approximately 130 students.Footnote 17  To facilitate further discussion, these workshops were followed by concise symposia on the same theme that reached approximately 2000 students at the officer cadet level.Footnote 18  While these symposia are attended by thousands of participants, they do not incur any additional costs to DMTC, beyond the cost of keeping course instructors abroad for a few extra days, enabling DMTC to expand its reach in a cost-effective manner.Footnote 19 

As part of the commitment to promoting increased awareness of gender, peace and security issues, the MTCP actively encourages the participation of women from national security forces on its training activities. As of FY 2014/15, the MTCP began systematically reporting on female participation in MTCP-sponsored activities conducted in Canada and abroad. In FY 2016/17, over 130 female participants took part in MTCP-sponsored trainingFootnote 20  – a 63 percent increase over the 80 female participants trained in FY 2014/15. Anecdotal evidence from document review, case studies and interviews indicates that gender-focused courses provide female participants from various countries an opportunity to engage and share their experiences with each other, raising awareness of the gender issues in the militaries and promoting an expanded role of women in military organizations.

2.2.4 Achieving influence in areas of strategic interest to Canada

Key Finding 6: There is anecdotal evidence that the MTCP achieves regional influence in areas of strategic interest to Canada by establishing relations with select countries in these regions.

This outcome is closely related to the previous outcomes as they are the means by which Canadian influence is achieved. Analysis of stakeholder interviews, document review and program data suggests the MTCP achieves influence in areas of strategic interest to Canada. As with previous outcomes, there are no specific performance measures available to assess whether the MTCP’s activities lead to actual influence in specific regions. The MTCP is an enabling tool that is intended to create connections in order to extend defence diplomacy relations and Canada’s influence, though much of what occurs after the connections are established is beyond the control of the program.

Program data and stakeholder interviews indicate that MTCP activities are targeted at countries and regions of strategic interest to Canada. As outlined briefly in the program description, the MASC is responsible for reviewing the MTCP’s membership and recommending adjustments of priorities to ensure alignment with strategic interests. The MASC adjusts the MTCP’s engagement with each region and member country on a regular (normally annual) basis to reflect current strategic aims.Footnote 21  The four regions are Africa and the Middle East, Asia-Pacific, Europe, and the Americas, and are each allocated a percentage of MTCP training. The countries within each region are subdivided into category A and B, where the countries in category A are considered of “strong strategic and operational relevance”Footnote 22  and are to receive more course and funding allocations. The MTCP’s annual target is for 85 percent of training participants to be “within the governmental priority regions” (i.e., category A). As shown in Table 3, the target was achieved in FY 2014/15 and almost achieved in the other 3 years for which data were available.Footnote 23  In years where the target was not met it was due to re-scheduling, re-location and cancellation of planned courses.

Table 3. Percentage of MTCP participants trained in priority regions FY 2013/14 to FY 2016/17.
  FY 2013/14 FY 2014/15 FY 2015/16 FY 2016/17

Foreign candidates trained within the governmental priority regions

82%

89.5%

84%

83%

Table 3. Percentage of MTCP participants trained in priority regions FY 2013/14 to FY 2016/17. This table displays the percentage of foreign candidates trained in the governmental priority regions for FYs 2013/14 to 2016/17.

Table 3 Details - Percentage of MTCP participants trained in priority regions FY 2013/14 to FY 2016/17.

Similarly, the MTCP strives to achieve 85 percent spending on priority regions. DMTC tracked this data from FY 2012/13 to FY 2015/16, when they discontinued collecting the data. However, DMTC is currently in the process of reactivating the collection of this data. As shown in Table 4, overall, the program met, or almost met, the 85 percent target, with Africa and the Middle East being an area where the target was consistently not achieved. However, the MASC has approved the realignment of regional prioritizations for FY 2019/20, including an increase in priority for Africa and the Middle East, and this may better align MTCP spending with priority regions. With the reactivation of the database to track regional spending, DMTC will be able to monitor the progress and adapt as necessary.

Table 4. Percentage of MTCP spending in priority countries by region, FY 2012/13 to FY 2015/16.
  FY 12/13 FY 13/14 FY 14/15 FY 15/16 Average

Africa/Middle East

73

76

66

65

70

Asia-Pacific

74

94

93

91

88

Europe

92

86

98

84

90

Americas

84

87

92

86

87

Overall

80

87

88

83

85

Table 4. Percentage of MTCP spending in priority countries by region, FY 2012/13 to FY 2015/16. This table displays the percentage of spending in priority countries in Africa and the Middle East, Asia-Pacific, Europe and the Americas, for FYs 2012/13 to 2015/16.

Table 4 Details - Percentage of MTCP spending in priority countries by region, FY 2012/13 to FY 2015/16.

Most stakeholders interviewed for the evaluation agreed that the countries targeted by the MTCP are aligned with strategic priorities. Also, many agreed that the MASC and DMTC are responsive to changing priorities and recommendations. For example, the Africa and the Middle East region will increase in priority for FY 2019/20 to reflect the “high number of members in this region and the substantial contribution of these countries to PSOs, as well as the GC’s appetite to support peace and stabilization efforts in the region.”Footnote 24  GAC representatives interviewed indicated this reprioritization was aligned with GAC priorities.

Although the extent of influence in strategically relevant countries cannot be measured in a quantifiable way to track progress, anecdotal evidence from documents, interviews and case studies indicates that the MTCP does in fact achieve influence. One approach of the MTCP is to establish strong relations with strategic partner countries in order to achieve broader influence in the region. One example of this engagement strategy is in Jamaica, which has been a key partner in building Canada’s relations with other Caribbean nations by providing a central location for a variety of training programs.Footnote 25  Jamaica has been a member of the MTCP since 1965 and has established a strong relationship with Canada which has led to many opportunities for engagement and cooperation. For example, through an arrangement signed in 2012 the Jamaican Operational Support Hub can provide support to CAF personnel, as it did during Exercise TRADEWINDS in 2016 in Jamaica. The MTCP has built on this strong relationship by assisting the Jamaican Defence Force in establishing or upgrading several Centres of Excellence in Jamaica which are leveraged to train personnel from other MTCP countries in the region, and beyond.Footnote 26  The Centres of Excellence “represent a lasting, positive legacy and unique resource for DND/CAF in the region” and the MTCP continues to leverage and support them to achieve influence throughout the region.Footnote 27  The MTCP uses a similar approach in Ukraine, which is a regional training hub that brings participants from across the region to partake in MTCP courses.

Evidence from interviews and case studies revealed that the MTCP is also effective as a tool to achieve influence in countries with emerging democracies, or that have historically not been as open to Western influence. For example, interviews found that the MTCP is crucial to maintain a Canadian presence and influence in the young democracies of Georgia and Tunisia where the MTCP is almost the only defence engagement by Canada. Both countries are strategically located, as Georgia is positioned on NATO’s eastern flank and faces heavy influence from Russia, while Tunisia is located between Libya and Algeria and important to maintain stability in the region. The MTCP also contributes to achieving influence in Vietnam and Mongolia, two countries that are trying to expand their engagement with the West, as traditionally, they have been heavily engaged with Russia and China. Mongolia refers to the expanded engagement as their “third neighbour policy.”Footnote 28 

2.3 Performance—Demonstration of Efficiency

The evaluation assessed whether the most appropriate and efficient means are being used to achieve the expected outcomes of the program.

2.3.1 Program Design – MTCP membership structure and criteria

Key Finding 7: Overall, the MTCP membership structure is appropriate and efficient, and the program’s activities and country prioritization are aligned with Canada’s strategic priorities.

Based on the document review and interviews with program stakeholders, the evaluation concludes that the overall MTCP membership structure is appropriate and efficient in relation to the program’s objectives. Most interviewees reported that the MTCP conducts its activities in the right locations around the world, and mechanisms are in place to ensure that program activities are aligned with Canada’s strategic priorities.Most senior managers within ADM(Pol) indicated that the program targets the right countries in its four regions, and the course allocation is policy driven. Additionally, senior managers felt DMTC makes efforts to ensure the membership structure provides a greater output for Canada by prioritizing countries based on current policy considerations, including the SSE, GES, foreign policies, PSO-related Memoranda to Cabinet, UN Security Council bid campaign, UN Peacekeeping Defence Ministerial, and Elsie Initiative for Women in Peace Operations.

Interview data suggests that the MTCP country prioritization is well managed through the MASC process. Most CDAs were satisfied with the oversight from ADM(Pol) and its regional policy directorates and GAC. Document review indicated that there was an issue with the flexibility allowed by the program’s membership structure (i.e., the uneven number of member states in each category and region combined with a fixed regional share of the training budget) as reported by the CDA network and the DGIS Pol geographic desks.Footnote 29  However, there is evidence that DMTC has been making efforts to improve its processes. For example, in preparation for the annual MASC deliberations, DMTC Policy section reviews the MTCP membership architecture with the aim of increasing the program’s responsiveness and flexibility.Footnote 30  As part of the 2014 review of the MTCP, DMTC convened three working group meetings to discuss the MTCP policy priorities, regional training requirements and program delivery for FY 2016/17.Footnote 31 

Overall, many interviewees felt that, in more recent years, the MTCP has been flexible in responding to changing political and other realities globally. External interviewees felt that DMTC and the MASC were fairly flexible in responding to changing global conditions. They noted that the recent reprioritization of the regions to provide higher priority to Africa and the Middle East was reflective of Canada’s foreign policy priorities. Most CDAs were generally satisfied with the current membership structure, although a few indicated that the program could be more flexible in responding to changes in countries’ capacity as well as political and governmental events. However, some interviewees, both internal and external to the program, emphasized that the need to adjust the MTCP priorities can be challenging to address in a timely manner in the context of constantly and rapidly changing government priorities and global geo-political environment, given that most activities must be planned many months out for logistic reasons.

Key Finding 8: The MTCP lacks a graduation mechanism beyond joining NATO, and the program’s membership criteria require updating.

While most interviewees felt that the overall MTCP membership structure is appropriate, a few issues were identified in relation to the MTCP’s terms and conditions, particularly concerning the exit mechanism from the program and the membership criteria. The concerns were related to the lack of a graduation mechanism from the program beyond joining NATO, and the need to clarify and streamline the membership criteria, including the need to potentially re-assess the PSO criterion and revise the GDP threshold.

Many stakeholders interviewed reported that the lack of a graduation mechanism from the MTCP is a key area of concern affecting program efficiency. Joining NATO is the only “true” graduation mechanism from the MTCP. External stakeholders indicated that the lack of graduation mechanism outside of joining NATO is a challenge, as many current MTCP members are not likely to join NATO. Additionally, a few interviewees suggested that a more flexible transition could be established when members join NATO to ensure that country member status is not changed prematurely. The interviewees noted that, once a country enters NATO, it becomes ineligible for MTCP training which may undermine the relations established through years of MTCP membership.

Currently, no exit mechanism exists for countries that no longer meet the membership criteria or are no longer a strategic priority to Canada, or those where a broader defence relationship has been established. A few senior managers within ADM(Pol) and program mangers indicated that continued engagement with countries that are no longer of strategic relevance to Canada, or that no longer need the same type of engagement, takes funds away from other countries where the program could be more valuable. The interviewees felt that there should be a different mechanism to maintain engagement with countries once they have achieved a certain level of capacity.

DMTC does not receive more resources when new member countries are added.Footnote 32  As the MTCP lacks a graduation or incremental funding mechanism, a zero-sum membership policy is maintained, whereby the addition of new members and increases in priority for some countries leads to decreases in priority for others.Footnote 33  At the same time, the number of members has grown but the MTCP budget and personnel resources have not increased since 2009. During the five-year evaluation period, the number of active members has increased by 13 percent from 47 countries in FY 2013/14 to 54 countries in FY 2017/18; over the past ten-year period, the number of members increased by 23 percent.Footnote 34  As a result, MTCP funding is spread thinly between the program’s members.

Some interviewees also indicated that the MTCP membership criteria need to be revised. A few interviewees within ADM(Pol) and DMTC said that the membership criteria are outdated and could be clarified and streamlined to better reflect current needs. Program managers reported that the terms and conditions are vague, and it is not always clear which criteria or their combination are mandatory in order to be considered for MTCP membership. There is also some possible duplication between the criteria (e.g., meet foreign and defence policy strategic interests, meet CAF operational goals and interests, demonstrate political/military relevance to Canada) and a lack of clear indicators for assessing the criteria.

A few senior managers, internal managers and the interviewees from GAC reported that there is a potential need to re-assess the relevance of the PSO criterion. One program manager commented that the MTCP membership should not be limited to countries that contribute to PSOs, and a senior manager from ADM(Pol) stated that this criterion may not be realistic in absence of a graduation mechanism as troop contributions for any country can vary significantly, and “it is unclear what strategic value this requirement serves.” DMTC staff commented that, in the absence of a graduation mechanism, some long-standing member countries have not historically contributed to PSOs or do not currently do so, and as the MTCP fills not only a capacity building but also a long-term relationship-building and defence diplomacy role, it has been challenging to realign activities based on the number of UN troops contributed. Interviewees from GAC also reported that this criterion is challenging: “it is currently being interpreted broadly which works, but this criterion can be limiting.”

Evidence suggests that there is a potential need to revise the current requirement that countries should have a per capita GDP of US$15,000 or less per year at the time of consideration for MTCP membership. A few interviewees indicated that the current maximum of US$15,000 has not been reviewed since 2009 and should be revised to reflect inflation. DMTC staff explained that some countries of potential strategic interest may have a per capita GDP that exceeds this limit. Additionally, some existing MTCP members have current GDP levels that are higher than they were at the time of admission to the program. The MTCP manages the variation in GDP among its members through a two-tiered structure, whereby existing members are categorized according to the level of financial assistance they require, and countries with a per capita GDP in excess of US$15,000 must pay for their own expenses.Footnote 35  Senior managers commented that the current GDP cap is low, and the threshold could be revised for increased flexibility to accommodate different countries.

Document review and interview data indicated that DMTC intends to launch a comprehensive review in the near future, to enable the MTCP to function effectively and meet evolving strategic needs.Footnote 36  The issues outlined in this section could be potentially addressed as part of this renewal process, which DMTC should make a priority.

In addition to the MTCP evaluation, ADM(RS) conducted an independent investigation of MTCP’s terms and conditions, and the associated eligibility criteria and expenditure authorities, and recommended that the terms and conditions should be updated/renewed to ensure delegated expenditure and contracting authorities are clearly stipulated to avoid misinterpretation, and to ensure all MTCP expenditures conform to eligibility criteria. All the previously-noted issues were outside the scope of the MTCP evaluation.

ADM(RS) Recommendation

2.3.2 Program Governance

Key Finding 9: In general, the MTCP governance structure and processes are clearly defined and efficient, and there is effective communication between DMTC and key program stakeholders and partners; however, there is a need to clarify the roles and responsibilities related to capacity building efforts within DND.

Governance Structure

Document review and interviews with key stakeholders indicated that the MTCP governance structure and processes are clearly defined and revised regularly to make the necessary adjustments. The MTCP’s standard operating procedures (SOP) outline the steps and stages of the program’s planning processes related to annual activities and allocations planning. The program is governed by two standing committees, the MASC and the Professional Military Education International Allocation Board, both of which have terms of references in place clearly outlining their governance, operations, activities and responsibilities. In preparation for annual MASC deliberations, the MTCP has at times augmented the plenary session with interdepartmental working groups to discuss policy priorities, regional training requirements and program delivery. DMTC also engages in biannual operational design workshops which allow for discussion concerning MTCP activities plans, troops-to-task, SOPs, budget, battle rhythm, human resources and other critical matters.

Most interviewees were satisfied with the overall governance structure of the MTCP and felt it was appropriate and efficient. As one senior manager commented, “the MTCP’s governance structure is enduring, flexible and responsive enough to permit a wide range of ongoing engagement that is planned well in advance and realigned in a deliberate, strategic fashion as priorities change.” Most CDAs interviewed for this evaluation were satisfied with the oversight DMTC receives from ADM(Pol), the regional policy directorates and GAC.

The MASC – the program’s key decision-making tool – provides an appropriate venue for discussion among the various program stakeholders according to most interviewees. Senior managers were generally satisfied with the support the MASC provides to the MTCP, and said that the MASC offers a forum where different perspectives and priorities are discussed, and facilitates consultation. The committee also provides a way for DMTC to brief key internal and external stakeholders on the MTCP, which helps to reduce in-year questions. External stakeholders said that the MASC is a useful governance mechanism that promotes open discussion between GAC and DND, and is receptive to GAC’s suggestions.

Communication and consultation

There is regular and sufficient communication and consultation among the MTCP stakeholders, according to interviewees. Many internal interviewees felt there is an ongoing effective dialogue between DMTC and senior management within DND. Documentary evidence indicated that DMTC and the DGIS Pol regularly engage with key program stakeholders on key program governance, planning and implementation issues. For example, the network of CDAs accredited to MTCP member countries are consulted on regional priorities and training requirements.Footnote 37  These consultations take place throughout the year and there is daily communication with DMTC on issues ranging from strategic priorities to routine administration. All of DMTC interviewees and most of interviewed CDAs agreed that there is sufficient consultation between DMTC and the CDAs.

Most interviewees also indicated that there is sufficient interaction between DND and GAC; however a few senior managers noted some challenges in working with GAC. In particular, a few interviewees expressed concern about GAC’s decision-making role as the MTCP is a program fully funded and administered by DND; however, most felt that the nature of GAC’s engagement in the program is appropriate given GAC’s lead role in foreign relations. A few interviewees also stated it is challenging to work with GAC because of the complexity of its internal organizational structure and the time it takes GAC to collect input from its various groups and stakeholders. Nevertheless, senior managers indicated that the MTCP maintains regular communications with GAC on issues related to program delivery. Furthermore, DGIS Pol geographic desks and CDAs liaise with their GAC counterparts to ensure in-year recommendations to DMTC on allocations and activity design are aligned.

DMTC also tracks, feeds into, and is informed by a large number of fora led by other entities, such as monthly desk officer meetings for specific regions, staff talks (led by DGIS Pol and/or SJS with member countries), bilateral consultations led by GAC with member countries, and multilateral fora at which related training and capacity building are discussed. This includes, among others, G7 Experts Group on Peacekeeping and Peacebuilding, ASEAN Regional Forum, UN Peacekeeping Defence Ministerial, Conference of Defence Ministers of the Americas, Caribbean Security Conference, Western Hemisphere Trilateral (CAN-US-UK) Framework Agreement Dialogue, Tokyo Defence Forum, and NATO Defence Education Enhancement Program.

Roles and Responsibilities

Most interviewees agreed that the roles and responsibilities related to the delivery of the MTCP are generally clear and well understood. There were inconsistencies in the management of the MTCP in the past, including coordination and awareness challenges, parallel yet divergent documentation, and duplication of efforts. Internal program managers commented that DMTC has taken steps to address these issues. Initiatives launched by DMTC since 2017 included biannual operational design workshops, SOPs for finance, policy, and operations, an internal decision matrix outlining the process of MTCP-related decision making, a new battle rhythm/process for activities planning, quarterly budget briefs to DGIS, yearly consultations with CDAs and a new MOU template. DMTC is currently working on developing SOPs for the personnel outside Canada, and establishing SOPs for training officers.

Evidence shows that there is a need to determine how capacity building efforts are delivered within DND and clarify the respective roles and responsibilities of DMTC and SJS. Engaging in capacity building to support the security of other nations and their ability to contribute to security abroad is one of the core missions of the CAF identified in SSE.Footnote 38  Many internal interviewees indicated that, despite this renewed commitment to capacity building, no adequate new organizational structure or resources have been added, and there is currently no effective, central clearing house for training and capacity building requests at DND/CAF. SJS has been tasked with leading the implementation of this initiative, as it is responsible for the drafting of the Global Engagement Plan that will, on promulgation of the revised GES, more precisely task the Operational Commands and other CAF organizations which will enable the capacity building activities.Footnote 39  However, the SJS lacks Vote 10 funds that could be used for capacity building initiatives, and has to seek funds through time/mandate-limited operational Memoranda to Cabinet, in addition to relying on leveraging of GAC programs such as the Counter-Terrorism Capacity Building Program and the Peace and Stabilization Operations Program. Internal interviewees stressed that the MTCP is currently the only program with Vote 10 funds within DND that does work in relation to capacity building. As the SJS will potentially be looking to create a capacity-building unit/organization whose roles and responsibilities are yet to be defined, its creation could have an impact on the MTCP.

Both internal and external interviewees emphasized the need to clarify how the MTCP aligns with the broader capacity building initiative under SSE, particularly in relation to the SJS’s new Directorate of Strategic Engagement and Capacity Building (SECB) that will help guide pan-CAF capacity building activities and facilitate coordination with GAC. All interviewees who commented on the matter felt that there is a need to clarify the roles and responsibilities between DMTC and the SJS, to ensure the linkages between the two organizations are clear and there are no gaps, and to avoid duplication of effort. While interviewees had mixed views on how the two initiatives could best fit together, a few interviewees from ADM(Pol) and DMTC expressed concern about a possible merging of the MTCP with SECB, arguing that the MTCP’s current place within DND’s governance structure makes it an effective defence diplomacy mechanism that achieves its objectives through building softer skills and domains versus operational tools.

ADM(RS) Recommendation

2.3.3 Resource Efficiency

Key Finding 10: The MTCP activities have increased despite decreasing resources and staffing shortages over the evaluation period; however, anecdotal evidence suggests the MTCP continues to achieve its program outcomes.

Document review, interviews with key stakeholders and analysis of program data indicated the MTCP activities have increased, as program resources have decreased. It appears that the program is being delivered in a more efficient manner, but without a comprehensive performance measurement framework, this is not possible to determine. As shown in Table 5, the number of individuals participating in the MTCP has increased by 39 percent. Overall expenditures and program disbursements have remained relatively stable but there has been a significant decline (20 percent) in the expenditures to administer the MTCP. The majority of the decline is due to a 38 percent reduction in civilian salaries, from $2.76 million in FY 2013/14 to $1.72 million in FY 2017/18. These reductions in administration costs have decreased the proportion of the program’s overall administration budget from 34 percent to 28 percent over the evaluation period.Footnote 40 

Table 5. Number of MTCP participants and expenditures by fiscal year.

  FY 13/14 FY 14/15 FY 15/16 FY 16/17 FY 17/18 5-year growth

Number of participants

1,122

1,162

1,475

1,628

1,561

39%

Salaries and Operating Expenditures ($M)

5.1

3.7

3.8

3.9

4.1

-20%

Grants and Contributions Funds ($M)

10.1

9.7

10.4

10.8

10.5

4%

Total Funding ($M)

15.1

13.4

14.1

14.8

14.6

-3%

Table 5. Number of MTCP participants and expenditures by fiscal year.Footnote 41  This table displays the total number of MTCP participants, the salaries and operating expenditures, Grants and Contributions funds and total expenditures for the MTCP for FYs 2013/14 to 2017/18.

Table 5 Details - Number of MTCP participants and expenditures by fiscal year.

Data on full-time equivalents (FTE) or the number of positions vacant versus staffed was not available; however DMTC staff interviewed indicated that they were operating at 50 percent capacity, and faced several staffing challenges during the evaluation period, including difficulty staffing vacant positions and high staff turnover. As a result, DMTC program managers reported that staff are overworked and employee burnout is a risk. Additionally, DMTC has conducted an internal review of resource allocation versus assigned responsibilities and identified a need for an increase in personnel assigned to deliver the MTCP, in addition to staffing the vacant positions.

ADM(RS) Recommendation

Most interviewees stated that the MTCP is delivered efficiently given the resources available, and that current levels of funding are adequate for the program, but should the need arise to increase the level of activity, additional resources would be required. As discussed in the Performance Measurement section of this report, the absence of a performance measurement strategy to accurately assess achievement of outcomes limited the evaluation’s ability to assess the program’s value versus resources expended. However, all senior managers and internal program managers indicated the MTCP provides good value for money in the context of achieving the program’s outcomes, and the results achieved are satisfactory given relatively minimal investment.

2.3.4 Performance Measurement

Key Finding 11: While the MTCP collects some performance data as part of its Annual Assessment, the program lacks a comprehensive performance measurement strategy, and requires improvements to its collection and management processes of performance and financial data to ensure program performance is measured and monitored consistently.

The evaluation found that the MTCP lacks a comprehensive performance measurement framework to effectively monitor progress towards its intended outcomes and its overall efficiency. Many interviewees cited difficulty attributing progress on the program’s intended outcomes and objectives directly to MTCP activities, and noted that these outcomes and objectives are nearly impossible to quantify by design, due to the nature of the MTCP as a defence diplomacy program. Many interviewees within ADM(Pol) and DMTC, as well as CDAs, stressed that the MTCP needs to develop a formal performance measurement system to enable the monitoring and evaluation of program implementation on a regular basis. A draft performance measurement strategy was prepared in 2009, and outlined proposed performance indicators that were grouped into three main categories with the purpose of measuring the program’s success, cost-effectiveness and relevance. However, the performance measurement strategy was neither formalized nor fully implemented.

A review of MTCP performance data indicated that performance data currently collected by the program is primarily output-based. The evaluation team found that the current indicators in the Annual Assessment, the program’s key performance measurement tool designed to collect data at the program level, are largely perception-based and provide limited value in capturing program outcomes. The Annual Assessment collects input from all service delivery partners, stakeholders and beneficiaries through three different questionnaires sent to CDAs, students and directing staffs. It is meant to collect data on the program’s intended outcomes, with particular emphasis on CDAs’ feedback, as they maintain the closest level of relationship with foreign country recipients.Footnote 42  Evidence suggests that, while some performance measurement data is collected through the Annual Assessment, the performance indicators used to assess progress and the program logic model that outlines the MTCP’s intended outcomes have not been revised since 2009.

While the program collects a range of data related to the delivery of the MTCP, inconsistent data management and reporting have precluded the program from using this data for performance measurement. Documentary and interview evidence suggests that the MTCP has relied on unrefined data management systems. For example, the DMTC regional and country expenditures database was not maintained on a consistent basis during the evaluation period, and was discontinued in 2016. The program is currently working on reviving it by updating it to the latest Excel version and populating it with missing data. Internal program managers interviewed for this evaluation also reported that data collection and management have been too time-consuming as the MTCP lacked the resources and the expertise to analyze and track data consistently. The interviewees attributed the gaps in data management to staffing issues within DMTC (see Key Finding 10), the increasing requirements for data collection and analysis, and a lack of an integrated software.Footnote 43  Senior managers and program staff commented that a better data management system (e.g., a software tool and dedicated personnel to create and maintain a consolidated database of program data) would reduce miscommunications, improve planning, and allow DMTC policy and operational staff to engage in more high-value activities.

DMTC has recently launched an attempt to improve data management and performance monitoring; however, the program needs to prioritize these efforts. Initiatives launched by DMTC in 2017 include:

As mentioned previously, DMTC is planning to launch a renewal in the near future. As part of this process, DMTC should review the performance measurement strategy and revise the program’s logic model and performance indicators.

ADM(RS) Recommendation


Back to Table of Contents

Annex A—Management Action Plan

Key Finding 8: The MTCP lacks a graduation mechanism beyond joining NATO, and the program’s membership criteria require updating.

ADM(RS) Recommendation

Management Action

Table A-1. Management Action Plan for Recommendation 1.

Action Item Target Date
  1. Create a DMTC working group to examine and modernize DMTC’s Terms and Conditions including:
    1. Membership criteria
    2. Graduation criteria

February 2019

  1. Develop concept options based on ADM guidance to modernize the DMTC’s Terms and Conditions.

April 2019

  1. Seek ADM option decision.

Fall 2019

  1. Convert selected option into a submission plan for Assistant Deputy Minister (Finance) (ADM(Fin)).

September 2020

Table A-1. Management Action Plan for Recommendation 1. This table outlines the action items and target dates for Recommendation 1.

Table A-1 Details - Management Action Plan for Recommendation 1.

OPI: DMTC

Key Finding 9: In general, the MTCP governance structure and processes are clearly defined and efficient, and there is effective communication between DMTC and key program stakeholders and partners; however, there is a need to clarify the roles and responsibilities related to capacity building efforts within DND.

ADM(RS) Recommendation

Management Action

Table A-2. Management Action Plan for Recommendation 2.

Action Item Target Date
  1. Establish a Director level engagement between ADM(Pol)/DMTC and SJS/SECB.

Completed

  1. Establish a staff-level coordination working group with DND stakeholders.

Completed

  1. Develop a training and capacity-building alignment concept paper for the Defence Executive Policy Committee (DXPC) (ideally joint DMTC/SECB).

April 2019

  1. Extract guidance from DXPC approved concept paper and incorporate into the renewal for ADM(Fin) per Recommendation #1 Action item #4.

September 2019

Table A-2. Management Action Plan for Recommendation 2. This table outlines the action items and target dates for Recommendation 2.

Table A-2 Details - Management Action Plan for Recommendation 2.

OPI: DMTC

Key Finding 10: The MTCP activities have increased despite decreasing resources and staffing shortages over the evaluation period, and anecdotal evidence suggests the MTCP provides good value for money in the context of the achievement of program outcomes.

ADM(RS) Recommendation

Management Action

Table A-3. Management Action Plan for Recommendation 3.

Action Item Target Date
  1. Identify DMTC staffing establishment for Defence Team Establishment Plan (DTEP) Submission.

Completed

  1. Conduct a SWEFootnote 45 /FTE/military personnel needs analysis to inform DTEP.

Completed

  1. Seek DG approval for staffing recommendations.

Completed

  1. Submit DTEP.

Completed

  1. Investigate and provide recommendations to DG on personnel backfill plan prior to fulfilling final staffing strategy.

Completed

Table A-3. Management Action Plan for Recommendation 3. This table outlines the action items and target dates for Recommendation 3.

Table A-3 Details - Management Action Plan for Recommendation 3.

OPI: DMTC

Key Finding 3: Anecdotal evidence suggests that the MTCP training contributes to enhancing the skills and abilities of course participants; however there is a lack of a consistent follow-up mechanism to track MTCP graduates.

Key Finding 11: While the MTCP collects some performance data as part of its Annual Assessment, the program lacks a comprehensive performance measurement strategy, and requires improvements to its data collection and data management processes to ensure program performance is measured and monitored consistently.

ADM(RS) Recommendation

Management Action

A data management strategy is a long-term investment (five years or more) which necessarily will need to be implemented in phases and synchronized with the Department’s nascent analytics capability. DMTC’s efforts in the short term will be to focus on easily implementable foundational processes which can mature over time as corporate tools and systems are developed. This strategy is dependent implementing staffing changes as prerequisites. DMTC is also dependent on Assistant Deputy Minster (Data, Innovation and Analytics) and Assistant Deputy Minister (Information Management) Director Business Relationship Management as enablers to our data management and performance measurement desires.

Table A-4. Management Action Plan for Recommendation 4.

Action Item Target Date
  1. Assess need for additional positions related to data management. If positions are needed add requirement to Recommendation #3.

Completed

  1. Establish an information/data management plan focusing on the following elements:
    1. Governance
    2. Data management
    3. Architecture (how users will access data)
    4. Data quality
    5. Data integration
    6. Data literacy
    7. Activity plan and financial management integration
    8. Data reporting, analytics and business intelligence

May 2019

  1. Initiate engagements with departmental data management stakeholders to align DMTC information management efforts with corporate systems and to solicit support.

June 2019

  1. Develop a performance management framework for submission in support of Recommendation #1.

September 2020                          

  1. Start implementation of the information/data management plan in item 2. Implementation will occur over multiple years.

January 2021

Table A-4. Management Action Plan for Recommendation 4. This table outlines the action items and target dates for Recommendation 4.

Table A-4 Details - Management Action Plan for Recommendation 4.

OPI: DMTC


Back to Table of Contents

Annex B—Evaluation Methodology and Limitations

1.0 Methodology

1.1 Overview of Data Collection Methods

The findings and associated recommendations of this evaluation are supported by multiple lines of evidence collected through qualitative and quantitative research methods. Information and data collected from relevant sources were analyzed to inform conclusions on the relevance and performance (effectiveness and efficiency) of the program. Multiple data sources provide information from various perspectives, which helps triangulate and validate evaluation findings, thus increasing their accuracy and credibility, and strengthening the validity of conclusions.

Key activities of the evaluation were grouped to determine the expected outcomes of the program as defined in the logic model. Indicators were identified for each of the evaluation questions to assess the relevance and performance of the MTCP. Evaluation questions, their associated indicators and related data collection methods are listed in the Evaluation Matrix in Annex D.

The following methodologies were used to collect evidence for the evaluation:

1.2 Details on Data Collection Methods

1.2.1 Document Review

A preliminary document review was conducted, as part of the planning phase of the evaluation, to garner a foundational understanding of the MTCP, its purpose, scope and mandate. Additionally, the document review enabled visibility into the larger organizational and operational context in which the program operates. Core program documents like the initiating documents were used to establish parameters regarding program design and delivery and as a source of data to support the evaluation of the effectiveness and efficiency of the program.

A comprehensive document review was conducted as part of the conduct phase of the evaluation, focusing on the relevance and performance of the MTCP activities. The documents reviewed during the conduct phase of the evaluation included the following:

1.2.2 Key Informant Interviews

Key informant interviews and discussions with program representatives and other key stakeholders provided important sources of qualitative information and informed opinion and observations on select evaluation questions.

Key stakeholders who contributed to interviews were located within the vicinity of Ottawa or were contacted by phone. Interviews were conducted with key senior stakeholders from DMTC, from ADM(Pol) (senior officials), SJS (senior officials) and from GAC (senior officials). In addition, interviews were conducted with a select group of CDAs abroad.

Informal pre-scoping interviews were conducted with stakeholders within DMTC to determine appropriate areas of inquiry within the initially prescribed scope and develop the foundation for the development of comprehensive key informant interview guides.

To support this area of inquiry, 23 interviews were conducted with 25 key informants:

Interview guides were tailored specifically to each group of key informants and shared with them in advance. Interviews were semi-structured and conducted in-person or by telephone either one-on-one or in a group setting. Detailed notes were taken during the interviews and then transcribed and analyzed according to research themes. Some of the CDAs responded to the interview questions in writing. When necessary, interviewees provided written responses to the interview questions after the interview was completed to provide more detailed responses

Table B-1 outlines the methodology for the analysis and reporting of interview data.

Table B-1. Interview Response Analysis Grid.

Responses Percentage

A few

1 – 24%

Some

25 – 49%

Many

50 – 74%

Most

75 – 99%

All

100%

Table B-1. Interview Response Analysis Grid. This table displays the classification for interview responses and the corresponding percentage of interviewees.

Table B-1 Details - Interview Response Analysis Grid.

1.2.3 Review of Program Performance Data

MTCP performance data review was conducted based on available data from the MTCP Annual Assessment – DMTC’s qualitative and quantitative monitoring system. The Annual Assessment is designed to collect data at the program level, seeking input from specific MTCP stakeholders and beneficiaries. It consists of three questionnaires sent annually to CDAs, students and schools’ directing staffs.

The review of performance data from the Annual Assessment applicable to the period covered by the evaluation (FY 2013/14 to FY 2017/18) provided data at the program level on the MTCP’s effectiveness at achieving its intended outcomes, as well as the program’s efficiency.

1.2.4 Case Studies

To provide further insight, six case studies were conducted. Case studies were selected based on intensity of MTCP activities during the evaluation period, and were intended to be representative of the four geographic regions where the MTCP activities are conducted. Case studies targeted the following member countries:

Case studies enabled the evaluation team to collect the following additional evidence:

Case studies involved the document and data review and additional interviews with Canadian ambassadors or high commissioners in the selected countries. Case study interview guides were developed and were designed to address specific evaluation questions in the evaluation matrix.

1.2.5 Review of financial data

Financial and administrative data for the MTCP were reviewed with respect to the indicators provided in the Evaluation Matrix (see Annex D) to evaluate the efficiency and economy of the program. Financial data for FY 2013/14 to 2017/18 was obtained from DMTC.

2.0 Limitations

Table B-2 describes the limitations and mitigation strategies employed in the Evaluation of the Military Training and Cooperation Program:

Table B-2. Evaluation Limitations and Mitigation Strategies.

Limitation Mitigation Strategy

The evaluation had a limited time period and resources to conduct the evaluation in order to meet required deadlines. In addition, the program documentation and files were obtained late in the evaluation process, limiting time available to conduct the main data collection and analysis phase of the evaluation.

In consultation with DMTC staff, the decision was made to focus the evaluation on key intermediate outcomes of the MTCP Logic Model. Short-term outcomes were grouped together and assessed as a combined outcome “Contribution to enhancing the abilities of military trainees.” The short-term outcome, “developing capacity of member countries to undertake peace support operations,” and long-term outcome, “contributing to international peace and security,” were scoped out. This permitted the evaluation team to focus their efforts and assess this limited scope.

Program performance data was only available for part of the evaluation period, from FY 2013/14 to FY 2016/17. The Annual Assessment for FY 2017/18 was not yet available during the conduct phase of the evaluation.

The evaluation team assessed program performance data for the first four years of the evaluation period (FY 2013/14 to FY 2016/17), and attempts were made to fill information gaps through other sources (e.g., interviews, case studies).

Attribution of activities and outputs to intended outcomes was challenging due to the nature of the program targeting defence diplomacy and the difficulty in measuring the program’s objectives. This means that the evaluation evidence relies heavily on perceptions of key stakeholders.

The evaluation consulted multiple sources of quantitative and qualitative information to determine the performance of the program. The evaluation team attempted to mitigate this limitation by supplementing data from key informant interviews and MTCP’s Annual Assessments with evidence from document review.

There was the possibility that the interviewees would provide biased information and only positive stories about their program.

The evaluation consulted various groups of stakeholders, both internal and external to the program. A comparison was made between interviewees and other people from the same organization or group, and information from other sources (e.g., documents and files).

Table B-2. Evaluation Limitations and Mitigation Strategies. List of the limitations of the evaluation and the corresponding mitigation strategy.

Table B-2 Details - Evaluation Limitations and Mitigation Strategies.


Back to Table of Contents

Annex C—Logic Model

Figure C-1.  Logic Model for MTCP.
Figure C-1. Logic Model for MTCP. This flowchart shows the relationship between the MTCP’s main activities, outputs and expected outcomes.
Figure C-1. Details - Logic Model for MTCP.


Back to Table of Contents

Annex D—Evaluation Matrix

Table D-1. Evaluation Matrix.

Evaluation Question Indicators Finding Number

Relevance

   

1. Is there a continued need for the MTCP?

1.1 Evidence of/perceptions of the continued need for the MTCP activities

1

1.2 Presence/absence of other programs that complement or duplicate the objectives of the MTCP

2. Is the MTCP aligned to federal government priorities?

2.1 Extent to which the MTCP’s outcomes correspond to recent/current federal government and departmental priorities

2

Effectiveness

   

3. To what extent has the MTCP contributed to enhancing the abilities of military trainees?

3.1 Evidence of/views on demand for MTCP training

3

3.2 Evidence of/views on whether course graduates apply the knowledge they gained in their current job

4. Does the MTCP contribute to expanding and reinforcing Canada’s bilateral defence relations?

4.1 Evidence of/views on the extent MTCP has improved bilateral relations with member countries

4

4.2 Evidence of/views on the extent member countries have demonstrated improvement in interoperability with Canada and NATO allies

5. Does the MTCP contribute to promoting Canadian democratic principles, the rule of law, and human rights in the international arena?

5.1 Evidence of/views on the MTCP’s success in promoting Canadian principles abroad

5

5.2 Evidence of/views on the extent the MTCP encourages the participation of female trainees in its training courses and activities

6. Does the MTCP contribute to achieving influence in areas of strategic interest to Canada?

6.1 Percentage of foreign candidates trained within the governmental priority regions (countries in category A versus category B)

6

6.2 Evidence of/views on MTCP’s contribution to establishing influence in specific areas of strategic interest to Canada

Efficiency

   

7. Is the MTCP’s governance structure appropriate and efficient?

7.1 Clearly defined governance structure, including program processes

9

7.2 Clarity of roles and responsibilities for program delivery

7.3 Evidence of/views on the appropriateness and effectiveness of governance structure

7.4 Evidence of/views on the whether there is sufficient communication between program stakeholders

8. Is the MTCP’s membership structure appropriate and efficient?

8.1 Evidence of/views on the appropriateness and efficiency of MTCP’s membership structure

7

8.2 Evidence of/views on the appropriateness and efficiency of MTCP’s membership criteria

8

9. Is the program implemented in an efficient and economical manner?

9.1 Proportion of MTCP spending in priority regions

10

9.2 Analysis of actual program administrative costs in relation to Grants and Contributions funding

9.3 Trend analysis of program resources (expenditures and FTEs)

9.4 Amount of MTCP funding as a proportion of the overall Defence budget

9.5 Views on whether the progress made toward expected outcomes is adequate for the resources expended

9.6 Evidence of/views on how the efficiency of program activities can be improved and alternative ways of delivering program activities and outputs

Table D-1. Evaluation Matrix. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the relevance, effectiveness and efficiency of the Military Training and Cooperation Program.

Table D-1 Details - Evaluation Matrix.


Back to Table of Contents

Page details

Date modified: