Evaluation of the NATO Contribution Program
Air Command and Control System
Allied Command Operations
Assistant Deputy Minister (Finance)
Assistant Deputy Minister (Materiel)
Assistant Deputy Minister (Policy)
Assistant Deputy Minister (Review Services)
Alliance Ground Surveillance
Canadian Armed Forces
Canadian Representative to the Military Committee
Canadian Special Operations Forces Command
Canadian Forces Intelligence Command
Canadian Joint Operations Command
Combined Joint Operations from the Sea
Centre of Excellence
- Comd CA
Commander of the Canadian Army
- Comd RCAF
Commander of the Royal Canadian Air Force
- Comd RCN
Commander of the Royal Canadian Navy
- Comd RCN
Commander of the Royal Canadian Navy
Defence Acquisition Guide
Defence Capabilities Board
NATO Defence Ministerial Meeting
Director General International and Industry Programs
Department of National Defence
North Atlantic Treaty Organization Naval Forces Sensor and Weapon Accuracy Check Sites
Global Affairs Canada
Government of Canada
International Board of Auditors for the North Atlantic Treaty Organization
Joint Air Power Competence Centre
Joint Intelligence, Surveillance, and Reconnaissance
Munitions Safety Information Analysis Centre
North Atlantic Council
North Atlantic Treaty Organization Airborne Early Warning and Control
North Atlantic Treaty Organization Canadian National Technical Expert
North Atlantic Treaty Organization
North Atlantic Treaty Organization Communications and Information Agency
North Atlantic Treaty Organization Intelligence Fusion Centre
North Atlantic Treaty Organization Special Operations Headquarters
North Atlantic Treaty Organization Security Investment Program
North Atlantic Treaty Organization Support and Procurement Agency
Office of Collateral Interest
Office of Primary Interest
Royal Canadian Air Force
Royal Canadian Navy
Supreme Allied Commander Europe
Supreme Allied Commander Transformation
Supreme Headquarters Allied Powers Europe
Strategic Joint Staff
- SWG EW
Special Working Group Electronic Warfare
Vice Chief of the Defence Staff
- NATO membership is more relevant for Canada in the current threat climate compared to ten years ago.
- Membership in NATO has provided Canada with access to equipment, military capabilities and strategic information.
- DND/CAF management of the NATO Contribution Program would benefit from additional improvements, such as establishment of a single point of contact within DND/CAF, knowledge management mechanisms and performance measurement systems in order to access and assess program outputs and outcomes.
|Key Finding 1: NATO membership affords Canada with access to common military infrastructure and capabilities and an equal voice in high-level decisions within the Alliance. Evidence suggests that NATO is more relevant in the current threat climate compared to ten years ago.||See Recommendation 3|
|Key Finding 2: The NATO Contribution Program aligns with federal roles and responsibilities; it operates within the framework of the North Atlantic Treaty (Washington Treaty, 1949) and the National Defence Act. During the July 2016 NATO summit; Canada renewed its past and future commitments, indicating that Canada will continue to be a committed member within the Alliance.|
|Key Finding 3: The management of the NATO Contribution Program is part of an integrated approach undertaken by individuals from both DND/CAF and the GAC. Although DND/CAF works with GAC in the management of the NATO Contribution Program, the evaluation found no evidence of duplication of roles.|
|Key Finding 4: The NATO Contribution Program aligns with federal government priorities and supports DND/CAF’s strategic interests as identified in the Global Engagement Strategy. The membership responds to the need to maintain a “network of defence partners” which allows DND/CAF opportunities for networking, information sharing, and increased interoperability.|
|Key Finding 5: NATO Standardization was described as one of the main benefits of NATO membership. Canada ranks as 7th among 28 NATO countries in the implementation of the NATO Standards.||Recommendation 1: Review and actively implement NATO Standardization Agreements (STANAG), where appropriate.|
|Key Finding 6: DND/CAF has taken an active role in the development of NATO Standards. However, efforts in the implementation of Standards at the working level could be increased.||
OPI: Comd CA, Comd RCN, Comd RCAF
|Key Finding 7: According to Allied nation’s representatives and based on the activities of Brussels–NATO (BNATO), Canadian delegation’s active participation in NATO’s resource committees, particularly in the implementation of best practices, such as accountability and transparency have been significant.||See Recommendation 3|
|Key Finding 8: The DND/CAF withdrew from NATO’s NAEW&C and AGS acquisition group in 2013.|
|Key Finding 9: DND/CAF’s contributions to NATO provide access to a number of unique capabilities through both core and non-core activities that enable interoperability and enhance the testing of existing national capabilities.|
|Key Finding 10: Membership in NATO allows DND/CAF to access strategic information developed through various NATO activities, which enables DND/CAF to save resources, maintain expertise in unique subject areas, and access data to build national capabilities.|
|Key Finding 11: During the evaluation period, the CAF has participated in a number of NATO operations and exercises. In 2011, a CAF member became in charge of the Combined Joint Task Force Unified Protector.|
|Key Finding 12: The DND/CAF management structure and reporting process for the NATO Contribution Program has improved since the last evaluation (2011). However, additional improvements, such as establishment of a single point of contact within DND/CAF for reporting would assist in effectively using program outputs and outcomes in program validation.||
Recommendation 2: Establish a strategic coordination role within ADM(Pol) that would take a consolidated approach to providing L1s access to consistent information and improving awareness of DND/CAFs larger NATO engagement.
OCI: ADM(Fin), VCDS
Recommendation 3: Continue to provide funding through the NATO Contribution Program to the core and non-core programs for NATO membership; however, going forward, develop processes for the DND/CAF to monitor and/or comprehensively review core and non-core program results. This will provide a holistic perspective of the program and also assist in determining which non-core program activities may be funded for the next program cycle through the NATO Contribution program.
OCI: ADM(Pol), VCDS
Recommendation 4: Develop mechanisms to make NATO program information (outputs and outcomes) accessible to DND/CAF members.
|Key Finding 13: Information from core and non-core programs is not being effectively used within the DND/CAF to inform or to validate the comprehensive impact of the Program. This issue has been raised in the previous evaluation report (2011). Further, the Contribution Program does not have a holistic performance measurement strategy; very few of the programs have started measuring program performance. However, these are not collated and reported.|
|Key Finding 14: In terms of knowledge management, the DND/CAF NATO Contribution Program does not formally solicit feedback from core and noncore program recipients. In some areas, information collected on program outputs and outcomes has not been well promoted or made easily accessible.|
|Key Finding 15: The Washington Treaty (1949), recent Warsaw Summit (2016), and the Global Engagement Strategy (DND, 2015) support the involvement and development of member nations’ industries in order to provide domestic economic benefits and benefits to NATO as a whole. Participation and awarding of Canadian industries in NATO contracts have been lower compared to other similar NATO countries. As an example, France and Belgium industries gain about four times their national contribution and Norway about three times in total value of contracts.||
Recommendation 5: Improve efforts to assist industry with potential contracts as per the Global Engagement Strategy (DND, 2016) and the Industrial and Technological Benefits Policy (2014) to align Canadian industrial participation with potential NATO contracts.
|Key Finding 16: France and the US have larger delegations to support industry compared to Canada. These countries have also developed comprehensive strategies to support national industry.|
|Key Finding 17: There are areas where interoperability can further be improved through implementation of NATO standardization.||See Recommendation 1|
|Key Finding 18: According to NATO records, in 2015 Canada spent the equivalent of 0.98 percent of GDP on defence in comparison to the 2 percent guideline to be moved towards by 2024. Canada has also demonstrated commitment to NATO by agreeing to lead a multi-national NATO battlegroup in Latvia.|
|Key Finding 19: DND/CAF fulfills hundred percent of their personnel support commitment in NATO Command.||See Recommendation 3|
|Efficiency and Economy|
|Key Finding 20: In order to deal with NATO’s management of funding, ADM(Fin) optimizes the use of resources to the best of their ability by using historical financial information and trending.||See Recommendation 3|
|Key Finding 21: A number of NATO contribution program activities offer benefits to DND/CAF by providing access to unique capabilities and expertise. The evaluation noted opportunities to improve and optimize DND/CAF’s involvement in other program activities to fully take advantage of their outcomes.|
|Key Finding 22: DND/CAF is unable to access common fuel contracts through the NSPA program due to Treasury Board contracting policies. Improvements to existing contracting policies could lead to cost savings.|
|Key Finding 23: DND/CAF’s contributions to NATO dropped from $230 million in 2010-2011 to $130 million in 2015/16. A contributing factor to the reduction was DND/CAF’s withdrawal from NAEW&C and AGS acquisition group. Canada has the sixth largest cost share of Military Budget and NSIP contributions, in comparison to other NATO members.|
Table 1 Details - Summary of Key Findings and Recommendations
Table 1 Summary: This table has two columns. The left-hand column lists the key findings and the right-hand column lists the recommendations. Read across each row for the key finding and its corresponding recommendation.
|Change from previous year (%)||-||-26.8||-11.2||-9.7||-21.3||20.5||-|
|Average annual change (%)||-||-||-||-||-||-9.7||-|
|Change from previous year (%)||-||0.5||-16.6||15.8||-28.5||48.2||-|
|Average annual change (%)||-||-||-||-||-||3.8||-|
|Change from Previous Year (%)||-||-26.6||-11.3||-9.4||-21.4||20.9||-|
|Average annual Change (%)||-||-||-||-||-||-9.6||-|
Source: Public Accounts of Canada
Table 2 Details - Total Expenditures for the NATO Contribution Program
Key Finding 8: The DND/CAF withdrew from NATO’s NAEW&C and AGS acquisition group in 2013.
Within the DND/CAF, the NATO Contribution Program is primarily managed by ADM(Fin) as the expenditure authority and is supported by ADM(Pol), ADM(Mat) and SJS, who provide functional guidance, planning and policy oversight to the program.Footnote 70 In addition, other Level Ones such as, CFINTCOM, CJOC and the three Environments provide functional guidance to mainly the non-core programs. The Program is managed by a program manager within the ADM(Fin) and a Director General, who directly responds to queries regarding financial matters. ADM(Fin) has three additional staff in Brussels, NATO. BNATO handles financial matters and represent Canada in key NATO committees. The BNATO staff work in coordination with the program manager to provide comments, advice and input with regard to policy, programs, capability and resources. Staff from other Level Ones such as ADM(Mat), also participate in NATO committees and report financial matters to ADM(Fin) and BNATO.
Problems associated with the management of the program were cited in the previous ADM(RS) evaluation of the NATO Contribution Program (2011). A NATO Steering Committee was formed as a direct response to one of the recommendations of the previous evaluation.Footnote 71 The aim of the Steering Committee has been to increase information sharing among DND/CAF Level One organizations, NATO Allied Command Transformation and Canada’s Joint Delegation to NATO thereby facilitating coordination on issues and collaboration in the provision of strategic advice to senior management. The Committee meets on a bi-monthly basis to discuss DND/CAF's ongoing work at NATO and to identify upcoming issues and priorities.Footnote 72 Based on interviews with program staff, bi-monthly Steering Committee meetings have enabled a better flow of information on current initiatives and have resulted in a more coherent engagement with NATO. However, it was observed that the Steering Committee meetings have been primarily geared towards updates and discussions on policy, planning and capability related issues. Issues concerning the management of the NATO Contribution Program or financial updates are not discussed at every meeting. According to ADM(Pol) staff, including representatives from all program activities (in addition to Level One representatives) could create a cumbersome structure of reporting.
The evaluation observed that the management of the Program by DND/CAF has generally improved with the implementation of the Steering Committee and other recent reporting measures. Based on interviews with stakeholders, current management structure of the Program generally works well through coordinated efforts; however concerns were expressed in regards to the following reporting and other management issues:
(1) Lack of a single point of contact within DND/CAF. There is evidence to support that lack of a centralized point of contact within DND/CAF demonstrates a need for a coordinating and strategic guidance unit for all NATO activities within the Department. In the absence of such entity, a couple of the non-core programs have established their own contacts within the DND/CAF through chain of command. As cited in the previous evaluation (2011), it appears that very few staff within the DND/CAF could make the distinction of which NATO activities were covered under the NATO Contribution Program and which were covered by other entities within DND/CAF. Some program staff interviewed was also unaware of a point of contact for funding change requests and/or program related issues. This evidence suggests that the management set-up could benefit from further improvements both at the Program Manager and the Steering Committee levels in order to oversee the NATO Contribution Program’s core and non-core program activities.
(2) Lack of effective knowledge management system in place and dissemination of program outputs. It was noted that the lack of knowledge transfer mechanisms within DND/CAF for the NATO Contribution program, particularly for the non-core programs prevent the distribution and /or storage of valuable documents, such as newly developed concepts, doctrines and updates from the Centres of Excellence Outputs of programs are published in silos and not being used to derive best practices. According to the MSIAC program representative, more could be done in regards to dissemination of information and need for more outreach. For example, MSIAC outputs with insensitive munitions could be put to greater use in CAF projects. There is some evidence, however, that knowledge transfer occurs in informal ways. Interviews with various stakeholders, including CJOC and ADM(Mat) explained that activities towards knowledge transfer happens regularly, such as presentations made at committees are brought back to the Department. However, to date there is no formal process in place. This suggests an increased risk that knowledge might be being inefficiently used and/or duplicated.
Going forward, opportunities could be investigated for knowledge management and/or communication systems for NATO related activities. For example, a SharePoint system might offer a communication mechanism for program outcomes. The SharePoint can be made available to all stakeholders involved, and be optimized to deliver program information, provide guidance on financial issues, publish program summaries, and share knowledge and best practices related to all NATO Contribution program activities.
(3) Reporting requirements. Key elements of resource management are to maintain supervision and awareness of the program’s activities. While most core and non-core program staff do liaise with the Program Manager’s office within ADM(Fin), weaknesses in communication were identified concerning the non-core programs. Further, as cited in the previous evaluation and also evident during interviews, there is a general lack of awareness of the Program Manager position and the representatives of the core and non-core are not systematically reporting back to DND/CAF or receiving guidance from the Department. Formalized assessments (at least annually) should be conducted by the current Program Manager and discussed at the Steering Committee to determine whether or not Canada should continue participating in all current non-core programs and/or drop out of some of them.
It was noted that during the evaluation period regular feedback from program recipients and/or completed projects has not been formally sought to identify potential program improvements based on their experiences with the Contribution program. This led to a lack of systematic review of the program design or delivery beyond program renewal activities. In addition, consultations with program leads have only been conducted on ad-hoc basis during steering committee meetings, which were mostly held at strategic policy level.
(4) Performance measurement. Based on interviews, some project leads reported that very few of the programs have started using performance measurement indicators. They do not include details on results, participant feedback, optimal use of equipment etc. This could be hindered by a lack of Performance Measurement Strategy.
The evaluation examined the management structures of other NATO countries. ADM(Fin)’s BNATO staff and NATO committee chairmen interviewed pointed out that similar to Canada, 95 percent of the countries’ NATO structures are decentralized. The Civil and Military Budget contributions are administered by two separate government departments: Defence and Foreign Affairs. The financial and policy sides are also separately managed. However, some nations have additional personnel and/or bigger defence headquarters compared to Canada. For example, the US and Germany have similar structures to Canada but additional personnel and larger headquarters.
Going forward, despite its decentralized structure the management of the NATO Contribution program should ensure that direction and guidance, processes and procedures are set and followed by all program activities. Information on core and non-core program results are not compiled, monitored or comprehensively reviewed. This finding was echoed in previous evaluation of the Contribution program. Enhanced reporting to the Program Manager and involvement of the Steering Committee in regular program updates would improve the management structure. In this regard, the Program Manager should play a key role and ensure that communication and accountabilities are enhanced with core and non-core program activities, and where necessary, using the NATO Steering Committee as an important platform to engage stakeholders.
2.4.6 Intermediate Outcome 1 – Inclusion, promotion, and support of Canadian industry at NATO
To what extent does the DND/CAF support and promote Canadian industry considering or currently doing business with NATO?
The following indicators were used to assess this outcome:
- Support provided to Canadian industry by DND/CAF; and
- Participation of Canadian industry in NATO contracts.
|Change from previous year (%)||-26.6||-11.3||-9.4||-21.4||20.9|
|Department Actual Spending||20,298,257||20,218,758||19,978,190||18,764,374||18,453,938||18,666,073||116,379,591|
|Percentage of total Departmental Expenditure (%)||1.15||0.85||0.76||0.73||0.59||0.70||0.80|
Table 3 Details - Trends in Contribution Program Spending and Program Expenditure as a Percentage of the DND Budget
|-||2010||2012||2014||2016||Change from 2010 to 2016 (%)|
Table 4 Details - Canada’s Cost Share of the Military Budget and NSIP
3. Continue to provide funding through the NATO Contribution Program to the core and non-core programs for NATO membership however, going forward, develop processes for the DND/CAF to monitor and/or comprehensively review core and non-core program results. This will provide a holistic perspective of the program and also assist in determining which non-core program activities may be funded for the next program cycle through the NATO Contribution program.
Distinguishing the NATO Contribution Program from the entirety of DND/CAF’s NATO involvement.
The purpose of this evaluation was to assess the transfer payments made to NATO by the DND/CAF. However, transfer payments are only a small part of the DND/CAF’s total involvement in NATO. At times it was difficult for the evaluation team to determine what activities could be directly attributed to the Contribution Program. Most stakeholders interviewed were also unaware of the distinction between contributions made to NATO and other NATO activities.
|The evaluation team was careful in the scoping phase of the evaluation to determine the outputs and outcomes of the NATO Contribution Program. Although there are many outputs and outcomes of DND/CAF’s total NATO involvement, the evaluation only assessed what was determined to be the most important aspects of the Program.|
Identification of stakeholders.
Due to the decentralized management structure of DND/CAF’s NATO involvement, it was difficult to identify all stakeholder involved in the NATO Contribution Program. It is possible that key stakeholders were not consulted.
|The evaluation team interviewed as many stakeholders as possible. The interview list continued to develop throughout the evaluation process and interviews were conducted as new stakeholders were identified.|
Assessment of high level outcomes.
Some intermediate and ultimate outcomes were difficult to quantify and as a result, it could be difficult to assess their achievement.
|The assessment of intermediate outcomes included the use of document review, interview responses, and quantitative data where available. Information was triangulated in order to assess the achievement of the outcomes to the best of the evaluation team’s ability.|
Table B-1 Details - Evaluation and Mitigation Strategies
Figure C-1 Details - Logic Model for the NATO Contribution Program
|Evaluation Issues/Questions||Indicators||Literature and Document Review||Key Informant Interviews||Expert opinion|
1.1 Continued Need for the NATO Contribution Program
|1.1.1 Evidence of ongoing need for NATO Contribution Program activities||Yes||No||No|
1.2 Evidence of future need for NATO contribution program activities
|1.2.1 Evidence that the international security context indicates a need for NATO in the future||Yes||No||No|
1.3 Alignment with federal roles and responsibilities
|1.3.1 Alignment of the NATO Contribution Program with government acts and legislation||Yes||No||No|
|1.3.2 Extent of duplication of NATO Contribution Program activities that are the responsibility of other government departments, agencies, or private sector (role of GAC vs DND/CAF)||Yes||Yes||No|
1.4 Alignment with government priorities, DND/CAF priorities, and NATO strategic objectives
|1.4.1 Alignment between NATO Contribution Program activities and federal government priorities||Yes||No||No|
|1.4.2 Alignment between NATO Contribution Program activities and DND/CAF priorities||Yes||No||No|
|1.4.3 Alignment of NATO non-core activities with NATO’s overall strategic objectives||Yes||No||No|
Table D-1 Details - Evaluation Matrix – Relevance
|Evaluation Issues/Questions||Indicators||Administrative and financial data||Literature and Document Review||Key Informant Interviews||Expert opinion||Comparison with allies|
2.1 Immediate Outcome – 1 Increased standardization across NATO countries
|2.1.1 Extent that Canada meets NATO standards/comparison to other NATO countries||Yes||Yes||Yes||Yes||Yes|
|2.1.2 Stakeholder satisfaction with use of STANAGs||No||No||Yes||No||No|
|2.1.3 Extent of Canada’s participation in the development of NATO Standards||No||Yes||Yes||No||No|
2.2 Immediate Outcome – 2 Core and non-core activities contribute to increased, relationship building and networking
|2.2.1 Evidence that Canada has a large and effective network based on its participation in NATO/Canada’s profile is raised||Yes||Yes||Yes||No||Yes|
|2.2.2 Degree of impact of Canada’s withdrawal from the NATO Airborne Warning and Control and Air Ground Surveillance||Yes||Yes||Yes||Yes||No|
2.3 Immediate Outcome 3– DND/CAF gains access to equipment, capabilities, and strategic information as a result of the NATO Contribution Program
|2.3.1 Examples of equipment and capabilities available to DND/CAF over the last five years as a result of the Contribution Program||No||Yes||Yes||No||No|
|2.3.2 Examples of strategic information shared and received through NATO activities||No||Yes||Yes||No||No|
2.4 Immediate Outcome 4 – Support to deployed operations and missions
|2.4.1 DND/CAF participation in operations through NATO membership||No||Yes||Yes||Yes||No|
|2.4.2 DND/CAF participation in exercises through NATO membership||Yes||Yes||Yes||Yes||No|
2.5 Immediate Outcome 5–NATO Contribution Program is Well Managed
|2.5.1 Representation at NATO committees and DND Steering Committee||Yes||Yes||Yes||No||No|
|2.5.2 Level of awareness on the resources and effects of the Contribution Program||Yes||No||Yes||No||No|
|2.5.3 Effective knowledge transfer, and best practices mechanisms in place within DND/CAF, utilized and is accessible||No||No||Yes||No||No|
|2.5.4 Level of effectiveness of the management and reporting structure||No||Yes||Yes||No||Yes|
|2.6 Intermediate Outcome 1 - Inclusion, promotion, and support of Canadian industry at NATO||2.6.1 Support provided to Canadian industry by DND/CAF||No||Yes||Yes||No||Yes|
|2.6.2 Participation of Canadian industry in NATO contracts||Yes||Yes||Yes||No||Yes|
2.7 Intermediate Outcome 2– NATO Contribution Program activities improve interoperability with NATO countries
|2.7.1 Canada’s implementation rate for NATO Standards||Yes||Yes||No||No||No|
|2.7.2 Opinions on how to improve CAF’s interoperability through standardization||No||Yes||Yes||No||No|
2.8 Intermediate Outcome 3 –NATO Contribution Program activities demonstrate Canada's commitment to NATO
|2.8.1 Assessment of NATO funding throughout the Alliance||Yes||Yes||Yes||No||Yes|
|2.8.2 Qualitative and quantitative assessment of allied nation's/other experts’ opinions on Canada's contribution to NATO|
|2.8.3 Level of Canadian representation in NATO Command structure and International Military Staff||No||Yes||Yes||No||Yes|
Table D-2 Details - Evaluation Matrix - Performance: Achievement of Expected Outcomes (Effectiveness)
|Evaluation Issues/Questions||Indicators||Administrative and financial data||Literature and Document Review||Key Informant Interviews||Expert opinion||Comparison with allies|
3.1 Demonstration of efficiency
|3.1.1 Funding process for NATO and how it affects DND’s management of the program||Yes||Yes||Yes||No||No|
|3.1.2 Cost avoidance as a result of membership in NATO activities||Yes||Yes||Yes||No||No|
|3.2.3 Opportunities for further efficiencies and value for money||Yes||Yes||Yes||No||No|
3.2 Demonstration of economy
|3.2.1 NATO Contribution program budget in comparison to DND budget/trends in contributions to NATO||Yes||Yes||Yes||No||No|
|3.2.2 Cost share and comparison of Canada’s NATO contributions to other NATO countries||Yes||Yes||No||No||Yes|
|3.2.3 Perception that the benefits and outcomes of NATO activities are worth the resources allocated to the Program—value for money||No||No||Yes||Yes||No|
Table D-3 Details - Performance: Demonstration of Efficiency and Economy
Report a problem or mistake on this page
- Date modified: