Evaluation of the Defence Engagement Program (DEP)
PDF Version
Assistant Deputy Minister (Review Services)
May 2016
1258-3-001 (ADM(RS))
Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.
- ADM(Pol)
-
Assistant Deputy Minister (Policy)
- ADM(RS)
-
Assistant Deputy Minister (Review Services)
- CAF
-
Canadian Armed Forces
- CIHR
-
Canadian Institutes for Health Research
- DEP
-
Defence Engagement Program
- DND
-
Department of National Defence
- FTE
-
Full-time Equivalent
- FY
-
Fiscal Year
- ISROP
-
International Security Research and Outreach Program
- NATO
North Atlantic Treaty Organization
- TBS
Treasury Board Secretariat
- OGD
Other Government Department
- OPI
Office of Primary Interest
- PAA
Program Alignment Architecture
- SDF
Security and Defence Forum
- SOP
Standard Operating Procedure
- SSHRC
Social Sciences and Humanities Research Council
- NSERC
Natural Sciences and Engineering Research Council
Overall Assessment
- The DEP Grant Program has informed, confirmed and challenged policy thinking.
- The DEP contributes to a need for independent analysis and discussion of Canadian security and defence issues.
- Coordination between the DEP and external outreach programs of other government departments (OGD) and agencies has promoted a whole-of-government approach and economy of effort.
Overall Assessment
The DEP Grant Program has informed, confirmed and challenged policy thinking.
The DEP contributes to a need for independent analysis and discussion of Canadian security and defence issues.
Coordination between the DEP and external outreach programs of other government departments (OGD) and agencies has promoted a whole-of-government approach and economy of effort.
This report presents the findings and recommendations of the Evaluation of the Defence Engagement Program (DEP) Grant Program. The evaluation was conducted by Assistant Deputy Minister (Review Services) (ADM(RS)) between May and September 2015 in compliance with the Treasury Board Secretariat (TBS) Policy on Evaluation. As per the policy, the evaluation examined the relevance and the performance of the DEP Grant Program since its inception in 2012.
Background
The DEP facilitates access to external expertise that is intended to inform, confirm and challenge the Defence Team’s policy thinking on engagement priorities that are established and updated annually by the program’s Defence Team Steering Committee. The DEP was developed to build on the successes of the former Security and Defence Forum (SDF) Grant Program, which had over 40 years of engagement with the Canadian security and defence community.
Program Description
At its inception, the DEP comprised two initiatives: the Targeted Engagement Grant Program and the Fellowship and Scholarship Awards Program. An additional DEP initiative, the Expert Briefing Series, is funded with Assistant Deputy Minister (Policy) (ADM(Pol)) Vote 1 funds and is managed by the DEP Secretariat.
The Targeted Engagement Grants fund security and defence organizations, universities and individual experts to conduct conferences, workshops, roundtables and topical research projects. The Fellowship and Scholarship Awards Program provided a postdoctoral fellowship, and scholarships at the Master’s and PhD levels. The Awards Program was discontinued in fiscal year (FY) 2014/15. The Expert Briefing Series brings defence and security experts to the Department of National Defence (DND) to provide presentations and/or hold discussions with departmental officials about relevant, trending defence issues.
The DEP Grant Program is one component of a wide range of DND / Canadian Armed Forces (CAF) external engagement activities designed to strengthen the knowledge and policy thinking of the Defence Team and to support the public interest in Canadian security and defence issues.
Relevance
The DEP has worked to fill the departmental need for independent analysis and discussion of Canadian security and defence issues. The program, through the exchange of arm’s length research, analysis and perspectives on security and defence issues, has served the public interest and aligns with federal roles and responsibilities. The annual Defence Team Engagement Priorities have ensured that program activities align with federal government priorities and DND/CAF strategic outcomes.
Performance
The DEP was designed to assist the Defence Team to respond to the myriad of security and defence challenges facing Canada through two mandated objectives: (1) to support a strong Canadian knowledge base in contemporary security and defence issues that is relevant to Canadian defence policy and capable of informing, confirming and challenging defence policy thinking; and (2) to foster the next generation of security and defence scholars in the Canadian academic community.
With the cancellation of DEP Fellowship and Scholarship funding in 2014, the DEP is no longer fully achieving one of the grant program’s initial objectives: to foster the next generation of security and defence scholars. However, through the Targeted Engagement Grants, the DEP has promoted dialogue, established limited networks and supported the knowledge base of security and defence issues for Canadians. The Expert Briefing Series has provided an opportunity to acquire pertinent information, in a timely manner, on key security and defence policy issues. Both the Targeted Engagement Grants and the Expert Briefing Series have addressed the annual Defence Team Engagement Priorities and have contributed to increasing the understanding of new and emerging security and defence challenges. Nevertheless, there are opportunities to improve the dissemination of information resulting from the two DEP components. Moving forward, additional funding flexibility for the Targeted Engagement Grant program may also be beneficial to the DEP.
The DEP is managed in an effective and responsive manner. The coordination and consultation between the DEP Secretariat and corresponding external outreach programs offered by OGDs and agencies has promoted a whole-of-government approach and economy of effort.
Conclusion
The DEP has facilitated access to external expertise that has informed, confirmed and challenged the Defence Team’s policy thinking. The program has filled a departmental need for independent analysis and discussion of Canadian security and defence policy issues. The DEP effectively supports the Defence Team and should continue to build on its successes.
Key Findings and Recommendations
Relevance
Key Finding 1: The DEP contributes to a departmental need for independent analysis and discussion of Canadian security and defence issues.
Key Finding 2: The DEP is aligned with federal roles and responsibilities.
Key Finding 3: The DEP is consistent with federal government priorities and DND/CAF strategic outcomes.
Performance (Effectiveness)
Key Finding 4: The DEP provides a means to establish limited networks, promote dialogue and support the knowledge base of security and defence issues for Canadians.
Key Finding 5: There are opportunities to improve communication and the dissemination of information resulting from the Targeted Engagement Grants and the Expert Briefing Series, within the DND/CAF and throughout the security and defence community.
Key Finding 6: The Expert Briefing Series has provided an opportunity to acquire pertinent information on key security and defence policy issues in a timely manner.
Key Finding 7: With the cancellation of DEP Fellowship and Scholarship funding in 2014, the DEP is no longer fully achieving one of the initial program objectives: to foster the next generation of security and defence scholars.
Key Finding 8: The Targeted Engagement Grants and the Expert Briefing Series have each addressed the annual Defence Team Engagement Priorities and have contributed to increasing the understanding of new and emerging security and defence challenges.
Key Finding 9: The attribution of DEP activities to timely and relevant policy advice can be inferred, but is not quantifiable.
Performance (Efficiency and Economy)
Key Finding 10: The indirect costs required to administer the DEP are appropriate and are comparable to similar programs.
Key Finding 11: The coordination and consultation between the DEP Secretariat and external outreach programs offered by OGDs and agencies has promoted a whole-of-government approach and economy of effort.
Key Finding 12: The DEP application process is efficient and the DEP Secretariat is effective and responsive.
Key Finding 13: The funding limit for individual Targeted Engagement Grants is generally sufficient; however, additional funding flexibility would be beneficial to the DEP.
Recommendations
ADM(RS) Recommendation 1. In addition to the DEP Annual Report, establish and implement the means to disseminate information throughout the year on the results of DEP activities. Recipients should include the Defence Team Steering Committee, OGD representatives and the security and defence academic community.
ADM(RS) Recommendation 2. When seeking the renewal of the DEP, formally incorporate the Expert Briefing Series as a component of the program.
ADM(RS) Recommendation 3. Assess whether there is a need and tangible benefit to the DND/CAF and the Government of Canada to directly support the development of Canadian scholars in the fields of security and defence.
ADM(RS) Recommendation 4. Develop a feedback mechanism to assess the extent to which funded DEP activities contribute to better awareness and understanding of new and emerging security and defence challenges.
ADM(RS) Recommendation 5. To provide the DEP with additional funding flexibility, consider increasing the funding ceiling and funding period for a limited number of Targeted Engagement Grants.
Note: Please refer to Annex A—Management Action Plan for the management responses to the ADM(RS) recommendations.
1.0 Introduction
The evaluation of the DEP was conducted between May and September 2015. The evaluation was completed by ADM(RS) in accordance with TBS policies1 and approval authority,2 and it assesses the extent to which prescribed DEP objectives were achieved during the observation period (FY 2012/13 to FY 2014/15). As the DEP was established in 2012 as a successor to the former SDF Grant Program, this program has not previously been audited or evaluated by ADM(RS). This evaluation meets the requirement for the program to be evaluated prior to FY 2016/17, which is the end of the initial five-year funding period for the DEP.
1.1 Context for the Evaluation
An evaluation of the former SDF Grant Program was conducted by Chief Review Services3 in 2010.4 The evaluation’s overall general assessment was that the SDF program “continues to be relevant and successful in sustaining and expanding academic competence in Canada with respect to security and defence issues.” Separate from the 2010 Chief Review Services evaluation, the 2010 Strategic Review determined that the needs that led to the creation of the SDF had been fulfilled, and ADM(Pol) was directed to remodel the SDF Grant Program to better meet the evolving needs of the DND/CAF. The former SDF program expired in March 2012.
Based on the Strategic Review direction, DND launched the DEP in FY 2012/13 as a new approach to external engagement, aimed at providing a more nimble and flexible means to receive timely and relevant policy advice within a security environment that is fast paced, unpredictable and constantly evolving. As such, the DEP five-year program was designed to engage Canadian and international experts in academia, in addition to non-governmental organizations, think tanks, the private sector and other external organizations.
The purpose of this ADM(RS) evaluation is to examine both the relevance and performance of the DEP.
1.2.1 Program Description
The DEP facilitates access to external expertise that is intended to inform, confirm and challenge the Defence Team’s policy thinking. The program seeks timely, thought-provoking and multi-disciplinary insight from external experts on evolving and trending topics of direct policy relevance. The focus of DEP engagements with external experts is guided by the program’s Defence Team Engagement Priorities, which are established and updated annually.
The DEP was developed to build on the successes of the former SDF Grant Program, which had over 40 years of engagement with the Canadian security and defence community,5 and which had been established to support Canadian expertise in contemporary security and defence issues, foster informed public discussion and enhance communication between the academic community and the DND/CAF.
When the DEP commenced in FY 2012/13, the grant agreement comprised two initiatives supported through Vote 10 funds, as follows:
- Targeted Engagement Grant Program. These grants fund individual experts and security and defence organizations to conduct projects, including conferences, workshops, roundtables and topical research projects, that are directly relevant to the annual engagement priorities.
- Fellowship and Scholarship Awards Program. This program provided scholarships at the Master’s and PhD levels, as well as a postdoctoral fellowship, for selected scholars studying defence issues. The decision to discontinue this component of the DEP in FY 2014/15 was taken by the Defence Executive Policy Committee in 2014.
The DEP subsequently evolved to include an additional initiative that is not funded through the grant. The initiating documentation for the DEP states, “In the event that the Defence Team does not receive proposals on one of its engagement priorities, the program provides flexibility, through the use of contracts or grants, to organize events itself to meet targeted needs and objectives.” This opportunity to meet targeted needs and fill grant application gaps has become the Expert Briefing Series.
- Expert Briefing Series. This program brings defence and security experts and thinkers to DND to provide presentations and/or hold discussions with officials from the DND/CAF and OGDs and agencies about relevant, trending defence issues. While the Expert Briefing Series is managed by the DEP Secretariat and is categorized as a component of the DEP, it is funded through ADM(Pol) Vote 1 funds, and not through the annual $500,000 DEP grant.
1.2.2 Program Objectives
The DEP was designed to assist the Defence Team respond to the myriad of security and defence challenges facing Canada by supporting projects on multidisciplinary issues that affect the DND/CAF and its Government of Canada partners. At its inception, the DEP was mandated to achieve the following two key objectives:6
- support a strong Canadian knowledge base in contemporary security and defence issues that is relevant to Canadian defence policy and capable of informing, confirming and challenging DND/CAF defence policy thinking; and
- foster the next generation of security and defence scholars in the Canadian academic community.
The grant program is designed to be results-based, and therefore a performance measurement strategy was developed for the program in 2012. ADM(Pol) consolidates the performance measurement results annually and provides a roll-up of results in the DEP annual reports.
1.2.3 Stakeholders
The primary stakeholder of the DEP is the ADM(Pol) organization. The coordination of DEP activities is conducted by the DEP Secretariat within the Directorate of Public Policy in ADM(Pol). The Secretariat consists of one program manager, assisted by a director on a part-time basis. They are assisted by the following:
- A Defence Team Steering Committee is convened annually to refresh the Defence Team Engagement Priorities. The Committee includes a core group of representatives from the following DND/CAF groups: ADM(Pol), Vice Chief of the Defence Staff, the three Environmental Chiefs of Staff, Assistant Deputy Minister (Public Affairs), Chief of Military Personnel (Directorate of History and Heritage and the Canadian Defence Academy) and Assistant Deputy Minister (Science and Technology).
- An External Advisory Panel comprised of four experts on defence issues from Canada and abroad was appointed to assist the Department in evaluating applications for grants and academic awards. The engagement of the DEP External Advisory Panel was brought to an end in FY 2014/15 in order to streamline the grant evaluation and approvals process. This was tied, in part, to the discontinuation of the Fellowship and Scholarship Awards Program.
Within the DND/CAF, the DEP has provided research and briefings to military and civilian members of the Defence Team, up to the level of Deputy Minister. Externally, recipients of DEP funding include Canadian and international security and defence scholars and experts in academia, non-governmental organizations, Canadian and international think tanks and other external security and defence-related organizations.
1.3.1 Coverage and Responsibilities
The DEP links to Program Alignment Architecture (PAA) program 5.0 – Defence Capability Development and Research,7 which falls within the PAA strategic outcome of “Defence remains continually prepared to deliver national defence and defence services in alignment with Canadian interests and values.” The DEP supports PAA program 5.0 by providing knowledge to inform, confirm and challenge policy thinking. It does this through engagements that seek to anticipate changes in the security and defence environment and provide the means to assess international and national defence and security trends.
1.3.2 Resources
The total approved funding of the DEP is up to $2,500,000 Vote 10 (Grant and Contribution) funding over the five-year period between FY 2012/13 and FY 2016/17. The available allocation of Vote 10 funds is up to $500,000 annually and the program expended a total of $854,763 in the three years between FY 2012/13 and FY 2014/15. The Expert Briefing Series expended $54,862 between FY 2012/13 and FY 2014/15 from the ADM(Pol) Vote 1 budget.8
1.3.3 Issues and Questions
In accordance with the TBS Directive on the Evaluation Function (2009),9 the evaluation addresses the five core issues related to relevance and performance. The methodology used to gather evidence in support of the evaluation questions can be found at Annex B. An evaluation matrix listing each of the evaluation questions, with associated indicators and data sources, is provided at Annex D.
2.0 Findings and Recommendations
The following sections discuss the relevance and performance of the DEP. The evaluation examined the extent to which the program addresses a demonstrable need, is aligned with federal roles, responsibilities and priorities, achieves its expected outcomes and demonstrates efficiency and economy in resource utilization.
Relevance
2.1 Relevance—Continued Need
To determine whether the DEP continues to address a demonstrable need, the following key indicators were used:
- proportion of stakeholders who believe that the Targeted Engagement Grant projects serve a valid need in Canada’s security and defence stakeholder community; and
- proportion of stakeholders who believe that the Expert Briefing Series serves a valid need in Canada’s security and defence stakeholder community.
The following findings are based on evidence from key informant interviews and/or a stakeholder questionnaire engaging:
- program staff;
- DND/CAF stakeholders;
- recipients of Targeted Engagement Grants and/or participants in the Expert Briefing Series from the broader defence and security community; and
- program staff from comparable programs in OGDs and agencies.
Key Finding 1: The DEP contributes to a departmental need for independent analysis and discussion of Canadian security and defence issues.
In assessing the continued need for the DEP, the evaluation reviewed the activities of the program, specifically the Targeted Engagement Grants and the Expert Briefing Series, to determine if they support Canadians in general, the security and defence stakeholder community and the DND/CAF.
The stakeholder questionnaire and interview results indicated a strong consensus among program staff, stakeholders and academics that there remains a need for the DEP. The majority of interviewees believe strongly that DEP-related activities remain relevant and that the Targeted Engagement Grants have contributed to a greater awareness of and interest in security and defence issues by the Canadian public. The activities have led to broader debates on defence and security issues and continue to highlight key national and international issues that are of interest to Canadians. Specifically, the grants have directly contributed to enhancing the Canadian knowledge base on security and defence issues through conferences, media engagements and research papers.
The DEP grants have directly engaged the Canadian community of academics who have an interest in defence and security issues and have contributed to developing networks and partnerships within this community of interest, both nationally and internationally. As a result, the Canadian and international academic communities have provided independent analysis on defence and security issues for Canadians. Most importantly, given existing and emerging security challenges facing Canada and Canadians, the need for distinctive and independent Canadian research to inform and assist decision makers and policymakers remains extant. The Targeted Engagement Grants ensure that security and defence issues are addressed within the Canadian context and perspective. Through the modest investments in the short and medium term, the grants have also supplemented the expertise within the DND/CAF and have bridged knowledge gaps on key issues.
The Expert Briefing Series is designed to directly serve the immediate policy information and knowledge needs of ADM(Pol) and the DND/CAF in general. There was a strong consensus between the stakeholders and program staff interviewed that the Expert Briefing Series has provided an opportunity for the DND/CAF to acquire further knowledge about key security and defence issues and view them with new or differing perspectives. The knowledge conveyed through the briefing series is immediate, and stakeholders have engaged directly with the speakers to inform, confirm or challenge conventional thinking. Series speakers have briefed on key issues to multiple levels of leadership within the DND/CAF and to senior managers within DND and OGDs. The speakers have supplemented departmental policy expertise by also contributing knowledge to inform departmental and ministerial participation in specific high-level international symposia.
2.2 Relevance—Alignment with Federal Roles and Responsibilities
This section examines the extent to which the DEP aligns with federal roles and responsibilities.
The following indicator was used in this assessment:
- evidence of alignment between federal roles and responsibilities and the delivery of the DEP by DND.
The evaluation findings are based on evidence from document reviews, in addition to the validation of evidence through interviews with key informant stakeholders and/or a stakeholder questionnaire, including the following:
- program staff;
- DND/CAF stakeholders;
- recipients of Targeted Engagement Grants and/or participants in the Expert Briefing Series from the broader defence and security community; and
- program staff from comparable programs in OGDs.
Key Finding 2: The DEP is aligned with federal roles and responsibilities.
The DEP supports the exchange of ideas and serves the public interest. Through the Targeted Engagement Grants and the Expert Briefing Series, the DEP provides arm’s length research and analysis and independent perspectives to address important defence issues. Through an informed and engaged academic community, the DEP contributes to a better-informed Canadian public.
As DND is a lead department in the Canadian security and defence community, the Department is expected to develop well-informed defence policies, based on up-to-date analysis. As such, it is appropriate for the Department to solicit independent viewpoints through external engagement in order to supplement internal departmental capacity and improve defence policies. The DEP is an appropriate means to acquire this independent analysis as there are few other external sources for defence research support in Canada.
2.3 Relevance—Alignment with Government Priorities
This section examines the extent to which the DEP’s objectives were consistent with federal government priorities and DND/CAF strategic outcomes.
The following indicators were used to make this determination:
- evidence of alignment between DEP objectives and federal government priorities; and
- evidence of alignment between DEP objectives and DND/CAF strategic outcomes.
The observations and findings made in this section are based on evidence originating from official document reviews and interviews conducted for the evaluation.
Key Finding 3: The DEP is consistent with federal government priorities and DND/CAF strategic outcomes.
The DEP aligns with the extant defence policy10 by addressing emerging defence issues that are important to Canadians. In the PAA of 2013, the DEP falls under the departmental PAA strategic outcome: Defence remains continually prepared to deliver national defence and defence services in alignment with Canadian interests and values. Academic expertise derived from the range of DEP activities should contribute directly to understanding the strategic environment and facilitate appropriate policy planning.
The DEP has provided knowledge to inform, confirm and challenge policy thinking within the Department. To ensure alignment with current and emerging DND/CAF policy issues, DEP-funded projects address at least one of the Defence Team Engagement Priorities developed by the Defence Team Steering Committee.11 These program priorities, which are reviewed and established annually, ensure that funded projects are aligned with strategic priorities and departmental outcomes and remain relevant to the DND/CAF. For FY 2014/15, the Defence Team Engagement priorities focused on three areas, which were further defined by indicative areas of interest. The three priority areas were as follows:
- The Business of Defence in the Twenty-first Century
- Advanced Technology, New Domains and the Future of Warfare
- Canada’s Global Defence Engagement
Performance
2.4 Performance—Achievement of Expected Outcomes (Effectiveness)
The effectiveness of the DEP was assessed by measuring the extent to which program outcomes were achieved. The outcomes considered in this evaluation are identified in the program logic model (Annex C) as immediate, intermediate and ultimate program outcomes. These outcomes are as follows:
- Immediate outcomes. Support a strong Canadian knowledge base in contemporary security and defence issues that is relevant to Canadian defence policy and capable of informing, confirming and challenging DND/CAF defence policy thinking; and foster the next generation of security and defence scholars in the Canadian academic community;
- Intermediate outcome. Increase ability of the DND/CAF and the Government of Canada to identify implications of new global security and defence challenges as they arise; and
- Ultimate outcome. Timely and relevant policy advice on current security and defence issues is provided to the Department and the Government of Canada.
The observations and findings to assess the DEP’s effectiveness are based on evidence collected through document reviews, program data reviews, a questionnaire administered to program stakeholders, a comparative analysis with programs administered by OGDs and agencies and interviews with program staff, stakeholders and grant recipients.
2.4.1 Immediate Outcome – Support a strong Canadian knowledge base in contemporary security and defence issues that is relevant to Canadian defence policy and capable of informing, confirming and challenging DND/CAF defence policy thinking
The following performance measures were used to assess the achievement of this outcome:
- the extent to which Targeted Engagement Grant projects have supported (1) the Canadian knowledge base, and (2) the DND/CAF’s knowledge base in contemporary security and defence issues; and
- the extent to which the Expert Briefing Series has supported the knowledge base in contemporary security and defence issues.
Targeted Engagement Grants
To measure the extent to which the grants have supported the knowledge base of Canadians and the DND/CAF, the following quantitative indicators were used:
- number of funded engagement grant projects;
- proportion of engagement grant projects by type and recipient;
- proportion of engagement grant project applications funded;
- proportion of international engagement grant recipients who partnered with a Canadian institution and/or experts through their project;
- number of Canadian and international experts attending an engagement grant project event or otherwise participating in an engagement grant project;
- proportion of engagement grant projects that had media engagement and the number of media engagements per engagement grant project;
- proportion of DND/CAF stakeholders who believe that engagement grant projects establish links and promote research and dialogue with defence and security experts;
- number of DND/CAF personnel attending and/or participating in engagement grant project events; and
- number of publications resulting from engagement grants.
Key Finding 4: The DEP provides a means to establish limited networks, promote dialogue and support the knowledge base of security and defence issues for Canadians.
There is consensus among stakeholders that the DEP grants support the Canadian knowledge base by providing funds to workshops, conferences or research initiatives and result in connections between academic and government institutions. Specifically, the DEP provides an opportunity to establish links and promote dialogue between defence officials and security experts, in addition to creating opportunities for dialogue amongst security and defence experts themselves.
As shown in Table 1, during the first three years of the DEP, from FY 2012/13 to FY 2014/15, the program funded 51 of the 109 grant applications received and disbursed $596,018 in grant funding. Program staff reported the number of funded grant applications increased each year as the quality of applications improved.
Table 1. Applications for Grant Funding and Funded Applications. This table provides the number and status of funded grant applications by fiscal year.
Table Summary
This table provides the number and status of funded grant applications by fiscal year. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the number of grant applications, number of funded grant applications, percentage of funded applications and the total amount funded in dollars. The bottom row provides the sum of the amounts listed in each column.
FY | Grant Applications | Funded Grant Applications | Funded Applications | Amount Funded |
---|---|---|---|---|
2012/13 | 29 | 10 | 34% | $110,943 |
2013/14 | 45 | 15 | 33% | $156,905 |
2014/15 | 35 | 26 | 74% | $328,170 |
Total | 109 | 51 | 47% | $596,018 |
Of the 51 funded Targeted Engagement Grants, almost two thirds of the grants (32 of 51) were awarded to universities, with the remainder awarded to non-governmental organizations and think tanks. Forty-nine percent of the funded applications involved events only, such as conferences, symposia or roundtable consultations. Another 49 percent of successful grant applications involved events combined with associated publications and/or research projects. Two percent of successful grant applications involved only publications / research projects.
To assess the extent to which these grants contribute to a strong Canadian knowledge base in contemporary security and defence issues, the evaluation conducted interviews with stakeholders and grant recipients, reviewed participation levels at events held from FY 2012/13 to FY 2014/15 and reviewed the number of media engagements related to these events and the number of publications resulting from the events.
One of the conditions of Targeted Engagement Grant funding specifies that Defence personnel may attend the events funded through the DEP. As shown in Table 2, at least 206 DND/CAF personnel have attended funded grant events since FY 2012/13 (noting partial FY 2014/15 data), resulting in an average of almost eight DND/CAF attendees per event. Attendance provides an opportunity to network with national and international subject matter experts and facilitates the transfer of information and knowledge, in addition to what may be provided through post-event media engagement.
Table 2. DND/CAF Attendees to Funded Grant Projects. This table provides the number of DND/CAF attendees to funded grant projects by fiscal year.
Table Summary
This table provides the number of DND/CAF attendees to funded grant projects by fiscal year. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the number of grant projects, the number of DND/CAF participants and the average number of DND/CAF participants. The bottom row provides the sum of the amounts listed in each column.
FY | Grant Projects | DND/CAF Participants | Average Number of DND/CAF Participants |
---|---|---|---|
2012/13 | 10 | 39 | 3.9 |
2013/14 | 10 | 49 | 3.3 |
2014/15 (*) | 15 of 26 | 118 | 7.9 |
Total | 40 | 206 | 5.2 |
*Note: For FY 2014/15, data were only available for some of the funded projects (58 percent of the total). Statistical data collection for most of the projects awarded funding in FY 2014/15 was still ongoing at the time of evaluation data gathering.
As shown in Table 3, available program data indicate that between FY 2012/13 and FY 2013/14, an increasing number of Canadian and international security and defence experts participated in the funded grant events. The extrapolation of available data for FY 2014/15 also continues this trend. Since FY 2012/13, 367 Canadian experts and 373 international experts participated in 40 funded grant events.
Table 3. Experts Participating in Targeted Engagement Grant Events. This table provides the number of Canadian and international experts participating in DEP Targeted Engagement Grant events.
Table Summary
This table provides the number of Canadian and international experts participating in DEP Targeted Engagement Grant events. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the number of grant projects, the number of Canadian experts, the number of international experts, the average number of Canadian experts per project and the average number of international experts per project. The bottom row provides the sum or the average of the amounts listed in each column.
FY | Grant Projects (with data available) | Canadian Experts | International Experts | Average Number of Canadian Experts per Project | Average Number of International Experts per Project |
---|---|---|---|---|---|
2012/13 | 10 of 10 | 54 | 38 | 5.4 | 3.8 |
2013/14 | 15 of 15 | 175 | 181 | 11.7 | 12.1 |
2014/15 (*) | 15 of 26 | 138 | 154 | 9.2 | 10.3 |
Total | 40 | 367 | 373 | 9.2 | 9.3 |
*Note: For FY 2014/15, data were only available for 15 of the 26 funded projects (58 percent of the total). Statistical data collection for projects that were awarded funding in FY 2014/15 was still ongoing at the time of evaluation data gathering.
In addition, program data indicate that 16 percent (8 of 51) of grant recipients were international organizations and 87.5 percent of these recipients (seven of eight) partnered with a Canadian institution and/or Canadian security and defence experts during the project. The majority of stakeholders and grant recipients confirmed that the DEP provides the academic community with opportunities to develop interdisciplinary teams and communities of practice within academic institutions in Canada and the United States and with the broader defence and security community. However, due to the “single project relationship” between grant recipients and the DEP, some academic grant recipients reported it is a challenge to develop enduring networks. Grant recipients reported that many of the existing and enduring defence-related academic networks were initiated during the former SDF program.
Notwithstanding this mutually beneficial networking opportunity, many stakeholders and grant recipients perceive that the links and level of dialogue between the academic community and DND are not as strong as they were under the former SDF program and continue to wane. There was broad consensus from stakeholders and members of the academic community that the DEP only promotes a limited “transactional relationship” between the Department and the security and defence community. They also opined that under the DEP, academic collaborations are now more regionalized, and the DEP program does not foster the pan-Canadian conversation of defence and security issues within the Canadian academic community that was present under the SDF program.
As shown in Table 4, 51 percent of grant projects reported a related media engagement, and the average number of media engagements per project has increased significantly each year during the first three years of the DEP. Media engagement refers to known mentions and/or coverage by major media organizations, as well as press conferences and media releases, such as articles, web articles by major news outlets, television news coverage and radio coverage as self-reported by the grant recipients. In FY 2014/15 there were, on average, four media engagements reported per project.12
Table 4. Media Engagements for Targeted Engagement Grant Projects or Events. This table provides the number of media engagements by fiscal year reported for Targeted Engagement Grants.
Table Summary
This table provides the number of media engagements by fiscal year reported for Targeted Engagement Grant projects or events. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the number of grant projects, the total number of media engagements and the average number of media engagements per project. The bottom row provides the sum or the average of the amounts listed in each column.
FY | Grant Projects | Total Media Engagements | Average Number of Media Engagements per Project |
---|---|---|---|
2012/13 | 10 | 3 | 0.3 |
2013/14 | 15 | 40 | 2.7 |
2014/15 (*) | 15 of 26 | 60 | 4.0 |
Total | 40 | 103 | 2.6 |
*Note: For FY 2014/15, data were only available for 15 of the 26 funded projects (58 percent of the total). Statistical data collection for most of the projects awarded funding in FY 2014/15 was still ongoing at the time of evaluation data gathering.
Key Finding 5: There are opportunities to improve communication and the dissemination of information resulting from the Targeted Engagement Grants and the Expert Briefing Series within the DND/CAF and throughout the security and defence community.
The sharing of information and knowledge derived from DEP activities should be a key aspect of the program. Most notably, after the end of each fiscal year, the DEP produces and distributes a comprehensive annual report that provides a synopsis of the previous year’s activities for each program component: the Targeted Engagement Grant Program; the Fellowship and Scholarship Awards Program (until FY 2013/14); and the Expert Briefing Series. Many departmental stakeholders and grant applicants, however, identified room for improvement in the dissemination of DEP-related information during the year. Some members of the Defence Team Steering Committee and other personnel who evaluated Targeted Engagement Grant Program applications indicated they would appreciate receiving notification of the successful applications at the end of each round of grant funding assessments. On the other hand, some unsuccessful grant applicants indicated they would appreciate feedback on why their application proposals were not selected.
The dissemination of funded project event reports or synopses, research papers and publications can contribute to an improved knowledge base in contemporary security and defence issues. The DEP Standard Operating Procedures (SOP) for Grants13 provides a list of financial and administrative data requirements that are to be submitted no later than 60 days after the completion of the project or event. This document indicates that grant recipients must identify how the funded project furthered the public policy debate on the Defence Team’s Engagement Priorities that were developed for that particular round of Targeted Engagement Grant funding. This latter requirement could be further emphasized with grant recipients, and the DEP Secretariat could distribute the resulting synopses periodically during the year to keep the Defence Team Steering Committee members and others informed, as applicable.
As shown in Table 5, since FY 2012/13, the DEP Secretariat received eight publications resulting from funded engagement grants. However, only five of the eight publications received were distributed as there was no procedure for the distribution of publications until FY 2013/14.
Table 5. Publications Expected, Received and Distributed. This table provides the status by fiscal year of publications expected, received and distributed.
Table Summary
This table provides the status by fiscal year of publications expected, received and distributed. The fiscal years are indicated in the left hand column. For each fiscal year, read across the row to determine the number of publications expected, number received, publications distributed, percentage distributed and number pending. The bottom row provides the sum of the amounts listed in each column.
FY | Publications Expected | Received | Publications Distributed | Percent Distributed | Pending |
---|---|---|---|---|---|
2012/13 | 5 | 4 | 1 | n/a | 1 |
2013/14 | 8 | 3 | 3 | 100 | 5 |
2014/15 | 12 | 1 | 1 | 100 | 11 |
Total | 25 | 8 | 5 | - | 17 |
Since that date, all publications received by the DEP Secretariat have been distributed. However, as of October 2015, 17 publications are still pending, of which 4 are overdue. As some members of the Defence Team Steering Committee reported that they had not received any products resulting from funded events, the DEP Secretariat should ensure that the resulting products from Targeted Engagement Grants are received on time and distributed widely.
ADM(RS) Recommendation
1. In addition to the DEP Annual Report, establish and implement the means to disseminate information throughout the year on the results of DEP activities. Recipients should include the Defence Team Steering Committee, OGD representatives and the security and defence academic community.
OPI: ADM(Pol)
Expert Briefing Series
To measure the extent that the DEP Expert Briefing Series has strengthened the knowledge base in contemporary security and defence issues, the following indicators were used:
- number of experts engaged;
- number of DND/CAF personnel attending expert briefing sessions;
- number of DND/CAF executives attending expert briefing sessions;
- average time in days for expert briefing reports to be prepared and distributed within the DND/CAF;
- proportion of DND/CAF stakeholders who believe that hosting expert briefing sessions establishes links and promotes dialogue with defence and security experts; and
- proportion of staff from OGDs attending DEP-sponsored expert briefing sessions.
Key Finding 6: The Expert Briefing Series has provided an opportunity to acquire pertinent information on key security and defence policy issues in a timely manner.
There was strong consensus from stakeholders and ADM(Pol) staff that the Expert Briefing Series established links between defence officials and academic experts, and that it promoted knowledge sharing on a variety of topics related to key security and defence issues within the thematic areas of the Engagement Priorities and with direct policy relevance to the Department. Stakeholders agreed strongly that the Expert Briefing Series is a successful initiative, and has provided an opportunity for the DND/CAF to acquire relevant knowledge about key security and defence issues in a timely manner.
During the first three years of the DEP, 30 expert speakers were engaged for the DEP expert briefings, and these briefings were attended by over 600 DND/CAF personnel. Briefings engaged a broad range of personnel, such as the Deputy Minister of National Defence, senior executives,14 military officers and senior analysts. As shown in Table 6, the number of departmental executives attending the expert briefings was notable and averaged 37 percent of the total participation in both FY 2013/14 and FY 2014/15. Departmental executives were able to directly engage with the briefers, thereby increasing their understanding of key security and defence issues. Stakeholders remarked that the FY 2014/15 briefing sessions related to the themes of European and transatlantic security, Asia-Pacific defence and security issues and cyber security, as well as the future of the North Atlantic Treaty Organization (NATO) and Canada’s role in the Alliance were particularly valuable to departmental officials.
Table 6. DND/CAF Personnel Attending Expert Briefings. This table provides the number of DND/CAF attendees to the expert briefings by fiscal year.
Table Summary
This table provides the number of DND/CAF personnel attending the expert briefings by fiscal year. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the number of expert speakers, the number of expert briefings, the number of DND/CAF personnel attending the expert briefings and the number of executives attending the expert briefings. The bottom row provides the sum of the amounts listed in each column.
FY | Expert Speakers | Expert Briefings (*) | DND/CAF Personnel Attending Expert Briefings | Executives Attending Expert Briefings |
---|---|---|---|---|
2012/13 | 12 | 10 | 209 | n/a |
2013/14 | 8 | 7 | 175 | 65 |
2014/15 | 10 | 9 | 224 | 82 |
Total | 30 | 26 | 608 | 147 |
*Note: This is the number of briefings for which data were available. One expert briefing in FY 2013/13 and one in FY 2014/15 had no data available.
A total of 36 representatives from OGDs and agencies15 attended over 80 percent of applicable briefings in FY 2013/14 and FY 2014/15.16 Based on the subject matter presented at the expert briefings, 56 percent of briefings were of relevance to Global Affairs Canada staff, and 45 percent were judged to be of interest to Privy Council Office staff. Less than one third of expert briefings were relevant to Public Safety Canada staff, while only 2 of 16 were deemed in the general area of interest for Canadian Security Intelligence Service staff.
To disseminate information within the DND/CAF on each briefing session, the DEP Secretariat prepared reports after each of the expert briefings. In FY 2014/15, these reports were distributed quickly, on average within 3.3 working days.17 Notwithstanding the timely distribution of expert briefing reports, the DEP Secretariat should review the distribution structure to ensure the reports are distributed widely.
ADM(RS) Recommendation
2. When seeking the renewal of the DEP, formally incorporate the Expert Briefing Series as a component of the program.
OPI: ADM(Pol)
2.4.2 Immediate Outcome – Foster the next generation of security and defence scholars in the Canadian academic community
The following indicators were used to assess the achievement of this outcome:
- number of fellowships and scholarships funded per year; and
- proportion of available fellowships and scholarships funded per year.
Key Finding 7: With the cancellation of the DEP Fellowship and Scholarship funding in 2014, the DEP is no longer fully achieving one of the initial program objectives: to foster the next generation of security and defence scholars.
The DEP Fellowship and Scholarship Awards Program was initiated in FY 2012/13. This program was developed to increase links with defence and security experts and subsequently to develop a cadre of defence-oriented graduates capable of being engaged in the security and defence policy environment. The program was designed and funded for six awards annually for a total available annual funding of $135,000. The number of scholarships awarded under each category varied annually, depending upon the quality of applications.
As shown in Table 7, during the two-year period from 2012 to 2014, the DEP awarded a total of nine fellowships and scholarships to Canadian defence scholars undertaking defence-related research at Canadian or accredited international academic institutions. In 2014, the decision was taken by the Defence Executive Policy Committee to discontinue the fellowship and scholarship component of the DEP.
Table 7. DEP Scholarships and Fellowships Awarded from FY 2012/13 through to FY 2014/15. This table provides the number of funded scholarship and fellowship awards by fiscal year compared to the number of available awards.
Table Summary
This table provides the number of funded scholarship and fellowship awards by fiscal year compared to the number of available awards. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the number of funded fellowships and scholarships, the number of available fellowships and scholarships, and the percentage of fellowships and scholarships funded each year. The bottom row provides the sum or the average of the amounts listed in each column.
FY | Fellowships and Scholarships Funded | Fellowships and Scholarships Available | Fellowships and Scholarships Funded |
---|---|---|---|
2012/13 | 4 | 6 | 67% |
2013/14 | 5 | 6 | 83% |
2014/15 | n/a | n/a | n/a |
Total | 9 | 12 | 75% |
During the conduct of this evaluation, unsolicited comments from some academic grant recipients and stakeholders within the Department raised concerns about the cancellation of the DEP Fellowship and Scholarship Awards Program. They noted that the fellowship and scholarship component of the DEP contributed towards achieving a viable and robust security and defence academic community in Canada, with a depth of knowledge and experience that can provide an independent Canadian perspective.
The 2010 Evaluation of the SDF noted that the former program maintained scholarly exposure to defence issues and engaged the Canadian academic community to generate information and opinion on security and defence issues. Findings of that evaluation supported the assertion that despite security and defence-related courses being popular among undergraduate students, security and defence issues had long been marginalized within the academic community – particularly the political science community. During this evaluation, the academic grant recipients and stakeholders also noted that research and university-based programming on security and defence issues in Canada continues to decline. They stated that potential security and defence graduate students are now seeking funding opportunities internationally, as the DEP fellowships and scholarships were the primary vehicle for fostering new generations of graduate-level students in security and defence studies in Canada. There are few other opportunities to receive Canadian defence-related post-graduate scholarships or fellowships. While some other components of the federal government are involved directly or indirectly in providing scholarship award programs, a scan of Government of Canada websites clearly indicates that the vast majority of federal government research and scholarship award programs primarily focus on the social sciences and humanities, with a limited number supporting programs within natural sciences and engineering.18
With the cancellation of DEP fellowship and scholarship funding in 2014, the DEP no longer directly fosters the next generation of security and defence scholars and is no longer fully achieving this initial DEP objective. It was beyond the scope of this evaluation to assess the impact of the cancellation of the fellowship and scholarship funding, and whether the near-term and long-term health of the security and defence academic community in Canada has the capacity and depth of expertise to meet the immediate and future needs of the DND/CAF, the Government of Canada and Canadians. The Department must assess the potential long-term effects of cancelling the DEP Fellowship and Scholarship Awards Program and determine whether there is a need and tangible benefit to the DND/CAF and the Government of Canada to directly support the development of Canadian scholars in the fields of security and defence.
ADM(RS) Recommendation
3. Assess whether there is a need and tangible benefit to the DND/CAF and the Government of Canada to directly support the development of Canadian scholars in the fields of security and defence.
OPI: ADM(Pol)
2.4.3 Intermediate Outcome – Increase ability of the DND/CAF and the Government of Canada to identify implications of new global security and defence challenges as they arise
To measure the extent to which this immediate outcome was achieved, the following indicators were used:
- coverage of Defence Team Engagement Priorities by DEP program activities;
- proportion of DND/CAF stakeholders who believe that (1) the Targeted Engagement Grant projects and (2) the Expert Briefing Series have contributed to increasing the Department’s ability to understand new and emerging security and defence challenges; and
- proportion of stakeholders from OGDs who believe that the Expert Briefing Series has contributed to understanding new and emerging security and/or defence challenges.
Key Finding 8: The Targeted Engagement Grants and the Expert Briefing Series have each addressed the annual Defence Team Engagement Priorities and have contributed to increasing the understanding of new and emerging security and defence challenges.
As outlined in the DEP SOPs for Grants, projects funded through DEP grants must address at least one of the Defence Team Engagement Priorities established each fiscal year. While it is not mandatory for an application to address a specific topic listed under each priority, the SOPs stipulate that at least one priority must be generally addressed in order for a project to receive DEP grant funding.
A review of the available program data for FY 2012/13 through to FY 2014/15 indicates that each of the Defence Team Engagement Priorities was covered by the Targeted Engagement Grants and the Expert Briefing Series together. As shown in Table 8, Targeted Engagement Grants have often covered multiple priorities. During the first three years of the DEP, all Defence Team Engagement Priorities were addressed by Targeted Engagement Grants and 40 percent of individual grants covered more than one priority.
Table 8. Grants Awarded by Fiscal Year, Covering Each Defence Team Engagement Priority. This table provides the Defence Team Engagement Priorities by fiscal year and the number of grants that cover each of those priorities.
Table Summary
This table provides the Defence Team Engagement Priorities by fiscal year and the number of grants that cover each of those priorities. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine each of the three Defence Team Engagement Priorities and the number of grants / percentage of coverage for each priority.
FY | Priority 1 | Grants (%) | Priority 2 | Grants (%) | Priority 3 | Grants (%) |
---|---|---|---|---|---|---|
2012/13 | New Context for Defence | 4 of 10(40%) | Global Security Environment and Trends; Advanced Technology and New Domains | 9 of 10(90%) | Canada’s Defence Relations with the United States | 2 of 10(20%) |
2013/14 | New Context for Defence | 4 of 15(27%) | Global Security Environment and Trends; Advanced Technology and New Domains | 12 of 15(80%) | Canada’s Defence Relations with the United States | 6 of 15(40%) |
2014/15 | Business of Defence in 21st Century | 13 of 26(50%) | Advanced Technology and New Domains; Future of Warfare | 12 of 26(46%) | Canada’s Global Defence Engagement | 20 of 26(77%) |
The DEP Expert Briefing Series Forward Plan provides recommendations for expert briefers during the fiscal year. However, the Expert Briefing Series also permits the engagement of experts on an ad hoc basis, outside of the Forward Plan but within the thematic areas of the Engagement Priorities. From FY 2012/13 through to FY 2014/15, the Expert Briefing Series addressed the same Engagement Priorities as the grants, albeit with a differing emphasis and perspective. During these three years, 30 percent of individual expert briefings also covered more than one Defence Team Engagement Priority.
It is assessed that the Expert Briefing Series has provided the Department with a mechanism to supplement departmental expertise and address global security and defence challenges as they arise. There was consensus that the timely transfer of knowledge or expertise has been of direct and immediate benefit to departmental officials. The extent to which the DEP has increased the ability of the DND/CAF, and in turn, the Government of Canada, to identify implications of new global security and defence challenges as they arise is only supported through anecdotal evidence. Many stakeholders suggested that there should be a formal feedback mechanism to verify the utility or impact of the funded engagements. Some program staff and stakeholders suggested that improved dissemination of information resulting from the Expert Briefing Series, and the broader distribution of research reports or post-event synopses from Targeted Engagement Grant events could increase awareness and improve understanding of new and emerging security and defence challenges.
ADM(RS) Recommendation
4. Develop a feedback mechanism to assess the extent to which funded DEP activities contribute to better awareness and understanding of new and emerging security and defence challenges.
OPI: ADM(Pol)
2.4.4 Ultimate Outcome – Timely and relevant policy advice on current security and defence issues is provided to the Department and the Government of Canada
The following indicator was used to assess the achievement of this outcome:
- proportion of DND/CAF stakeholders for whom defence policy thinking has been informed by (1) a Targeted Engagement Grant project, and/or (2) an Expert Briefing Series event.
Key Finding 9: The attribution of DEP activities to timely and relevant policy advice can be inferred, but is not quantifiable.
The majority of stakeholders and ADM(Pol) staff indicated it is difficult to measure and assess the transfer of knowledge. They acknowledged that the Expert Briefing Series and Targeted Engagement Grants could inform or influence considerations or decisions when developing defence and security policy, but that it was very difficult to directly attribute policy advice to specific DEP activities or products. However, stakeholders noted that DEP-funded conferences provide background knowledge on important issues and can contribute to professional development within the Department and within the broader defence community. Other Targeted Engagement Grant events can promote knowledge transfer on specific issues, such as the grant to Chatham House (London), which resulted in a research paper and a briefing to departmental staff prior to a NATO ministerial meeting.
The majority of stakeholders agreed that the Expert Briefing Series has contributed directly to the general professional knowledge of departmental staff. The series has also targeted specific areas of security and defence interest. Expert engagements that focused on issues pertaining to Russia and Syria have informed and shaped departmental thinking up to senior levels. In addition, a multi-expert executive briefing panel, which was convened in 2015 to inform senior officials and policy staff prior to the Minister’s attendance at the International Institute for Strategic Studies 14th Asia Security Summit, was deemed effective and influential. While funded DEP activities may directly inform or influence policy considerations and decisions, the attribution of specific DEP activities to timely and relevant policy advice can be inferred, but is not quantifiable.
2.5 Performance—Demonstration of Efficiency and Economy
The approach utilized to demonstrate efficiency and economy of the DEP addressed the following evaluation questions:
- Is the DND/CAF delivering the DEP in an efficient and economical manner? What funds are being expended?
- Are there alternative, perhaps more economical, ways of delivering the DEP?
- Is the progress made toward expected outcomes adequate for the resources expended?
To assess the efficiency and economy of the DEP, the evaluation analyzed trends in expenditures, staffing levels and program administration from FY 2012/13 to FY 2014/15. This assessment was based on information provided from the DEP annual reports, program internal monitoring statistics, and data from the Defence Resource Management Information System.
Key Finding 10: The indirect costs required to administer the DEP are appropriate and are comparable to similar programs.
Expenditure of Funds to Deliver the Program
During the first three years of the DEP, the program budget was, on average, 1.5 percent of the overall ADM(Pol) budget, spending $821,018 Vote 10 funds for the Targeted Engagement Grants and the academic awards. A further $54,862 was spent on the Expert Briefing Series from ADM(Pol) Vote 1 funds, bringing overall spending to $875,880, or an average of $292,000 annually. During these three years, the DEP Vote 10 spending increased by 45 percent, from $225,000 in FY 2012/13 to $328,170 in FY 2014/15.
As shown in Table 9, a total of $596,018 was expended between FY 2012/13 and FY 2014/15 on the Targeted Engagement Grant Program. Over that time period, the number of funded grant applications increased from 10 to 26, with a concomitant 200 percent increase in grant expenditures. The average annual funding per grant remained relatively constant and ranged from $10,460 to $12,622 during the three years.
Table 9. Funded Targeted Engagement Grant Applications and Funding Amounts by Fiscal Year. This table provides the number of Targeted Engagement Grant applications by fiscal year and the funding attributed to those grants.
Table Summary
This table provides the number of Targeted Engagement Grant applications by fiscal year and the funding attributed to those grants. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the number of funded applications, the total amount for the grants and the average funding per grant. The bottom row provides the sum or the average of the amounts listed in each column.
FY | Funded Applications | Total Amount for Grants | Average Amount per Grant |
---|---|---|---|
2012/13 | 10 | $110,943 | $11,094 |
2013/14 | 15 | $156,905 | $10,460 |
2014/15 | 26 | $328,170 | $12,622 |
Total | 51 | $596,018 | $11,687 |
As shown in Table 10, the Fellowship and Scholarship Awards Program expended $225,000 between FY 2012/13 and FY 2013/14, and the Expert Briefing Series expended $54,862 between FY 2012/13 and FY 2014/15. The Terms and Conditions for the DEP Vote 10 grant allocated $265,000 for the Targeted Engagement Grant Program and $135,000 for the Fellowship and Scholarship Awards Program. The discontinuation of academic awards in FY 2014/15 allowed for those Vote 10 funds to be allocated to the Targeted Engagement Grant program. By FY 2014/15, the DEP was expending 82 percent of available Vote 10 funds.
Table 10. DEP Expenditures by Year (FY 2012/13 to FY 2014/15). This table provides the total funding provided for each of the program components by fiscal year.
Table Summary
This table provides the total funding for each of the program components by fiscal year. The fiscal years are indicated in the left-hand column. For each fiscal year, read across the row to determine the total funding expended for Targeted Engagement Grants, the total expended for Fellowship and Scholarship Awards, the total funding expended for the Expert Briefing Series, and the total funding expended by fiscal year. The bottom row provides the sum of the amounts listed in each column.
FY | Total Amount Expended for Targeted Engagement Grants | Total Amount Expended for Fellowship and Scholarship Awards | Total Amount Expended for the Expert Briefing Series | Total Expended |
---|---|---|---|---|
2012/13 | $110,943 | $115,000 | $19,778 | $245,721 |
2013/14 | $156,905 | $110,000 | $18,197 | $285,102 |
2014/15 | $328,170 | 0 | $16,887 | $345,057 |
Total | $596,018 | $225,000 | $54,862 | $875,880 |
For the two years that the Fellowship and Scholarship Awards Program was offered, 83 percent of allocated awards program funding was expended, for a total expenditure of $225,000 Vote 10 funds. In FY 2012/13, 15 applications were received and four successful applications were funded for an average award of $28,750 and a total expenditure of $115,000. In FY 2013/14, the number of applications doubled to 30, and five fellowships/scholarships were granted, for an average award of $22,000 and a total expenditure of $110,000.
For the Expert Briefing Series, the expenditure of $54,862 Vote 1 funds provided an average cost of $18,287 per year. The ADM(Pol) Business Plan 2015/16 allocated $35,000 ADM(Pol) Vote 1 funds for the briefing series. The average cost per speaker over these three years, which includes speaker fees and hospitality costs, was $2,095 and ranged between $1,728 in FY 2012/13 to $2,412 in FY 2014/15.19
The DEP Secretariat is designated 1.25 full-time equivalent (FTE) positions for the day-to-day management of the program, as defined within the program’s Performance Measurement Strategy. This consists of one program manager (EC-05), assisted by the Director Public Policy on a quarter-time basis. The 1.25 FTE positions incur an approximate annual salary cost of $120,000. As such, the ratio of the salary expenditures or indirect costs20 of the DEP Secretariat to manage the program has decreased from 49 percent and now approaches 35 percent of the value of funds disbursed. This ratio is comparable to Canadian university and college research centres, which average 40 percent.21 The trend line comparing administrative expenses to funds disbursed continues to improve as the program increases program spending due to increased numbers of funded Targeted Engagement Grant applications.
In 2015, due to the DEP’s higher level of activity and increased coordination requirements, the program used an additional FTE (EC-05) from Director Public Policy on an “as required basis” to deliver on specific major projects, or when two or more priority events occurred at the same time.22 To sustain an ongoing high level of DEP activities, similar to what was achieved in the first half of 2015, it is assessed that an FTE capacity of 1.5 policy officers, plus a portion of the FTEs provided by the Director Public Policy (EX-01) and the directorate’s Administrative Assistant (AS-02), will be required. The part-time engagement of a staff member is assessed to be an economical and appropriate measure to support the program’s increased coordination requirements.
The funds expended to administer the DEP and for the DEP activities are easily measured. However, the value of the DEP program and the extent to which it informs programs, activities or policies within the DND/CAF and across government may be significant but cannot be quantified.
Delivering the Program with a Whole-of-Government Approach
Key Finding 11: The coordination and consultation between the DEP Secretariat and external outreach programs offered by OGDs and agencies has promoted a whole-of-government approach and economy of effort.
During the program’s three-year time frame, the expert briefings were regularly attended by officials from OGDs and agencies. As a tangible example of creating efficiencies, the coordination and consultation between the DEP and the external outreach programs offered by the OGDs and agencies facilitated the sharing of expert briefers when appropriate. Nine of the 29 experts (30 percent) engaged by the DEP were shared with OGDs and six experts were received from OGDs to brief DND/CAF officials. This coordination has been enabled by regular contact between the various departments’ program staffs, who have reviewed engagement priorities and plans to minimize the potential for duplication and to support complementary activities. In addition, some DEP grant applications are provided to officials of the OGDs and agencies for assessment.
Comparison with OGDs
The management of the DEP is comparable to the support provided for similar external outreach programs. Global Affairs Canada, Public Safety Canada and the Canadian Security Intelligence Service, for example, each offer external outreach programs that are complementary to each other and the DEP. The following programs serve as a comparative benchmark:
- ISROP, coordinated by Global Affairs Canada, informs and supports the development of Canada’s international security policy by drawing upon the expertise of academic and think tank communities in Canada and abroad.23 This program generally expends $75,000 to $100,000 Vote 1 funds annually and is coordinated by one staff officer. ISROP expenditures are nominally allocated to an Annual Conference (20-25 percent), commissioned research (50 percent), the Graduate Research Awards event (10 percent) and Fast Talks or “virtual roundtable” events (15 percent); and
- The Kanishka Project is a five-year, $10-million contribution program administered by Public Safety Canada. This initiative invests Vote 10 funds in research on pressing issues about terrorism and counter-terrorism and is intended to produce more effective policies, tools and resources for law enforcement and other officials. The Kanishka Project also supports other activities necessary to build knowledge and create networks of researchers and students.24 The Kanishka Project is staffed by one program coordinator who conducts most of the analytical and liaison work and is supported by a finance administrator (AS-02) and a project manager (EC-04).
Some stakeholders suggested that the DEP should investigate a closer relationship with the granting organizations of NSERC, CIHR and SSHRC as an alternative means of program delivery, while others decried the administrative processes of those organizations and noted that the Department would likely have to relinquish control of the DEP. They suggested that, at present, the DEP ensures departmental needs are met in an effective and responsive manner.
Program Administration
Key Finding 12: The DEP application process is efficient and the DEP Secretariat is effective and responsive.
Key Finding 13: The funding limit for individual Targeted Engagement Grants is generally sufficient, however additional funding flexibility would be beneficial to the DEP.
Recipients of Targeted Engagement Grants indicated that with the introduction of a streamlined review process, the review of DEP applications has become more efficient and is very efficient compared to the majority of other national and international granting institutions.25 Most significantly, stakeholders and grant recipients alike reported that the DEP Secretariat was effective and very responsive to program queries and administrative requirements.
Officials who have been asked to review grant applications, such as members of the Defence Team Steering Committee and representatives of OGD and agency grant programs, all agreed that the grant review process was not onerous. While some grant proposals require a more comprehensive review,26 interviewees indicated that each application generally took less than one hour to review and score. Departmental stakeholders generally agreed that the time and effort expended on administering the Targeted Engagement Grants was worthwhile. They did suggest that the program might consider a scalable review process, adjusting the number of reviewers to the value of the requested grant. The dissolution of the DEP External Advisory Panel reduced the review process timeline, which contributed to a more efficient review process.
The Terms and Conditions of the Targeted Engagement Grant Program indicate the maximum payable is $25,000 per event and that proposals may receive partial or full funding depending on the assessment of the application. Members of the academic community and some departmental stakeholders agreed that the limit of $25,000 is generally more than sufficient for a specific targeted event or a specific research project. Most Targeted Engagement Grant funding recipients, however, indicated a desire to see the DEP adopt more funding flexibility, with an opportunity to increase funding limits27 or funding periods (e.g., two years) for specific projects, with pre-established objectives or outputs. Increased funding opportunities could attract a broader range of applicants and longer-term projects could contribute to building networks and foster long-term thinking, analysis and debate in particular areas of interest.
ADM(RS) Recommendation
5. To provide the DEP with additional funding flexibility, consider increasing the funding ceiling and funding period for a limited number of Targeted Engagement Grants.
OPI: ADM(Pol)
Annex A—Management Action Plan
ADM(Pol) Comment: I would like to thank ADM(RS) staff for all of the work that they have put into this review of the DEP. On behalf of the DEP, I fully accept the following recommendations and support the target dates for implemented responses. Building on the successes of this modest yet influential program, the recommendations will help to inform us as we look to expand the DEP when we seek to renew this program.
ADM(RS) Recommendation
1. In addition to the DEP Annual Report, establish and implement the means to disseminate information throughout the year on the results of DEP activities. Recipients should include the Defence Team Steering Committee, OGD representatives and the security and defence academic community.
Management Action
For DND/CAF stakeholders, improvements to the DEP Intranet site will be a key step toward fulfilling this recommendation. In addition, beginning July 2016, the DEP will disseminate a quarterly email (newsletter) to OGDs and other external stakeholders for the three quarters of the fiscal year in which the annual report is not published.
OPI: ADM(Pol)
Target Date: July 2016
ADM(RS) Recommendation
2. When seeking the renewal of the DEP, formally incorporate the Expert Briefing Series as a component of the program.
Management Action
The DEP intends to formally request the incorporation of the Expert Briefing Series as a core component of its program in the upcoming TBS submission.
OPI: ADM(Pol)
Target Date: July 2017
ADM(RS) Recommendation
3. Assess whether there is a need and tangible benefit to the DND/CAF and the Government of Canada to directly support the development of Canadian scholars in the fields of security and defence.
Management Action
The DEP will assess whether there is a need and tangible benefit to the DND/CAF and the Government of Canada to directly support the development of Canadian scholars in the fields of security and defence. This will be done through internal (ADM(Pol) and Defence Team Steering Committee) and external consultations (universities and think tanks). Such a re-examination would also need to be submitted to the appropriate departmental governance body. All consultations would be completed prior to the upcoming TBS submission.
OPI: ADM(Pol)
Target Date: May 2017
ADM(RS) Recommendation
4. Develop a feedback mechanism to assess the extent to which funded DEP activities contribute to better awareness and understanding of new and emerging security and defence challenges.
Management Action
Feedback forms will be developed for internal and OGD stakeholders, asking about the extent to which DEP grant reports and Expert Briefings are useful for their work.
OPI: ADM(Pol)
Target Date: July 2016
ADM(RS) Recommendation
5. To provide the DEP with additional funding flexibility, consider increasing the funding ceiling and funding period for a limited number of Targeted Engagement Grants.
Management Action
The DEP will examine the notion of funding flexibility internally (Director Strategic Coordination and Outreach), as well as through local consultations (ADM(Pol) and Defence Team Steering Committee). Results will be proposed in the upcoming TBS submission.
OPI: ADM(Pol)
Target Date: May 2017
1.0 Methodology
1.1 Overview of Data Collection Methods
The findings and associated recommendations of this evaluation are supported by multiple lines of evidence collected through qualitative and quantitative research methods. Information and data collected from relevant sources were analyzed to inform conclusions on the relevance and performance (effectiveness and efficiency/economy) of the program.
Key activities of the evaluation were grouped together to determine the expected outcomes of the program as defined in the logic model. Performance indicators were identified for each outcome to assess the performance of the DEP. Outcomes and their associated performance indicators are listed in the Evaluation Matrix (Annex D).
The data for performance indicators was collected using the following research methods:
- literature review
- document review
- key informant interviews
- stakeholder questionnaire
-
administrative and financial data reviews
1.2 Details on Data Collection Methods
1.2.1 Literature Review
A literature review was undertaken to identify the existence of comparable programs in OGDs and agencies.
1.2.2 Document Review
A document review was completed to provide a thorough picture of the purpose, scope and mandate of the DEP and the context in which the program operates. Core program documents like the initiating documents were used to establish parameters regarding program delivery and as a source of data to support the performance evaluation of the program.
The documents reviewed during the conduct phase of the evaluation included:
- official initiating documents for this grant program;
- policy, legislation and related accountability documents, such as TBS Policy and Directive on Evaluation, DND Plans and Priorities, and DND PAA;
- Government of Canada direction and related documents, such as Canada First Defence Strategy; government priorities; and Government of Canada whole-of-government framework;
- program-specific documents, including reporting documents, internal reports and extractions from the DEP website; and
- other reviews conducted by ADM(RS) that are relevant to the present evaluation.
1.2.3 Key Informant Interviews
Interviews were conducted with the DEP Secretariat (2) and other members of ADM(Pol) (5), the Defence Team Steering Committee (4), officials from OGDs or agencies that conduct complementary programs (3), and a selection of national and international DEP grant recipients (6). Most of the key stakeholders who contributed to interviews were located within the National Capital Region. Others interviewees located outside of the National Capital Region were interviewed by telephone. Some respondents were interviewed more than once.
The interviewees were given an interview guide in advance. Interview guides were tailored to each individual, and interview questions were aligned with the performance indicators to facilitate analysis.
1.2.4 Stakeholder Questionnaire
The questionnaire concerning the Targeted Engagement Grant Program and the Expert Briefing Series was provided to departmental stakeholders and to officials from OGDs or agencies (9).
1.2.5 Administrative and Financial Data Reviews
Financial and administrative data for the DEP were reviewed to support the evaluation performance indicators, as applicable.
2.0 Validation
Validity and credibility of data is a concern in all evaluations. To address these concerns, the following mechanisms were employed:
- Peer Review. At each step of the evaluation where documents are created (i.e., work plan, logic model, evaluation matrix, evidence matrix, interview guides, portion of draft report, etc.), an internal review was conducted.
- Triangulation. Triangulation was achieved through a variety of methods to capture the same information while using different approaches. For instance, information that had been gathered in the initiating document and confirmed in financial statements was discussed in interviews. Another triangulation method was to ask the same question for the same performance indicator to different stakeholders. Obtaining the same data from multiple stakeholders was considered data validation.
3.0 Limitations
In this section, limitations affecting the evaluation are identified and assessed. A mitigation strategy is also stated to acknowledge how the evaluation attempted to overcome those limitations. The limitations and mitigation strategies are summarized in Table B-1.
Table B-1. Evaluation Limitations and Mitigation Strategies. This table lists the evaluation methodology limitations and the mitigation strategies associated with those limitations.
Table Summary
This table lists the evaluation methodology limitations and the mitigation strategies associated with those limitations. The limitations are listed in the left-hand column. For each limitation, read across the row for the associated mitigation strategy.
Limitations | Mitigation Strategies |
---|---|
Interview bias. Interviews may lead to perceptions and interpretations of facts and evidence. | Insights derived from interviews require corroboration from multiple sources, either from quantitative data, or from agreement with other interviewees. Validation through triangulation was employed. |
Period for data collection. The contribution program has run for 48 months, but only the first 30 months of data were readily available. |
Validity of data is ensured when taking into consideration triangulation between methods. |
Figure C-1. Logic Model for the DEP. This flow chart shows the relationship between the program’s inputs, main activities, outputs and expected outcomes.
Text description for Figure C-1.
The bottom of the flow chart provides the three domains of the Defence Engagement Program:
- Targeted Engagement Grants
- Expert Briefing Series
- Academic Awards
Starting Point 1: Targeted Engagement Grants
Inputs: $365,000 per year for engagement and research grants
Activities:
- Defence Team Engagement Priorities established
- Grant opportunities publicized
- Applications assessed
- Funding awarded
Outputs:
- Conferences, workshops
- Panels, roundtables
- Research projects
- Papers/publications
Immediate Outcome: Support a strong Canadian knowledge base in contemporary security and defence issues that is relevant to Canadian defence policy and capable of informing, confirming and challenging DND/CAF defence policy thinking.
Intermediate Outcome: Increase ability of the DND/CAF and the Government of Canada to identify implications of new global security and defence challenges as they arise.
Ultimate Outcome: Timely and relevant policy advice on current security and defence issues is provided to the department and the Government of Canada.
Starting Point 2: Expert Briefing Series
Inputs: Vote 1 funds, used through contracts
Activities:
- Topics selected
- Experts identified
- Experts contracted
Outputs:
- Expert briefing sessions
- Multi-expert panels
- Other events/initiatives
- Event summaries/reports
Immediate Outcome: Support a strong Canadian knowledge base in contemporary security and defence issues that is relevant to Canadian defence policy and capable of informing, confirming and challenging DND/CAF defence policy thinking.
Intermediate Outcome: Increase ability of the DND/CAF and the Government of Canada to identify implications of new global security and defence challenges as they arise.
Ultimate Outcome: Timely and relevant policy advice on current security and defence issues is provided to the department and the Government of Canada.
Starting Point 3: Academic Awards
Inputs: $135,000 per year for scholarships and fellowship
Activities:
- Award opportunities publicized
- Applications assessed
- Fellowships and Scholarships awarded
Outputs:
- Fellowships
- Scholarships
Immediate Outcome: Foster the next generation of security and defence scholars in the Canadian academic community.
Intermediate Outcome: Increase ability of the DND/CAF and the Government of Canada to identify implications of new global security and defence challenges as they arise.
Ultimate Outcome: Timely and relevant policy advice on current security and defence issues is provided to the department and the Government of Canada.
Annex D—Evaluation Matrix
Table D-1. Evaluation Matrix—Relevance. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the DEP’s relevance.
Table Summary
This table indicates the data collection methods used to assess the evaluation issues/questions for determining the DEP’s relevance. The evaluation questions are indicated in the left-hand column. For each evaluation question, read across the row to determine the associated indicators. For each indicator, continue reading across the row to determine whether the following sources were used: document and file review / program data; key informant interviews with (a) program staff, (b) stakeholders, or (c) OGDs; stakeholder questionnaire; or comparative analysis.
Evaluation Matrix – Relevance | |||||||||
---|---|---|---|---|---|---|---|---|---|
Evaluation Questions | Indicators | Document and File Review /Program Data | Key Informant Interviews | Stakeholder Questionnaire | Comparative Analysis | ||||
ProgramStaff | Stakeholders | OGDs | |||||||
1.1 | To what extent does the DEP continue to address a demonstrable need and respond to the needs of Canadians? | 1.1.1 | Proportion of stakeholders who believe that engagement grant projects serve a valid need in Canada’s security and defence stakeholder community | No | Yes | Yes | Yes | No | No |
1.1.2 | Proportion of stakeholders who believe that expert briefings serve a valid need in the DND/CAF | No | Yes | Yes | Yes | Yes | No | ||
1.2 | How do the roles and responsibilities of the federal government align with the delivery of the DEP? | 1.2.1 | Degree of alignment between federal roles and responsibilities and the delivery of the DEP by the DND/CAF | Yes | Yes | Yes | No | No | |
1.3 | How does delivery of the DEP align with federal government priorities and departmental strategic outcomes? | 1.3.1 | Degree of alignment between DEP objectives and current federal government priorities | Yes | No | No | No | No | No |
1.3.2 | Degree of alignment between DEP objectives and DND/CAF strategic outcomes | Yes | No | No | No | No | No |
Table D-2. Evaluation Matrix—Performance (Effectiveness). This table indicates the data collection methods used to assess the evaluation issues/questions for determining the DEP’s performance in terms of achievement of outcomes (effectiveness).
Table Summary
This table indicates the data collection methods used to assess the evaluation issues/questions for determining the DEP’s performance in terms of achievement of outcomes (effectiveness). The evaluation questions are indicated in the left-hand column. For each evaluation question, read across the row to determine the associated indicators. For each indicator, continue reading across the row to determine whether the following sources were used: document and file review / program data; key informant interviews with (a) program staff, (b) stakeholders, or (c) OGDs; stakeholder questionnaire; or comparative analysis.
Evaluation Matrix – Performance: Achievement of Expected Outcomes | |||||||||
---|---|---|---|---|---|---|---|---|---|
Evaluation Questions | Indicators | Document and File Review /Program Data | Key Informant Interviews | Stakeholder Questionnaire | Comparative Analysis | ||||
ProgramStaff | Stakeholders | OGDs | |||||||
Immediate Outcome: Support a strong Canadian knowledge base in contemporary security and defence issues that is relevant to Canadian defence policy and capable of informing, confirming and challenging DND/CF defence policy thinking | |||||||||
2.1 | To what extent have the engagement grant projects strengthened the external Canadian knowledge base in contemporary security and defence issues? | 2.1.1 | Number of engagement grant projects funded | Yes | No | No | No | No | No |
2.1.2 | Proportion of engagement grant projects by recipient type | Yes | No | No | No | No | No | ||
2.1.3 | Proportion of engagement grant projects by type | Yes | No | No | No | No | No | ||
2.1.4 | Proportion of engagement grant project applications funded | Yes | No | No | No | No | No | ||
2.1.5 | Proportion of overseas engagement grant recipients who partnered with a Canadian institution and/or expert(s) through their project | Yes | No | No | No | No | No | ||
2.1.6 | Number of Canadian and international experts attending an engagement grant project event or otherwise participating in an engagement grant project | Yes | No | No | No | No | No | ||
2.1.7 | Proportion of engagement grant projects that had a media engagement | Yes | No | No | No | No | No | ||
2.1.8 | Number of media engagements per engagement grant project | Yes | No | No | No | No | No | ||
2.1.9 | Proportion of DND/CAF stakeholders who believe that engagement grant projects establish links and promote research and dialogue with defence and security experts | No | Yes | Yes | No | No | No | ||
2.2 | To what extent have the engagement grant projects strengthened the DND/CAF’s knowledge base in contemporary security and defence issues? | 2.2.1 | Number of DND/CAF personnel attending and/or participating in engagement grant project events | Yes | No | No | No | No | No |
2.2.2 | Number of publications resulting from engagement grant projects | Yes | No | No | No | No | No | ||
2.3 | To what extent have the expert briefing sessions strengthened the Canadian knowledge base in contemporary security and defence issues? | 2.3.1 | Number of experts engaged | Yes | No | No | No | No | No |
2.3.2 | Number of DND/CAF personnel attending expert briefings | Yes | No | No | No | No | No | ||
2.3.3 | Number of DND/CAF executives attending expert briefings | Yes | No | No | No | No | No | ||
2.3.4 | Average time in days that expert briefing reports are prepared and distributed within the DND/CAF | Yes | No | No | No | No | No | ||
2.3.5 | Proportion of DND/CAF stakeholders who believe that hosting expert briefing sessions establishes links and promotes dialogue with defence and security experts | No | Yes | Yes | No | No | No | ||
2.3.6 | Proportion of OGDs attending applicable DEP expert briefings | Yes | No | No | No | No | No | ||
Immediate Outcome: Foster the next generation of security and defence scholars in the Canadian academic community | |||||||||
2.4 | To what extent has the DEP fostered the next generation of security and defence scholars in the Canadian academic community? | 2.4.1 | Number of fellowships and scholarships funded per year | Yes | No | No | No | No | No |
2.4.2 | Proportion of available fellowships and scholarships funded per year | Yes | No | No | No | No | No | ||
Immediate Outcome: Increase ability of the DND/CAF and the Government of Canada to identify implications of new global security and defence challenges as they arise | |||||||||
2.5 | To what extent has the DEP increased the ability of the DND/CAF and the Government of Canada to identify implications of new global security and defence challenges? | 2.5.1 | Coverage of Defence Team Engagement Priorities by DEP program activities | Yes | No | No | No | No | No |
2.5.2 | Proportion of DND/CAF stakeholders who believe that the engagement grant projects have contributed to increasing the Department’s ability to understand new and emerging security and defence challenges | No | Yes | Yes | No | No | No | ||
2.5.3 | Proportion of DND/CAF stakeholders who believe that the expert briefing sessions have contributed to increasing the Department’s ability to understand new and emerging security and defence challenges | No | Yes | Yes | No | Yes | No | ||
2.5.4 | Proportion of OGD stakeholders who believe that the expert briefing sessions have contributed to increasing their ability to understand new and emerging security and/or defence challenges | No | No | No | Yes | No | No |
Table D-3. Evaluation Matrix—Performance (Efficiency and Economy). This table indicates the data collection methods used to assess the evaluation issues/questions for determining the DEP’s performance in terms of efficiency and economy.
Table Summary
This table indicates the data collection methods used to assess the evaluation issues/questions for determining the DEP’s performance in terms of efficiency and economy. The evaluation questions are indicated in the left-hand column. For each evaluation question, read across the row to determine the associated indicators. For each indicator, continue reading across the row to determine whether the following sources were used: document and file review / program data; key informant interviews with (a) program staff, (b) stakeholders, or (c) OGDs; stakeholder questionnaire; or comparative analysis.
Evaluation Matrix – Performance: Demonstration of Efficiency and Economy | |||||||||
---|---|---|---|---|---|---|---|---|---|
Evaluation Questions | Indicators | Document and File Review / Program Data | Key Informant Interviews | Stakeholder Questionnaire | Comparative Analysis | ||||
ProgramStaff | Stakeholders | OGDs | |||||||
3.1 | Is the DND/CAF delivering the DEP in an efficient and economical manner? What is being spent on the DEP? | 3.1.1 | Amount spent on the DEP | Yes | Yes | Yes | No | No | No |
3.1.2 | Amount of DEP funding as a proportion of the overall Defence budget | Yes | No | No | No | No | No | ||
3.1.3 | Amount of DEP funding as a proportion of ADM(Pol)’s budget | Yes | No | No | No | No | No | ||
3.1.4 | Staffing levels for administering the DEP | Yes | No | No | No | No | No | ||
3.1.5 | DEP budget versus actual expenditures | Yes | No | No | No | No | No | ||
Engagement grant projects: | |||||||||
3.1.6 | Amount spent on engagement grants | Yes | No | No | No | No | No | ||
3.1.7 | Engagement grant budget versus actual expenditures | Yes | No | No | No | No | No | ||
3.1.8 | Number of engagement grant applications | Yes | No | No | No | No | No | ||
3.1.9 | Proportion of engagement grant applications that were funded | Yes | No | No | No | No | No | ||
Expert briefing series: | |||||||||
3.1.10 | Amount spent on expert briefings | Yes | No | No | No | No | No | ||
3.1.11 | Expert briefing series budget versus actual expenditures | Yes | No | No | No | No | No | ||
3.1.12 | Number of experts shared with, and received from, OGDs for briefing sessions | Yes | No | No | No | No | No | ||
Fellowships and scholarships: | |||||||||
3.1.13 | Amount spent on fellowships and scholarships | Yes | No | No | No | No | No | ||
3.1.14 | Fellowship/scholarship budget versus actual expenditures per year | Yes | No | No | No | No | No | ||
3.1.15 | Number of fellowship and scholarship applications | Yes | No | No | No | No | No | ||
3.2 | Are there alternative, perhaps more economical, ways of delivering the DEP? | 3.2.1 | Alternative approaches to external engagement, their costs and pros and cons | No | No | No | Yes | No | Yes |
3.2.2 | Stakeholders’ perceptions as to the existence of alternative and/or more efficient ways to deliver the program | No | Yes | Yes | No | Yes | No | ||
3.3 | Is the progress made toward expected outcomes adequate for the resources expended? | 3.3.1 | Proportion of stakeholders who believe the time and effort expended on administering No engagement grants is worthwhile | No | Yes | Yes | No | Yes | No |
3.3.2 | Proportion of stakeholders who believe the time and effort expended on hosting expert briefing sessions is worthwhile | No | Yes | Yes | No | No | No | ||
3.3.3 | Estimate of time spent on reviewing engagement grant applications | No | No | No | No | Yes | No |
___________________________________________________________________________________________________________________________
Footnote 1 TBS Policy on Evaluation, 2009; consulted December 2, 2015, from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024 and the TBS Policy on Transfer Payments, 2012, consulted December 2, 2015, from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=13525. The TBS Policy on Transfer Payments, 2012 requires a review of the relevance and effectiveness of each ongoing non-statutory grant and contribution program every five years.
Footnote 2 DND/CAF Five-Year Evaluation Plan 2014-15 to 2018-19 (January 2014).
Footnote 3 DND/CAF Five-Year Evaluation Plan 2014-15 to 2018-19 (January 2014).
Footnote 4 Chief Review Services Evaluation of SDF Class Grant Program, May 2010. The Evaluation of the SDF was approved by the Chief of the Defence Staff on August 26, 2010, and by the Deputy Minister on September 2, 2010.
Footnote 5 In 1967, Cabinet approved the Military and Strategic Studies Program, which aimed to establish chairs of military studies at Canadian universities and provide grants to Canadian scholars specializing in the field. In 1996, the program was renamed the SDF with a broadened mandate to support Canadian expertise in contemporary security and defence issues, foster informed public discussion and enhance communication between the DND/CAF and the academic community.
Footnote 6 Security and Defence Engagement Grant Program – Performance Measurement Strategy, February 2012.
Footnote 7 The ADM(Pol) Business Plan 2015/16 links the DEP to PAA programs 2.0 and 5.0. However, the DEP Secretariat subsequently determined the DEP is not well placed under PAA program 2.0, as this area refers to a separate program run by Defence Research and Development Canada..
Footnote 8 The ADM(Pol) Business Plan 2015/16 allocated $35,000 Vote 1 funds to the DEP for the Expert Briefing Series.
Footnote 9 TBS. Directive on the Evaluation Function, April 1, 2009. http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681§ion=text. Last consulted on December 2, 2015,
Footnote 10 The defence policy during the observation period (FY 2012/13 to FY 2014/15) was the Canada First Defence Strategy.
Footnote 11 The Defence Team Steering Committee is responsible for the strategic oversight and guidance of the DEP. It is mandated to meet annually to review lessons learned and establish and approve the Defence Team Engagement Priorities. The Committee member organizations are also included in the grant assessment process as subject matter experts.
Footnote 12 Statistical data collection for most of the projects awarded funding in FY 2014/15 was still ongoing at the time of evaluation data gathering.
Footnote 13 Defence Engagement Program. Standard Operating Procedures – Grants, Version 1.0, updated May 1, 2015.
Footnote 14 Director level and above.
Footnote 15 Other Canadian federal government departments or agencies include: Privy Council Office; Public Safety Canada; Global Affairs Canada (previously the Department of Foreign Affairs, Trade and Development); and the Canadian Security Intelligence Service.
Footnote 16 No data are available for FY 2012/13 as attendance by other Canadian federal government departments and agencies was not tracked during that year.
Footnote 17 Based on data for 8 of 10 Expert Briefing Sessions held in FY 2014/15. No data are available for two briefings. Source: DEP internal monitoring.
Footnote 18 Some of these programs include: The Canada Graduate Scholarships – Master’s Program. This program supports 2500 students annually in five broad fields of study: health, natural sciences and/or engineering, and social sciences and/or humanities. It is administered jointly by the three federal granting agencies: the Social Sciences and Humanities Research Council (SSHRC), the Natural Sciences and Engineering Research Council (NSERC) and the Canadian Institutes for Health Research (CIHR). For programs involving OGDs or agencies, Export Development Canada runs the Youth Education Program to help build the capacity of the next generation of business leaders by awarding scholarships to Canadian students with the demonstrated interest in international business. Global Affairs Canada has the International Security Research and Outreach Programme (ISROP), which supports the Graduate Research Awards to enhance Canadian graduate-level scholarship on non-proliferation, arms control and disarmament issues. The actual Graduate Research Awards are now offered by The Simons Foundation, in partnership with ISROP. For selected candidates who participate in Graduate Research Awards debates, Global Affairs Canada funds the travel for participants and hosts a working lunch. Graduate Research Awards recipients and officials attend the debates, which have provided a unique opportunity for exchange among departmental officials and Canadian opinion leaders in the non-proliferation, arms control and disarmament field.
Footnote 19 The baseline professional services fee offered to speakers by the DEP is $1000 per day. However, when a speaker originates from outside Ottawa, expenditures may also include transportation, accommodation and meals. Since FY 2012/13, four speakers provided their services free of charge.
Footnote 20 As per the website for the federal government’s Research Support Fund (which replaced the Indirect Costs Program after FY 2014/15), the term “indirect costs of research” includes, for example, the costs of maintaining modern labs and equipment; costs of providing researchers with access to up-to-date knowledge resources; costs of managing and administering research; costs of meeting regulatory and ethical requirements; and costs of transferring research results to knowledge users. http://www.rsf-fsr.gc.ca. Last consulted on January 4, 2016.
Footnote 21 Association of Universities and Colleges of Canada, Funding the Institutional Costs of Research: An International Perspective, Ottawa, May 2009. https://www.uoguelph.ca/research/system/files/institutional-costs-international-2009-05-e.pdf. Last consulted on January 4, 2016.
Footnote 22 Examples include the March-April 2015 time frame, during the evaluation of the second round of grants for FY 2014/15, while simultaneously running the DEP Expert Briefing Series, refreshing the Defence Team Engagement Priorities, and preparing to submit an event request to engage eight external experts simultaneously (the DEP Executive Panel); and in April-May 2015, planning and execution of the DEP Executive Panel event, which took place on May 6, 2015.
Footnote 23 ISROP acts as a focal point for timely, high-quality and policy-relevant research on international security and defence issues related to North America; regional and multilateral security and defence cooperation; and non-proliferation, arms control and disarmament.
Footnote 24 One of the successes of the program was the creation of the Canadian Network for Research on Terrorism, Security and Society. The Network is a national strategic initiative designed to foster communication and collaboration between academic researchers working on these topics in Canada; communication and collaboration between academic researchers and policy officials in these subject fields; and links with research on these topics in other countries.
Footnote 25 Organizations cited by interviewees included NATO, NSERC, CIHR and SSHRC.
Footnote 26 Assessment time depends on the application itself (length, level of detail and complexity of argumentation). For applications sent to Global Affairs Canada, the coordinator may also consult a geographic desk if the grant application addresses a particular country or region.
Footnote 27 A revised funding ceiling of $40,000 to $50,000 may be appropriate.
Page details
- Date modified: