Evaluation of the IDEaS Program
Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.
- Executive Summary
- Program Context
- Evaluation Scope
- Key Results
- Q1: How can the program be supported to stay relevant?
- Q2: Is the IDEaS Program effective at fostering innovative networks and engaging innovators?
- Q3: To what extent were the identified challenges successful in producing solutions that were of benefit to the defence and security community?
- Q4: To what extent is the program being delivered in a cost-effective manner?
- Annex A: Management Action Plan
- Annex B: Findings and Recommendations
- Annex C: Evaluation Methodology
- Annex D: Evaluation Limitations
- Annex E: Program Profile
- Annex F: Definitions
- Assistant Deputy Minister (Defence Research and Development Canada
- Assistant Deputy Minister (Review Services)
- Canadian Armed Forces
- Call for Proposal
- Competitive project
- Director General of Research and Development Innovation
- Department of National Defence
- Defence Research and Development Canada
- Departmental Results Report
- Employee Salary and Benefit Plan
- Force Development
- Fiscal Year
- GBA Plus
- Gender-based Analysis Plus
- Gross Domestic Product
- Grants and Contributions
- Innovation for Defence Excellence and Security
- Management Action Plan
- Operations and Maintenance
- Office of Collateral Interest
- Other government departments
- Office of Primary Interest
- Public Service and Procurement Canada
- Research and Development
- Strategic Focus Area
- Small or Medium-sized Enterprise
- Solution Readiness Levels
- Canada’s defence policy: Strong, Secure, Engaged
- Science and Technology
- Treasury Board of Canada
- Treasury Board of Canada Secretariat
- Transfer Payment Program
- Technology Readiness Level
The evaluation of the Innovation for Defence Excellence and Security (IDEaS) Program focused on the relevancy of the program, the program's ability to engage innovators, benefits of program outcomes to the defence and security community, and the efficiency of the program for the period between Fiscal Years (FY) 2017/18 and 2021/22. The evaluation also considered the extent to which the program has achieved expected results, in line with its short-term and medium-term objectives. Over the short-term, the program aimed to engage innovators and solicit concepts and initiatives to solve Department of National Defence/Canadian Armed Forces (DND/CAF) issued challenges (i.e., competitions in need of solutions to specific defence and security issues). In the medium-term, IDEaS aimed to engage Canadian innovators to solve defence and security challenges. The evaluation is in accordance with the DND/CAF Five-Year Departmental Evaluation Plan FY 2017/18 to FY 2021/22.
IDEaS was launched in 2018 consistent with the direction given through initiative 105 of Canada's defence policy: Strong, Secure, Engaged (SSE). IDEaS is based on three pillars: 1) to build an innovation community by bringing industry, researchers and academics together; 2) to find solutions through the open competition of ideas with the goal of finding multiple solutions to challenges faced by DND/CAF; and 3) to push technology forward through validating innovative solutions and advancing the most promising technologies along the innovation continuum. In 2018, the program received a funding commitment of $1.6 billion over the next 20 years, ultimately in support of the Department’s Science, Technology and Innovation Program. As of August 2022, the program has invested over $314.2 million.
The findings show that the IDEaS Program remains relevant in the evolving global defence and security environment, and has effectively engaged Small or Medium-sized Enterprises (SME) and academia, further cultivating those networks to strengthen the innovation ecosystem within Canada. The program has effectively facilitated the advancement of a range of solutions to address defence and security challenges, and has had varying levels of success in delivering its allocated funding throughout the evaluation period. Challenges were identified within the IDEaS Program, notably the need for better strategic direction, as well as a need to improve internal processes, governance of the program, and use of the Grants and Contributions (Gs&Cs) portions of program funding. Levels of engagement between innovators, clients and the program show room for improvement.
Overall, the program received positive feedback from all program stakeholders including program personnel, DND/CAF members, and innovators. IDEaS remains relevant in the context of SSE and in the face of the evolving defence and security environment and is currently funded until 2037 to continue to deliver on its mandate. IDEaS has been effective in cultivating innovation within Canada, specifically with SMEs and academia. Through these relationships, solutions have been curated to address a wide range of challenges. Nonetheless, some areas within IDEaS have been identified requiring improvement.
- Strengthen the strategic direction and governance of the IDEaS Program (particularly related to the generation of challenges) to ensure alignment with departmental priorities and enhance internal processes.
- To ensure the successful continuity of partnerships, increase engagement throughout the program processes.
- Explore the benefits of increasing the utilization of Transfer Payment Program (TPP) (Vote 10) funds.
In accordance with initiative 105 of SSE, the IDEaS Program was launched in 2018 with a commitment of $1.6 billion until 2037. The IDEaS Program is a sub-component of the larger Defence Capability Development & Research activity that aims to develop and synthesize new knowledge and integrate advancements in Science and Technology (S&T) to provide a competitive advantage in military operations and enhance the effectiveness of other defence programs.Footnote 1 Underpinned by its three pillars to: Build an Innovation Community; Find Solutions; and Push technology forward, the program was developed to provide mechanisms to allow the government to better tap into technology development and innovation. IDEaS further supports the Departmental outcomes of “a robust Canadian innovation base that can support Canada’s defence and security is established and enabled”Footnote 2 and “current and future military capabilities and related areas of scientific investigation are established and fulfilled.”Footnote 3
As of August 2022, the program has issued 70 challenges, and invested over $314.2 million into 624 projects across its five funding elements: Competitive Projects; Innovation Networks; Contests; Sandboxes; and Test Drives.Footnote 4 Funding recipients have spanned across Canada with 284 unique recipients being represented (either as part of an SME, large industry, academia, not-for-profit or even as an individual).
Figure 1 Summary
This figure is circular in design and contains three smaller circles representing the three main program stakeholders: Clients, Innovators, and Program. Clients – an L1 organization (or the L1 appointed contact) who has identified and submitted a challenge to the IDEaS program. Innovators – A small business or large enterprise, an academic institution, a non-profit, or an individual with no preference for any specific industry, region or domain seeking to present potential solutions to the challenges issued by DND/CAF. Program – IDEaS staff and integrators.
The overall objective of IDEaS is to shape, shift and promote innovation from Canadian innovators to benefit defence and related security challenges posed by DND/CAF. IDEaS aims to support creating a vibrant innovation community, open competition of ideas and validated innovative solutions.
The IDEaS Program possesses two funding components accessible to individuals and organizations. Contracts are the mechanisms used to fund Competitive Projects, Sandboxes and Test Drives. The program also possesses a TPP composed of grant and contribution funding used to support other project elements such as Contests and Innovation Networks.
(Pers + EBP)
* Vote 1 Expenditures exclude $22.74M spent on Test Drives.
** Vote 1 Expenditure data for FY 2022/23 is as of Oct 31, 2022.
Table 1 Summary
This table compares annual program allocations against program expenditures by Vote 10 and by Vote 1 from fiscal year 2017/18 to 2022/23. The table has 10 rows. The top three rows contain the table headings as well as the column sub-headings. Beginning with row four, each row represents the allocations and expenditures for that fiscal year. The bottom row contains the overall totals for each column for the six fiscal year period. The last column provides the annual overhead costs which include personnel costs and employee benefit plan costs (EBP).
Coverage and Responsibilities
The evaluation of the IDEaS Program considered the extent to which the program has achieved expected results in line with its short-term and medium-term objectives. Over the short-term, the program aimed to engage innovators and solicit concepts and initiatives to solve DND/CAF issued challenges. In the medium-term, IDEaS aimed to engage Canadian innovators to solve defence and security challenges. The evaluation was structured around four main evaluation questions:
Figure 2 Summary
This figure is comprised of 4 rectangular shapes each containing one of the main evaluation questions. Read the text in each rectangular shape for each of the evaluation questions.
The IDEaS evaluation was undertaken in alignment with the Treasury Board of Canada (TB) Policy on Results, which requires Gs&Cs programs of $5 million or more per year, to be evaluated every five years. This is the first time this program has been evaluated; although in 2015, an evaluation of the Science and Technology Program was conducted.
Out of Scope
The departmental procurement process was scoped out.
The findings and recommendations within this report were informed by multiple lines of evidence, including:
- Document Review;
- Review and analysis of program data, including financial data;
- Key Informant interviews (n=6: external innovators engaged with IDEaS, n=9: program analysts);
- International Benchmarking: Czech Republic, France, Italy, Netherlands, New Zealand, Norway, UK and US; and
- Surveys (n=19: internal DND innovation clients, response rate of 54%; n=23: program analysts, excluding executives, response rate of 61%).
Figure 3 Summary
This figure is comprised of six rectangular shapes. The two rectangles under each of the two top-most rectangles or header rectangles, contain the name and a short description of the targeted groups for each of the surveys and interviews conducted during the evaluation. Under Survey, the first box reads DND Clients, Internal DND Innovation stakeholders (L1 level). And the second box reads IDEaS Program Analysts, Program personnel/staff (excluding executives). Under Interview, the first box reads Innovators, Innovators external to DND/CAF who have engaged with the IDEaS Program. And the second box reads IDEaS Program Managers, Program personnel/staff at the manager level and above.
Q1: How can the program be supported to stay relevant?
The IDEaS Program remains relevant in the evolving global defence and security environment.
As outlined in SSE, adapting and innovating quickly are crucial to developing and maintaining capabilities that address the challenges presented by today’s evolving global defence and security environment. A program document on SSEFootnote 5 identifies IDEaS as a program that "provides the tools to access this capacity (knowledge, solutions, and technologies) to meet both immediate and future needs of defence and security communities.” In fact, since its inception, IDEaS has had well-balanced social impact targets (e.g., force protection, cyber security, pandemic preparedness) that aligned with SSE.Footnote 6
The overall mission for the IDEaS Program is to help DND/CAF address critical challenges to Canada’s defence and security posture through a demand-side innovation approach that links DND/CAF needs with, and helps grow, Canadian innovation capabilities and opportunities.Footnote 7 According to foundational documents by DND/CAF, the program was designed to provide new opportunities to support science, technology and innovation for the defence and security of Canada, and facilitate the integration and adoption of new solutions and capabilities for the CAF and the Public Safety and Security communities. In doing so, it not only transforms the way DND/CAF generates solutions to the new threats it faces, but it can also stay ahead of potential adversaries and meet evolving defence and security needs, while generating economic benefits for Canada.
Figure 4 Summary
This figure is a process flowchart. It is comprised of three rectangular text boxes, four directional arrow symbols indicating the process flow, as well as seven icons with text below each. The flowchart is read beginning from the top left text box that reads DND/CAF Priorities. From there, follow the flow indicated by the arrow symbols to the right and down.
DND/CAF Priorities ➔ Challenge selection and prioritization (down arrow)
Innovation Challenges (three-pronged down arrow)
Integrators (bracket) Competitive projects; Contests; Innovation Networks ➔ Defence Capability Test Drives; Sandboxes
Challenge selection and prioritization may not reflect the highest priorities of DND/CAF.
Regardless of the intended alignment of IDEaS to departmental priorities, 60% of program analyst survey respondents did not believe that IDEaS has adequate policies in place to support the program. Similarly, results from the client survey showed that DND/CAF stakeholders are also unclear about the strategic alignment of the IDEaS Program to, or its impact on, Government of Canada priorities. Nonetheless, 94% agreed that the program is valuable to DND and to the defence and security community.
Program analyst survey respondents also identified that there are difficulties with how the challenges are prioritized and governed, given the lead role Defence Research and Development Canada (DRDC) plays in this process. Specifically, the majority (74%) of respondents disagreed that IDEaS has the necessary strategic direction to effectively prioritize submitted challenges.
The lack of strategic direction and the program's reliance on DRDC to source and prioritize challenges is a problem that program managers are aware of and currently addressing.
"Challenges" within the context of the program are competitions that invite innovators to present viable solutions to specific defence and security needs that are posted on the IDEaS website.
Concerns were raised about internal processes and lack of governance.
In interviews with program managers, it was repeatedly raised that the IDEaS Program is not releasing enough challenges, as there is a need for a constant stream of solutions required from DND/CAF stakeholders. They also acknowledged that they do not have enough integrators (i.e., internal DND/CAF subject-matter experts who can help refine the development of solutions) to manage these challenges.
However, it should be noted that the IDEaS Program is not responsible for developing challenges or sourcing its own integrators. It was acknowledged by program manager interviewees that the greatest obstacle to putting out the most relevant challenges is the reliance on the broader DRDC organization, which is responsible for distributing challenges to IDEaS.
Through interviews it was found that more frequent check-ins and direct communication between DND/CAF innovation clients (L1 level) and innovators would allow both sides to have a better idea of project progression throughout challenges, as well as anticipated next steps. While IDEaS Program managers are aware of these issues, they are careful not to overburden DND/CAF clients with additional meetings; instead, they are seeking ways to optimize governance internally and provide clearer and more consistent messaging to partners.
Strengthen the strategic direction and governance of the IDEaS Program (particularly related to the generation of challenges) to ensure alignment with departmental priorities and enhance internal processes.
Q2: Is the IDEaS Program effective at fostering innovative networks and engaging innovators?
The IDEaS Program has effectively engaged SMEs and academia, further cultivating those networks to strengthen the innovation ecosystem within Canada.
The IDEaS Program has been successful in attracting a wide range of innovators to enrich innovation within Canada. In fact, program data show that since the outset of IDEaS, 100% of posted challenges have received proposals. This outpaced the 80% target outlined in the program’s foundational document. Further, benefitting from geographically dispersed innovators has positioned IDEaS to maximize established national and provincial systems of innovation –significant sources of knowledge, expertise, skills, facilities and equipment.Footnote 9
Program data also show that to date, SMEs and academia have been especially participatory in the funding elements (the various avenues to obtaining funding from the program, which includes Competitive Projects (CP), Innovation Networks, Sandboxes, Contests and Test Drives). In particular, SMEs and academia have been identified as having the highest involvement rate compared with other innovator types (e.g., Large Industry, Individuals and Not-for-profits), as well as those who have been awarded the most funding. The partnerships built during challenges have led all interviewed innovators to indicate that they would apply to the IDEaS Program again if the challenge aligned with their work.
SMEs have also been awarded a higher funding percentage of second-milestone contracts, furthering the access to the creativity and knowledge of SMEs, and reflecting the effectiveness of the process for these innovators. In fact, 96% of program analyst survey respondents agreed that the IDEaS Program helped foster connections with Canadian industry and academia.
Large industries received 16% of IDEaS awards with this proportion decreasing over time, which is in contrast to some of the innovator interviewees who believe that the program benefits larger companies. At the onset of IDEaS, the reporting requirements from the program to receive funding were more onerous, making these tasks more burdensome for smaller businesses compared with larger ones. The reporting requirements have been streamlined in recent years.
Figure 8 Summary
This figure is horizontal line chart. The vertical axis indicates the number of contracts from 0 to 100 in increments of 20 and the horizontal axis indicates the fiscal year beginning with FY 2018/19 and ending with FY 2021/22. The 3 line graphs included on the chart represent contracts for Large Industry, SMEs, and Academia respectively.
- 2018/19 = 88
- 2019/20 = 54
- 2020/21 = 63
- 2021/22 = 67
- 2018/19 = 39
- 2019/20 = 23
- 2020/21 = 31
- 2021/22 = 19
- 2018/19 = 26
- 2019/20 = 24
- 2020/21 = 23
- 2021/22 = 15
The decrease in CPs after FY 2018/19 is the result of COVID-19 impacts as well as an intentional decrease to the volume of awarded contracts. Lessons learned and after action reviews following the first Call For Proposal (CFP) recommended reducing the number of received proposals to match program capacity to manage them and the resultant projects.
The program has effectively attracted both new and repeat innovators to submit proposals. In particular, there were 71 unique return innovators to related domain challenges within the CP’s program element during the evaluation period. According to innovator interviewees, the main sources which attracted them to the program were industry meetings, innovator networks, conferences, previous engagement with DND, the program website and emails.
Figure 9 Summary
This figure is comprised of text and three circles with a percentage in each. Read each line of the figure from left to right with the corresponding percentage inside each of the circles.
The program is well known within industry
The program is well known within academia
The program is well known within DND/CAF
Gender-based Analysis Plus (GBA Plus) Considerations:
The IDEaS Program attracts a wide range of innovators and as part of a government-wide initiative to examine GBA Plus indicators, the diversity of innovators was assessed during the evaluation. One-third of innovator interviewees felt that their companies naturally met the expectations for company diversity due to the types of individuals that work in their fields in terms of gender, neurodiversity and visible minority groups. IDEaS Program data show that there is a wide variety of individuals listed as owners in participating innovator organizations, confirming the interview responses that innovator organizations are diverse in ownership. Specifically, 51% of innovator organizations had some visible minority ownership, followed by 46% that had a portion of youth ownership, and 36% that included female ownership.
Figure 10 Summary
This figure is comprised of four variably-sized circles each containing text describing the number of unique IDEaS program funding recipients for each of the program’s four funded elements: competitive projects, contests, innovation networks, and test drives.
- 262 Unique Recipients (71 return)
- 34 Unique Recipients
- 30 Unique Recipients
- 4 Unique Recipients
Figure 11 Summary
This figure is comprised of text and three variably-sized rectangles each of which contain a percentage. The three sets of text on the left side of the figure describes what each of the percentages within the rectangles represent.
The degree to which diversity and inclusion are considered in partnership choices is satisfactory.
The degree to which GBA Plus elements are incorporated into the selection process is satisfactory.
The funding mechanisms provided by IDEaS ensure that diversity in the selection of Canadian innovators.
Many compliments about the program were noted; however, levels of engagement between innovators, clients and the program show room for improvement.
The program received positive feedback from innovators, clients and personnel alike. All innovator interviewees appreciated the broadness of the challenges, which allowed for creativity and stated that the challenge statements were clear and understandable. The overall application process was noted as being easy and clear by many innovator interviewees. Similarly, DND/CAF client survey respondents scored the IDEaS Program highly in:
- Providing value for DND and the broader defence and security community;
- Positively impacting domestic partnerships;
- Effectively contributing to innovative designs and concepts; and
- Transferring knowledge to DND/CAF.
However, current levels of engagement with both IDEaS Program staff and DND/CAF clients was an area where many innovator interviewees hoped enhancements could be made. The Attracting Innovators consultationFootnote 10 produced similar results, with barriers for innovators listed as: the inability of innovators to engage end-users (i.e., DND/CAF members that could potentially utilize the solution in their day-to-day activities) and clients (i.e., those representing an L1 organization who identified/submitted a challenge and could potentially acquire the solution in the future); the lack of personal interaction between innovators and IDEaS Program staff and integrators, and insufficient feedback from integrators to innovators.Footnote 11
Figure 12 Summary
This figure is the same as Figure 1. The figure is circular in design and contains three smaller circles representing the three main program stakeholders: Clients, Innovators, and Program. Read under each of these circles for a description of each program stakeholder.
Clients – an L1 organization (or the L1 appointed contact) who has identified and submitted a challenge to the IDEaS program. Innovators – A small business or large enterprise, an academic institution, a non-profit, or an individual with no preference for any specific industry, region or domain seeking to present potential solutions to the challenges issued by DND/CAF. Program – IDEaS staff and integrators.
Innovator – Program
The relationship between the program and the innovators is significant as it is the principal connection in achieving the intended outcomes of the program. However, half of innovator interviewees desired improvements to this partnership noting that it was is challenged by factors such as: the liaison between the two parties (known as the integrator), clarity on timelines and feedback levels.
All interviewed innovators had positive experiences working with program integrators and benefited from its collaborative nature. However, access to those individuals was said to be limited with the frequency of engagements being perceived as minimal. Of innovators interviewed, some shared that they would like more frequent and descriptive communication as well as meetings with integrators between formal milestones.
Clarity on timelines seemed to be the biggest challenge for innovator interviewees. While interviewees agreed that the program is responsive when contacted, it was stated that they would prefer to have clearly communicated timelines and proactive engagement during each stage of the process. Specifically, approximately half of the innovator interviewees felt there was still a level of uncertainty in the timelines for the selection process.
Feedback given during milestone reviews was said to be helpful for innovators, many of whom indicated that the feedback they received was easy to understand. They mentioned that the scoring system was helpful for understanding which projects were selected and which aspects of the project were important to the client. However, half of innovator interviewees wanted more feedback from program personnel, indicating that a clear reason for not being chosen for follow-on phases would be valuable for future applications.
Innovator – Client
All innovator interviewees desired increased engagement with DND/CAF clients (i.e., those organizations that identified/submitted their respective challenges and are invested in the outcome of the challenge to potentially acquire the solution in the future), stating that it would be beneficial to receive feedback directly from clients and expand upon opportunities for improved visibility and exposure. This step was perceived to aid innovators in gaining access to the later stages in a process if they can further align their innovations with DND/CAF expectations. However, in interviews with program managers, they did not view innovator participation in IDEaS as either an advantage or disadvantage to future procurement. The key to fairness is ensuring no kind of information is shared that could preclude innovators from participating in procurement activities outside of IDEaS.
The newly launched MarketplaceFootnote 12 event in 2022 was an occasion whereby innovators presented their projects to defence and security stakeholders both in the public and private sectors and had the additional benefit of providing an opportunity for innovators to ask questions and get feedback on their solutions from DND/CAF stakeholders (e.g., both clients and end-users). Only a few innovator interviewees had participated in the Marketplace event, while the others had not heard of it. As this was the first piloted instance of Marketplace, participation and awareness is expected to increase in future years, especially in light of the informal engagement that occurred as a result of this event.
Client – Program
Both DND/CAF client and program analyst survey respondents agreed that current engagement levels within DND between the client and the IDEaS Program staff are insufficient, and that engagement throughout the process could be improved. There was further agreement that updates from both innovators and program staff to DND/CAF clients should be increased and consistent. Although 76% of program analyst survey respondents signaled that the IDEaS Program is well-known within DND/CAF, there are still desired improvements. Specifically, 67% of program analyst survey respondents disagreed that current engagement levels between the program and DND/CAF are sufficient, followed by 57% of respondents who disagreed that DND/CAF clients are engaged throughout the process effectively by the program.
Sustained engagement between the client and the program was said to be challenged for several reasons, including a perceived insufficient number of dedicated integrators associated with the program. Similarly, program manager interviewees noted challenges in sustaining consistent and prolonged client buy-in, particularly in light of lengthy project timelines and cyclical CAF turnover.
To ensure the successful continuity of partnerships, increase engagement throughout program processes.
Q3: To what extent were the identified challenges successful in producing solutions that were of benefit to the defence and security community?
The IDEaS Program has effectively facilitated the advancement of a range of solutions to address defence and security challenges. However, concerns were raised regarding the next steps, including commercialization, for solutions to fully support defence innovation.
Overall, respondents to both the DND/CAF client survey and the program analyst survey are confident in the quality of the challenges to derive the solutions needed.
Of DND/CAF clients surveyed, 93% agreed that the potential solutions accurately address the identified defence challenges, and 100% agreed that the submitted challenges are, or will be beneficial to DND/CAF. Similarly, the majority of program analyst survey respondents agreed that the innovations produced to date have largely matched the original intent of the client.
In addition to the 100% of posted challenges that received proposals, program data also show that 100% of posted challenges resulted in projects or initiatives to address them. This outpaced the 75% target indicated in the program’s cornerstone document.
The program data also show that between FY 2018/19 and October 2022, 150 funding awards were granted to organizations who succeeded to the second milestone of a challenge (16 challenges total) within the CP program element, representing 79% of projects that have received funding for a higher level of development since the inception of IDEaS. Innovators and their solutions are moving further into the process, meaning that the solutions are increasing their Technology Readiness Levels (TRL), and the maturity of solutions is increasing.
SMEs received 51% of the total number of funding awards to advance to second milestone contracts.Footnote 13 Of those who progressed to this stage, innovator interviewees shared that they were optimistic about future opportunities to share their product or technology outside of DND to benefit the entire defence and security community.
Figure 14 Summary
This figure is combination bar graph and line graph. The bars indicate the percentage of second-milestone contracts awarded for SMEs, Academia, and Large Industry, respectively from FY 2018/19 to October 2022. The line chart indicates the total dollar amount in millions of second milestone contracts awarded to SMEs, Academia, and Large Industry, respectively from FY 2018/19 to October 2022.
- 51% of awarded second-milestone contracts
- $84 million of awarded second-milestone contracts
Large Industry received
- 22% of awarded second-milestone contracts
- $34 million of awarded second-milestone contracts
- 22% of awarded second-milestone contracts
- $32 million of awarded second-milestone contracts
There is a lack of clarity about next steps, including commercialization, once a solution is developed.
In interviews with program managers, it was indicated that IDEaS has not yet reached the point where solutions, once developed, can be procured. It was also stated by some interviewees that the IDEaS Program is not funding projects that are market ready outside of Test Drives. As program manager interviewees acknowledged, the acquisition of the solution by DND/CAF was not the initial scope of the program. As such, the most asked question from innovators is “what happens next?” For this, the program has no clear answer. Commercialization of solutions was identified as an international best practice for similar innovation programs within the governments of the USA, UK, Australia and New Zealand. As such, this could be an area that is considered when further developing the IDEaS Program that would enable it to foster marketable innovation and to keep pace with equivalent programs. In fact, IDEaS Program manager interviewees shared that knowing how innovation fits into overall capability development within DND/CAF could benefit the program. As noted in open-ended survey questions, internal DND innovation clients are also interested in the commercialization of solutions.
In alignment with this finding, the 2021 Integrated Strategic Analysis Force Development (FD) Evaluation recommended that DND/CAF work towards enhancing transition from innovation to capacity delivery. Management Action Plans (MAP) were developed in response to this recommendation. As of September 2022, the Vice Chief of the Defence Staff, in consultation with Project Sponsors, has fully implemented the MAP that prioritizes awareness of the transition of innovation to Force Development. More specifically:
- Development of a relationship with IDEaS and further awareness of innovation solutions are obtained through IDEaS;
- Various IDEaS Workshops, such as Innovation for the Future: A Critical Review of Global Defence and Security Innovation Programs (March 2022); and
- Engagement through the IDEaS Marketplace. “These efforts enable further awareness of the Innovation program and FD.”Footnote 14
Effective performance metrics could further inform the direction of the program.
Of the program analysts surveyed, 60% disagreed that there are effective metrics to track the progress of IDEaS. As noted in open-ended survey questions from program analysts, most metrics are not outcome-based. Without awareness of effective metrics, program personnel cannot confidently speak to the success of the program and, in turn, cannot effectively determine if changes or adjustments are required to reach program goals.
Performance indicators and targets contained in program foundational documents aimed at long-term outcomes should be examined in the next evaluation of this program. The long-term outcome of "Defence and security communities adopt advice and innovative solutions created from the Canadian innovation system, improving their capabilities to protect, secure and defend Canada and its citizens as well as for Canadian companies to gain marketshare"Footnote 15 is measured by two indicators.
- A target of 20% is set for the percentage of IDEaS projects that resulted in useful advice, adoption, integration or eventual procurement of a new defence and security capability.
- The effectiveness of the program is being measured by the percentage of defence and security senior leadership reporting satisfaction that their S&T requirements are addressed by IDEaS, with a target of 75% positive response rate.
Enhancing the transition from innovative solutions to capability delivery is an essential component in the program continuum to ensuring that the innovator’s products are leveraged and are available to support defence innovation.
Table 2 Summary
This table is comprised of four columns and four rows and compares the percentage of program overhead of three federal programs, including IDEaS. The first column lists the federal programs being compared. The second column lists the total annual funding for each program. The third column lists the annual overhead costs for each program, while the fourth and final column lists the overhead percentage for each program which is derived from dividing the overhead costs of each program by its total annual funding.
Figure 15 Summary
This figure is a bar graph. The vertical axis indicates the annual Vote 1 estimates and expenditures in millions from 0 to 70 million in increments of 10 million. The horizontal axis indicates the fiscal year beginning with FY 2018/19 and ending with FY 2022/23. Above each fiscal year along the horizontal axis are two bar graphs. The first bar indicates the dollar amount of Vote 1 cost estimates and the second bar indicates the amount of Vote 1 expenditures for that same fiscal year. This is repeated along the horizontal axis for each of the five fiscal years represented on the chart.
FY 2018/19 (in millions of dollars)
- $27.12 Vote 1 cost estimates
- $29.05 Vote 1 expenditures
FY 2019/20 (in millions of dollars)
- $65.18 Vote 1 cost estimates
- $66.01 Vote 1 expenditures
FY 2020/21 (in millions of dollars)
- $61.08 Vote 1 cost estimates
- $50.76 Vote 1 expenditures
FY 2021/22 (in millions of dollars)
- $54.98 Vote 1 cost estimates
- $47.41 Vote 1 expenditures
FY 2022/23 (in millions of dollars)
- $54.88 Vote 1 cost estimates
- $49.86 Vote 1 expenditures
The program’s Vote 1 funding includes allocations for O&M costs as well as overhead costs such as personnel. As mentioned in interviews with program staff, the program does not have enough overhead funding within its annual Vote 1 allocation to alleviate their actual overhead costs and the pressures this creates in terms of staff capacity and administrative workload.
The program reduced the number of projects following CFP1 (Call for Proposal – phase one). This was intentional as the number of challenges released and the volume of proposals received were far larger than initially anticipated. Lessons Learned and After Action Reviews completed following CFP1 recommended the total number of contracts awarded in a given fiscal year be reduced in the future to ensure the number of proposals and resulting projects were feasible for the program and DND capacity to support.
Although IDEaS' overhead costs in Table 1 appear to reflect a high measure of program efficiency, inadequate levels of overhead funding appear to be impacting the program’s capacity to continue to maintain the ability to generate and manage the initial volume of challenges and resulting program elements.
IDEaS has not fully expended Vote 10 funds allocated in the IDEaS’ TPP which includes all Gs&Cs spending. According to data reported in the Public Accounts of Canada, the program had spent $31.7M, or 42%, of total allocated Vote 10 funding of $75M (per Main Estimates) as of the end of FY 2021/22. Program staff mentioned the unspent portions of Vote 10 funding were either reprofiled or lapsed at fiscal year-end.
As noted in program manager interviews, this funding is ideally suited to seed the initial Canadian innovation ecosystem to promote and support the research and development of lower TRL projects to address defence and security challenges that could eventually result in a procured solution.
Program staff interviews cited an insufficient number of relevant challenges issued as one of the headwinds the program faced in spending its Vote 10 funding during the evaluation period. Furthermore, it was noted that the program was initially focused on competitive projects, funded through Vote 1 contracts. There was also a perception that the relevance of issued challenges through Vote 10 funds during this timeframe was potentially diminished given that this spending focused on lower solution-ready projects.
The COVID-19 pandemic presented unexpected challenges to the program and recipients of program funding. Particularly impacted were academic institutions (i.e., the majority recipient of Vote 10 funds) who had difficulties spending their funding as most universities had no students on campus, and few to no graduate students working in university laboratories. Program staff noted Vote 10 spending should improve in the coming years with the easing of COVID-19 restrictions.
Figure 16 Summary
This figure is a bar graph. The vertical axis indicates the annual Vote 10 funding allocations and expenditures in millions from 0 to 30 million in increments of 5 million. The horizontal axis indicates the fiscal year beginning with FY 2018/19 and ending with FY 2022/23. Above each fiscal year along the horizontal axis are two bar graphs. The first bar indicates the dollar amount of Vote 10 funding allocations and the second bar indicates the amount of Vote 10 expenditures for that same fiscal year. This is repeated along the horizontal axis for each of the five fiscal years represented on the chart. Note that for FY 2018/19 and 2022/23 there is not a bar graph representing Vote 10 expenditures since the amount of expenditures for those two years was zero.
FY 2018/19 (in millions of dollars)
- $12.00 Vote 10 funding allocation
- $0.00 Vote 10 expenditures
FY 2019/20 (in millions of dollars)
- $14.00 Vote 10 funding allocation
- $2.67 Vote 10 expenditures
FY 2020/21 (in millions of dollars)
- $21.00 Vote 10 funding allocation
- $12.46 Vote 10 expenditures
FY 2021/22 (in millions of dollars)
- $28.00 Vote 10 funding allocation
- $16.54 Vote 10 expenditures
FY 2022/23 (in millions of dollars)
- $27.00 Vote 10 funding allocation
- $0.00 Vote 10 expenditures
During four of the first five years following the program's launch, Vote 10 dollars were not fully expended.
The program is encouraged to maximize the use of Vote 10 funds to achieve the full mandate and intended benefits of the program. Doing so will further continue to seed the creation and support of innovation networks and further enable the funding of successful proposals from non-profit organizations and academia. A greater focus on lower TRL proposals will also promote early experimentation that can ultimately lead to solutions that can become commercialized in the defence and security domain.
Explore the benefits of increasing the utilization of Transfer Payment Program (Vote 10) funds.
Overall, the program received positive feedback from all program stakeholders including program personnel, DND/CAF members and innovators. IDEaS remains relevant in the context of SSE and in the face of the evolving defence and security environment, and is currently funded until 2037 to continue to deliver on its mandate. IDEaS has been effective in cultivating innovation within Canada, specifically with SMEs and academia. Through these relationships, solutions have been curated to address a wide range of challenges that the department is facing, and which DND/CAF members have found beneficial.
Current barriers to fully supporting the future of innovation within the department include:
Governance; prioritization of challenges to align with departmental priorities; limited engagement opportunities between innovators and integrators/end-users; unclear processes between innovation and capability development; and the under-utilization of Vote 10 funds. Further developing the strategic direction for the program, enhancements with stakeholder engagement, and optimizing the program’s Vote 10 funds should alleviate some of the difficulties to ensure the future of IDEaS will effectively contribute to the departmental priorities and increase stakeholder confidence.
Annex A: Management Action Plan
1. Strengthen the strategic direction and governance of the IDEaS Program (particularly related to the generation of challenges) to ensure alignment with Departmental priorities and to enhance internal processes.
Action 1.1 Director General of Research and Development Innovation (DGRDI) will develop a consistent method of defining its innovation program of work to better facilitate the prioritization, definition and sourcing of the required support for the challenges. Working with Director General Research and Development Program, DGRDI will develop new tools including a new standardized intake method for its innovation challenges through the DRDC call for S&T requirements, as well as a sourcing framework to identify the “best fit” for delivery for all new challenges as part of an integrated DRDC program. This approach will allow for efficiencies to be realized that will minimize the impact on stakeholders, and will take into account key considerations for streamlining the process including: the security classification of the S&T required; the Scientific/Technical Readiness Level; dollar value; expected economic benefit; and the current capability within DRDC research centre resources.
OCI: DG Research and Development Program, DG Research and Development Science and Engineering
Target Date: December 2024
Deliverable: DGRDI will develop an innovation program sourcing framework. This framework will ensure innovation challenges are consistently collected from clients and streamed to the most appropriate program, based on a flexible, yet pre-determined, set of criteria.
2. To ensure the successful continuity of partnerships, increase engagement throughout the competitive process.
Action 2.1 The IDEaS program relies on successful partnerships and engagements with its stakeholders within the CAF and the DND; other government departments (OGD); academia and industry in order to meet the goals of the program. While IDEaS has demonstrated success in meeting the short and medium-term goals, there is recognition that there is a need for a refreshed and more deliberate approach to engagement in order to meet the long-term goals. The planned evolution of program governance will enable IDEaS to strengthen its relationship with its clients and stakeholders across the Defence Team and OGDs. IDEaS will also explore options to strengthen relationships with innovation partners (e.g., academic and industry) throughout the delivery of innovation projects in a manner that is more aligned with agile innovation project management. This will require consultation with colleagues at Public Services and Procurement Canada (PSPC) to ensure that such engagements would not present any risks to the integrity of the procurement process, and would remain aligned with the outcome-based approach to soliciting and investing in concept and advanced technology development and demonstration projects. It is important that all stakeholders remain engaged throughout this multi-year process, given that the development of solutions to innovations challenges requires increased financial investment as the maturity level of these solutions increases.
Target Date: September 2024
Deliverable: The IDEaS team will develop an options analysis report outlining possibilities to update its program delivery governance in order to reduce the programmatic burden on stakeholders, strengthen the relationship with DND/CAF and facilitate an agile innovation project delivery approach throughout its processes.
3. Explore the benefits of increasing the utilization of Transfer Payment Program (Vote 10) funds.
Action 3.1 IDEaS is currently leveraging its Vote 10 capacity to advance projects in new ways to test different investment models. For example, a particular project under the CP element was no longer viable for higher Solution Readiness Level (SRL) testing/demo purposes based on the stakeholder’s needs; however, it was determined that there would be value in a deviation to the original project to acquire knowledge in the domain. Therefore, a contribution agreement that defined the new requirements is in development and will be managed using Vote 10 funds within the Innovation Networks element. In addition, a more optimized use of Vote 10 funding will help mitigate some of the capacity challenges faced by PSPC in processing IDEaS Vote 1 contracts.
Target Date: December 2024
Deliverable: IDEaS will research, develop and implement new methods of leveraging Vote 10 capacity to advance projects in innovative ways to test different investment models. The MAP will be considered fully implemented once a Concept paper has been finalized outlining an overview of the findings and resulting recommendations.
Annex B: Findings and Recommendations
|Evaluation Questions||Results and Recommendations|
|1. How can the program be supported to stay relevant?||
Result: The IDEaS Program remains relevant in the evolving global defence and security environment.
Recommendation 1: Strengthen the strategic direction and governance of the IDEaS Program (particularly related to the generation of challenges) to ensure alignment with departmental priorities and to enhance internal processes.
|2. Is the IDEaS Program effective at fostering innovative networks and engaging innovators (e.g., diversity of innovators, awareness of challenges, continuity of partnerships)?||
Result: The IDEaS Program has effectively engaged SMEs and academia, further cultivating those networks to strengthen the innovation ecosystem within Canada; however, levels of engagement between innovators, clients and the program show room for improvement.
Recommendation 2: To ensure the successful continuity of partnerships, increase engagement throughout program processes.
|3. To what extent were the identified challenges successful in producing solutions that were of benefit to the defence and security community?||
Result: The IDEaS Program has effectively facilitated the advancement of a range of solutions to address defence and security challenges. However, concerns were raised regarding the next steps, including commercialization, for solutions to fully support defence innovation.
|4. To what extent is the program being delivered in a cost-effective manner?||
Result: Overall, the program has had varying levels of success in delivering its allocated funding.
Recommendation 3: Explore the benefits of increasing the utilization of Transfer Payment Program (Vote 10) funds.
Table B-1 Summary
This table has 2 columns and 5 rows. The top row contains the headers Evaluation Questions and Results and Recommendations.
Read across the row to find the evaluation question, followed by the result and recommendation.
Annex C: Evaluation Methodology
The findings and recommendations of this report were informed by multiple lines of evidence collected throughout the conduct phase of the evaluation, which were then triangulated to provide a fulsome overview. The research methodology used in the scoping and conduct of the evaluations are as follows:
Annex D: Evaluation Limitations
Annex E: Program Profile
In the short term, the program aims to steer more external innovator expertise towards DND/CAF challenges by engaging with external innovators across Canada, ensuring these challenges provide good development opportunities and by streamlining the submission and review process, where possible. In the medium term, the IDEaS Program’s main goal is to mature relationships and partnerships among innovators and improve the quality of the solutions to the challenges issued by DND/CAF. Finally, in the long term the program will aim to demonstrate increased defence capabilities and increased economic benefits to Canada.
The main focus of the program is to:
Connect: Engaging S&T networks to promote multilateral organizational collaboration on Defence Innovation. IDEaS conducts outreach and engagement through external and internal networks to create an ‘innovation ecosystem’;
Develop: IDEaS develops innovative solutions through program elements (e.g., CPs, Contests, Sandboxes, and Test Dives) by creating a competition of ideas around defence and security issues to promote innovation and breakthroughs;
Evaluate: Post-program follow-up on Research and Development (R&D) investment with mission-qualified knowledge; and
Transition: Providing support through the R&D process to transition innovation to operators. IDEaS is developing capacity to transition the industry solutions being developed for utilization and exploitation by DND/CAF by working collaboratively with PSPC partners.
The IDEaS Program reports under the Defence Results Framework Program entitled “Science, Technology and Innovation,” with the outcomes of “[A] robust Canadian innovation base that can support Canada’s defence and security needs is established and enabled” and “[C]urrent and future military capabilities and related areas of scientific investigation are established and fulfilled.”
Assistant Deputy Minister (Defence Research and Development Canada (ADM((DRDC)) manages the IDEaS Program on behalf of DND. IDEaS challenges are derived from DND/CAF stakeholders. Level 1 organizations and public safety partners such as the Royal Canadian Navy, Canadian Army, Royal Canadian Air Force, Canadian Joint Operations and Command, Royal Canadian Mounted Police, Canadian Special Operations Forces Command, Veteran Affairs Canada and Crown Corporations are considered program clients.
Internal to IDEaS Program
- ADM(DRDC)'s Director General Innovation
Internal to DND/CAF
- Assistant Deputy Minister (Public Affairs)
- Assistant Deputy Minister (Materiel)
- Assistant Deputy Minister (Infrastructure and Environment)
- Assistant Deputy Minister (Information and Management)
- Royal Canadian Navy
- Canadian Army
- Royal Canadian Air Force
- Canadian Joint Operations and Command
- Canadian Special Operations Forces Command
External to DND/CAF
Key external partners include innovators from SMEs, industry, academia, non-governmental organizations and not-for-profits. The IDEaS Program maintains strategic partnerships with the Office of Small and Medium Enterprise, Canadian Intellectual Property Office, National Research Council, PSPC and Industrial Research Assistance Program. Clients include the Royal Canadian Mounted Police, Veteran Affairs Canada and Crown Corporations.
The program is measured against three set of indicators: one contained in the program foundational documents, and two as part of a Gs&Cs program through the Departmental Results Report (DRR) (one supporting Departmental Result (DR) 4.2 and the other specific to IDEaS service delivery).
Foundational Document Performance Measurement Indicators
The program is expected to achieve the following results:
Focus of this evaluation
- DND has effective mechanisms to engage innovators, solicit ideas and initiatives to address defence, public safety and security challenges.
- % of posted challenges that receive proposals, target 80%.
- Canadian innovators are engaged in solving defence and security challenges.
- % of posted problems that result in projects or initiatives to address them, target 75% in FY 2019/20.
- The knowledge is generated to and used by the innovation community.
- % of projects that receive funding for further development, target 10% by FY 2019/20.
- Defence and security communities adopt advice and innovative solutions created from the Canadian innovation system, improving their capabilities to protect, secure and defend Canada and its citizens as well as for Canadian companies to gain market share.
- % of IDEaS projects that resulted in useful advice, adoption, integration or eventual procurement of a new defence and security capability. Target 20% of IDEaS projects.
- % of defence and security senior leadership reporting satisfaction that their S&T requirements are addressed by IDEaS. Target 75% of surveyed managers providing a positive response.
DRR Measurement: IDEaS results supporting DR 4.2: Defence and security challenges are addressed through innovative solutions.
Figure E-1 Summary
This figure is bar graph. The bars indicate the percentage of IDEaS projects that resulted in useful advice, adoption, integration or eventual procurement of a new defence and security capability for FY 2018/19, FY 2019/20, and FY 2020/21 respectively. There is a horizontal dotted line that runs across the bar graph indicating the target of at least 20%.
- No percentage amount indicated
* For FY 2018/19, results are not available. New indicators as of 2018/19.
** In FY 2019/20, projects were still in their first phase of work or had just transitioned to a second phase of funding (Component 1b). Therefore, most solutions were still in the development stage. In FY 2020/21, projects began completing the second phase of funding (Component 1b), and their broader solutions were delivered to DND/CAF for consideration. Therefore, DND/CAF began to benefit from completed Component 1b projects.
The date to achieve the departmental result indicator is March 31, 2023.
Source: Departemental Plan 2022-23
DRR Measurement: IDEaS Service Delivery
Program standard timeliness performance measures are publicly reported. For FY 2019/20, six performance measures were listed although no results data is available. This can be attributed to the relatively recent launch in 2018 of the IDEaS Program. For FY 2020/21, three performance results are available. The DRR indicates that the program has succeeded to meet one target over the last fiscal year.
Source: Departmental Results Report, GC InfoBase
Figure E-2 Summary
This figure is a bar graph. The graph contains three separate bars. Each bar indicates the IDEaS program performance as measured by three performance results for FY 2020/21. There is a horizontal dotted line that runs across the top of the bar graph indicating the 100% performance target.
Issuing a payment following receipt of all required documentation within 45 calendar days.
Acknowledgement of the receipt of a funding request within 21 calendar days.
A funding decision, after receiving a completed application, is communicated within 90 Calendar days.
In 2022, the program sought an analysis of its socio-economic impacts to-date. The analysis report titled Socio-Economic Impact and Supply Chain Analysis of the IDEaS Program was completed in July, 2022 and examined the period from program inception in 2018 to December 31, 2021 during which the program invested approximately $215 million in Canadian innovators.
As a result of this investment, IDEaS achieved a significant upstream economic impact for Canada with a total economic output of approximately $428 million and over 1,800 jobs created. IDEaS’ upstream impact on Canada’s Gross Domestic Product (GDP) is approximately $270 million. The report notes this impact is well-balanced on a regional basis with distribution patterns similar to national GDP and population figures.
Based on a downstream impact scenario, downstream GDP was estimated at $1 billion with over 7,000 full time jobs created. Potential direct downstream revenues associated with IDEaS-sponsored projects are estimated to be $902 million. The report also notes the distribution of these revenues across Canada is similar to key national patterns.
Annex F: Definitions
- Date modified: