Integrated Strategic Analysis: Force Development

November 2021

1258-3-050 - ADM(RS)

Reviewed by ADM(RS) in accordance with the Access to Information Act.  Information UNCLASSIFIED.


Assistant Deputy Minister (Defence Research and Development Canada)
Assistant Deputy Minister (Infrastructure and Environment)
Assistant Deputy Minister (Information Management)
Assistant Deputy Minister (Materiel)
Assistant Deputy Minister (Review Services)
Assistant Deputy Minister (Science & Technology)
Air and Space Force Development
Canadian Army
Canadian Armed Forces
Canadian Special Operations Forces Command
Chief of Combat Systems Integration
Conceive, Design, Build, Manage
Chief of Force Development
C Prog
Chief of Programme
Defence Capabilities Board
Department of National Defence
Force Development
Fiscal Year
GBA Plus
Gender-based Analysis Plus​​
Integrated Strategic Analysis
Level 1
Land Force Development
North Atlantic Treaty Organization
Naval Force Development
Office of Collateral Interest
Other Government Department
Office of Primary Interest
Project Approval Directive
Personnel, Research/Development, Infrastructure, Concepts, IM/IT, Equipment, Gender-based Analysis Plus
Royal Canadian Air Force
Royal Canadian Navy
Canada’s defence policy: Strong, Secure, Engaged
Science and Technology Investment Steering Committee
Vice Chief of the Defence Staff

Back to Table of Contents

Executive Summary

This report presents the department-level, cross-cutting issues as a result of the three individual FD evaluations conducted during Fiscal Year (FY) 2020/21 by Assistant Deputy Minister (Review Services) (ADM(RS)) in compliance with the 2016 Treasury Board Policy on Results. The evaluations examined the relevance and performance of the Land, Air and Space, and Naval FD Programs over a five-year period, FY 2015/16 to FY 2019/20, and were conducted in accordance with the Department of National Defence (DND) and the Canadian Armed Forces (CAF). Force Development was previously evaluated by ADM(RS) in the Evaluation of Defence Capability Development Program in 2017. 

Project Description

The three FD Programs that were evaluated and included in this ISA are the Land FD (LFD) Program, Air and Space FD (ASFD) Program and Naval FD (NFD) Program. All three FD Programs contribute to Core Responsibility 4 – Future Force Design, which aims to “[d]evelop and design the future force through a deep understanding of the future operating environment and security risks to Canada and Canadian interests. Enhance Defence’s ability to identify, prevent, adapt and respond to a wide range of contingencies through collaborative innovation networks and advanced research." Footnote 1

The responsibility for the individual Programs falls within their respective services, and as military organizations, they report to the Chief of the Defence Staff. As part of the Capability Development processes, there is critical collaboration that occurs with other Level 1 (L1) organizations within DND/CAF, such as Chief of Force Development (CFD), Assistant Deputy Minister (Information Management) (ADM(IM)), Assistant Deputy Minister (Infrastructure and Environment) (ADM(IE)), and Assistant Deputy Minister (Materiel) (ADM(Mat)). 


The ISA was conducted using the evidence gathered as part of the three individual FD evaluations, which were focused on the immediate outcomes of those Programs.

Overall Conclusions

There is a continued need for the FD Programs and value added was confirmed across environments. Connections between the FD Programs and DND priorities/Strong, Secure, Engaged (SSE) were established; however, without additional emphasis placed on collaboration and enablers, technological integration, training and agile processes, the FD Programs will be challenged to address current and future capability requirements.


The FD Programs meet a continuing need for DND/CAF but are faced with challenges that reduce timeliness, agility and flexibility.

The main issues identified by the ISA that impact the FD Programs are: 

  1. Advancing technologies and innovation;
  2. FD Capability process;
  3. PRICIEG (Personnel, leadership, individual training; Research and development; Infrastructure and environment; Concepts, doctrine, collective training; Information management and information technology; Equipment, support and sustainability; Gender-based Analysis +);
  4. Collaboration;
  5. Interoperability; and
  6. Resources and Training.

Back to Table of Contents

Project Scope

The ISA is a high-level comparative analysis of the three individual FD evaluations, contained as separate chapters in this package. The ISA compares the main environmental FD Programs with the aim of identifying common areas of concern that require a departmental approach. The systemic challenges the FD Programs face are outside of the sphere of control of any one environment and require collaborative methods to improve capability outputs. Within the individual evaluations, findings are separated into findings internal to the services and findings external to the services. Findings that are related to ISA issues, will be indicated by an ISA icon within the individual evaluation chapters.ISA

The Defence Program Inventory Programs examined within this ISA are: Program; 4.3- Land Force Development (LFD) Program; 4.4- Air and Space Force Development (ASFD) Program; and 4.2- Naval Force Development (NFD), from FY 2015/16 to FY 2019/20. All three Programs contribute to Core Responsibility 4 – Future Force Design, within the Departmental Results Framework. Within the three Programs, the evaluations examined the identification of capability requirements and development.

Out of Scope

The ISA did not include in-depth analysis of cross-cutting issues outside of the data collected as part of the three FD Evaluations. Additionally, procurement, acquisition and the Joint Force Development Program were excluded from the scope and are expected to be evaluated in the future.

As outlined in Canada’s defence policy, SSE, in order for the CAF to be equipped to participate in a variety of operations, it “requires targeted and strategic investment in capabilities and equipment that can be used on domestic and international military operations.” The intention is for SSE to provide the policy direction to support this goal by increased attention on innovation to keep pace with rapidly changing technology and streamlining the defence procurement system. Each FD Program directly contributes to the fulfillment of multiple SSE commitments.


The issues and recommendations within this ISA are validated across the three separate FD evaluations, and the information is triangulated within the individual evaluations, allowing for high confidence in these results. The ISA is a summative report based on those supporting evaluations.

The main systemic challenges identified below are interconnected and interdependent. In order to address one issue, they all must be the focus of change and improvement.

Figure 1
Figure 1. The main systemic challenges identified are interconnected and interdependent.
Figure 1 Summary

This is a Venn diagram which represents the main systemic challenges identified are interconnected and interdependent.

  • Advancing technologies and innovation
  • Interoperability
  • Collaboration
  • Resources
  • FD Capability process

Force Development Expenditures

Force Development organizations receive between 0.18% and 1.33% of DND expenditures.

Figure 2
Figure 2. Force Development Program Expenditure as a Percentage (%) of DND Spending FY 15/16 – 19/20.
Figure 2 Summary

This figure represents fiscal year expenditures for the RCAF, CA and RCN Force Development programs between FY 15/16 and FY 19/20 with numbers in percentage of total DND spending. It is a bar graph for each Program with bars for FY 15/16, FY 16/17, FY 17/18, FY 18/19 and FY 19/20.

FY 15/16

  • ASFD: 1.18%
  • LFD: 0.52%
  • NFD: 0.21%

FY 16/17

  • ASFD: 1.33%
  • LFD: 0.50%
  • NFD: 0.18%

FY 17/18

  • ASFD: 0.85%
  • LFD: 0.51%
  • NFD: 0.20%

FY 18/19

  • ASFD: 0.97%
  • LFD: 0.42%
  • NFD: 0.23%

FY 19/20

  • ASFD: 0.30%
  • LFD: 0.40%
  • NFD: 0.23%

Program expenditures for the period include the restated figures for FY 2015/16 to FY 2018/19 and the actual amount in expenditures for FY 2019/20 as a percentage of  DND spending from FY 2015/16 to FY 2019/20. Relative expenditures for ASFD were considerably higher for all years, except FY 2019/20. NFD consistently had the smallest relative expenditures amongst the three environments.

Back to Table of Contents

Force Development Profile


The CAF FD system uses the Conceive, Design, Build, Manage (CDBM) model as part of the Defence Planning and Management Framework. 

Figure 3
Figure 3. Conceive, Design, Build, Manage model.

Modified from Capability Based Planning Handbook, CFD, June 2014, Figure 1-2, p.12

Figure 3 Summary

This figure contains four blocks which outline a description of each of the Conceive, Design, Build, Manage components.  Overlying this is the Defence Planning and Management Framework, which begins with DND/CAF strategy, followed by Capability-based Planning Process, then VCDS Project Approval Process and Force Posture &N Readiness Process.


The Canadian Army (CA) is responsible for the LFD Program, the Royal Canadian Air Force (RCAF) is responsible for the ASFD Program, and the Royal Canadian Navy (RCN) is responsible for the NFD Program.

Within DND/CAF, other stakeholders include: 

Stakeholders external to DND/CAF include: 

Back to Table of Contents

Innovation: There are challenges incorporating innovation.

ISSUE 1: The FD Programs' ability to innovate was present, but it faces some challenges in incorporating innovation through the lifespan of a project.

In order to maintain relevancy, FD Programs need to adapt current thinking and processes to embed innovative thinking throughout the capability development cycle, and drive actions through a capability development innovation agenda.

Across all environments, there have been examples of innovation being used to support FD activities. Despite this, it was noted that there are challenges limiting their capacity to incorporate innovation throughout the lifespan of a project. 

Observation: NFD has created a Directorate of Naval Innovation to support innovation through their FD activities. This is a new position, and its effectiveness could be monitored in order to determine if it could be considered as a best practice for ASFD and LFD.

Within DND/CAF, there are several cells that conduct research and development in support of FD activities. Once a concept is considered viable, the transition from the innovative research to the pre-commercialization of the product to produce it at the required scale is challenging. Exploring different partnerships with industry and academia could ease the difficulties in mass producing capabilities, possibly shorten project timelines, and more seamlessly incorporate innovation. 


1. DND/CAF needs to work towards enhancing the transition from innovation to capability delivery.

Advancing Technology & Innovation
Evaluation LFD ASFD NFD
Related Findings 4 & 5 6 3

Back to Table of Contents

FD Capability process: Departmental processes can reduce the agility of the capability development lifecycle.

ISSUE 2: Current FD governance and accountability processes, including procurement structures, inhibit programs' ability to deliver on time, on budget and technologically relevant capabilities.

The inflexibility of the Project Approval Directive (PAD) process was a recurring issue that was highlighted by stakeholders across the three FD programs. However, upon further examination it became clear that the PAD offers different levels of accountability for projects, but Program Managers and Project Directors were often opting for the most stringent level. A full analysis as to the reasons for this was not included as part of the evaluations but could potentially be a result of incomplete training of FD personnel. Additionally, it was noted that by following a faster process path, additional institutional concern is triggered, leading to amplified processes.

Across the FD environments, the need to simplify the procurement process was identified in order to be able to procure capabilities while they are still relevant. The issue was not simply one of the procurement process being lengthy, but also that of policies and guidelines pertaining to capability acquisition being out of date and too rigid. 

The pace of technology outruns traditional government procurement processes. Technological change gives the Forces the opportunity to enhance the capabilities being delivered, but also shapes the threats to which they will have to respond. The delta between the speed of technological change and our adversaries' ability to keep pace is where DND/CAF will face the greatest risk. ​

CDBM, the current model of capability development is linear, and as such, may be ill-suited to meet the changing capability gaps of the future and to keep pace with evolving technology. There is evidence to suggest that lines between the pillars are starting to be blurred. For example, aspects of the conceive pillar are starting to be integrated more into the design and build pillars. However, incorporating any changes after the early stages can seriously stall projects, negatively affecting both timelines and cost or the effectiveness of the resulting capability.


2. In order to implement a more agile and flexible system,  and to be better able to rapidly deliver technologically relevant capabilities, the FD community should encourage: 

  • application of the entirety of process options available
  • level and extent of early engagement with key FD enablers
FD Capability Processes
Evaluation LFD ASFD NFD
Related Findings 5 & 6 6 5

Back to Table of Contents

PRICIEG: Standardized incorporation of PRICIEG elements could improve outputs.

ISSUE 3: The integration of PRICIEG elements is beneficial in the development of a successful capability. However, inconsistent application and understanding of the PRICIEG tool impacts its effectiveness. 

According to the Capability-based Planning Handbook and PAD, capability projects must conduct a PRICIEG analysis. PRICIEG elements are intended to be considered and integrated throughout capability development. Initial considerations should be done in the Conceive phase, indicative analysis in the Design phase and substantive analysis in the Build phase.

Personnel, leadership, individual training

Research and development, operational research

Infrastructure and environment

Concepts, doctrine, collective training

Information management and information technology

Equipment, support and sustainability

Gender-based Analysis Plus (GBA Plus)

GBA Plus has only recently been incorporated into the PRICIEG framework across all three environments. As such, we cannot conclude definitively the effectiveness of this element.  

Suggestion for follow-up: Examine the effectiveness of the incorporation of GBA Plus in capability projects since its mandatory enactment in the PAD.

When integrated comprehensively and in a timely manner, PRICIEG analysis provides strategic insight into crucial considerations in the development of a capability.  Current application is sometimes inconsistent and could lead to incomplete understanding of infrastructure, personnel, training/doctrine, equipment and information management needs associated with the implementation of a new capability.

Inconsistent and incomplete application of PRICIEG was reported across FD environments.


3. In order to ensure optimal use of PRICIEG, review application, timing and training of PRICIEG.

Evaluation LFD ASFD NFD
Related Findings 7 7 4

Back to Table of Contents

Collaboration: Supporting L1s are perceived to take a project lens as opposed to holistically examining the capability gap identified. 

ISSUE 4: A project focus versus a focus on fulfilling capability gaps can create disconnect among organizations working in FD. 

Force Development activities are spread out across multiple groups and L1s, and often operate quite independently from each other. There is a need for areas to work together to maintain momentum and keep up with project changes and decisions. This support ideally begins during the identification and options analysis phase, and continues through to the definition and implementation phases, where organizations such as ADM(IM), ADM(Mat) and ADM(IE) become the project lead. 

As resource challenges are perceived to be experienced across L1s within the department (refer to issue 6), requested support can often be delayed. This extends timelines further and increases costs if component considerations are not included early enough in the project lifecycle. 


Please refer to recommendation 2.

Evaluation LFD ASFD NFD
Related Findings 8, 9 &10 8 5

Back to Table of Contents

Interoperability: The joint fabric of the interconnected platforms, sensors and networks is critical to the design process.  

ISSUE 5: Technical interoperability is challenged due to rapidly evolving technology and the spectrum of integration imperatives across architectures, systems and spectrum of Allied necessity. 

At the tactical and operational level, a rapid, joint, multi-faceted approach is vital to the success of current and future threat encounters. The achievement of operational effects to overcome opponents is better positioned with common situational awareness, the exchange of data and streamlined logistics both within the CAF and with key allied partners. Strengthened interoperability with key allied partners advances the ability to make informed decisions. The ability to deliver capabilities that are integrated hinges on being able to keep pace with evolving technology and the ability to effectively collaborate with other L1s throughout the FD process including the integration of best practices. 

In addition to evidence from the FD evaluations in support of this finding, several department-level documents highlight the need for the services to focus on the seamless integration of joint enablers, including space, cyber and exploiting intelligence functions.

Within DND/CAF, multiple IM platforms, both corporate and operational, are created in isolation and are often incompatible resulting in inconsistent information management across the department, out-of-date information, and general inefficiency in the management and sharing of information.  This has been noted in past ADM(RS) evaluations as well.  

Environmental strategic documentation identifies interoperability within the CAF as a key consideration when planning future capabilities. Overall survey results from the individual evaluations identified agreement that delivered capabilities are interoperable within the CAF. 

Domestically, DND/CAF is tasked to integrate and collaborate with OGDs on challenges. This was identified as a priority in strategic documents across environments. Survey respondents noted interoperability with OGDs as an area where improvements could be gained. 

Interoperability with allies is a priority for DND/CAF. Allied working groups highlight interoperability as a key capability component, and oversight over all DND/CAF FD projects will support the progression towards meeting interoperability standards. Overall survey results suggest that all three Programs are having some success in delivering capabilities that are aligned with their key allied partners. However, concern with keeping pace with technological advancements of allies was noted, particularly related to upgrades of capabilities. Canadian processes and requirements were not felt to facilitate DND/CAF's ability to react in a timely manner to advancements of allied partners. Interoperability was also identified as a key priority for the United Kingdom and Australia who use working groups, Memoranda of Understanding and agreed upon protocols through venues such as NATO, to work towards this goal.

ADM(IM) restructured its FD organization in June 2020, so that there is now a dedicated peer group established to facilitate early and ongoing collaboration with the services. This is expected to ease the process for future capability development projects. 


Please refer to recommendations 1 and 2.

Technical Interoperability
Evaluation LFD ASFD NFD
Related Findings 10 9 5

Back to Table of Contents

Resources & Training: Adequate and stable expertise are required for effective force development throughout all pillars.

ISSUE 6: Resource challenges and lack of training has impacted the FD Programs' ability to support a timely capability development process. 

There was a perceived shortage of resources (both financial and full-time employees) and FD-specific training across all three FD environments. The following challenges related to resources and capacity were identified across FD:

Note: A professional development regime for CFD employees has been implemented based on a recommendation from the 2017 Defence Capability Development Evaluation. However, this has not been applied to individual environments' FD training and career management efforts. 

The FD Programs are primarily composed of military members, with civilian members only making up a minority of resources. Human resources data also show that vacancy levels are moderate. 

Figure 4
Figure 4. FD Expenditures as a % of Total L1 Expenditures.
Figure 4 Summary

This graph illustrates the composition of the FD programs, broken down into military members and civilians. The graph also highlights the vacancy rate for each program.


  • 77% are military members
  • 23% are civilian members
  • 27% is the vacancy rate


  • 80% are military members
  • 20% are civilian members
  • 16% is the vacancy rate


  • 82% are military members
  • 18% are civilian members
  • 0% is the vacancy rate
Figure 5
Figure 5. HR Data for the FD Programs.
Figure 5 Summary

This chart represents fiscal expenditures as a percentage of total L1 expenditures for each of the three elements for FY 18/19 and 19/20.

FY 18/19:

  • ASFD: 19.6%
  • LFD: 10.83%
  • NFD: 6.03%

FY 19/20:

  • ASFD: 7.21%
  • LFD: 8.99%
  • NFD: 6.88%

Overall, the  CA and RCAF FD expenditures declined at a greater rate than their respective total L1 expenditures. In contrast, although RCN FD expenditures make up a smaller percentage of overall RCN expenditures, RCN FD expenditures increased at a greater rate (17.7%) than total RCN expenditures (3.2%).

Observation: In future planning, consideration needs to be given to the cost of the early stages of capability development. A system to track progress of identified capability gaps could help showcase impacts of resource constraints. This would lead to more informed decision making and prioritization.


4. In order to support capability development, FD organizations should examine: 

  • FD training 
  • sustainment of corporate of knowledge
Evaluation LFD ASFD NFD
Related Findings 11 10 & 11 6 & 7

Back to Table of Contents


The CAF FD organizations fill a continuing need within the department to produce the defence capabilities of the future as identified in SSE and other forward-looking strategic documents, yet face some ongoing challenges. The common issues are integral and thus require a collaborative approach to address. Challenges include advancing technology and interoperability, collaboration and workforce management, and FD processes and training.

Back to Table of Contents

Annex C – Management Action Plan

ADM(RS) Recommendation

1. DND/CAF needs to work towards enhancing the transition from innovation to capability delivery.

Management Action 1.1

To explore the potential to acquire innovations directly after an innovation process, without having to re-compete through a competitive acquisition process.        

Target Date: June 2023

Management Action 1.2

VCDS, in consultation with Project Sponsors, will improve awareness of the transition of innovation to FD. This is to include:   

  1. Training needs analysis to identify the level of understanding that FD staff have in this space; 
  2. Resourcing options to satisfy identified training shortcomings; and
  3. Continuing the development of a relationship with the Innovation for Defence Excellence and Security  to explore this space. 

This approach, once developed, should be presented to CFD for approval and incorporated into guidance. 

Target Date: July 2022

ADM(RS) Recommendation

2. In order to implement a more agile and flexible system, and to be better able to rapidly deliver technologically relevant capabilities, the FD community should encourage: 

  • application of the entirety of process options available
  • level and extent of early engagement with key FD enablers

Management Action 2.1

VCDS, in consultation with the three environments and other applicable project sponsors, will review process path decision making. This is to include: 

  1. Examination of Defence Service Project Portal – Project List to determine where the project path detail can be captured for departmental use and exploitation; and 
  2. Examination of process options for new capabilities compared to replacement or modernization of current capabilities, to include Template and Detail, Documents, and Governance. 

This approach, once developed, should be presented to CFD for approval and incorporated into guidance. 

Target Date: July 2022

Management Action 2.2

VCDS, in consultation with the three environments and other applicable project sponsors, will review the Defence Capabilities Board’s (DCB) visibility on how tasks assigned within the Force Capability Plan are progressing.  This is to include: 

  1. Implementing an update to the DCB membership about where all the projects are and what process path they are on; and 
  2. Implement periodic sponsor updates to DCB on how they are progressing the tasks given to them in the Force Capability Plan which will speak to the capability gaps and how they are being closed/mitigated, thus giving DCB membership the context in which to nest the individual projects they see at DCB.  

This approach, once developed, should be presented to DCB for approval and incorporated into guidance. 

Target Date: October 2022

ADM(RS) Recommendation

3. In order to ensure optimal use of PRICIEG, review application, timing and training of PRICIEG.

Management Action 3.1

VCDS, in consultation with the three environments and other applicable project sponsors, will review PRICIEG information for decision making. This is to include: 

  1. Examination of current PRICIEG direction within PAD; 
  2. Staffing of applicable PAD amendment requests, as applicable; 
  3. Training needs analysis; and 
  4. Resourcing options to satisfy identified training shortcomings. 

This approach, once developed, should be presented to CFD for approval and incorporated into guidance. 

OCI: All L1s
Target Date: July 2022

ADM(RS) Recommendation

4. In order to support capability development, FD organizations should examine: 

  • FD training 
  • sustainment of corporate of knowledge  

Management Action 4.1

VCDS, in consultation with the three environments and other applicable project sponsors, develop an approach to train FD personnel. This is to include: 

  1. Training needs analysis; 
  2. Resourcing options; and 
  3. Succession planning strategies. 

This approach, once developed, should be presented to Force Development Forum for approval and incorporated into guidance. 

Target Date: July 2022

Back to Table of Contents

Annex D – Methodology

Data Sources

The issues and recommendations within this report were informed by multiple lines of evidence collected throughout the conduct phase of the evaluations. These lines of evidence were triangulated with each other and verified with relevant stakeholders to ensure their validity.  The research methodology used to design and source data for the ISA are as follows:

Literature Review: As part of the planning phase of the FD evaluations, preliminary document reviews were conducted to develop a foundational understanding of the ASFD, LFD and NFD programs to create logic models and scopes. This was expanded upon during the conduct phase of the evaluations and throughout the ISA, as other documents were examined to find data that would help in the assessment. Documents included: government websites; departmental administrative reports; and program documents, both in draft and finalized. 

Administrative and Financial Data: Administrative data was collected from the three FD programs as well as from central databases to determine the number of individuals working in the program and overall vacancy rates. Expenditure data was collected from the individual FD programs and verified with DND central financial analysts. 

Comparative Analysis: Each of the three FD evaluations surveyed their respective stakeholders. The ISA team conducted a comparative analysis of the aggregate data to inform the issues and recommendations highlighted herein. ASDF also conducted a questionnaire.

Benchmarking: The evaluations conducted a comparative analysis by benchmarking Canada’s FD programs with the military force development efforts of the United Kingdom and Australia. NFD only received responses from the United Kingdom. Comparison was enabled using various sources such as government publications available online and data collected via questionnaires sent to representatives of the British and Australian Armed Forces.

Interviews: The evaluation teams of the FD Programs conducted interviews with stakeholders. These responses were aggregated to inform opinion and perspectives in support of the ISA. All references to “senior program managers” only refer to those who are at the director level and above in the RCN, RCAF and CA. Other organizations external to the Programs were also interviewed as part of the ISA.

Survey:  Surveys were conducted in 2020. The survey was sent to internal DND stakeholders. The surveys remained online for 2 – 4 weeks, were administered in English and French, and integrated GBA Plus principles. The target audience for the surveys was comprised of senior managers, military members, project directors, project managers and other DND/CAF stakeholders. Survey distribution relied on the Points of Contact identified within each organization.

Case Study: Case studies were conducted in support of the evaluations to analyze capability projects as they progressed through the four pillars of the FD process. In particular, they were examined through the perspectives of:

The cases selected included: 

Back to Table of Contents

Annex E – Limitations

Table E-1. Limitations and Mitigations
Limitation Mitigation
Timelines: The ISA was conceived and green lit mid-evaluation cycle which greatly reduced the resources that could be dedicated to its development. In order to successfully meet tight timelines, the evaluation team designed the ISA to be a chapeau piece, aggregating data from the three FD evaluations and deducing common themes.
Personnel Shortage: Due to the tight deadlines associated with completing the ISA and pre-existing personnel shortages across ADM(RS), the ISA evaluation team was understaffed. The ISA team received additional staffing support to undertake multiple activities.
Data Source Constraints: As the themes highlighted in the ISA are based on data collected by the ASFD, LFD and NFD evaluations, it is likely that it could have benefitted from exploring sources of data other than those utilized by the FD evaluations. LFD also experienced a lack of Program Data access. Data from the FD program was validated through triangulation with other lines of evidence and data collection tools within the individual evaluations.  Caveats around program data are identified in the report to ensure clarity that the list of projects was only based on a status point at one period of time. Trends/issues identified in program data were triangulated with other data sources to provide a holistic picture.
Inherited/Secondary Biases: As the ISA employed data collected by the ASFD, LFD and NFD evaluations, any biases present in the collection or analysis of data in the respective evaluations may surface in the ISA. Interview comments from each FD program evaluation were corroborated with other sources to ensure validity. Interview notes were conducted by more than one individual to confirm understanding of discussions and decrease the likelihood of bias.
Low Survey Response Rate: The number of respondents to the surveys was limited resulting in a possibility of skewed results. Survey data was validated through triangulation with other lines of evidence and data collection tools. Interviews with senior program management confirmed the accuracy of results, which facilitated high-level discussions during these interviews.
Case Study Data:  A relatively small sample size of case studies complicates extrapolation of findings to make wider claims about the FD Programs. Case studies were used primarily to support findings that had already been established by other evidence.  ASFD also added four cases which included the new PRICIEG tool to analyze.
Ongoing Pandemic: The evaluation was conducted during the global outbreak of COVID-19 which prevented site visits that would contribute additional data. Multiple data sources were explored and employed to compensate for the limitations of the ongoing health crisis.

Back to Table of Contents

Page details

Date modified: