Evaluation of Basic Training List Management

Alternate Formats

PDF Version
(1,503 KB, 40 pages)

Assistant Deputy Minister (Review Services)

ADM(DIA): Assistant Deputy Minister (Data, Innovation and Analytics)
ADM(IM): Assistant Deputy Minister (Information Management)
ADM(RS): Assistant Deputy Minister (Review Services)
AMOR: Annual Military Occupation Review
ARA: Authorities, Roles and Accountabilities
Ares: Army Reserve
BMOQ: Basic Military Officer Qualification
BMQ: Basic Military Qualification
BTAG: Basic Training Administrative Guide
BTL: Basic Training List
BTLMS: Basic Training List Management System
CA: Canadian Army
CAF: Canadian Armed Forces
CANSOF: Canadian Special Operations Forces
CBI: Compensation and Benefits Instructions for the Canadian Forces
CDA: Canadian Defence Academy
CFINTCOM: Canadian Forces Intelligence Command
CFLRS: Canadian Forces Leadership and Recruit School
CFRG: Canadian Forces Recruiting Group
CMP: Chief of Military Personnel
COT: Compulsory Occupation Transfer
CTC: Combat Training Centre
D Mil C: Director of Military Careers
DAOD: Defence Administrative Orders and Directives
DMCA: Director Military Careers Administration
DMPGPC: Director Military Personnel Generation Planning and Coordination
DTA: Designated Training Authority
FTE: Flying Training Evaluation
FY: Fiscal Year
GBA+: Gender Based Analysis Plus

HR: Human Resources
HRMS: Human Resource Management System
IT: Information Technology
IT&E: Individual Training and Education
MILPERSGEN: Military Personnel Generation
MOS ID: Military Occupational Structure Identification
MPGG: Military Personnel Generation Group
MPGTG: Military Personnel Generation Training Group
NCM: Non-Commissioned Member
OCI: Office of Collateral Interest
OFP: Operationally Functional Point
Op GEN: Operation GENERATION
OPI: Office of Primary Interest
OSS: Occupation Speciality Specification
PARRA: Production, Attrition, Recruiting, Retention and Analysis
PAT: Personnel Awaiting Training
PLAR: Prior Learning Assessment and Recognition
PML: Preferred Manning Levels
RCAF: Royal Canadian Air Force
RCN: Royal Canadian Navy
SOP: Standard Operating Procedure
SSE: Canada’s defence policy: Strong, Secure, Engaged
SUTL: Subsidized University Training List
SWE: Salary and Wage Envelope
TA: Training Authority
TB: Treasury Board
TE: Training Establishment
TEE: Trained Effective Establishment
TES: Trained Effective Strength
VCDS: Vice Chief of the Defence Staff

Overall Assessment

BTL directives should be assessed to ensure alignment with DND/CAF policies and priorities.
BTL-related databases and Information Technology (IT) solutions should be harmonized to ensure a greater enterprise-wide tracking and controlling of members’ data.
Resourcing needs should be assessed to ensure sufficient capacity to support BTL training and operational oversight.

The BTL is the list of Canadian Armed Forces (CAF) members who have been recruited undergoing basic and common training. While on the BTL, members are considered un-trained and remain so until they reach their Operationally Functional Point (OFP). The OFP is the point at which a military member completes all qualifications required for first employment in their assigned military occupation. At this point a member is posted off BTL into a Trained Effective Establishment (TEE) position and counted as Trained Effective Strength (TES).

BTL management is a set of processes designed to take recruits from the initial intake stage of their career through to OFP. It is the mechanism used by the CAF to manage recruits and their training before their first posting. BTL management is not a program per se but rather a series of activities that link several programs across recruiting, military training, occupational training and subsidized post-secondary education.

Key Findings and Recommendations

Table 1 lists a summary of evaluation findings and associated recommendations.

Table 1. Key Findings and Recommendations. This table lists the key findings along with the associated recommendations.
Key Findings	Recommendations
Relevance
1. The current policies and directives supporting BTL management do not fully align with DND/CAF priorities.	1. In collaboration with Training Authorities/Designated Training Authorities (TA/DTA), review and/or develop policies and directives regarding BTL management, ensuring they: Are aligned with DND/CAF priorities (e.g., Canada’s defence policy: Strong, Secure, Engaged (SSE), Operation GENERATION (Op GEN)).
Effectiveness
2. Due to a lack of readily available data, it is not possible to fully ascertain the extent to which BTL management contributes to meeting the CAF’s needs for appropriately trained members at OFP.	1. In collaboration with TAs/DTAs, review and/or develop policies and directives regarding BTL management, ensuring they: Include provisions on reporting and tracking disaggregated member data, and establish optimal paths to OFP by occupation.
Efficiency
3. The optimal duration to reach OFP is not well-defined nor consistently tracked.	2. BTL management, Training Establishments (TE), and all elements of the CAF harmonize their individual databases and IT solutions to ensure that these systems align functionally and contribute to the CAF common operating picture, including tracking key performance indicators. This would align with Defence Policy expectations, and improve the ability to view, track and analyze members’ Human Resources (HR) data.
4. Multiple factors affect the efficiency and the duration that members spend on BTL, including inconsistent information management, communication, capacity, language and individual members’ life events.	3. Assess resourcing needs to help ensure that: Adequate capacity is available in TEs to implement the full potential of BTL training for members and meet increased throughput objectives in line with SSE targets.
5. There is an inconsistent approach to the employment and utilization of Personnel Awaiting Training (PAT).	1. In collaboration with TAs/DTAs, review and/or develop policies and directives regarding BTL management, ensuring they: Include provisions on how to effectively utilize PAT, and how to more clearly communicate and publish potential pathways to OFP.
Design and Governance
6. Authorities, Responsibilities and Accountabilities (ARA) are not well established, leading to inconsistencies in how BTL is managed.	1. In collaboration with TAs/DTAs, review and/or develop policies and directives regarding BTL management, ensuring they: Clarify ARA for BTL management across the CAF.
7. The existing capacity of BTL managers was found to be insufficient in order to ensure effective processing of members from basic training to OFP.	3. Assess resourcing needs to help ensure that: There is sufficient capacity of BTL managers to support and sustain effective operational oversight.

Table 1 Details - Summary of Key Findings and Recommendations

Table 1 Summary:

This table represents the key findings and recommendations. There are two columns and 12 rows including titles and labels throughout the table. The columns are labelled ‘Key Findings’ and ‘Recommendations’ and the rows list the evaluation issues, the key findings for each issue, and if applicable, recommendation(s) corresponding to each key finding.

Key finding of relevance #1: The current policies and directives supporting BTL management do not fully align with DND/CAF priorities.

Associated recommendation: In collaboration with TAs/DTAs, review and/or develop policies and directives regarding BTL management, ensuring they:

Are aligned with DND/CAF priorities (e.g., SSE, Op GEN).

Key finding of effectiveness #2: Due to a lack of readily available data, it is not possible to fully ascertain the extent to which BTL management contributes to meeting the CAF’s needs for appropriately trained members at OFP.

Associated recommendation: In collaboration with TAs/DTAs, review and/or develop policies and directives regarding BTL management, ensuring they:

Include provisions on reporting and tracking disaggregated member data and establish optimal paths to OFP by occupation.

Key finding of efficiency #3: The optimal duration to reach OFP is not well-defined nor consistently tracked.

Associated recommendation: BTL management, TEs, and all elements of the CAF harmonize their individual databases and IT solutions to ensure that these systems align functionally and contribute to the CAF common operating picture, including tracking key performance indicators. This would align with Defence Policy expectations, and improve the ability to view, track and analyze members’ HR data.

Key finding of efficiency #4: Multiple factors affect the efficiency and the duration that members spend on BTL, including inconsistent information management, communication, capacity, language and individual members’ life events.

Associated recommendation: Assess resourcing needs to help ensure that:

Adequate capacity is available in TEs to implement the full potential of BTL training for members and meet increased throughput objectives in line with SSE targets.

Key finding of efficiency #5: There is an inconsistent approach to the employment and utilization of PAT.

Associated recommendation: In collaboration with TAs/DTAs, review and/or develop policies and directives regarding BTL management, ensuring they:

Include provisions on how to effectively utilize PAT, and how to more clearly communicate and publish potential pathways to OFP.

Key finding of design and governance #6: Authorities, Responsibilities and Accountabilities (ARA) are not well established, leading to inconsistencies in how BTL is managed.

Associated recommendation: In collaboration with TAs/DTAs, review and/or develop policies and directives regarding BTL management, ensuring they:

Clarify ARA for BTL management across the CAF.

Key finding of design and governance #7: The existing capacity of BTL managers was found to be insufficient in order to ensure effective processing of members from basic training to OFP.

Associated recommendation: Assess resourcing needs to help ensure that:
There is sufficient capacity of BTL managers to support and sustain effective operational oversight.

Back to Table of Contents

1.0 Introduction

1.1 Context for the Evaluation

This report presents the results of the ADM(RS) Evaluation of Basic Training List Management. The evaluation was conducted as a component of the DND Five-Year Evaluation Plan, as approved by the Performance Measurement and Evaluation Committee in July 2017. Initially, this report was to evaluate Individual Training and Professional Military Education (Departmental Results Framework program 3.2). However, in consultation with the Chief of Military Personnel (CMP) in March 2018, it was determined that such an evaluation would overlap with other ongoing audit and evaluation activities. As such, it was decided to focus the evaluation on BTL management, which also covers Core Responsibilities #2 and #3 (Ready Forces; Defence Team), and is a key component of the Individual Training and Education (IT&E) Program.

The evaluation was conducted in accordance with the Treasury Board (TB) Policy on Results (2016). It examined the relevance, effectiveness, efficiency, and design and governance of BTL management for FYs 2013/14 to 2017/18.

Back to Table of Contents

1.2 Program Profile

1.2.1 Program Description

The BTL is comprised of newly recruited CAF members within Development Period 1 who require basic, environmental and occupational training. It is a Human Resources Management System (HRMS) accounting list for members undergoing basic-level training required to reach their OFP. The OFP is the point at which a military member completes all qualifications required for first employment in their assigned military occupation. At this point, a member is posted off the BTL or Subsidized University Training List (SUTL), and into a TEE position, where they are counted as TES.

The term ‘BTL management’ describes the activities that oversee personnel while they progress from intake to OFP. From 2014 to 2018, there was an average of approximately 8,500 personnel on the BTL.^{Footnote 1}

BTL management is not a program per se but rather a series of activities that link several programs, across recruiting, military training, occupational training and subsidized post-secondary education. Examples of BTL management activities include:

Career management functions for new personnel;
Assigning members to occupational training serials;
Coordinating summer training for Royal Military College and civilian university students (e.g., on the job education/training);
Posting members to TEs for core or pre-requisite courses;
Posting of PAT to operational units; and
Preparation and dispatch of posting/TC instructions for qualification of officers.

While variations exist between the more than 100 occupations in the CAF, new members typically follow a similar path from intake into OFP:

Intake: Once an individual accepts an offer to join the CAF, their enrolment is communicated to BTL managers via an Enrolment or Transfer and Posting Instruction. Military Personnel Generation (MILPERSGEN) is responsible for recruits upon intake into the Forces.
Basic Training: After being sworn in, the recruit is sent to the Canadian Forces Leadership and Recruit School (CFLRS) for basic military training, which is common to all CAF members: Basic Military Qualification (BMQ) for Non-Commissioned Member (NCM) recruits and Basic Military Officer Qualification (BMOQ) for officer recruits. A local Canadian Forces Recruiting Group (CFRG) is responsible for ensuring that BMQ/BMOQ courses are loaded to capacity. The Military Personnel Generation Training Group (MPGTG) BTL team, within CMP, is responsible for posting messages for recruits at week five of BMQ/BMOQ. Training Authorities (TA) determine where NCMs go, and the Director of Military Careers (D Mil C) in CMP determines where officers go.
Environmental and Occupational Training: Once a recruit leaves CFLRS, they enter the third stage of BTL management which is the responsibility of the individual TAs/DTAs. Training requirements are completed in accordance with the requirements of the environment and occupation (as outlined in the respective occupations specifications). Upon completion of all necessary training, the TE notifies D Mil C about a member’s graduation from all necessary courses and their reaching OFP. D Mil C then releases the member’s first post-BTL posting message, which marks their departure from the BTL into effective strength, and posting to a unit.

1.2.2 Program Objectives

The overarching priorities for defence are enshrined in the Government’s defence policy: Strong, Secure, Engaged (SSE).^{Footnote 2} The policy includes the main objectives for recruitment, training and retention across the CAF. In support of SSE, the Operation GENERATION Directive provides the corporate-level expectations and direction to grow the Force to SSE levels by 2026. A key component to the directive is an improved BTL management system.

The Operation GENERATION Directive commits National Defence to optimize and synchronize BTL management practices to improve the predictability, efficiency and transparency of occupation training to OFP, with a particular focus on reducing unproductive (non-training) time. The extent to which BTL is managed in an efficient and effective way is crucial to meeting policy priorities for recruitment and force augmentation.

1.2.3 Stakeholders

New and potential recruits are considered external stakeholders of BTL program activities. They are the most impacted by BTL decision making.

Internal stakeholders include CMP as the Functional Authority. The Operation GENERATION Directive states that Chief Military Personnel Command (CMPC) will act as the strategic integrator of military personnel production. This extends from enrolment to basic military training, through occupational training to the attainment of OFP. CMPC is to work to increase capacity in the recruiting and training system in concert with L1s.

Other internal stakeholders include the Royal Canadian Air Force (RCAF), Canadian Army (CA) and the Royal Canadian Navy (RCN), as well as the TAs and the various DTAs who are responsible for managing BTL in their assigned specifications. This includes 11 TAs/DTAs across more than 100 occupations, which are key to operationalizing basic training and other recruitment-related functions.

As per the directive, other internal stakeholders are responsible for building an IT-enabled Basic Training List Management System (BTLMS), along with the requisite system infrastructure to ensure accuracy of reporting. Specifically, the Director General Military Careers (DGMC) is to assist in building the BTLMS. The Assistant Deputy Minister (Information Management) (ADM(IM)) will assist in the development and fielding of an IT solution.

1.2.4 Resources

As it is not a program, direct spending on BTL management has not been consistently identified or tracked. The only formal spending data relevant to BTL management is from the overarching IT&E Program. Table 2 summarizes the overall cost breakdown for IT&E.

Table 2. Individual Training and Professional Military Education Expenditures. This table outlines the expenditures for individual and professional military education by FY ($ millions).^{Footnote 3}
-	2014/15	2015/16	2016/17	2017/18
Services	1,487	1,565	1,637	2,245
Goods, lands, buildings and works	83	93	86	95
Other expenditures	13	14	15	16
Total	1,584	1,672	1,738	2,356

Table 2 Details - Expenditures on Individual Training and Professional Military Education, by standard object, fiscal years 2014/15 to 2017/18 ($ millions)

Table 2 Summary:

Table 2 shows the expenditures for the DND/CAF’s Individual Training and Professional Military Education program, broken down by standard object, from fiscal year 2014/15 to 2017/18. It has five columns and five rows, including titles. The left-hand column lists the standard objects. Each of the four columns to the right of it show one fiscal year, from 2014/15 to 2017/18.

In fiscal year 2014/15, program expenditures amounted to $1,487 million for services; $83 million for goods, lands, buildings and works; and $13 million for other expenditures; for a total of $1,584 million for the year.

In fiscal year 2015/16, program expenditures amounted to $1,565 million for services; $93 million for goods, lands, buildings and works; and $14 million for other expenditures; for a total of $1,672 million for the year.

In fiscal year 2016/17, program expenditures amounted to $1,637 million for services; $86 million for goods, lands, buildings and works; and $15 million for other expenditures; for a total of $1,738 million for the year.

In fiscal year 2017/18, program expenditures amounted to $2,245 million for services; $95 million for goods, lands, buildings and works; and $16 million for other expenditures; for a total of $2,356 million for the year.

1.2.5 Previous Reviews

There have been previous evaluations and other reviews regarding individual training, which have addressed BTL management issues. ADM(RS) evaluated the following programs: Recruitment and Basic Military Training (November 2012) and the Canadian Defence Academy (January 2015). The Ombudsman of the Department of National Defence and the Canadian Armed Forces also conducted a preliminary assessment of the management of PAT (January 2014). Across all three reviews, the same recurring themes can be found: the progress of trainees is not managed nor tracked effectively leading to deficiencies in retention and OFP target attainment, and a disproportionate number of PAT due to lack of course availability and training alignment.

Back to Table of Contents

1.3 Evaluation Scope

1.3.1 Coverage and Responsibilities

Within DND/CAF’s Departmental Results Framework, BTL management falls within the strategic outcome of “Defence Remains Continually Prepared to Deliver National Defence and Defence Services in Alignment with Canadian Interests and Values.” BTL management is a component of Program 3.2: Individual Training and Professional Military Education (IT&E),^{Footnote 4} but some of its activities are covered in several programs within Core Responsibility 2: Ready Forces.^{Footnote 5}

The evaluation focussed on how effective BTL Management is at providing the CAF with appropriately trained members at the OFP; how efficiently members are managed from recruitment to OFP; and on the clarity of BTL governance and design. The evaluation examined BTL management over the period of FY 2013/14 to FY 2017/18.

The evaluation covered BTL management for the Regular Forces only. It excluded related activities that do not specifically fall under BTL management, such as training and recruitment. The evaluation scope also excluded the SUTL, as well as BTL management for the Reserve Forces.

1.3.2 Issues and Questions

In accordance with the Treasury Board Secretariat Directive on Results (2016),^{Footnote 6} the evaluation report addressed issues related to relevance, effectiveness, efficiency, as well as governance and design. An evaluation matrix listing each of the evaluation questions, with associated indicators, is provided at Annex D.

1.3.3 Methodology

Evaluation methods included a detailed document review, financial and HR data analysis, key informant interviews and a survey of school commandants. Scoping for the evaluation commenced in April 2018, and analysis was undertaken between February and August 2019. The methodology included the following sources of information:

A comprehensive review of 68 documents (corporate, policy and program-level)
Survey of 26 commandants of CAF TEs, for their input on the effectiveness and efficiency of BTL management
260 in-depth interviews with primary and secondary stakeholders as an important source of qualitative and informed opinions
Analysis of HR data
- Career progression of members recruited in 2013
- Historical PARRA data
Analysis of financial data

Details on the methodology used to gather evidence in support of the evaluation questions can be found at Annex B.

Back to Table of Contents

2.0 Findings and Recommendations

2.1 Relevance

This section examines the extent to which BTL management is aligned with government and DND/CAF priorities.

Key Finding 1: The current policies and directives supporting BTL management do not fully align with DND/CAF priorities.

Policy Gaps

BTL management is not currently supported by policies and directives specific to its activities, nor is there clear alignment with corporate-level policies and priorities. Although Defence Administrative Orders and Directives (DAOD) assign a Commander CDA responsible for establishing policy in the area of professional development, there are no corporate-level policies, directives or instructions for BTL.^{Footnote 7} The only orienting document available regarding BTL management is the Basic Training Administrative Guide (BTAG). However, it is perceived as outdated, with an intended use as internal guidance only, and originally designed for a specific base. Multiple interviewees highlighted the need for a clear policy and updated guidance on BTL in order to clarify relationships across L1s and reduce inconsistencies in treatment of BTL members. In a survey of TE commandants, the majority (73 percent) agreed or strongly agreed that improving and updating BTL policy could significantly reduce time on BTL.

Other CAF policies that affect BTL members may not be aligned with government and DND/CAF priorities. For instance, the Compensation and Benefits Instructions allow BTL members to move their dependants at public expense when the member is required to move “other than temporarily.”^{Footnote 8} The word “temporarily” is not defined in this instruction. According to the BTAG, the way in which a member is sent to training can vary significantly from one case to another (e.g., restricted posting, prohibited posting, attached posting or temporary duty), each of which entails different benefits entitlements. The manner in which this is decided, according to the BTAG, is dependent on the member’s career status and the duration of training. However, given sometimes lengthy times awaiting training and unclear BTL pathways as described in later findings, members may often spend longer at one location than anticipated. According to many BTL managers interviewed across the CAF, the lack of clear direction on how to determine the type of posting leads to inconsistent practice.

Certain grievances have arisen as a result of this issue, some of which have reached the Military Grievances External Review Committee, and the Chief of the Defence Staff (CDS) (Final Authority). The CDS has acknowledged that members who remain on the BTL in a single place for more than 12 months should be entitled to family relocation benefits. In the CDS’ ruling:

“I direct CMP to conduct a review of the relevant administrative policies for postings of members on the BTL whose training exceeds one year in a single geographic location. As I see it, those members whose training exceeds one year in a single geographical location and want to relocate their dependents should be posted “restricted” (not “prohibited”).”

The Operation GENERATION Directive has outlined plans to ensure future alignment with Government and DND/CAF priorities. For example, in support of SSE, the Operation GENERATION Directive provides the corporate-level expectations and direction to grow the Force to SSE levels by 2026. A key component to the Directive is an improved BTL management system. Specifically, it has committed to optimizing and synchronizing BTL management practices to improve the predictability, efficiency and transparency of occupation training to OFP, with a particular focus on reducing unproductive (non-training) time. The extent to which BTL is managed in an efficient and effective way is crucial to meeting policy priorities for recruitment and force augmentation.

MILPERSGEN is in the process of developing a directive and updated guidance on BTL administration. This work is ongoing at the time of this report.

ADM(RS) Recommendation

Back to Table of Contents

2.2 Effectiveness

This section examines the extent to which BTL-related program objectives and activities are being effectively delivered.

Meeting the CAF’s Needs for Trained Members

Key Finding 2: Due to a lack of readily available data, it is not possible to fully ascertain the extent to which BTL management contributes to meeting the CAF’s needs for appropriately trained members at OFP.

Although there is limited data available, there is some indication that the CAF is not meeting its need for appropriately trained members reaching OFP in a timely manner. BTL management’s contribution to this outcome is not known. There is limited evidence that graduation rates from BMQ are in line with international comparators.

BTL management, in conjunction with other activities, supports the key outcome of ensuring the supply of appropriately trained members at OFP. The extent to which this outcome is being met is not currently measured: the number of personnel reaching OFP each year is not systematically tracked and no targets have been set (except those in the annual Strategic Intake Plan). However, there is some evidence that this outcome is not fully being met. For example, in a survey of TE Commandants undertaken for the evaluation, the majority of respondents (54 percent) disagreed or strongly disagreed that the number of members reaching OFP each year in the occupations trained in their TE is sufficient to meet the CAF’s needs. A comparison of the CAF’s current manning levels (TES) to targets (Preferred Manning Levels (PML)) reinforces this notion: as shown in Figure 1, 77 percent of CAF occupations were understaffed (i.e., TES levels were lower than PML) in 2018, and half (49 percent) were chronically understaffed (i.e., TES levels were lower than PML in each year from 2014 to 2018).^{Footnote 9}

**Figure 1. CAF Occupations Understaffed.** This bar graph shows the percentage of occupations understaffed, by environment.

Figure 1 Details - Percentage of occupations understaffed, by environment

While a contributing factor to this gap, BTL is only one of many factors involved in affecting the achievement of PML targets. Some of the factors external to BTL management include:

Recruitment: While recruitment comes close to meeting targets (i.e., Strategic intake plan) CAF-wide, the recruitment rate in several occupations is less than 50 percent of the target.^{Footnote 10}
Releases: Members releasing from effective strength diminish the CAF’s abilities to meet manning targets. In the evaluation period, an average of about 5,200 personnel were released from the CAF each year, a number roughly equivalent to the annual intake.^{Footnote 11}
Quality of training and management of training: Many interviewees, and 77 percent of survey respondents, have confirmed that TEs have established appropriate mechanisms to ensure the quality and effectiveness of training. The perception from most survey respondents (88 percent) and interviewees is that the quality of training is high in most occupations, and that members reaching OFP are capable of meeting expectations of their occupation.

The factors affecting this gap that are at least partly within the control of BTL management include: the BTL graduation rate and time on BTL. The latter is discussed in the following finding.

Graduation Rates

Graduation rates are not systematically tracked for the CAF. According to a 2018 study by CMP, the graduation rate in that year for NCMs, from BMQ to OFP, was estimated at 72 percent.^{Footnote 12} While a total was not calculated for officers, the average across occupations was approximately the same.

Although it is not possible to determine if graduation rates for BTL as a whole are within acceptable levels, there is some evidence that the graduation rates from BMQ are. Graduation rates for NCM recruit training (i.e., the equivalent of Canada’s BMQ) in the other Five Eyes countries appear to be in the same range as those of Canada (see Figure 2).^{Footnote 13}

**Figure 2. Graduation Rates.** This bar graph shows an international comparison of graduation rates from recruit training.

Figure 2 Details - Graduation rates from recruit training, by environment, international comparison

Figure 2 Summary:

Figure 2 is a bar graph showing a comparison across countries of the graduation rates from recruit training for each country’s navy, army and air force, and for Canada, CMP.

Recruit training graduation rates for the navy were as follows. The New Zealand Navy graduation rate was 97%. The US Navy graduation rate was 92%. The Australian Navy graduation rate was 89%. The Canadian Navy graduation rate was 88%. The UK Navy graduation rate was 86%.

Recruit training graduation rates for the army were as follows. The UK Army graduation rate was 90%. The Canadian Army graduation rate was 88%. The US Army graduation rate was 88%. The New Zealand Army graduation rate was 87%. The Australian Army graduation rate was 87%.

Recruit training graduation rates for the air force were as follows. The New Zealand Air Force graduation rate was 95%. The UK Air Force graduation rate was 95%. The US Air Force graduation rate was 94%. The Australian Air Force graduation rate was 90%. The Canadian Air Force graduation rate was 88%.

Finally, the recruit training graduation rate for Canada’s CMP was 86%.

Similarly, there is currently no systematic tracking of graduation rates broken down by demographic grouping. SSE notes that the CAF has committed to an increase in its population of female members from 15 percent to 25 percent by 2026. The defence policy also prioritizes that activities and programs to grow the CAF should be inclusive of a Gender-based Analysis Plus (GBA+) lens. While BTL is impacted by this commitment, a systematic breakdown of personnel who completed BMQ/BMOQ and BTL up to OFP by gender and minority group is not currently available.

Overall, in the absence of systematic data on BTL duration and graduation rates, it is impossible to assess the extent to which BTL contributes to the outcome ensuring a sufficient number of trained members to OFP.

ADM(RS) Recommendation

Back to Table of Contents

2.3 Efficiency

This section examines to what extent members reach OFP in a timely manner and the factors that affect the time spent on BTL.

Key Finding 3: The optimal duration to reach OFP is not well-defined nor consistently tracked.

Monitoring and Tracking of Members

In evaluating BTL management, ADM(RS) was unable to locate systematically tracked data for the duration of time it took members to reach OFP. A study carried out by MPGTG indicates, for a sample of five NCM occupations in the Navy, Air Force and CMP, a median time to OFP of just over a year.^{Footnote 14} Anecdotally, the Army has estimated time to OFP from 10 to 24 months, while the Air Force noted that pilots can take up to three years to reach OFP.^{Footnote 15} However, these studies are not comprehensive nor systematic, and the methodology used to obtain the data is not clear. Therefore, any resulting observations are limited in value.

Subjective opinion appears to favour the notion that time to BTL should be shorter. In a survey of TE Commandants, the majority of respondents (62 percent) agreed or strongly agreed that, for the Regular Force overall, the amount of time members spend on BTL is too long.

Feedback from a few stakeholder interviews revealed that the duration to reach OFP varied widely by occupation. It was difficult for stakeholders to provide specific details regarding duration, because the databases do not provide accurate readings, or (in the case of some TEs) access to databases for tracking member data is unavailable leading to the frequent use of spreadsheets and emails to organize, track and analyze data. There is presently no centralized IT system for BTL managers or TEs to use. In a survey of TE Commandants, the large majority (81 percent) of respondents agreed or strongly agreed that time on BTL could be significantly reduced by improving the electronic tracking of BTL members, and 65 percent agreed or strongly agreed that consolidating all BTL tracking into one IT system could also significantly reduce time on BTL.

As of the writing of this report, CMP is in the process of developing an IT-enabled BTLMS “to provide CAF-wide visibility, tracking and status of all BTL personnel with a view to optimizing the path to OFP.”^{Footnote 16} Such a system would also allow for analytic capabilities. However, no deadline has been set for the implementation of this new system.

This system would support an analysis of actual time to OFP but would not allow for an assessment of the appropriateness of this time in the absence of established optimal BTL durations by occupation. At present, optimal timelines have not been set for each occupation, but work is underway in this area.^{Footnote 17}

Optimal Path to OFP

While it is impossible to determine the optimal path, or accurately assess the duration of time it takes to complete BTL, it is clear that there are inefficiencies present, which impact the ability of members on BTL to reach OFP within appropriate timelines as evidenced through document review, and interview and survey data. Factors affecting efficient time on BTL include information management, communication, capacity, demographic barriers and individual members’ life events.

Key Finding 4: Multiple factors affect the efficiency and the duration that members spend on BTL, including inconsistent information management, communication, capacity, language and individual members’ life events.

Information Management and Communication

Information management and communication have been identified as key factors in generating inefficiencies within BTL management. In a survey of TE commandants, the majority of respondents (81 percent) agreed or strongly agreed that improving communications both within and across L1s could reduce time on BTL. The majority (77 percent) also agreed or strongly agreed that coordinating training schedules across L1s could reduce time on BTL.

Based on the documents reviewed and a significant number of stakeholder interviews, a number of recurring issues were identified with respect to information management and communication, which included:^{Footnote 18}

The lack of common operating procedures and standard electronic notifications between TEs and BTL staff;
The lack of formal file management procedures, such as reliance on emails and personal drives, a lack of a backup filing system and inconsistent handling of Protected B documents;
The lack of business analytics in order to track student performance; and
The wait time for security clearances to be completed, which can take from 3-12 months (several respondents mentioned that the issue could be improved by submitting the clearance requests while a member is still on BMQ/BMOQ, rather than afterwards).

Capacity

Regarding capacity, the large majority of TE commandant survey respondents (92 percent) agreed or strongly agreed that time on BTL could be significantly reduced by increasing TE capacity. The majority (65 percent) also agreed or strongly agreed that their TE’s capacity will be insufficient in order to accommodate the planned growth in recruitment over the coming years. A widespread issue, according to stakeholder interviews, is that there are capacity issues in terms of basic training operational support. For example, interviewees mentioned the shortage of trainers, the use of incremental or as-needed TE staffing and, in some cases, a lack of access to equipment (e.g., ships and aircraft), which prevents the schools from consistently offering courses.

Feedback from the evaluation survey, as well as key stakeholder interviews indicated that there are capacity issues in terms of basic training operational support. The shortage of trainers, as well as challenges in retention, were viewed as placing substantial strain on the training capacity of the TEs. Similarly, in some cases, interviewees mentioned that a lack of equipment also limited the TEs’ ability to sufficiently offer training opportunities and on-the-job learning.

Language

An analysis of NCM data for this evaluation showed that language was a factor across the CAF in time to OFP rank when controlling for occupation, entry type and demographic factors.^{Footnote 19} Francophones took longer than Anglophones to reach OFP rank. According to many interviewees in multiple environments, Francophones are often at a disadvantage when on BTL as they must sometimes wait for French courses which can have less availability, or they are required to attend language training prior to enrolling in courses only offered in English.

GBA+ Considerations

Gender-based factors in BTL have not previously been examined, due to a lack of available data. An informal data analysis of gender differences in time to OFP rank was undertaken as part of this evaluation, but no conclusive findings could be drawn.^{Footnote 20} Further research, using the GBA+ tool, would be useful and aligned with SSE.

Life Events

Key stakeholders consistently noted that other factors affecting the efficiency and time that members spend on BTL include a member’s life events, such as injuries, course failures, occupational transfers and family situations. If members are unable to join in time to align with the course cycle, which typically commences in September, they could be waiting on BTL for the next available course start. However, these life events are not being formally coded or tracked.

ADM(RS) Recommendation

Personnel Awaiting Training

Key Finding 5: There is an inconsistent approach to the employment and utilization of personnel awaiting training (PAT).

Throughout the BTL process, officers and NCM may obtain the status of PAT. This status is designated to personnel who are enrolled in BTL but who are not presently registered for training. Given the limited availability of certain training components, trainees can sometimes spend months as PAT, waiting to complete the next phase of their training.

There appears to be some disagreement about the extent to which PAT time is longer than it needs to be. In a survey of TE commandants, 35 percent agreed or strongly agreed, while 46 percent disagreed or strongly disagreed, that wait times between BMQ/BMOQ and the start of the next training are longer than they need to be in their occupations.

Nonetheless, a common response from stakeholders is that spending too much time as a PAT can have negative effects. All respondents unanimously agreed that when members spend too much time awaiting training it negatively impacts their morale, and it hurts the CAF’s operational readiness. Respondents were also nearly unanimous (96 percent) in agreeing that when members spend too much time awaiting training they are more likely to leave the CAF.

It is not known how many PAT there are across the CAF, or how long they spend on PAT on average, because a dedicated database to track PAT status is currently not in place. In 2017, the Army estimated it had 875 PAT, about 39 percent of its total BTL complement.^{Footnote 21} Anecdotal evidence from document review analysis identified the average wait time for PAT for certain occupations, as shown in the following table.^{Footnote 22}

Table 3. Average time on PAT. This table outlines the average days on PAT for selected occupations.
Environment	Occupation	Average time on PAT (days)
RCN	Boatswain	30
CA	Infantrymen	83
RCAF	Aircraft Structures Technician	181
CMP	Cook	122
CMP	Medical Technician	219
CMP	Military Police	120
CMP	Intelligence Officer	348-372

Table 3 Details - Average time on PAT (days), selected occupations

Table 3 Summary:

Table 3 presents the average number of days on PAT for selected occupations. It has three columns and eight rows, including titles. The left-hand column lists the environment the occupation is found in, the middle column lists the occupation, and the right-hand column provides the number of days on PAT.

Within the Royal Canadian Navy, the occupation of Boatswain has an average PAT time of 30 days.

Within the Canadian Army, the occupation of Infantryman has an average PAT time of 83 days.

Within the Royal Canadian Air Force, the occupation of Aircraft Structures Technician has an average PAT time of 181 days.

The table provides PAT times for four CMP occupations: Cook, Medical Technician, Military Police, and Intelligence Officer.

The occupation of Cook has an average PAT time of 122 days.

The occupation of Medical Technician has an average PAT time of 219 days.

The occupation of Military Police has an average PAT time of 120 days.

The occupation of Intelligence Officer has an average PAT time of 348 to 372 days.

The Navy estimated an average PAT time of 3 months for its members.^{Footnote 23} Furthermore, as of September 2017, 1,608 NCM personnel had been on BTL for over 12 months (577 who are identified as PAT).^{Footnote 24} This represents almost one third (32 percent) of all NCM BTL members at this time.^{Footnote 25}

Many interviewees revealed that PAT morale was dependent on the TE’s approach to managing their time during these periods, rather than the duration of time spent as PAT. Personnel who were provided with meaningful work and the opportunity to participate in activities exhibited higher morale than those who were not permitted or able to work or participate in activities. For example, in some cases PAT were not allowed to play on sports teams for fear of injury. Some PAT reported very limited availability of computers and/or internet, limiting their ability to take online courses while waiting.

Existing Oversight

As per a Combat Training Centre (CTC) Directive to all Army schools, “PAT must feel valued and given every chance to succeed on their training. COs/Cmdts must ensure that PAT are properly employed and supervised.”^{Footnote 26}

In terms of constructively using wait times between trainings, members are, in some cases, provided with informal opportunities to enhance their experiences. Document reviews and feedback from interviews highlight multiple opportunities, including unofficial employment in field training (e.g., weapons, fitness, tactics or acting as enemy force on training exercises), official languages, first aid, navigational and compass usage, and other opportunities.^{Footnote 27}

Operation GENERATION Directive noted that performance measures to track PAT time should be developed in order to minimize unproductive time awaiting training. It includes objectives to maximize the efficiency of basic military and occupational training and to minimize unproductive time for trainees. The aim is to develop and implement performance measures and tracking processes to identify productive (training) and unproductive (awaiting training) times for each occupation.

ADM(RS) Recommendation

Back to Table of Contents

2.4 Design and Governance

This section discusses findings related to the design and governance structure of BTL. Indicators regarding roles and responsibilities, governance, delegation of authorities, policy and guidance, and recommendations from previous evaluations were used to inform this section.

Need for Enterprise-wide Stewardship

Key Finding 6: Authorities, Responsibilities and Accountabilities (ARA) are not well established, leading to inconsistencies in how BTL is managed.

TEs tend to work in silos, and there is a lack of communication between them and with BTL management, which causes challenges in coordinating and tracking personnel data. In addition, the governance structure is not clearly established, which further contributes to inconsistencies.

The majority of school commandants that responded to the survey (62 percent) disagreed or strongly disagreed that the roles and responsibilities across the CAF for BTL Management are clear.

The current delegation of authorities is divided as follows:

Silo approach: CMP delegates down to MILPERSGEN, who then distributes responsibility to the TAs (RCAF, CA, and RCN). The TAs then distribute responsibilities to each of the DTA. While accountabilities are delegated down, progress reporting and issue management are not formally shared up the chain of command within each TA/DTA, or aggregated up to CMP at the CAF-wide level.
Working Groups: A working group under CMP has been established at the Capt (N)/Col and Cdr/LCol level to support Military Personnel Command in addressing challenges within the BTL process as they arise. The overall objective of this working group is to improve members’ satisfaction; minimize moves; reduce the time required to achieve OFP; and enhance information management and sharing practices.
Operation GENERATION Directive: The directive acknowledges that the current method of managing BTL and individual training is “poor or not synchronized,” and has “no enabled entity to oversee and coordinate the throughput of individuals from the point of recruitment, through basic training and environmental training to OFP.” The directive further notes that “environmental throughput equally fails to meet the OFP requirement, due in part to a lack of centralized coordination and synchronization.” It seeks to resolve these inefficiencies through the employment of a standardized IT solution, oversight of a high-level working group, and to implement common standardized operating procedures.

While the scope of this evaluation covers up to FY 2017/18, it is noted that progress has since been made in terms of establishing a responsibility matrix and management oversight systems.

ADM(RS) Recommendation

BTL Management Capacity

Key Finding 7: The existing capacity of BTL managers was found to be insufficient in order to ensure effective processing of members from basic training to OFP.

DND/CAF defence policy has outlined objectives aimed at increasing the effectiveness and efficiency of processing members’ throughput from intake to OFP. It is unclear, however, whether the existing capacity is sufficient to address both the existing concerns and these upcoming objectives. The Operation GENERATION Directive has committed BTL management to the following goals^{Footnote 28} regarding its support of CAF member production:

Reducing Prior Learning Assessment and Recognition (PLAR) processing time by 25 percent;
Increasing CFLRS capacity;
Modularizing Developmental Period level 1 training;
Increasing numbers of skilled/semi-skilled enrollees;
Improving University Liaison Office Programme;
Optimizing of path to OFP;
Drafting a BTL administration directive; and
Developing a BTLMS IT solution work plan.

The interviewees consistently pointed out that resourcing was also causing pressures on BTL management with respect to updating personnel files, as many staff are currently using spreadsheets and/or databases with limited analytics capabilities to process and store BTL member data.^{Footnote 29}

Issues related to resourcing have been acknowledged by BTL management through its Tactical Level Working Group meetings, which took place from April-May 2018.^{Footnote 30} An annex of the working group findings noted that turnover is high; that there are not enough staff to do the work; and that current staff are left to compensate for this gap by working overtime. The result of this has the potential to lead to increased employee burnout, turnover, and an increased backlog of work related to ensuring the effective throughput of BTL members from intake to attainment of OFP.

Without an adequate number of staff, the management of BTL throughput from intake to OFP will continue to perform at a less than optimal level. Given that the objectives outlined in Operation GENERATION Directive are tied to increased effectiveness and efficiency, increased capacity with respect to staff would help to address not only the future objectives but the current capacity-related gaps identified through this evaluation.

ADM(RS) Recommendation

Management Action

Action 1.1 - Directive Op GEN has identified significant gaps in BTL policy, and Commander Military Personnel Generation Group (MPGG) has been tasked to produce updated policy.

Deliverable: A wide-ranging policy and guidance review is currently underway, led by Director Military Personnel Generation Planning and Coordination, in collaboration with key stakeholders including TAs/DTAs. The policy will provide detailed ARA for BTL management and include direction on reporting and tracking. It will also include direction on managing BTL members including communicating the pathway to OFP and employment of PAT.

OPI: CMP-DMPGPC/MPGG
OCI: TAs/DTAs
Target Date: December 2020

Action 1.2 - Directive Op GEN has identified the requirement to establish optimal paths to OFP by occupation.

Deliverable: On behalf of Commander MPGG, Director Personnel Generation Requirements is responsible for conducting Military Employment Structure studies on occupations to ensure that the tasks, knowledge, skills and abilities are clearly delineated by job/rank, which in turn serve as the foundation for occupation training requirements amongst other Mil-HR applications. DAOD on the Military Employment Structure and other related policy documents will be reviewed and updated to ensure that the criteria and process for establishing optimal occupational training paths to OFP are included in the occupational studies. Policy review complete by July 2020.

OPI: CMP-DMPGPC/MPGG
Target Date: July 2020

Action 1.3 - Once defined, these criteria and instructions will be consistently applied as Standard Operating Procedures (SOP) for any occupational studies or reviews being conducted. The “optimal paths” will be published in flow/process maps in the Job Based Occupational Specifications for each occupation. Implementation of “Optimal Path to Operationally Functional Point (OFP)” criteria and SOPs as part of the upcoming occupation analysis studies and life cycle management to be completed by September 2020.

OPI: CMP-DMPGPC/MPGG
Target Date: September 2020

Key Finding 3: The optimal duration to reach Operationally Functional Point (OFP) is not well-defined nor consistently tracked.

ADM(RS) Recommendation

Management Action

Action 2.1 - Op GEN has also identified the requirement for an IT-enabled BTLMS solution with the intent to manage critical aspects of BTL administration and track members’ data.

Deliverable: MPGG is currently in the initial stages of developing this tool led by MPGG’s Enhanced Performance Innovation Centre team. Requirements definition is currently well underway led by Director Military Personnel Generation Planning and Coordination (DMPGPC). Key stakeholders including TAs/DTAs are engaged. A target date has been set, but this will be contingent on resourcing, contracting and the capacity of the developer. Given alignment requirement across DND, this will take significant time to achieve for synchronization within DND/CAF.

OPI: DMP-DMPGPC/MPGG
OCI: TAs/DTAs; ADM(IM); ADM(DIA)
Target Date: March 2021

Key Finding 7: The existing capacity of BTL managers was found to be insufficient in order to ensure effective processing of members from basic training to OFP.

ADM(RS) Recommendation

Management Action

Action 3.1 - To inform senior management decision making, MPGG will coordinate an assessment of TE/Developmental Period training capacity to meet assigned throughput requirements. Each TA/DTA will be responsible for conducting their own internal evaluation. DMPGPC will synthesize and produce the final report.

OPI: CMP-DMPGPC/MPGG
OCI: TAs/DTAs
Target Date: December 2020

Action 3.2 - To inform senior management decision making, MPGG will coordinate an assessment of BTL manager capacity to support and sustain effective operational oversight. The report will identify key requirements, performance gaps and recommendations. Each TA/DTA will be responsible for conducting their own internal evaluation. DMPGPC will synthesize and produce the final report.

OPI: CMP-DMPGPC/MPGG
OCI: TAs/DTAs
Target Date: December 2020

Back to Table of Contents

Annex B—Evaluation Methodology and Limitations

1.0 Methodology

1.1 Overview of Data Collection Methods

The evaluation used multiple lines of evidence and complementary qualitative and quantitative research methods to help ensure the reliability of information and data supporting findings. The evaluation assessed both BTL Management CAF-wide, and examined in greater depth BTL Management for a sample of occupations, adopting a multiple case study approach.

1.2 Details on Data Collection Methods

The following data collection methods were used to gather qualitative and quantitative data in the evaluation:

Document review (n=68)
Survey of TE Comdts (n=26)
Key Informant Interviews
- BTL managers and other TA reps (n=52)
- PAT (n=208)
Analysis of HR data
- Career progression of members recruited in 2013 (n=1,997)
- Historical PARRA data
Analysis of financial data

1.2.1 Document Review

A comprehensive document review was conducted as part of the conduct phase of the evaluation, focusing on the relevance and performance of BTL Management activities.

The following documents were reviewed during the scoping and conduct phases of the evaluation:

Corporate documents, such as Departmental Plans and SSE;
Policy documents, such as DAOD and CF Military Personnel Instructions;
Program-level documents, including the BTAG, findings from the study on PAT; and materials from the BTL Working Group.

1.2.2 Surveys

A survey of commandants of CAF TEs was undertaken, seeking their input on the effectiveness and efficiency of BTL management, both in the occupations of their TEs and of the CAF overall. The survey was administered online between June 11, 2019 and July 5, 2019. Invitations were sent via email.

Out of 32 individuals targeted by the survey, 26 responses were received, for an overall response rate of 81 percent. Response was highest in CMP (93 percent) and lowest in the Air Force (50 percent).

Data were analyzed primarily at group level. Responses for the Army and CMP were analyzed for differences against other respondents using Chi Square tests (α = 0.05). Due to low sample sizes, breakdowns were not assessed for other environments.

1.2.3 Key Informant Interviews

In-depth key informant interviews with primary and secondary stakeholders served as an important source of qualitative information and provided informed opinions and observations on select evaluation questions, as well as assisted in the interpretation and understanding of qualitative and quantitative data from other lines of enquiry.

Two hundred and sixty interviews were conducted with the following categories of interview respondents:

Table B-1. Key Informant Interviews. This table outlines the categories of interview respondents.
Organization	Units	# of Interviewees
Organization	Units	TA, BTL Managers	PAT
CMP	MPG, CFHS, CFRG, CFSTG HQ, Comd CFB Borden, MP	16	12
RCN	NPTG, NOTC, NPT, Coastal MOC Advisors	22	70
CA	CTC	10	115
RCAF	CFSATE, 2 Cdn Air Div BTL Coordinator	4	11
Total		52	208

Table B-1 Details - Categories of Interview Respondents

Table B-1 Summary:

This table presents the number of interviewees by organization and unit. It has four columns and six rows, including titles. The left-hand column lists the organization, the second column lists the units, and the third and fourth columns lists the number of interviewees. The third column lists the TA and BTL Manager interviewees and the fourth column lists the PAT interviewees.

Within CMP, representatives from the following units were interviewed: MPG, CFHS, CFRG, CFSTG HQ, Comd CFB Borden, and MP. Sixteen TA and BTL Managers and 12 PAT were interviewed.

Within RCN, representatives from the following units were interviewed: NPTG, NOTC, NPT, and Coastal MOC Advisors. Twenty-twp TA and BTL Managers and 70 PAT were interviewed.

Within CA, representatives from CTC were interviewed. Ten TA and BTL Managers and 115 PAT were interviewed.

Within RCAF, representatives from the following units were interviewed: CFSATE and 2 Cdn Air Div BTL Coordinator. Four TA and BTL Managers and 11 PAT were interviewed.

In total, 52 TA and BTL Managers and 208 PAT were interviewed.

1.2.4 Review of Human Resources Data

Data was extracted from HRMS, the legacy HR system of the CAF, of Regular Force members recruited in 2013, to assess time spent on BTL and the percentage of members releasing while still on BTL. Other HR and program data from non-HRMS sources were not reviewed.

Out of a population of 3,513, the analysis included only those in the 46 occupations for which OFP data was available (from the Occupation Specifications). Excluding errors, unexpected values, and those who changed occupations prior to OFP, resulted in a sample size of 1,997. This sample was compared to data from the PARRA report and found to be representative by occupation.

Due to data quality issues, it was not possible to assess time on BTL, so time to OFP rank was assessed instead. Therefore, this analysis could not be used to estimate time on BTL but could help to indicate sub-groups that may spend longer than others to reach effective strength.

1.2.5 Review of Financial Data

Although no expenditure data was available for BTL management since it is not a program, some financial data were received, mostly on the cost of PAT or the cost of time on BTL. Although limited in scope, these were analyzed to the extent possible.

2.0 Limitations

The anticipated limitations associated with this evaluation, and the mitigation strategies that were applied to address them are:

Table B-2. Evaluation Limitations and Mitigation Strategies. This table lists the limitations of the evaluation and the corresponding mitigation strategies.
Limitation	Mitigation Strategy
Not a true program – lack of clear roles, responsibilities and no IT tools shared across the TAs/DTAs	Engage stakeholders across the CAF
Stakeholders dispersed across the CAF	Ensure representation from all relevant L1s

Table B-2 Details - Evaluation Limitations and Mitigation Strategies

Figure C-1 Details - Logic Model

Figure C-1 Summary:

The flowchart represents the logic model for BTL Management.

On the left-hand side of the graphic, there are six boxes, one on top of the other, showing six stages as follows, from top to bottom: Ultimate Outcome, Intermediate Outcome, Immediate Outcomes, Outputs, Activities and Inputs.

To the right of these boxes, at the level of Ultimate Outcome, the top box states: Canadian Armed Forces is growing towards its end state of 101,500 personnel.

There is one box underneath the above box, at the level of Intermediate Outcome, which states: Defence Team Members have the full range of competencies required for their rank or level and are developing the required competencies for the next rank or level.

There is one box underneath the above box, at the level of Immediate Outcomes, which states: CAF members reach Operational Functional Point (OFP) in correct numbers, on time and with the proper qualifications.

There are four boxes, side by side, underneath the above box, at the level of Outputs.

The first box states:

Annual Military Occupation Review
Strategic Intake Plan

The second box states:

Canadian Forces
Basic Military Qualification/ Basic Military Officer Qualification
Subsidized University Training List (SUTL)
Occupational Training

The third box states:

PCN (Personnel Classification Number)
Posting Messages
Landing Messages
Career Review Board (CRB)

The fourth box states:

Central Command of BTL
Decentralized execution of BTL

There is one box underneath each of the four boxes above, at the level of Activities.

The first box states:

Planning
Director Personnel Generation Requirements develops, maintains and implements SIP
Chief of Military Personnel conducts a systematic review of an occupation that provides a comprehensive assessment of personnel management issues and trends (AMOR)

The second boxes states:

Training
Canadian Forces
Develop courses
Course validation questionnaires
Occupational Support
Prior Learning Assessment and Recognition (PLAR)
Develop courses
Course validation questionnaires

The third box states:

Career Management Administration
Personnel Awaiting Training (PAT) Management
Gap Management
Releases
Sequence Career Training Events

The fourth box states:

Governance
Military Personnel Generation commands BMQ and academic programming stages for recruits (initial stage)
Training Authorities and Designated Training Authorities manage “On Job” and “Environmental” training,

There are three boxes, side by side, underneath the above boxes, which represent Inputs.

The first box states:

Policy Documents: Canada’s defence plan, Strong, Secure, Engaged/Defence Requirements Review; CANFORGEN 257/10, 031/97

The second box states:

Recruits

The third box states:

Resources: MILPERSGEN; CMP; Recruiting Centres; CAF Bases

Back to Table of Contents

Annex D—Evaluation Matrix

Table D-1. Evaluation Matrix. This table indicates the data collection methods used to assess the evaluation issues/questions for determining the overall effectiveness of Basic Training List management.
Relevance
Evaluation Issues/Questions	Indicators	Document Review	Interviews	Survey	Program Data
1.1 Is BTL Management aligned with departmental priorities?	1.1.1 Extent to which policy and directives around BTL Management reflect the new priorities of DND/CAF	YES	YES	-	-
Effectiveness
Evaluation Issues/Questions	Indicators	Document Review	Interviews	Survey	Program Data
2.1 To what extent does BTL management meet the CAF’s needs for appropriately trained members at OFP in each occupation?	2.1.1 Number and proportion of personnel that complete BMQ/BMOQ, by occupation and gender	YES	-	-	YES
	2.1.2 Number and proportion completed training to OFP, by occupation, gender and minority group	-	-	-	YES
	2.1.3 Gap in PML to TES, by occupation	-	-	-	YES
	2.1.4 Extent to which the number of members reaching OFP is considered sufficient to meet CAF needs	-	YES	YES	-
	2.1.5 Perceptions of extent to which BTL members reaching OFP are fully capable of meeting the expectations of their occupation (quality of members)	-	YES	YES	-
	2.1.6 Extent to which mechanisms are in place to ensure quality and effectiveness of training across the CAF	YES	YES	YES	-
	2.1.7 Explanations of reasons for gaps in numbers of members reaching OFP and in quality of BTL training	YES	YES	YES	-
Efficiency
Evaluation Issues/Questions	Indicators	Document Review	Interviews	Survey	Program Data
3.1 How quickly are BTL members reaching OFP?	3.1.1 Duration on BTL for sample occupations, controlling for occupation specifications, size of annual intake and size of PML.	YES	-	-	YES
4.1 What are the primary factors affecting BTL efficiency?	4.1.1 Extent to which the following affects duration on BTL for sample occupations: Intake type; Prior experience and qualifications; Gender; Attrition; Occupational transfers / component transfers	YES	YES	YES	YES
4.1 What are the primary factors affecting BTL efficiency?	4.1.2 Perceptions of extent to which the following affects duration on BTL for all occupations: TE capacity; PAT management; Communication issues; Technical issues; Policy; Training requirements; and other issues	YES	YES	YES	YES
5.1 To what extent is existing CAF training capacity adequate to meet future needs?	5.1.1 Capacity (number of personnel, hours to deliver training, available vs required)	YES	YES	-	YES
	5.1.2 Projected increase in number of personnel on BTL (based on SSE recruitment targets, average time on BTL)	YES	YES	-	-
	5.1.3 Projected increased capacity needed to accommodate increased number of personnel on BTL	YES	YES	-	-
6.1 To what extent can the BTL pathway accommodate individual differences in recruits?	6.1.1 Availability of alternative BTL pathways for trained recruits, for sample occupations, relative to equivalent civilian careers	YES	YES	YES	-
7.1 What effect, if any, does time spent awaiting training have on individual members and on the CAF’s production?	7.1.1 Wait time, by occupation, between BMQ/BMOQ and start of next training	YES	YES	-	YES
	7.1.2 Number of PAT by occupation	YES	YES	-	-
	7.1.3 Perceptions of impacts of time spent awaiting training on individuals and on the CAF	-	YES	YES	-
8.1 What return on investment can be gained from greater efficiencies in the BTL process?	8.1.1 Salary cost of increased time on BTL due to issues identified	-	-	-	YES
	8.1.2 Recruitment cost due to attrition arising from issues identified in the BTL process	-	-	-	YES
	8.1.3 Total training expenditures divided by number of graduates	-	-	-	YES
	8.1.4 Estimated return on investment of other efficiencies	-	-	-	YES
Design and Governance
Evaluation Issues/Questions	Indicators	Document Review	Interviews	Survey	Program Data
9.1 Is the BTL management governance structure appropriate and efficient?	9.1.1 Clarity of roles and responsibilities for program delivery	YES	YES	YES	-
	9.1.2 Perceptions of barriers to more effective governance	YES	YES	YES	-
	9.2.3 Estimate of benefits and disadvantages of current delegation of authorities across L1s vs alternative arrangements	YES	YES	YES	-
10.1 Has BTL Management been implemented as planned?	10.1.1 Consistency of policy and guidance with practice	YES	YES	YES	-
10.1 Has BTL Management been implemented as planned?	10.1.2 Extent to which recommendations from previous reviews were acted upon	YES	-	-	-

Table D-1 Details - Evaluation Matrix

Table D-1 Summary:

This table indicates the data collection methods used to assess the evaluation questions and indicators for assessing the relevance, effectiveness, efficiency, and design and governance of BTL Management.

The chart has six columns. The first column on the left lists the evaluation questions, and the second column lists the indicators for each question. The remaining four columns represent one of the four data collection methods for the evaluation: document review, interviews, survey and program data.

There are 35 rows which are separated into four sections: Relevance , Effectiveness, Efficiency, and Design and Governance.

There is one Relevance question, which is as follows: Is BTL Management aligned with departmental priorities? This question has one indicator, as follows: Extent to which policy and directives around BTL Management reflect the new priorities of DND/CAF. This indicator was assessed through the following collection methods: document review, interviews.

There is one Effectiveness question, as follows: To what extent does BTL management meet the CAF’s needs for appropriately trained members at OFP in each occupation? This question has seven indicators.

The first indicator for this question is as follows: # and % of personnel that complete BMQ/BMOQ, by occupation and gender. This indicator was assessed through the following collection methods: document review, program data.

The next indicator for this question is as follows: # and % completed training to OFP, by occupation & gender and minority group. This indicator was assessed through the following collection methods: document review, program data.

The next indicator for this question is as follows: Gap in PML to TES, by occupation. This indicator was assessed through the following collection methods: document review, program data.

The next indicator for this question is as follows: Extent to which the number of members reaching OFP is considered sufficient to meet CAF needs. This indicator was assessed through the following collection methods: interviews, survey.

The next indicator for this question is as follows: Perceptions of extent to which BTL members reaching OFP are fully capable of meeting the expectations of their occupation (quality of members). This indicator was assessed through the following collection methods: document review, interviews, survey.

The next indicator for this question is as follows: Extent to which mechanisms are in place to ensure quality and effectiveness of training across the CAF. This indicator was assessed through the following collection methods: document review, interviews, survey.

The next indicator for this question is as follows: Explanations of reasons for gaps in numbers of members reaching OFP and in quality of BTL training. This indicator was assessed through the following collection methods: document review, interviews, survey.

There are six Efficiency questions. The first question is as follows: How quickly are BTL members reaching OFP? This question has one indicator, as follows: Duration on BTL for sample occupations, controlling for occupation specifications, size of annual intake and size of PML. This indicator was assessed through the following collection methods: document review, interviews, program data.

The next question in Efficiency is as follows: What are the primary factors affecting BTL efficiency? This question has two indicators. The first indicator is as follows: Extent to which the following affects duration on BTL for sample occupations: Intake type; Prior experience and qualifications; Gender; Attrition; Occupational transfers/component transfers. This indicator was assessed through the following collection methods: document review, interviews, survey, program data.

The next indicator for this question is as follows: Perceptions of extent to which the following affects duration on BTL for all occupations: TE capacity; PAT management; Communication issues; Technical issues; Policy; Training requirements; Other issues. This indicator was assessed through the following collection methods: document review, interviews, survey.

The next question in Efficiency is as follows: To what extent is existing CAF training capacity adequate to meet future needs? This question has three indicators. The first indicator is as follows: Capacity (number of personnel, hours to deliver training, available vs required). This indicator was assessed through the following collection methods: interviews, survey.

The next indicator for this question is as follows: Projected increase in number of personnel on BTL (based on SSE recruitment targets, average time on BTL). This indicator was assessed through the following collection methods: document review, survey, program data.

The next indicator for this question is as follows: Projected increased capacity needed to accommodate increased number of personnel on BTL. This indicator was assessed through the following collection methods: document review, interviews, survey, program data.

The next question in Efficiency is as follows: To what extent can the BTL pathway accommodate individual differences in recruits? This question has one indicator, as follows: Availability of alternative BTL pathways for trained recruits, for sample occupations, relative to equivalent civilian careers. This indicator was assessed through the following collection methods: document review, interviews, survey.

The next question in Efficiency is as follows: What effect, if any, does time spent awaiting training have on individual members and on the CAF’s production? This question has three indicators. The first indicator is as follows: Wait time, by occupation, between BMQ/BMOQ and start of next training. This indicator was assessed through the following collection methods: document review, interviews, survey, program data.

The next indicator for this question is as follows: Number of PATs by occupation. This indicator was assessed through the following collection methods: document review, program data.

The next indicator for this question is as follows: Perceptions of impacts of time spent awaiting training on individuals and on the CAF. This indicator was assessed through the following collection methods: document review, interviews, survey, program data.

The next question in Efficiency is as follows: What return on investment can be gained from greater efficiencies in the BTL process? This question has four indicators. The first indicator is as follows: Salary cost of increased time on BTL due to issues identified. This indicator was assessed through the following collection methods: program data.

The next indicator for this question is as follows: Recruitment cost due to attrition arising from issues identified in the BTL process. This indicator was assessed through the following collection methods: document review, program data.

The next indicator for this question is as follows: Total training expenditures divided by number of graduates. This indicator was assessed through the following collection methods: program data.

The next indicator for this question is as follows: Estimated return on investment of other efficiencies. This indicator was assessed through the following collection methods: document review, interviews, program data.

There are two Design and Governance questions. The first question is as follows: Is the BTL management governance structure appropriate and efficient? This question has three indicators. The first indicator is as follows: Clarity of roles and responsibilities for program delivery. This indicator was assessed through the following collection methods: document review, interviews, survey.

The next indicator for this question is as follows: Perceptions of barriers to more effective governance. This indicator was assessed through the following collection methods: document review, interviews.

The next indicator for this question is as follows: Estimate of benefits and disadvantages of current delegation of authorities across L1s vs alternative arrangements. This indicator was assessed through the following collection methods: document review, interviews.

The next question in Design and Governance is as follows: Has BTL Management been implemented as planned? This question has two indicators. The first indicator is as follows: Consistency of policy and guidance with practice. This indicator was assessed through the following collection methods: document review, interviews.

The next indicator for this question is as follows: Extent to which recommendations from previous reviews have been acted upon. This indicator was assessed through the following collection methods: document review.

Footnotes

Footnote 1

Data from Production Attrition Recruiting Retention Analysis (PARRA) Report as of July 13, 2018.

Return to footnote1 referrer

Footnote 2

See http://publications.gc.ca/site/eng/9.835971/publication.html.

Return to footnote2 referrer

Footnote 3

Financial Analysis Summary.

Return to footnote3 referrer

Footnote 4

See https://www.canada.ca/content/dam/dnd-mdn/migration/assets/FORCES_Internet/docs/en/about-reports-pubs/dnd_2018-19_departmental-plan.pdf.

Return to footnote4 referrer

Footnote 5

Specifically, programs 2.2 (Ready Naval Forces), 2.3 (Ready Land Forces), 2.4 (Ready Air and Space Forces), 2.5 (Ready Special Operations Forces), 2.6 (Ready Cyber Forces), 2.8 (Ready Joint and Combined Forces), and 2.9 (Ready Health Services and Military Police Forces).

Return to footnote5 referrer

Footnote 6

Treasury Board Secretariat. Directive on Results, July 1, 2016. https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=31306.

Return to footnote6 referrer

Footnote 7

DAOD 5031-8, Canadian Forces Professional Development. DND, 2012. https://www.canada.ca/en/department-national-defence/corporate/policies-standards/defence-administrative-orders-directives/5000-series/5031/5031-8-canadian-forces-professional-development.html.

Return to footnote7 referrer

Footnote 8

Compensation and Benefits Instructions, Chapter 208 - Relocation Benefits, s 82. DND, 2018.

Return to footnote8 referrer

Footnote 9

Historical PARRA Report, DND, 2018. Although TEE would represent a better target than PML, as the latter includes approved future positions, TEE data by occupation are not available in this data source.

Return to footnote9 referrer

Footnote 10

Canadian Forces Recruiting Group, FY 18/19 Year End Brief. DND, 2019.

Return to footnote10 referrer

Footnote 11

Historical PARRA report, DND, 2018.

Return to footnote11 referrer

Footnote 12

Internal DND/CAF records, 2018.

Return to footnote12 referrer

Footnote 13

Canadian data from internal DND/CAF records, 2018. Data for other countries from Recruit Training: Assessing the Quality of Recruit Training in the New Zealand Defence Force. New Zealand, 2017 (data from 2014).

Return to footnote13 referrer

Footnote 14

Cost to OFP (internal document), DND, 2018.

Return to footnote14 referrer

Footnote 15

BTL Path to OFP Working Group, Army and Air Force Updates, November 2017.

Return to footnote15 referrer

Footnote 16

Annual Op GEN Report, Annex A, DND/CAF, 2019.

Return to footnote16 referrer

Footnote 17

Ibid.

Return to footnote17 referrer

Footnote 18

See, for example, Executive Summary, Basic Training List Management Tactical Level Working Group (Annex A), May 23, 2018; Operation GENERATION, Production Working Group, May 2018.

Return to footnote18 referrer

Footnote 19

Linear regression of HR data from HRMS. See Annex B for methodology.

Return to footnote19 referrer

Footnote 20

Ibid.

Return to footnote20 referrer

Footnote 21

BTL Path to OFP Working Group, Army Update, November 2017.

Return to footnote21 referrer

Footnote 22

Internal records, regarding cost to OFP for various occupations, November 2018.

Return to footnote22 referrer

Footnote 23

RCN PAT and PAD Quad Slide, DND/CAF, 2019.

Return to footnote23 referrer

Footnote 24

BTL SOPG Mapping 20171005.

Return to footnote24 referrer

Footnote 25

PARRA data.

Return to footnote25 referrer

Footnote 26

Directive to all CTC Schools, guidance and procedures relating to the employment of PAT, September 23, 2014.

Return to footnote26 referrer

Footnote 27

See, for example, PAT Employment, communiqué to all CTC Schools, TCs, Units Employing PATs, September 23, 2014; BTL Management Working Group, Optimizing Path to OFP and Management of PAT Resources, November 1, 2018.

Return to footnote27 referrer

Footnote 28

20181211-UU-CMPC-OP_ORDER_OP_GENERATION_ANX_D_LOO3_PRODUCTION.

Return to footnote28 referrer

Footnote 29

20180523 Annex A - BTL Tactical Level WGs Findings.docx

Return to footnote29 referrer

Footnote 30

Ibid.

Return to footnote30 referrer

Page details

Date modified:: 2020-07-20

Language selection

Search

Evaluation of Basic Training List Management

Table of Contents

Alternate Formats

Assistant Deputy Minister (Review Services)

Overall Assessment

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

ADM(RS) Recommendation

Page details