Evaluation of the DND/CAF Airworthiness Programme

March 2016

1258-220 (ADM(RS))

Reviewed by ADM(RS) in accordance with the Access to Information Act. Information UNCLASSIFIED.

Acronyms and Abbreviations

Executive Summary

This report presents the findings and recommendations of the Evaluation of the Department of National Defence and the Canadian Armed Forces (DND/CAF) Airworthiness Programme (AWP). The evaluation was conducted by Assistant Deputy Minister (Review Services) (ADM(RS)) in response to a request from the Chief of the Air Force (C Air Force). In compliance to this request and to the Treasury Board Secretariat (TBS) Policy on Evaluation (2009), the evaluation examined the relevance, performance and efficiency of the approach used by the DND/CAF to assure airworthiness to CAF military aviation.

Program Description

The objective of the AWP is to provide an acceptable level of safety for military aviation.¹ This objective is derived directly from the Aeronautics Act. The AWP is based on the fundamental principles that airworthiness-related activities are completed to accepted standards, performed by authorized individuals, accomplished within accredited organizations and done using approved procedures. These principles are fundamental to the application of airworthiness and aviation safety programs worldwide.

As detailed in the Aeronautics Act, the Minister of National Defence (MND) is accountable to Parliament for military aviation safety in Canada. The AWP is managed by four authorities assigned through formal delegation from the MND to the Chief of the Defence Staff (CDS): the Airworthiness Authority (AA), the Technical Airworthiness Authority (TAA), the Operational Airworthiness Authority (OAA) and the Airworthiness Investigative Authority (AIA). The C Air Force is designated as the AA, responsible for the development, promotion, supervision and management of the DND/CAF AWP.

Relevance

The DND/CAF AWP is relevant and is aligned with federal government and departmental roles, responsibilities and priorities. The current AWP strongly supports the safety of existing and future military aeronautics in the CAF through compliance with appropriate airworthiness standards and regulations.

Effectiveness

The DND/CAF AWP contributes to the sustainment of Royal Canadian Air Force (RCAF) readiness within an acceptable level of safety. The evaluation determined that the AWP is effective based on a sound foundation of formal delegations and clear scopes of authority, thorough documentation, comprehensive airworthiness and risk management processes, annual fleet airworthiness review boards and periodic audits of operational and technical organizations. In the last five years, several new fleets were successfully and safely introduced into service, and significant programmatic improvements have been made in several areas, most notably in risk management, operational airworthiness (OA) documentation and technical auditing.

However, its effectiveness is dependent upon its ability to deliver adequate governance of the program and the operational, technical and investigative functions of airworthiness. To assess the effectiveness of each of these aspects, several indicators were examined; namely, scope of authority, organizational structure, processes, risk management, documentation, training, resources and the Performance Measurement Framework (PMF). Each of the governance, operational, technical, and investigative airworthiness aspects is discussed in terms of the indicators within the report and in greater detail in annexes A, B, C and D.

While the AWP is considered effective, the evaluation found several areas for improvement. A number of recommendations are made to improve its governance framework and each of the operational, technical and investigative airworthiness sub-programs. These recommendations are based on international best practices and seek to strengthen the AWP by proposing several measures to improve program monitoring and management, independent oversight, airworthiness risk management, oversight of DND/CAF and foreign military aviation operations, training of regulators and other associated personnel and documentation of AWP policies, regulations and processes.

Efficiency and Economy

The AWP was largely implemented by adding responsibilities to existing personnel and redirecting a limited number of engineering staff into regulatory positions. Because most personnel are double-hatted,² the total cost of the AWP is difficult to estimate. The implementation of performance measures would provide the ability to assess the efficiency and costs of the program and associated improvement initiatives. Other measures, proposed to improve the efficiency of the AWP, include establishing closer ties with Transport Canada (TC), recognizing allied aircraft certifications and increasing the use and the retention of civilian expertise within the regulatory and investigative staff.

Management Action Plan

The evaluation findings and recommendations were regularly discussed and presented to key stakeholders throughout the course of the evaluation, including the AA staff. While the AA plans to address all of the findings and recommendations within the report, a set of six key summary recommendations were agreed upon with the AA staff for the purposes of the Departmental Evaluation Committee presentation and ADM(RS) evaluation monitoring. These recommend measures to improve the independence and overall management of the AWP, to improve the oversight of aviation operations under the responsibility of DND, to improve training in support of the AWP, and to improve the investigative response to a major DND accident in an austere location. These recommendations, along with their corresponding management actions, are provided at Annex H.

1.0 Introduction

1.1 Context for the Evaluation

This report presents the findings and recommendations of the Evaluation of the DND/CAF Airworthiness Programme. ADM(RS) conducted the evaluation at the request of the C Air Force.³ The evaluation sought to evaluate the relevance and performance of the program, with a specific focus on the effectiveness, comprehensiveness and efficiency of the approach used to assure airworthiness to CAF military aviation. Recommendations resulting from the evaluation will be used to inform DND/CAF senior management of the health of the AWP, its key enablers and the areas which may require additional focus.

1.2 Program Profile

1.2.1 Program Description

The DND/CAF AWP encompasses the development, regulation, supervision and investigations of all matters related to military aeronautics. It includes the design, manufacturing, maintenance, material support, personnel, facilities and operations of aeronautical products. The Programme is based on the fundamental principle that airworthiness activities are completed to accepted standards, performed by authorized individuals, accomplished within accredited organizations and done using approved procedures. For example, the DND/CAF AWP ensures CAF military aviation operations are executed in accordance with DND/CAF published regulations and orders, and any deviations to approved orders are documented. Risks are formally identified, mitigated when applicable and adjudicated by competent and approved authorities. Air occurrences are thoroughly and independently investigated, and recommendations are made to regulators and implementers.

The activities of the program are separated into three distinct roles that are conducted by airworthiness regulators, implementers and investigators. The regulators develop the rules and standards and ensure compliance; the implementers conduct the aviation activities in compliance with the rules; and the investigators investigate airworthiness-related aviation safety occurrences or issues. The program is managed by four authorities assigned through formal delegation from the MND to the CDS; the AA, the TAA, the OAA, and the AIA. The C Air Force is designated as the AA, responsible for the development, promotion, supervision and management of the DND/CAF AWP. Reporting to the AA, the Director General Aerospace Equipment Program Management (DGAEPM) is normally designated as the TAA, the Commander (Comd) 1 Canadian Air Division (CAD) is designated as the OAA and the Director Flight Safety (DFS) is designated as the AIA. As illustrated in Figure 1, a structured governance framework compromised of the AA, TAA, OAA and AIA has been established to manage the programme and provide regulatory oversight.

Airworthiness for CAF aviation has evolved considerably over the last 15 years. In 1993, Chief Review Services (CRS)⁵ published a review of the DND/CAF Airworthiness Programme, including its policy, legislative framework and application,⁶ which led to fundamental changes. In 1998, the MND directed the implementation of a more robust airworthiness program to more effectively meet the Minister’s legal obligations under the Aeronautics Act.⁷ This transformation introduced the management framework and delegation of authorities that are in place today. In 2005, the MND further delegated to the CDS the power to make or revoke airworthiness designations and delegations.⁸ In 2008, at the C Air Force’s request, CRS performed a Review of the DND/CAF Airworthiness Risk Management Process,⁹ which led to recommendations that were formally monitored by the DND airworthiness authorities at annual airworthiness boards.

1.2.2 Program Objectives

The objective of the DND/CAF AWP is to provide an acceptable level of safety for all military aviation. The program must also allow operational commanders the flexibility to balance mission accomplishment by considering risk, context and urgency against aviation safety goals. A logic model that details the intended outcomes of the program is at Annex G and is used as the framework to assess the performance of the program.

1.2.3 Stakeholders

The AWP is a DND/CAF-wide program involving all personnel designing, maintaining, operating or otherwise supporting military air operations in Canada and abroad. The primary stakeholders for the AWP include, but are not limited to, the following:

• the MND;
• the CDS;
• the RCAF, as the principal regulator and implementer of the program;
• DGAEPM, as regulator and implementer for the technical aspects of the program;
• Director General Major Project Delivery (Air) (DGMPD (Air)), as implementer of major aircraft capital projects;
• all other DND operators of aircraft systems including the Canadian Army (CA) and Royal Canadian Navy (RCN);
• the Air Cadet League of Canada;¹⁰
• all foreign militaries operating aircraft within Canada;
• all civilian companies designing, maintaining or supporting DND aircraft; and
• all civilian aircraft operators supporting DND training, exercises or operations.

1.3 Evaluation Scope

1.3.1 Coverage and Responsibilities

This evaluation reviewed the roles and responsibilities of the regulators (AA, TAA and OAA) and the investigators (AIA), as well as the coordination between them. Key aspects of the evaluation, as per the request of the C Air Force, include an assessment of the methods used to assure airworthiness, the governance and management of the program, the resources allocated to the regulator and investigator functions and the capacity of the programme to adjust to future aeronautical activities.

The AWP falls under the following strategic outcome and programs within DND’s Program Alignment Architecture (PAA) (February 6, 2014):

• Strategic Outcome: Defence Remains Continually prepared to Deliver National Defence and Defence Services in Alignment with Canadian Interests and Values.
  • Program 3.0: Defence Ready Force Element Production
  • Sub-program 3.1: Force Element Readiness Sustainment
  • Sub-sub-program 3.1.3: Aerospace Roles – Readiness Sustainment
  • Program 4.0: Defence Capability Element Production
  • Sub-program 4.2: Material Life Cycle
  • Sub-sub-program 4.2.7: Material – Strategic Coordination, Development and Control

1.3.2 Resources

The AWP is essentially a regulatory framework spanning multiple operational and headquarters functions and activities within the RCAF and DND. As such, there is no specific AWP budget, and annual spending that could be attributed to the AWP is apportioned to operational, maintenance and capital project activities within those organizations’ budgets.

Similarly, while the AWP involves most RCAF and many other DND/CAF personnel in the implementation of its regulations, few personnel resources are actually allocated to the program itself. The AWP organizational structure was implemented by overlaying airworthiness responsibilities onto existing operational command and engineering management structures and reallocating a limited number of support staff to dedicated regulatory positions. The latter includes approximately 57 regulatory positions in the TAA organization and 4 regulatory positions in the OAA organization. In addition, the AIA has 15 investigator positions.

2.0 Findings and Recommendations

The following sections discuss the relevance and performance of the AWP. The evaluation examined the extent to which the program addresses a demonstrable need, is aligned with federal roles and responsibilities and government priorities, achieves its expected outcomes and demonstrates efficiency and economy in resource utilization.

2.1 Relevance—Continued Need

This section examines the continued need for the AWP. The findings in this section are based on evidence from document reviews.

Key Finding 1: In order to contribute to sustaining RCAF readiness with an acceptable level of safety, there is an on-going need for the DND/CAF to conduct its own AWP.

The evaluation used the following indicators to assess the extent to which the program continues to address a demonstrable need:

• evidence of a continued need for the regulation of aviation activities, facilities and services;
• perceived extent to which the AWP is needed in order to achieve the sustainment of RCAF readiness with an acceptable level of safety; and
• perceived extent to which the AWP is needed in order to effectively manage current and future aeronautical activities.

As detailed in the Aeronautics Act, the MND is accountable to Parliament for military aviation safety in Canada. As such, in 1998 the MND directed that a more robust DND/CAF AWP be created and operated to provide a framework for regulating all aspects of military aviation in accordance with the Aeronautics Act. In the directive, the MND stated that, “the Department has a ‘Duty of Care’ to persons who may be harmed by the operation in Canada of military aircraft which are not airworthy. Therefore, it is implicit that the Department should operate under an aviation safety system which is no less effective than required by civil aviation.”¹¹

Aviation safety for both civil and military aeronautics has advanced significantly over the last two decades and continues to evolve. The present DND/CAF AWP strongly supports the safety of existing and future military aeronautics through compliance with appropriate airworthiness standards and regulations.

2.2 Relevance—Alignment with Federal Roles and Responsibilities

This section examines the extent to which the AWP aligns with departmental and federal roles and responsibilities. The findings in this section are based on evidence from document reviews.

Key Finding 2: The AWP aligns with federal roles and responsibilities.

The evaluation used the following indicators to assess the alignment of the AWP:

• alignment with government acts, legislation and policies; and
• the extent to which the AWP conducts activities that are the responsibilities of other government departments, other levels of government or the private sector.

The MND has national and international obligations by law for the management of airworthiness of all Canadian military aviation. The responsibilities of the Minister are outlined in Section 4.2 of the Aeronautics Act, “The Minister is responsible for the development and regulation of aeronautics and the supervision of all matters connected with aeronautics.”¹² The discharge of these responsibilities applies to any matter relating to military personnel, military aircraft, military aerodrome or military facility of Canada or a foreign state or any other matter relating to defence.

As part of Canadian law, the Aeronautics Act assigns the responsibilities for the regulation of civil aviation to the Minister of Transport and for the regulation of military aviation to the MND. As such, the MND is explicitly responsible for the development and regulation of aeronautics and the supervision of all matters connected with the MND’s span of control over Canadian airspace.

2.3 Relevance—Alignment with Government Priorities

This section examines the extent to which the AWP aligns with government priorities and departmental strategic outcomes. The findings in this section are based on evidence from document reviews.

Key Finding 3: The AWP aligns with government priorities and DND/CAF strategic outcomes.

The following indicators were used to assess the outcome of the finding:

• alignment between AWP priorities and federal government priorities; and
• alignment between AWP priorities and DND/CAF strategic outcomes.

In 1998, the MND directed that a more robust DND/CAF AWP be created and operated to provide a framework for regulating all aspects of military aviation in accordance with his responsibilities for military aviation safety under the Aeronautics Act. The AWP is specifically identified in the DND PAA as a regulatory programme that contributes directly to the strategic outcome “Defence Remains Continually Prepared to Deliver National Defence and Defence Services in Alignment with Canadian Interests and Values.” As such, the AWP is a key enabler for the RCAF to maintain its aerospace readiness in support of the Canada First Defence Strategy¹³ and the CAF Force Posture and Defence Readiness.

2.4 Performance—Achievement of Expected Outcomes (Effectiveness)

This section provides an evaluation of the overall effectiveness of the AWP. The AWP is a broad program involving several DND and RCAF organizations. The evaluation team worked with stakeholders to identify key program activities and associated outputs and outcomes. The principal outcomes identified correspond to the scope of responsibility of each of the four airworthiness authorities (AA, TAA, OAA and AIA) and can be summarized as follows:

• effective airworthiness governance;
• effective technical airworthiness (TA);
• effective OA; and
• effective investigative airworthiness (IA).

In order to assess each of the outcomes, indicators were used to assess their effectiveness. The following key areas were questioned/examined:

• Scope of Authority: Was the scope of authority appropriate, well documented and exercised in accordance with policy and directives?
• Organizational Structure: Was the organizational structure adequate to meet the scope of authority?
• Processes: Were the processes functional and well documented?
• Risk Management: Was the risk management well-defined, functional and appropriately managed?
• Documentation: Was the documentation relevant, complete and updated regularly?
• Training: Was the training adequate to meet the needs of the program?
• PMF: Was the PMF well-defined, functional and appropriately managed?
• Resources: Were the resources adequate to meet the needs of the program?

In order to assess the performance of the AWP against these key areas, the evaluators conducted a review of program documentation and reports, researched open source benchmarking to comparable programs, interviewed senior leaders and personnel at the command and staff levels and obtained feedback from stakeholders.

Since the findings and recommendations arising from the evaluation of the performance are extensive, the following sub-sections (2.4.1 through 2.4.4) provide a concise summary of the main issues. Detailed descriptions of findings and recommendations are provided at Annexes A, B, C and D.

2.4.1 Governance and Management of the AWP

The ultimate outcome of the DND/CAF AWP is to contribute to the sustainment of RCAF readiness with an acceptable level of safety. This is accomplished by the establishment and management of an AWP with effective governance to ensure safe aeronautical products and flying operations, thorough investigations and effective regulatory oversight. While the CDS has designated the C Air Force as the AA for the DND/CAF AWP, responsible for the overall management and monitoring of the AWP,¹⁴ the AA is assisted by the OAA, TAA and AIA, who are responsible for regulating and overseeing the implementation of the operational, technical and investigative aspects of airworthiness, respectively. Together, these four authorities manage and oversee the regulation and implementation of the overall DND/CAF AWP. The management of the AWP, the inter-relations between the authorities and their sub-programs, as well as the processes involving all of the sub-programs are outlined within this Governance section and are discussed in more detail in Annex A.

The evaluation made 20 findings and 19 recommendations regarding the governance of the AWP. These are summarized in this section and explained in detail at Annex A.

In general, the AWP is appropriately based on a sound foundation of formal delegations and documentation, comprehensive TA and risk management processes, annual fleet airworthiness review boards and periodic audits of operational and technical organizations. During the last five years, several new fleets were successfully introduced into service, and significant programmatic improvements have been made in several areas, most notably in risk management, operational airworthiness documentation and technical auditing.

While the AWP is a generally effective program with clearly defined scopes of authority and well-documented policies and processes, there are concerns that its governance structure does not have sufficient resources and mechanisms in place to properly oversee and manage the program at a strategic level. A Reserve Force position exists to assist the AA with the AWP, but this position has not been manned consistently and is assessed as insufficient to properly oversee, coordinate and update airworthiness issues and guidance at the strategic level. The net effect is that AWP issues are addressed only when necessary and primarily in support of the annually scheduled Airworthiness Advisory Board (AAB). In addition, monitoring of the AWP has been delegated to the AIA. While this has some benefits by allowing the AIA to monitor TAA and OAA activities and products, it also has the potential of compromising the independence of the AIA for flight safety (FS) investigations. Further, the scope of AIA monitoring is limited as the evaluation did not find any evidence of overall AWP strategic monitoring, audits or performance measurement. As a result, there is no strategic assessment of the effectiveness, efficiency and costs of the overall program and associated improvement initiatives. For these reasons, the evaluation recommends staffing an AA support section reporting directly to the AA or to the Assistant C Air Force to provide more objective monitoring and strategic management of the AWP and oversight of the TA/OA/IA programs and to improve the coordination and timeliness of AA guidance and publication updates. The addition of dedicated staff would also improve management of new initiatives and awareness of best practices of foreign military airworthiness programs. In concert with this, a PMF should also be implemented to more effectively monitor and manage the AWP. These elements are key to managing the AWP and ensuring continuous improvement.

Strategic oversight and governance mechanisms for the DND/CAF AWP are limited. The AAB is the only meeting attended by all the airworthiness authorities. Chaired by the AA, it provides a forum to update the members on the AWP and to address any issues or concerns. It also provides the basis for the Annual Airworthiness Programme Report to the CDS and the MND. The AAB was assessed to be tightly scripted and time-compressed, with limited opportunity to discuss program challenges and issues impacting the AWP. The evaluation recommends allocating more time to the AAB. For similar reasons, the evaluation also recommends establishing a formal senior airworthiness forum between DND and TC to promote communication, mutual development and resolution of issues to the mutual benefit of Canada’s two airworthiness regulators. Subjects of mutual concern include unmanned aerial vehicle (UAV) clearances and operational oversight, contracted civilian aircraft operators, new international aviation regulations and navigation requirements, training courses and personnel qualifications and simulator certification.

The other key DND airworthiness forum is the annual Airworthiness Review Board (ARB). Its purpose is to annually review the airworthiness clearance of all the in-service fleets. The ARB meets twice a year and is attended by all the airworthiness authorities, including a representative for the AA. The ARB reflects airworthiness best practices and provides an important opportunity for most of the airworthiness authorities to meet and review fleet issues and discuss more general airworthiness concerns. It is an effective oversight forum and a key component of the AWP due to its requirement for annual fleet updates and reports.

The lack of independence of the DND/CAF AWP from the operational chain of command is another area of concern. One of the key principles of an effective airworthiness program is an appropriate level of independence between the regulators and the implementers of the program, in order to avoid any potential conflict of interest. Allied defence departments typically address the principle of independence by organizational constructs that provide as much separation as possible between the Military Airworthiness Authority (MAA) and the implementers. The United States of America (USA), the United Kingdom (UK) and the Netherlands, for example, have implemented an MAA separate from their operational chains of command. Other countries, such as Canada and Australia, are at the other end of the spectrum, with airworthiness regulators reporting to an airworthiness authority that is double-hatted with an operational command role. However, Australia adds independent oversight by holding annual ARBs chaired by retired Australian generals outside the chain of command. The evaluation assessed that the current construct of the DND/CAF AWP involves undue risk, since all of the DND airworthiness authorities are double-hatted and thus are subject to conflicts of interest between their operational demands and their airworthiness responsibilities. Independent airworthiness oversight would increase the objectivity and robustness of the DND/CAF AWP. The fact that the DND/CAF AWP appears to have been working well to date should not prevent the DND/CAF from following international best practices and implementing some level of independent oversight. The UK airworthiness program also appeared to be working well until a Nimrod accident review¹⁵ led the UK military to completely overhaul its program to increase the independence of its MAA and the accountability and liability of its airworthiness authorities.

One of the key processes involving all of the DND/CAF airworthiness authorities is airworthiness risk management. The evaluation found the processes for quantifying and approving airworthiness risk assessments and control plans to be well documented and airworthiness risks to be well recorded and monitored until mitigation or resolution. The AWP’s risk management process has shown improvement since 2008; risk management training has been provided to technical and operational staff and to contractors maintaining RCAF aircraft, and an airworthiness risk management (AWRM) database was introduced to efficiently standardize, document and track risk assessments as Records of Airworthiness Risk Management (RARM). That said, the current AWRM database does not address the previous CRS recommendation¹⁶ for a single, universally recognized, easily accessible and current RARM database. In addition, senior airworthiness authorities may not have a full appreciation of all the fleet RARMs when renewing the fleet Airworthiness Clearance at ARB. Accordingly, the evaluation made a number of recommendations to improve the sustainability and accessibility of the AWRM database, to improve senior management awareness of fleet risks and risk acceptance decisions and to improve airworthiness risk management in general.

Another aspect of the AWP affecting all of the airworthiness authorities is the need for training. A key airworthiness principle is that authority assigned to individuals must be based upon their demonstrated skills, knowledge and experience. Interviews indicate that the overall level of understanding of the AWP within the DND/CAF implementer and operational communities is often poor, and there are training gaps within the regulatory community and with some delegated authorities. The evaluation recommends introducing an on-line airworthiness familiarization course to provide a fundamental level of understanding to all military and civilian personnel directly or indirectly involved with aircraft systems, as well as establishing appropriate airworthiness qualifications and training for all DND/CAF regulator and implementer personnel with airworthiness responsibilities. The National Defence website, which currently only mentions the Technical AWP, should also be amended to provide information on the entire DND AWP, including its key authorities, roles and responsibilities and publications.

Finally, while UAV clearances and the issuance of Specific Purpose Flight Permits (SPFP) appear to function effectively when OAA or TAA staff are contacted to oversee the process, there are concerns that this may not always occur due to a lack of DND/CAF-wide awareness of the UAV Release to Service (RTS) process. The evaluation identified a need for AA policy and guidance regarding the UAV clearance process, approval authorities and operational responsibilities to ensure airworthy flying operations. The evaluation also recommends implementing dedicated personnel resources to assure appropriate oversight and compliance.

2.4.2 Technical Airworthiness

Following the 1998 MND direction to formally implement a more robust airworthiness program for military aviation, the TAA revamped its program to align with accepted worldwide principles. As such, the DND Technical Airworthiness Program (TAP) follows the principle of authorized individuals working in accredited organizations and following approved procedures and accepted standards. The TAP thus regulates and ensures the implementation of the technical aspects of airworthiness.

The evaluation made 14 findings and 8 recommendations regarding the TAP. These are summarized in this section and explained in detail at Annex B.

The CDS normally designates the DGAEPM as the TAA; however, this authority can alternatively be assigned to the Director of Technical Airworthiness and Engineering Support (DTAES) if the existing DGAEPM has limited background in airworthiness issues. In general, the TAA has developed and embodied the structure and instruments necessary to successfully manage the regulatory functions and implementation of the TAP. The TAA has extensively reviewed and improved the TAP by implementing many recommendations arising from comprehensive independent evaluations and internal reviews. The TAP has rigorous, well-documented TA certification and clearance processes for initial and continuing airworthiness that reflect accepted worldwide airworthiness practices for approving type designs, design changes and airworthiness products. While the regulatory independence of the TAA is considered adequate, being outside the operational chain of command, resource constraints and competing pressures within DGAEPM have restricted progress of the TAP. Initiatives to improve the regulatory functions such as improved performance measurement, improved training and mutual recognition of allied MAAs are competing against DGAEPM implementer demands and force generation activities. While an aging demographic within the regulator staff suggests an imminent need for the TAA to implement minimum airworthiness training requirements, there are insufficient resources and no formal training authority to implement the identified training. The evaluation recommends prioritizing initiatives and recommends that issues and decisions impacting regulatory airworthiness functions and initiatives receive high-level visibility at the AAB and in the Annual Airworthiness Report (AAR) to the CDS and the MND.

Outside the regulator community, the implementers’ understanding of the regulators’ roles and specific functions is inconsistent. Interviews revealed that some implementers still did not fully understand the DTAES regulator-implementer relationship. This was particularly evident in project offices, but also in certain DGAEPM directorates where misunderstandings and expectations regarding mutual responsibilities of the regulators and engineering staff had often been a source of confusion and delays. The evaluation recommends identifying and implementing the necessary training and for regulators and project staff to establish expectations, mutual responsibilities and work requirements at the beginning and periodically during the course of projects.

While the TAP has developed a sound performance measurement tool to monitor its regulatory aspects, as well as a strong audit program to assure implementation of the program, performance measures to monitor and assess the overall implementation of the TAP should be implemented. Similarly, the TAA has comprehensive regulatory documents that have evolved considerably over several years, but additional effort is required to finally complete them. Completion of these documents should be monitored by the TAA, and thereafter a periodic review process should be implemented to ensure they are reviewed and updated on a regular basis.

2.4.3 Operational Airworthiness

In support of the DND/CAF AWP, the outputs and immediate outcome of the OA program is to promulgate DND regulations, standards and competencies for flying operations and ensure the operations conform to regulations and standards. The OA program thus regulates and ensures the implementation of the operational aspects of airworthiness.

The evaluation made 12 findings and 22 recommendations regarding the OA program. These are summarized in this section and explained in detail at Annex C.

The CDS designates the Comd 1 CAD as the OAA. The OAA has a clearly established scope of authority to regulate and oversee operations for all Canadian and foreign military aeronautical activities within the Canadian airspace and for all Canadian military aeronautical activities around the world. In addition to four OA staff members, the OA program relies heavily upon existing operational staff within 1 CAD and 2 CAD, which are dual-hatted with OA responsibilities to provide the necessary airworthiness oversight of aviation operations. While this is efficient and economical, the primary divisional focus on operations risks compromising regulatory independence. Double-hatting of OA authorities and operational command chains is common in many allied countries, but the DND/CAF is distinct in lacking an OA regulatory authority independent from the OAA. Accordingly, the evaluation recommends implementing an OA regulatory authority that is separate from the OAA and the operational chain of command. The small size of the regulatory staff is also a concern because the ability of the organization to retain expertise and fully qualified and experienced staff is difficult with the regular turnover of military personnel, which risks compromising corporate knowledge. The evaluation recommends improving the retention of OA expertise and the stability of OA regulatory positions by hiring some civilian subject matter experts (SME) and further documenting OA procedures.

The evaluation also raised concerns regarding the minimal resources and oversight in certain areas of OAA responsibility. OAA responsibility is much broader than core RCAF fleet activities; however, the reliance on the divisional organizational structure has resulted in extremely limited airworthiness oversight of military aviation activities in which the RCAF has little or no direct operational involvement, such as CA, RCN, and foreign UAVs operating in Canada, contracted aviation operators, and foreign military aviation operations in Canada. The evaluation recommends the OAA review the OA organization to ensure that sufficient resources, including dedicated audit teams, are available to regulate, oversee and assure the operational airworthiness of all aviation operations under OAA responsibility. The OAA structure should also involve Comd 2 CAD in the OA program and ensure adequate OA oversight exists in 2 CAD.

While the OAA has made significant progress in documenting the Operational Airworthiness Manual (OAM) and establishing OA processes, the evaluation recommends implementing a fixed-cycle review process to ensure OAA regulatory documents and procedures are regularly reviewed and updated. Several processes should also be re-examined and improved. In particular, the appropriateness of the procedures, resources and information requirements for approving foreign military aviation operations and overflights in Canada should be reviewed. The involvement of countries without a recognized military airworthiness program potentially elevates the corresponding risk of such operations. However, there is no documented OAA process regarding how to regulate and ensure the airworthiness of such operations. The evaluation recommends implementing a formal process as well as monitoring and oversight resources to ensure compliance with the DND/CAF AWP.

Similarly, the evaluation team is concerned that Statements of Operating Intent (SOI) are not being reviewed yearly, and it recommends including an update on the fleet SOI in the annual fleet Operational AAR. Such SOI reviews are important to ensure that changes to an aircraft’s roles/missions/usage are technically and operationally evaluated to assess their impact on aircraft maintenance and service life.

While the OAA has a well-defined process for quantifying and accepting airworthiness risks, there are concerns that the recent introduction of the Operational Risk Assessment Tool (ORAT), intended to document operational risks that do not have any airworthiness implications, could potentially circumvent the airworthiness risk management process in cases where the airworthiness implications are not evident. The evaluation recommends further clarifying the definitions to avoid any ambiguity, as well as the monitoring of ORATs by OA staff to ensure valid airworthiness risks are correctly documented.

The evaluation also raised concerns regarding airworthiness training of OA regulators and implementers. In particular, all OA personnel authorized to approve or accept RARMs should be trained, qualified and explicitly delegated to do so. While some personnel have received training, it is often informal and inconsistent. The evaluation recommends appointing an OA training authority, formally identifying training requirements for each regulatory and operational position with OA responsibilities and monitoring training to ensure OA staff are appropriately trained before they receive any airworthiness delegation of authority. Basic airworthiness training/knowledge should also be provided to operational aircrew, and RARMs involving aircraft restrictions should be communicated to the operational level to provide aircrew with a better understanding of the rationale and implications of the associated restrictions.

While OA-specific performance measures and audits are lacking, the 1 CAD Standards and Evaluation Teams (SET) already conduct similar audits of operational and maintenance units to ensure appropriate standards and procedures are being followed. The evaluation recommends adapting these existing audits to ensure OA activities and airworthiness aspects are included and to ensure the OA program benefits from SET feedback. Finally, the evaluation recommends the OAA implement a PMF to better monitor, manage and improve the OA program.

2.4.4 Investigative Airworthiness

The CDS designates the DFS as the AIA. In support of the AWP, the primary responsibility of the AIA is to conduct independent investigations of airworthiness-related accidents. The AIA is also tasked to monitor the AWP to identify deficiencies and recommend preventative measures.

The evaluation made 10 findings and 7 recommendations regarding the AIA program. These are listed in this section and explained in detail at Annex D.

Overall, the evaluation assessed that the core investigative and FS functions are being performed effectively, and progress has been made on some significant issues. The mandate of the AIA has been broadened to include investigation in zones of conflict of all occurrences where the aircraft, its equipment or its operation may have been contributing factors. Also, the recent passing of Bill C-3 “Safeguarding Canada’s Seas and Skies Act” now provides the AIA with the statutory powers to investigate civilians involved in military accidents and incidents.

While the AIA is adequately staffed to perform most accident investigations, investigator experience is an issue and its capacity to handle large scale accidents is correspondingly limited. The training curriculum for aircraft accident investigators is well established and managed, but the training time and high turnover of military accident investigators, due to posting cycles, create a loss of expertise and are a vulnerability to the AWP investigative capability. The evaluation recommends improving the stability of key investigator positions by changing some military investigator positions into qualified civilian investigator positions. A consolidation of FS organizations within National Defence Headquarters (NDHQ) and 1 CAD might also provide more flexibility for the AIA to fulfill assigned roles.

The time required to complete FS investigation reports and the availability of FS training for unit FS personnel are other issues that need to be carefully managed. Both of these issues have the potential to negatively impact the investigative effectiveness and efficiency of the AWP. The evaluation suggests establishing a small production section capable of managing and monitoring the quality and structure of investigative reports in order to reduce the time required to complete investigation reports. Tracking of unit-level FS personnel qualifications, and improvements in the frequency and methods of FS course delivery are recommended to improve unit-level FS training availability.

Finally, while the AIA tracks extensive accident/incident data, it does not have a formal PMF to monitor and improve AIA activities. Performance management is important because it assesses progress towards goals and objectives. The evaluation recommends integrating and augmenting current monitoring activities into a PMF aimed at continuously monitoring and improving the AIA program. A formal AIA documentation review process is also recommended to ensure the periodic review and update of all AIA documents.

2.5 Performance—Demonstration of Efficiency and Economy

The Aeronautics Act and resulting AWP brought a legal basis, a responsibility/accountability framework and a regulatory system that reinforced the importance and safety aspects of pre-existing RCAF maintenance, engineering and operational processes. The AWP was largely implemented by adding responsibilities to existing senior personnel, along with reshuffling a limited number of engineering staff resources into DTAES regulatory positions. As a result, the AWP is considered efficient in that most personnel involved in the program are double-hatted, including all the senior airworthiness authorities and all the implementers. The only exception are the 57 regulatory personnel within DTAES 2, 3, 4 and 5 and four personnel within 1 CAD filling positions solely dedicated to airworthiness. Precisely because most personnel are double-hatted, the total cost of the AWP is difficult to determine since it is difficult to estimate what portion of their work should be attributed to the AWP itself. However, the extra cost of the AWP itself is considered to be marginal since airworthiness is a framework for providing an acceptable level of safety for military aviation, something the DND/CAF would have to do regardless. As previously indicated, the implementation of a performance management framework is recommended as it would provide a means to track improvements to the efficiency and costs of the program.

Other improvements that should be considered to achieve greater efficiency of the AWP include, but are not limited to, establishing closer coordination and exchanges with TC, recognizing allied certifications and accreditations and increasing the use and retention of civilian expertise within the regulatory and investigative staff.

Annex A—Governance Effectiveness

The ultimate outcome of the DND/CAF AWP is to contribute to the sustainment of RCAF readiness with an acceptable level of safety. This is accomplished by the intermediate outcome of establishing an airworthiness program to effectively manage current and future aeronautical activities. The AWP requires effective governance and management to ensure regulatory oversight and effective management of initial airworthiness and continuing airworthiness. While the CDS has designated the C Air Force as the AA for the DND/CAF AWP, responsible for the overall management and monitoring of the AWP,¹⁷ the AA is assisted by the OAA, TAA and AIA, who are responsible for regulating and overseeing the implementation of the operational, technical and investigative aspects of airworthiness, respectively. Together, these four authorities manage and oversee the regulation and implementation of the overall DND/CAF AWP.

Indicator 1: The scope of authority is appropriate, well-documented and is exercised in accordance with policy and directives.

Finding A1: The AA has a clearly defined scope of authority to manage and develop the AWP. However, the responsibility for monitoring the AWP has been delegated to the AIA.

As indicated in the AWP policy document,¹⁸ the C Air Force is the DND/CAF AA responsible for managing and developing the AWP, whereas monitoring and auditing of the AWP has been delegated to DFS as the AIA.¹⁹ While this has some benefits in allowing the AIA to review TAA and OAA activities and products, the evaluation found no evidence of strategic monitoring or auditing of the AWP, likely due to the lack of AA and AIA resources to support this function. The delegation of this monitoring role to the AIA is also a concern because it is distinct from the core investigative and FS functions of the AIA and could potentially affect the AIA’s mandated requirement to be impartial and independent during accident/incident investigations. A second concern is that it requires the AIA to assess itself as part of the AWP and to monitor and audit the performance of the OA and TA programs, each of which is managed by more senior DND/CAF officials. To strategically and independently monitor the performance of the overall AWP, it would be more appropriate to have a small dedicated airworthiness section reporting to the AA or to the Assistant C Air Force to monitor and audit the AWP, thereby directly assisting the AA to more effectively oversee and manage the AWP. This is further discussed in the next section under organizational structure.

Indicator 2: The organizational structure is adequate to meet the scope of authority.

Finding A2: The AWP has been fulfilling its mandate. However, more effective governance mechanisms would improve the AWP and potentially yield efficiencies.

In general, the AWP is working well based on a sound foundation of formal delegations and documentation, comprehensive TA and risk management processes, annual fleet airworthiness review boards and periodic audits of operational and technical organizations. During the last five years, several new fleets were successfully introduced into service, and significant programmatic improvements have been made in several areas, most notably in risk management, operational airworthiness documentation and technical auditing. However, the AWP is to a large extent a passively-led, bottom-up program that succeeds and improves due to the professionalism and diligence of its personnel. That said, there are many issues that would benefit from increased involvement of senior airworthiness authorities. While the TAA, OAA and AIA all have dedicated airworthiness organizations or sections to help them oversee and manage their airworthiness responsibilities, the C Air Force does not have a dedicated airworthiness organization to support the AA role. An Airworthiness Coordination Cell exists within the AA organizational structure, but in practice only consists of one Reserve Force officer and is inconsistently manned and not strictly dedicated to airworthiness. As a result, the AA does not have the capacity to effectively manage the AWP or become familiar enough with the intricacies of the program to provide the necessary challenge function and proactively guide the program forward. AWP issues are addressed only when necessary and primarily in support of the annually scheduled AAB. While the AAB is an opportunity for the AA to track some AWP initiatives, it is an insufficient mechanism for actively monitoring and managing the program. Active and visible senior management engagement would contribute to energize the AWP. The creation of a dedicated airworthiness section would allow the promulgation of new airworthiness policies and documentation updates in a more timely manner. It would also assist the AA in providing strategic monitoring of the AWP, objective oversight of the TA, OA and IA programs, review and advice regarding airworthiness issues, awareness of new initiatives and best practices of foreign militaries and coordination and implementation of AA AWP guidance. As described at Annex E, Australia has a very similar military airworthiness structure to that of the DND/CAF, but has established the Airworthiness Coordination and Policy Agency, a strategic airworthiness organization supporting the Australian Defence Forces AA, who is also its Chief of the Air Force.

Finding A3: The lack of a formal senior airworthiness forum between DND and TC impedes communication, mutual development and resolution of issues between Canada’s two airworthiness regulators.

There are only two airworthiness regulators within Canada: TC and DND. Interviews indicated that while there are some agreements, informal discussions and exchanges between the two organizations, there is no established forum for senior DND/CAF airworthiness authorities to regularly discuss issues and best practices with their counterparts at TC. Current subjects of mutual concern would include UAV clearances and operational oversight, contracted civilian aircraft operators, new international aviation regulations and navigation requirements, training courses and personnel qualifications and simulator certification. In addition, leveraging each other’s organizations could be pursued to investigate synergies, alleviate organizational constraints and increase efficiencies. For example, a 2012 TC decision²⁰ to stop certifying DND/CAF flight simulators will require the DND/CAF to expend resources to implement a duplicate capability. The continued use of TC certification staff would represent efficiencies and economies of scale beneficial to the DND/CAF even if some form of cost recovery were involved. Regular senior-level discussions with TC staff would provide both parties the opportunity to discuss and resolve such issues proactively and to the mutual benefit of both organizations.

Finding A4: The current DND/CAF AWP lacks objective, independent oversight.

One of the key principles of an effective airworthiness program is an appropriate level of independence between the regulators and the implementers of the program in order to avoid any conflict of interest. While this is normally the case in civil aviation, where government agencies regulate private operators, military organizations typically self-regulate since their operations often involve greater risk than what is acceptable in civilian practice and therefore cannot be held to civilian airworthiness standards and regulations. As Annex E demonstrates, defence departments address the principle of independence by organizational constructs that provide as much separation as possible between the MAA and the implementers. At one end of the spectrum, the UK and the Netherlands’ defence departments have an entirely independent MAA reporting to their respective ministers of defence. MAAs in the US Armed Forces are also separate from their operational chains of command. At the other end of the spectrum, Canada and Australia have airworthiness regulators that report to an airworthiness authority who is double-hatted with an operational command role. However, Australia partly compensates for this by holding annual airworthiness review boards chaired by retired generals independent of the chain of command. The evaluation assessed that the current construct of the DND/CAF AWP involves undue risk, since all of the DND/CAF airworthiness authorities are double-hatted and thus inevitably subject to conflicts of interest between their operational pressures and their airworthiness responsibilities. The fact that the DND/CAF AWP appears to work reasonably well should not prevent the DND/CAF from following international best practices and implementing independent oversight. Independent airworthiness oversight would increase the objectivity, robustness and effectiveness of the DND/CAF AWP.

Indicator 3: Processes are functional and well-documented.

Finding A5: The airworthiness processes function generally well but often rely on RARMs to document remaining risks and to permit airworthiness clearances to be issued in a timely manner.

In order to support AA approval of the RTS of new aircraft systems, the TAA ensures its technical airworthiness through the progressive issuance of Military Type Certificates (MTC) to approve their type designs; Certificates of Airworthiness to certify the resulting aircraft products; and Technical Airworthiness Clearances (TAC) to confirm that all TA elements have been implemented to introduce and support a new aircraft into service. Similarly, the OAA and AIA issue OA Clearances and IA Clearances, respectively, to confirm that all operational and investigative airworthiness elements have been implemented. This DND/CAF process has developed and matured over the last 15 years with the acquisition of several new aircraft fleets. The process is complex and rigorous and conforms to the same process and standards as employed by allied military and civilian airworthiness authorities (see benchmark analysis at Annex E).

Since military aircraft are procured to fill an operational need, there is often real or perceived pressure to quickly introduce a newly acquired aircraft system into service, particularly if similar aircraft are already cleared for use by allies. Ultimately, airworthiness exceptions are tolerated by the AA, OAA and TAA, and a provisional airworthiness clearance is issued if mitigation and residual risks allow for an acceptable level of safety, which may involve several restrictions on aircraft use. The issuance of formal documentation and formal acceptance by the TAA and OAA of aircraft limitations and remaining actions required to acquire full Airworthiness Clearance is a critical step to manage leadership expectations and ensure the safe release to service. For example, the CH147 initial certification process, which led to the issuance of an SPFP (vice a full Certificate of Airworthiness) and a Provisional TAC as part of the RTS documentation. The documentation was well presented with a coherent plan to attain initial operational capability and full operational capability. All of the temporary TA limitations were well identified and understood by the OAA and AA, and corresponding restrictions were implemented. The remaining risks were documented and are now being managed and tracked as RARMs. Nevertheless, such cases raise concerns regarding the extent to which RARMs and restrictions should be used to clear a new aircraft for use. The excessive use of RARMs could potentially undermine the purpose of the clearance process and the AWP itself. This concern was also expressed in the Mansfield Report.²¹ Policy guidance would be appropriate to clarify the acceptability or limits of this practice.

Additional recommendations regarding the management and use of RARMs for in-service aircraft are included in the Governance Risk Management section.

Finding A6: A DND/CAF-wide policy regarding UAV airworthiness clearances and approval authorities is required.

While the RCAF operates most airborne systems, the CA, RCN and foreign militaries (primarily the US Army) also operate UAVs in Canadian airspace. All of these airborne systems fall under the oversight of C Air Force as the designated AA who ultimately issues the airworthiness clearances and RTS necessary to ensure safety and airspace coordination. For RCAF fleets, airworthiness regulatory and oversight activities essentially follow the same airworthiness process as other manned aircraft. For non-RCAF operators, UAV clearances and foreign SPFPs appear to function effectively when OA or TA staff are contacted with sufficient forewarning to initiate the process; however, there are concerns that this may not always occur due to a lack of DND/CAF-wide awareness of the UAV RTS process. This issue has been recently discussed at the ARB and AAB.²² The evaluation was advised that documents are being drafted by an informal CA UAV Working Group²³ to ensure all DND organizations are formally apprised of the process and approval authorities, but these have yet to be promulgated.

The lack of dedicated, expert staff to support the clearance and RTS process of these specialized airborne systems is also an issue. The lack of resident SMEs, especially within the OA organization, may introduce risk to the oversight responsibilities of the AA and OAA. AA guidance on UAVs is an essential step, but adequate personnel resources must also be available to ensure appropriate oversight and compliance.

Finding A7: Oversight mechanisms to ensure overall management of the DND/CAF AWP are few and may not adequately address key issues and developments.

Oversight and governance mechanisms for the DND/CAF AWP are limited and essentially consist of two DND airworthiness forums: the AAB and the ARB. The ARB is chaired jointly by the OAA and TAA and is attended by the other airworthiness authorities and a representative from the AA.²⁴ The ARB was established to manage the interface between the OA and TA of each aircraft fleet, annually review the airworthiness clearance of all in-service aircraft types, recommend the airworthiness clearance of new aircraft types to the AA and address observations and concerns raised by other DND airworthiness authorities and advisors.²⁵ Such annual fleet reviews reflect airworthiness best practices, with Australia and the UK conducting similar reviews. The ARB meets twice a year and reviews half of the fleets at the first meeting and the remaining fleets at the second meeting. In support of the ARB, the operational and technical managers and the AIA representative of each fleet submit individual AARs for members of the ARB to review. In general, the ARB is an effective oversight forum and a key engine for the AWP due to its requirement for annual fleet updates and reports. It also provides a rare bi-annual opportunity for most of the airworthiness authorities to meet and discuss specific fleet airworthiness issues. While the evaluation did not find evidence of any fleet whose Airworthiness Clearance was not renewed, this is undoubtedly due to the fact that mitigation measures, such as increased maintenance or other restrictions, have normally already been implemented by the fleet airworthiness authorities well before the ARB. More general airworthiness issues are also discussed the previous day at a preliminary Executive ARB meeting.

The annual AAB is chaired by the AA and is the only meeting that the AA and all the other airworthiness authorities attend. It provides a forum to update the AA on the AWP and to address any issues or concerns. It also provides the basis for the Annual Airworthiness Programme Report to the CDS and the MND. The AAB was assessed to be time-compressed, scripted and controlled, and it mainly addresses high-visibility fleet airworthiness issues. As such, many airworthiness programmatic issues are not discussed or given appropriate visibility. This has impeded the opportunity to openly discuss program challenges and resource constraints that hamper the progress of airworthiness initiatives and the AWP in general. Terms of reference for the AAB should ensure adequate review of programmatic aspects of the AWP and key issues impacting the separate authorities and other stakeholders.

Indicator 4: Risk management is well defined, functional and appropriately managed.

Finding A8: Airworthiness risks are well documented and monitored until mitigation or resolution.

Airworthiness risks that do not meet the pre-defined Acceptable Level of Safety (ALOS) for a fleet are documented in detail as RARMs.^{26 27 28 29} New and revised RARMs are approved by delegated TAA or OAA authorities. In order to be approved, RARMs require a detailed assessment of risk hazards and risk control activities, as well as a risk index forecast table to be completed. This captures milestones and ensures most RARMs implement risk control activities to resolve or mitigate the risk in a timely manner. All open RARMs and those closed within the past year are reviewed at least annually by each fleet Senior Design Engineer (SDE) and reported in the fleet Technical AAR. Any missed approvals or milestones are flagged by the AWRM database and normally resolved prior to the AAR. In addition, Directorate of Technical Airworthiness and Engineering Support (DTAES) 4 staff, on behalf of the TAA, annually audit the RARM database for each fleet in preparation for the ARB. These RARM reviews and audits in support of the AAR and ARB ensure that airworthiness risks are properly documented, approved and monitored until the problem has either been eliminated or mitigated to within the fleet ALOS. Any contentious issues between regulatory staff and implementers are briefed at the ARB.

Finding A9: There are some discrepancies between the TAM and the OAM regarding the delegated RARM airworthiness approval and acceptance authorities.

New and revised RARMs must be approved by authorized TA and OA authorities and finally accepted by an authorized Operational Command Risk Acceptance Authority (OCRAA). The level of the approval and acceptance authorities varies in accordance with the risk index of the RARM; however, there are discrepancies in the TAM³⁰ and OAM³¹ regarding the OA approval and OCRAA authorities. Further, since these authorities are key to the airworthiness risk management process as a whole, it is suggested that these changes should only be authorized by the AAB, with the authoritative table included within the AWP manual.³²

Finding A10: Since current risks (i.e., open RARMs) are not explicitly presented at ARBs and AABs, senior airworthiness leadership may not have sufficient awareness of the airworthiness status of each fleet.

The mandate of the ARB³³ is to review the airworthiness status of each fleet. The TAA and OAA staff thoroughly vet all AARs in advance of the ARB, and they pre-brief the TAA and OAA on all fleet issues, highlighting those that require ARB attention. When the ARB takes place, the chairs determine the continued validity of each fleet’s Airworthiness Clearance as presented.³⁴ Although all open RARMs and those closed within the past year are included in the fleet AAR, a review of past ARB minutes made no reference to the content of fleet AARs, and there is no criteria that requires a RARM to be briefed to the ARB, except for non-fleet-specific RARMs, which are reviewed annually (since May 2013). As a result, the ARB process briefs about RARMs by exception when issues are considered significant enough to warrant ARB attention, and when there is no traceability of all the fleet RARMs or AARs to the ARB and the renewed fleet Airworthiness Clearance. Further, since the TAA, OAA and AA only approve RARMs at the risk level of “extremely high” (approval of RARMs of a risk level of “high” or lower has been delegated^{35 36}), there is a risk that senior airworthiness authorities do not obtain a full appreciation of all the open risks for each fleet. To address these issues, the ARB should reference the AARs and review all open RARMs above ALOS. Since most fleets have very few non-RARMs (only three fleets have 10 or more), the added time and effort involved would be minor and would ensure appropriate awareness by the senior airworthiness authorities prior to renewing the fleet clearances.

Finding A11: The airworthiness risk management process has shown improvement since 2008.

Several improvements have been made in the management of airworthiness risk since the 2008 CRS Review. Risk management training has been provided to technical and operational staff, in both DGAEPM and 1 CAD, as well as to contractors maintaining RCAF aircraft. In addition, in 2010-2012, DGAEPM introduced the AWRM database as an interim solution to effectively and efficiently standardize, document and track all RARMs. As all RCAF airworthiness risks are documented as RARMs, the monitoring of all RARMs is essential for effective airworthiness risk management. Fleet RARMs are reported annually in the fleet Technical AAR and audited annually by TAA staff in preparation for the ARB. The AWRM database application ensures a consolidated, consistent standard format for each RARM, including hazard assessments, risk control plans and necessary approvals, in addition to the monitoring of upcoming and overdue risk control activities and approvals. The description and use of the AWRM database is well documented.^{37 38}

Finding A12: The current DGAEPM AWRM database is a good interim solution but should be improved to address deficiencies and ensure long-term sustainability.

The current AWRM database was initiated as a requirements definition prototype and was never intended to be a permanent solution. The robustness and long-term viability of the AWRM database is a concern since it was not designed to be an enterprise-level application suitable for a large number of users, and there is no in-house expertise to maintain the software. Also, since the database resides on a local DGAEPM server, direct access for users external to the DGAEPM, such as 1 CAD and other non-National Capital Region personnel, is extremely slow. 1 CAD addresses this by maintaining a separate website database.³⁹ As a result, there are several discrepancies between the quantities and status of RARMs within the two sites. A new AWRM application would address these issues and could also improve the management of airworthiness risk by automating notifications, assigning tasks, tracking more metrics and generating fleet management and performance analysis reports.

Finding A13: The airworthiness risk management process is effective but lacks measures to efficiently monitor and improve its performance.

The need for an airworthiness risk management process can be demonstrated by the fact that the 1 CAD database holds a total of 903 RARMs initiated between 1998 and 2014. The effectiveness of the process has resulted in the vast majority of these risks being eliminated or mitigated to ALOS. About 77 percent have been closed, 5 percent have been cancelled or incorporated into a newer RARM, and 18 percent are still open. Half of these open risks are already at ALOS but are still being monitored. Overall, only about 9 percent of the database constitutes open risks that are non-ALOS. As a result, in October 2014 there were 83 open, non-ALOS RARMs.⁴⁰

While the need and general effectiveness of the airworthiness risk management process is clear, improvements in effectiveness are much more difficult to assess due to a lack of performance metrics and annual airworthiness risk management reports. There are very few metrics to assess and monitor the entire AWRM database and how well each fleet manages its risks. While the AWRM database “Dashboard” tab does provide a few measures, such as overdue activities and improperly closed RARMs, these metrics are not regularly reported or tracked. Other basic metrics, such as the yearly number of open RARMs, the average time for approving and closing RARMs, the average extent of schedule delays, or the percentage of trained staff, are not measured. Tracking and reporting of such metrics for each fleet would allow senior airworthiness managers to periodically compare the performance of airworthiness risk management within each fleet and the RCAF as a whole and flag potential issues for improvement and management action. While such metrics were proposed by DTAES in 2011, the Fall 2013 Executive ARB Report⁴¹ confirmed that the DTAES project to implement performance metrics to illustrate airworthiness risk by fleet has been held back due to “capacity limitations.” As a result, the RCAF currently does not track and report airworthiness risk management performance measures and trends. It is, however, possible to estimate a few measures of performance using the AWRM database. For example, the RARM open and close dates can be used to tabulate the yearly number of open and closed RARMs, as well as the yearly average RARM completion times. This exercise was performed using the RARM metadata entered in the database. The results are shown in Figures A-1 and A-2. Similar charts examining the data pertaining to individual fleets might also provide useful information for fleet airworthiness authorities.

Figure A-1 indicates that the total number of open RARMs increased until peaking in the 2008-2010 time period and has been generally decreasing since then. The increase in open RARMs was primarily due to several issues with the aging in-service fleets and the fact that, as shown on the chart, few RARMs were being closed until 2007. The decreasing number of open RARMs since 2010 suggests that airworthiness risk management effectiveness has improved since 2007, resulting in the resolution of many airworthiness risks. However, the drop in the number of closed RARMs since 2011 suggests further improvements might be needed to address the remaining open RARMs. The open RARMs include about 80 ALOS in addition to the 83 non-ALOS RARMs mentioned previously. While it would have been interesting to chart the cumulative yearly total of open non-ALOS RARMs, the database metadata unfortunately does not record when the risk index status first becomes non-ALOS.

Incomplete database fields, especially the lack of an “Open” date, meant that only 274 out of 528 closed RARMs could readily be used for estimation of the completion times. Nevertheless, based on that sample size, Figure A-2 indicates that the average RARM completion time has generally decreased since 2007, and that the cumulative average of about 2.25 years has remained fairly consistent since 2010. This again suggests that airworthiness risk management effectiveness has improved since 2007. In order to complete the picture, another useful airworthiness risk management performance metric would be to determine how long current non-ALOS RARMs have been open. As mentioned earlier, a review of the existing open RARMs indicates that there are actually only about 83 open non-ALOS RARMs. Figure A-3 provides a breakdown of these by year of origin and risk level.

As one would expect, the majority of the open RARMs are recent, having originated since 2008. However, there are a number of older RARMs that appear to warrant increased monitoring and attention. Involvement by senior management and additional resources may be necessary to resolve these. Unresolvable or “permanent” RARMs might potentially be closed by amending the fleet Airworthiness Clearance.

In summary, three areas that warrant additional review and attention include the following:

• inconsistencies in risk index status and incomplete data fields in the AWRM database;
• non-ALOS RARMs that have been open for several years; and
• the reduction in numbers of RARMs being closed in recent years to determine whether further process improvements are necessary to address the remaining open RARMs, about half of which are assessed to be at ALOS.

Finding A14: The consolidated impact of multiple fleet risks is difficult to assess and is not reported.

Most fleets currently have several open RARMs of varying hazard severities and probabilities, so there are legitimate concerns regarding the overall impact of multiple fleet risks. Inherently, a fleet with several risks is more hazardous than a fleet with just one or two risks of comparable severity. Combining multiple fleet risks of equal hazard severity is feasible and the resulting combined probability will be higher than that of the individual risks, such that the combined risk level for a fleet could potentially be higher than that of its individual risks.^{42 43} Unfortunately, there is no practical way to combine fleet risks of different hazard severities, and to do so would probably be misleading. For this reason, the consolidated impact of fleet risks has never been calculated or reported. However, given the small number of non-ALOS RARMs for each fleet, a basic but practical alternative would consist in simply listing and/or tallying the numbers of high, medium and low risks. This would allow senior airworthiness authorities to monitor and compare fleets on an annual basis and, where necessary, invest additional resources to assist fleets with higher numbers of risks.

Finding A15: There is no policy guidance or incentive limiting the routine, peacetime use of aircraft with medium or higher residual risk levels.

As the OAM⁴⁴ indicates, RARMs above ALOS must be accepted by a suitable senior OCRAA in order for the aircraft to continue operating.⁴⁵ Much work is done to develop risk control plans that will mitigate the risks as much as possible. Additionally, guidance on the suitability of missions for fleets with residual risks is provided in the OAM⁴⁶ and often within the RARM itself. As a result, RARMs are nearly always accepted and the RCAF currently operates several fleets with designated medium risks and some with high risks. The OCRAA role is considered to be a command function where the authority for a specific mission accepts the airworthiness risk after ensuring that the importance of the mission outweighs the risk. While such command decisions may be justified, they could also eventually undermine the AWP and should therefore be subject to independent review, such as by the independent fleet review board recommended under Indicator 2. More fundamentally, the current AA and OAA designation orders do not mention the OCRAA role or any associated risk acceptance responsibility. Introducing personal risk acceptance responsibility and accountability, within the text of the designation orders and within the acceptance text of each RARM, would emphasize the importance of the OCRAA role and clearly remind authorities of their risk acceptance responsibility. This should serve to emphasize the significance of risk acceptance decisions, especially in the case of peacetime operations. Of note, annual air safety risk reviews and legal personal accountability for the safe operation of aircraft on the part of risk acceptance authorities are two of the key changes made to the UK Ministry of Defence’s airworthiness program in 2010, as a result of the Hadden-Cave review of the Nimrod accident.⁴⁷ The new UK airworthiness program is briefly described at Annex E.

Indicator 5: Documentation is relevant, complete and updated regularly.

Finding A16: The AA airworthiness documentation is usable, but needs to be completed and regularly updated.

The AA airworthiness documentation consists of two Defence Administrative Orders and Directives (DAOD),^{48 49} an AWP policy manual,⁵⁰ and an associated Air Force Order on the RTS process.⁵¹

The DAODs are very brief documents essentially introducing the DND/CAF AWP and its key concepts and linking it to the Aeronautics Act and other key references.

The DND/CAF AWP policy manual was first published in 2006 and last updated in 2011. The manual expands on the DAODs and is a strategic policy and management document describing the AWP governance mechanisms, airworthiness authorities and their responsibilities, airworthiness safety criteria and key airworthiness processes, including risk management, airworthiness clearances, flight tests and foreign aircraft operations. While most of the manual is complete and usable, it does have some shortcomings. The section describing Aviation Life Support Equipment (ALSE) is vacant, indicating that it is “to be promulgated.” This is an important omission since ALSE includes survival equipment that cannot be assessed using the same airworthiness criteria as other aircraft components, especially regarding safety thresholds and risk. Another issue is the section on Medical Standards. While the other parts of the AWP manual are written at a strategic level, the section on Medical Standards is written in significant detail and takes up 33 pages of the entire 110 page manual. While the information may be correct, it would be simpler and more consistent with the strategic nature of the document and the rest of the manual to simply reference the appropriate tactical level medical publications. This would also avoid having to amend the strategic manual to keep pace with tactical changes in medical standards or conditions. The sections on ALSE and Medical Standards were extensively revised in 2012, but that revision has yet to be officially completed and promulgated.

In addition, within the AWP Manual, the quantitative safety objectives⁵² are inconsistent with the TAM⁵³ and OAM⁵⁴ due to the fact that nearly all of the “less than” (<) signs on the figure are backwards or missing. The AWP Manual also seems to fall a bit short by including only a very cursory note on the RTS process. Even though RTS involves other aspects besides airworthiness, it is recommended to add a brief description of RTS in the AWP Manual since it is essential for the operation of CAF aircraft and does require the completion of airworthiness clearances as prerequisites.

The Air Force Order⁵⁵ is another AA document that has not been updated for several years. As a result, it omits to include the Investigative Airworthiness Clearance as a prerequisite for a full Airworthiness Clearance and RTS.

A general issue underlying all of the AA documents is that they lack a formal periodic review process with appropriate stakeholder input to ensure the content of the document is up-to-date. A formal review process with a fixed update cycle would ensure the documents are regularly improved and updated, thereby increasing their accuracy and relevance.

Indicator 6: Training is adequate to meet the needs of the program.

Finding A17: In general, the RCAF implementer community needs to improve their understanding of the DND/CAF AWP.

Interviews indicate that the overall level of understanding of the AWP is often poor. In field units, in project offices and sometimes even within DGAEPM, it was apparent that some personnel, sometimes in leadership positions, did not have a good understanding of the AWP. On the operational side, from a practical point of view, an aircraft may have restrictions that are the result of mitigation initiatives stemming from an identified airworthiness risk. Squadron personnel clearly understood the restriction but did not realize where it came from and why it was imposed. Improved understanding of the various components of the AWP would contribute to enhance aircrews’ awareness and airmanship. Some foreign MAAs, such as the Australian Defense Forces, have implemented minimum training requirements, on-line familiarization courses and quick reference handbooks to acquaint their personnel with the AWP and its key principles, processes and points of contact. Similarly, while information on allied airworthiness programs can be found relatively easily, there is a lack of publicly available information on the DND/CAF AWP beyond the TA portion of the program.⁵⁶ Likewise, a Defence intranet search leads to divisional airworthiness information vice departmental-level information.

Finding A18: A lack of governance oversight has led to a bottom-up approach with regards to training, resulting in training disparities and challenges.

In evaluating the AWP, the evaluation found a bottom-up approach towards training evolving out of necessity for both the OA and TA programs. These commendable efforts have led to some level of training being delivered with success. That said, no appropriate training oversight or identified training needs/objectives have been enunciated to date. With training requirements growing, this has put undue pressure on the existing resources that are presently managing and delivering the current semi-formalized airworthiness training. There is a growing need to oversee and manage the airworthiness training from a top-down approach to ensure staff within the airworthiness authorities and, to a lesser extent, across the DND/CAF are appropriately qualified, and that current and future needs are appropriately addressed.

Other training findings and recommendations specifically applicable to the TA, OA and IA programs have been mentioned in the annexes pertaining to those programs (annexes B, C and D).

Indicator 7: The PMF is well defined, functional and appropriately managed.

Finding A19: The AWP does not have a strategic-level PMF.

According to the TBS Directive on the Evaluation Function,⁵⁷ program managers are responsible for developing and implementing performance measurement strategies for their programs. As previously discussed under Scope of Authority and Organizational Structure, while the AA has been delegated the management of the program, the monitoring function has been delegated to the AIA. However, the evaluation did not find any evidence of strategic-level monitoring or performance measures to assess and manage the overall program and its improvements. While progress has been made in some aspects of the AWP, many improvements are largely bottom-up developments or recommendations from external evaluations that would benefit from more strategic guidance or involvement. The annual AAB is an opportunity for management to track some initiatives, but this only occurs once a year and is therefore an insufficient means to actively monitor and manage the program. Implementation of a PMF to monitor and assess the AWP would provide many of the tools necessary for senior airworthiness authorities to manage and improve the program. Suggested performance measures would include monitoring and reporting on the performance of the AWP, completeness and adequacy of airworthiness training among regulatory and implementation staff, performance of the fleet risk management process, airworthiness-related expenditures, resource requirements, the TA and OA audit programs and progress on AWP initiatives and improvements.

Indicator 8: Resources are adequate to meet the needs of the program.

Finding A20: The governance construct of the AWP lacks the resources to effectively monitor and manage the AWP.

As confirmed by interviews, the AA does not currently have any staff specifically dedicated to the AWP. As a result, the AA has a very limited ability to effectively monitor, manage and improve the AWP or to address its current and future issues. Current staff work only on airworthiness when necessary and struggle to simply update the airworthiness policy documents for which they are responsible. The postponement of the 2014 AAB and the lengthy delays in publishing past AAB minutes are other examples of limited AA resources struggling to address competing priorities. As a result, while the TAA, AIA, OAA and their senior delegates have established a productive working relationship, the AA is not fully apprised of program challenges. The implementation of an AA support cell to more effectively monitor and manage the AWP is further discussed and recommended under Indicator 2 (Organizational Structure).

Annex B—Technical Airworthiness Effectiveness

Following the 1998 MND direction to formally implement a more robust airworthiness program for military aviation, the TAA revamped its program to align with accepted worldwide principles. As such, the DND TAP follows the principle of authorized individuals working in accredited organizations following approved procedures and accepted standards. The TAP thus regulates and ensures the implementation of the technical aspects of airworthiness.

Finding B1: The TAA has extensively reviewed and improved the TAP by implementing many recommendations arising from a comprehensive independent evaluation.

In 2011, an independent review of the TAP was conducted by Mr. Ken Mansfield, a recognized aviation safety and security consultant and former Director of Aircraft Certification at TC. The resulting 106-page report, hereafter referred to as the Mansfield Report, provided a strong endorsement of the TAP and its efficacy but also presented 22 key recommendations for consideration, as well as other observations not completely captured in the recommendations. ADM(RS) found the report to be still highly pertinent and endorses all of the report’s observations and recommendations. DTAES endorsed the findings of the report and promptly addressed 14 of the recommendations; however, completion of the remaining recommendations has been slow or delayed due to resource constraints. Table B-1, located at the end of this annex, lists the Mansfield Report recommendations and their status as provided by DTAES in 2013.⁵⁸ This evaluation will not duplicate the work of the Mansfield Report, but will instead highlight some of the recommendations and other areas of the TAP that seem to need additional attention.

Indicator 1: The scope of authority is appropriate, well documented and exercised in accordance with policy and directives.

Finding B2: The TAA has a clearly defined scope of authority.

The TAA has a clearly established scope of authority for regulation of the technical aspects of airworthiness as defined in the CDS designation letter⁵⁹ and AWP policy manual.⁶⁰ This includes regulating the design, manufacture and maintenance of aeronautical products and determining their airworthiness acceptability for DND use. The CDS normally designates the DGAEPM as the TAA, although this authority can alternatively be assigned to DTAES if the current DGAEPM has a limited background in airworthiness.

Indicator 2: The organizational structure is adequate to meet the scope of authority.

Finding B3: The TAA has developed and embodied in its program the structure and instruments necessary to successfully manage its regulatory functions and implementation of the TAP.

The TAP has worked well based on expert resident knowledge, support from senior management, adequate resources, solid customer audit and risk management programs and formal and controlled documentation for both its processes and technical orders. However, it is largely a bottom-up program that is balanced with regular management involvement and oversight to address significant issues and strategic long-term improvements to the program, such as through the conduct of periodic internal and external evaluations, such as the Mansfield Report, previous CRS evaluations and audits and comprehensive internal program reviews. Regular oversight is provided by two key DGAEPM management committees that have been established to periodically review airworthiness issues and processes. The DTAES Technical Airworthiness Regulatory Committee (TARC)⁶¹ meets quarterly to oversee the TAP and its rules and standards, and it recommends approval to the TAA for changes to the TAM⁶² and other publications. On the implementation side, a similar DGAEPM Airworthiness Control Committee (ACC) involving all of the fleet SDEs also meets on a quarterly basis to review airworthiness implementation issues and recommend changes to processes. DGAEPM staff review and audit all processes and procedures on fixed cycles and they forward recommendations to significantly amend airworthiness procedures to the ACC for review and endorsement. Both the TARC and ACC provide an important oversight and communication role and mechanism to exchange best practices and continuously review and improve the TAP. These committees must continue to provide a discussion forum and challenge function for issues brought forward and avoid becoming a repository of status updates from different sub-programs and initiatives. Issues requiring TAA attention are brought up either directly to DGAEPM (or DTAES as applicable) at regular DGAEPM Board of Directors meetings, and/or at the Executive ARB. Through all of these forums, the DGAEPM and the TAP are examples of a DND organization and program striving for continuous improvement.

Finding B4: The independence level of the TAA is considered adequate. However, competing resource pressures within DGAEPM have impacted progress of the TAP.

As discussed in Annex A, one of the principles of an airworthiness program is an appropriate level of independence between the regulators and the implementers in order to avoid any conflict of interest. Within DGAEPM, DTAES 3, 4 and 5 clearly perform solely regulatory functions, whereas the rest of DGAEPM, along with all the RCAF field units and contractors, implement the TAP. While both the TAA and DTAES are double-hatted, their engineering responsibilities complement their airworthiness responsibilities and, as part of Assistant Deputy Minister (Materiel) (ADM(Mat)), the entire DGAEPM organization is independent of the RCAF operational chain of command. For this reason, while complete regulatory independence would be ideal, the evaluation generally agrees with the findings of the Mansfield Report, which concluded that the TAP operates sufficiently independently within the current DND/CAF organizational structure. That said, the current fiscal pressure within DGAEPM does elicit some concern as demonstrated in the DGAEPM Fiscal Year (FY) 2015/16 Business Plan,⁶³ regulator resource demands to maintain and improve the regulatory functions and to progress new initiatives such as improved training and mutual recognition of allied MAAs are directly competing against implementer resource demands and force generation activities. It is important that decisions or priorities that impact airworthiness activities and initiatives be assessed in terms of risk to the program and briefed to senior airworthiness authorities for their awareness and potential guidance or resolution.

Indicator 3: Processes are functional and well documented.

Finding B5: The TAA has a rigorous, well-documented airworthiness certification and clearance process that reflects accepted worldwide airworthiness practices.

In order to support AA approval of the RTS of new aircraft systems, the TAA ensures its TA through the progressive issuance of MTCs to approve their type designs, Certificates of Airworthiness to certify the resulting aircraft products and finally TACs to confirm that all the elements of the TAP required to introduce and support a new aircraft into service have been implemented. Occasionally, issues arise when new aircraft systems do not fully comply with all of the TAM requirements. In those cases, project staff must work closely with the regulator and aircraft manufacturer to satisfactorily resolve the issues. This process has developed and matured over the last 15 years with the acquisition of several new aircraft. The process is complex and very rigorous, but essentially it is the same as that employed by other military and civilian airworthiness authorities (see benchmark analysis at Annex E). The certification and airworthiness clearance processes work well, but require close coordination between the TAA regulatory staff, the Project Office staff and the aircraft manufacturer.

Finding B6: Within DTAES, the TA responsibilities of the regulators are well understood; however, outside DTAES, the implementers’ understanding of the regulators’ roles and specific functions could be improved.

Communication is identified as a key initiative within the DTAES Strategic Plan,⁶⁴ and DTAES tracks the progress of its communication plan via its Balanced Scorecard.⁶⁵ Notwithstanding a communication plan to better inform the DGAEPM community of its roles and functions, interviews revealed that some implementers still did not fully understand the DTAES regulator-implementer relationship. This was particularly evident in project offices but also in certain directorates, where misunderstandings and expectations regarding the mutual responsibilities of the regulators, engineering support staff and project staff had often been a source of confusion, frustration and delays. Another related source of confusion for many implementers is the distinction between the regulatory functions of DTAES 3, 4 and5 vice the engineering support functions of DTAES 6, 7 and 8. This confusion is further exacerbated by some DTAES engineering support staff occasionally being called upon to perform specialist regulatory functions, and other engineering support staff reportedly acting as regulators. For these reasons, additional communication measures should be implemented for each specific project in order to clarify and clearly establish the responsibilities and work required of each participant, including the DTAES regulators, Weapons System Manager (WSM) or Project Office staff, DTAES engineering support staff and other key stakeholders. In addition, identifying the right individual within the regulator organization to be the focal point of contact for the project is key to ensuring successful synergy. This delineation of roles and responsibilities should be performed for all projects or tasks where DTAES involvement is required.

Finding B7: The TAA has a good continuing airworthiness process that includes an annual fleet review process.

Continuing airworthiness involves ensuring an aircraft system remains airworthy throughout its service life. As stated in the TAM, continuing airworthiness involves conducting approved maintenance, certification of any design changes, configuration management and monitoring of the usage of the product to ensure it remains within its intended certified usage. By its nature, continuing airworthiness applies to in-service aircraft and is therefore primarily the responsibility of the DGAEPM WSMs and fleet SDE, along with DND and contractor maintenance and support organizations. The qualifications and requirements for the fleet SDE and associated Authorized Individuals are well defined within the TAM as they are key roles within the fleet implementer community providing qualified individuals with delegated airworthiness authority to approve some design and maintenance changes or deviations. DTAES regulator staff and the TAA maintain oversight of such approvals through the AAR and ARB process. The use of SDEs and authorized individuals represents an efficient use of TAA resources by appropriately leveraging senior implementer experience for functions and approvals for which they are normally already well trained. This alleviates the TAA and DTAES regulatory workload, which in turn permits DTAES to focus its limited regulatory resources primarily on initial aircraft certification and regulatory assurance functions such as audits and AAR reviews.

The yearly ARB has proven to be a key mechanism to ensure an appropriate level of review of the continuing airworthiness of each fleet. The process for maintaining the Airworthiness Clearance is well documented in the TAM, and the supporting Engineering Process Manual (EPM) and AF9000 Plus documentation provide clear procedures defining the activities and validation process that must be undertaken by the WSMs and SDEs to renew the Airworthiness Clearance. A key supporting activity is the production of the Technical AAR by the fleet SDE for review and comment by DTAES regulatory staff and the TAA several weeks prior to the ARB. The Technical AAR essentially summarizes all the recent and ongoing airworthiness issues for a particular fleet over the annual reporting period, including design changes, maintenance deviations, RARMs that are raised, closed or still open, new restrictions and flight manual changes, flight permits issued and other pertinent fleet issues. In support of the ARB, DTAES also produces a Technical Airworthiness Oversight Requirements Presentation that indicates the accreditation status and issues of each fleet WSM, maintenance organization and its support network that could potentially impact fleet airworthiness.

Indicator 4: Risk management is well defined, functional and appropriately managed.

Finding B8: The TAA has a well-defined process for quantifying and approving airworthiness risk assessments and control plans.

The technical aspects of the airworthiness risk management process are comprehensive and well described in the TAM⁶⁶ and in associated procedures.⁶⁷ A key aspect of TA risk management is the documentation of airworthiness risks as RARMs and inclusion in the AWRM database, which is currently hosted on a DGAEPM server. While most RARMs are initiated by WSM or AEO staff within DGAEPM, they can also be initiated by project offices, contracted maintenance organizations, and Divisional HQ staff. RARMs must then be approved by the TAA or a delegated airworthiness representative such as an SDE or Authorized Individual, before being forwarded to the OAA staff for approval and acceptance. The TA risk management process itself is rigorous and well-monitored through AWRM database tracking and annual ARB reviews. That said, a number of findings and recommendations regarding the DND airworthiness risk management process have been made in Annex A.

Indicator 5: Documentation is relevant, complete and updated regularly.

Finding B9: The TAA has comprehensive documentation that has evolved considerably in recent years, but additional effort is required to complete TAA regulatory documents.

The TAA has established key regulatory documents and internal procedures that are extensive, understood and accepted. The primary regulatory document, the TAM, provides the rules, standards and advisory material for the implementation of the TAP. The TAM is an excellent, comprehensive and very readable document and is essential to ensure a common understanding of the requirements for technical airworthiness certification and continuing airworthiness. It has evolved considerably since first published in 2001 and was significantly updated in 2012; another update is reportedly underway.⁶⁸ That said, it has been updated only once in the past seven years, and several TAM annexes have yet to be written. Most of these annexes are intended to detail the continuing airworthiness in-service monitoring requirements of aircraft systems. These would include aging assessment requirements considered especially important for the DND/CAF given the age of several of its aircraft fleets and the DND/CAF’s recurring need to extend service lives well beyond that of the original design. These annexes are expected to be part of the next update scheduled for release in fall 2015.

In order to ensure TA requirements are consistently implemented throughout the RCAF and by DND/CAF contractors, all design, maintenance and engineering organizations supporting DND/CAF aircraft are in turn required to publish and implement an Airworthiness Process Manual that describes how their organization complies with the regulatory TAM. This includes DGAEPM itself, whose airworthiness processes are described in the Aerospace Equipment Program Management EPM and further amplified in its on-line AF9000+ Manual of Aerospace Procedures⁶⁹ and TAA-approved EPM Supplements for each aircraft fleet. While the EPM Supplements and the Manual of Aerospace Procedures are formally reviewed at regular, fixed intervals (18 months for the EPM Supplements and typically three to five years for the Manual of Aerospace Procedures), the Aerospace Equipment Program Management EPM has not been updated since its original publication in 2008. The evaluation team was advised that a significant update was underway, and it was expected to be published in 2015.

Besides the TAM, two other TAA regulatory documents are being developed: the Airworthiness Design Standards Manual (ADSM) and the Aircraft Equipment Requirements (AER). The ADSM and AER complement the TAM by providing guidance to project staff and other implementers on standards and requirements applicable to aircraft designs and acceptable to the TAA. Like the TAM, these documents require a very high degree of airworthiness knowledge. While the development of both documents has been underway for many years, limited staff resources and other priorities within DTAES have delayed their completion. An ADSM has been published with many chapters remaining to be promulgated, whereas the AER has been reportedly drafted and is under review.⁷⁰ Finalization of both of these documents would improve program effectiveness and likely offset some of DTAES’ resource demands by assisting implementers to improve the quality of their certification plans. This need is all the more pressing since it is anticipated that there will be a significant reduction of airworthiness experience and technical knowledge within the organization during the next few years with the retirement of many senior SMEs.

Indicator 6: Training is adequate to meet the needs of the program.

Finding B10: There is a pressing need for the TAA to identify and implement minimum airworthiness training requirements for all personnel involved with the TAP.

Personnel involved with the technical airworthiness process need extensive knowledge and training. DTAES regulators ensure that implementers with airworthiness responsibilities, such as SDEs who work within accredited organizations, meet airworthiness standards defined in the TAM for skills, knowledge and experience prior to being authorized for airworthiness responsibilities. However, the evaluation identified a lack of standardized training and a need to formalize required qualifications especially within the regulatory staff. Some foreign MAAs and TC clearly mandate specific airworthiness courses and qualifications before specific airworthiness functions can be carried out by individuals. This need was also identified in the Mansfield Report with a series of recommendations (see Table B-1, items 5, 6, 7, 8 and 9). Presently, a cadre of very senior personnel with extensive experience and knowledge has provided expertise to perform the specific airworthiness activities required by the TAP. Since many of these are expected to retire within the next few years, the need to adequately train future replacements is pressing. As mentioned under Indicator 3, there is also a need to educate project staff (and other implementation staff working on design or maintenance changes) regarding airworthiness and regulator roles and functions.

Finding B11: Although a training plan exists, the TAA does not have sufficient resources to properly implement identified airworthiness training requirements.

The TAA has recognized the need to enhance airworthiness training for both the technical regulators and implementers. Consequently DTAES 2 carried out a training needs analysis⁷¹ to assess the number of courses required, the target audience and the course content. This led to the identification of 22 courses covering 8 fields of expertise required by 1622 people/positions. DTAES 2 identified a $1 million start-up cost and a total of $4.5 million requirement over 10 years to implement all the courses. However, a Training Development Officer and SMEs will be needed to develop and deliver the courses. In the interim, DTAES has made some progress by using contracted personnel with airworthiness expertise. Eight of the 22 courses have been developed, and some training has been delivered. However, there is a need to ensure the courses meet DND/CAF standards, and that they are properly developed, administered and delivered. Proper training administration, development of training plans, course content and qualification standards would ensure required airworthiness training is verified and appropriate. Unfortunately, resource constraints at the ADM(Mat)/DGAEPM level have created a competition between the regulator and implementer communities for securing the funds and resources necessary to fully carry out their programs and initiatives. This is considered a major obstacle in the allocation of the necessary resources to ensure that airworthiness training requirements are addressed with a sustainable solution. The requirement to properly train airworthiness personnel and the impact of delaying the implementation of adequate training due to lack of resources require management and oversight at both the TAA and AA level.

Indicator 7: The PMF is well defined, functional and appropriately managed.

Finding B12: The TAA has developed a strong audit program to monitor the implementation of the TAP.

An essential component of an airworthiness program involves audits to ensure proper standards and processes are being followed. Within the DND/CAF AWP, engineering and maintenance organizations, including original equipment manufacturers and WSMs, must be accredited by the TAA as meeting the TAM requirements considered necessary to safely design and support CAF aircraft. Accreditation and subsequent audits provide assurance to the TAA that airworthiness processes are being adhered to.

The evaluation reviewed sample audit reports and interviewed some DTAES auditors and audited organizations and determined that the audit function works well and provides the necessary oversight of the technical implementers. Civilian contractors recently audited by DTAES have supported the DND accreditation process and favorably compared it to Transport Canada Civil Aviation (TCCA) audits.⁷² An annual summary of the accreditation status and audit issues involving each fleet WSM, maintenance organization and its support network is summarized in the Technical Airworthiness Oversight Requirement Matrix provided as part of the annual Technical AAR submitted to the AAB.

Finding B13: The TAP has developed a sound performance measurement tool to monitor regulatory aspects of the TAP. However, performance measures to monitor and assess the implementation of the TAP are lacking.

It should be noted that the 1 CAD/A4 Maintenance section also has an audit function auditing squadron maintenance organizations. The audits being performed are very similar to those performed by DTAES; however, their purpose is to both validate compliance of maintenance activities and inform Comd 1 CAD of the health of the maintenance program. To avoid duplication, the periodicity of DTAES audits of RCAF maintenance organizations is adjusted to take into account the A4 Maintenance Audit program and its findings.

As previously discussed, DGAEPM strives to improve the TAP and has benefitted tremendously by applying recommendations from past external and internal evaluations. Prior to the 2011 Mansfield Report, the AWP had been the subject of a CRS Airworthiness Audit in 1993, and a CRS Review of Airworthiness Risk Management in 2008. The TAA also conducted an internal review⁷³ of the UK Nimrod accident report. Other recommendations for regulatory improvements have been captured as strategic initiatives within the 2012 DTAES Strategic Plan. Ongoing progression of those initiatives is monitored via the DTAES Dashboard.

While such assessments are useful, TBS policy requires each department to implement performance measures to monitor and improve program effectiveness. A good example of this is the DTAES Balanced Scorecard, which provides a quarterly assessment of DTAES client feedback, employee development, process improvements and resource management. The scorecard is a DTAES management tool, but because DTAES is the regulatory arm of the TAP, it does cover several activities impacting the entire TAP, including accreditations, client feedback from accreditation and certification activities, the status of regulatory resources and the progress of strategic initiatives to improve the TAP. While it is an excellent tool for TAP regulatory aspects, the evaluation found nothing similar to assess the implementation aspects of the TAP, such as performance measures to monitor the adequacy of airworthiness training among the implementation and project staff, fleet risk management metrics, airworthiness expenditures, resource requirements and process improvements. As discussed in Annex A, while there was an initiative to implement some airworthiness risk management metrics, this has been on hold since 2011 due to resource limitations.

Indicator 8: Resources are adequate to meet the needs of the program.

Finding B14: DTAES is adequately structured and staffed to perform its normal regulatory functions, but it has limited capacity to implement initiatives to improve the program.

The few specialist personnel increases, which were recommended by the Mansfield Report, are being effected. The only other steady-state shortcoming that the evaluation identified is the lack of administrative support for the AWRM database, a critical risk management application introduced without any staff resources to sustain it. Since there is no redundancy in many of the regulatory specialist positions, any reduction of resources must be avoided. Presently, DTAES relies heavily on a core of very senior section heads and specialists with extensive knowledge. Careful succession planning will be critical to ensure suitable replacement staff and minimal impact to the program. An important enabler for the regulatory program is the use of contracted support to supplement current personnel when additional specialist expertise is required. DTAES section heads indicate that they could not accomplish their current mandate without this resource. While suitable for surge SME requirements, contractors are costly and not always readily available, and they cannot be relied upon to provide the core expertise or DND regulatory authority.

Additionally, there are many initiatives to improve the TAP, but these have been slow to progress due to resource constraints. For example, the strategic initiative to recognize the airworthiness certification of some foreign MAAs could significantly improve program efficiency by reducing the resources and level of effort required to certify aircraft systems already accepted by a recognized authority. Similarly, recognition of civilian organizations possessing TC airworthiness authorization would yield economies of effort. Other initiatives discussed elsewhere in this evaluation report include development of training courses, completion of publications (TAM, ADSM and AER), implementation of a more accessible risk management database and performance measurement. All of these improvements would result in more efficient use of regulator and implementer resources. While many of these initiatives are identified in the DTAES Strategic Plan and the Balanced Scorecards, they all require substantial one-time efforts to implement, and progress has therefore been slow. Better oversight and endorsement by the TAA may be necessary to ensure adequate resource allocation and timely completion. Given limited staff resources, it may be worthwhile to set up a project team to implement each of these in turn, based on an evaluation of their cost and level of effort versus their benefits and risks. Progress of initiatives and impacts of continued delays should be reported at the Executive ARB and AAB until completion.

Table B-1 shows the Mansfield Report recommendations and their status as provided by DTAES in 2013.

Table B-1. Mansfield Report Recommendations. This table lists the recommendations of the Mansfield Report and the status of the resulting TAA actions.

Annex C—Operational Airworthiness Effectiveness

In support of the DND/CAF AWP, the outputs and immediate outcome of the OA program is to promulgate DND regulations, standards and competencies for flying operations and ensure the operations conform to regulations and standards. The OA program thus regulates and ensures the implementation of the operational aspects of airworthiness.

Indicator 1: The scope of authority is appropriate, well documented and exercised in accordance with policy and directives.

Finding C1: The OAA has a clearly defined scope of authority to meet its OA responsibilities.

The CDS has designated the Comd 1 CAD as the OAA.⁹³ The OAA has a clearly established scope of authority to regulate and oversee operations for all military aeronautic activities within the Canadian airspace and for all Canadian military aeronautic activities around the world. This includes regulating and ensuring safe and effective operations of all RCAF aircraft fleets, simulators, aerodromes and airspace and ranges, as well as all CA and RCN aeronautical activities (including shipborne air operations), all transiting and/or foreign military aircraft activities within the Canadian airspace, all civilian-owned and DND-operated aircraft and all DND-contracted aircraft operating on a Canadian Aviation Regulations (CARs) 701-705 exemption.⁹⁴

Indicator 2: The organizational structure is adequate to meet the scope of authority.

Finding C2: The majority of positions with key OA responsibilities are dual-hatted. The primary focus on operations risks compromising regulatory independence.

The OA program relies heavily upon existing operational staff within 1 CAD and 2 CAD who are dual-hatted with OA responsibilities to provide the necessary airworthiness oversight of aviation operations. Day-to-day airworthiness-related activities are primarily conducted by Senior Staff Officers (SSO) and their staff, who are responsible for specific fleets or areas of activity. SSOs complete operational airworthiness documents, such as OA Clearances and operational aspects of RARMs, and they provide input to regulatory documents, such as the RCAF Flying Operations Manual.

The SSOs report to three Divisional Operational Airworthiness Managers (DOAM): one responsible for airspace/aerodrome activities, one overseeing the 1 CAD aircraft fleets and one overseeing the 2 CAD fleets. In turn, as illustrated in Figure C-1, the DOAMs report to the Senior Operational Airworthiness Manager (SOAM), who is dual-hatted as the 1 CAD Deputy Commander Force Generation. The SOAM reports to the OAA, himself double-hatted as Comd 1 CAD. In addition, a dedicated four-person OA staff, headed by the SSO OA, reports directly to the SOAM. This small cell is charged with the day-to-day management of the OA program including oversight of all RCAF fleet airworthiness activities and Operational Test and Evaluation (OT&E) projects, monitoring civilian companies conducting contract work for DND and foreign military overflights, completing and promulgating OA regulations and ensuring OA policy, SOPs and training are instituted within the broader divisional activities.

The dual-hatting of the SSOs and senior OA authorities, up to and including the OAA, is clearly an efficient and economical method that takes advantage of existing staff resources and existing operational communication channels to ensure both operational oversight and SME input to the regulatory documents. As explained at Annex E, similar double-hatting of OA authorities and operational command chains is common in many allied countries, including the USA, the UK and Australia. However, the DND/CAF is distinct in lacking an OA regulatory authority independent from the OAA and the operational chain of command. As explained in Annex A, one of the tenets of an airworthiness program is an appropriate level of independence between the regulators and the implementers in order to avoid any potential conflict of interest. The small SSO OA cell is competent but cannot act as a strong independent regulatory authority since it reports to the operational command chain. Allied countries address this issue by either including the Operational Airworthiness Regulator (OAR) as part of their independent MAA (e.g., the UK Military Aviation Authority) or by organizationally separating the OA regulatory authority from the operational chain of command, such as in the USA. Australia, which has a generally comparable airworthiness system, has assigned its Deputy Chief of Air Force as the OAR, thereby separating this authority from the OAAs and the operational chains of command. The DND/CAF situation is all the more concerning given the lack of independent oversight within both the OA program and the overall AWP.

Another issue, which can be observed from Figure C-1, is that, unlike Comd 1 CAD, Comd 2 CAD is not in the formal OA structure (except as an OCRAA for high risks), even though Comd 2 CAD is operationally responsible for the safe and effective operations of the training fleets. Rather, the OA of 2 CAD aircraft (both contracted and organic to the RCAF) is the responsibility of the 2 CAD DOAM and a sole SSO. Currently, the 2 CAD DOAM’s authority is delegated to the SSO due to the limited airworthiness expertise within 2 CAD. As a result, one person essentially has both the SSO and DOAM responsibilities for the nine 2 CAD aircraft fleets, as well as the oversight of all the 2 CAD SETs. This situation is not ideal as it prevents the important second review of airworthiness documentation and approvals. The lack of dual oversight is mitigated by ensuring that Comd 2 CAD is engaged in any documents requiring DOAM authority. Although this is an acceptable interim strategy, formally designating Comd 2 CAD as a DOAM or alternative SOAM would formally involve Comd 2 CAD within the OA organizational structure and establish the requisite redundancy.

Finding C3: The OA program has performed effectively over the evaluation period but should improve its oversight of UAVs, contracted aviation operators, flight simulators and foreign military aviation operations in Canada.

As the OA program relies heavily upon existing RCAF operational staff within the air divisions to provide the necessary airworthiness oversight of aviation operations, the current organizational structure lends itself well to the airworthiness oversight of RCAF fleets and air operations. However, OAA responsibility is much broader than RCAF activities and encompasses all Canadian military aviation activities and foreign military activities within Canada. Reliance on the pre-existing RCAF-centric organizational structure has resulted in extremely limited airworthiness oversight of military aviation activities in which 1 CAD and 2 CAD have little or no operational involvement.

One such area of activity involves UAV operations. UAV activities in Canada and abroad can be divided into three main categories: RCAF past and future fleets, CA and RCN operations and foreign military UAVs in Canadian airspace. For past RCAF UAV fleets, SSOs were established, enabling OA dual-hatting, and thus airworthiness oversight activities were established similar to that of the other RCAF fleets. A similar arrangement would be anticipated for future RCAF UAV fleets. Unfortunately, this SSO no longer exists, largely because there is presently no RCAF UAV capability. As a result, for UAVs operated by the CA and RCN, OA Clearances and RARMs are reviewed by 1 CAD/SSO Maritime Air and forwarded to the OAA, with available UAV SMEs within the RCN and CA providing operational information and subsequent operational oversight. Foreign military UAVs typically operate under an SPFP approved by the OAA, with the Kingston-based Joint Task Force Headquarters Air-Land Integration Cell currently coordinating and overseeing CA and foreign military UAV operations within Canada. Although UAV clearances and foreign SPFP processes are said to be functioning effectively when requested, the lack of dedicated operational and airworthiness oversight for these aircraft is a concern, especially given the emerging proliferation of UAV applications and UAV operators.

Similarly, as discussed under Indicator 8, there is very limited operational staff to oversee and assure the airworthiness of aviation operations of foreign military aircraft within Canada, the airworthiness of contracted aircraft operating in support of the CA, RCN and RCAF and the certification of DND flight simulators. The lack of operational and airworthiness oversight of these aircraft operations may introduce risk to the regulatory and oversight responsibilities of the OAA and AA.

Indicator 3: Processes are functional and well documented.

Finding C4: The OAA has made significant progress in establishing OA processes within the operational headquarters but several processes need to be reviewed and improved.

In order to support AA approval of the RTS of new aircraft systems, the OAA staff ensure operational airworthiness through the completion of an SOI to identify the missions and usage of a new aircraft type, and conduct OT&Es to assess the operational effectiveness and suitability of the new aircraft systems and operational support elements including maintenance, manuals and training. This culminates in the issuance of an OA Clearance to confirm that all operational elements required to introduce and support a new aircraft into service have been implemented. These processes are described in detail in the OAM,⁹⁵ which was first published in 2010. The entire OA Clearance process has developed and matured over the last 15 years with the acquisition of several new aircraft fleets. Based on these accomplishments and the limited number of operational staff involved, the process is assessed to be effective and efficient, but success requires continued close coordination and communication between the OAA, TAA and Project Office staff.

Once an aircraft fleet enters service, continuing airworthiness activities ensure it remains airworthy throughout its service life. The yearly ARB has proven to be a key mechanism to ensure an appropriate level of review of the continuing operational and technical airworthiness of each fleet. The end-purpose of the ARB is the annual renewal of the Certificate of Airworthiness of each aircraft fleet. A key OA supporting activity is the production of the Operational AAR prepared by the fleet SSO and OA staff. The Operational AAR summarizes all the recent and ongoing operational airworthiness activities and issues for a particular fleet over the annual reporting period.

One area of concern involves SOIs. Initial SOIs are well documented since they are required to support initial aircraft type certification. As stated in the OAM, SOIs are required to be reviewed yearly, since changes to an aircraft’s roles/missions/usage can significantly impact its airworthiness, and thus technical and operational analysis is required to properly assess the impacts on the aircraft, its maintenance and its service life. A good example of this is the change in role of the CC130H from transport to SAR, which altered its operational usage with consequent effects on its fatigue life. Unfortunately, interviews and document reviews have revealed that the importance of SOIs is often not well understood and that many are not annually reviewed and updated as required.

Another OAA responsibility involves oversight of foreign military aircraft activities within Canada. These activities fall into two broad categories: transient overflights and exercise or operational flights. The airworthiness of these activities falls under the authority of both their home nation and the DND/CAF. The OAA has been delegated the responsibility of ensuring that the airworthiness of a foreign aircraft and its crew is to an acceptable standard before operating within Canada. In general, this is occurring with varying degrees of OAA oversight. Global Affairs Canada initiates transient military overflight requests and directs them to the overflight section of 1 CAD for approval. This process has proceeded uneventfully over the evaluation period, and interviews reveal that the process is largely viewed as a political/sovereignty exercise and less as an actual airworthiness certification exercise. Several countries have already been pre-approved for overflights by the OAA based on an assessment of their airworthiness program, as well as other factors.⁹⁶ If the country has not been pre-approved, SSO OA staff conduct an Airworthiness Impact Assessment to assess the risks involved.⁹⁷ Given the limited time, resources and information available, the Airworthiness Impact Assessment is not always rigorously completed; however, any concerns are highlighted up the chain of command.⁹⁸ One problem may be that the information required by TC for states requesting overflights⁹⁹ is much less than that required by the OA staff to complete the Airworthiness Impact Assessment. Unfortunately, there are no metrics regarding foreign military flights to understand the extent of the problem and risks to DND. That said, even if the number of such occasions is limited, the potential risks and consequences should be appropriately documented, and senior staff should be willing to deny flights if key information is inadequate or missing.

The second area of foreign military operations in Canada has been from NORAD¹⁰⁰ and NATO¹⁰¹ exercises.¹⁰² While these exercises involve some high-risk activities such as air combat manoeuvres, the participants have historically been from allied foreign militaries with similar airworthiness programs and standing OA Clearances. However, exercises such as Maple Flag have recently been inviting non-traditional international participants from within Southeast Asia, South America and Eastern Europe. For example, in 2013, Maple Flag 46 included forces from Columbia and Singapore, as well as observers from future potential participants including Chile, India, Oman, Peru, South Africa, South Korea and the Ukraine.¹⁰³ The involvement of countries without a recognized military airworthiness program potentially elevates the corresponding risk of the exercise. The desire for international participation must not outweigh airworthiness safety. Unfortunately, there is not a documented OAA process regarding how to regulate and ensure the airworthiness of such operations. As a workaround, SSO OA staff use the foreign military overflight procedure¹⁰⁴ as a baseline for a more in-depth review. A formal process should be documented to ensure all necessary assessments are completed and approved. In addition, since the OAA is the airworthiness authority for these aircraft and their operations within Canada, careful monitoring and a corresponding investment in assessment and oversight resources should be implemented to ensure compliance with the DND/CAF AWP.

Indicator 4: Risk management is well defined, functional and appropriately managed.

Finding C5: The OAA has a well-defined process for quantifying and, if appropriate, accepting airworthiness risks.

The OA risk management process is well described in the OAM.¹⁰⁵ A key aspect of OA risk management is the documentation of airworthiness risks as RARMs. While most RARMs have been initiated by TA staff, RARMs can also be initiated by the OA staff. As Table C-1 indicates, RARMs are normally approved by both a TA and OA representative and, if appropriate, accepted¹⁰⁶ by an OCRAA. As a result, RARMs that are low risk or higher are typically reviewed by at least three authorized individuals. The exceptions are RARMs that do not require mitigation plans to meet the pre-defined aircraft ALOS. Such RARMs only need one signature since, by definition, the risk is already considered to be within acceptable limits. While the OAM indicates that such ALOS RARMs are to be signed by an authorized section head, OA staff indicated that such RARMs are sometimes delegated to the subordinate staff level when the section head is absent. This raises concerns that there may not be sufficient review of such RARMs by qualified and authorized individuals.

Table C-1. Airworthiness Approval and Operational Command Risk Acceptance Authorities. This table lists the technical and operational airworthiness approval and acceptance authorities for different risk index levels.¹⁰⁷

Another issue is that, although RARMs are well understood in 1 CAD and 2 CAD Headquarters, interviews suggest that they are less understood by operational staff at the squadron/tactical level. As a result, the rationale for aircraft restrictions is not always well understood. Better communications to the operational units would improve airworthiness by improving the understanding of why restrictions are in place.

The airworthiness risk management process and its issues have been further discussed in Annex A.

Finding C6: The introduction of operational risk records needs careful monitoring to ensure there are no airworthiness implications.

In September 2012, operational staff introduced the Record of Operational Risk Management process to document risks from threats other than airworthiness that may impact the successful conduct of operations.¹⁰⁸ The name of the process was subsequently changed to ORAT. While ORATs are a useful tool to quantify and accept non-airworthiness risks, under some circumstances, there can be some debate as to whether a risk is purely operational or has airworthiness implications. A definition is provided in the OAM, but it is not entirely clear since there are no specific criteria to distinguish one from the other. Interviews with operational staff also highlighted the need for increased clarity. One example raised was the Moose Jaw Harvard II aircraft avoidance issue, which some interviewees indicated should be a RARM, whereas others indicated should be an ORAT. This confusion was also evidenced at the June 2013 AAB in the case of the CT156 flying into icing conditions.¹⁰⁹ The concern, previously raised by the AIA in May 2012,¹¹⁰ is that valid airworthiness risks might be recorded as ORATs instead of RARMs and thus bypass the airworthiness risk management process. For this reason, the OAM suggests that ORATs should be staffed through SSO OA, to ensure the identified risk is operational and not an airworthiness risk.¹¹¹ Unfortunately, this is not a requirement and is not mentioned in the 1 CAD Order describing ORAT applicability and completion.¹¹²

Indicator 5: Documentation is relevant, complete and updated regularly.

Finding C7: The OA documentation is comprehensive; however, it lacks a formal review process at fixed intervals.

Significant improvements have been made towards developing OA documentation, which has greatly aided the codification and awareness of the AWP and its associated processes. The OAA uses three key regulatory documents to execute its program.

• National Defence Flying Orders (B-GA-100-001/AA-000)
• Aerospace Control Flight Inspection Procedures Manual (B-GA-164-000/AA-001)
• OAM (B-GA-104-000/FP-001)

The National Defence Flying Orders and the Aerospace Control Flight Inspection Procedures Manual serve as overarching documents for regulating Canadian military air operations. In addition, 1 CAD publishes the RCAF Flight Operations Manual to supplement the National Defence Flying Orders and other 1 CAD Orders. These include some key OA processes, such as the foreign military aircraft overflight process¹¹³ discussed previously.

That said, most of the OA processes (besides flying regulations and orders) are described in the OAM. The OAM describes the Operational Airworthiness Program and provides a framework for regulating the operational aspects of military aviation activities, facilities and services to achieve and maintain aeronautical products as airworthy. In 2010, the OAA published the first version of the OAM, which was superseded by version 2.0 in 2013. In-depth document review as well as feedback from stakeholder interviews revealed that the OAM provides adequate guidance and direction from the OAA to all DND/CAF personnel involved in OA activities to ensure that the latter are performed in compliance with established aeronautical regulations and orders and operational airworthiness instructions and standards. However, there is no evidence of a formal review process at fixed intervals with appropriate stakeholder input to maintain the OAM. Formalizing the review process would ensure the content of the document is kept up-to-date and valid.

Indicator 6: Training is adequate to meet the needs of the program.

Finding C8: The OAA has not formally identified its training requirements for both OA regulators and operational staff.

The OAA has recognized the need for airworthiness training. A series of courses offered either encompass some airworthiness component or are purely airworthiness. Efforts to date have been focused toward the regulators and some implementers. Key courses are the Airworthiness Risk Management course delivered by the TAA and the OA Seminar delivered by the OAA. As well, to bridge the airworthiness knowledge gap, the OAA has successfully incorporated an airworthiness component in various other training courses. For example, the Flying Supervisor course and OT&E Project Officer course recently have added an airworthiness component. Interviews with key 1 CAD personnel validated that the courses were instrumental in their knowledge of airworthiness. However, interviews have indicated that staff with delegated OA responsibility are not consistently receiving training prior to assuming their delegated role. Formalizing the operational staff training requirements and making this training mandatory for all OA personnel with airworthiness responsibilities would increase the effectiveness of the program by enhancing their understanding of the AWP and their roles within the program.

Finding C9: Airworthiness knowledge dissemination does not always reach crews and squadrons.

Interviews conducted with key informants at the regulatory and user level revealed airworthiness discourse is largely concentrated at the regulatory level. The importance of airworthiness is well understood within 1 CAD Headquarters and has gained significant visibility over the last few years. However, crews and squadrons do not understand the nuances behind operational restrictions because airworthiness knowledge is not comprehensively circulated. The appointment of a formal training authority could manage educational standards across the airworthiness organization. Basic on-line courses or handbooks distributed to all personnel would enhance the overall understanding of how airworthiness impacts operations and procedures.

Indicator 7: The PMF is well defined, functional and appropriately managed.

Finding C10: While OA-specific performance measures and audits are lacking, 1 CAD SETs conduct audits of operational and maintenance units to ensure appropriate standards and procedures are being followed.

An essential component of an airworthiness program consists of conducting audits to ensure proper standards and processes are being followed. While the OAM does include requirements for auditing new aerodromes and contracted civilian operators, it does not include a requirement to audit DND/CAF operational units. This is because divisional SETs already perform a similar function to evaluate RCAF operational squadrons and maintenance organizations. While the SETS do not evaluate airworthiness requirements per se, several SET reports were reviewed, and these cover much of what would be examined under an OA audit, although no reference is made to the OAM or OA requirements. Interviews confirm that SETs have limited understanding of the OAM and OA requirements. Each SET only reports to its associated fleet SSO, and there is limited coordinated and standardization between the different SSOs and OAA staff. Inclusion of SETs as an audit function of the AWP would standardize the airworthiness aspects of SET audits and provide invaluable feedback to the OAA staff. Along with personnel airworthiness requirements, the airworthiness audit should evaluate units regarding the OA Clearances, flight permits, operations restrictions, RARMs and SOIs directly impacting them. In addition, non-RCAF units subject to DND airworthiness regulations, such as those operating DND UAVs and contracted aircraft operators, should also undergo periodic operational airworthiness audits. The OA program should consider emulating the TAP and establish an authority able to address these audit shortcomings, document and incorporate airworthiness audit requirements into the current SET program and monitor and report on the audit program. While complete SET standardization may not be feasible, a core set of standard airworthiness requirements should be part of each evaluation.

Apart from the SET audits, the evaluation did not find any evidence of specific performance measures or reports assessing the OA program itself. While progress has been made in several aspects of the OA program, there were no benchmarks or performance measures to assess or monitor the program and its improvements. TBS policy calls for each department to implement performance measures to monitor and improve program effectiveness. Suggested measures would include monitoring and reporting on the completeness and adequacy of airworthiness training among the regulatory and implementation staff, the fleet risk management process, airworthiness-related expenditures, resource requirements, the OA audit program, as well as other airworthiness process initiatives and improvements.

Indicator 8: Resources are adequate to meet the needs of the program.

Finding C11: The OAA has limited dedicated airworthiness regulatory staff, and the retention of expertise is a concern.

The SSO OA section in 1 CAD is critical to the health of the OA program. The SSO OA section is made up of three personnel and one part-time reservist tasked with running and overseeing the OA program. Responsibilities include facilitating OA training and providing subject matter expertise on OA Clearances, RARMs, test and evaluation projects, and other OA issues. In addition to the formal OA staff within SSO OA, there are other regulatory personnel in Air Force Standards responsible for the flying regulations. Despite the stark contrast to the much larger TAA organization, the OA program functions well by leveraging heavily on operational 1 CAD staff expertise and by dual-hatting fleet SSOs and other senior operational staff with OA responsibilities. The SSO OA regulatory staff fill a key role by diligently monitoring the resulting OA activities and documents. While the small size of the SSO OA section is viable, it must have solid continuity and stability to retain corporate OA knowledge, oversee the OA program activities and implement improvements to the program. However, because all positions are military and subject to regular posting rotations, the ability of the organization to retain expertise and fully qualified and experienced staff is limited. Methods to partly mitigate this periodic loss of experience would include documenting corporate knowledge by further developing SOPs and terms of reference and by hiring some civilian SMEs to ensure long-term expertise and stability.

The only funding available to SSO OA for executing OA responsibilities is $450,000. This is a reduction from $650,000 after recent budget cuts. The SSO OA budget is used primarily to fund OT&E activities. The lack of fenced funding for OAA means the organization is subject to the overall fluctuations and pressures of DND/CAF programs and must compete within this environment. It is difficult to quantify to what extent these financial restrictions impair airworthiness activities, but ongoing cuts have impacted progress of multi-year projects and are expected to hamper the planning of long-term airworthiness activities. The reduction of the SSO OA OT&E budget may lead to delay in testing and implementation of equipment/procedures that may be linked to risk reduction of identified issues. For example, budget constraints have delayed the implementation of the Terrain Collision Avoidance System for the Harvard fleet. Furthermore, other DND/CAF programs have fenced funding envelopes to ensure key programs are not adversely affected by budget fluctuations.

Finding C12: There is very limited staff capacity to oversee airworthiness requirements and operations involving UAVs, simulators, contracted aviation operators and foreign military aircraft operations within Canada.

As previously mentioned, there is a need for the OA program to review its oversight of UAV and foreign military aircraft activities to ensure adequate processes and resources are in place to assure the airworthiness of these operations. In addition, in the last decade, the DND/CAF has moved towards using more contracted aircraft operators to provide services that were traditionally provided by aircraft owned and crewed by the RCAF. Every area, from basic pilot training, aggressor aircraft, airlift and UAVs, are now being provided, in some part, by civilian aviation operators. While this may be a more efficient delivery of a non-core service, it nevertheless requires airworthiness oversight by DND. An interdepartmental memorandum of understanding delineates the respective TCCA and DND responsibilities for airworthiness oversight of these civilian operators and their aircraft.¹¹⁴ These responsibilities are also summarized in the AWP policy manual.¹¹⁵ These civil-registered aircraft all operate with a TCCA Certificate of Airworthiness and, depending on the level of interaction with DND, DND airworthiness oversight varies from nil (such as for contracted cargo flights) to comprehensive. For instance, DND completes a full airworthiness clearance of civil aircraft used for DND basic pilot training but only provides a Temporary Authority to Operate for civil-operated aircraft involved in combat support training. Beyond the initial DND airworthiness clearance or Temporary Authority to Operate, DND also has the responsibility to oversee and ensure the continuing airworthiness of the aircraft and the operations involved. This is accomplished through Temporary Authority to Operate audits and operational SETs. For example, 414 Squadron undergoes regular Temporary Authority to Operate audits and also has two SET oversight pilots for Discovery Air’s provision of Alpha Jets for the Contracted Airborne Training Services program. As the use of contracted aviation increases to support DND training and operations, increased oversight and liability is placed upon the Department. The OAA’s scope of authority requires the ability to regulate and audit conformance. While document and audit reviews indicate that the OAA is engaged in both the regulatory and operational audit of many of these non-core activities, staff limitations likely impede full oversight. Growth in the use of contracted aircraft by DND/CAF, in combination with reduced levels of TC oversight and limited DND oversight, may represent an increased risk for the Department. Similarly, a decision by TCCA to stop certifying flight simulators for CAF aircraft¹¹⁶ will require additional OA resources to regulate, certify and oversee these systems.

A recommendation for an OAA review of its organization and resources to ensure sufficient OA oversight of these activities has been made under Indicator 2 (Organizational Structure).

Annex D—Investigative Airworthiness Effectiveness

In support of the AWP, the primary responsibility of the AIA is to regulate the airworthiness aspects of the FS Program and conduct independent investigations of airworthiness-related accidents. The AIA is also tasked to monitor the AWP to identify deficiencies and recommend preventative measures.

Indicator 1: The scope of authority is appropriate, well documented and exercised in accordance with policy and directives.

Finding D1: Until Bill C-3 “Safeguarding Canada’s Seas and Skies Act” was passed, the AIA did not have statutory powers to investigate civilians involved in military accidents and incidents.

The CDS designated the DFS as the AIA. In support of the AWP, the primary responsibility of the AIA is to conduct independent investigations of airworthiness-related accidents. However, until 2015, the AIA had no authority to apply established FS investigative processes to civilians involved in military accidents or incidents. As no statutory requirement existed to require civilians or civilian companies to participate in DND FS investigations, DND relied instead on contractual obligations and cooperation to conduct such investigations. Bill C-3 “Safeguarding Canada’s Seas and Skies Act,”¹¹⁷ which came into force in February 2015, amends the Aeronautics Act to give the MND, through the AIA, full statutory powers to investigate accidents and incidents involving civilians. While regulations still need to be developed, Bill C-3 will enable the AIA and the military investigators to carry out their duties for all aspects of military aviation safety, and it will promote the independence and integrity of military FS investigations. A working arrangement¹¹⁸ between the Canadian Transportation Safety Board (TSB) and DND further delineates the situations and the extent to which DND will lead or participate in investigations involving civilians.

Finding D2: The scope of FS investigations in zones of conflict had been an issue until it was clarified in 2013.

“The purpose of the Flight Safety Investigation is to determine all factors, which contributed to the occurrence and to establish the required preventative measures.”¹¹⁹ AIA/DFS investigations follow the International Standard and Recommended Practices for Aircraft Accident and Incident Investigation.¹²⁰ However, reviews of accident reports in the last five years indicated that the scope of investigations for accidents occurring in zones of conflict had been limited, in accordance with a note in the FS manual¹²¹ excluding events caused by enemy action. Unfortunately, this was preventing the AIA from investigating other factors that may also have contributed to the incident. For example, in 2010, a Chinook aircraft transiting at low altitude was forced down due to an explosion and in-flight fire during a mission outside Kandahar Airfield. The source of ignition was reported as being due to insurgent fire that was directed towards the aircraft. The aircraft landed safely but aircrew and passengers sustained minor injuries. The DFS scope of the investigation was limited to the review and analysis of the effectiveness of ALSE and egress procedures, as well as other issues pertaining to occupant safety.¹²² It is unclear which agency, if any, conducted the remaining portion of the investigation to review the operational circumstances that led to the accident. In 2013, the need to investigate all aspects of accidents in a zone of conflict was recognized and led to an amendment of the FS manual that now requires investigation of all occurrences in which the aircraft, its equipment or its operation contributed to the event.

Finding D3: The AIA does not fully exercise its mandated responsibility of monitoring and auditing the AWP.

The AWP policy manual states that “the AIA is responsible for monitoring the Airworthiness Programme with a view to identifying any deficiencies, and reporting them to the Airworthiness Authority.”¹²³ While the AIA does monitor FS occurrences throughout the CAF and performs some beneficial basic monitoring of airworthiness activities and products as a participant in the ARB, AAB, and RTS processes, the evaluation found no documented evidence of AWP strategic monitoring or auditing, on behalf of the AA, to assess the status, deficiencies and development of the overall AWP. Based on interviews with AIA staff, such strategic program monitoring and auditing appears to be beyond the capability of current AIA resources.

In addition, as mentioned in Annex A, the evaluation is not convinced that strategic monitoring and auditing of the DND/CAF AWP should be an AIA function. It would not only require the AIA to assess its own involvement in the AWP, but there is also a concern that such a function could impact on the AIA’s mandated requirement to be impartial and independent during accidents/incident investigations. As well, the AIA organization is not currently structured to perform these functions. A small section reporting to the AA would be better positioned to formally monitor and audit the AWP, as well as fulfilling other airworthiness functions on behalf of the AA. This issue and the resulting recommendation can be found under Indicator 1 of Annex A.

Indicator 2: The organizational structure is adequate to meet the scope of authority.

Finding D4: The AIA is organized to conduct investigations, but lacks flexibility to handle major accidents and does not effectively monitor the overall AWP.

The AIA has established a fairly robust and efficient investigative program with clear lines of responsibilities.¹²⁴ However, even with the addition of an AIA role to the pre-existing DFS organization, the introduction of UAVs and the use of contracted air services and foreign military operations, its governance and organizational structure has not changed significantly. Instead, investigative personnel have been tasked with concurrent monitoring roles to act as advisors with respect to ARB and AAB documentation, RTS and the review of RARMs, which has hampered investigation timelines. While AIA involvement is beneficial at a tactical level, its resources are not organized to conduct strategic monitoring and auditing of the overall AWP. This is discussed in more detail under Indicator 1 of Annex A.

The AIA organization is adequately structured to deliver its mandate of investigating accidents. However, the investigation of a complex, large-scale accident, particularly in a remote location, would require significant investigator resources and coordination with other GC departments. Interviews with AIA staff confirmed the need to develop a response plan in consultation with other affected GC departments in order to prepare for this type of emergency. However, the DND/CAF FS structure could also be reviewed to provide more flexibility. Organizationally speaking, there is no direct command line between the AIA and divisional FS sections. Besides the 15 investigator positions within DFS, 1 CAD has an FS section comprised of 9 positions and which reports directly to Comd 1 CAD to advise on FS issues, promote FS to field units and deliver FS courses to unit personnel. Similarly, 2 CAD FS is also creating a divisional FS office to advise Comd 2 CAD on FS matters. The 1 CAD and 2 CAD FS sections do not report to DFS and do not currently perform an active investigative role for the AIA. With the addition of the AIA roles, the lack of DFS surge capacity to handle large accidents or an increase in accident rates and the creation of 2 CAD and the complexity of aviation issues, a review of the entire RCAF FS structure is deemed to be warranted. Should the divisional FS sections report directly to DFS, it would allow for a better oversight of the FS program, give more flexibility and robustness to the AIA investigative team and provide more independence to the AIA. Additionally, it would give the AIA better control over the delivery and monitoring of investigative training for which the position is ultimately responsible. This change of reporting structure would still allow for the 1 CAD and 2 CAD FS sections to continue advising their commander on FS issues under their command. This type of reporting structure is successfully used by the Military Police and Judge Advocate General. They are attached to commands and formations as advisors but directly report to their chain of command (e.g., Provost Marshal and Judge Advocate General) for a better oversight and continuity of the specialist services performed.

Indicator 3: Processes are functional and well documented.

Finding D5: The AIA carries out independent and thorough aviation investigations, but report completion times are an issue.

The mandated role of the AIA is to investigate airworthiness-related aviation safety occurrences;¹²⁵ this process results in Flight Safety Investigative Reports (FSIR)¹²⁶ that follow the International Civil Aviation Organization format.¹²⁷ These investigations are expected to be independent, thorough, and timely. Independence is paramount for objective reporting and according to the AWP manual, “The Investigator…must remain ‘independent’ from the Regulator and the Implementor.”¹²⁸ In the present organizational structure, the AIA reports to the AA, and to ensure independence, the AIA has the option to inform the CDS/MND via the AA.¹²⁹ Further, FSIRs are subject to three levels of review: Senior Investigator, Chief Investigator, and finally the DFS. That, coupled with positive feedback from operational and technical airworthiness personnel confirms that AIA produces FSIRs that are thorough.

According to the Airworthiness Investigative Manual (AIM), the AIA’s goal is to complete and publish FSIRs within a twelve month timeframe.¹³⁰ Considering the relatively small size of the AIA section in Ottawa, service level agreements and memoranda of understanding are essential tools for the AIA to cooperate with other agencies to leverage the necessary expertise and support for the investigations. As such, the AIA works closely with Aerospace Engineering Test Establishment, Quality Engineering Test Establishment, National Research Council, Defence Research and Development Canada and TSB. It has been determined via stakeholder input that these arrangements are adequate and provide the required aviation and technical expertise. However, these organizations also have work priorities that can make it difficult to provide timely support to specific AIA taskings. The capacity of supporting expertise in other organizations, long wait times for translation services, lengthy translation reviews, lack of experience in writing investigative reports and additional tasks to investigators such as preparation for ARBs and AABs, are all issues contributing to the difficulty in meeting formal timelines. For a complex accident, completing an investigation within a year has become very difficult and, at times, impossible. Table D-1 demonstrates that between 2009 and 2015, no FSIRs met the one-year timeline goal, whereas 54 percent were completed in approximately two years. The minimum time taken to complete an FSIR was 15 months and the maximum time was 58 months, with the average time being 31 months. Nonetheless, all stakeholders agree that there must be an effort to re-enforce the one-year timeline goal to maintain the relevance and credibility of findings. To this end, development of a production cell could alleviate timeline pressures by increased monitoring to reduce the time required for report completion, provide continuity and establish a standard for reports.

Table D-1. FSIR Completion Times. This table provides completion times of FSIRs from 2009 to 2015. ¹³¹

Finding D6: The majority of incidents investigated at the squadron/wing level meet the 30-day completion timeline.

Another AIA process involves investigating incidents, the output of which is a Supplementary Report (SR). SRs give details revealed from the investigation related to the cause of an occurrence, make findings, assign cause factors, and recommend preventive measures. There are approximately 3000 reported incidents per year. There are approximately 300 investigators at the unit/wing level in the RCAF and most of these investigators are embedded within the squadrons as the Unit Flight Safety Officers, Deputy Unit Flight Safety Officers and Flight Safety Non-Commissioned Members. These unit FS teams are responsible for the investigations of occurrences in their unit. According to the AIM, SRs are to be completed within 30 calendar days of the occurrence.¹³² DFS did not collect data on SRs completion timelines prior to 2010; however, tracking was initiated that year to better monitor the performance of the program. Although there is no identified target threshold, SRs are now closely reviewed, monitored and tracked for timeliness. This data can serve as a useful indicator within an eventual AIA PMF. As such, most SRs are completed within the target timeline as demonstrated in Table D-2.

Table D-2. SR Timeliness. This table provides the total number of SRs and the percentage that met the 30-day deadline from 2011 to 2014.¹³³

Overdue occurrence reports have a detrimental effect on the AIA’s ability to analyse and trend cause factors and the distribution of preventive measures information. The reasons for overdue SRs appear to stem from Wing and Unit FSOs managing competing priorities between primary and secondary duties and the operational tempo of the unit. The AIA continues to strive for excellence as is demonstrated in the Director’s Comments of the 2013 Annual Report; “We shall continue to improve on those statistics, ideally to less than 5%. The flight safety staff must remain focused on core activities of investigating occurrences, recommending preventive measures and monitoring their implementation and effectiveness.”¹³⁴

Indicator 4: Risk Management is well defined, functional and appropriately managed.

Since the AIA is not directly involved in the airworthiness risk management process, the evaluation did not make any findings or recommendations regarding this indicator with respect to the AIA. However, it should be noted that, as part of their AWP monitoring activities, AIA staff are forwarded copies of completed RARMs and can question them if necessary.

Most of the findings and recommendations regarding the DND airworthiness risk management process can be found in Annex A.

Indicator 5: Documentation is relevant, complete and updated regularly.

Finding D7: The AIA documentation is comprehensive, but it lacks a formal review process.

The DFS/AIA publishes the following two key documents to execute its program:

• Flight Safety for the Canadian Forces (A-GA-135-001/AA-001)
• AIM (A-GA-135-003/AG-001)

Published in 2009, the AIM is an extension of the DND/CF AWP manual, comprehensively delineating the AIA policies and amplifying standards, procedures and instructions for investigators. Its first revision was published in March 2015. The AIM lacks a formal revision process with appropriate stakeholder input. Formalizing the review process would ensure the content of the document is kept up-to-date and valid.

Aside from the AIM, the AIA has a series of SOPs that serve as directives for the staff. These describe AIA roles when dealing with airworthiness activities such as Temporary Authority to Operate, RARM, RTS, airworthiness reports, preventive measures management and IA Clearances. The content of these SOPs contributes to the standardization and quality of AIA activities. As with the AIM, implementing a scheduled process to review and update these procedures will improve their overall effectiveness.

Indicator 6: Training is adequate to meet the needs of the program.

Finding D8: Training for investigators is well established and of high quality, but requirements for FS training of field personnel exceed availability.

The AIM adequately details the qualifications required for investigator competencies,¹³⁵ and the AIA has a well-established and documented training plan¹³⁶ to properly qualify investigators. Interviews revealed that although training for accident investigators is closely tracked at NDHQ, it is not clear that the same tracking rigor is in place for unit-level FS investigators. Interviews with wing and unit FS personnel revealed that the FS Course was instrumental in their ability to carry out their tasks. The results of a DFS course validation survey conducted in 2014 indicated that 86 percent of the personnel agreed that the FS Course provided them with the knowledge and skills required to investigate FS occurrences and hazards.¹³⁷ The FS Course is offered five times a year with space for 32 persons per session, but the demand typically represents two to three times the number of places available. With such a high demand, succession planning and prioritization are imperative to control and monitor the training of personnel. Interviews with FS personnel revealed that sometimes personnel are assigned to FS positions before having received the required training, which impairs their ability to properly conduct and report FS investigations. However, the exact number is unknown because tracking of FS qualifications is done informally and only at the wing level. With the arrival of new RCAF fleets, UAVs managed by army/navy personnel and increased contractor roles in some maintenance support concepts, the FS Course will continue to be in high demand. Additional measures should be implemented by divisional FS to ensure that all personnel involved in FS investigations are suitably trained and qualified.

Indicator 7: The PMF is well defined, functional and appropriately managed.

Finding D9: The AIA tracks extensive accident/incident data, but does not have a formal PMF to monitor and improve AIA activities.

According to the TBS Directive on the Evaluation Function, program managers are responsible for developing and implementing performance measurement strategies for their programs.¹³⁸ While the AIA tracks data as part of its activities, it has not implemented a formal PMF. Current AIA measurement activities include, but are not limited to accident/incident statistics and data analysis, report completion timelines, wing monitoring, cause factor analysis and surveys of different wings. The evaluation assessed that many of these AIA activities could form the basis of an AIA PMF once supplemented by measures to assess its own internal activities. The 2013 AIA Annual Report states that “Surveys are conducted to measure the effectiveness of the FS Program, to identify deficiencies that would otherwise have gone undetected, and to make recommendations for enhancements.”¹³⁹ This highlights the use and benefit of one monitoring activity to improve the AIA program. Augmenting and formalizing several such measures into performance indicators would contribute to a more robust PMF to continuously monitor and improve the AIA program.

Indicator 8: Resources are adequate to meet the needs of the program.

Finding D10: The AIA is adequately staffed to perform most accident investigations, but has limited capacity to strategically monitor the AWP and handle a large scale accident.

The AIA organization is adequately staffed to deliver its mandate of investigating accidents based on existing workload requirements with 12 out of 15 investigator positions currently staffed. Service level agreements and memoranda of understanding with other organizations also support the conduct of investigations.

The CAF has had a low accident rate in the last few years and has not recently experienced a complex accident situation. The investigations of major accidents require significant expertise, especially in the lead investigator role. It takes approximately three years to fully train an investigator and, with a three-year military posting cycle, the ability of the organization to retain expertise and fully qualified and experienced military investigators is limited. Should a large-scale accident or a higher than normal accident rate occur, the evaluation team believes that the AIA would have difficulty providing enough qualified and experienced investigators. The time required to become a fully trained investigator, coupled with a high turnover of military personnel, runs the risk of a resource shortfall of trained personnel in the event of a large-scale accident. This situation could be alleviated by adding permanent civilian investigator positions.

AIA personnel responsibilities are not limited to accident investigations. Responsibilities include, but are not limited to reviewing each fleet, training coordination, sustaining publications, maintaining service level agreements/memoranda of understanding and supporting RTS and risk management, ARB/AAB coordination, statistical data analysis and surveys to monitor the DFS/AIA program. However, as discussed under Indicator 1 of this annex, these activities do not monitor the overall strategic health of the AWP. This activity appears to be beyond the scope of current AIA resources. As suggested under Indicator 2, a review of FS organizations in NDHQ and 1 CAD could help resolve some of the AIA resource issues.

Annex E—Benchmark Study

Introduction

It is worthwhile to review how airworthiness is conducted in the defence services of some allied nations in order to objectively assess if the DND/CAF approach to airworthiness reflects best practices and suggest areas where some improvements could be made. The DND/CAF airworthiness process will first be summarized, followed by descriptions of the equivalent processes in Australia, the UK and the USA. These three countries were chosen because of their similar principles as allied countries, availability of information describing their processes and their different implementations of airworthiness. In addition, TC will also be reviewed as a representative of the civilian airworthiness authority. It is interesting to note the diversity of these approaches to accomplish similar airworthiness objectives. These different implementations can be an invaluable source of information and best practices to further improve the DND/CAF AWP. Best practices are summarized at the end of this annex.

It is beyond the scope or purpose of this review to examine each nation’s airworthiness processes in complete detail. Rather, this benchmarking review will focus on the following key aspects where differences may be more significant:

• airworthiness structure
• regulatory independence
• regulatory staff resources
• airworthiness review boards
• training requirements and resources
• airworthiness risk management
• primacy of operations over airworthiness risk.

In order to standardize the different terms used by the different nations, the following definitions are used:

• AA – the authority responsible for the implementation and management of the airworthiness program;
• Technical Airworthiness Regulator (TAR) – the authority responsible for establishing and maintaining the TA regulations;
• OAR – the authority responsible for establishing and maintaining the OA regulations;
• TAA – the authority responsible for determining the TA of aircraft in conformance with the regulations;
• OAA – the authority responsible for determining the OA of aircraft in conformance with the regulations; and
• AIA – the authority responsible for monitoring the airworthiness program and investigating aviation safety issues.

Canada

Canada has a population of approximately 34 million people and a defence establishment of about 90,000 full-time personnel (military and civilian). Unlike most other nations, all of the Canadian military aviation resources, with the exception of UAVs, have been consolidated within the RCAF. In accordance with the Aeronautics Act, military airworthiness is the responsibility of the MND, and DND has a self-regulating and self-investigating responsibility for airworthiness. The MND and CDS have formally delegated C Air Force, DGAEPM, Comd 1 CAD, and DFS as the AA, TAA, OAA and AIA, respectively. The AA, TAA, OAA and AIA manage the DND airworthiness program and ensure formal airworthiness regulations and processes are in place. The OAA and AIA are both within the RCAF chain of command, whereas DGAEPM, as the TAA, falls under ADM(Mat) and is independent of the RCAF chain of command. The OAA is responsible for regulating and implementing OA and has limited staff to look after regulatory aspects. The TAA has a significant number of regulatory staff within the DTAES directorate, which is organizationally independent from the implementers within the other DGAEPM directorates. The TAA is also independent of the project offices within DGMPD (Air) and the operational maintenance units. To execute airworthiness oversight and reduce regulatory workload, the TAA delegates limited airworthiness authorities to a suitably qualified SDE and, to a lesser extent, other Authorized Individuals for each aircraft fleet within DGAEPM.

The TAA is responsible for both regulating and implementing technical airworthiness. The TAA ensures the technical airworthiness of new aircraft systems through the issuance of MTCs, Certificates of Airworthiness and TACs. Similarly, the OAA ensures initial operational airworthiness through the issuance of OA Clearances. Subsequently, the AA issues an RTS upon the recommendation of an RTS Board that all airworthiness and non-airworthiness issues have been addressed. Once in service, the TAA and OAA oversee continuing airworthiness by chairing annual ARBs to review and confirm the airworthiness of each fleet and assure compliance by conducting periodic audits of operational, maintenance and design organizations. There are formal processes for approving design and maintenance changes, and a risk management process to identify, assess, mitigate and accept airworthiness risks. Low and medium risk assessments can be approved by Authorized Individuals, SDE and a DOAM, but high and extremely high risk assessments must be approved by DTAES and the TAA, and the 1 CAD SOAM or OAA. Risks must also be accepted by a more senior OCRAA. Operational commanders have the flexibility to undertake missions if they deem these to be more important than the documented fleet airworthiness risks.

Australia

References:

• Australia Directorate General Technical Airworthiness (DGTA) Website: www.defence.gov.au/dgta/
• Defence Instructions (General) – Defence Aviation Safety Program (DI(G) OPS 02-2)
• Operational Airworthiness in the Australian Defence Force (ADF)
• The ADF Airworthiness Management System
• Management and Communication of Risk (DGTA(I) TECH 4-2)
• DTAES Trip Report (2182-01775-01), dated June 4, 2012
• ADF Airworthiness Education Strategy
• Australian Air Publication 7001.053 – Technical Airworthiness Management Manual

Australia has a population of approximately 24 million people, but its defence establishment of about 80,500 personnel (full-time military and civilian) is comparable to that of Canada. Unlike Canada, its military aviation resources are dispersed amongst the air, navy and army services comprising the ADF. As a result, while the Chief of Air Force is the ADF AA and the Deputy Chief of Air Force is the OAR, the OAAs are distributed among the three services. Each OAA is appointed by the AA based on nominations submitted by the Service Chiefs and is accountable to the AA and responsible to the relevant Service Chief for the operational airworthiness of their aircraft and making informed decisions on the treatment of risks.

OAAs can formally delegate operational airworthiness authority to an Operational Airworthiness Authority Representative. This ensures that operational airworthiness is managed at the lowest practicable command level. Each Operational Airworthiness Authority Representative normally has responsibility for a specific aviation system/aircraft type or a group of aircraft in similar roles. Importantly, Operational Airworthiness Authority Representatives may not further delegate their authority.

While the distinct OAR role and the distribution of OAA responsibilities differ from those of DND, technical airworthiness is similar. ADF technical airworthiness is the responsibility of the DGTA-ADF, a tri-service agency organisationally located within Air Force Headquarters. DGTA is both the TAR and the TAA. As TAA, DGTA is responsible to the AA and responsible to the Service Chiefs for the technical airworthiness of ADF aircraft and for communicating technical risks to the relevant OAAs. The AA, OAR and TAR ensure formal processes are in place to develop regulatory policy and assure compliance. The OAR staff conducts audits of the OAAs, whereas DGTA staff conducts audits of Authorised Engineering Organisations and Authorised Maintenance Organisations.

A significant difference from DND is that the ADF TAR/TAA is independent of the ADF Defence Materiel Organisation, which is responsible for acquisition and sustainment of defence aviation systems. Within each Defence Materiel Organisation Systems Program Office, an aircraft Design Acceptance Representative (DAR) (typically the Chief Engineer) is delegated by the TAR to manage the Design Acceptance certification process for type designs and major design changes on that position’s behalf. While the DAR cannot delegate responsibilities, he/she can authorize SDEs within other aircraft engineering organisations to approve minor design changes for systems in service. SDEs are nominated by aircraft engineering organizations and endorsed by the TAR.

The aircraft clearance process is very similar to that of DND. Following DAR Design Acceptance certification and an Australian MTC, a Service Release is issued to indicate that the operational and technical airworthiness infrastructure is in place to support flight operations of the new aircraft type. The Australian MTC and Service Release are both issued by the AA on the recommendation of the TAA, OAA and Airworthiness Board (AwB). Subsequent major changes to the aircraft type design undergo the same acceptance process to issue a Supplemental Type Certificate, which certifies that the aircraft continues to meet appropriate airworthiness standards.

A marked difference between the DND and ADF airworthiness process is the composition and independence of the annual AwB. The ADF AA annually convenes the AwB to review the type certification, Service Release and airworthiness of each aircraft in service, being introduced into service and undergoing major modifications. The AwB consists of two officers of star rank (one operational and one technical) who are independent of the ADF airworthiness regulatory system and the chain of command. These officers are generally retired senior officers who perform their board functions as members of the active reserve. The board members review extensive documentation, which is summarized in a full day’s worth of presentations and questions and answers. As reported at reference F, paragraph 40, the ADF believes that the AwB provides significant benefit in its independence and methodology. At reference F, paragraph 21, DGTA also indicated that they plan to implement a “hazard log” of accepted aircraft risks, which will be reviewed annually at the AwB to help assess an aircraft’s composite risk and continued airworthiness.

Two other key tri-service agencies include the Airworthiness Coordination and Policy Agency and the Directorate Defence Aviation and Air Force Safety. The Airworthiness Coordination and Policy Agency represents the AA and OAR on matters concerning airworthiness management and is responsible for airworthiness training policy, regulatory audit functions, issuance of Australian MTCs, and maintenance of the state register. Also, as the Airworthiness Board (AwB) secretariat, the Airworthiness Coordination and Policy Agency is responsible for the high-level Military Aviation Regulations from which Operational Airworthiness Regulations and Technical Airworthiness Regulations are developed. The Directorate Defence Aviation and Air Force Safety, while not formally part of the ADF airworthiness management system, assists defence organizations to successfully implement aviation safety management and is responsible to the Chief of Air Force (in his role as Defence Aviation Authority), the Service Chiefs and the Group Heads for investigating aviation safety matters.

The objective of ADF defence aviation is that it be conducted at acceptable levels of risk and to a level “as low as reasonably practicable.” However, commanders retain the flexibility to deviate from airworthiness requirements due to compelling operational imperatives. Operational clearances can be issued by the OAA or Operational Airworthiness Authority Representative to operate aircraft in roles, environments or configurations for which they have not been certified, in extenuating situations where the increased risk is considered acceptable given the operational requirement. Such clearances are to be based on aviation risk management and must include specialist advice. There are also cases where, for operational imperatives, an aircraft or aviation system must be modified or must operate beyond its certification basis without completion of the Australian MTC / Service Release process. The ADF AA may permit such modifications to take place under an Airworthiness Directive, potentially accepting a higher level of risk.

Mandatory airworthiness training is required for all key airworthiness appointments, including the AA, OAR, TAR, OAAs, DARs and AwB, as well as aviation-related command appointments and airworthiness support staff. The training consists of a personal briefing given by the Airworthiness Coordination and Policy Agency, DGTA or a senior airworthiness authority, as well as an on-line ADF Airworthiness Course. The airworthiness training for these positions must be completed within three months of assuming responsibilities but is preferably achieved prior to assuming responsibility. The Aerospace Engineer, Senior Maintenance Manager, and Senior Non-Commissioned Officer Technical Manager courses all include airworthiness training as part of their qualifications. DGTA also provides risk training to its staff and is planning to provide a Technical Airworthiness Management Manual course (reference F, paragraph 24.a.vi). To support this training, DGTA has a small training cell consisting of three staff to coordinate training, but the actual training is given by SMEs as a secondary duty; according to reference F, annually this requires the equivalent of two person years of effort.

The UK

References:

• Charles Haddon-Cave, QC. Nimrod Review – An Independent Review into the Broader Issues Surrounding the Loss of the RAF Nimrod MR2 Aircraft XV230 in Afghanistan in 2006, dated October 28, 2009.
• UK Military Aviation Authority Website: www.maa.mod.uk
• UK Military Aviation Authority PowerPoint Brief to The Technical Cooperation Program Aerospace Systems Group, dated June 2011
• UK Military Aviation Authority Regulatory Policy
• UK Military Aviation Master Glossary
• UK Regulatory Articles: 1000 Series

While the UK population of 64 million is not quite double that of Canada, it has a much larger defense establishment of approximately 225,000 personnel (full-time military and civilian). The authority to operate and regulate UK military registered aircraft is vested in the Secretary of State for Defence. As with most other nations, its military aviation resources are dispersed amongst its air force, navy and army services such that, until a few years ago, airworthiness responsibilities were similarly dispersed. However, the loss of a Nimrod aircraft in 2006 led to an extensive review (reference A) of how airworthiness was regulated and conducted in the UK Ministry of Defence. Subsequent implementation of the report’s recommendations resulted in a major overhaul of the Ministry of Defence airworthiness system. One of the key changes was the Secretary of State for Defence’s establishment, in 2010, of a Military Aviation Authority as an independent Ministry of Defence organization responsible for the regulation and assurance of UK defence aviation safety. The UK Military Aviation Authority is led by a three-star director general and includes approximately 250 positions (reference C). The Military Aviation Authority has a regulatory role over defence aviation similar to that of regulatory civilian airworthiness authorities over civilian aviation. This includes promulgating and enforcing air safety regulations, approving certification of all new UK military air systems and major changes, maintaining the UK Military Aircraft Register and training personnel with airworthiness responsibility. In addition, the Military Aviation Authority includes a joint Military Air Accident Investigation Branch that undertakes military air accident investigations in support of Service Inquiries convened by Director General Military Aviation Authority.

While the Military Aviation Authority is the sole regulator for UK military aviation, implementer airworthiness authority ultimately rests with each of the Service Chiefs of Staff, who are assigned the role of Senior Duty Holders (DH) and the Assistant Chiefs of Staff, who are designated as the Release to Service Authorities. Subordinate command staff are also assigned as Operational DH and Delivery DH. The DH role parallels that of the command chain, and legally individuals are personally accountable for the safe operation of air systems within their area of responsibility. DHs are responsible for the upkeep of Air System Safety Cases. A risk management system enables them to manage risk to life arising from the support and operation of military air systems. It is their responsibility to cease routine aviation operations if risks to life are identified that are not demonstrably tolerable and “as low as reasonably practicable.” A Senior Operator and a Chief Air Engineer advise each DH such that both the operational and technical sides are represented. The Operational DH must hold an annual Air Safety Steering Group to evaluate air safety risk across his/her area of responsibility, as well as periodic type-specific Air System Safety Working Groups to evaluate the airworthiness, operation and maintenance of specific air systems.

In addition to the Chief Air Engineer advising each DH, the other key technical airworthiness authorities are the Type Airworthiness Authorities, who are typically aircraft Project Team Leads working within the joint Defence Engineering and Support organization. Each Type Airworthiness Authority oversees the airworthiness of specified aircraft types throughout their full life cycle and can approve minor design changes. Military aircraft Type Certificates and subsequent major changes must be approved by the Military Aviation Authority.

Even though the UK structure of airworthiness responsibilities differs considerably from that of DND, the aircraft airworthiness processes are generally similar. Essentially, for an aircraft to operate on the UK Military Aircraft Register, an RTS must be issued by the Release to Service Authority on behalf of the Senior DH, supported by an MTC issued by the Military Aviation Authority. Where operational imperatives may result in high levels of risk exposure or where supporting evidence is still immature, the Release to Service Authority may consider issuing Operational Emergency Clearances or Clearances with Limited Evidence.

Regulations require DHs, commanders and project team leads to ensure that the training experience and qualifications of personnel involved in airworthiness activity are assessed and documented. Certain airworthiness positions, including DHs and Type Airworthiness Authorities, are subject to MAA endorsement based on set criteria with respect to qualifications and experience. To support these qualification requirements, the MAA Skills Training and Talent Sustainment section sponsors and delivers air safety training courses for the Defence aviation community.

USA

References:

• USA DoD Directive 5030.61: DoD Airworthiness Policy

The USA has a population of approximately 319 million people with a Department of Defense (DoD) establishment of some 2.13 million full-time personnel. Its military aviation resources are dispersed amongst its three military services, namely the Department of the Army, the Department of the Navy, and the Department of the Air Force. The DoD does not have a central airworthiness authority; rather, each military department is responsible for its own airworthiness authority, policy and implementation in accordance with reference A. Since each military department is a self-certifying organization, the DoD policy requires each one to establish sufficient independence between its airworthiness organization and its implementers (system program offices and operational organizations) for the airworthiness organization to exercise engineering oversight and present an objective assessment of airworthiness and safety of flight risk.

US Army

References:

• Army Regulation 70-62: Airworthiness Qualification of Aircraft Systems
• Aviation Engineering Directorate (AED) / Department of the Navy (DoN) Military Authorities Recognition Questionset, 2013
• Army Regulation 385-16: System Safety Management Guide

The US Army defines its policies, responsibilities, processes and procedures for airworthiness qualification and system specification compliance of Army aircraft at reference A. The Army AA is the Commanding General of the Aviation and Missile Command, who delegates responsibility for execution of the airworthiness program to the AED by a delegation memorandum. The AED has delegated AA status and is the single airworthiness office with engineering cognizance and delegated authority for execution of all Army airworthiness qualifications and continued airworthiness actions. The AED is responsible for regulations, policies, procedures and standards relating to airworthiness, issuance and renewal of certificates of airworthiness, approval of modifications and mandatory inspections, approval and inspection of approved maintenance organizations and monitoring and control of mandatory continuing airworthiness information. The AED reports through the Research, Development and Engineering Command in the Army up through the Chief of Staff of the Army and is organizationally independent from the regulated community of aircraft operators and from the programs and the projects for which it makes airworthiness determinations.

The US Army fleet of developmental, production and fielded aircraft systems is managed by the Program Executive Officer Aviation through a series of Program Management Offices responsible for their entire lifecycle. The Program Executive Officer reports through the Assistant Secretary of the Army for Acquisition, Logistic and Technology. The Program Managers are responsible for ensuring that the AED-approved airworthiness requirements have been met and for obtaining an airworthiness release prior to fielding a system. AED assesses all design data, test results and analyses provided by the Program Management Office against the pre-approved airworthiness criteria. For permanent aircraft configurations, a Statement of Airworthiness Qualification (SAQ) is issued when all airworthiness requirements have been met, all technical publications are fully complete and current, and all maintenance and sustainment provisions are in place to support continuing airworthiness. The SAQ is analogous to an MTC. Aircraft configurations may also be released to the field without a full SAQ through issuance of an Airworthiness Release. The AED issues and renews Airworthiness Releases, SAQs or Airworthiness Approvals for every Army aircraft. These documents function as both the authority to operate and as the certificate of airworthiness within the Army system, and AED has the authority to cancel, revoke, not issue or otherwise amend them. Airworthiness Releases are also required for modifications impacting airworthiness.

The total AED airworthiness expertise consists of approximately 800 work years with 560 of those man years being provided by direct hire government employees and the remainder as contract engineering support. All employees must take the New Employee Orientation training, which provides an overview of the Army airworthiness process and the responsibilities of all the divisions within AED to achieve airworthiness. Each employee also has an Individual Development Plan and Career Development Guide based on their assigned duties and responsibilities. The Career Development Guide defines all the education, training and experiential development required to become a fully functioning and empowered airworthiness engineer.

The US Army does not issue production certificates or approvals. The Army conducts production readiness assessments of contractor facilities, expertise, processes and equipment, and Defense Contract Management Agency (DCMA) personnel are responsible for overseeing the production process and accepting the delivered aircraft. They assure the aircraft are in compliance with the specifications and that all discrepancies have been identified and accepted. Minor changes to a manufacturing process or technical data that do not affect a critical process can be approved locally by DCMA, but discrepancies or modifications with airworthiness implications are required to be evaluated and approved by AED. The Army maintains a registry of aircraft in inventory, but registration is used principally for accountability and inventory and does not indicate the airworthiness of any Army aircraft.

US Navy

References:

• Naval Air Systems Command Instruction 13034.1D: Flight Clearance Policy for Air Vehicles and Aircraft Systems
• AED / DoN Military Authorities Recognition Questionset, dated 2013

The DoN has equivalent airworthiness authorities and processes. The policies, responsibilities, processes and procedures for airworthiness qualification of aircraft are at reference A. The Commander, Naval Air Systems Command (AIR-00) is the AA for all aircraft owned and/or operated by the Navy and Marine Corps. The Research and Engineering Group (AIR-4.0) is designated as the TA for all DoN aircraft and the Airworthiness Directorate (AIR-4.0P) is the single authority for the issuance of flight clearances for all DoN aircraft, which provides direction and executes the day-to-day airworthiness process on behalf of AIR-00. AIR-00 and AIR-4.0P are organizationally independent from the operators (who report to the Commander, Naval Air Forces) and from the Program Executive Officers. AIR-00 reports to Chief, Naval Operations, while the Program Executive Officers report to the Assistant Secretary of the Navy, Research, Development and Acquisition. The Program Executive Officers manage the fleets of DoN aircraft systems through a series of Program Management Aviation offices that are responsible for the entire lifecycle of their defined aircraft types. The Program Managers and the Program Executive Officers have responsibilities to ensure airworthiness requirements have been met and to obtain an airworthiness release (flight clearance (FC)) prior to fielding an aviation system.

While AIR-4.0P conducts the airworthiness oversight required for type certification, the DoN does not issue type certificates specifically. AIR-4.0P establishes the airworthiness of new or modified aircraft type through a combination of requirements documents and engineering review processes. The Program Management Aviation offices provide evidence of compliance with requirements to AIR-4.0P for review, analysis and approval. AIR-4.0P then issues and renews FCs or airworthiness assessments on behalf of AIR-00. All FCs requiring acceptance of unusual risk require documentation of acceptance of that risk prior to FC release. The FC is functionally equivalent to an MTC and a certificate of airworthiness within the DoN system. Modifications impacting airworthiness require issuance of an updated FC. AIR-4.0P is constantly issuing, renewing and validating the airworthiness of aviation systems through the process of issuing FCs, and it has the authority to cancel, revoke, not issue or otherwise amend any FCs.

Naval Air Systems Command has training and education programs to maintain a consistently qualified workforce. New engineering hires are part of the Engineer, Scientist Development Program that formalizes mentoring and rotational opportunities during their first three to four years. Standard skill packages provide employees with progression requirements to ensure appropriate training and competence. SMEs and other employees required to execute a role within the airworthiness process are trained and empowered to do so upon completion of additional training as defined by AIR-4.0P. This includes an Airworthiness Process and Procedures course and instruction on airworthiness issue resolution. Key AIR-4.0P personnel and chief engineers are trained and empowered to manage the flight clearance processes and release DoN flight clearances. Similarly, Test Wing Flight Officers and other specific individuals are certified by AIR-4.0P to release certain limited flight clearances. AIR-4.0P manages the certification and recurring training requirements of all personnel empowered as airworthiness authorities.

The US DoN currently does not issue military certificates of registration or certificates of airworthiness for individual tail numbers. DCMA acts as the Navy’s trusted agent in providing oversight at the production facility and verifies that the delivered aircraft meet Navy engineering design and configuration management standards. DCMA accepts the aircraft from the contractor on behalf of the Navy and the aircraft is brought into the Aircraft Inventory Readiness and Reporting System, which provides the naval aviation community with up-to-date and consistent aircraft inventory. While the DoN does maintain a registry of aircraft in the inventory, the AA does not have direct control over the aircraft registry process or maintenance of the registry.

USAF

References:

• Air Force Policy Directive 62-6, USAF Airworthiness, dated June 11, 2010
• Air Force Instruction 62-601, USAF Airworthiness, dated June 11, 2010
• USAF Brief on Airworthiness Policy and Implementation, undated
• USAF Airworthiness Authority Assessment Questions, dated September 7, 2011

In 2010, the USAF amended its airworthiness process to bring it more in line with that of the US Army and Navy and worldwide aviation practice. The new USAF airworthiness policy is documented in references A and B, and summarized in references C and D. A key change was the introduction of a USAF TAA independent of the aviation system program management offices and operational command chains. The TAA is the Director of Air Force Life Cycle Management Center Engineering within Air Force Materiel Command. The TAA defines and enforces airworthiness standards, approves certification basis, makes findings of compliance, issues flight authorizations and MTC and chairs the USAF Airworthiness Board. The Airworthiness Board is composed of senior engineering technical personnel and its main function is to assist the TAA in making airworthiness determinations. The Airworthiness Board is supported by airworthiness staff and accredited SMEs who interface with Air Force Materiel Command System Program Offices (SPO) and are involved in detailed reviews of compliance with SPO staff.

To avoid becoming a bottleneck, the independent TAA airworthiness determination and certification is intended to focus on risky or non-routine items. More routine airworthiness determinations are managed by delegation to Delegated Technical Authorities. These are individuals assigned to senior SPO engineering positions who have met accreditation requirements defined by the TAA. They are empowered to exercise certain airworthiness authorities in addition to executing their regular program responsibilities. Individuals assigned to Chief Engineer and Director of Engineering positions are required to obtain accreditation as a Delegated Technical Authority. Delegated Technical Authorities can approve modifications that do not have a significant airworthiness impact.

For new aircraft type designs and major modifications, the certification basis must be defined by the SPO Program Manager and approved by the TAA. Once the Program Manager demonstrates compliance to the certification basis, the TAA makes findings of compliance and issues MTCs for the type design. The Program Manager can then issue military certificates of airworthiness for individual aircraft based on confirmation from the Delegated Technical Authority of product compliance with the MTC. Thereafter, the Program Manager ensures the airworthiness of fielded aircraft throughout their lifecycle. The Program Manager is also the acceptance authority for medium/low safety risks. If necessary, and if an MTC is not feasible, the TAA can issue flight authorizations when the required risk assessment, risk mitigation and risk acceptance have been accomplished and documented by the Program Manager in coordination with the operational major commands.

In terms of operational airworthiness and risk management, all three US military departments are very similar. In all three cases, there is no explicit OAA, although the USAF does recognize operational airworthiness as the responsibility of the major commands. Instead, there is reliance on the fact that departmental regulations require operational commanders to comply with all published procedures necessary to maintain assigned aircraft in an airworthy condition. All operational and maintenance units are subject to frequent independent no-notice inspections and audits to ensure compliance with approved aircraft configurations and continued airworthiness requirements.

With respect to risk management, all of the departments use the DoD System Safety Risk Acceptance Process (documented in MIL-STD-882 and DoD I 5000.2) to identify and mitigate all hazards that have the potential to injure or kill personnel, damage or destroy equipment or otherwise impact mission effectiveness. Program Managers must identify hazards, assess the risks, mitigate the risks to acceptable levels and then report on the status of residual risk acceptance decisions at technical reviews and at the appropriate management levels. The user representative must be part of this process and provide formal concurrence prior to all serious and high-risk acceptance decisions. Low and medium risks can be accepted at the Program Manager level, serious risks at the two-star Program Executive Officer level, and high risks at the Assistant Secretary level. The risk acceptance process is applicable for the entire lifecycle of a system.

While each of the departments has an agency that investigates aircraft accidents and makes available FS information related to airworthiness, none of them has a formal airworthiness role. These investigation agencies are independent of the airworthiness authorities and the operational units, although the airworthiness authorities often support them with technical expertise.

A Civilian Regulatory Agency: Transport Canada

References:

• CARs
• TC Aeronautical Information Manual, October 16, 2014

For civil aviation in Canada, as is the case in most if not all countries, the airworthiness regulator and implementer are totally independent from each other. TC is a typical civilian aviation transportation regulatory agency, responsible for promulgating, regulating and enforcing aviation safety and airworthiness requirements for all civilian aircraft in Canada. TC is responsible for the CARs and their associated standards that govern civil aviation in Canada. Airworthiness requirements are described in Part V of the CARs. TC is also concerned with developing and administering policies and services for transportation systems in Canada with respect to the federally regulated aviation transportation mode. Implementers, typically airlines and other civil aviation manufacturers and operators, conduct the aviation activities associated with the engineering, manufacture, maintenance, materiel support and operation of aeronautical products. TC is also responsible for all aspects of licensing of pilots and flight engineers in Canada, as well as establishing regulations, standards, policies and procedures pertaining to Aircraft Maintenance Engineer licensing and training. Finally, the role of the investigator rests with the TSB, an agency that became independent of TC in 1990. Unlike military aviation in Canada, these three key airworthiness roles (i.e., regulator, implementer and investigator) are independent in the case of civil aviation in Canada and in most developed countries.

TC issues Canadian type certificates to certify that the type design of an aircraft meets the applicable airworthiness, noise and engine emission standards. Following type certification and production, TC issues a flight authority in the form of a Certificate of Airworthiness for aircraft that conform to a certified type design and are safe for flight. Aircraft flown in Canada must then be registered in accordance with Part II of the CARs or under the laws of an International Civil Aviation Organization member state, or a state that has a bilateral agreement with Canada concerning interstate flying. Canadian nationality and registration marks for new or imported aircraft are issued upon request by the appropriate TC regional office. TC exercises regulatory oversight to ensure the continuing airworthiness of Canadian-registered aircraft. Corrective action engineers, specialist engineers, technical inspectors and associated technical support staff take the corrective action necessary to resolve in-service aircraft airworthiness issues. This is achieved in part by the development, issuance and distribution of Airworthiness Directives and the approval of alternate means of compliance.

The Minister of Transport empowers certain persons to act on his/her behalf. Such delegates exercise authority in many different fields of aviation. For design approval of aeronautical products, delegation is shared between the regions and the Aircraft Certification Branch in headquarters. The regions are responsible for delegates who are primarily involved in the modification and repair of aeronautical products. The Aircraft Certification Branch is responsible for delegates who are primarily involved in the design and manufacture of aeronautical products and for the appointment and management of all flight test delegates.

Consistent with Civil Air Aviation Directives, TC provides training to its employees to develop their core, business and job-specific competencies prior to full delegation of authority. Training to maintain and enhance competencies is also provided. TC also schedules and delivers courses to provide aircraft certification engineers and technologists, DARs and approved persons/candidates (from delegated organizations) with a comprehensive foundation in the regulatory framework and with procedures and responsibilities governing the certification of aeronautical products in Canada. These courses provide required knowledge and skills used by regulatory engineers and industry delegates in carrying out their certification functions.

Through its CARs, TC mandates the requirement for a Safety Management System to specified certificate holders. The Safety Management System provides the framework for continuous improvement of aviation safety and is intended to provide an organization with the capacity to anticipate and address safety issues before they lead to an incident or accident. It also provides management with the ability to deal effectively with accidents and near misses so that valuable lessons are applied to improve safety and efficiency.

The TSB is responsible for investigating all aviation occurrences in Canada involving civil aircraft registered both in Canada and abroad. Its mandate is to advance transportation safety through the conduct of independent investigations, the identification of safety deficiencies and the making of recommendations to eliminate or reduce such deficiencies. When the TSB investigates an accident, no other federal department (except DND and the Royal Canadian Mounted Police) may investigate for the purpose of making findings as to the causes and contributing factors of the accident. TC may investigate for any other purpose, such as regulatory infractions.

Best Practices of Military Airworthiness Programs

A truly independent military airworthiness regulatory authority: Like its civilian counterpart, the military regulatory authority, responsible for developing and enforcing the technical and operational airworthiness regulations, is independent of the technical and operational implementers. While some might argue that the UK was able to implement this because of its large defence establishment, it should be noted that independent MAAs have also been successfully established by much smaller nations, such as the Netherlands.

Strong letters of designation: The text of letters of designation for airworthiness authorities clearly state their legal and personal accountability and responsibility for air safety.

Qualified personnel: The minimum/mandatory experience and qualifications required for operational and technical airworthiness authorities and staff are identified. The qualifications of personnel assigned to these positions is formally documented and endorsed, preferably by an independent regulatory agency, prior to assuming airworthiness responsibilities.

Sufficient training resources: Training resources are sufficient to ensure technical and operational personnel can obtain the necessary qualifications before assuming airworthiness responsibilities. Basic airworthiness training for subordinate personnel involved in airworthiness is also provided.

Independence of regulators and implementers: The OAR and OAA should be independent from each other, thereby clarifying individual airworthiness responsibilities and reducing the potential for conflicts of interest. Similarly, the TAR should be independent of the organizations responsible for aircraft acquisition and continuous airworthiness.

AA and OAR airworthiness resources: The AA and OAR should have the staff resources required to assist and represent them regarding airworthiness policy, airworthiness management, airworthiness training policy, RTS, maintenance of the aircraft register and regulatory audit functions.

Mandatory airworthiness qualifications: The minimum experience and qualifications required for operational and technical airworthiness positions should be specified. The qualifications of personnel assigned to airworthiness positions should be formally documented and endorsed, preferably by an independent regulatory agency, and preferably prior to assuming airworthiness responsibilities.

Airworthiness training: Training resources should be sufficient to ensure technical and operational personnel can obtain the necessary qualifications before assuming airworthiness responsibilities. Training resources should also be sufficient to train subordinate personnel involved in airworthiness.

Independent Airworthiness Boards: An independent Airworthiness Board to annually review the airworthiness of each aircraft fleet would provide increased assurance of fleet airworthiness and overall airworthiness management. This is considered especially important where there is a lack of independence between the regulatory and implementer authorities, such as is the case within DND.

Table E-1 provides a comparison of the different military airworthiness programs discussed. TC is also included as an example of a typical civilian regulatory agency.

Table E-1. Comparison of Airworthiness Programs. This table compares different features of DND/CAF, TC and some allies’ airworthiness programs.

Annex F—Evaluation Methodology and Limitations

1.0 Methodology

1.1 Overview of Data Collection Methods

The evaluation of the DND/CAF AWP included the use of multiple lines of evidence and complementary research methods to strengthen the rigour and reliability of the assessment. The methodology used a consistent approach when collecting and analyzing data to help ensure the reliability of the evaluation findings, conclusions and recommendations. Quantitative and qualitative data collection methods were used and included reviews of literature and program documents, key informant interviews and a benchmark analysis. Following data collection and analysis, preliminary evaluation findings were presented to the key stakeholders. Discussions from these presentations helped to further refine and clarify the findings and recommendations that are presented in this report.

1.1.1 Literature and Program Document Review

A review of program documents was conducted in the initial phase of the evaluation to establish an understanding of the background and context of the DND/CAF AWP. These documents included the following:

• federal/departmental accountability documents;
• strategic and operational program documents (i.e., orders, directives, briefing notes);
• regulatory documents (i.e., DND/CAF AWP manuals);
• website contents;
• guidance documents and process and procedure manuals;
• previous internal and external assessment reports;
• relevant academic literature and publications; and
• program products/outputs.

The document review was integral to assess the relevance of the program and to support performance findings from other lines of evidence.

1.1.2 Key Informant Interviews

While there was no formal advisory group, consultations were conducted at different phases in the evaluation’s planning phase with key stakeholders (i.e., Level 2, director general or director level) and other staff officers and informants from the C Air Force organization, 1 CAD and DGAEPM. These consultations focused on project scope and evaluation frameworks (e.g., logic model) for the project. Informants were consulted throughout this process to facilitate the identification of key documents, personnel and SMEs. Stakeholders were periodically briefed on the progress of the evaluation and its findings.

Interviews were conducted with DND/CAF personnel directly and indirectly involved in the AWP. Upwards of 60 interviews were conducted with relevant personnel within the C Air Force staff, 1 CAD, 2 CAD, DGAEPM, DGMPD (Air) and various wings/units. These interviews were used to discuss the relevance and performance of the AWP and to gather evidence of any issues affecting the program. Interviews also provided context and elaboration of trends observed in the program data. Information gathered from the interviews was cross referenced against documentation to assess performance.

1.1.3. Benchmark Analysis

A benchmark analysis was conducted to compare the DND/CAF AWP to the programs of some allied nations. The countries of Australia, the UK and the USA were chosen because of their similar principles as allied countries and the availability of open source information describing their processes. In addition, TC was reviewed as a representative civilian airworthiness authority. This was used to inform and illustrate key similarities and differences between the DND/CAF AWP, allied countries and TC and to summarize their best practices. Suggestions for improvement of the DND/CAF AWP based on these best practices were noted in applicable sections of the report.

2.0 Limitations

The following table shows the limitations related to the data sources used to assess the program and the mitigation strategies applied to them.

Table F-1. Evaluation Limitations and Mitigation Strategies. This table shows the limitations related to the data sources used to assess the program and the mitigation strategies applied to them.

Annex G—Logic Model

Annex H—Management Action Plan

Airworthiness Authority Preamble

The DND/CAF continues to deliver a world-class AWP that is designed to ensure military aviation is conducted in a fashion that is as safe as reasonably practicable. The Programme’s great success is a direct result of the highly professional personnel who continuously strive to make Canada a leader in military airworthiness amongst our allies. The Programme enables the RCAF to meet its operational mandate while mitigating any known risks. In order for the DND/CAF to continue to deliver such a high quality Programme, there remains an ongoing requirement for assessment in order to continue to strengthen it. As such, the Comd RCAF, as the AA, formally requested that ADM(RS) conduct an evaluation of the AWP.

The evaluation report indicated excellent results, but there are always opportunities for improvement in a Programme of this magnitude. The evaluation report provides crucial insight into the areas where the DND/CAF could better refine the AWP by identifying 56 findings and 56 recommendations across several broad areas. Of the 56 recommendations, 6 have been selected to be addressed through management action plans, which will be tracked through the well-established ADM(RS) reporting process. The remaining 50 recommendations will be addressed through the AWP’s self-assessment and oversight mechanism, the AAB.

The AWP continues to meet similar challenges facing the entirety of the DND/CAF as aging demographics leads to a loss of expertise. The key to combating this “brain drain” is to both ensure that the AWP remains institutionalized in our culture and that we have a robust training plan to ensure we pass along current airworthiness knowledge to the future DND/CAF. Furthermore, as the technical complexity of our fighting systems evolves and challenges our current airworthiness practices, the Programme must become more agile and integrated. The demand for airpower will remain high and a robust and first-rate AWP must remain relevant and effective to deliver air effects across the full spectrum of warfare.

Recently, the AWP has experienced several key successes. Most importantly, the departments of defence of both the US and Australia have recognized the strength and validity of the DND/CAF AWP comparable. This was accomplished through interoperability mechanisms such as the Air and Space Interoperability Council and NATO airworthiness working groups, during which best practices have been shared. Programme commonality amongst key allies helps reduce the extensive test and evaluation requirements for certain projects, lowering costs and potentially accelerating project timelines while maintaining high airworthiness standards. Additionally, the AWP has been adjusted to adapt to emerging technologies such as unmanned aerial systems. Unlike manned aircraft, the airworthiness of the unmanned aerial systems in respect to occupants is not required, but there are many other areas where risk must be mitigated to ensure mission success. We continue to remain flexible in our application of our AWP to support the current and future needs of the CAF.

The DND/CAF AWP continues to set a very high standard in ensuring that air operations are conducted at the appropriate level of risk to meet the needs of the GC. This evaluation and its resultant findings and recommendations are significant contributions to the continued success of the Programme and will shape it for years to come, enabling the RCAF to deliver agile and integrated airpower to support CAF operations at home and abroad.

M.J. Hood

Lieutenant-General

Comd RCAF

DND/CAF Airworthiness Authority

ADM(RS) Recommendations and AWP Management Action Plans

Management Action

The AA will review current oversight and regulatory mechanisms and explore options to establish an independent oversight and regulatory body.

OPI: AA

Target Date: November 2017

Management Action

The AA will establish a minimum training standard for all personnel working with direct responsibilities in the airworthiness program to include ‎an informative training plan for personnel with indirect responsibilities in the airworthiness process.

OPI: AA

Target Date: September 2017

Management Action

The OAA will conduct a review of all processes and provide an update on any required improvements to the AA. This is currently already underway.

OPI: AA

OCI: OAA

Target Date: January 2017

Management Action

The Airworthiness Coordination Cell will develop a PMF in order to measure, monitor and manage the performance, issues and improvements of the AWP.

OPI: AA

Target Date: January 2017

Management Action

The Airworthiness Coordination Cell will establish full-time and part-time positions that will be dedicated solely to support the AA and all other airworthiness authorities.

OPI: AA

Target Date: September 2017

Management Action

The AA, in collaboration with Canadian Joint Operations Command Headquarters, will develop an emergency response plan in the form of a contingency plan to detail the various tasks and responsibilities to respond to a major DND aircraft accident. As part of this effort, the AIA will also be involved to provide expert advice on the investigative requirements and responsibilities.

OPI: AA

OCI: AIA

Target Date: September 2017

Additional Management Action

The AA will also ensure that all the other evaluation recommendations are addressed internally through the AWP.

OPI: AA

Target Date: November 2017

___________________________________________________________________________________________________________________________

Footnote 1 DND/CF AWP Manual (A-GA-005-000/AG-001), part 1, section 1, paragraph 4, dated May 9, 2011.

Footnote 2 That is, performing two roles.

Footnote 3 Request for Evaluation of DND/CAF AWP (C Air Force letter), dated April 16, 2008.

Footnote 4 DND/CF AWP Manual, Figure 1-1-2, dated September 5, 2011.

Footnote 5 CRS is the former designation of ADM(RS), the latter having come into effect on May 13, 2015.

Footnote 6 CRS. Audit of Airworthiness, May 1993.

Footnote 7 Memorandum from the MND to the CDS and the Deputy Minister (DM), dated September 16, 1998.

Footnote 8 Ministerial Instructions on Airworthiness, July 18 2005.

Footnote 9 CRS. Review of the DND/CF Airworthiness Risk Management Process, April 2008.

Footnote 10 The Air Cadet fleet is owned by the Air Cadet League, but operated by the DND. TC classifies the Air Cadet operation as a civil enterprise subject to civil airworthiness regulations and provides airworthiness oversight. However, since the fleet is operated by the DND, the Air Cadet Gliding Program is also being monitored under the DND/CAF AWP, and work is underway to establish the foundational elements to issue a DND airworthiness clearance by November 2016.

Footnote 11 Ministerial Direction Regarding DND/CF Airworthiness Program, dated September 16, 1998.

Footnote 12 Aeronautics Act (R.S.C., 1985, c. A-2) Section 4.2: Responsibilities of Minister. http://laws-lois.justice.gc.ca/eng/acts/A-2/page-2.html#h-5. Last consulted on March 24, 2015.

Footnote 13 Canada First Defence Strategy. http://www.forces.gc.ca/en/about/canada-first-defence-strategy.page. Last consulted on March 24, 2015.

Footnote 14 CDS Order, dated July 28, 2008.

Footnote 15 Charles Haddon-Cave, QC. Nimrod Review – An Independent Review into the Broader Issues Surrounding the Loss of the RAF Nimrod MR2 Aircraft XV230 in Afghanistan in 2006, October 28, 2009.

Footnote 16 CRS. Review of DND/CF Airworthiness Risk Management Process, April 2008.

Footnote 17 CDS Order, dated July 28, 2008.

Footnote 18 DND/CF AWP Manual.

Footnote 19 CDS Order, dated July 28, 2008.

Footnote 20 Email from TC National Simulator Program, dated November 30, 2012.

Footnote 21 Ken Mansfield. Review of the DND Technical Airworthiness Program, dated November 4, 2011.

Footnote 22 Minutes of the 2014 AAB, dated December 17, 2014.

Footnote 23 Record of Discussion, CAF Unmanned Aircraft System Working Group – February 27, 2015, dated April 2015.

Footnote 24 The AA is normally represented at the meeting by a representative from Director Air Readiness and Plans, who is responsible for airworthiness coordination.

Footnote 25 DND/CF AWP Manual.

Footnote 26 ibid.

Footnote 27 Technical Airworthiness Manual (TAM) (C-05-005-001/AG-001), dated June 22, 2012.

Footnote 28 OAM (B-GA-104-000/FP-001), dated February 20, 2013.

Footnote 29 Airworthiness Risk Management Process (DG01.003), dated November 6, 2012.

Footnote 30 TAM, Figure 5-1-2-7.

Footnote 31 OAM, Table 5-8.

Footnote 32 DND/CF AWP Manual.

Footnote 33 ibid.

Footnote 34 Manual of Aerospace Procedures, Annual ARB (TAA01.003).

Footnote 35 TAM, Figure 5-1-2-7.

Footnote 36 OAM, Table 5-8.

Footnote 37 Airworthiness Risk Management Process.

Footnote 38 AWRM Database User Guide, December 12, 2011.

Footnote 39 While the DGAEPM database is universally recognized as the master, 1 CAD maintains a separate database that is updated monthly with new information from the DGAEPM database. The 1 CAD database also includes about 200 older closed RARMs that were not included in the DGAEPM database.

Footnote 40 The 1 CAD database actually listed 85, and the DGAEPM database listed 101, the difference being primarily due to some risk index discrepancies and draft RARMs within the DGAEPM database.

Footnote 41 DTAES 2-3. Minutes of the Fall 2013 Executive ARB, dated February 5, 2014.

Footnote 42 Airworthiness Risk Management Process.

Footnote 43 OAM

Footnote 44 ibid, Table 5-8.

Footnote 45 The AA is the OCRAA for extremely high risks, and the OAA is the OCRAA for high and medium risks.

Footnote 46 OAM, Annex 5E.

Footnote 47 Haddon-Cave. Nimrod Review, op. cit., October 2009.

Footnote 48 DAOD 2015-0 – Airworthiness. Last modified September 11, 2014.

Footnote 49 DAOD 2015-1 – DND/CAF Airworthiness Programme. Last modified September 11, 2014.

Footnote 50 DND/CF AWP Manual.

Footnote 51 Air Force Order 8001-2, Release to Service – New and Modified Aircraft Fleets. Last modified August 3, 2010.

Footnote 52 DND/CF AWP Manual, Figure 2-1-1.

Footnote 53 TAM, Figure 5-1-2-3.

Footnote 54 OAM, Table 5-3.

Footnote 55 Air Force Order 8001-2.

Footnote 56 http://www.forces.gc.ca/en/business-regulations-technical-airworthiness/index.page. Last consulted on April 15, 2015.

Footnote 57 TBS. Directive on the Evaluation Function – 6.2 Responsibilities of Program Managers. http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681&section=text#sec6.2. Last consulted on June 30, 2015.

Footnote 58 Mansfield Report – Closing Action, August 2013.

Footnote 59 CDS Order, dated December 20, 2015.

Footnote 60 DND/CF AWP Manual.

Footnote 61 DGAEPM Procedures (TAA01.007-01).

Footnote 62 TAM.

Footnote 63 DGAEPM Fiscal Year 2015/16 Business Plan, Annex A.

Footnote 64 DTAES Strategic Plan 2012.

Footnote 65 DTAES Balanced Scorecard Dashboard FY 2014/15 Q2.

Footnote 66 TAM, Part 5, Chapter 1.

Footnote 67 Airworthiness Risk Management Process.

Footnote 68 DTAES Strategic Plan 2012.

Footnote 69 Manual of Aerospace Procedures.

Footnote 70 Minutes of the 2014 AAB, dated December 17,2014.

Footnote 71 BNH Expert Software Inc. Airworthiness Training Needs Analysis, dated March 27, 2014.

Footnote 72 DTAES tracks audit progress and client feedback via its quarterly Balanced Scorecard, and feedback is generally very positive for both DND and contractor organizations.

Footnote 73 Nimrod Review – DND Assessment Report, February 23, 2011.

Footnote 74 Mansfield Report – Closing Action, August 2013.

Footnote 75 Annex A to Mansfield Report – Closing Action (2182D-1027-810-01), August 2013.

Footnote 76 EMT: Equipment Management Team.

Footnote 77 OPI: Office of Primary Interest.

Footnote 78 PAESS: Professional Airworthiness and Engineering Support Services.

Footnote 79 DAEPM: Director Aerospace Equipment Program Management.

Footnote 80 CFSATE: Canadian Forces School of Aerospace Technology and Engineering.

Footnote 81 AEOBC: Aerospace Engineering Officer Basic Course.

Footnote 82 DLN: Defence Learning Network.

Footnote 83 ACP-T: Airlift Capability Project – Tactical.

Footnote 84 FMS: Foreign Military Sales.

Footnote 85 ASIC: Air and Space Interoperability Council.

Footnote 86 AWIL: AgustaWestland International Limited.

Footnote 87 NFTC: NATO Flying Training in Canada.

Footnote 88 Refers to Director Aerospace Equipment Program Management (Fighters and Trainers).

Footnote 89 D Air CFG: Director Air Contracted Force Generation.

Footnote 90 BMAT: Bombardier Military Aviation Training.

Footnote 91 Role of RARM Process within Larger Framework of Certification Process.

Footnote 92 DGMSSC: Director General Materiel Systems and Supply Chain.

Footnote 93 CDS Order, dated December 17, 2013.

Footnote 94 Memorandum of understanding between TC and DND (DTAES 3-6) (2182D-1027-835-2-5).

Footnote 95 OAM.

Footnote 96 Briefing Note for OAA on Approving Foreign Military Aircraft Overflights, dated December 11, 2009.

Footnote 97 1 CAD Orders, Volume 3, 3-401 Foreign Military Aircraft Overflight Process.

Footnote 98 Some overflights need to be approved within hours and the available OA staff occasionally have to resort to open internet sources to try to assess country/aircraft/aircrew background.

Footnote 99 TC. Foreign Air Operator Certification and Inspection Manual, dated June 1998.

Footnote 100 North American Aerospace Defence Command.

Footnote 101 North Atlantic Treaty Organization.

Footnote 102 For example: Exercise Maple Flag, Maritime Command Operational Training and OPEN SKIES flights.

Footnote 103 Market Wired. http://www.marketwired.com/press-release/canada-welcomes-international-forces-to-exercise-maple-flag-46-1795273.htm. Last consulted on May 4, 2015.

Footnote 104 1 CAD Orders, Volume 3, 3-401 Foreign Military Aircraft Overflight Process.

Footnote 105 OAM.

Footnote 106 In this context, approval essentially indicates TA and OA endorsement of the content of the RARM, whereas acceptance essentially indicates that an authorized Operational Command representative endorses flight operations given the airworthiness risk documented in the RARM.

Footnote 107 OAM, Table 5-8.

Footnote 108 OAM.

Footnote 109 Director Air Program. Item III, dated October 23, 2013.

Footnote 110 2011/12 ARB Annual Airworthiness Report (Technical).

Footnote 111 OAM.

Footnote 112 1 CAD Orders, Volume 3, 3-310 Operational Risk Management for Air Operations.

Footnote 113 1 CAD Orders, Volume 3 – Foreign Military Aircraft Overflight Process.

Footnote 114 Memorandum of understanding between TC and DND (DTAES-3-6).

Footnote 115 DND/CF AWP Manual, Figure 1-4-1.

Footnote 116 Email from TC National Simulator Program, sent November 30, 2012.

Footnote 117 Bill C-3, chapter 29, dated 2014.

Footnote 118 Working Arrangement Between TSB and DFS, dated April 9, 2009.

Footnote 119 Occurrence Investigation Techniques for the CF (A-GA-135-002/AA-001), Chapter 1.

Footnote 120 Convention on International Civil Aviation. Annex 13 – International Standard and Recommended Practices.

Footnote 121 Flight Safety for the Canadian Forces (A-GA-135-001/AA-001).

Footnote 122 DFS. CH147202 Chinook – Flight Safety Investigation Report, paragraph 3.1.1.

Footnote 123 DND/CF AWP Manual, Part 1, section 4, paragraph 6.

Footnote 124 ibid.

Footnote 125 DND/CF AWP Manual, Part 1, section 1, paragraph 11c.

Footnote 126 FSIRs are designed to provide a comprehensive report on an occurrence, are usually produced by a team of investigators chosen for the particulars surrounding the occurrence and are intended to be completed within twelve months. Source: Airworthiness Investigative Manual, Chapter 3 – Investigation Classification System: Report Types.

Footnote 127 International Civil Aviation Organization. Annex 13 – Incident Reporting, Data Systems and Information Exchange. http://www.icao.int/safety/ism/ICAO%20Annexes/Annex%2013.pdf. Last consulted on April 13, 2015.

Footnote 128 DND/CF AWP Manual, Part 1, section 1, paragraph 11c.

Footnote 129 CDS order Airworthiness Investigative Authority, dated July 28, 2008.

Footnote 130 Airworthiness Investigative Manual (AIM) (A-GA-135-003/AG-001), Chapter 3, paragraph 8, dated November 26, 2009.

Footnote 131 Source: RCAF Investigation Reports. http://www.rcaf-arc.forces.gc.ca/en/flight-safety/investigations-list.page?cid=1372838830907&page=1#1372838830907. Last consulted on October 19, 2016.

Footnote 132 AIM, chapter 3, paragraph 5.

Footnote 133 AIA/DFS 2012/2013 Annual Report.

Footnote 134 2013 AIA Annual Report, Director’s Comments. http://www.rcaf-arc.forces.gc.ca/en/flight-safety/article-template-statistical-report.page?doc=2013-annual-report-airworthiness-investigative-authority-and-flight-safety-program-s-activities/hyyl14ow. Last consulted on June 4, 2015.

Footnote 135 AIM, chapters 4 and 5.

Footnote 136 ibid, chapter 5.

Footnote 137 Flight Safety Course Validation Survey, slide 27, February 2014.

Footnote 138 TBS. Directive on the Evaluation Function, section 6.2.

Footnote 139 2013 AIA Annual Report.

Footnote 140 Air Force Life Cycle Management Centre / Engineering Services Directorate.

Footnote 141 For comparison purposes, training of investigators is not included since only DND includes the AIA in its AWP.