Follow-up on Internal Audit: Fixed-Wing Search and Rescue (FWSAR) Project
Alternate Formats
Assistant Deputy Minister (Review Services)
March 2014
7050-11-48 (CRS)
Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.
Caveat
The result of this work does not constitute an audit of the FWSAR project. Rather, this report was prepared to provide reasonable assurance that the Management Action Plans (MAP) that resulted from the May 2009 FWSAR project audit were implemented as stated and as such have addressed the associated recommendations.
- ADM(Mat)
-
Assistant Deputy Minister (Materiel)
- CID
-
Capability Investment Database
- Comd
-
Commander
- CRS
-
Chief Review Services
- DND
-
Department of National Defence
- FWSAR
-
Fixed-Wing Search and Rescue
- ISSC
-
In-Service Support Contract
- L1
-
Level 1
- MAP
-
Management Action Plan
- ADM(Fin CS)
-
Assistant Deputy Minister (Finance and Corporate Services)
- OPI
-
Office of Primary Interest
- PAD
-
Project Approval Directive
- PO&M
-
Personnel, Operations and Maintenance
- RCAF
-
Royal Canadian Air Force
- SOR
-
Statement of Requirements
- SRB
-
Senior Review Board
- SSI
-
Statement of Support Intent
- VCDS
-
Vice Chief of the Defence Staff
- AERMS
-
Audit and Evaluation Recommendation Management System
1.0 Introduction
In keeping with the Treasury Board Policy on Internal Audit,1 Chief Review Services (CRS) is required to undertake audit follow-ups to assess the implementation status of MAPs developed in response to previous CRS audit recommendations. In accordance with the CRS Risk-Based Audit Plan for Fiscal Year 2013/14 to 2015/16, this audit follow-up was selected to ensure that some systemic capital acquisition process issues, as well as concerns specific to the FWSAR project, have been addressed.
In May 2009, CRS completed an audit of the ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ FWSAR project. The objective of the audit was to provide assurance that the FWSAR project had effective governance, risk management and control frameworks in place to ensure a cost-effective and timely operational capability.
The May 2009 audit concluded that improvements were needed in identifying capability deficiencies, project risk management, as well as departmental governance for projects. Nevertheless, good management practices had been observed. The project options analysis included a wide range of options, including a lease option, to address funding limitations. The Statement of Requirements (SOR) was developed based on independent research in collaboration with internal stakeholders. The findings of the 2009 audit are summarized as follows:
Project Governance. To fully adhere to the requirements of the Project Approval Guide,2 a Senior Review Board (SRB) checklist for the project was to be submitted prior to every SRB meeting. Further, the SRB oversight needed to improve when the FWSAR project was reactivated, in order to ensure that a broad range of issues would be brought to the attention of senior management.
Capability Deficiency. Project approval documents had not fully addressed the fleet capability deficiencies with respect to impact on operations. Capability deficiencies such as fleet availability, load capacity, and fleet maintenance costs due to project delays, were not included in the SOR or any other key project document.
Statement of Requirements. Although there was sufficient evidence to support the FWSAR project’s mandatory requirements in the October 2006 version of the SOR, some capabilities were understated. There could have been more detailed descriptions for items such as flying speed requirements, manoeuvrability, fleet size, and cargo size. Capability shortfalls were attributed to limited departmental guidance on the development of mandatory requirements in the SOR.
In-Service Support Concept. The 2005 FWSAR Statement of Support Intent (SSI) needed to be revised to align with the Department of National Defence (DND)’s In-Service Support Contract (ISSC) framework. While the FWSAR will operate out of the main operating bases in Canada, the intention of the Canadian Armed Forces performing first- and second-line support was not consistent with three other “critical role” fleets,3where all support had been outsourced for non-deployable aircraft.
Financial Management. The estimated savings in personnel, operations and maintenance (PO&M) of ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ in the FWSAR project submission were overstated because some of the assumptions used in the calculation were inaccurate. In addition, cost validation was provided only on project acquisition PO&M costs, but not on recurring post-acquisition costs.
Risk Management. The FWSAR project did not have a detailed risk management plan to manage risks in accordance with DND risk management guidelines.
In order to address these issues, six recommendations were put forward. The Vice Chief of the Defence Staff (VCDS), Assistant Deputy Minister (Finance and Corporate Services) (ADM(Fin CS)) and Assistant Deputy Minister (Materiel) (ADM(Mat)) developed a set of MAPs to address the audit recommendations specific to their areas of responsibility.
2.0 Methodology
This audit follow-up is the outcome of a review of documentation and evidence to assess the progress made in implementing the MAP, based on the assessment criteria in Annex A. The following methods were used:
- Analysis of data from the Audit and Evaluation Recommendation Management System (AERMS)4and the Capability Investment Database (CID).
- Interviews with personnel from the FWSAR project office and VCDS, ADM(Mat) and ADM(Fin CS) staff.
- Review of key project documentation and policies pertaining to the MAP.
Statement of Conformance
The audit follow-up conclusions contained in this report are based on sufficient and appropriate audit evidence gathered in accordance with procedures that meet the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing. The audit follow-up thus conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The opinions expressed in this report are based on conditions as they existed at the time of the audit follow-up, and apply only to the entity examined.
3.0 Overall Assessment
The follow-up on the May 2009 audit identified that progress has been made in some areas. For example, there have been improvements in the areas of the project’s SOR, risk management, and SSI. As well, there were improvements in the Project Approval Directive (PAD) to develop mandatory requirements.
At the time of follow-up, the project had 7 out of 11 MAPs designated as fully completed in the AERMS. Upon further examination of the 11 MAPs, three of the completed MAPs were found to require additional measures to fully address the issues, and the four MAPs that were under way were found to be complete as portrayed in the scorecard in Annex B. As for those MAPs that have yet to be fully addressed at the time of the follow-up, the majority are scheduled to be implemented by 2014 due to the systemic nature of the issues. An implementation status of the MAP items can be found in Annex C.
Until all MAPs are fully implemented, there remain risks in the following areas:
- Capability Deficiency. The absence of a more thorough options analysis (including the analysis of capability deficiencies with the status quo) will hinder the ability to identify the full impact of the limitations of the existing FWSAR aircraft in operations and training.
- Project Governance. Until all project SRBs meet on an annual basis and the CID is updated on a regular basis, this will contribute to shortfalls in project oversight.
For the purpose of monitoring the implementation of the action plans that have not yet been fully implemented, the AERMS status of these action plans will be revised by CRS to fully reflect the results of this audit follow-up.
Annex A—Assessment Criteria
Line of Enquiry: Progress made on the 2009 Audit Recommendations
Progress has been made on the implementation of the MAPs identified in the FWSAR audit tabled in 2009.
The following criteria were used to assess the level of completion for each MAP item.
1. No Progress or Insignificant Progress (0-24% complete)
No action taken by management or insignificant progress. Actions such as striking a new committee, having meetings and generating informal plans are insignificant progress.
2. Planning Stage (25-49% complete)
Formal plans for organizational changes have been created and approved by the appropriate level of management (at a sufficiently senior level, usually at the Executive Committee level or equivalent) with appropriate resources and a reasonable timetable.
3. Preparation for Implementation (50-74% complete)
The entity has begun necessary preparation for implementation, such as hiring or training staff, or developing or acquiring the necessary resources to implement the recommendation.
4. Substantial Implementation (75-99% complete)
Structures and processes are in place and integrated in some parts of the organization, and some achieved results have been identified. The entity has a short-term plan and timetable for full implementation.
5. Full Implementation (100% complete)
Structures and processes are operating as intended and are implemented fully in all intended areas of the organization.
6. Obsolete
Audit recommendations that are deemed to be obsolete or have been superseded by another recommendation.
Table Summary:
Eleven recommendations are listed in the left-hand column. For each recommendation, read across the row to determine the title of the recommendation, the MAPs, the OPI and the CRS assessment of progress on the MAP.
| Recommendation # |
MAPs |
OPI |
CRS Assessment of Progress on MAPs |
|---|---|---|---|
| 1.a) Overdue SRBs |
Projects in the CID will be reviewed for overdue SRBs; affected Level 1s (L1) will be contacted. |
VCDS |
No Progress or Insignificant Progress |
| 1.b) SRB Checklists |
Director Force Planning and Program Coordination analysts will ensure Programme Management Board submissions contain SRB checklist. |
VCDS |
Obsolete |
| 1.c) Quarterly Review of CID |
Director Force Planning and Program Coordination will implement quarterly review of CID projects to determine out of date material. |
VCDS |
No Progress or Insignificant Progress |
| 2. Capability Deficiencies |
SOR will be revised to include operational impact of maintaining status quo. |
Comd |
Preparation for Implementation |
| 3.a) Developing Mandatory Requirements |
The Department will determine where these guidelines should reside. |
VCDS |
Full Implementation |
| 3.b) SOR |
All items identified by original audit will be reviewed for latest SOR. |
Comd RCAF |
Full Implementation |
| 4.a) SSI |
The SSI is undergoing review and all necessary revisions will be made. |
Comd RCAF |
Full Implementation |
| 4.b) SSI Aligned with the ISSC Framework |
The 2005 version of SSI will be modified so that the project is aligned with current ISSC guidelines. |
ADM |
Full Implementation |
| 5.a) PO&M Costs |
The recurring PO&M costs will be reviewed to ensure accuracy of figures from both an operational and maintenance perspective. |
ADM |
Full Implementation |
| 5.b) PO&M Estimates Methodology |
The Department will explore methodologies to improve recurring PO&M estimates. |
ADM |
Substantial Implementation |
| 6. Project Risk Management |
The risk management plan will align with current departmental guidelines. |
ADM |
Substantial Implementation |
Annex C—Summary of Management Action Plan Status
Table C-1. Status of the Implementation of the MAP Items for Recommendation 1 Table Summary.
Table Summary:
The MAPs are listed in the left-hand column. For each action, read across the row to determine the OPI, the target date, the progress to date, and the status of the action item.
| MAPs | OPI | Target Date | Progress to Date | Status of Action Item |
|---|---|---|---|---|
| Recommendation 1—“For all projects, ensure project leaders ensure SRBs are held in accordance with the Project Approval Guide and that checklists are submitted accordingly. Implement measures to improve capital project information in the CID and follow up where information is not reliable.” |
||||
| 1.a) Projects in the CID will be reviewed to determine overdue SRBs. Affected L1s will be contacted to set a date for SRB completion. |
VCDS |
April 2009 |
The PAD5 states that, as a minimum, the SRB will meet at least annually to review project performance, progress, and the risk management plan. The project’s Director Defence Programme Coordination analyst is responsible to ensure that the project holds at least an annual SRB.6 An SRB health report7 completed in October 2013 showed that 52 percent of projects8 had no SRB data (down from 59 percent in 2009); 27 percent of SRBs were on schedule (up from 26 percent); and 21 percent of projects had an overdue SRB (up from 15 percent). VCDS staff will address this issue by April 2014. | No Progress or Insignificant Progress |
| 1.b) Director Defence Program Coordination analysts will ensure Annex G of Programme Management Board submissions will include SRB checklists in accordance with the Program Guidance Memo of 02/08. |
VCDS |
April 2009 |
A sample of 11 projects tabled at the Programme Management Board between January and August 2013 showed that only 1 of the 11 included an SRB checklist. Although the SRB checklists were mandatory under the Project Approval Guide, they were superseded by the PAD in October 2011, which does not explicitly require the checklists.9 | Obsolete |
| 1.c) Director Defence and Programme Coordination staff will implement a quarterly review of CID projects to determine out-of-date material, and then work with Project Leaders to provide updates. |
VCDS |
June 2009 |
Rather than undertaking quarterly reviews of the CID, the Director Defence Programme Coordination ensures that key project documents, such as the Project Brief, and Project Complexity and Risk Assessment, are entered in the CID prior to any expenditure approval of the project. Although the Force Capability Plan 2013 requires quarterly updates of project information in the CID for investment planning purposes, and ADM(Mat)10 has directed that the CID be updated monthly for equipment projects; there is no requirement under the PAD to perform quarterly reviews of the CID. The VCDS intends to provide more specific direction on the frequency of CID project information updates in the PAD depending on the project phase, by April 2014. | No Progress or Insignificant Progress |
Table C-2. Status of the Implementation of the MAP Items for Recommendation 2
Table Summary:
The MAPs are listed in the left-hand column. For each action, read across the row to determine the OPI, the target date, the progress to date, and the status of the action item.
| MAPs | OPI | Target Date | Progress to Date | Status of Action Item |
|---|---|---|---|---|
| Recommendation 2—“Ensure that the FWSAR project office includes the impact of the capability deficiencies, as well as the cost of delaying the project in the SOR, and any other pertinent documents.” Overall Assessment—Preparation for implementation |
||||
| 2. The FWSAR SOR operational impact of maintaining the status quo beyond 2010 will be revised to address the impact of maintaining these fleets beyond 2015—the current FWSAR project timeline. The Project Profile and Risk Assessment is currently being revised. |
Comd RCAF |
April 2009 |
Although the project documents have been revised, the SOR (version 6.1) and the latest Project Brief (9 February 2012) do not factor in the impact of the current capability deficiencies. For example, in terms of fleet availability, there was no mention of the effect on operations and training due to the current fleet’s reduced yearly flying rate. The SOR will be revised by April 2014. | Preparation for Implementation |
Table C-3. Status of the Implementation of the MAP Items for Recommendation 3
Table Summary:
The MAPs are listed in the left-hand column. For each action, read across the row to determine the OPI, the target date, the progress to date, and the status of the action item.
| MAPs | OPI | Target Date | Progress to Date | Status of Action Item |
|---|---|---|---|---|
| Recommendation 3—“Develop guidelines in the Project Approval Guide for project staff to develop mandatory requirements. Re-assess the requirements in the FWSAR SOR and revise the SOR and draft Statement of Interest and Qualifications accordingly.” Overall Assessment—Full implementation |
||||
| 3.a) Chief of Programme will meet with Chief Force Development to determine where these guidelines should reside. |
VCDS |
May 2009 |
The Capability-Based Planning Handbook provides a conceptual look at how to develop mandatory requirements against the capabilities that are assessed. Also, the PAD refers to this handbook as the guide to determining the capability-based planning process. | Full Implementation |
| 3.b) Re-assess the requirements in the FWSAR SOR and revise the SOR and draft Statement of Intent and Qualifications accordingly. | Comd RCAF | June 2009 | The SOR has been revised in light of the independent review done by the National Research Council and thus also the procurement strategy. A capability-based approach will be used where bidders will be required to propose the number of aircraft and basing locations to meet operational requirements. A Request for Proposal will be issued. | Full Implementation |
Table C-4. Status of the Implementation of the MAP Items for Recommendation 4
Table Summary:
The MAPs are listed in the left-hand column. For each action, read across the row to determine the OPI, the target date, the progress to date, and the status of the action item.
| MAPs | OPI | Target Date | Progress to Date | Status of Action Item |
|---|---|---|---|---|
| Recommendation 4—“In conjunction with ADM(Mat), consider updating the SSI plan to align with the ISSC framework.” Overall Assessment—Full implementation |
||||
| 4.a) The SSI is undergoing review and all necessary revisions will be made. |
Comd RCAF |
By Effective Project Approval |
The project has revised its SSI and has addressed issues relating to maintenance responsibilities and intellectual property rights. | Full Implementation |
| 4.b) In cooperation with Comd RCAF, ADM(Mat) intends to modify the 2005 version of the SSI where required. Materiel Group Program Management Committee was briefed in October 2008 that the project will be aligned with current ISSC guidelines. | ADM (Mat) | By Effective Project Approval | Defence Administrative Orders and Directives 3022-1, regarding the ISSC framework, has included the FWSAR as a critical role fleet. Therefore, DND is to perform first-line and limited second-line maintenance to ensure deployable flexibility. In accordance with the framework, the current SSI has identified the need for in-service support performance metrics, and has also indicated that DND and contractor integration will be achieved through the Defence Resource Management Information System and the Air Force Integrated Information Learning Environment. | Full Implementation |
Table C-5. Status of the Implementation of the MAP Items for Recommendation 5
Table Summary:
The MAPs are listed in the left-hand column. For each action, read across the row to determine the OPI, the target date, the progress to date, and the status of the action item.
| MAPs | OPI | Target Date | Progress to Date | Status of Action Item |
|---|---|---|---|---|
| Recommendation 5—“Revisit the recurring PO&M model and ensure accuracy of financial data in submission documents. In conjunction with ADM(Mat), validate the major assumptions for recurring life cycle PO&M costs for new combat system acquisitions.” Overall Assessment—Substantial implementation |
||||
| 5.a) The recurring PO&M costs will be reviewed during the definition phase with Director Aerospace Requirement to ensure the accuracy of the replaced fleet equipment life expectancy, phase out timelines, and assess the impact on main operating base, personnel and other elements from both an operational and maintenance perspective. Director Costing Services is currently reviewing its policy on the level of validation required concerning recurring PO&M. Director Costing Services will be requested to review and validate the costing to ensure accuracy of data, use of applicable DND financial documentation (such as the Cost Factors Manual and the Economic Model), use of compounding escalation factors, general ledgers, and other costing elements. |
ADM (Mat) |
By Project Approval Implementation |
The project has revisited its recurring PO&M costs in order to address the issues identified in the May 2009 audit. The yearly flying rate figure has been revised for a more accurate estimate of operating costs; ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ In addition, a Director Costing Services analyst dedicated to the project management office is validating all elements of the project costs, including recurring PO&M costs. |
Full Implementation |
| 5.b) Director Costing Services, in conjunction with Director Materiel Group Comptroller, will explore the methodologies to improve recurring PO&M estimates. This improvement will be incorporated in the Strategic Cost Model to support the Canada First Defence Strategy implementation. | ADM (Fin CS) | July 2010 | Since June 2013, cost analysts with project office staffs have used a cost template developed in 2011 to capture PO&M costs. These actions will enhance the rigour and accuracy of the life cycle estimates, and the cost data will eventually be incorporated in the Strategic Cost Model. CRS recommends that this MAP be closed. |
Substantial Implementation |
Table C-6. Status of the Implementation of the MAP Items for Recommendation 6
Table Summary:
The MAPs are listed in the left-hand column. For each action, read across the row to determine the OPI, the target date, the progress to date, and the status of the action item.
| Recommendation 6—“Develop an FWSAR risk management plan that incorporates the DND/Canadian Armed Forces integrated risk management guidelines, and re-evaluate the risks identified and their level of impact on the project.” Overall Assessment—Substantial implementation |
||||
| MAPs | OPI | Target Date | Progress to Date | Status of Action Item |
|---|---|---|---|---|
| 6. The FWSAR project management office is committed to incorporating a comprehensive risk management process. The risk management plan will align with current departmental guidelines, and the Project Profile and Risk Assessment has been updated. It should be reiterated that the original FWSAR project never formally entered the definition phase; therefore, the risk management process was not as rigorous as it would have been if the project had progressed. | ADM (Mat) |
By Formal Definition phase |
The project has revised its risk management plan to align with the DND Integrated Risk Management Guidelines. The risk management plan now includes clear information on addressing risks, stakeholder roles and responsibilities, and criteria for the assessment of risk impact and probability. The risk management plan is also in line with the Project Management Body of Knowledge. The project risk radar maintains and tracks project risks while allowing the project to assess impact and probability on five levels. However, there is still room for improvement, as there are inconsistencies between the project brief and the risk radar results with regards to the risk severity level. Due to the absence of the log of historical events in risk radar, it is not possible to determine the cause for inconsistent reporting of inherent risk severity levels in the project brief. Consistent reporting of risk severity assessments will be addressed by April 2014. | Substantial Implementation |
___________________________________________________________________________________________________________________________
Footnote 1 Policy on Internal Audit http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/ia-vi/ia-vi_e.asp.
Footnote 2 AERMS is a system that tracks the progress of the MAP items.
Footnote 3 The RPC is used to tell the contractor/DND facility which items should be repaired first. The RPC reflects the number of months of serviceable stock available (http://admmat.ottawa-hull.mil.ca/material/mmi/1500_e.pdf).
Footnote 4 The Maximum Repair Cost is a standard established by DND to guard against the possibility of an item being repaired at a cost that exceeds its replacement value to DND.
Footnote 5 Chapter C.15.11.10 of the PAD.
Footnote 6 Chapter A.2.1.8 of the PAD states the first SRB will be held as soon as possible after project starts but not before the Project Charter and options analysis plan are ready for approval.
Footnote 7 For the selected projects, the SRB health report takes the milestone actual date or, if null, the expected forecast date for last SRB meeting. This date is compared to last year’s meeting date.
Footnote 8 The SRB health report included all 513 projects at Options Analysis, Definition and Implementation stage at the time of the audit follow-up.
Footnote 9 PAD, section C.15.11.14, states “the project team consults the SRB checklist to ensure all necessary topics have been covered prior to SRB.”
Footnote 10 ADM(Mat) 1000-1 Memorandum, 14 August 2003.
Page details
- Date modified: