Public Services and Procurement Canada
Summary of conclusions and recommendations: Evaluation of the risk-based defence procurement pilot project

Initiative description

The risk-based defence procurement pilot (RBDPP) program allowed for a temporary delegation of authority where the minister of Public Services and Procurement Canada (PSPC) can enter into and amend contracts and contractual arrangements for lower risk and low-medium complexity defence procurements exceeding PSPC's contracting limits for the period between November 2018 and April 2020. PSPC led the implementation of the pilot, in partnership with Treasury Board of Canada Secretariat (TBS) and the Department of National Defence (DND).

Evaluation objective, scope and methodology

The RBDPP evaluation examined the program's achievement of its objectives in accordance with Treasury Board's Policy on Results. The evaluation examined the following 4 objectives:

• approval levels for defence procurements undertaken by PSPC are appropriately identified based on assessment of risk
• administrative processes are streamlined and lead to faster approvals for a large number of PSPC defence procurements
• Treasury Board (TB) is able to focus its approval on higher risk and complex contracts
• the implementation of DND's Strong, Secure, Engaged (SSE) is supported and the needs of the Canadian Armed Forces (CAF) are better met

The evaluation of the RBDPP was undertaken by the Office of Program Evaluation, PSPC, between December 2019 and May 2020. Largely, the composition of the RBDPP Program consisted of 3 stakeholder departments: PSPC, DND and TBS.

A variety of methods were used to assess the objectives. The evaluation examined the following lines of evidence; interviews, surveys, review of performance measurement data, review of operational and reporting information and, review of risk and risk mitigation assessments.

Evaluation constraints and limitations

The evaluation addressed the limitations of specific methods by triangulating findings across multiple lines of evidence in preparation of the evaluation's conclusions. Limitations included:

Interview

Interview questionnaires were partially customized to suit the role of the interviewee. As a result not all questions were asked of all interviewees and interviewees could not provide a response to all questions.

Survey

Survey participation was targeted to officials providing operational support to the pilot. The survey included a comprehensive list of questions seeking respondent's perspectives, with some questions asked of a single department. As a result of its broad distribution, the survey encountered high rates (up to 40% in some questions) of respondents stating they did not know enough about the topic. Some survey participants were also included in the interviews.

Operational and reporting information

Operational and reporting information obtained from the program were reviewed; however, the evaluation was unable to collect data to conclude on certain evaluation indicators–both due to a lack of available information as well as the need to further refine indicators used in the assessment. Of note, there was a lack of pre-pilot information. Additionally, due to operational constraints resulting from the COVID-19 pandemic, the evaluation team was unable to complete a media review and departmental correspondence review.

Review of risk and risk mitigation material review

A review of risk and risk mitigation material review was completed by a third-party. Due to cost and timeliness considerations, all materials created, consulted or referenced in preparation of the final risk assessments could not be included in the review. Despite this constraint, the evaluation team determined the sample size given was sufficient to support an analysis of tools and their application to draw conclusions. Ultimately, the reviewer was able to provide a best judgement assessment by seeking clarification where necessary.

Largely, the 2 main constraints of the evaluation were: (i) the fixed timeframe for the RBDPP implementation and (ii) the evaluation occurring 13 months into the 18 month pilot period (well before the pilot's completion).

None of the identified issues and/or constraints were significant enough to prevent evaluation reporting.

Evaluation findings

The RBDPP assessment reached the following findings on the achievement of the pilot's objectives:

Approval levels were appropriately identified

While there is potential to refine PSPC tools and their application, the approval levels for defence procurements undertaken by PSPC were appropriately identified as lower-risk and low-medium complexity.

Service standards met overall

PSPC service standards for the pilot have been met overall resulting in expedited timelines for key activities. The pilot has supported faster approvals for defence procurements with the potential for further efficiency.

More time and resources to focus on higher risk/complexity projects.

PSPC approval of lower risk procurements meant that TBS had more time and resources to focus on higher risk/complexity projects on the TB agenda. Continued improvements by PSPC to the usefulness and completeness of its information products provided to TBS would help maximize efficient delivery on Canada's defence policy.

Expedited contracting

The pilot has supported the implementation of SSE through expedited contracting, demonstrating PSPC's capacity to support an increase in the number and type of procurements approved by the department. DND believes that CAF needs have been met through the pilot and would continue to be better met through the continuation of PSPC approval of lower-risk and lower-medium complexity defence procurements.

Overall, across the 18-month period, the evaluation found the RBDPP program produced recognized benefits without harm to the Crown. As a result, the evaluation concluded there is merit in PSPC pursuing the continuation of a revised contracting limit whereby the department can enter into certain contracts without Treasury Board approval for procurements determined to be low risk and low-medium complexity.

Recommendations

The following 3 recommendations have been developed to refine the RBDPP program:

• recommendation 1—it is recommended that PSPC pursue the continuation of a revised authority to approve lower-risk, low-medium complexity defence procurement contracts to better allow the continued achievements of results in support of the Government of Canada's defence policy.
• recommendation 2—it is recommended that PSPC continue to refine its risk assessment tools and guidance to ensure greater precision in the methodologies, templates and guidance supporting the assessment and documentation of inherent and residual risks as well as of risks to the Crown by the proposed procurement.
• recommendation 3—it is recommended that PSPC continue to identify and streamline operational practices to further enhance efficiencies in procurement.

PSPC acquisition management accepts the recommendations and recognizes each as an important step in the department's procurement modernization efforts. Acquisition management notes that the RBDPP has allowed PSPC, in partnership with TBS and DND, to test a different way to approve procurements with the goal of improving administrative efficiency and service delivery.