National Digital Evidence Management System and Body Worn Cameras

Attestation of assurance

The fairness monitor (FM) hereby provides the unqualified assurance statement below concerning the National Digital Evidence Management System (DEMS) and Body Worn Cameras (BWC) procurement process:

It is our professional opinion that the procurement process we observed or monitored was carried out in a fair, open and transparent manner.

Original signed by:

Project requirement

RFP Solutions Inc. was engaged on October 28, 2020, as a FM to observe the competitive procurement process for the DEMS-BWC, initiated by Public Works and Government Services Canada (PWGSC) on behalf of the Royal Canadian Mounted Police (RCMP) under solicitation M7594-212120. RFP Solutions Inc. is an independent third party with respect to this activity.

This report covers the activities of the FM during the request for information (RFI) phase, including the invitation to qualify (ITQ), the review and refine requirements (RRR) phase, the request for proposal (RFP) phase and the evaluation of bids leading to the recommendation of a winning bidder to enter the field test phase of the procurement. It covers the period from October 28, 2020, to December 14, 2022.

The DEMS-BWC procurement process was implemented with the objective of rolling out more than 10,000 BWCs to be worn by the RCMP’s front-line officers. The RCMP was also working towards finding an effective and secure national solution to assist with the proper collection, storage, and disclosure of its digital evidence. This DEMS solution will support BWCs and other important digital evidence collection tools.

The procurement process under solicitation M7594-212120 tied together concepts that were previously explored by the RCMP through several RFIs.

Solicitation M7594-162680/A was issued in November 2015 to request industry feedback as part of an evidence-gathering initiative on the sustainability of body worn video for the RCMP and to help inform a plan for its possible implementation.

Solicitation M2989-190834/A was issued in June 2018 to request industry feedback on a software-based solution that would support the collection, storage, organization, protection, and disposal of digital evidence using a cloud-based software-as-a-service solution.

Solicitation M7594-192290/A was issued in May 2019 to request industry feedback on a field evidence extraction application that can be deployed on a mobile device, considering the various types of digital evidence that the RCMP is required to capture in the field.

The FM was not involved in these previous procurement activities.

The current procurement process started with 7 RFIs issued by PWGSC under solicitation M7594-212120 to provide industry with information and solicit feedback about the procurement process, including the ITQ process. Three provisionally qualify respondents (PQRs) were invited to participate in the subsequent RRR and RFP phases. During the RRR phase, each PQRs was evaluated to be a qualified respondent (QR), on the completion of preliminary supply chain integrity (SCI) and foreign ownership, control and influence (FOCI) assessments, and permitted to continue in the RRR and RFP phases.

We reviewed all the information provided and observed all relevant activities.

This report includes:

• our attestation of assurance
• a summary of the scope and objectives of our assignment
• the relevant observations from the activities undertaken

Fairness monitoring engagement and observations

In accordance with the terms of our engagement, we:

• familiarized ourselves with the relevant documents
• observed the preparation and communications with PQRs and QRs, including the receipt of submissions
• identified fairness-related matters to the contracting authority (CA)
• ensured that responses and actions were reasonable and appropriate

In this section

• Initial request for information
• One-on-one vendor demonstration sessions
• Second request for information
• Third request for information
• Fourth request for information
• Fifth request for information
• Invitation to qualify
• Invitation to qualify evaluation
• Sixth request for information
• Review and refine requirement
• Request for proposal
• Bid evaluation instructions
• Bid administrative review
• Bid technical evaluation
  • Point rated technical evaluation of written bids
  • Point rated solution performance evaluation testing
  • Point rated solution usability evaluation testing
• Bid financial evaluation, verification and bidder selection
• Communications and debriefing

Initial request for information

RFP Solutions Inc. was engaged to begin fairness monitoring activities after the RFI (solicitation M7594-212120/A) had been issued via buyandsell.gc.ca on October 20, 2020. To become familiar with the DEMS-BWC requirements and the procurement process, the FM reviewed the RFI documentation.

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• the clarity of the documentation to support understandability by interested suppliers

A 2-hour industry information session (industry day) was conducted on November 16, 2020, to provide industry with the proposed procurement and engagement approach, the DEMS-BWC requirements and to answer questions on each. The industry information session was conducted via Microsoft Teams with representatives from 13 vendors participating. The FM was present and observed the industry information session.

Four amendments to the RFI were issued to respond to 27 industry questions, make 11 modifications to the RFI terms and conditions and provide an update of the industry information session proceedings. The FM reviewed the amendments before they were issued.

The initial RFI closed on November 27, 2020, at 2:00 pm Eastern Standard Time.

The FM reviewed the way the responses to the RFI were received and handled with the CA.

Twenty-two responses were received and handled in accordance with deadline and instructions stated in the RFI. One additional response was received following the deadline.

In reviewing the RFI documentation and amendments, observing the industry information session, and reviewing the receipt and handling of the RFI responses, the FM did not observe any fairness issues.

One-on-one vendor demonstration sessions

The initial RFI requested that vendors interested in demonstrating their solutions register to participate in a vendor demonstration session (which would occur after the RFI close). The purpose of the vendor demonstration sessions was to demonstrate how the proposed solutions met the requirements of 2 predefined scenarios, with related questions, and for the RCMP to ask questions about the solutions. A total of 11 vendors participated in the sessions which were 2.5 hours each and conducted between December 7 to 15, 2020. Each meeting was conducted via Microsoft Teams and organized by PWGSC.

The FM was present and observed each of the online one-on-one vendor demonstration sessions and did not observe any fairness issues.

Second request for information

The second RFI was issued to engage with and solicit additional feedback from industry about DEMS-BWC, to:

• publish additional questions that were asked during the one-on-one vendor demonstrations under the initial RFI
• receive additional feedback on a managed service approach, costing methodologies and basis of payment structure
• request for input on proposal requirements

The FM reviewed the RFI before it was issued.

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by interested suppliers

The RFI was issued on February 22, 2021, via buyandsell.gc.ca under solicitation M7594-212120/B with the final closing date on March 17, 2021, at 2:00 pm Eastern Daylight Saving Time (EDT).

Five amendments to the RFI were issued to respond to 15 vendor questions and to make 2 modifications to the RFI terms and conditions. The FM reviewed the amendments before they were issued.

Eleven responses were received and handled in accordance with the closing requirements and deadline stated in the RFI.

The FM reviewed the way the responses to the RFI were received and handled with the CA

In reviewing the RFI, the responses to questions and amendments to the RFI, and reviewing the receipt and handling of the RFI responses, the FM did not observe any fairness issues.

Third request for information

The third RFI was issued to inform industry of Canada’s intention to proceed with an ITQ to pre-qualify respondents to participate in the RRR and RFP phases of the procurement and that prior to the ITQ, draft ITQ requirements, including a draft statement of work (SOW) and evaluation criteria would be issued, to allow the industry to submit feedback and raise any concerns.

The FM reviewed the RFI before it was issued.

The RFI was issued on March 26, 2021, via buyandsell.gc.ca under solicitation M7594-212120/C with the final closing date on April 30, 2021, at 5:00 pm EDT.

No amendments to the RFI were issued. No responses from industry were received.

In reviewing the RFI and its closing, the FM did not observe any fairness issues.

Fourth request for information

The fourth RFI was issued to solicit feedback from industry on the draft overview and high-level description of the requirement and evaluation criteria that would be included within the formal ITQ to qualify suppliers to be able to bid on a future solicitation to acquire a DEMS and BWCs.

The FM reviewed the RFI before it was issued.

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by interested suppliers

The RFI was issued on April 1, 2021, via buyandsell.gc.ca under solicitation M7594-212120/D with the final closing date on April 15, 2021, at 2:00 pm EDT.

Two amendments to the RFI were issued to respond to 3 supplier questions and to make 3 modifications to the RFI terms and conditions. The FM reviewed the amendments before they were issued.

Other than the questions from suppliers already noted, no additional responses were required.

In reviewing the RFI, the responses to questions and amendments to the RFI, and its closing, the FM did not observe any fairness issues.

Fifth request for information

The fifth RFI was issued to describe and summarize the feedback received from industry in response to the first RFI (solicitation M7594-212120/A).

The FM reviewed the RFI before it was issued.

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by interested suppliers

The RFI was issued on April 16, 2021, via buyandsell.gc.ca under solicitation M7594-212120/E and closed administratively on August 31, 2021, at 2:00 pm EDT. No responses from industry were requested with respect to this RFI.

One amendment to the RFI was issued to describe and summarize feedback received from industry in response to the second RFI (solicitation M7594-212120/B). The FM reviewed the amendment before it was issued.

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by interested suppliers

In reviewing the RFI and its closing, the FM did not observe any fairness issues.

Invitation to qualify

An ITQ was issued to qualify interested suppliers in accordance with the terms and conditions to become PQRs and QRs for any later phase of the DEMS-BWC procurement process.

The FM reviewed the ITQ before it was issued.

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by interested suppliers

The ITQ was issued on May 28, 2021, via buyandsell.gc.ca under solicitation M7594-212120/F with a final closing date on July 6, 2021, at 2:00 pm EDT.

Ten amendments to the ITQ were issued to respond to 22 supplier questions and to make 9 modifications, including changes to the closing date and the ITQ terms and conditions. The FM reviewed the amendments before they were issued.

The FM reviewed the way the ITQ responses were received and handled with the CA.

Six responses were received and handled in accordance with deadline and instructions stated in the ITQ.

In reviewing the ITQ, the amendments and the receipt and handling of the ITQ responses, the FM did not observe any fairness issues.

Invitation to qualify evaluation

Prior to response evaluations, an evaluation plan was prepared to provide instructions for the evaluators.

The FM reviewed the evaluation guidelines, including the:

• evaluator responsibilities
• code of conduct
• security, confidentiality and conflict of interest requirements
• evaluation methodology

An evaluation kick-off meeting for all evaluators was held on July 8, 2021, to review the evaluation plan. The meeting was conducted via tele/web conference. The FM was present to observe the proceedings.

The evaluation of submissions was subject to a phased compliance process for the mandatory submission and technical requirements; there were no rated requirements.

The FM provided assurance as to the impartiality of the evaluation methodology.

Between July 15 and 20, 2021, a review of mandatory submission and technical requirements was undertaken by the CA and evaluation committee respectively of the 6 responses received. Compliance assessment reports (CAR) were issued to 4 respondents. After the remedy period was closed, the CAR responses of the noncompliant respondents were assessed by the evaluation committee which concluded its review on August 9, 2021.

Through the evaluation, 3 of the respondents were deemed compliant to the requirements as stated in the ITQ, and therefore deemed to be PQRs that could participate in the RRR and RFP phases. Three respondents were deemed not to be eligible to be PQRs. All respondents were notified of the results in a letter dated August 13, 2021, from the CA.

The FM observed the evaluation committee meetings and reviewed the results of the mandatory review and phased compliance processes against the requirements in the ITQ. The FM also reviewed the notifications prior to them being sent to respondents.

In reviewing the evaluation plan, observing the evaluation kick-off meeting and reviewing the assessment of respondents’ compliance with the mandatory submission and technical requirements, as well as the notifications, the FM did not observe any fairness concerns.

Sixth request for information

The sixth RFI was issued to inform industry of the list of PQRs and to indicate that the SCI and FOCI assessments would be conducted in parallel with the RRR phase and that PQRs may be disqualified if they did not meet Canada’s security requirements.

The RFI was issued on August 16, 2021, via buyandsell.gc.ca under solicitation M7594-212120/G. The RFI was issued directly as a notice via buyandsell.gc.ca with no associated attachments. The RFI closed administratively on September 27, 2021, at 2:00 pm EDT. No responses from industry were requested with respect to this RFI.

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by industry

In reviewing the RFI and its closing, the FM did not observe any fairness issues.

Review and refine requirement

The 3 PQRs were formally invited to participate in the RRR phase via an e-mail on August 31, 2021.

The objective of the RRR phase was to share information and obtain feedback from PQRs on Canada’s preliminary requirements, notional procurement process, and on subsequent phases of the procurement.

The FM was copied on the e-mail to the PQRs which included a document describing the purpose and process for the RRR phase, including terms and conditions.

The RRR phase consisted of 3 streams:

• stream 1 pertained to
  • technical requirements for the BWC and the DEMS
  • cross functional services
• stream 2 pertained to
  • security and privacy requirements
  • the field test
  • implementation portions of the contract
• stream 3 pertained to
  • contractual terms and conditions
  • transition-out services
  • contract services
  • additional services
  • regulatory and policy requirements

The FM provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by the PQRs

An initial formal RRR phase kick-off meeting, held on September 8, 2021, via Microsoft Teams to describe the process was attended by all PQRs. The FM was present and observed the kick-off meeting.

Draft documentation for stream 1, functional and technical requirements for the BWC, the DEMS and cross functional services, was released to the PQRs by email on August 31, 2021. The FM was copied on the e-mail.

Prior to the release, the FM reviewed the documentation and provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by the PQRs

On September 22, 23 and 24, 2021, full-day Microsoft Teams sessions were scheduled with each of the PQRs to review the stream 1 requirements and their feedback.

Based on feedback received during the initial session, subsequent 2-hour meetings were held with each of the PQRs on October 15, 2021, via Microsoft Teams to review requirements and receive feedback related to solution bandwidth and architecture. Prior to the meeting, a document providing more context and details related to the solution bandwidth and architecture requirements was shared with the PQRs.

The FM reviewed the document prior to release and provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by the PQRs

In response to the document, each PQRs submitted a presentation deck to guide the discussion during the meeting.

The FM was present and observed each of the stream 1 RRR one-on-one meetings.

Draft documentation for stream 2, security and privacy requirements and the field test and implementation portions of the contract, was released to the PQRs by email on September 13, 2021. The FM was copied on the e-mail.

Prior to the release, the FM reviewed the documentation and provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by the PQRs

On October 5, 7 and 8, 2021, full-day Microsoft Teams sessions were scheduled with each of the PQRs to review the stream 2 requirements and their feedback.

Based on feedback received during the initial stream 2 session, subsequent 3-hour meetings were held with each of the PQRs on November 2, 3 and 4, 2021 via Microsoft Teams for further review and discussion of the field test and implementation requirements of the contract, including how sites with low bandwidth could be addressed. Prior to the meeting, a document providing more context and details related to the field test and implementation requirements of the contract was shared with the PQRs.

The FM reviewed the document prior to release and provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by the PQRs

PQRs were not requested to provide written feedback prior to the meetings.

The FM was present and observed each of the stream 2 RRR one-on-one meetings.

Draft documentation for stream 3, contractual terms and conditions, transition-out services, contract services, additional services and regulatory and policy requirements, was released to the PQRs by email on September 29, 2021. The FM was copied on the e-mail.

Prior to the release, the FM reviewed the documentation and provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by the PQRs

On November 22, 24 and 30, 2021, full-day Microsoft Teams sessions were scheduled with each of the PQRs to review the stream 3 requirements and their feedback.

The FM was present and observed each of the stream 3 RRR one-on-one meetings.

The draft RFP was released iteratively to PQRs on a component basis starting between February 15, 2022, and June 15, 2022, with final comments due June 16, 2022.

Prior to its release, the FM reviewed the documentation and provided assurance as to the:

• neutrality of the requirements
• objectivity of requirements and process
• clarity of the documentation to support understandability by the PQRs

On April 21, 2022, PWGSC issued a presentation to PQRs highlighting security and privacy requirements described in the draft RFP to ensure all PQRs had a common and fulsome understanding of the requirements prior to the commencement of the formal bidding process. The issuance of the presentation was followed by a 1.5-hour Microsoft Teams meeting with each of the PQRs on April 25, 26 and 27, 2022. The FM reviewed the presentation prior to it being issued and was present and observed each of the one-on-one sessions.

On May 6, 2022, PWGSC issued a series of standardized questions to the PQRs requesting feedback on recovery point objectives. The FM reviewed the questions prior to their issuance.

Following receipt by PWGSC of all feedback from industry on the draft RFP components, PWGSC prepared and issued a summary of changes to the PQRs on June 27, 2022, to identify intended updates to the RFP components based on feedback received. The FM reviewed the summary of changes prior to it being issued.

Throughout the RRR process reviews, preliminary assessments of the PQRs’ FOCI and their SCI were being conducted by the PWGSC Contract Security Program and the Communications Security Establishment.

On May 18, 2022, each of the PQRs was notified by email to indicate that the preliminary FOCI and SCI assessments had been completed and they were deemed to be QRs. The list of QRs was also made publicly available on buyandsell.gc.ca on May 25, 2022.

In being qualified to respond to the RFP, each of the QRs was notified on July 13, 2022, that they should have a valid Canada Post ePost Connect account to be able to receive the RFP.

The RRR phase was administratively closed with the release of the RFP to the QRs via the Canada Post ePost Connect system on July 19, 2022.

In reviewing the RRR phase invitation and the documentation, the draft RFP, communication with suppliers, notification of QRs, and observing all meetings with suppliers, the FM did not observe any fairness issues.

Request for proposal

The RFP was issued to the QRs via the Canada Post ePost Connect on July 19, 2022, as solicitation M7594-212120/I. The submission date for information relating to FOCI and SCI evaluations and personnel security clearance information (field test) was August 4, 2022, at 2:00 pm EDT. The RFP final closing date was August 30, 2022, at 2:00 pm EDT.

On August 5, 2022, PWGSC established a new Canada Post ePost Connect conversation to communicate with and distribute documentation to all the QRs. The new conversation was established to mitigate privacy concerns resulting from the submission of information from 1 QR that did not follow the established protocols. The new conversation included the original RFP and associated documentation and amendments 1 through 5. The FM reviewed the approach to cancel and restart the Canada Post ePost Connect conversation. The FM did not observe any fairness concerns.

The CA received a total of 23 questions from QRs during the posting period. Responses to these questions, as well as 22 amendments to the terms and conditions of the solicitation were issued in 12 solicitation amendment documents; all were issued to the QRs via Canada Post ePost Connect. The FM reviewed all the amendment documents. Minor fairness concerns raised by the FM were addressed by the CA prior to issuing the amendment documents.

Two submissions, relating to the FOCI, SCI and personnel security clearance information (field test), were received by 2:00 pm EDT on August 4, 2022, and handled in accordance with the RFP requirements. The FM reviewed the way the FOCI, SCI and personnel security clearance information were received and handled with the CA.

Two written bids and associated equipment for test evaluation were received by 2:00 pm EDT on August 30, 2022, and handled in accordance with the RFP closing requirements.

The FM reviewed the way the RFP responses were received and handled with the CA.

In reviewing the receipt and handling of the bids, the FM did not observe any fairness issues.

Bid evaluation instructions

Prior to bid evaluations, the FM again reviewed the evaluation guidelines, including:

• evaluator responsibilities
• code of conduct
• security,
• confidentiality and conflict of interest requirements
• evaluation methodology

The evaluation of submissions was subject to a phased bid compliance process for the mandatory submission and technical requirements.

The FM provided assurance as to the impartiality of evaluation methodology.

The CA and procurement team conducted evaluation training for the bid evaluation teams, including presentations to describe the evaluation methodology for the written bids and the performance and usability evaluations. The FM reviewed the presentations prior to finalization. The 2 evaluator training sessions were conducted on August 26, and September 6, 2022, via Microsoft Teams. The FM observed each of the training sessions.

In reviewing the evaluation methodology and evaluator training presentations, and observing the evaluator training, the FM did not observe any fairness concerns.

Bid administrative review

Following RFP closing, a review of mandatory submission and technical requirements was undertaken by the CA and evaluation committee respectively of the 2 bids received.

The administrative review included a check of the hardware samples submitted by the bidders. The samples had been shipped to and were opened at the RCMP facility at 73 Leiken Drive, Ottawa. The FM observed the opening of the samples.

The evaluation team members independently reviewed the submissions against the mandatory technical requirements. Consensus evaluation meetings for the mandatory technical requirements were held on September 16, 21 and 27, 2022. Based on this review, both bidders were identified as being non-compliant with the mandatory technical requirements. The CA issued a CAR to each bidder in accordance with the terms of the RFP, advising them of their respective non-compliance and invited each to submit additional information to support compliance.

Upon receipt of the Foreign ownership, control and influence (CAR) responses, the evaluation team members independently reviewed the information provided. Consensus evaluation meetings to evaluate the information provided in the CAR responses were held on October 4 and 5, 2022.

Based on the initial submissions and the response to the CARs, it was determined that both bidders met the requirements. Subsequently, CARs were issued to both bidders, informing them that they were compliant with the mandatory submission and technical requirements and eligible to proceed to rated technical evaluation.

The FM observed each of the consensus evaluation meetings for the mandatory technical requirements, including the meetings to evaluate the CAR responses. The FM also reviewed the CARs prior to issuance and the results of the phased bid compliance process to ensure alignment with the requirements in the RFP.

In observing the consensus evaluation meetings and reviewing the assessment of the bidders’ compliance with the mandatory submission and technical requirements, the FM did not observe any fairness issues.

Bid technical evaluation

The rated technical evaluation was conducted on both bids, as each met all mandatory requirements. The rated technical evaluation had 3 parts:

• evaluation of the written bids against the point rated management bid and technical compliance requirements, including Indigenous participation criteria
• performance evaluation testing of the bid solutions (sample hardware and cloud-based software) against the point rated requirements and to validate the evaluation of the mandatory and point rated technical compliance criteria in the written bids
• point rated usability evaluation testing of the bid solutions

All bids, for which performance testing validated that the mandatory technical compliance criteria were met, were eligible to proceed to the financial evaluation and received a total technical score (TTS) that was used in the basis of selection.

Point rated technical evaluation of written bids

The first technical evaluation was of the written bids against the management bid, which included confirmation of the bidder team members’ experience by way of reference check with references for previously completed projects, point rated technical compliance criteria, and Indigenous participation component evaluation.

The bids were first reviewed and scored individually by each member of the evaluation team. A series of consensus evaluation meetings were conducted to determine the consensus score of each bidder. The evaluators found that the responses to some of the point rated criteria were difficult to locate within the written responses from each bidder. Questions were sent to each of the bidders to clarify where in the bid each of the responses was located. The evaluators independently reviewed each of the responses and additional consensus meetings were conducted to determine the evaluated score.

At the same time as the written bids were being evaluated, the CA sent emails with response templates to the bidders’ references to confirm team members’ experience was as claimed. In the case of both bidders, some references failed to respond within the requested time periods and others failed to respond to the full scope of the reference questions. As per the RFP, the CA informed the bidders of the references’ unresponsiveness, and they were permitted to substitute new references if the original references were no longer available. The CA contacted those references with response templates to confirm team members’ experience. The responses from the references were first reviewed and scored individually by each member of the evaluation team and the consensus point rated scores for the bidders’ team members were determined during the consensus evaluation meetings.

Consensus evaluation meetings were conducted via Microsoft Teams on October 19, 20, and 21, and November 17, 2022. A final consensus evaluation, related to a response obtained through a reference check, was conducted secretarially by each evaluator submitting their confirmation of their associated score and rationale via email, on November 21, 2022.

The FM observed each of the consensus meetings and was included on the communication for the consensus evaluation of the point rated criterion that was conducted secretarially. The FM also reviewed communication with bidders and their references before it was issued.

In observing the consensus evaluation meetings, the FM noted minor issues during the discussion that were resolved in real-time and before the final scores were set. In reviewing the communication with bidders and their references, the FM noted minor issues with clarity that were resolved before the communications were issued. There were no fairness issues with the final scores or the communications that were issued to bidders or their references.

Point rated solution performance evaluation testing

Once the evaluation of the point rated technical compliance criteria of the written bids was complete, the bid solutions were evaluated against the performance evaluation criteria. The performance evaluation was undertaken concurrently with part of the evaluation of the point-rated management bid criteria as none of the scoring of the management bids would have affected the ability to proceed with the performance evaluation.

The point rated solution performance evaluation was conducted in a dedicated boardroom at the RCMP facility at 73 Leikin Drive, Ottawa. The solution from each of the bidders was configured according to instructions provided within their respective bid package and based on the requirements within the RFP. In set-up for the performance evaluation, a web conference was held on October 24, 2022, with 1 of the bidders to confirm network connectivity set-up. During the performance evaluation, clarifications about where to find instructions to perform activities within their bids had to be obtained from both bidders.

In conducting the performance evaluation, each evaluator independently conducted each test against the identified mandatory requirements and conducted and scored each test against the identified point-rated criteria, and then a consensus discussion was held to arrive at the consensus outcomes for each criterion.

Performance evaluation testing was conducted on October 26, 27, 28 and 31, and November 1, 9, 10, and 15, 2022.

The FM reviewed the set-up and protocols for the conduct of the performance evaluations which aligned to the detailed testing methodology set out in Annex G of the RFP and observed elements of the performance evaluation sessions on October 26 and 28, 2022, and November 1, 10, and 15, 2022. The CA was present through the whole process and was in frequent contact with the FM to discuss and review any issues that came up during the testing. The FM also reviewed communication with bidders during the performance evaluation before it was issued and made inquiries of the CA as appropriate. Once the process was complete, the FM reviewed the records of results with the CA.

In observing the performance evaluation and understanding proposed resolution of issues, the FM noted minor issues that were resolved in real-time, before communication with bidders was issued, and before the final scores were set. In reviewing the communication with bidders, the FM noted minor issues with clarity that were resolved before the communications were issued. There were no fairness issues with the final scores or the communications that were issued to bidders.

Point rated solution usability evaluation testing

Once the evaluation of the point rated solution performance evaluation was complete, the bid solutions were evaluated against the point-rated usability testing criteria. The usability evaluation was undertaken concurrently with part of the evaluation of the point-rated management bid criteria as none of scoring of the management bids would have affected the ability to proceed with the usability evaluation.

The point rated solution usability evaluation was conducted in a dedicated boardroom at the RCMP facility at 73 Leikin Drive, Ottawa. The solution from each of the bidders was configured according to instructions provided within the respective bid package and based on the requirements within the RFP.

In conducting the usability evaluation, the evaluators were comprised of a representative sample of front line RCMP officers. Nine evaluators tested and scored each usability test, with the same evaluators evaluating both bids. Each evaluator independently conducted each test and completed a questionnaire for each test by assigning a score on a 4-point Likert-type scale. The total score for each bidder for each criterion was the average score from all the evaluators scoring the criterion.

Performance evaluation testing was conducted on November 21 and 22, 2022.

The FM reviewed the set-up and protocols for the conduct of the usability evaluations which aligned to the detailed methodology and user trials set out in Annex G of the RFP and observed elements of the usability evaluation testing sessions on November 21, 2022. The CA was present through the whole process and was in frequent contact with the FM to discuss and review any issues that came up during the testing and made inquiries of the CA as appropriate. Once the process was complete, the FM reviewed the records of results with the CA. There were no fairness issues with the final scores.

Bid financial evaluation, verification and bidder selection

In accordance with the RFP, there were no minimum point rated scores and both bids met the mandatory requirements so both financial bids were opened, and the CA determined the total evaluated price of each bid. The FM reviewed the total evaluated price and the total financial score of each bid with the CA.

The FM also reviewed the overall weighting and combination of scores of each bid with the CA.

In reviewing the results of the financial evaluation, overall score verification and bidder selection, the FM did not observe any fairness concerns.

Communications and debriefing

This section will be completed as an Addendum to this report following completion of contract award, which is subject to confirmation of security and integrity provisions and, as requested, debriefing of Bidders.

Reference documents

Documents related to the RFI (including the ITQ) phase are available under solicitation numbers M7594-212120/A, B, C, D, E, F and G through CanadaBuys and through the project office.

Documents related to the RRR phase of the DEMS BWC procurement are available through the project office.

The public notification of the QRs is available under solicitation number M7594-212120/H through CanadaBuys.

Documents related to the RFP phase of the DEMS-BWC procurement are available through the project office.

Addendum to the final report

April 21, 2023

Addendum to the Final Report dated January 3, 2023, for the DEMS-BWC.

This addendum covers the period following the conclusion of the evaluation phase including contract award and debriefings.

The contract was formally awarded, and the successful bidder was officially notified of the result on January 12, 2023 with an e-mail that included the signed contract. The unsuccessful bidder was notified of the result on January 13, 2023 via an e-mail that included a formal letter of regret. The letter of regret identified the successful bidder and the contract value and informed the unsuccessful bidder that that they were entitled to a debrief on the results. The email also included the results of the unsuccessful bidder’s consensus evaluation scoring on the mandatory and rated requirements, and of the financial evaluation. A contract award notice was published on CanadaBuys on January 13, 2023. The FM reviewed the contract and the communications with the successful and unsuccessful bidders before being issued.

On January 24, 2023, the unsuccessful bidder requested additional feedback on where it had lost points in the evaluation. The unsuccessful bidder resubmitted this request to the CA as a formal notice of challenge on January 26, 2023. The CA provided written feedback on February 22, 2023. The FM reviewed the feedback before it was sent to the bidder and did not observe any fairness issues.

Attestation of Assurance

The FM hereby provides the unqualified assurance statement below concerning the DEMS-BWC procurement process contract award and debriefing processes.

It is our professional opinion that the contract award and debriefing processes we observed or monitored were carried out in a fair, open and transparent manner.

Original signed by: