Audit of the development of VaccineConnect

October 2021

On this page

• Executive summary
• Criterion 1: Governance
• Criterion 2: Risk management
• Criterion 3: Project management controls
• Criterion 4: End-user engagement
• Appendix A: Scorecard

Executive summary

Introduction

The Public Health Agency of Canada (PHAC) has been working closely with Canadian provinces and territories to develop a pan-Canadian strategy for COVID-19 vaccine logistic management and rollout strategy. As part of PHAC’s immunization campaign efforts, a project was launched to develop a technology solution, known as VaccineConnect, to provide logistical support for the administration, surveillance, and reporting of vaccines. This pioneering work represents the first major digital project for PHAC, as it aims to develop a cloud-based solution using an Agile approach to systems development. 

To accomplish this, the Agency quickly assembled a multi-disciplinary team with expertise in IT, logistics, epidemiology, and project management, under the leadership of the Chief Technology Officer (CTO). These partnerships brought the required skills and ideas needed to develop an integrated solution.

Within the last eight months, the team has been working diligently to develop the following three modules that make up VaccineConnect:

1. Intelligent Supply Chain (ISC): processes vaccine orders and tracks inventory and logistics in near real-time;
2. Immunization Information System (IIS): collects, stores, and enables the analysis of national  immunization safety and coverage data; and
3. Immunization Program Management (IPM): functions as an end-to-end management process that includes registration, scheduling, vaccine forecasting and reminders, in-clinic administration, supply management functionality, and reporting and analytics.

VaccineConnect’s three modules are currently in various stages of production. Following a major release in June 2021, the ISC module was used successfully to process orders and distribute vaccines to federal, provincial, and territorial (FPT) jurisdictions. Capacity is being developed for the IIS module to allow provinces and territories to transmit safety and coverage data to PHAC safely through direct input in the system by using formats that require less manual manipulation compared to current practices. It should be noted that following the completion of this audit, IIS business owners expressed concerns about the progress of the IIS module’s development, as it did not yet have the functionality nor flexibility needed. The IPM module has had extensive engagement with provinces, territories and federal partners, and two provincial clients are currently using IPM.

Out of necessity, the development of VaccineConnect has had to move at very fast pace, which is unusual for a project of this magnitude and complexity within the Government of Canada. Such a project would normally be subject to rigorous Treasury Board of Canada Secretariat (TBS) oversight and be required to follow an established corporate project management methodology. However, due to the urgent need to develop a timely solution, an Agile approach to project management was adopted, with modified governance. TBS exceptions were granted in support of this approach, acknowledging the need for the technology system to adapt to unusual requirements quickly in order to provide timely information on vaccine distribution, coverage, and safety.

Engagement objective

The objective of this audit was to provide reasonable assurance that appropriate controls were in place to meet the functional objectives of the VaccineConnect IT platform.

Engagement scope

The scope of this audit engagement included the VaccineConnect IT platform and its related governance, risk management, project management systems, and system development processes and controls, from November 1, 2020, to July 15, 2021. This included all files, documents, and data pertaining to the project within this timeframe. The audit did not include a review of procurement, financial, and IT security controls. Finally, the audit did not include a review of vaccine certification functionality, which has not entered the development phase for this IT initiative.

What was found

We found that most elements of governance, risk management, communications, testing, and data-sharing controls were in place to support VaccineConnect in meeting its functional objectives. The Project Steering Committee and Product Management Council have been providing sufficient oversight and decision making for the project. Also, most of the required project documents have been developed and approved in accordance with the Agile Systems Development Checklist. In addition, communication strategies and plans have been developed to engage all stakeholders in VaccineConnect. Furthermore, rigorous user acceptance testing is in place with business owners and the developer. Finally, an information-sharing agreement between the Agency and the provinces and territories is in place to ensure proper handling of information.

We also identified some areas that need further improvement. The lack of a formal risk escalation mechanism in the risk management process could delay progress for VaccineConnect if risks remain unresolved at the working group level. In addition, the risk register does not include broader strategic risks and, if left unattended, these could affect further development of VaccineConnect. Although a managed service for VaccineConnect has been arranged for the short term, PHAC is only now formulating a long-term vision and strategy for implementing services that could affect future planning and operations for the IT platform. In addition, even though there are some controls in place to ensure timely delivery of VaccineConnect, their effectiveness is limited due to a lack of formal vendor management, work breakdown structure, critical path dependencies, and corresponding mitigation strategies. Without appropriate vendor management in place, project functionality may be delayed and over budget.

Criterion 1: Governance

Context

As VaccineConnect was developed to support evolving COVID-19 vaccine management logistics, rollout requirements, and services, the project team has had to respond to exceptionally tight timelines. Due to the urgent nature of this project, established corporate project management practices and gating processes were modified.

An Agile System Development Life Cycle (SDLC) approach was established to allow VaccineConnect to evolve from a minimum viable product that provided immediate support, to a more complete ongoing functional solution using an agile and flexible approach.

The project has benefited from ministerial and senior executive level support by receiving early approval and financing, as well as exceptions granted by the Treasury Board, that supported the adoption of Agile project management approaches.

What we expected to find

We expected to find that PHAC maintained appropriate governance frameworks, policies, and controls for the development of VaccineConnect.

Findings

In an effort to reduce administrative burdens and streamline project management, a newly-developed Agile project governance framework was put in place to oversee the project. This included scrum teams for each of the three modules, a Product Management Council, and a Project Steering Committee.

The VaccineConnect Steering Committee served as a decision-making and oversight body for the project, and helped ensure that project management practices were being followed and that project progress was monitored. The Committee helped ensure appropriate controls were in place prior to release. For example, we observed that “Go/No-Go” votes occurred once assurance was given that the product release contained the proper functionalities and that the acceptance criteria for that phase of work had been met. The audit also found that the Product Management Council provided program-level guidance and integration of overall project objectives at the operational level. Overall, we found that the project management structure was working as intended to support project planning, development, and decision making.

The project tracked the progress of key project documents using an SDLC Compliance Checklist for Agile projects. While we found that there were gaps in the completeness of project documentation, as some documents were left unsigned, incomplete, and in some cases, were bypassed completely, due in part to limited staff at the beginning of the project, as well as staff turnover. The project has continued to develop key project artifacts and is on track to meet the requirements of the SDLC checklist.

Roles and responsibilities have been identified and communicated through organizational charts found in the draft Project Management Plan, as well as in a project Responsibilities, Accountabilities, Support, Consult, Information (RASCI) chart. While most of the roles, responsibilities, and accountabilities were clearly defined and communicated, we found that the vendor management role remains unclear and should have been prioritized at the beginning of the project, as it can affect vendor accountability and overall operational effectiveness. This is addressed in further detail in the Project Management section of this report.

Conclusion

We found that an appropriate governance framework was in place for the development of VaccineConnect, through an active Steering Committee that provides oversight for decision making. There is also a Product Management Council in place to provide governance and oversight on the evolution and implementation of the three modules product roadmaps. A RASCI chart has been developed and communicated to all stakeholders, however the vendor management role was not prioritized sufficiently at the onset of the project to help monitor and report on vendor performance.

Criterion 2: Risk management

Context

The development of VaccineConnect introduced a new systems development approach (Agile) for the Agency, consequently there is little experience on which to rely.

The project has been deemed to be complex, with a Project Complexity Rating Assessment (PCRA) higher than the Agency’s project management rating, and would normally require Treasury Board Secretariat oversight. This requirement has been exempted for this project. In addition, the project has also been given an exemption from the Agency’s internal Project Management Methodology, due to the urgency of developing an IT solution in response to the pandemic. As a result, governance, administration, and oversight have all been lessened for this project, thereby increasing the inherent risk to the project.

A risk management plan was not developed for this project. Instead, a simplified risk management process was used in order to align with Agile project management and governance approaches. 

What we expected to find

We expected to find a risk management plan that includes governance controls, policies, and associated processes for identifying, communicating, and mitigating risk, which is assessed routinely. In addition, we expected to find a risk management process that includes an escalation mechanism to ensure that risks are being managed in a timely manner.

Findings

We found a simplified risk management process for this project. This streamlined process was needed to align with the Agile project methodology and the fast-paced nature of this project. We found that tactical and operational risks relating to the development of VaccineConnect were being well managed by both the vendor and PHAC through this process, which includes daily or weekly scrum meetings. These meetings have been taking place since the inception of the project. The vendor records risk information with input from PHAC into a register (Risks, Actions, Issues, and Decisions: RAID). This document contains most of the key elements to manage risk and, to date, has been effective in managing these risks. However, we found limited information relating to broader strategic risks in the RAID, such as vendor performance management, human resources transition from Canadian Armed Forces personnel to PHAC, transition between the vendor and PHAC for managed services, and few users of the IPM module. If these risks are not given appropriate oversight and left unattended for a lengthy amount of time, they could affect the current and future progress of VaccineConnect. 

Finally, the current documented risk management process, which is identified in the draft Project Management Plan, does not reflect a risk escalation mechanism. However, given the nature of the mission and the expediency required to execute on successful vaccine delivery, informal risk escalation takes place between the vendor, Chief Technology Officer Branch (CTOB) Product Leads and officials from the Information Management Systems Directorate (IMSD) to address unresolved tactical risks in event that risks are not resolved at the operational level. The Agile Project Management Governance structure, endorsed by Treasury Board and the Agency’s Project Management Office, has identified the Project Steering Committee as having maximum empowerment. In its capacity, the Project Steering Committee has a role to play in risk management, specifically in terms of risk resolution for strategic project risks. Risk resolution involving all key stakeholders engenders greater transparency to all who have in interest in this project. Risk escalation is important for risk response strategy options, as it ensures that each risk is managed where it matters, and is owned by the stakeholder who would be most affected.

Conclusion

While no risk management plan was developed, tactical and operational risks were well managed through the simplified risk management process. The risk register contains most of the elements for managing risks; however, it is missing broader strategic risks that could affect the progress of VaccineConnect project. In addition, the risk management process is lacking a formal risk escalation mechanism, which should be invoked to address strategic project risks. This is an important risk response option in the event that risks cannot be resolved using the current process.

Recommendation

Recommendation #1: The Senior Vice-President of the Chief Technology Officer Branch should ensure that all risks are included in the risk register. In addition, the risk management process should be updated and documented to include a risk escalation mechanism that reflects greater transparency and the responsibility and accountability of the Project Steering Committee in this process. See also Recommendation 3, related to vendor management.

Criterion 3: Project management controls

Context

The Treasury Board Directive on the Management of Projects and Programmes requires that government projects be effectively managed, implemented, monitored, controlled, and closed to enable the realization of the expected benefits and results for Canadians. The Directive expects that:

1. Governance and controls over projects are effective;
2. Decisions are made throughout the life of the project and program with a view to maximize efficiency and ensure the realization of benefits; and
3. Performance measurement data is used to support regular monitoring of project and program health and evidence-based decision making.

What we expected to find

We expected to find project management controls to ensure that VaccineConnect objectives and deliverables were achieved on time and working as intended.

Findings

We found that there were controls in place to manage the VaccineConnect project, including a reporting structure detailing the frequency, content, and reporting of project statuses at all levels. Furthermore, there was a rigorous user acceptance testing regime that involved all key stakeholders. The effectiveness of the controls was partly limited due to lack of incorporation and planning to address risks, some of which are strategic, including:

1. Risk of slowed or limited participation in VaccineConnect by provinces and territories, especially in the context of the viability of continued development and maturation of the individual modules. Risks, uptake thresholds, and contingencies should be factored in validating the continued development and support of the individual modules. This would ensure that budgets and efforts are used justifiably in implementing VaccineConnect services. For example, the participation in IPM has been limited, irrespective of the hard work and commendable effort that has already been put into marketing it to interested federal, provincial, and territorial clients.
2. Risks arising from the lack of implementing a vendor management role, which is in and by itself a key control. Considering the size and budget of this IT initiative, we expected vendor performance to be routinely monitored, assessed, and reported on to senior management. Introducing and judiciously implementing a vendor management role will serve to curb risks of scope creep, delays in delivery, and compromised deliverable quality.
3. The overall Project Complexity Risk Assessment Rating (PCRA) for this project was scored at three, however there were some risks scored at five (on a scale of one to five). Some examples include risks arising from dependency on other project outcomes, dependency of other projects on VaccineConnect outcomes, and on the degree of integration with other projects, systems, infrastructure, or organizations.

Nowhere in the project management plan, nor in the risk register, were any of the risks incorporated and, if left unacknowledged and unresolved, these could adversely affect current and future development of VaccineConnect.

We did not find sufficient evidence that PHAC has a plan for a full transition and implementation of VaccineConnect, even though the contract with the vendor is expected to expire in January, 2022. In addition, PHAC did not yet have a long-term vision and strategy beyond the current 2-dose COVID-19 vaccine campaign. Considering the pressure on the development and management teams to deliver in these most exceptional circumstances, it is understandable that little time has been devoted to developing a long-term vision and accompanying strategy for VaccineConnect. Rather than operating the three modules in silos, the implementation process should consider integrating them under the management of one functional area, resulting in synergetic benefits and potentially reduced cost.

In addition, the effectiveness of controls is reduced further due to a lack of work breakdown structures, critical path dependencies, and corresponding contingencies in the project management plan. This creates the potential of leaving project teams blind to risks that may remain dormant for an unreasonable length of time. Examples of resulting impacts include confusing project objectives, overlooked dependencies, and adverse impacts on the timeliness and quality of VaccineConnect deliverables. Equally significant are unforeseen costs and requirements for re-planning deliverables.

Delivery of some module features were delayed, are pending delivery, and, in some cases, were scoped out. Through sampling and testing, we found evidence of a number of features, as per the contract and associated task authorizations that had been paid for, but had not yet been delivered. We have not been able to obtain records of decisions at the Project Steering Committee level to demonstrate due and adequate justification.

Conclusion

A managed service for VaccineConnect has been arranged for one year, plus three optional years. PHAC is only now forming a long-term vision and strategy for implementing services. The existing controls ensure timely delivery of VaccineConnect; however, their effectiveness is limited due to a lack of formal vendor management, of work breakdown structures, critical path dependencies, and corresponding contingencies. The project management plan and risk registers missed certain strategic risks and have not allotted due attention, nor a corresponding action plan to them. Some features identified in the Task Authorizations have not been fully met for the ISC and IIS modules and are pending delivery in future releases. No timelines have been provided for these items. Finally, we found that PHAC had implemented appropriate user acceptance testing (UAT) for the VaccineConnect deliverables to ensure quality and proper functionality.

Recommendations

Recommendation 2: The Vice-President of the Chief Technology Officer Branch should, in consultation with the Logistics and Operations Branch and the Immunization Branch, develop and implement a strategy for transitioning and implementing VaccineConnect in the context of a long-term vision. The strategy should include the three modules (ISC, IIS, and IPM).

Recommendation 3: The Vice-President of the Chief Technology Officer Branch should introduce a Vendor Management role with clearly defined responsibilities to manage and routinely report on vendor performance in meeting contractual obligations, and ensure that software releases are delivered on time, within budget, and with minimal requirements for change management. Key Performance Indicators for vendor performance measurement should be identified and implemented.

Recommendation 4: The Vice-President of the Chief Technology Officer Branch should revise the project management plan and incorporate a clearly laid out work breakdown structure and corresponding critical path dependencies, including associated risks and mitigation measures.

Criterion 4: End-user engagement

Context

As part of Canada’s response to COVID-19, the Public Health Agency of Canada is actively monitoring the vaccine rollout. Part of the Agency’s duty is to collect and analyze coverage and safety [Adverse Event Following Immunization (AEFI)] data. Given that health care is a provincial jurisdiction, AEFI and coverage data is directly collected and owned by the provinces and territories (PTs).

Since coverage data is considered aggregated and non-identifiable data, Canada is not required to use a data sharing agreement. However, safety (AEFI) data is considered personal data, given the risk of identification posed by the combination of multiple data points captured in AEFI data.

AEFI data is currently shared between every province and territory, except BC, and Canada on a voluntary basis, and in accordance with the principles of the Multi-Lateral Information Sharing Agreement (MLISA).

What we expected to find

We expected to find a data-sharing framework in place between PHAC, provinces and territories, and other potential stakeholders to ensure appropriate end-user engagement.

Findings

Long-standing data-sharing agreements exist between Canada and the provinces and territories, such as the Multi-Lateral Information Sharing Agreement (MLISA), and the British Columbia Centre for Disease Control (BCCDC) agreements.

We found that, while the MLISA agreement has been in negotiation for 10 years, it serves as an effective tool that can be leveraged for IIS module purposes. We also found that PHAC is in the process of developing a separate Memorandum of Agreement (MoA) that is aligned with the principles of MLISA and further clarifies roles and responsibilities between Canada, provinces, and territories regarding the IIS module. These agreements help ensure proper use, handling, and protection of shared data, which if misused may affect the privacy, reliability, and confidentiality of data.

While data-sharing agreements are in place, the Agency currently receives coverage and AEFI data from the provinces and territories on a voluntary basis. The MLISA, BCCDC, and MOAs are providing guiding principles for data exchange, but there is an ongoing risk that provinces and territories may stop sharing coverage and AEFI data.

We also examined whether a framework was in place to ensure that there was end-user engagement. We found that communication strategies were in place for engaging with provinces and territories for each module of VaccineConnect. Engagement calendars demonstrated PHAC’s past and planned outreach to various stakeholders. While there has been significant outreach to encourage participation in the IPM module, only two provinces, Alberta and British Columbia, have adopted an IPM process to date. There are no federal clients using the IPM module.

Conclusion

We found that data-sharing agreements are in place, including the Multi-Lateral Information Sharing Agreement, which provides guiding principles for data exchange between provinces, territories, and the Government of Canada. A memorandum of agreement is also being drafted to support data sharing for the IIS module. A communication strategy was developed to engage with provinces and territories for all three modules of VaccineConnect.

Appendix A: Scorecard

Criterion	Risk rating  (residual risk without implementing the recommendation)Table 1 Footnote a	Risk remaining without implementing recommendation	Recommendation number
Risk management 2.1: Risk thresholds, mitigating responses, and escalation mechanisms are identified clearly.	3: Moderate risk	Risks that could jeopardize the progress of the project may not get resolved at the working level, and therefore deliverables may not be provided on time. In addition, the lack of inclusion of all risks to the risk register can leave the project vulnerable if these risks are not provided the necessary oversight and corresponding mitigation plans.	1
Program management 3.1: Requirements for the transitioning and future ownership of VaccineConnect have been identified, and a commensurate plan to ensure a fault-free transition, including resources required for sustaining the solution, has been established.	4: Significant risk	Lack of long-term vision and strategy leaves the program at risk of the following: confusing or ill-defined objectives; incoherent planning; costly duplication of operational and infrastructure requirements for continued program support; impaired visibility to threats, some of which may prove severe, and opportunities that the program faces; and inability to develop coherent plans for growing and efficiently promoting program services.	2
Program management 3.2: Controls are in place to ensure that monitoring and reporting on vendor performance, contracts, task authorizations, and change requests are adequately managed.	4: Significant risk	Lack of a vendor management role can lead to the risk of increased project cost, scope creep, delayed and suboptimal quality deliverables, and reduced effectiveness in managing the project.	3
Program management 3.3: Controls are in place to manage scope evolution, timelines, change management, and solution quality.	3: Moderate risk	A project management plan that is lacking a work breakdown structure, corresponding critical paths, dependencies, and mitigation plans can leave PHAC vulnerable to risks that may remain dormant for an unreasonable time. Examples of resulting impacts include confusing project objectives, ill-defined dependencies to the adverse impact on timeliness and quality of VaccineConnect deliverables, and unforeseen costs or requirements for re-planning project deliverables in light of new information.	4
Table 1 Footnote a Levels of risk: Minimal risk; Minor risk; Moderate risk; Significant risk; Major risk Table 1 Return to footnote a referrer