Introduction: Evaluation of the National Emergency Stockpile System (NESS)
This evaluation was initiated in response to a recommendation in the 2010 Public Health Agency of Canada Audit of Emergency Preparedness and Response to assess the relevance of the National Emergency Stockpile System (NESS) program.
The evaluation focused on the continued contextual relevance of the program, including the needs, roles, priorities and considerations related to the current NESS program for both domestic and international responses. It explored the alignment of the program with federal and Public Health Agency roles and responsibilities, as well as its alignment with Government of Canada and Public Health Agency priorities. There was also an opportunity to explore challenges faced by the program and examine how public health stockpiles are managed in other jurisdictions.
Lines of inquiry included:
- What needs does the NESS intend to address? Do these needs still exist?
- Have needs changed over time? Has the program design changed to accommodate these needs? Are there further changes that should be implemented?
- Should the federal government be delivering the services provided by NESS, either in its entirety or for each type of asset?
- Is the NESS program consistent with the current Government of Canada and Public Health Agency mandate and strategic priorities?
- Which types of assets remain relevant for the Public Health Agency?
- Is there overlap/duplication or complementarity in service delivery?
While there was an opportunity to review domestic and international deployment patterns for NESS program assets, little other program performance data was available for analysis. As a result this report focuses primarily on addressing relevance, rather than performance, issues.
This report provides a description of the NESS program, its history and current context. It presents an assessment of linkages, outlines challenges and opportunities, and culminates in four recommendations. The appendices provide additional detail on NESS supplies, context and relationships, and considerations for improved life-cycle management. The appendices also present two domestic and three international case studies.
Management and staff should use the findings and recommendations of this evaluation for planning and decision making in preparation for future national, and potentially international, public health events.
The National Antiviral Stockpile (NAS) is a separate federal/provincial/territorial stockpile outside of the NESS and therefore falls outside of the scope of this evaluation.[Link to footnote B]
The evaluation was conducted by Evaluation Services, a group that is internal to the Public Health Agency but that is not involved in the program area responsible for administration of the NESS program. The evaluation analyzed information from multiple sources:
- key internal documents including:
- previous audits and reviews of the NESS program
- records of decision from various internal committee meetings
- correspondence and communication related to the NESS program
- program records on the deployment of NESS supplies
- a self administered survey based on open ended questions and focus group feedback from provincial/territorial representatives through the Council of Health Emergency Management Directors (CHEMD) and the Council of Emergency Social Services Directors (CESSD) of the Public Health Network
- a self administered survey based primarily on open ended questions from selected other government department representatives
- interviews with key senior managers and staff from the Public Health Agency, selected other government departments and non-governmental organizations, and external experts in the field of emergency preparedness and response
- interviews with specialists in a selection of other countries (Australia, Norway and the United States) involved in the management of public health stockpiles within their respective jurisdictions.
This evaluation incorporates multiple lines of evidence and a combination of qualitative and quantitative measures to ensure a balanced analysis of relevance and performance of the NESS program. This evaluation report has been written so that senior managers and external readers can readily focus on and understand the important issues being reported. To provide the reader with the appropriate context, it is necessary to clearly expose the limits of the evaluation in terms of scope, methods and conclusions. (For a detailed description of methodology, see Appendix B.)
Most evaluations are confronted with constraints that may have implications for the validity and reliability of evaluation findings, conclusions and recommendations. This section discusses the limitations with respect to the design and methods for this particular evaluation. Also discussed are the mitigation strategies put in place by the evaluation team to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.
Due to the large number of potential informants (internal and external) that could have been involved, the evaluators were reliant on the recommendations of the Public Health Agency program staff. Some of these individuals work with the program staff and may have a vested interest in the program. Attempts were made to compensate for this by interviewing a variety of stakeholders who were within and external to the program.
Short time frame
The short timeframe for completion of this evaluation did not permit an in-depth analysis of some data, such as program finances or the composition and the contexts of other federal and provincial stockpiles in Canada. However, through case studies and surveys, evaluators were able to gain a general understanding of both the program finances and the composition of other stockpiles, both domestic and international. The time frame did limit the number of international and domestic case studies reviewed for comparison. Importantly, the two domestic case studies include two key alternate organizations in Canada that provide domestic emergency supplies. The international case studies in particular are not intended to be a representative sample of jurisdictions with public health stockpiles, rather they were selected purposefully for their anticipated best practices.
Validity of qualitative data sources
There were several validity issues related to the data sources.
The questionnaires completed by federal and provincial government representatives were asking for retrospective data over the past five years. The validity of this information could be questioned as a result of such things as staff turnover and challenges with memory or recall. The interviews and focus groups were also in part retrospective and respondents' accuracy may also be questioned.
While detailed notes were taken during interviews and sessions, due to financial and time constraints, some interviews and focus groups were not transcribed verbatim.
Because the evaluation team was not able to initially contact the entire roster of the Council of Health Emergency Management Directors (CHEMD) and the Council of Emergency Social Services Directors (CESSD) through the survey and focus groups, this line of evidence is subject to some degree of non-response bias. That said, because both methods were employed, the participation rate was at 92% (only one jurisdiction was not represented through either the survey or the focus groups).
There have been a number of NESS reviews over the past decade and saturation may have affected perceptions of the value of input into the current review. In some cases, program staff turnover meant limited knowledge of the history of the NESS program and the location of some documents, such as inter-jurisdictional agreements dating from the 1960s.
Limited performance data available
There was limited quantitative performance data available. The evaluation team attempted to mitigate the imbalance between qualitative and quantitative data by incorporating multiple lines of evidence (i.e., the literature review, the document review, the surveys and focus groups, the key informant interviews, and the case studies) into its methodological approach. However, many of these lines of evidence – including the survey, which allowed for some quantification of results – relied on measuring individuals' perceptions.
Due to the lack of a reliable electronic inventory, deployment data about supplies in the National Emergency Stockpile System was not consistently tracked. The limitations of the qualitative performance data associated with deployment of the National Emergency Stockpile System, and mitigation strategies, have been noted in section 2.3 of the report.
Despite these potential methodological limitations, the evaluation team implemented a number of mitigation strategies to increase the validity of the findings. As multiple lines of evidence were examined throughout the course of the evaluation, Evaluation Services is confident that senior management can rely on this report for planning and decision-making needs.
- Date modified: