Audit Report - Immunization Program

Approved by Chief Public Health Officer
on July 27, 2012

Table of Contents

• Executive Summary
• Introduction
  • Context
  • Authority for Audit
• Background
  • Organizational Structure
  • Financial Information
  • Stakeholders and Partners
• About the Audit
  • Objective
  • Scope
  • Approach and Methodology
• Audit Findings and Recommendations
  • Governance and strategic directions
  • Risk Management
  • Human Resource Planning
  • Program Delivery
  • Results and Performance
• Conclusion
• Appendix A: Audit Criteria
• Appendix B: Management Action Plan
• Appendix C: List of Acronyms

Executive Summary

1. The overall objective of the audit was to provide assurance that the management control framework supporting the Immunization Program is adequate and facilitates decision making within the Agency.

2. The audit scope consisted of an examination of management processes in relation to core Immunization Program functions within the Centre for Immunization and Respiratory Infectious Diseases. The audit focused on governance, mandate and strategic direction, operational planning and monitoring, human resource management and performance measurement considerations. Observations relating to these areas are described below.

3. The audit was conducted from September 2011 to March 2012. The audit team conducted interviews with Public Health Agency of Canada (PHAC or the Agency) key personnel and program partner personnel including selected provincial/territorial and Non-Government Organizations (NGOs) representatives. The audit team also reviewed key documents and processes related to Immunization Program activities. The audit criteria were derived from the Audit Criteria related to the Management Accountability Framework: A Tool for Internal Auditors issued in March 2011 by the Office of the Comptroller General of Canada (OCG).

Mandate & Strategic Direction

4. The Agency's Immunization Program operates within a complex environment, with various federal/provincial/territorial (F/P/T) jurisdictions and organizations having responsibility for different components of immunization activities in Canada. Within this context, we found that the Agency has not yet clearly defined, nor communicated within the Agency and among its external stakeholders, its mandate and intended role for participating in immunization activities.

5. The absence of a clearly defined and communicated mandate, including roles and responsibilities for the Agency's Immunization Program, has contributed to a lack of clarity among external stakeholders on the Agency's purpose and intended leadership roles in its immunization activities. This constitutes a potential risk to the Agency's reputation.

Governance

6. From an external governance perspective, we found that the Agency has established governance structures through national committees that enable communication with key stakeholders of the Agency's Immunization Program, including representatives of P/Ts, NGOs and the vaccine industry.

7. From an internal governance perspective, we found that the Centre's organizational structure and management practices provide the key governance structures and processes to support the planning and management of Immunization Program activities that it undertakes.

Risk Management

8. The Centre has introduced elements of risk management into its work planning and monitoring processes. However, we found no evidence that a systematic risk management approach had been adopted as part of the Centre's broader strategic and operational management processes.

Human Resource Planning

9. The Centre has implemented certain elements of human resource planning; however, continued management focus is required in areas such as succession planning and work load analysis to better support Centre management in its delivery of Program activities.

Program Delivery

10. In completing its various Immunization Program activities, such as surveillance and immunization education, the Centre collaborates with a variety of external partners through a variety of program delivery models. We found no evidence to indicate that the methods by which program activities are delivered, and the business arrangements supporting program delivery, are regularly or systematically evaluated to ensure that the Agency is delivering activities related to the Immunization Program in an optimal manner.

Operational Planning and Monitoring

11. The Centre has made significant progress in implementing organizational and process changes to improve the consistency of business planning and monitoring activities across the Centre. Continued management focus is required to ensure full implementation of these changes, including work planning and monitoring approaches across the Centre.

Performance Measurement

12. Summary performance measures have been established for the Immunization Program within the Agency's Performance Measurement Framework. In addition, we noted that Centre management has implemented practices to support enhanced measurement of the resources consumed (e.g. salary and operations and maintenance costs) within the program. We found, however, that Centre management has yet to develop and implement comprehensive performance measures for the Immunization Program to enable broader measurement of program activities, outputs and results.

Conclusion

13. While progress has been realized in implementing a management control framework in relation to the Immunization Program, management attention is still required to:

• Develop and communicate a comprehensive mandate statement and related strategic objectives for the Agency in relation to its Immunization Program delivery;
• Expand and formalize the Centre's risk management process in relation to achievement of its broader program objectives;
• Ensure that program delivery models, including the roles played by and business arrangements with, delivery partners, effectively support program delivery goals; and,
• Enhance the Centre's performance measurement approach to include a focus on Immunization Program outputs and results in order to better inform management decision making.

Statement of Assurance

14. In the professional judgment of the Director General, Audit and Evaluation, sufficient and appropriate audit procedures have been performed and evidence gathered to support the accuracy of the audit conclusion. The audit findings and conclusion are based on a comparison of the conditions that existed at the time of the audit, against pre-established audit criteria that were agreed upon with management. Further, the evidence was gathered in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.

Management Response

15. The Agency's management agrees with our findings and recommendations and a management action plan is presented in Appendix B.

Introduction

Context

16. Immunization programs have had a major impact on the health status of the population, as many diseases are prevented through immunization. Since the introduction of mass immunization efforts, infectious diseases which used to be common are now rare.^{Footnote 1}

17. The National Immunization Strategy (NIS), established in 2003, guides the way the federal government provides leadership and works in partnership with provincial and territorial jurisdictions to improve the effectiveness and efficiency of immunization programs and to reduce inequalities in immunization coverage. This leadership position has arisen partly in response to the differences in vaccine costs, availability and knowledge across the country^{Footnote 2}.

18. Jurisdictional questions remain with regard to immunization, particularly with regard to the roles and responsibilities that federal, provincial/territorial (F/P/T) and non-governmental partners should assume in creating an effective Canadian vaccine strategy^{Footnote 3}.

19. In Canada, immunization is a shared responsibility among the federal, provincial and territorial governments. The federal government is responsible for the approval and regulation of vaccines, vaccine safety monitoring, and for providing evidence-based recommendations on the use of vaccines in Canada. The provinces and territories are responsible for funding, program planning, and the delivery of immunization programs in their respective jurisdictions. There is, however, a lack of uniformity across provinces and territories on some fundamental aspects of immunization programs such as approved and publically supported vaccines and vaccine schedules.

20. Immunization activities carried out by the Agency include nationally coordinated surveillance and research on vaccine-preventable and respiratory diseases and immunization coverage, ensuring vaccine supply, vaccine safety in post-market surveillance of adverse events following immunization, and P/T and international collaboration through the provision of technical assistance and policy/program advice. Most of these activities are carried out by the Centre for Immunization and Respiratory Infectious Diseases (CIRID, the Centre). For the purposes of this report, we focused exclusively on the immunization activities within the Centre and we refer to these activities as the Agency's core immunization functions, referred to collectively as the Immunization Program.

21. The Agency's Immunization Program activities, as carried out by the Centre and as examined in this audit, operate within a complex and multi-jurisdictional environment. In addition to the P/Ts, the Agency's program delivery partners and clients consist of a broad range of Non-Government Organizations (NGOs), including a number of medical profession associations, as well as academia and vaccine industry associations and manufacturers.

Authority for Audit

22. The Agency's Risk-Based Audit Plan for 2011-14 identified the audit of the Immunization Program as a priority for 2011-12.

Background

23. The Centre for Immunization and Respiratory Infectious Diseases, within the Infectious Disease Prevention and Control (IDPC) Branch, provides the Agency with strategic leadership in the development and implementation of an integrated approach to immunization against vaccine-preventable and infectious respiratory diseases in Canada. The Centre has four key objectives:

• Prevent, reduce or eliminate vaccine-preventable and infectious respiratory diseases;
• Reduce the negative impact of emerging and re-emerging respiratory infections;
• Facilitate pandemic preparedness and response activities on behalf of the Government of Canada; and,
• Maintain public and professional confidence in immunization programs in Canada.

24. To achieve these goals, the Centre collaborates with the P/Ts, other federal departments and other national and international stakeholders. Program areas of focus include nationally-coordinated surveillance, epidemiology, and research for vaccine-preventable infectious diseases; implementation of the National Immunization Strategy, including immunization registry development, national goals and objectives, vaccine supply, vaccine safety, and public and professional education; enhancing preparedness; national and international collaboration; and developing guidelines and protocols.

Organizational Structure

25. As of September 2011, there were 118 positions within the Centre distributed across the following organizations:

Figure 1 – Centre for Immunization & Respiratory Infectious Diseases

26. Areas of focus of the Centre's various Divisions are detailed below.

27. The Executive Director's Office is accountable for the identification, research and analysis of issues and the provision of strategic input, policy advice and management oversight to support the Director General, CIRID, in the delivery of national/ international public health initiatives, policies and programs.

28. The Immunization Division (ID) provides leadership, advice, coordination and support for the timely, equitable and sustainable recommendations for the use of vaccines and immunization programs for Canadians. These activities are delivered through the framework of the National Immunization Strategy, in collaboration with key stakeholders, and include:

• Supporting development of scientific and programmatic immunization recommendations for the safe and effective use of vaccines in Canada;
• Contributing to the increased knowledge and competency of health professionals in order to better protect Canadians from vaccine-preventable diseases; and
• Providing Canadians with access to reliable and accurate information about immunization and immunization programs, which allows them to make informed vaccination decisions for themselves and their children.

29. The Pandemic Preparedness Division (PPD) undertakes activities to strengthen national pandemic preparedness and response capacity by:

• Leading, coordinating and supporting activities of P/Ts and national stakeholders;
• Supporting improvements to Canadian capacity to respond to a pandemic, including responsibility for an annual grant and contribution fund; and
• Providing federal expertise on antivirals and vaccines for pandemic response.

30. This Division has been a major contributor to the execution of a five year $400 million health portfolio initiative which began in 2006 and was aimed at strengthening Canada's capacity to prevent and respond to health and economic impacts of avian influenza while increasing preparedness for a potential pandemic.

31. The Surveillance and Outbreak Response Division (SORD) undertakes activities to strengthen national surveillance and response capacity through:

• National coordination of surveillance and epidemiologic analysis dealing with the incidence and prevalence of disease in large populations, and with detection of the source and the cause of epidemics of infectious disease;
• Enhancement of the Canadian Adverse Events Following Immunization Surveillance System through the F/P/T Vaccine Safety Network to enhance vigilance with monitoring and reporting of adverse events following immunization; and
• Enhancement to vaccine safety capacity through a variety of mechanisms, such as harmonized and routine reporting mechanisms, networks and committees, and improved communications.

Financial Information

32. The following table outlines expenditures of the Centre by fund description and by fiscal year, starting in 2009-10. In addition to these expenditures, during the reference period the Centre provided oversight on additional expenditures, including the purchase of vaccines and antivirals as part of pandemic preparedness activities within the Agency.

Stakeholders and Partners

33. The Agency works with a variety of stakeholders and partners in providing national and international leadership in addressing the health and economic risks and impacts of vaccine-preventable and infectious respiratory diseases, they include:

• Provinces and Territories
• Other Government Departments
• Non-Government Partners
• International Partners
• Vaccine Industry Partners

About the Audit

Objective

34. The objective of this audit was to provide assurance that the management control framework supporting the Immunization Program is adequate and facilitates decision-making within the Agency.

35. A management control framework is intended to provide a common language for discussing and understanding the relationship between objectives, risks, controls and performance.

Scope

36. The audit scope consisted of an examination of Agency management processes related to the planning and delivery of the Immunization Program. Specifically, the audit addressed management processes in relation to the core Immunization Program functions within the Centre for Immunization and Respiratory Infectious Diseases.

37. The audit scope covered program activities undertaken during the fiscal years 2009-10, 2010-11 and 2011-12.

38. The audit scope excluded assessment of management activities related to the National Immunization Strategy due to the currently evolving nature of this initiative.

Approach and Methodology

39. The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.

40. The audit criteria (see Appendix A) were derived from the Audit Criteria related to the Management Accountability Framework: A Tool for Internal Auditors issued in March 2011 by the OCG.

41. The audit work was carried out between September 2011 and March 2012, and included an assessment of processes, procedures and management practices related to the administration of the Agency's Immunization Program. The audit team conducted interviews with PHAC key personnel and representatives of program partners, and reviewed documents and processes.

Audit Findings and Recommendations

Governance and strategic directions

Mandate and strategic direction

42. The complex operating environment for the F/P/T jurisdictions in the area of public health and immunization programs suggests a need for clarity in defining the roles and responsibilities of the respective jurisdictions and stakeholders involved in the various facets of immunization program development, delivery and monitoring.

43. The P/Ts have primary responsibility for making decisions on vaccines and delivering immunization programs within their jurisdictions. In addition, there are various Non-Government Organizations (NGOs) focused on immunization activities, particularly in the areas of promotion and education of the Canadian public. Given this program environment, there is a need for clear definition of the roles and responsibilities of the federal government, as carried out by PHAC, in its Immunization Program delivery.

44. We expected to find that a clearly defined mandate, with documented roles and responsibilities, had been established for the Agency's Immunization Program, and communicated widely within the Agency and among its external stakeholders. We also expected to find that CIRID had clearly defined and communicated strategic directions and objectives, and that these were aligned with its mandate. In addition, we expected that the Centre had established clear governance structures that support delivery against program objectives and that incorporate all key program delivery partners.

45. We reviewed the only strategic planning document available in IDPC and CIRID - the draft IDPC Strategic Plan, 2011-2016. The draft plan identifies the Branch's strategic goals and the results it is committed to achieving over the planning period. The draft plan also identifies the IDPC mandate and provides a summary description of the activities of CIRID.

46. The draft IDPC Strategic Plan demonstrates progress in defining the Branch's mandate and primary activities of CIRID in relation to the Immunization Program, as well as in shaping the Agency's interactions with its external stakeholders in national and international immunization activities. In our opinion, however, it represents only a first step toward achieving an approved and communicated plan. The Plan is currently a discussion document, last updated in December 2011. The stated focus of the Plan over the first year is to validate its goals and objectives and endorse an implementation strategy. As such, the Plan's commitments have yet to be reflected in the strategic planning efforts of CIRID and its operating divisions.

47. We were advised that a strategic plan for the Immunization Program does not exist within the Centre, nor have documented statements of roles and responsibilities for the divisions and the Centre, in relation to immunization activities, been established and communicated within the Agency. We were also advised, however, that Centre-level strategic planning, linked to the Agency's priorities, has been carried out for the past three years. These planning activities reflected a “bottom-up” approach to defining the Centre's objectives for its immunization activities. The operating divisions developed projects based on their interests and those of their collaborators in the Public Health Network. We could not find any examples of “top-down” strategic direction-setting and mandate definition by the Centre for its immunization activities.

48. We interviewed representatives of the Agency's program delivery partners and clients and solicited their views and understanding of several of the Agency's immunization initiatives. These included initiatives for immunization promotion, education, program monitoring, recommendation development, research, vaccine surveillance activities and vaccine supply.

49. Based on the responses provided and documentation reviewed, it was apparent that the Agency had not clearly defined nor communicated its mandate and intended role for participating in these various immunization initiatives. For some initiatives, such as the development of immunization recommendations, the Agency's mandate was implicitly understood by external stakeholders. For other initiatives, such as immunization education, awareness and promotion, the Agency's mandate and ultimate goals had not been clearly articulated and communicated. Stakeholder opinions included a desire for better communication of the Agency's roles and responsibilities in its immunization activities.

50. In our view, the IDPC Branch has made progress in documenting its draft strategic plan, and has described the primary activities of CIRID. However, the Branch mandate as it pertains to the Immunization Program has not been clearly defined nor communicated. Specific roles and responsibilities of the Centre and the Branch in the delivery of Immunization Program activities have not been comprehensively defined and communicated within the Agency and to its program delivery partners and clients.

51. Furthermore, the absence of a clearly communicated mandate with roles and responsibilities for the Agency's Immunization Program has contributed to a lack of clarity for external stakeholders on the Agency's purpose and intended leadership roles in its immunization activities. This constitutes a potential risk to the Agency's reputation.

Governance

52. From an external governance perspective, we found that the Agency has established governance structures via the NACI and the CIC and their respective working groups. These organizations provide membership opportunities for the various stakeholders of the Immunization Program, including representatives of the P/Ts, NGOs and, in specific cases, the vaccine industry. We also noted that this current governance structure is under review by the Agency in order to improve its effectiveness in supporting decision-making in immunization activities across Canada.

53. From an internal CIRID governance perspective, we found that the key governance processes in place to support immunization activities included an organizational structure that divides responsibilities into three functional areas/Divisions: Immunization, Pandemic Preparedness, and Surveillance and Outbreak Response. This internal governance structure includes regular meetings between the Centre's Director General and the three Division Directors to ensure effective direction and oversight is provided to support achievement of the Centre's objectives in relation to the Immunization Program.

Recommendation

54. The Assistant Deputy Minister, Infectious Disease Prevention and Control Branch, should develop and broadly communicate:

1. comprehensive mandate statement that clearly articulates the Agency's roles and responsibilities in relation to Immunization Program partners; and
2. related strategic objectives for the Agency's immunization Program activities.

Risk Management

55. The Agency's Policy on Integrated Risk Management states that risk management involves systematically considering the potential effects of risk on achieving objectives, and addressing key risks to objectives through appropriate decision-making and risk treatment actions. The Policy also states that “risk management shall be a consideration in strategic and operational planning and reporting activities”.

56. We expected to find that risk management processes existed to support the identification, assessment, monitoring, mitigation, and reporting of strategic and operational risks related to Immunization Program objectives and key initiatives.

57. We noted that the Centre has introduced limited elements of risk management into its work planning and monitoring processes. The activity/initiative work plans typically contain a section on the risks that may impact a Division's ability to carry out individual program activities. The Centre has chosen to refer to these risks as “pressures and issues”. The majority of the risks identified in the work plans are tactical in nature and do not represent broader program or strategic risks. Illustrative examples of these tactical types of risks include the risk that budgeted funds for an activity will not be spent prior to the fiscal year end, and the risk that a staff resource is not available to support a specific project.

58. Beyond the tactical risk management approaches noted above, we found no evidence that a systematic risk management approach had been adopted as part of the Centre's broader strategic and operational management processes. We view the lack of defined and approved strategic objectives for the Immunization Program as a limiting factor in the Centre's capacity to adopt a broader and more comprehensive risk management process.

Recommendation

59. The Director General, Centre for Immunization and Respiratory Infectious Diseases should expand and formalize the Centre's risk management process to include a greater focus on risks and mitigation approaches related to the achievement of its broader program objectives once these have been defined.

Human Resource Planning

60. Human resource planning is critical to ensure that resources are managed effectively in order to support program objectives. We expected to find alignment between CIRID human resource planning and monitoring practices and its strategic and operational planning practices.

61. Human resource planning as carried out within the Centre involves estimating the required resources to carry out activities contained in the divisional activity work plans. This practice generally includes estimated levels of required Full Time Equivalents (FTEs). However, we found limited evidence that Centre management has sufficient information on required levels of effort for the Centre's activities to validate the human resource estimates allocated to them. To that end, we noted that the Centre has recently begun an analysis to better understand workloads and their impacts on human resource requirements for certain Immunization Program activities.

62. Our interviews with Centre staff and external stakeholders also identified CIRID staff turnover, at all levels, as one of their principal concerns. Centre staff and external stakeholders noted that ongoing vacancies and turnover of staff that support some Immunization Program activities, such as the secretariat and technical support of NACI and CIC, had a significant impact on the effectiveness of the work of these committees. Similar observations regarding the adequacy of PHAC staff support to these committees were contained in a recent review performed by a task group focused on improving the structures and processes related to immunization recommendation development activities.

63. We noted that, in response, the Centre has begun to address employee turnover by introducing a process to document employee career goals as a means of addressing employee retention. The Centre has also introduced limited elements of succession planning through the identification of its key positions and inclusion of planned staffing processes in its work plans. While these initiatives represent an initial attempt to address these issues, a comprehensive approach has yet to be implemented.

64. In our view, insufficient workload and human resource planning information, particularly in relation to the Agency's support to NACI and CIC, constitute a potential risk to the Agency's reputation and its capacity to deliver on its Immunization Program commitments. We have confirmed these issues with CIRID management for consideration in their continued development of a comprehensive human resource planning and monitoring approach.

Program Delivery

Stakeholder Communication

65. The Centre relies on regular communication with external stakeholders to effectively deliver several components of the Immunization Program. External stakeholders include representatives of other federal departments (e.g. Health Canada), P/Ts, NGOs and the vaccine industry.

66. We expected to find that effective communication channels existed between the Centre and its external stakeholders to support timely, relevant and reliable information sharing, given the variety of partners and stakeholders with whom the Centre collaborates and interacts in supporting Immunization Program development and delivery.

67. The Agency works through multiple fora to oversee or support delivery of a number of Immunization Program activities. These fora include committees such as NACI and CIC and their related working groups, and other partnerships forged with, for example, NGOs focused on immunization. The level of PHAC involvement in these groups varies from that of taking a leadership role (e.g. committee Chair or Co-Chair) to and/or providing financial support, to that of a participant.

68. From our interviews with external stakeholders, we identified some concerns related to the adequacy of Agency communication practices in relation to some of the Immunization Program initiatives led by the Agency. Several external stakeholders indicated a need for better communication by the Agency concerning the status of specific initiatives. Examples include revisions to the Canadian Immunization Guide and NACI statements under development, both of which are referenced/utilized by the external stakeholders.

69. These external stakeholder observations constitute a potential risk to the Agency's reputation and its capacity to effectively deliver on its Immunization Program responsibilities. There is also a risk that PHAC may not capitalize on opportunities to influence those initiatives that are not led by the Agency. We have confirmed these stakeholder communication issues with CIRID management for consideration in their future planning and execution of Immunization Program activities.

Program Delivery Collaboration

70. In completing its various activities in support of the Immunization Program, CIRID collaborates with a variety of stakeholders through a variety of program delivery models. For example:

• The Surveillance and Outbreak Response Division contracts with the Canadian Pediatric Society, which manages the Immunization Monitoring Program, Active (IMPACT). IMPACT is a nation-wide pediatric hospital-based active sentinel surveillance initiative, which has, since 1991, provided Health Canada and PHAC with information on selected adverse events following immunization and vaccine preventable diseases.
• The Immunization Division provides annual funding to the Canadian Coalition for Immunization Awareness & Promotion to support the Coalition's delivery of the National Immunization Awareness Week and the National Influenza Immunization Campaign.

71. In relation to the different models of Immunization Program delivery, we expected to find evidence that the Centre systematically and comprehensively assesses and leverages, where appropriate, collaborative opportunities to enhance program delivery.

72. In reviewing the various areas of program delivery, we found no evidence to indicate that the methods by which program activities are delivered and the business arrangements supporting program delivery are systematically assessed to ensure that the Agency is delivering activities related to the Immunization Program in an optimal manner.

73. For illustrative purposes, we noted that the Centre supports a number of initiatives focused on the promotion of immunization, and the provision of immunization education to the Canadian public and public health care practitioners. Examples of these initiatives include:

74. Our interviews with external stakeholders confirmed that there is perceived value in PHAC's support and delivery of these initiatives. However, based on our discussions with internal and external stakeholders, we found no evidence to indicate that these initiatives had been assessed to determine whether they collectively represent the optimal approach to achieving program goals related to Immunization Program promotion and education.

Recommendation

75. The Director General, Centre for Immunization and Respiratory Infectious Diseases should undertake a systematic and comprehensive review of the Centre's current program delivery model, including the role played by, and business arrangements with, delivery partners to confirm the effectiveness of current program delivery approaches.

Results and Performance

Operational Planning & Monitoring

76. We expected that CIRID management had developed operational plans and objectives supporting the achievement of its strategic objectives and had implemented appropriate monitoring processes and practices.

77. In supporting the planning and delivery of program activities, the Centre has implemented an organizational structure in which each of the Divisional Directors for Surveillance and Outbreak Response, Immunization and Pandemic Preparedness is supported by a Business Planning Manager, who supports Divisional level business planning and reporting, resource management and internal business controls.

78. In addition to their divisional reporting relationships, the Business Planning Managers report to, and support, the Centre's Executive Director's Office in Centre-level business planning and reporting, performance and accountability reporting and resource management. This model has been introduced to improve the consistency and quality of business planning and monitoring activities across the Centre.

79. In addition to the Business Planning Manager organizational model, the Centre maintains a number of planning and monitoring tools including:

• Annually, the Centre undertakes a “strategic planning” exercise for each Division in which Division management document key activities/initiatives, the relative priority of activities and estimated activity resourcing;
• For 2011-12, the results of the “strategic planning” exercise were translated into detailed O&M budget tracking spreadsheets that are periodically (at least monthly) reviewed and updated by each Division; and,
• Divisions develop work plans for key activities/initiatives. Based on our review of selected activity work plans, elements generally include expected results for the activity/initiative, performance indicators, planned resources and key tasks, deliverables and timing.

80. We noted that for 2011-12, each of the three divisions of CIRID (SORD, PPD and ID):

• Has developed high-level plans that summarize key activities, priorities and estimated O&M resourcing by activity;
• Maintains a detailed budget planning spreadsheet to track the status of O&M spending by key activity;
• Maintains a “dashboard” to track the status of actual salary and O&M expenditures against Divisional budgets; and,
• Has developed, to an extent, work plans providing additional detail on planned Divisional activities and projects.

81. In reviewing selected planning and monitoring tools used, we noted that progress is being made in the introduction of a common operational planning and monitoring approach across the Centre's Divisions. However, additional focus is required to ensure implementation of a consistent and systematic work planning and monitoring approach across the Centre. Specific issues requiring attention include:

• Key activities/initiatives not supported by detailed work plans;
• Inconsistent levels of detailed information within work plans pertaining to, for example:
  • Expected results for the activity/project;
  • Performance indicators;
  • Planned resources (O&M and human resource levels);
  • Pressures/Issues (i.e. factors that may impact the meeting of project objectives); and,
  • Key tasks, deliverables and timing.
• Beyond the monitoring and updating of financial resources expended at an activity/project level that is currently performed, there is limited evidence that project work plans are systematically monitored and updated on a periodic basis to reflect actual work progress and results for activities/projects.

82. In order to address the observations noted above, management attention is required to clarify and communicate work planning and monitoring process expectations to ensure full implementation of a consistent planning and monitoring approach for Division activities/initiatives. We have confirmed these observations with CIRID management for consideration in their future planning and execution of Immunization Program activities.

Performance Measurement

83. A performance measurement framework refers to an objective basis for collecting information related to the intended results of a department or agency and its programs. It sets out the strategic outcomes and expected results to be achieved and specific outputs to be produced by its programs and it identifies the manner by which these elements will be measured.

84. Performance information is central to supporting effective management decision making. Management and external stakeholders require performance information to determine the extent to which expected results have been achieved.

85. We expected to find that Centre management had identified appropriate performance measures linked to planned results and that this information is used to support management decision making.

86. We found that the Agency has developed a Performance Measurement Framework (PMF) that includes performance indicators for the Immunization Program as presented in Table 1 for 2011-12.

87. While this PMF has been established at a summarized Agency level, Centre management has yet to develop or implement comprehensive performance measures within the Immunization Program. As a related observation, Centre management has yet to develop a logic model to articulate the relationship among program activities, program outputs and expected results. Articulation of a logic model would support Centre management's efforts to clarify its mandate and strategy, as observed elsewhere in this report.

88. In the absence of comprehensive performance measures, we noted that Centre management has implemented certain tactical elements to support enhanced performance measurement within the program.

89. In 2011-12, the Centre implemented a divisional dashboard for each of its three operational divisions. The information included in the dashboard (produced monthly) allows for detailed monitoring of program “inputs”, including actual salary dollars incurred versus planned for the division, and actual and forecasted O&M expenditures versus planned for key activities within the division. This information provides for more effective monitoring of overall program financial results and supports management decision making related to resource allocation during the fiscal year.

90. As part of the work planning template developed annually to document program activities and to support management of program activities, there is provision for project leaders to define the expected results for each project/activity and performance indicators. For project work plans reviewed during the audit, we noted that these elements are defined to varying levels of detail within the work plans. However, there is limited evidence that this planning information is compared to actual results in a systematic manner.

91. We recognize that these performance measurement tools provide a starting point to support enhanced performance planning and monitoring of program activities. However, additional management attention is required to translate these elements into a broader framework to support more comprehensive measurement of program activities, outputs and results.

92. We also noted that from a financial monitoring perspective, the Centre does not systematically or regularly measure or monitor financial resources required or incurred in relation to its various funding bases (e.g. A-base, TB submission funding, etc.).

Recommendation

93. The Director General, Centre for Immunization and Respiratory Infectious Diseases should enhance the Centre's performance measurement approach to include a focus on Immunization Program outputs and results in order to better inform management decision making.

Conclusion

94. While progress has been realized in implementing a management control framework in relation to the Immunization Program, management attention is still required to:

• Develop and communicate a comprehensive mandate statement and related strategic objectives for the Agency in relation to its Immunization Program delivery;
• Expand and formalize the Centre's risk management process in relation to achievement of its broader program objectives;
• Ensure that program delivery models, including the roles played by and business arrangements with, delivery partners, effectively support program delivery goals; and,
• Enhance the Centre's performance measurement approach to include a focus on Immunization Program outputs and results in order to better inform management decision making.

Acknowledgments

95. We wish to express our appreciation for the cooperation and assistance afforded to the audit team by management and staff during the course of this audit.

Appendix A: Audit Criteria

The audit criteria were derived from the Audit Criteria related to the Management Accountability Framework: A Tool for Internal Auditors issued in March 2011 by the OCG. The generic OCG audit criteria were adapted to relate to PHAC's Immunization Program.

Appendix B: Management Action Plan

Appendix C: List of Acronyms

Agency

Public Health Agency of Canada

CIC

Canadian Immunization Committee

CIRID

Centre for Immunization and Respiratory Infectious Diseases

CPHO

Chief Public Health Officer

DG

Director General

FTE

Full-Time Equivalent

IDPC

Infectious Disease Prevention and Control (Branch)

ID

Immunization Division

IIA

Institute of Internal Auditors

IMPACT

Immunization Monitoring Program Active

NACI

National Advisory Committee on Immunization

NGO

Non-Government Organizations

NIS

National Immunization Strategy

OCG

Office of the Comptroller General of Canada

PHAC

Public Health Agency of Canada

PPD

Pandemic Preparedness Division

SORD

Surveillance and Outbreak Response Division