Section 3: Evaluation of community associated infections prevention and control activities at the Public Health Agency of Canada – Findings – relevance

3. Findings

This section provides a summary of the findings organized under two broad headings:

• Relevance: the need, federal public health role and priority in preventing and controlling community associated infections.
• Performance: the effectiveness, efficiency and economy of the Public Health Agency's activities in this area.

This summary is based on an analysis of the descriptive information found in available documents and on the themes that emerged from interviews and surveys with key informants.

3.1 Relevance

3.1.1 Community associated infections affect many Canadians, especially vulnerable populations

Community associated infections affect many Canadians. Although anyone can become infected, certain sub-populations are disproportionately affected. Epidemiological data indicate that individuals who engage in certain behaviours are most at risk for acquiring community associated infections. However, the risk is further influenced by living conditions and the structures put in place around them to deal with illness. Factors such as lack of social support, inadequate housing conditions, and limited access to health care services are important challenges that contribute to an individual's susceptibility to infection.^{Footnote 2} Figure 9 provides some key surveillance information on the community associated infections covered in this evaluation.

It is important to note that the figures provided above may underestimate the true burden of infection. Community associated infections are reportable on a voluntary basis at the national level through the Canadian Notifiable Disease Surveillance System (cases of tuberculosis are reported through a separate stand-alone system) by each province and territory. The number of reported cases is likely underestimated for a variety of reasons but primarily because many infected people do not exhibit symptoms and may not present to a healthcare practitioner for testing. As a result, many infections are underdiagnosed and underreported. Additionally, individuals who have limited access to health care services (e.g. vulnerable populations, people living in rural or remote regions, etc.) may be less likely to be diagnosed and treated.^{Footnote 20},^{Footnote 21} As an example, an estimated 21 per cent of the people infected with hepatitis C are unaware of their status.

Susceptible populations are slightly different for each of the community acquired infections.

• People who share contaminated equipment for injection drug use are more likely to acquire bloodborne infections such as hepatitis B and C. Enhanced surveillance and other studies suggest that acute infection rate is five times higher in Aboriginal individuals compared to non-Aboriginal people.^{Footnote 22}
• People who engage in high risk sexual behaviour are more likely to be exposed to sexually transmitted bacteria. They include: sex trade workers, sexually active youth under 25 with multiple partners, inmates, people who inject drugs, street-involved youth, and people travelling to a country with high prevalence of sexually transmitted infections.^{Footnote 23}
• People living in poverty and crowded housing are at increased risk of developing tuberculosis. In Canada, the incidence of tuberculosis disease is highest in Aboriginal people and recent immigrants who come from countries where the rates are endemic. Preliminary 2010 data show that while people of foreign descent made up approximately 22 per cent of the Canadian population, they accounted for 63 per cent of all reported tuberculosis cases in Canada in that year. Similarly, while Canadian-born Aboriginals peopleconstitute about 4 per cent of the Canadian population, they represent a disproportionate proportion (21 per cent) of all tuberculosis cases in Canada in 2010.^{Footnote 24}
• People who are coinfected with HIV and tuberculosis are at a greater risk of developing active tuberculosis due to their weakened immune system.^{Footnote 25} Chronic health conditions, including diabetes, may also increase one's risk of tuberculosis.^{Footnote 26}

Health consequences for these community associated infections can also last a lifetime. Many community associated infections have no symptoms for long periods of time and escape early detection and clinical intervention. They can become chronic if they are left untreated.Viral community associated infections (hepatitis B and C) can become chronic conditions and result in serious long-term negative health outcomes such as liver damage. ^{Footnote 27} Bacterial infections, if left undiagnosed and untreated, can lead to serious reproductive consequences. ^{Footnote 28} The pathogen responsible for tuberculosis can live in the body without making the individual sick (latent tuberculosis infection). However, if tuberculosis bacteria become active in the body and multiply, the person will go from having latent tuberculosis infection to active tuberculosis disease and become infectious.^{Footnote 29}

Trends

National surveillance data show that the number of cases and overall rates of reported chlamydia and infectious syphilis are rising, rates of ghonorrhea are stabilizing, rates of bloodborne hepatitis infections are declining, and rates of tuberculosis are relatively stable. However, as mentioned earlier, key populations are disproportionately affected.

In Canada, steady decreases in the rates of reported cases of hepatitis B and C in the general population were observed between 1998 and 2010. Theincidence of hepatitis B has declined following the implementation of vaccination programs.^{Footnote 30} Possible explanations for the declining trend in hepatitis C among people who inject drugs and general population might be a change in pattern of drug use and possible positive impact of preventive work. International research also suggests a similar trend in other developed countries.^{Footnote 31}

Reported rates of gonorrhea increased in Canada between 2001 and 2010. However, 2010 rates were almost the same as those from the previous year, suggesting a stabilizing trend.^{Footnote 32} Reported rates of chlamydia and infectious syphilis in Canada have been rising since 1997. International reports demonstrated that the increase in bacterial sexually transmitted diseases is also present in other countries such as: Australia, United States and United Kingdom.^{Footnote 33} Over the last decade in Canada, both chlamydia and gonorrhea rates increased in both men and women. Infectious syphilis rates also rose, driven mainly by increases among men.

Overall, Canada has one of the lowest tuberculosis disease rates in the world.^{Footnote 34} While significant progress has been made towards elimination, tuberculosis continues to be a major public health problem in Canada among First Nations, Métis, Inuit and foreign-born populations, with incidence rates within these populations many times higher than the overall national rate.

Antimicrobial resistance is a growing concern for some community associated infections.As defined by the World Health Organization, antimicrobial resistance is "resistance of a microorganism to an antimicrobial medicine to which it was previously sensitive. Resistant organisms are able to withstand attack by antimicrobial medicine so that standard treatments become ineffective and infections persist and may spread to others."

Gonorrhea has a long-established resistance to certain antibiotics. A recently published study ^{Footnote 35} presents the first series of clinical treatment failure of gonorrhea in Canada among patients of a Toronto clinic following the treatment with the only oral antimicrobial option. A rise in this type of antimicrobial resistance among gonorrhea patients was identified in parts of Asia as early as the late 1990s, followed by similar reports from other regions of the world over the next two decades.

Compared with international reports, drug-resistant tuberculosis has not yet been identified as a major problem in Canada. Nevertheless, the potential exists due to the increase and ease of international travel. In response, the Public Health Agency of Canada, in collaboration with the Canadian Tuberculosis Laboratory Technical Network and participating laboratories (representing all provinces and territories), developed the Canadian Tuberculosis Laboratory Surveillance System which was developed to monitor emerging trends and pattern in tuberculosis drug resistance patterns across Canada. On average, ten per cent of all tuberculosis specimens tested through this system have shown some level of antimicrobial resistance. Between 1997 and 2011, Canada reported a total of six cases of extensively-drug-resistant^{Footnote 36} tuberculosis.^{Footnote 37}

Travel and Migration appear to impact community associated infections rates.Community associated infections are present worldwide. With increased travel around the world, the risk of acquiring these infections by Canadians is higher if they travel in countries where the prevalence is high and they engage in risky behaviours.^{Footnote 38} Travelers' risk of contracting bloodborne disease such as viral hepatitis is increased if they are exposed to contaminated blood through medical or dental procedures, acupuncture, tattooing, drug injecting, etc. Migrants to Canada from countries where tuberculosis is endemic have an increased risk of developing active disease, particularly in the first years after arrival.^{Footnote 39}

In summary, community associated infections can affect anyone. Rates of infection in Canada are similar to those in other developed countries. National surveillance data show that numbers of cases and rates of reported sexually transmitted infections cases, such as chlamydia, gonorrhea and syphilis, are on the rise while rates of bloodborne viral hepatitis infections and tuberculosis are stable or declining. However, despite the overall declining rates in viral hepatitis and overall stable rates of tuberculosis, some vulnerable groups are more affected by risk factors that are interrelated and interdependent. Antimicrobial resistance has become a worldwide problem for treatment of some infections following the inappropriate use of medicine. Drug-resistance threatens the effectiveness of treatment especially for tuberculosis and gonorrhea and needs to be closely monitored. Public health interventions aimed at reducing the burden of community associated infections may require sustained collaborative efforts by multiple players in the health system and other sectors that may have a great impact on the health of society and of an individual.

3.1.2 The prevention and control of infectious disease is a priority for the Government of Canada as well as the Public Health Agency

The 2010 Speech from the Throne states that "protecting the health and safety of Canadians and their families is a priority of our Government." The Budget Plan 2012 also commits to "Supporting Families and Communities by protecting the health and safety of all Canadians and their communities..." These statements broadly underline the importance that the Government of Canada places on the prevention and control of community associated infectious disease, which is a recognized public health risk to all Canadians.

In its most recent Report on Plans and Priorities (2012-13), the Public Health Agency of Canada establishes 'Managing Public Health Risks to Canadians' as its first priority. This follows a longstanding commitment to combatting infectious disease, which has been a key Agency activity since its creation and has been featured in all of its previous Report on Plans and Priorities. The 2012-13 Report on Plans and Priorities includes two specific plans to address this priority, namely:

• reduce the potential for, and the impact of, infectious disease events, such as epidemics and pandemics, as well as foodborne illness
• enhance the health security of Canadians by reducing the potential for antimicrobial resistance in disease-causing organisms.

The 2011 Public Health Agency of Canada's Corporate Risk Profile identifies infectious disease as one of its top ten public health risks, stating that there is a risk that emerging infectious diseases will continue to create the potential for epidemics and pandemics that will result in considerable health, social and economic impacts. The Corporate Risk Profile also states that there is a risk that antimicrobial resistance in disease-causing organisms will continue to be a threat to the Canadian population resulting in increased morbidity, mortality and impact on the health system. Antimicrobial resistance has also been identified as a federal, provincial and territorial public health priority through the Public Health Network Council.

The Government of Canada's G8 website proclaims its commitment to the fight against infectious disease both domestically and abroad, stating: "A vigorous response to the threat of infectious diseases, the leading cause of death worldwide, is essential to global development and to the well-being of the world's population. Major diseases such as HIV/AIDS, tuberculosis, malaria, and measles continue to exact a heavy toll on economies and societies around the world, particularly in developing countries, impeding achievement of Millennium Development Goals (MDGs)."

The Government of Canada also recognizes the priority of preventing and controlling community associated infectious diseases which disproportionately affect vulnerable populations, including recent immigrants and aboriginal Canadians. Health Canada has clearly identified the threat posed to aboriginal people from tuberculosis, stating: "The rate of tuberculosis in Canada is among the lowest in the world, with a steady decrease being seen over the past 30 years. Despite this low incidence for its overall population, certain populations in Canada, including Aboriginal populations, are disproportionately affected by tuberculosis. While a significant reduction in Aboriginal tuberculosis rates has been seen over the past 30 years, the rates remain much higher than those of the non-Aboriginal Canadian-born populations."

The community associated infections activities carried out by the Centre for Communicable Diseases and Infection Control directly support the Public Health Agency of Canada's mission, as set out in its corporate strategic documents, to promote and protect the health of Canadians through leadership, partnership, innovation and action in public health. Their activities are clearly aligned with and support the priorities established by the Public Health Agency of Canada and the Government of Canada.

3.1.3 There is a federal public health role in the prevention and control of community associated infections

The Public Health Agency's role in preventing and controlling community associated infections generally aligns with and supports the Agency's broader public health mandate.

The Department of Health Act (1996) establishes the Minister of Health's powers, duties and functions relating to health. These include:

• the protection of the people of Canada against risks to health and the spreading of diseases
• investigation and research into public health, including the monitoring of diseases
• the collection, analysis, interpretation, publication and distribution of information relating to public health
• cooperation with provincial authorities with a view to the coordination of efforts made or proposed for preserving and improving public health.

The Public Health Agency of Canada Act (2006) outlines the responsibilities of the Agency to assist the Minister in exercising or performing his/her functions. Under the leadership of the Chief Public Health Officer, and in collaboration with its partners, the Public Health Agency is mandated to lead federal efforts and to mobilize pan-Canadian action in preventing disease and injury. It is also mandated to promote and protect national public health.

Since the Lalonde report (1974), it has been recognized that there is a federal role in public health to promote overall health and prevent infectious and chronic diseases. It is also widely accepted that the federal government has a role to play in taking action on public health issues of national concern in order to protect, maintain and improve the health of all Canadians. Furthermore, the Naylor report highlights the federal government's role in ensuring coordination across the different jurisdictions.^{Footnote 40}

In addition to coordination, the government plays an important role in the surveillance of diseases in Canada. As discussed earlier, the Department of Health Act sets out the Minister of Health's duties, which includes the collection, analysis and dissemination of information, as was also reinforced in the Kirby Report: "An effective health protection and promotion infrastructure also requires a strong capacity to communicate authoritative, evidence-based, information in a timely manner."^{Footnote 41}

The federal role is further defined in program authorities received to address hepatitis C. The Public Health Agency received funding to address hepatitis C through activities to:

• develop and implement sustainable prevention efforts for vulnerable populations
• raise awareness of general hepatitis C facts, hepatitis C testing, issues of coinfection and common risk factors
• build research and surveillance capacity, and build and support community-based capacity.

Considering the number of partners involved in preventing and controlling community associated infections, the Public Health Agency has also developed a series of memoranda of understanding or letters of agreement with various partners and stakeholders to help clarify roles and responsibilities in this area. The roles and responsibilities outlined in these memoranda mirror those outlined in the aforementioned program authority for addressing hepatitis C.

The Public Health Agency and other government departments

As previously mentioned, the Public Health Agency works with a number of federal partners to address community associated infections. This role tends to be advisory in nature by providing public health expertise, information, support, advice through collaboration with other government departments. These departments tend to focus on their key federal population:

• The First Nations and Inuit Health Branch at Health Canada: provides health services in on-reserve First Nations communities. First Nations and Inuit Health Branch: provides public health expertise, information, support, advice and/or collaboration in the areas of communicable disease control and health promotion/disease prevention, First Nation and Inuit-specific public health education and awareness, capacity building, health research, surveillance, data collection, dissemination, and policy development. As First Nations on reserve and the Inuit population tend to exhibit higher rates of infection for tuberculosis, sexually transmitted infections and viral hepatitis, Health Canada is a natural partner for the Agency.
• The Canadian Institutes for Health Research: support, promote and enhance hepatitis C associated research and training in the areas of epidemiological, clinical, biomedical and psychological/behavioural research.
• Citizenship and Immigration Canada: responsible for the medical screening of immigrants and refugees. Areas of Citizenship and Immigration's responsibility include immigrant and refugee health screening (although it should be noted that Public Health Agency is responsible for the Quarantine Act which protects public health by taking comprehensive measures to prevent the introduction and spread of communicable diseases).
• Correctional Service of Canada: responsible for populations in federal correctional facilities. Offender populations are at high risk for acquisition and transmission of infectious diseases due to their demographic and behavioural characteristics.

To facilitate activities, there are a number of current agreements in place between the Centre for Communicable Diseases and Infection Control and other government departments such as those mentioned above. These government departments are responsible for certain populations or activities, but look to the Public Health Agency to provide public health advice to prevent and control community associated infections.

The Public Health Agency and provinces and territories

Provinces and territories are responsible for delivering healthcare services within their own jurisdictions. However, memoranda of understanding have been developed to clarify reporting and data sharing responsibilities of the Public Health Agency and the provinces and territories in the prevention and control of community associated infections.

Various memoranda have been developed with many provinces and territories. However, these memoranda do not universally cover all community associated infections, nor do they consistently cover all community associated infecctions. Generally, the Public Health Agency's role is to provide the national picture of diseases by publishing national surveillance reports and related publications. The memoranda stipulated that the provinces and territories will submit surveillance information, which tends to be 'routine, electronic, case-by-case non-identifying data' during set time periods.

In addition, there are specific requests for public health capacity support through the Public Health Agency's Field Service Officer Program. The memorandum with Alberta, for example, stipulates that the Public Health Agency of Canada will support a Field Surveillance Officer who will provide coordination/assistance in the collection of data from Alberta Health and Wellness and Northwest Territories.

Providing the national picture and public health advice and guidance appear to be a clear role for the Public Health Agency and this was confirmed by provincial and territorial representatives. The majority of the Communicable Infectious Disease Steering Committee provincial/territorial survey respondents understand the difference between the Public Health Agency's role and provincial/territorial activities. While there is overlap with provincial/territorial responsibilities in certain activities such as developing guidance and health promotion, respondents appear to accept activities such as developing guidance as a federal public health role.

The Public Health Agency and key stakeholders

While the majority of survey respondents from other government departments understood the differences between the Public Health Agency's role and their respective roles, a substantial minority did not (6 of 19). Comments ranged from overlap with the First Nations and Inuit Health Branch of Health Canada (this was also expressed by a few of the provincial/territorial respondents) and Citizenship and Immigration Canada, to no real understanding of role because of little exposure to the Public Health Agency on these topics.

Some academics also expressed confusion with the Public Health Agency's role, but comments revealed differences in the perception of the federal public health role (from a funding organization to a national leader/coordinator). A majority did find that the Public Health Agency duplicated research or expert roles in this domain.

This was reiterated by some international key informants; some did not have a clear idea regarding the Public Health Agency's activities in comparison to what other government departments did in community associated infections. Although clear on the Public Health Agency's role in surveillance and providing national guidance, others were simply not clear on what the Public Health Agency did, versus other government departments (federal and provincial). For example, some weren't able to distinguish the work of the Public Health Agency vis-à-vis that of the Canadian International Development Agency in the area of tuberculosis.

Generally, though, there is an understanding of the Public Health Agency's role in providing national public health advice, guidance and surveillance to aid in preventing and controlling community associated infections.

3.1.4 Work plans reflect community associated infections needs, federal public health roles and Public Health Agency priorities

As previously mentioned, the Centre for Communicable Diseases and Infection Control leads the community associated infections work for the Public Health Agency, specifically that which aims to prevent and control tuberculosis, hepatitis B and C, sexually transmitted infections, and activities that promote sexual health. The Agency as well as the Centre has undergone several organizational changes within the timeframe covered by this evaluation. The purpose of the 2011 organizational change was to realign operations to support an integrated approach to infectious disease prevention and control, particularly with respect to sexually transmitted and bloodborne infections. Prior to the reorganization, the Centre was organized by disease area, which was not considered the most efficient structure to leverage common approaches, resources, knowledge and stakeholders across disease areas. The current organizational structure aligns internal resources by function, allowing staff to work together across a number of diseases, along common divisional functions. As such, work previously divided between HIV/AIDS, tuberculosis, hepatitis B, hepatitis C, and sexually transmitted infections are now integrated into the following structures: surveillance and epidemiology, guideline development and public health practice, policy, program and partnership.

Work plans were analyzed and activities were found to align well with the need, priorities and role of the Public Health Agency in infectious disease prevention and control activity.

A review of the four divisional work plans of the Centre for Communicable Diseases and Infection Control demonstrates that the priorities of the Public Health Agency and the Infectious Disease Prevention and Control Branch are generally well integrated into the Centre's operations. The Centre has developed work plans for each of its three priority areas: sexually transmitted and bloodborne pathogens, tuberculosis, and antimicrobial resistance. The work plans combine all related work from each of the four divisions, creating a Centre-wide work plan for each priority area.

As discussed in section 3.1.1, a public health approach to community associated infections must include an understanding and awareness of the root causes of illness, including an assessment of the social, environmental and cultural influences on behaviour change. Without such an approach, interventions risk being irrelevant to the target population for which they are intended, and hence ineffective in promoting positive change. In the case of tuberculosis and sexually transmitted and bloodborne infections, extensive literature exists on how the broad social determinants of health influence individual and group behaviour, and the availability (or not) of healthy choices and environments.

3.2 Performance

The performance of the Centre for Communicable Diseases and Infection Control's activities are summarized along four outcome areas: knowledge (includes the first and second immediate outcomes); use of evidence (includes the first and second intermediate outcomes); community capacity; and collaboration.

3.2.1 There is evidence, while limited, that Agency products led to an increase in community associated infections knowledge

The purpose of this section is to determine the extent to which the Public Health Agency achieved its two immediate outcomes related to knowledge: creating knowledge on the state, incidence and trends of communicable disease in Canada, and creating knowledge of specific public health communicable disease practice in Canada (for both practitioners and individual Canadians). The first question in assessing this area of performance is process oriented, and examines the degree to which knowledge products were created (i.e. knowledge outputs): how many products were developed and disseminated to stakeholders? The second question in assessing this area of performance examines the degree to which these outputs increased the knowledge of their intended target group: how many stakeholders read, understood and enhanced their knowledge as a result of the Public Health Agency's knowledge products? Furthermore, to what extent were knowledge translation strategies applied in the Public Health Agency's knowledge creation activities?

Knowledge created by the Public Health Agency - what have we produced?

The Public Health Agency produced a number of knowledge products during the five-year timeframe of the evaluation study. While a complete list of all products is not available, Appendix B presents some significant pieces created during this timeframe. Products of direct Public Health Agency activities include but are not limited to: guidelines for health professionals and decision makers, professional desk references, surveillance reports, brochures and other public awareness products, training materials and tools, publications and research reports.

Knowledge disseminated by the Public Health Agency - who did we reach and by what means?

The Public Health Agency disseminates its knowledge products through a variety of means including web posting, networking, workshops, posters and presentations to national and international forums, continuing medical education credits. While there is not consistent and ongoing tracking of the knowledge products disseminated, there is some distribution information available for several of these mechanisms, which is presented below. While this information does provide evidence of a significant amount of activity in this area, the absence of clearly defined performance targets or base-line measures make it difficult to assess how well the Public Health Agency performed in this regard.

Posters, Presentations, Workshops and Journal Articles

Between 2008 and 2012, the Centre produced more than 40 scientific products, presented to national and international conferences in United States and United Kingdom, as well as peer reviewed journals such as Journal of Viral Hepatitis, Journal of Hepatology, Canadian Journal of Public Health and Journal of Urban Health. In addition, the Centre prepared reports, presentations, awareness products for stakeholders, distributed direct or through online tools and emails.

Canadian AIDS Treatment Information Exchange

Some of the knowledge products are distributed through Canadian AIDS Treatment Information Exchange Ordering Centre in both hard-copy and electronic PDF downloadable format. The Canadian AIDS Treatment Information Exchange website is typically accessed by health professionals and community based organizations responsible for the prevention and control of infectious diseases including hepatitis B and C, as well as sexually transmitted infections.^{Footnote 42} Some information exists about the distribution of some products. In 2011-12, the top three Agency products distributed by the Canadian AIDS Treatment Information Exchange Ordering Centre were: STI: Sexually Transmitted Infections Booklet (151,448), Hepatitis C: Get the Facts (15,220) (funded by the Public Health Agency of Canada but owned and distributed by the Canadian Liver Foundation), and Questions & Answers: Gender Identity in Schools (13,281).

Electronic Distribution

Many community associated infections products are also available on the Public Health Agency's website. Figure 10 represents the top 10 communicable disease web pages for the period of April 1, 2010 to March 31, 2011.^{Footnote 43} It is worth noting that eight of the top 10 pages are sexually transmitted infections related.

The number of "page views" refers to each time a web page was accessed and does not include the number of times a PDF document was downloaded. For example, the page views number represents the number of times a page is viewed, and counts as a separate view if someone opens the page multiple times. Page views can be used to indicate the overall popularity of each page. The page view numbers exclude all internal Public Health Agency and Health Canada employee traffic, as well as non-human traffic such as robots and spiders that search engines (e.g. Google) use to index websites (software applications that run simple automated tasks to collect and analyse information such as checking links to other websites).

Source: Cardinal Path - What Do Canadians Want from PHAC Communicable Diseases Content?

Furthermore, according to a web metrics analysis conducted in March 2012, the top four keywords used in communicable diseases searches are also related to sexually transmitted infections. They are as follows, in order of highest frequency: Chlamydia, ITS (infections transmissibles sexuellement), STI (Sexually Transmitted Infections), and Sexually Transmitted Infections.

The web metrics analysis also included a visitor profile chart, identifying the groups who access the Agency's communicable diseases website the most. From a review of this analysis, it seems the audience is diverse and includes Canadians (53.2 per cent), international stakeholders (22.2 per cent), post secondary institutions (4.9 per cent), provincial governments (3.1 per cent), health organizations (2.1 per cent) and municipalities (0.7 per cent). (see Appendix C for additional detail)

Increased knowledge from Agency products

While there is not consistent tracking of knowledge outcomes, there have been several recent attempts by the Centre for Communicable Diseases and Infection Control to measure and assess its impact in this area. While the performance measurement approach is not yet systematic, the Centre has initiated work to measure its impact in this area, demonstrating the Centre's progress towards a more comprehensive performance measurement strategy. Attempts to evaluate the following three products have been made, and results hint at what could be the Centre's impact on knowledge gain among its stakeholders.

1. The Centre for Communicable Diseases and Infection Control offers on-line tuberculosis training modules for health care practitioners. Pre- and post-test results from these modules demonstrate that participants in the course have, for the most part, increased their knowledge as a direct result of the course, as demonstrated by higher percentages of correct responses in the post-test results.
2. The Centre for Communicable Diseases and Infection Control distributed a document titled, Primary Care Management of Chronic Hepatitis C: Professional Desk Reference - 2009, intended for a broad range of health care providers working in the area of hepatitis C prevention and control. Results from a feedback survey (designed and distributed in 2010 to users of the professional desk reference) demonstrate that the product increased the hepatitis C knowledge of 71 per cent of health care practitioners who used the professional desk reference.
3. The Centre for Communicable Diseases and Infection Control developed a set of Sexual Health Education Guidelines to facilitate a national approach to sexual health education for Canadian youth. As accompanying documents to promote uptake and use of the guidelines, a series of "Questions and Answers" documents were developed, two of which were assessed in an evaluation pilot. Results from this pilot test evaluation are mixed.^{Footnote 44} While the pilot was intended to measure "use of the documents" as opposed to knowledge gain, less than half of the respondents had read the two documents. Of those who did read them, less than one quarter found them to be more informative than other documents used on these topics. However, those who did read the documents used the information to influence policy and program development within their schools, which will be explored further in section 3.2.3.

3.2.2 Public Health Agency funded hepatitis C projects had some success in achieving their knowledge and awareness objectives, although it may have plateaued in recent years

Increasing hepatitis C knowledge, prevention and awareness for a variety of populations was a key objective of many of the projects funded through the hepatitis C contribution funding. As demonstrated in Figure 11, over 148,000 people were reached through the funding between 2009-10 and 2011-12, with highest reach to the general population, followed by people who inject drugs, youth at risk, and practitioners.

Figure 11: Populations targeted by hepatits C projects

Source: Public Health Agency of Canada, Project Evaluation and Reporting Tool Analysis

Figure 12: Number of individuals reached versus those reporting increased knowledge of hepatitis C transmission and risk

A roll-up of the project performance measurement data demonstrates that projects had some success in achieving their knowledge and awareness objectives. As shown in Figure 12, over the last two years, some of the projects tracked reach versus knowledge and awareness outcomes for individuals. Results show that these projects reported reaching 9,636 individuals over the two years, with a success rate of increased knowledge of hepatitis C transmission and risk averaging 51 per cent (4,934 individuals).

3.2.3 There is evidence that some Public Health Agency products are used by public health professionals to enhance training and practice

The purpose of this section is to determine the extent to which the Centre for Communicable Diseases and Infection Control achieved its two intermediate outcomes related to use of evidence: application of communicable disease advice and guidance to professional practice, and application of communicable disease knowledge to the development of infectious disease interventions.

In assessing this area of performance, it is important to understand who has used the Public Health Agency's products, and in what ways. It is equally important to understand why a product was used or not. Intermediate outcomes are more difficult to assess in that they are not immediately observable following an activity, and are less within the direct control of the Public Health Agency (as compared to immediate outcomes which are observable in the short term and over which the Public Health Agency has more control). As such, understanding why one piece of knowledge was used over another is an important part of the performance story, and critical for adjusting activities to increase effectiveness.

Use of evidence - who used the knowledge produced through Public Health Agency's activities, and in what ways?

Overall, there is a lack of performance information to fully assess the extent to which the Public Health Agency achieved this intermediate outcome. The Centre for Communicable Diseases and Infection Control conducted a few isolated evaluation studies of some of its products, from which some performance information can be drawn. Figure 13 provides a summary of the results reported by the Centre for Communicable Diseases and Infection Control on several of their knowledge products.

Even though the self-learning modules were being used by stakeholders, they are no longer available online.^{Footnote 46}Both interviewand survey respondents reported that the online self-learning modules were removed from the Agency website without explanation and several survey respondents expressed their dismay with the fact that they are no longer available, noting that the sexually transmitted infections self-learning modules were an excellent online resource and survey respondents called the loss of the online modules unfortunate. Figure 14 describes what Public Health Agency products are being used by key informants and survey respondents.

Each of the four groups surveyed (other government departments, provincial/territorial representatives, non-profit sector and researchers) noted that the Agency's surveillance reports, guidelines (e.g. sexually transmitted infections guidelines) and standards (e.g. hepatitis and tuberculosis) are appreciated and used to help those groups in their community associated infections prevention and control activities.

3.2.4 Timeliness is a key challenge that limits stakeholders' use of Public Health Agency products

A common theme emerged from key informant interviews and surveys regarding timeliness of Public Health Agency products which warranted further exploration. Stakeholders frequently cited timeliness (meaning the extent to which documents are produced in time for them to be used in practice, programming or policy directions) as a challenge to the use of knowledge products created by the Public Health Agency.

Effective and timely evidence is critical for stakeholders to accurately track, plan for and respond to public health issues. As previously mentioned, the majority of survey respondents reported using the Public Health Agency's surveillance reports, guidelines and standards. However, key informants and stakeholder respondents also stated that the lack of timely information limited usefulness of the Public Health Agency's products. The Public Health Agency's inability to get out timely data is impacting the relevance of information, as stakeholders are forced to look elsewhere for information or gather it themselves. Providing more timely national surveillance data and more timely distribution of guidelines were identified by survey respondents as specific things the Public Health Agency could do to strengthen or improve its efforts to address community associated infections.

There are some natural time delays that are out of the Public Health Agency's control. The Public Health Agency depends on experts to provide advice on guidance and it also depends on provinces and territories to send the Public Health Agency public health surveillance data. It is possible to experience delays in receiving both, not to mention that the Public Health Agency must also verify the accuracy of the surveillance data which can take additional time to confirm. Other considerations, such as respecting Open Government, must also be factored into releasing information on public websites.^{Footnote 47}

However, there are built-in time delays within the Public Health Agency as well. To help understand the Public Health Agency's challenge with timeliness, two additional analyses were conducted: an analysis of selected products in the Management of Executive Correspondence System and a comparison of international and domestic community associated infections surveillance reporting.

Management of Executive Correspondence System

The Management of Executive Correspondence System (MECS) is an electronic correspondence tracking system used to manage the flow of correspondence between the Chief Public Health Officer, the Associate Deputy Minister and the Branches of the Public Health Agency. An examination of MECS was undertaken to assess the timeliness of the Public Health Agency's products. The search examined the audit trail of several Community Associated Infections reports, data tables and guidelines, including Tuberculosis in Canada, data tables on hepatitis B and C and the Canadian Guidelines on Sexually Transmitted Infections. The MECS analysis revealed a number of different approval points within the Centre, the Branch and across the Public Health Agency for releasing community associated infections products. However, there was no consistent point where approvals were delayed. It may be that the sheer number of approvals needed continuously slows down the whole process.

Comparison of international and domestic surveillance reporting

Timeliness is a key aspect of any public health surveillance system. It depends on a number of factors, including the nature of the disease under surveillance and how the data will be used. Various levels of public health system (local/provincial/territorial/federal) have different timeliness requirements for surveillance data. National surveillance data allow the federal public health system to support monitoring trends over time, to inform setting the priorities and allocation of public health resources, to monitor the effectiveness of disease control measures or identifying high risk populations.^{Footnote 48}

To provide an international perspective for the trends in surveillance data reporting, publication of epidemiological data of hepatitis B and C, sexually transmitted diseases and tuberculosis in Canada were compared to those in other western countries with similar population health status and well-established public health infrastructures. Countries selected for comparison are the United States, Australia, and United Kingdom, recognizing there may be other factors related to speedier collection and release of information. In addition, the frequency of epidemiological publishing at the provincial/territorial level was compared with the reporting at the national level. Statistics presented below are drawn from publicly published health reports on official websites.

The analysis identified a significant delay in publication of public health data at the national level compared to other countries or provinces. Other countries are reporting data on a monthly or weekly basis in a table format, to facilitate the identification of events that require prompt public health intervention. The reporting delay varies between one week and two months. National surveillance data in other countries is presented annually with a reporting delay of one year. Similar results were found for some of the Canadian provinces. In contrast, reporting on Canadian national surveillance of community associated infections is delayed by two or more years. In addition some of the national reports are not fully available to the public and requests for release have to be addressed to the Centre for Communicable Diseases and Infection Control. (see Appendix D for more detail)

Interviews with program staff in the Centre for Communicable Diseases and Infection Control showed that some surveillance reports are distributed through a list of stakeholders to avoid some of the publication delays.

A national meeting on Notifiable Disease in Canada held in March 2011 and organised by the Centre for Communicable Diseases and Infection Control, identified some of the challenges for the timeliness of reporting of notifiable disease, including community associated infections. They include: inconsistent reporting timelines from provinces and territories, financial constraints and capacity issues that hinder the ability of all levels of government to report, sharing information with stakeholders and the public as well the additional challenge of not enough attention from decision makers.

Internal key informants expressed their concern about delays in publishing some surveillance reports. Where possible, the Centre for Communicable Diseases and Infection Control employed alternative methods to ensure stakeholders received information without further delays - such as distributing some surveillance reports directly via email or through the Canadian AIDS Treatment Information Exchange.

While there is some evidence that key stakeholders appreciate and are using the Centre for Communicable Diseases and Infection Control's products, the Centre's ability to remain relevant is at risk because surveillance reports, guidelines and standards are not available in a timely manner.

3.2.5 Projects funded through the hepatitis C grants and contributions demonstrated use of evidence through improved access to health services, changes in practice and policy, and changes in individual behaviour

Projects funded through the hepatitis C grants and contributions demonstrated use of evidence through improved access to health services, changes in practice and policy, and changes in individual behaviour.

Figure 15: Improved access to health services

Source: Public Health Agency of Canada, Project Evaluation and Reporting Tool Analysis

Improved access to health services

Through the Project Evaluation and Reporting Tool questionnaire, project staff reported their projects had an impact on access to health or social services. As can be seen in Figure 15, approximately three quarters of the projects (ranging from 74 per cent to 78 per cent) intended to improve access to health and/or social services. While the projects reported a high rate of success in 2009-10 (80 per cent of projects), there were lower rates of success in the following two years of funding (54 per cent and 43 per cent). While only half of the projects (approximately) monitored change in access to services, the rate of success drops in each consecutive year of funding. The reason for this drop is not known, and would be important for determining how to direct programming in future years of funding.

It is important to note the limitations with this data however, which reflects self-reported data from funding recipients. While quality control checks are in place within the Public Health Agency to ensure the greatest degree of accuracy in reporting results, there is variability in terms of how access to health services is defined, measured, and assessed. This may lead to an inconsistent interpretation of how access is defined by project areas, how projects monitor their activities and how they report on improved access to healthcare services. There is no other evidence to substantiate or explain the results presented above.

Change in practice and policy

Projects were asked to describe evidence of successfully changed practice of practitioners, professionals, and/or service providers that occurred as a result of their project's activities. Projects described evidence of stronger collaboration and coordination of professional services, such as breaking down professional silos to provide a continuum of care for unreached and under-served populations. Greater knowledge and awareness of specific issues related to hepatitis C, such as the nature of addictions and harm reduction strategies, resulted in increased community referrals for care and greater interest among professionals to become involved in project activities. Other examples include changes in the way professionals engage and communicate with certain populations, such as at-risk youth, and shifts in primary care practice to fill service gaps. Heightened awareness of stigma and discrimination issues for individuals infected with HIV or hepatitis C resulted in shifts in the way in which practitioners apply confidentiality and client-based approaches to their practice.

Through the Project Evaluation and Reporting Tool, project staff reported on their project's impact on policy. As can be seen in Figure 16 below, less than half of the projects intended to change policy. Examples of policy targets include school board policies related to the availability of condoms, raising awareness of the importance of hepatitis C with decision makers, influencing community-based harm reduction policies such as safe needle disposal protocol, and changing organizational policies to promote a safe culture for at-risk youth to access services.

Figure 16: Policy change

Of the projects that did aim to change policy, there was a fair rate of success, ranging from 57 per cent in 2009-10, to 78 per cent in 2010-11, down to 30 per cent in 2011-12. Examples of successfully changes policy include the availability of needle drop boxes, the shifting/reorienting of services towards the provision of harm reduction supplies, expanded hours of service for at-risk populations, agreements among health centres to develop referral pathways for clients, and endorsement of local harm reduction strategies by municipal and provincial authorities.

Change in individual behaviour

Use of evidence can be inferred from the reports of individuals intending to change their behaviour as a result of their involvement in the hepatitis C project activity. While behaviour change is an intermediate outcome not typically observed in the short term, funding recipients measured their participants' intention to change their behaviour as a result of project activities. Based on the Project Evaluation and Reporting Tool data for the last two years of project funding, there are mixed results in terms of individuals reporting intention to change behaviour. As seen in Figure 17, in 2010-11 over 94 per cent of the individuals reached through project activity reported that they intended to adopt practices to reduce hepatitis C transmission. This rate was much lower in the following year, with only 41 per cent reporting this result. The reason for this drop in reported success is not known. While only speculation, possibilities could include anything from a threshold effect (individuals had already gained knowledge from the previous year of funding, therefore no increase in knowledge was reported) to ineffective knowledge development strategies. Research into the reason for the drop would yield a more accurate interpretation of its cause, and would be important for determining how to direct programming in future years of funding.

Figure 17: Intention to change behaviour

Source: Public Health Agency of Canada - Project Evaluation and Reporting Tool Analysis

Limited evidence on how Public Health Agency information is used internally to advance program and policy decisions

The Centre for Communicable Diseases and Infection Control does not necessarily identify staff within the Public Health Agency as targets of intended knowledge. However, as demonstrated by the Centre's logic model in Appendix A, coordination within the Centre is critical to successfully achieving success, as the collective outputs (i.e. products) of the Centre reply on input from each of the four divisions. Furthermore, while the Centre for Communicable Diseases and Infection Control does not have direct control over the degree to which external stakeholders use the evidence it produces, it can effectively ensure that its own decision making structures take all evidence into account when determining policy and programming directions. In order to do this however, there must be efficient and effective internal coordination and communication mechanisms which allow for joint initiatives, sharing of best practices and expertise, and input into policy and programming directions.

There are some positive indications that evidence produced by the Centre for Communicable Diseases and Infection Control is used internally, both across divisions within the Centre for Communicable Diseases and Infection Control and within some areas of the Public Health Agency. Internal key informants highlighted examples of internal use of evidence, such as:

• the use of surveillance data in mathematical modelling, statistics and predictions for hepatitis C and tuberculosis
• the use of surveillance data in the development of the guidelines (e.g. Sexually Transmitted Infections Guidelines)
• the use of surveillance data in the development of posters/abstracts at symposiums and conferences
• the use of mathematical modelling, statistics and predictions in surveillance estimates
• the Centre for Communicable Diseases and Infection Control provides advice and expertise to other areas of the Public Health Agency such as the Canadian Field Epidemiology Program for tuberculosis outbreak investigations, the Centre also provides advice and expertise to provinces and territories when support is needed and requested for a tuberculosis outbreak
• the Centre for Communicable Diseases and Infection Control also provides case information to Quarantine/Travel Health when travel restriction is required.

However, there are also areas in which improvements could be made, particularly with respect to the use of performance measurement information in internal decision making (see Section 3.2.9 for a full discussion of this issue). Internally, challenges in the development of knowledge products occur with the flow of information and with understanding the needs of knowledge users. Some internal key informants indicated that the exchange of information among knowledge creators within the Centre for Communicable Diseases and Infection Control is taking place to some extent but needs improvement. For example, surveillance information is helping in setting priorities for grants and contributions, and guides other programmatic activities, but more feedback is needed from the policy and programming area to better understand what the information users need. Constructive feed-back from front line non-governmental organizations through the Grants and Contributions Program is important but missing. They can provide information regarding gaps in surveillance data.

3.2.6 Collaboration activities are important to infectious disease work in Canada, and the Centre for Communicable Diseases and Infection Control's collaboration activities are well perceived by some stakeholders

The purpose of this section is to determine the extent to which the Public Health Agency achieved its third intermediate outcome of national and international stakeholder support, collaboration and responsiveness. Before assessing this area of performance, it is important to understand with whom the Public Health Agency collaborates in the area of infectious disease prevention and control, why this collaboration is important, and in what ways it occurs. Following this, an assessment of the results related to collaboration will be presented, examining the degree to which stakeholders are satisfied with the Public Health Agency's collaboration.

Why collaborate?

Collaboration can be defined as "an organizational phenomenon designed to achieve desired ends that no single organization can achieve acting unilaterally."^{Footnote 49} Collaboration is complex and implies more than the development and maintenance of a committee or working group. According to academic literature, collaboration can be theorized along a continuum, with a hierarchy of relational characteristics that denote five levels of collaboration: networking, cooperation, coordination, coalition and finally, collaboration.^{Footnote 50}

Assessing the appropriate level of stakeholder engagement on issues related to infectious disease prevention and control in Canada is important for a number of reasons, including:

• Shared jurisdiction and multiple roles
  • The Public Health Agency shares jurisdiction over public health and health promotion with numerous stakeholders, including the federal health portfolio partners (e.g. Health Canada), provincial and territorial governments, health charities, academics, researchers, subject-matter experts, and other health-related organizations. Each player has a different role and makes a unique contribution to the overall Canadian effort. In order to effectively leverage and complement the efforts of others in Canada, the Public Health Agency must collaborate with them.
• Levers outside the health system
  • The root causes for how and why an individual (or community in the event of an outbreak) contracts an infectious disease lie beneath the social, economic and environmental circumstances of his/her life. While individual behaviour change through education and social marketing campaigns can be a tempting focus for investment, an effective program must be comprehensive in nature and based soundly on an understanding of the influence of the broader determinants of health on behaviour. While the Public Health Agency does not directly control many of these factors, it can work collaboratively with other organizations to influence positive change and encourage uptake of knowledge and evidence in the development of intersectoral policies and programs.
• Better efficiency and effectiveness
  • Through collaboration, the Public Health Agency can reduce duplication of effort, maximize complementarity of programming, and pool resources to stretch value for money. In addition, collaboration can increase the effectiveness of the Public Health Agency's objectives by increasing organizational support (buy-in) for a specific approach, program or policy, and by leveraging organizational influence through the social networks of partnering organizations.^{Footnote 51}

With whom does the Public Health Agency collaborate?

Within the Public Health Agency, collaboration is principle guiding the work of the Centre for Communicable Diseases and Infection Control, as well as an outcome. Collaboration is achieved through the establishment of partnerships with other organizations in Canada working on issues of infectious disease prevention and control. The Centre's primary partners are: other federal government departments, provincial and territorial governments, non-governmental organizations, researchers and academics, and international agencies. Figure 18 summarizes the Centre's key partners and partnership mechanisms.

Collaboration results

In order to assess collaboration results, a web survey was designed and administered to stakeholders identified by staff within the Centre for Communicable Diseases and Infection Control. The survey assessed overall stakeholder satisfaction with the Centre for Communicable Diseases and Infection Control's collaborative activities. Results from the stakeholder surveys received are presented below.

Provincial and territorial governments

Due to shared jurisdiction over public health, provincial and territorial governments are key partners for the Public Health. The primary mechanism for collaborating with them is through the Communicable and Infectious Disease Steering Committee of the Public Health Network. For a more detailed description of the history of the Public Health Network and how it supports federal, provincial and territorial collaboration on infectious disease prevention and control activities, see Appendix E.

Provincial/territorial survey respondents were generally positive about the Agency's collaborative efforts with most agreeing or strongly agreeing that Agency-led activities had contributed to increased collaboration on community associated infections in Canada. Further, most respondents also agreed or strongly agreed that the Agency involved all relevant stakeholders in a meaningful way. While the majority of provincial/territorial survey respondents felt that Agency-led activities have contributed to increased collaboration, some commented that there remains an opportunity to expand stakeholder representation to include not only the provinces and territories, but other community representatives. Others noted that greater intergovernmental engagement must occur to effectively address community associated infections.

Internal

At the Public Health Agency, the Centre for Communicable Diseases and Infection Control collaborates regularly with the Office for Public Health Practice and the Centre for Emergency Preparedness and Response, particularly when addressing tuberculosis outbreaks, such as the recent outbreak in three Nunavik communities.

The Public Health Agency becomes involved in an outbreak when it receives a request for assistance from the affected province or territory. The initial request can come through the Office for Public Health Practice, who provides technical epidemiologic expertise or the Centre for Communicable Diseases and Infection Control, who provide the subject matter expertise. The two will be in contact through established notification processes, and because the collaborative relationship has been established any confusion over things like reporting requirements are quickly resolved through phone conversations.

According to key informants, outbreak response activities appear to be well coordinated and roles and responsibilities are generally clear and quickly clarified when questions arise. Key informants reported strong collaboration, especially in the area of tuberculosis. In the case of a recent tuberculosis outbreak in Nunavik, the field epidemiologists from the Office for Public Health Practice prepared a report for the Quebec Ministry of Health. This report was completed with input from the Centre for Communicable Diseases and Infection Control, and was designed to help the province learn from the outbreak and provide recommendations to help the province improve its monitoring of tuberculosis.

The Centre for Communicable Diseases and Infection Control collaborates with the Centre for Emergency Preparedness and Response by providing advice and assistance to the Quarantine Program to enact the Quarantine Act, supporting case management and contact investigation, placing travel restrictions on someone who is deemed infectious with tuberculosis and who indicates a willingness to travel against medical advice. The Centre for Communicable Diseases and Infection Control liaises with the International Health Regulation Program in the Centre for Emergency Preparedness and Response, keeping them informed on all the international communications.

One senior manager noted that strategic links between communicable and non-communicable diseases was important. For example:

• A great number of people infected with hepatitis B and C will develop cirrhosis of the liver or liver cancer.
• In addition, there are mental health implications of living with tuberculosis, viral hepatitis and sexually transmitted infections (such as depression). Likewise, mental health status, through its influence on an individual behaviour choices, can affect an individual's risk of acquiring an infectious disease.
• Social determinants of health such as education, gender, housing, income, race, ethnicity, social exclusion and unemployment influence individual vulnerability for acquiring tuberculosis, sexually transmitted infections and viral hepatitis.

3.2.7 Collaboration activities have recently declined with other partners and stakeholders although the impact of this decline is not readily evident

Other government departments

As previously mentioned, the Public Health Agency has signed Memoranda of Understanding or Intergovernmental Letters of Agreement with Citizenship and Immigration Canada, Correctional Service of Canada and Health Canada's First Nations and Inuit Health Branch. However, there is limited evidence of recent collaborative efforts as a result of existing Memoranda of Understanding or the Intergovernmental Letters of Agreement. As mentioned in figure 19, one of the Public Health Agency's primary responsibilities is to provide public health advice for specific populations covered by other federal government departments, such as First Nations living on reserve, offenders in federal prisons and those immigrating to Canada. These populations, as indicated in section 3.1.1, tend to have higher rates of community associated infections due to a variety of factors.

Survey respondents who responded positively noted that collaborative efforts have led to a stronger response to community associated infections, specifically with reference to priority populations such as street youth. In addition, the Public Health Agency is perceived as a trusted source of information/guidance on issues such as tuberculosis, which helps with work in their own areas. Specifically, there are indications that collaborative efforts between the Public Health Agency and Health Canada's First Nations and Inuit Health Branch appear to be working well. Key informants within the Agency confirmed this view.

However, not all government departments perceived that collaborative efforts were working well. Some survey respondents mentioned that materials or resources were not relevant to their population and that further information sharing (including what guidance was available or being drafted) was needed between the Public Health Agency and their department. While the impacts of declining collaboration were not evident from responses, it was apparent that public health advice was needed for the work they do with their respective populations. Concerns were also expressed specifically regarding the changes in collaborative activities on tuberculosis and what potential impact this may have on addressing this disease.

Finally, some federal government department survey respondents noted that data sharing between jurisdictions was one of their main challenges when addressing community associated infections, which they felt could be addressed by the Public Health Agency by ensuring that data sharing agreements are in place. They perceived the Public Health Agency as having a liaison role between the federal government and the provinces and territories on public health issues, including data sharing between jurisdictions.

Some provincial/territorial respondents echoed these concerns when asked how effectively the Public Health Agency has engaged other government departments on public health issues. Most responded that they either did not know about engagement activities between government departments or that the Public Health Agency was not doing enough to engage with other federal departments. One respondent expressed that 'all departments should be involved to address the underlying determinants of health' as an important reason for this type of collaboration. Some specifically noted that there has been a decline in engagement and a loss of established communication links since the reorganization of Public Health Agency structures.

Non-governmental organizations

As mentioned in figure 18, the Public Health Agency partners with non-governmental organizations to develop and revise guidance for public health professionals. For example, the Canadian Tuberculosis Standards are developed by the Canadian Thoracic Society with Public Health Agency involvement. However, the majority of non-governmental organization stakeholders surveyed noted that there has been a decline in engagement with most (73 per cent) either unaware of any engagement activities or dissatisfied with how effectively the Public Health Agency has engaged non-governmental organizations on public health issues.

Academics and researchers

Similarly, the Public Health Agency collaborates with academics and researchers to develop and revise guidance for public health professionals. These individuals provide expert advice on the prevention and control of community associated infections. Specifically, the Public Health Agency coordinates the work of the Expert Working Group that has been tasked with developing the Canadian Guidelines on Sexually Transmitted Infections. And although the majority of researchers surveyed (63 per cent) were either unaware of any engagement activities or expressed dissatisfaction with how effectively the Public Health Agency has engaged that sector, there is no evidence to say that this group has not been involved in the development of the guidelines.

The lack of information on the impact of declining collaboration could be a result of the survey tool that was employed to gather the views of non-profit organizations, academics and researchers. While researchers were asked about the effectiveness of the Public Health Agency's engagement of the research community, there was no follow-up question on the impact of these collaborative activities, be they effective or lacking. Therefore, it is difficult to deduce impact of declining collaboration based on the evidence available.

International partners

Collaboration, once extensive and appreciated, also appears to be declining with international partners. Key informants from the United States Centers for Disease Control and Prevention and the Pan American Health Organization recalled a close working relationship with the Public Health Agency in the past. In particular, the Pan American Health Organization counted on technical collaboration with the Public Health Agency on activities such as:

• the development of a document entitled Sexual Health for the Millennium
• the development of the Blueprint for the Provision of Care for Gay Men and Other MSM in Latin American and Caribbean Countries
• support of issues related to gender identity in school settings and issues related to sexual orientation in school settings.

International key informants expressed satisfaction with the collaboration with the Public Health Agency prior to 2010; however, key informants reported a loss of contact with the Public Health Agency in recent years, pointing to the Public Health Agency's recent reorganization. From the evidence provided, the Public Health Agency's activities appear to primarily focus on providing information, support and advice to various organizations. Therefore the potential impact of the loss of these types of activities on the Public Health Agency is difficult to deduce.

However, there are benefits of working with international partners that should also be considered. The Public Health Agency is a member of the International Circumpolar Surveillance Tuberculosis Working Group, a part of the Arctic Network for the Surveillance of Infectious Diseases. Involvement of Public Health Agency staff include exchanges of data (including sharing epidemiological information as well as ideas regarding prevention and control strategies) that could be used to address Canadian outbreaks.

In summary, finding an appropriate level of engagement is an ongoing challenge; however, according to the program's logic model and various corporate documentation, collaboration is critical to the Centre for Communicable Diseases and Infection Control's activities to prevent and control community associated infections. Collaboration appears to be working well with the provinces and territories, as well as some key internal partners, especially during an outbreak situation. However, collaboration appears to be declining with other partners. The impact of this decline on the Public Health Agency's ability to achieve community associated infections goals is difficult to determine at this stage, although it is apparent that partners look to the Public Health Agency to provide public health advice for their own respective mandates and populations.

3.2.8 The Public Health Agency encourages partnerships through hepatitis C grant and contribution funding with reported beneficial effects

The prevention and control of hepatitis C in Canada requires a multi-disciplinary approach that engages stakeholders across a variety of disciplines. Through the hepatitis C contribution funding, the Public Health Agency supported stakeholder collaboration in addressing community associated infections, as the establishment of partnerships was a requirement of the funding. Over the course of three years (2007-08 to 2011-12)^{Footnote 52}, the Public Health Agency funded 42 projects which, in turn, supported almost 800 partnerships across Canada. The majority of funding recipients come from the not-for-profit sector, working across a number of disciplines including health associations, universities, outreach centres, and Aboriginal organizations. As can be seen in Figure 19, the majority of partners (more than 450) came from the not-for-profit sector, while almost a third were from the public sector. A small percentage came from the private sector. Furthermore, the complex nature of the prevention and control activities funded through the hepatitis C projects was evidenced by the diversity of partners engaged in the process. As Figure 20 displays, partners came from a variety of disciplines, including health, education, social services, Aboriginal organizations, justice and housing.

Figure 19: Hepatitis C project partnerships by sector

Source: Public Health Agency of Canada - Project Evaluation and Reporting Tool Analysis

Figure 20: Hepatitis C project partnerships by discipline

Source: Public Health Agency of Canada - Project Evaluation and Reporting Tool Analysis

An analysis of performance measurement data by the Evaluation Directorate collected from the hepatitis C projects^{Footnote 53} reveals that the partnerships were fruitful in helping funding recipients achieve their project objectives more efficiently (i.e. using fewer resources) and more effectively (i.e. having greater impact). Along the lines of efficiency, projects reported project implementation support and coordination, increased access to target group, and organizational capacity building. Along the lines of effectiveness, projects reported more effective reach to the target group, greater access to health services, and increased knowledge gain of the target group. These are described below.

Partnership Benefit - Efficiency

Project implementation support and coordination

The vast majority of funding recipients reported receiving support from their partners in implementing their projects. Support varied in its form and intensity, but included things such as providing staff to facilitate project workshops, providing advice to the project on the steering committee, providing planning and administrative support and assisting with specific language, cultural or technical expertise. Partners also assisted in helping to coordinate project activities with complementary services and programs being offered in the community. For example, projects partners would provide referrals to the project from their own client-base, assist in developing coordinated community or provide-wide strategies, and develop common messaging for a shared clientele.

Access to the target population

As a result of the partnerships, funding recipients reported having greater access to a specific, and sometimes hard to reach, target group. Project partners served at times as a bridge to the target group, providing access through established programs, services and relationships. In addition, the coordination in service delivery described above encouraged higher numbers of people to access activities funded through the projects, as funding recipients frequently cited client referrals as an immediate benefit of the partnership.

Organizational capacity building

As a result of the partnerships, funding recipients reported direct benefits in terms of organizational growth and development. The most concrete example of this benefit stems from the increase in resources that funding recipients gained through the partnerships, both financial and human. Funding recipients cited partnership benefits such as access to funding, in-kind staff, volunteers, program materials, and meeting space. Partnerships also resulted in organizational gains in subject-matter knowledge and skills, as partners shared their knowledge and expertise. Finally, as a result of the partnerships, funding recipients established linkages throughout their community, which enabled them to be more strategic about leveraging and coordinating their efforts with the broader community.

Partnership Benefit - Effectiveness

More effective reach to the target group

In addition to reaching higher numbers of people as described above, partnerships also facilitated more effective reach, in that the strategies utilized were more accessible to the target group in terms of comfort, safety and/or cultural relevance.

Greater access to health services

As a result of the coordination of efforts described above, funding recipients reported direct project impacts in terms of increased access to health services. Project partners coordinated their services and referred clients between programs to enable more comprehensive and seamless program delivery in the community. As a result, there was greater access for the target group to health services such as needle exchange programs, primary care treatment and prevention services for addictions, and shelters, to name but a few.

Increased knowledge gain

Partners were instrumental in increasing the breadth and depth of funding recipients' dissemination activities, and also in developing more effective knowledge products that integrated partner knowledge and expertise. As a result of partnerships, funding recipients reported more effective gains in their knowledge and awareness objectives.

3.2.9 Although some performance data are collected, there is limited performance measurement information available for programmatic decision making

There is limited performance measurement in place to help the Centre for Communicable Diseases and Infection Control monitor progress and assist in programmatic decision making.

Centre for Communicable Diseases and Infection Control activities

In 2011, the Centre changed their organizational structure to help transition to a more integrated approach to the prevention and control of certain communicable diseases. To help aid this process, a Centre-wide logic model was developed and presented at an all-staff meeting (with opportunities for additional questions built into the meeting as well as follow-up meetings with managers).

While the logic model basically identified desired outcomes for target audiences, there was no accompanying narrative to help describe fully the activities, outputs and desired outcomes for the Centre. Narratives could increase understanding of the overall logic of the program or elements within it.

When discussed in June, 2012 with a group of working level program staff and within many working level key informant interviews, it was apparent that the Centre's logic model was not fully understood. Outcomes were not fully recognized by many program staff interviewed for this evaluation. For example, it was difficult to determine a common understanding of community capacity, a short term outcome of the program. A narrative would have defined this outcome and probably provided some examples to help explain it.

In addition, the Centre for Communicable Diseases and Infection Control does not have one overarching performance measurement strategy to monitor the performance of its community associated infections activities.

The Centre does track the production, dissemination and use of some of its outputs (the direct products of its work, such as reports, fact sheets, training modules, etc.), however, it is difficult to determine why some are tracked and others are not as there is no link to an overall performance measurement strategy.

As such, the Centre has evidence of activities/outputs, such as the Canadian Guidelines on Sexually Transmitted Infections, in which it is has achieved its desired results, but this is not consistently employed across key activities or products.

Grants and contributions activities

With respect to Public Health Agency grants and contributions activities, outputs and outcomes were generally tracked for project work funded through the hepatitis C grants and contribution program. However, despite the national roll-out of this tool over three full fiscal years, the data was never rolled up into a national report.

An analysis of performance measurement data collected from the hepatitis C projects has been conducted by the evaluation team. It demonstrates that projects generally had some success in achieving their knowledge and awareness objectives. It reveals that the partnerships were fruitful in helping funding recipients achieve their project objectives more efficiently (i.e. using fewer resources) and more effectively (i.e. having greater impact).

However, there is no evidence that the performance information was used to inform program or policy decisions regarding the overall hepatitis C program. As such, while the information was collected and available, it was never used to measure, assess and adjust performance for the program as a whole. As there are questions regarding effectiveness in certain areas - increased knowledge may have reached a threshold, reported behaviour change appears to have decreased - performance reviews are needed to adjust programs appropriately.

There are promising activities currently in the works which hint at the development of a more comprehensive performance measurement culture within the Centre for Communicable Diseases and Infection Control. The Centre is beginning to measure impact in the area of knowledge products and demonstrate the Centre's progress towards a more comprehensive performance measurement strategy.

Some products have been assessed and results hint at what could be the Centre's impact on knowledge gain among its stakeholders. During the five-year timeframe of this evaluation study, there were several outcome reports available to inform the analysis of the Centre's three intended outcomes: collaboration, knowledge gain, and use of evidence. This demonstrates that the Centre is beginning to develop tools and methodologies to measure and assess performance. In addition, in 2011 the Centre began developing a performance measurement framework to respond to its obligation to report on the outcome of "use of evidence", as per the Public Health Agency's corporate reporting requirements for the Departmental Performance Report.

3.2.10 Leveraging of volunteer hours led to efficiencies for hepatitis C funding

As outlined in section 2.4, the Public Health Agency received both ongoing and time-limited funding for its community associated infection activities. Financial information was tracked in different ways for different years due to organizational and program activity architecture changes resulting in difficulty to obtain consistent financial information. A lack of consistently tracked financial information for expenditures, especially by disease area, makes it difficult to completely assess the efficient use of overall funding for community associated infections activities.

Hepatitis C Program funding

As part of the government's response to the report of the Commission of Inquiry on the Blood System in Canada (Krever Commission), funding was received to support persons infected with, and affected by, hepatitis C to provide a stronger evidence base for policy and programming decisions and to strengthen partners' capacity to address hepatitis C in Canada.

The Public Health Agency received funding to address hepatitis C for the following activities:

• develop and implement sustainable prevention efforts for vulnerable populations
• raise awareness of general hepatitis C facts, testing, issues of co-infection and risk factors
• build research, surveillance capacity and support for community based capacity.

Although the original allocated investment for hepatitis C activities has been reduced somewhat over the last five years, this was primarily due to reductions in administration, travel and professional services and realignment of program efforts to address common risk factors in key populations. The Public Health Agency allocated over $7 million directly for hepatitis C activities (see Appendix F for detailed description of financial allocations and expenditures for this program). Grants and contributions spending was also less than allocated in the first two years of the program, although some key informants noted that funds were received late, delaying the launch of the program.

Financial data not tracked for analysis at the disease or activity level

As mentioned in section 2.4, the Centre for Communicable Diseases and Infection Control is responsible for a number of activities for programs that fall outside the scope of this evaluation such HIV/AIDS work and for activities that support more than one disease, such as surveillance, associated infections prevention and control, modeling and projection, development of professional guidelines and research coordination. This integrated approach has limited the ability to assess whether funding is efficient and appropriate to achieve the intended outputs and outcomes of each activity, and aligned with the Treasury Board allocations.

Economy and efficiency gains at the regional and national level; optimization of resource utilization through leveraging

In 2009-10, the British Columbian region of the Public Health Agency of Canada, provided $2 million in funding to 19 AIDS Community Action Program projects and just under $400 thousand in funding to four hepatitis C projects. In 2011, the regional office of the Public Health Agency conducted a study of the performance measurement information gathered through the Performance Evaluation and Reporting Tool.

The analysis of the data revealed that leveraging of funding for the 2009-10 AIDS Community Action program and the hepatitis C program provided $690,345 in additional funding and in-kind support and volunteers with notable results in collaboration, training, awareness and outreach activities with target populations. The proportion of hepatitis C leveraging dollars appears to be lower than dollars leveraged by the Aids Community Action Program. No further information is available on why this may be the case, although many factors could be involved. Efficiency gains were realized in other critical areas as well, including increased access to vulnerable groups affected by hepatitis C. These are outlined in Appendix G.

Leveraging of volunteer hours led to efficiencies for hepatitis C funding realized over the period 2009-10 through to 2011-12. To gauge efficiencies more broadly, the same logic employed in the British Columbian report was applied to all hepatitis C projects funded by Public Health Agency over the past three years. By applying the economic baseline reported for 2009-10 ($23.26/volunteer/hour), the study revealed that the hepatitis C projects leveraged $368,611 through in-kind contributions and $944,540 through volunteering and loan staff participation. This is a very conservative cost evaluation given that the participation of loaned staff would have an hourly value far exceeding the baseline used; based on Project Evaluation and Reporting Tool data, 47 per cent of loaned staff were health professionals and practitioners (program directors, managers and nurses) and 24 per cent were social workers, outreach and community workers.