ARCHIVED - Findings

The intent of this evaluation is to assess the successes and limitations experienced to date in the Surveillance Functional Component’s current level of implementation, describe progress in improving engagement and integration, and understand the current perceptions on achievements made towards outcomes. The findings are based on both document review and key informant interviews in response to the three evaluation questions guiding this formative evaluation report.

3.1 Relevance of the Surveillance Functional Component

Evaluation question: Does the Surveillance Functional Component continue to be consistent with departmental and government-wide priorities, and does it realistically address an actual need?

In describing the need for PHAC to lead on national public health surveillance, internal PHAC respondents referenced various authorities, such as PHAC’s Report on Plans and Priorities (RPP) (Minister of Health, 2007-2008), the PHAC Act, the role of the Chief Public Health Officer (CPHO), and emphasized the direct linkages with program decision-making and demonstrating value for Canadians. External respondents also recognized the critical role PHAC plays in national public health surveillance.

“You cannot manage what you cannot measure and…surveillance is the basis in which we benchmark and measure how we’re doing” (PHAC Policy Leader).

“The provinces and territories look to the federal government for this leadership and skill base” (PHAC Manager).

“This is one of the few initiatives I’ve worked with at PHAC that I believe truly was guided by the concept of enhancing public health capacity... It’s very respectful, very clear on roles, and driven by what I’d call a true altruistic purpose that is consistent with PHAC’s purpose and existence” (External Policy Leader)

Key informants identified that P/Ts have been very appreciative of PHAC’s support in enhancing the surveillance function.

“(The) Public Health Agency has certainly been very supportive over the last number of years in helping build capacity at the provincial level, and not just in our province, but other provinces and territories” (External Policy Leader).

Through the document review, an additional demonstration of relevance was identified with regards to the action areas of the Surveillance Functional Component; these directly reflect three of the four strategies1 of the Enhancing Capacity for Surveillance of Chronic Disease Risk Factors and Determinants report released in June 2005 (Advisory Committee on Population Health and Health Security: Surveillance Systems for Chronic Disease Risk Factors Task Group, 2005). Surveillance is also an organizational priority, and is identified in the 2007-2012 PHAC Strategic Plan, ensuring the continued development of surveillance capacity.

Alignment with Organizational Priorities

To examine the alignment between the surveillance actions of the Surveillance Component Logic Model with current priorities and directions of jurisdictions and sectors, respondents were asked to rate this alignment and provide a rationale for their rating.

A majority of respondents (84%) indicated the Surveillance Component of the ISHLCD aligned “very well” with the priorities of PHAC. This was expected, given the strategic focus on surveillance within PHAC. Respondents indicated one of the main mandates/objectives of PHAC is to support surveillance in priority areas. Of those who responded, the majority of respondents identified that the Surveillance Component “somewhat” aligned with the priorities of P/T governments (55%) and NGOs (75%) (see Table 3).

Collectively, respondents indicated P/T governments appreciate the need for a national perspective on chronic disease in Canada for comparison purposes. Respondents emphasized leveraging existing mechanisms and resources to conduct surveillance, rather than building new databases or registries. On the other hand, in some cases, P/Ts operate independently to address their own surveillance needs, given their focus on the health care system, which falls outside of PHAC’s jurisdiction.

¹ The fourth strategy, acted on outside the ISHLCD, is to “build capacity across jurisdictions for congruent public health legislation supportive of chronic disease surveillance”

Note: Total respondents for each area depicted in Table 3 equals n = 25.  Responses provided as “Don’t Know” are not captured as a valid response in tabulating overall percentages. Responses may not equal 100% due to rounding.

While coordination with P/Ts in the area of surveillance remains a challenge, particularly in the area of data consistency, engagement with the federal government is welcomed as providers of capacity (human and financial) in mutually beneficial relationships.

From a provincial perspective: “I think in our case anyways, it aligns very well with what we're doing. We have the same kind of aims and requirements, and so I think it aligns very well with our provincial perspective on surveillance” (External Policy Leader).

From a federal perspective: “Having them (P/Ts) on as advisors in our work on planning and surveillance pretty much ensures that we're working with them to give them the information they need. They're working with us to give us the information we need. It's a pretty symbiotic relationship” (PHAC Manager).

For NGOs, surveillance activities generally support their advocacy work. While the relationship with the Canadian Cancer Society is particularly strong, respondents also identified increased collaboration between PHAC and the Lung Association as well as the Heart and Stroke Foundation.  While PHAC has made progress in aligning major chronic disease areas with its activities, collaboration on surveillance with other disease areas (e.g., Parkinson’s disease and Fibromyalgia) are in different phases of development depending on resources and priorities.

Generally, respondents felt there was alignment with the health portfolio within the federal government, in particular with Health Canada and CIHR. Findings from the document review also illustrate that action areas identified in the 2007-2008 PHAC Report on Plans and Priorities were signed off by the Minister of Health (Minister of Health, 2007-2008, p. 32).

Respondents commented that PHAC has a key role to play in surveillance, and PHAC would appear to be well established and on “solid ground” (PHAC Policy Leader) in enhancing national chronic disease surveillance capacity. Given the evolving policy context, with new single disease area strategies emerging or in development, PHAC will need to continue to build national coherence on an ongoing basis in collaboration with P/T partners.

In terms of an illustration of work underway to ensure alignment between different organizations, the document review identified the NDCSS governance model. This work seeks to align and integrate the needs of multiple parties in surveillance, across advisory bodies from disease-specific areas, scientific and technical perspectives, and chronic disease decision-making bodies.

3.2 Design & Delivery of the Surveillance Functional Component

This section explores respondent observations on design and delivery of the Surveillance Functional Component, in particular the application of resources, progress on activities and outputs, and issues encountered to date with implementation.

Evaluation question: Are the most appropriate means being used to achieve objectives?

The document review identified that PHAC monitors program delivery within the Surveillance Functional Component through the Surveillance Working Group. The purpose of this group is to ensure collaboration and integration between surveillance and other Functional Components. The group provides regular updates through the co-leads of the Regional Coordinator, the Centre for Health Promotion Coordinator, and the CCDPC Coordinator to the Integrated Strategy Coordinating Committee.

Perceptions of Component Resourcing

Respondents described the delay in the roll-out of funding, indicating “it’s still ramping up” (PHAC Manager). There is a delay of about a year, and there was optimism expressed that the Strategy will likely “catch up” within the next fiscal year or two. Both internal and external respondents noted the general challenge of recruiting qualified candidates for work in surveillance.

“Using this integrated approach now for a number of years seems to have provide a lot of dividends… We’re maximizing resources…It’s not (a) lack of money or positions. It’s getting warm bodies to fill the spots” (PHAC Manager).

Design of Surveillance Action Areas

Almost all respondents indicated that there were no identifiable gaps in the design, delivery, or resourcing of the Surveillance Functional Component. In response to the action areas of the Surveillance Logic Model, respondents reported these areas helped “keep us on track” and current action areas have adequately reflected the surveillance activities of the division.

“I think (the Logic Model is) quite thorough and keeps us on track…(It helps us to) pay…attention to each of these components, instead of just focusing on trying to get new data or whatnot” (PHAC Manager).

“The new things that we want to do still fit within analyzing the existing data and making better use of it, getting that out to people, expanding our data sources and doing a better job of coordination, which includes data quality, planning and evaluation” (PHAC Policy Leader).

While providing a useful and adequate picture of surveillance activities, respondents also noted there could be an additional emphasis on dissemination. Dissemination of chronic disease information was identified as an integral element that coincides with the action areas of using and expanding surveillance data, and may require a greater emphasis in the Logic Model. Within the ISHLCD, a separate Knowledge, Development, Exchange & Dissemination (KDED) Functional Component has been designated to relate directly to the Surveillance Functional Component, as indicated in the ISHLCD Logic Model. However, in practice, functional linkages have not been established.

“My vision of the future is us dedicating a whole lot more resources to surveillance and dissemination of information, particularly via the Internet…I really see us needing to…double and triple our efforts to go in that direction as an organization” (PHAC Manager).

One external respondent also identified the issues of training of human resources and future planning in surveillance. While the present evaluation did not include identification of P/T programs for recruiting public health graduates, the ES Development Program (ESDP) helps recruit recent graduates for positions within PHAC. PHAC also leads the Canadian Public Health Service, which aims to place public health staff in various areas of need across Canada. There may be the need to identify P/T and other federal initiatives in the area of training public health surveillance professionals.

“If we’re thinking about the national capacity or surveillance capacity nationally…Who is coming up behind us?...What’s the planning and priorization process for what comes next in the surveillance system?” (External Policy Leader)

In addition, respondents noted concerns that PHAC Divisions were operating according to the Functional Components for which they were responsible, and that integration in terms of matrix-oriented business practice has yet to be fully realized.  Some respondents suggested the issue reflects the hierarchical nature of reporting, and the fact that integrated functional operations do not coincide with funding profiles, which occur by program area.

Progress on the Achievement of Outputs

The document review identified that outputs described in the Surveillance Logic Model are reported at annual Coordinator Committee Meetings, with high level summaries reported in the ISHLCD Implementation Reviews (Performance Management Network Inc., 2007; Public Health Agency of Canada, 2007d; Public Health Agency of Canada, 2008c; Public Health Agency of Canada, 2008d). The current status of the generation of outputs for each action area of the Surveillance Logic Model is reported below.

Analysis, Interpretation and Use of Existing Data Sources

In support of enhancement of the analysis, interpretation and use of existing data sources for surveillance, a resource unit within the Surveillance Division was established to carry out activities under this action area, resulting in the following outputs.

Internet-based geographic mapping of chronic diseases and their risk factors: Web server statistics² for the Chronic Disease Infobase Mapping Application for January through September 2008 have seen a 149% increase in regional profiles and data downloads compared to 2007. This may be due to an increased emphasis in promoting this tool with regional stakeholders.

Web-based chronic disease data dissemination tool and inventories/library of databases, metadata and resources: The pilot phase for the Chronic Disease Data-Cube project was completed in June 2008 with the generation of a Project Success Plan by Cognos. Currently available internally, the Chronic Disease Data-Cube project is in its production phase and may be launched publicly for use by health professionals, researchers, policy analysts and the public at some point during calendar year 2009.

Timely responses to requests: The number of requests for surveillance information is tracked through an Information Requests Tracking System. The quarterly summary for October to December 2008 identified 26 requests, requiring 36 hours of preparation. This tracking system presents only a partial picture of requests and staff time required to complete these requests, with half of the analysts within the Surveillance Division participating in the tracking of requests. Current plans to enhance use of the tracking system are being implemented and may result in more analysts tracking requests in 2009. The tracking system currently does not assess timeliness of these responses, although it does provide information identifying the broad spectrum of stakeholders requesting surveillance information.

Reports and fact sheets published: The Surveillance Division generates comprehensive chronic disease-specific epidemiologic reports, containing primarily descriptive statistics, as well as fact sheets, Web pages and government executive correspondence. In general, three major reports are produced annually on a rotational basis through the main chronic disease areas. Major reports have included Life and Breath: Respiratory Disease in Canada (Public Health Agency of Canada, 2007b), Canadian Cancer Statistics (Canadian Cancer Society et al., 2008) and The Human Face of Mental Health and Mental Illness in Canada (Public Health Agency of Canada, 2006). Current plans include the generation of reports in the areas of CVD, arthritis, diabetes, and a comprehensive report on chronic diseases in Canada. While evaluation data on dissemination and an exploration of indicators of use has been generated for The Human Face of Mental Health and Mental Illness in Canada report, this would appear to be ad hoc rather than a widely implemented practice for monitoring dissemination of surveillance reports.

E-learning programs, workshops, and conferences: While there is no systematic tracking of training sessions, workshops and conferences, internal respondents identified that there has been a training session on program evaluation for the Surveillance Division. The Surveillance Division has conducted training sessions for other divisions within CCDPC on data-cubes, and has made presentations on the Chronic Disease InfoBase at major chronic disease conferences, over the last several fiscal years.

² As tracking is related to the structure of the web page, substantial changes in the application may interrupt the tracking of trends over time.

Expansion of Data Sources

The expansion of data sources for surveillance includes projects addressing P/T databases, surveys and sentinel surveillance systems. These areas are currently being addressed through the NDCSS, supplements to the CCHS and through grants and contributions.

NDCSS: The growth and refinement of the NDCSS has included the development of a governance structure and a four-year plan for the inclusion of specific disease areas. The governance structure channels the contributions of disease-specific advisory committees, the science working group, the technical working group and the Public Health Network through the Collaboration and Development Section and the Data Management and Analysis Section of the Chronic Disease Surveillance Division. Hypertension and mental illness are being added in 2008-2009.  Work to develop case definitions for the other chronic diseases is ongoing, and will be added in upcoming years.

Supplements to the CCHS data: Recent developments with the CCHS have facilitated surveillance work. In August 2008, an agreement was signed with Statistics Canada ensuring the information collected through the CCHS can be shared with the Minister of Health. In September 2008, a long-term proposal for CCHS for 2007-2012 identified injury as a content theme in 2009 and mental well-being as a potential focus in 2011. The specific supplements to CCHS for 2008-2009 include survey data for hypertension, arthritis, sleep apnea and osteoporosis. The complementary nature of CCHS information as a surveillance data stream is described in the quote below.

“We still need to bring in information from health surveys, like the Canadian Community Health Survey, because that gives us right now more information about risk factors and socioeconomics. Whereas the strength of the NDSS (National Diabetes Surveillance System) infrastructure is we're going to get a better measure of incidence and prevalence, and also able to do some measuring of sort of complications relating to diabetes, and then also health services utilization” (PHAC Manager).

Gs&Cs: The Chronic Disease Surveillance Gs&Cs Program has the same main objectives as the Surveillance Component, and provides funding to conduct general surveillance on chronic disease at the national level, as well as Aboriginal surveillance.  Calls for proposals have targeted projects designed to build capacity for public health chronic disease surveillance at the national, P/T and regional/local level.  This includes supporting priorities in the development of regional risk factor surveillance, developing tools and methods for public health surveillance, and developing capacity for public health surveillance (Public Health Agency of Canada, 2008b).  Funded Gs&Cs have been awarded to relevant NGOs, universities, P/T governments, and local/regional public health organizations and networks. A total of three solicitations have occurred between 2006 and 2008.

In total, 21 Grants have been awarded to eligible recipients in eight provinces and territories.  At this time, only five Contributions have been awarded across three provinces, with one Contribution identified as a national project.  Table 4 provides a breakdown of the proportion of Gs&Cs funded by target recipients.  The national Contribution provides funding to the College of Family Physicians of Canada, which in turn, is working with seven sites across the country to conduct surveillance on risk factors and chronic disease in family physician sites.  This work has leveraged a significant amount of in-kind support, and allows access to data from clinical sites coordinated through the College (see Case Study in Box 2 below).

Many projects are currently at the start of their implementation process, as a result of delays in approvals of projects for fiscal years 2007/2008 and 2008/2009.  Because of the short time frame in 2006/2007 (due to a delay in receiving funding approval for the program), the only option available for funding was via one-year grants. The Internal Review of the PHAC Gs&Cs program limited the ability to do multi-year project funding for eligible Contribution projects.  As the Second Implementation Report found, this “limits the ability of the Strategy to provide multi-year Gs&Cs funding, which hampers the communities’ ability to deliver on their projects. All this, in turn, compromises the ability to effectively attain the Strategy outcomes” (Public Health Agency of Canada, 2008d, p. 287). As such, Contributions were awarded to eligible recipients towards the start of fiscal year 2008/2009, leading to delays in implementing the activities and objectives of these projects.

In terms of other gaps, respondents cited delivery gaps due to the delay in the development and delivery of Gs&Cs with an Aboriginal component.

Internal and external key informants included in this evaluation noted the program has made progress in refining the solicitation process.  Potential recipients are kept apprised of their status through formal and informal communication.  The review process has also been streamlined.  A PHAC Standard Operating Procedures Database has been created to provide guidance to potential Gs&Cs on the mandatory requirements detailed within the request for proposals.  Proposals are reviewed for eligibility, and then by chronic disease subject matter experts.  A Reviewer Comments Matrix has been designed to aid management in organizing the input from various stages, and to facilitate timely approval of projects.  A dedicated Web page has been developed to provide information and updates to potential applicants, and this Web page will also be used to disseminate final project reports. Final reports for Grants funded under this program are expected in March 2009.

Respondents noted two areas where funding via Gs&Cs to enhance or expand surveillance data for chronic disease has addressed significant gaps:

• Consultations on autism were held with relevant external stakeholders.  This work aimed to determine the best approach for conducting national surveillance on autism. One respondent familiar with this work identified that this was well received by those involved in the process.
• Contribution funding was provided to three Métis organizations in three provinces to allow for a linking of Métis registry data.  It was felt this will increase the availability of incidence and prevalence data for diabetes and other chronic diseases for Métis people, and allow for future linkages to other databases (e.g., provincial cancer or NDCSS databases).

“And so I think we're really making progress on getting information on the Métis in Canada. By taking advantage of our other investments in the NDCSS, so we've invested $2.8 million in that with the provinces and territories. Now by putting a little money in with the Métis, they can then take their registry and hook into that other investment we have and get information on Métis health problems.  So I think that's a really nice sort of coming together of different projects” (PHAC Manager).

 

Coordination, Planning and Evaluation

The third action area aims to enhance coordination, planning and evaluation by focusing on liaison activities with other surveillance initiatives, including supporting surveillance advisory committees and formal F/P/T collaboration through the Pan-Canadian Public Health Network (PHN).

Indicator framework: The Chronic Disease Indicator Framework currently contains approximately 300 indicators in four of six chronic disease areas drawn from a broad range of data sources including population, risk factor, morbidity, mortality, prevention and drug data sources. Indicators are grouped according to five domains: Individual Risk Factors (88 indicators), Environmental (31 indicators), and Health Status (73 indicators) (these three domains represent the primary surveillance targets), as well as Health Promotion (38 indicators) and Management (96 indicators). The Management indicators focus on self-management, and whether individuals with chronic conditions have had follow-up according to clinical practice guidelines. The surveillance system tracks people, rather than health services, and is not involved in the monitoring of quality of care. The role of indicators will continue to be emphasized, as this reflects requirements for population health information and informs surveillance data requirements.

Committees and Reports: While committees have been created to support ISHLCD program authorities and coordinators, a range of advisory committees involving broader groups of stakeholders have also been created to develop the surveillance information requirements for diverse areas of chronic disease surveillance. Within the PHN, the Task Group on Surveillance of Chronic Disease and Injury, which reports to the Chronic Disease and Injury Prevention and Control Expert Group (CDIPCEG), is a place for all the provinces and territories, Statistics Canada, CIHI, and CIHR to plan collaborative projects.  Of particular importance is the report Strengthening Surveillance of Chronic Disease and Injury: An Action Plan released by the Task Group on Surveillance of Chronic Disease and Injury in January 2008 (Task Group on Surveillance of Chronic Disease and Injury, 2008). This action plan established priorities based on the June 2005 report Enhancing Capacity for Surveillance of Chronic Disease Risk Factors and Determinants approved by Deputy Ministers in June 2005 (Advisory Committee on Population Health and Health Security: Surveillance Systems for Chronic Disease Risk Factors Task Group, 2005). Priorities from the action plan reflect the action areas of the Logic Model and emphasize collaborative surveillance.

Evaluation reports:  This is the first evaluation of this new surveillance program. PHAC has been laying the foundation for the effective management of the public health evaluation function through the completion of a PHAC Surveillance Strategic Plan that includes evaluation of one of the action areas. A PHAC-wide approach to surveillance is being developed and this program will participate in the development and implementation of the evaluation plan. This will include the creation of a surveillance conceptual framework and evaluation process and tools.

Lessons Learned through Implementation

A number of lessons learned from the implementation of the Surveillance Functional Component of the ISHLCD were identified:

• Benefits of Chronic Disease Surveillance: People living with chronic disease, as well as people advocating on behalf of those affected by chronic disease (such as NGOs and public citizen advocates for autism spectrum and neurological disorders), are recognizing the need for surveillance data. These stakeholders have a role to play in identifying relevant surveillance information.  That said, with an increase in expectations from more and more stakeholders, one challenge PHAC may face is meeting these many stakeholder expectations in a timely manner with the resources available.

“The only downside is when you go wide, you go thin… The biggest criticism we probably have right now is that we're maybe thin on the knowledge side of certain diseases. We don't have disease experts in-house that we probably could benefit from having” (PHAC Manager).

• Partnerships: Involving partners in the development of the concept of an integrated approach to surveillance was identified as important for future communications with regards to national surveillance governance with partners. Setting clear expectations on goals and what can be achieved, and communicating this with stakeholders, was identified as critical; expectations pertaining to the amount of time required to obtain and analyze surveillance information needs to be better communicated.
• Communicating progress: Some key informants, both internal and external, noted that progress made, as well as funding processes and amounts, have not been properly communicated between PHAC and external stakeholders.  This is key to establishing and maintaining relationships with partners.

“I think communication is really essential. It hasn't always, as I said, been clear what exactly is being worked on, what exactly are the dollars behind it and how…people who are receiving those dollars, how did they actually receive them” (External Policy Leader).

• Corporate Support: Respondents recognized that certain aspects associated with human resources (HR) and Information Management and Information Technology (IM/IT) are critical in supporting the initiative, such as staffing and being able to implement innovative technologies. Infrastructure to support the Surveillance Component may have lagged behind the strategic direction and vision for an integrated approach to chronic disease surveillance.
• Performance Management: A good program performance story contains elements of a well-defined program, a strong performance management framework, the collection and use of performance data, an emphasis on learning and improvement, as well as periodic evaluations, studies and audits (Samaroo, 2007). The document review showed, since 1999, the Auditor General of Canada has observed the need to improve performance measurement of surveillance systems, and made a specific recommendation for this in 2002 (Office of the Auditor General, 2002). Currently, the work of the Division and PHAC’s Strategic Plan includes developing surveillance indicators to support the performance measurement of surveillance systems.

“Recommendation: (PHAC)… should strengthen its evaluation, performance measurement, and reporting of results of its health surveillance activities” (Office of the Auditor General of Canada, 2002).

Respondents identified performance measurement, in the form of documenting the processes involved in coordinating an integrated approach to chronic disease (including the systems and committee pieces), as essential to PHAC’s role in surveillance. However, there is no performance management system in place.

• Appropriate Leadership and Resourcing: Respondents indicated there was strong leadership in moving resources dedicated towards surveillance in a positive direction, and identified the direction PHAC was taking in terms of chronic disease surveillance was indeed the right approach. PHAC’s role in facilitating partners to work together was viewed as very important. The Surveillance Component has functioned well as a result of strong leadership and dedication from a number of committed individuals.

“I think a lesson learned is that if you have good leadership and you have adequate resources in terms of dollars and people, you can accomplish a lot... if you don't have those things in place, then no matter what your plans are you can't realize them, because you just don't have the resource(s) to do anything” (External Policy Leader).

In summary, the design of the Surveillance Functional Component, as shaped by the action areas of the Logic Model, has been validated by respondents, and reflects previous F/P/T strategic directions. While delays in implementation have been encountered, primarily due to external causes and process issues related to office space and staffing, PHAC will likely “catch up” in the next couple of years. The means for generating and monitoring the outputs specified in the Logic Model are in place, and likely to be significantly advanced during the current calendar year with the release of reports and the public launch of the data-cubes, supporting broader access to chronic disease surveillance information. Management of partnerships has been positive, and will continue to shape the success of surveillance activities.

3.3 Success with Engagement & Surveillance Outcomes

Evaluation Question: Is the Surveillance Functional Component effective in generating outputs, given its resources, and without unintended consequences?

PHAC identified a requirement to examine the extent and nature of both internal and external engagement as part of the Second Implementation Review.  Initial work was undertaken to identify an approach to define and measure engagement for the ISHLCD.  This work included a review of the literature, which noted previous efforts focusing on the measurement of engagement have not addressed engagement between a government organization and a wide variety of external stakeholders.  Generally, the review noted the concept of engagement can mean the cooperative way two or more entities work together toward a shared goal (Frey et al., 2006).  Furthermore, collaboration – in this case, both internally with regards to integrating chronic disease surveillance, and externally with stakeholders – is a “primary method for achieving ideal short and/or long-term goals that would not otherwise be attainable as entities working independently” (Gajda, 2004).

“We need to find a way of explaining to the Minister and staff and stakeholders what integration means. We have an integrated surveillance program so that we don't have to duplicate the same thing five times, one for each disease, right?” (PHAC Manager).

Respondents noted integration continues to present a challenge.  Despite some success in improving coordination for the diabetes program, consistent themes emerged internally that operations continue to be “siloed” or “stove-piped”, as opposed to being integrated.  As part of the implementation process, attempts have been made to transition the organization to function as a matrix.  However, the organization continues to operate largely along hierarchical program lines in relation to staffing and funding.

“There isn't good coordination...it's this matrix model mapped against a hierarchical model of how…the directors actually interact” (PHAC Policy Leader).

Consistent with the findings of the First and Second Implementation Reviews, internal respondents identified organizational design issues of coordination and a lack of a systems perspective as barriers in advancing integration. This may have an impact on progress to improve inter-organizational engagement, to a greater degree than inter- and intra-sectoral and multi-jurisdictional engagement. Championing by senior management was identified as a potential source for improved integration across the Functional Components of the ISHLCD.

“I think there's a real lack of systems understanding” (PHAC Manager).

Progress on Engagement

In assessing the immediate outcomes of enhanced engagement, quantitative and qualitative descriptions were collected in response to key factors of engagement and the intermediate outcomes specified in the Logic Model. The outcome of engagement is linked to the functioning of integration within the Operational Matrix of the ISHLCD, so a brief assessment of integration of surveillance was explored with respondents.

 

When respondents were asked about their overall rating of engagement with partners across sectors and jurisdictions, all respondents indicated the Strategy has been at least “somewhat” effective in increasing the engagement of organizations, sectors and jurisdictions to this point, with over a third (36%) feeling efforts had been “very effective”.  Respondents pointed to the slow start in implementing early stages of the Strategy.

Across the five attributes of engagement used to probe success with engaging organizations sectors and jurisdictions, few respondents identified issues of ineffective engagement across the various domains. However, there was variability across domains, such as in role clarity with partners (see Table 5).

Note: Total respondents for each area depicted in Table 5 equals n = 27. Responses provided as “Don’t Know” are not captured as a valid response in tabulating overall percentages. Responses may not equal 100% due to rounding.

The following section summarizes the main points raised by respondents in relation to queries on the aspects of engagement described in the table above.  Respondents were asked to provide some context on whether or not progress has been made towards improving inter-organizational, inter- and intra-sectoral, and multi-jurisdictional engagement.

Sharing of information: Respondents noted there has been increased support at PHAC for providing surveillance information, despite early implementation issues related to internal capacity.  While improvements could be made with certain partners to enhance efforts, continued developments, involving P/T administrative data and the provision of online dissemination tools, are expected from the Component in coming years.

Sharing of resources and consistency of funding: Most respondents felt efforts had been “somewhat” to “very effective” in this area.  Some lack of clarity emerged in both intra-organizational sharing, as well as in certain chronic disease areas.  New resources provided under the Strategy have allowed PHAC to leverage either monetary or in-kind support from various stakeholders.

Clarity of roles with partners: National surveillance of chronic disease remains a clear niche for PHAC, given their position. In many ways, PHAC has taken a lead on surveillance.

“There's no one else doing national public health surveillance. Other places collect data, but they don't do the interpretation (from a public health perspective) and they don't do the (public health) reports on it and they're very clear about that. So there's no difficulty with (PHAC doing) that role” (PHAC Policy Leader).

Intra-organizational roles and responsibilities may require further clarification, in order to move forward on surveillance activities for certain chronic disease areas. Similarly, clearly defining the role of PHAC compared with other partners remains a challenge, and the majority of respondents felt the Strategy had been only “somewhat” effective in engaging stakeholders to clarify roles and responsibilities around surveillance. In particular, roles in cancer surveillance remain unclear. This lack of clarity is also an issue internally, particularly between national and regional roles.  Some respondents noted there has been a redefinition of roles for many of the partners involved in chronic disease surveillance, and when you “have transitions you can't have complete clarity” (External Policy Leader). Hence, there is an expectation that the transition in management of surveillance is evolving, and takes some time.

“I think that because it's a developing situation, I think they've (PHAC) done absolutely the best they can. And it's still evolving and roles are clarifying as we move along. So I would say they (PHAC) have been effective” (External Policy Leader).

Level of trust: The majority of respondents (68%) felt the Component had been “very effective” in increasing the level of trust among partners.  Respondents identified that the Gs&Cs announcement could have been extended to a broader audience.  However, this is improving, particularly following the re-structuring of the management processes. Including a broader range of chronic diseases (e.g., autism and mental illness) has also led to improvements in the level of trust noted by stakeholders. This trust is critical, given the increased sharing of resources and planning.

“Because it could be that you realize…you get involved in these things where you're sharing resources and things like that…there is a huge requirement of trust, and that if somebody lets down the team, we're going to be set back. It's going to be hard to get back up” (PHAC Manager).

Overall, the Surveillance Component’s progress on improving inter-organizational, inter- and intra-sectoral, and multi-jurisdictional engagement for chronic disease surveillance is perceived as being "somewhat effective".  However, there are areas where engagement has had a greater impact, particularly in the realm of increasing trust among surveillance partners and in the sharing of information.  The importance of having a dedicated body such as PHAC for collecting, coordinating and disseminating national surveillance information on chronic disease was clearly identified by internal and external stakeholders. Engagement should continue with stakeholders and experts. While roles and responsibilities have been documented internally, there may be benefit in confirming and communicating these on a regular basis.  PHAC should further refine the concept of 'engagement' and its measurement in relation to chronic disease surveillance.  This will be important for evaluating PHAC’s contribution toward attaining intermediate outcomes.

Progress on Intermediate Outcomes

The intermediate outcomes are considered to be within the direct influence of the Surveillance Functional Component. However, they are further down the results-chain of the Logic Model, and given the limited amount of time the Surveillance Component has been underway, it is expected these findings provide only early evidence of progress.

The clearest ratings of effectiveness were found for increasing the “Information on the tracking of chronic disease in Canada”, “Quality of data”, and “Knowledge about trends in chronic disease, risk factors and disease outcomes” as evident in Table 6. Domains garnering more “somewhat effective” ratings included “Access to data and metadata”, “Knowledge, skills and resources to do surveillance”, and “Capacity of data users to better interpret surveillance products.”

Note: Total respondents for each area depicted in Table 6 ranges between n = 26 - 27.  In one instance, answers were not provided for one respondent beyond “Comparability of data”. Responses provided as “Don’t Know” are not captured as a valid response in tabulating overall percentages. Responses may not equal 100% due to rounding.

*The Surveillance Functional Component has a contributing influence, not a direct influence, on this attribute.

Generally, differences were observed between internal and external respondents, with a greater proportion of internal respondents reporting "very effective" than external respondents.  This was the case particularly for data quality, knowledge to do surveillance, and knowledge in chronic disease trends.  There was convergence between internal and external respondents in perceptions related to outcomes such as the use of surveillance data to guide and evaluate decisions, as well as those on effectiveness and efficiency.  However, in rating progress on these latter two outcomes, a significant proportion of both internal and external respondents reported “not applicable” or “don’t know”.  These distinctions may be attributable to the need for improved communication and dissemination of surveillance products and information on activities and strategies; as well, they may reflect the early stage of implementation of the Surveillance Functional Component.

Respondents also provided assessments in support of their rating of early progress towards intermediate outcomes.  The main themes that emerged are summarized below.

Information on the tracking of chronic disease in Canada: The goal of expanding the surveillance system has only been partially achieved in the first two years of the Strategy; however, expansion is continuing.  One respondent noted work to date has allowed there to be “a much better understanding of what's going on in the context of these people with chronic disease” (External Policy Leader).  A number of diseases have not been incorporated at this stage.  In this sense, progress towards this longer-term outcome is difficult to assess given initial collection of data is still underway.

“I mean, what they have planned and what they're putting into place, well, eventually it'll make it very comprehensive, so I think they're on the right track” (External Policy Leader).

Data Quality, Comparability and Access: Issues of comparability were handled collaboratively, with PHAC providing a stewardship role. Also, a number of respondents indicated they “don’t know” about efforts for engagement in this domain, or that it was not applicable given work is still at the data collection and/or analysis phase.  There may be room for further communicating efforts and direction in these areas.

“It's more that I'm not aware of any efforts aimed at either of these two activities” (External Policy Leader).

Increasing capacity to do surveillance and interpret products: Respondents indicated that delays, in achieving goals in terms of adding human resources and in funding Gs&Cs, may have had an impact on the development of increased knowledge and skills.  A requirement to better develop expertise in understanding certain chronic disease areas was identified by both internal and external respondents.  In interpreting surveillance products, data limitations need to be properly understood and disseminated in order to address expectations.

Knowledge about trends in chronic disease, risk factors and disease outcomes: Some respondents (33%) indicated increased knowledge about trends in chronic disease, risk factors and disease outcomes has only been “somewhat” effective.  In many instances, new data has yet to be collected, which was reflected in the views of many that it was ‘too early’ to know. There was suggestion that work related to indicators remains in isolation, and this can impact knowledge about interactions associated with multiple risk factors.

“The picture is incomplete, so we have knowledge, but our level of knowledge is not as high as it should be just because the work hasn't been done yet” (External Policy Leader).

Based on the analysis of documentation and key informant data described above, there is a perception that progress towards these outcomes is being made; however, perceptions differ between internal personnel and external stakeholders. As the high overall ratings of effectiveness for information and knowledge gains due to surveillance appear to mostly be the influence of internal respondents, enhanced communication and dissemination of surveillance products may result in higher effectiveness ratings by external respondents in the future.

The findings for intermediate results are consistent with a program that is in its second year of full implementation. As the Strategy is still implementing activities, particularly in the realm of enhancing and expanding chronic disease surveillance through programming and dissemination tools, it is reasonable to suggest the earlier intermediate outcomes will be attained to some degree within the specified timeframe. This being said, the current evaluation findings can be regarded as early findings and should be used to direct dissemination and communication with external respondents. Concerted monitoring of outputs in relation to these intermediate and longer-term outcomes will provide a greater indication of areas of success.