ARCHIVED - Integrated Strategy on Healthy Living and Chronic Diseases - Knowledge Development and Exchange Functional Component

Methodology

2.0 Methodology / Design

This evaluation used a mixed methods design including both qualitative and quantitative methods and data to address both questions of convergence of findings as well as completeness and explanatory value of the analysis.^{Footnote 3} The evaluation is informed by four lines of evidence: KD&E project file review, internal PHAC key informant interviews, external stakeholder interviews and the Canadian Best Practices Portal evaluation. Each of these lines of evidence is described below.

1. File Review: PHAC project reports and documents were reviewed as a primary source of evidence on KD&E Functional Component activities and progress on outputs and initial outcomes and as a supplementary data source on KD&E Functional Component relevance and need.^{Footnote 4} Managers and officers working in ERAD on KD&E initiatives were consulted in the selection process of the relevant documents. Documents and project reports spanning the Fall of 2005 to the Fall of 2008, and from all of the KD&E Functional Component funded initiatives were selected for analysis. In total, 61 files were reviewed for the purposes of this formative evaluation. They included such items as PHAC funded or PHAC-led literature syntheses, risk analyses reports, journal publications, contract and grant/contribution project reports as well as key stakeholder meeting minutes or records of decisions. All these KD&E documents were reviewed and information coded using a coding manual developed in consultation with an external contractor to categorize key information from the reports in an Excel database. The coding manual facilitated a quantitative content analysis and summary of key features of the reports and documents.

2. Internal interviews:Internal interviews with PHAC staff were a key source of evidence informing questions of relevance/continued need for KD&E initiatives, examples of progress and success and in particular were central to issues of design, delivery and overall functioning.^{Footnote 5}

   In total, 10 in-depth semi-structured interviews and one group interview were conducted with PHAC personnel working within the KD&E Functional Component. These interviews were conducted by one or two PHAC evaluation analysts and took place in January and February 2009. The selected key informants were managers and/or senior analysts including a KD&E Regional Specialist, and represented all of the KD&E Functional Component funded initiatives and program areas in the Risk Assessment, Prevention and Management Divisions of CCDPC. The one group-interview was with five Managers and the Director of the Surveillance Division of CCDPC.

   On average the interviews lasted one hour. Interview notes were transcribed and coded by two PHAC evaluation analysts and analyzed both thematically and by frequency, with the interview guide and a review of a sub-set of interviews forming the basis of the coding scheme. The interview findings were synthesized in a detailed results report.

3. External Stakeholder Interviews: External interviews were conducted to provide a supplementary data set that reflected external stakeholder perspectives from those working closely with the respective KD&E Functional Component initiatives. The interviews focused on questions of relevance, success/progress to date, and design and delivery issues.

   A purposive sampling strategy^{Footnote 6} was used by PHAC to identify external key informants to interview regarding their perspectives on PHAC’s KD&E initiatives in healthy living and chronic disease. The aim of purposive sampling was to select individuals who are most likely to generate informed, productive, and in-depth discussions of observations and outcomes of KD&E initiatives to date.  In total, nine interviews lasting about one hour each were conducted from January to February 2009 by an external evaluation contractor. The selection criteria were developed in consultation with the ERAD evaluation staff and the external contractor and were applied based on input from the relevant program managers who are most familiar with their project-specific stakeholder groups.  One to two external key informants were selected representing each KD&E initiative (Canadian Best Practices Initiative, Canadian Task Force on Preventative Health Care (CTFPHC), KD&E External Advisory Committee, and disease-specific advisory groups) based on the following:

   • Being involved in the respective initiative for more than one year and
   • Currently serving as a chair or lead of the respective initiative advisory or expert reference group

   The external key informants included academics (n=3), health or public health professionals (n=3), provincial or national non-governmental organizations (n=2) and one P/T government official. The external interviews were recorded and transcribed for thematic and frequency analysis.

4. CBPI’s Portal Evaluation report:  The CBPI is the most advanced KD&E initiative. Its Portal evaluation focused on measuring achievement of three outcomes: increased awareness of the Portal; increased networking and communication with respect to best practices; and increased access to evidence-based information. In particular, excerpts of the Portal evaluation findings were used as a key evidence source for progress on immediate and intermediate outcomes. Data collection methods included routine web statistics on users and usage since 2007, a web-based survey of current or potential Portal users, and a web-based survey of Portal stakeholder organizations. For the web statistics, “Google Analytics” was adopted as the analysis tool at the beginning of November 2007 to complete monthly reports of Portal use from visitor patterns including number and type of visitors, length of visit, pages visited, etc.

   As there was no formal registration required for users to access the Portal at the time of study, a purposeful sampling strategy was used to identify current or potential Portal users by asking chronic disease prevention and health promotion stakeholder organizations which have a means of efficiently distributing information to their client base (e.g., newsletter, listserv) to post a notice about the survey to their constituency, a large portion of which are the types of practitioners that the Portal targets. A total of 239 people completed the user survey questionnaire, 46% of which identified themselves as practitioners who develop and deliver health promotion interventions. The stakeholder survey invitation was sent to a PHAC complied list of 156 organizations representing pan-Canadian coverage of pan-Canadian coverage of key organizations that support decision-makers in chronic disease prevention and health promotion and reflect current and potential CBPI partner groups, such as P/T level and national health NGOs, P/T government health departments, provincial and national chronic disease/healthy living alliances, and national and provincial public health associations. A total of 58 individuals completed the stakeholder survey (response rate=37%). The large majority of respondent organizations reported being involved in both developing capacity-building and knowledge exchange activities aimed at integrating evidence into their program/policy decision-making (86.2%) and respondents were distributed across Canada. The two web-based surveys which were conducted from May to June (user survey), and June to July 2009 (stakeholder survey) included questions about the level of awareness and use of the Portal and its potential impact, perceived needs for services and supports to help integrate evidence into program/policy-related decision-making; and respondent characteristics. The web survey data were coded and analyzed descriptively in SPSS. For further details on the Portal evaluation methods and results, please see the full report.Footnote 7

2.1 Methodological Considerations and Limitations

1. PHAC Project File Reviews, were limited by the availability and nature of specific reports completed at the time of study. The original intent of the reports was not to address evaluation questions, but to address knowledge gaps and this should be kept in mind. A breadth of types of reports and material were reviewed (e.g., journal publications, advisory committee minutes) from all KD&E initiatives and the range of knowledge products produced were accurately reflected.

2. Internal interviews with PHAC personnel were conducted by PHAC evaluation staff. The interview responses were analyzed by two PHAC evaluation analysts to ensure consistency in coding, analysis and interpretation. The approach for the internal interview analysis of conducting a thematic analysis accompanied by interview respondent comments had the advantage of retaining the full flavour of the qualitative data.

   Limitations from the internal interviews may be, due to the fact that they were done by PHAC evaluation staff. This may have had an influence on interview responses. In addition, internal interview findings may be positively biased due to the vested interest of those interviewed in the KD&E initiatives. While a small number of internal interviews were conducted, the respondents were the lead staff responsible for the respective KD&E initiatives. They represent all KD&E Functional Component funded initiatives and are most capable of providing feedback on activities to date, outputs, challenges and suggestions for improvement. Efforts were made to address limitations through dual note checking, clear guidelines for internal transcription, data coding checks and the use of systematic analysis protocols.

3. External stakeholder interviews were conducted by an external contractor. Based on the interview content from the purposefully selected external interview respondents, it is the view of the evaluation contractor that the external stakeholders selected for interview were knowledgeable. Several brought perspectives from outside Canada as well as their knowledge of the Canadian situation.  One limitation is that the persons interviewed are not a representative sample.  Another limitation is they may represent biased perceptions due to their possible vested interest in KD&E initiatives. At the same time, because the respondents were closely involved in PHAC KD&E initiatives they have both relevant content expertise and are well positioned to provide informed perspectives and insight into KD&E progress and challenges.

   Strategies undertaken to ensure rigor of data collection and analysis included use of an external interviewer to reduce bias, clear guidelines for verbatim transcription, use of systematic analysis protocols, provision of interview excerpts to support analysis in detailed results reporting (separate paper), and production of an audit trail of interview results for confirmatory analysis as needed.

4. In order to compensate for the limitations of each data source, there was planned evidence triangulation. For example there was planned overlap in data collection in the interview schedules to allow for external and internal perspectives on some core evaluation questions.  In addition, the evaluation combined both file review to document knowledge products, funding sources and target audiences as well as in-depth, qualitative approaches to collect evidence of process, progress, design/functioning and immediate outcomes. Further, the evaluation utilized multiple evaluation analysts both internally and externally to bring multiple perspectives to reach consensus on analysis and interpretation. All the data for this evaluation are accessible for verification.

5. Data Gaps

   It would have been desirable to include tracking data on KD&E products and outputs, dissemination and reach with key target audiences in this formative evaluation. However, to date a common or comparable set of key indicators for activity tracking either in KD&E or in ISHLCD overall has not been developed. In addition, when KD&E Functional Component initiative lead staff was asked if they had been tracking or collecting this information on their own, only a few members indicated that they had partial but not complete information. This was not collected consistently and was not readily available for this evaluation report. In addition, the complexity of KD&E concepts and activities posed challenges to designing questions for the interviews, which in the end were quite complex. Hence, it was hard to fully address the interview questions in a limited interview format and given the evaluation’s time constraints due to turnover in PHAC evaluation personnel. Future evaluations should utilize multiple interviews with respondents or plan for longer interview duration and/or focus groups or group interviews to allow for full probing of complex questions.