Appendix D: Details on evaluation methodology

Key informant interviews

Key informant interviews were conducted with program representatives and individuals external to the program. Since data collection for the NTAI was conducted separately as part of TC’s evaluation of the NTAI, the interviews covered nine of the 10 program elements and were conducted in two stages. Seven interviews covering five program elements were conducted during the first phase as part of the preliminary data collection (six interviews with a total of eight key informants internal to the federal government and one interview with an external stakeholder). The second phase consisted of 82 interviews covering nine program elements (one of which specifically focused on covering two different program elements). Interviews were approximately 45 to 90 minutes in length and focused on the program element(s) with which the individual key informant had the most experience and familiarity. Key informant interview data were analyzed using a thematic approach and reviewed to identify the themes and areas of consensus and disagreement. ^{Footnote 1} The number of interviews by program element is shown in Table D.1, while Table D.2 shows the number of interviewees who participated, according to their role within the Adaptation Theme. ^{Footnote 2}

Table D.1: Number of interviews by program element
Program element	Number of interviews
ACCASP	7
CCPSP	11
UCECCN	4
HARS	12
PPHSACC	11
CCHAP	9
CARPAN	11
NISI	12
NRCan Adaptation Program	13

Table D.2: Number of interviewees by role
Role	Number of interviewees
Senior management	10
Program management	25
Federal representatives	10
Provincial and territory representatives	11
First Nations, Indigenous or Inuit community	6
Municipal representatives	4
Academics	7
Industry	7
Other	9

Survey

A survey was conducted between January 27, 2015 and February 24, 2015 to collect feedback from external stakeholders on the relevance and achievement of outcomes. The survey was sent to external stakeholders for the participating program elements for whom email contact information was obtained. ^{Footnote 3} The survey was primarily conducted online and respondents were initially invited to participate via email. Additional reminder emails were sent and follow-up telephone calls (which included the option to complete the survey by telephone) were conducted to maximize participation. A total of 148 individuals responded to the survey from a sample of 638, representing a 23% response rate. Respondents were allowed to answer questions for up to two of the program elements with which they were familiar. The survey was therefore able to collect 182 program element-specific responses, with 114 respondents providing data concerning a single program element and 34 respondents providing data concerning two program elements.^{Footnote 4} A breakdown of the types of survey respondent organizations is provided in Table D.3 and the number of responses and the corresponding percentage of the total number of responses for each program element are provided in Table D.4.

Table D.3: Survey respondents by organization type
Respondent organization	Number of respondents	Percentage of total number of respondents
Provincial and territorial government	37	25%
Federal government	28	19%
Non-profit organization	16	11%
First Nations, Indigenous or Inuit community	12	8%
Research or academic	12	8%
Municipal government	11	7%
Industry	10	7%
Health care	7	5%
Professional association	2	1%
Other^*	12	8%
No response	1	1%

Note: Question: “What best describes the type of organization you belong to?”
*Examples of “other” organizations provided by respondents include: regional organizations, community programs and unique partnerships of municipal, provincial and First Nations leadership.

Table D.4: Survey responses, by program element^{Footnote 5}
Program element	Number of responses	Percentage of total number of responses
ACCASP	6	3%
UCECCN	5	3%
HARS	39	21%
PPHSACC	14	8%
CCHAP	17	9%
CARPAN	22	12%
NISI	26	14%
NRCan Adaptation Program	53	29%

Note Questions: “Please select the Adaptation Theme program that you have the most experience with.” and “If you would like to answer survey questions about another Adaptation Theme program that you have experience with please select it from the list below.”

Case studies

Case studies were used to examine specific initiatives or projects within the Adaptation Theme. A total of four case studies were conducted. Each case study focused on a project from a different program element. Data was collected for the case studies through document and file reviews and interviews with key stakeholders (two to four interviews per case study). Interviews were conducted following a document review to facilitate informed interviews. Detailed consideration was given to the impacts of these projects in terms of how they supported, or are expected to support, the targeted community or sector in planning for and implementing adaptation measures. While it is not possible to generalize the results of case studies to the program element as a whole, the case study findings highlight aspects such as achievements, external factors, findings related to design and delivery and best practices and lessons learned to date. The case studies focused on the following four projects:

SakKijanginnatuk Nunalik: Building Sustainable Communities in the Face of Changing Climate, with a focus on the Municipal Water Sub-Project (from the CARPAN)
Coastal Infrastructure Vulnerability Index (from the ACCASP)
Forest Change Initiative (from the NRCan Adaptation Program)
Climate Change, Adaptation and Acute Gastrointestinal Illness in the Canadian North (from the PPHSACC)

Footnote 1

In summarizing the degree of consensus in interview findings, the following guidelines were used: “no” key informants (0%); “some/a minority” (25 to 44%); “approximately half” (45 to 55%); “a majority” (56 to 75%); “most” (76 to 94%); “almost all” (95 to 99%); and “all” (100%).

Return to footnote1 Referrer

Footnote 2

Since some interviews were conducted with more than one key informant, and some key informants provided feedback on more than one Adaptation Theme element, a total of 90 interviews (Table D.1) was conducted with a total of 89 key informants (Table D.2).

Return to footnote2 Referrer

Footnote 3

Eight of the 10 program elements were targeted with the survey. Qualitative evidence related to ECCC’s CCPSP element was gathered through key informant interviews, while TC completed a separate evaluation of the NTAI.

Return to footnote3 Referrer

Footnote 4

When undertaking the analysis of the survey data, “don’t know” answers were excluded for the purposes of the analysis; however, “don’t know” data was included in tabulations of responses for demographic data. Survey respondents were able to provide responses for up to two program elements and, for some outcomes, were able to provide multiple responses based on their familiarity with different products or activities, or with different aspects of intended outcomes. Survey results are therefore generally presented as percentages based on the number of responses received for a given question.

Return to footnote4 Referrer

Footnote 5

Table D.4 provides the number of survey responses for each program element targeted by the survey. As noted previously, two program elements, ECCC’s CCPSP element and TC’s NTAI element, were not included in the survey, as feedback for these programs was captured through other means. Percentages were calculated based on 182 responses.

Return to footnote5 Referrer

Page details

Date modified:: 2018-12-06

Language selection

Search

Appendix D: Details on evaluation methodology

Key informant interviews

Survey

Case studies

Page details