Evaluation of the Welcoming Communities Initiative

3. Evaluation Methodology

Due to the similarity of ISAP, Host and WCI in terms of objectives, target population, expected outcomes, delivery structure, and data sources, a single evaluation design was developed to cover the evaluation issues and questions of the three programs. In the course of the evaluation, the decision was made to produce a separate evaluation report for each program.

The evaluation of WCI involved the collection and analysis of data from a variety of primary and secondary sources. The evaluation made use of multiple lines of evidence, including both qualitative and quantitative methodologies. The following describes these methods as well as the strengths and limitations of the study.

3.1. Primary data sources

Primary data is data collected through interviews, surveys and case studies. Primary data sources for this evaluation included interviews with CIC Director Generals, Directors, Managers and Program Officers, interviews with Provincial Representatives, a survey of senior SPO representatives and those involved directly in the delivery of WCI projects, surveys of project users and participants, and case studies. The methodology used for each line of evidence is outlined below.

Interviews with CIC and Provincial Representatives

Nine CIC directors and senior managers provided feedback on WCI, including six at NHQ and three in regional offices. Among these nine people, the average number of years involved with CIC was 6 years.

Six CIC program officers and local managers provided feedback on WCI. They have been with CIC an average of 7 years. This group will be referred to as program officers in this report.

Five provincial representatives participated in this evaluation, including representatives from Alberta, Manitoba, Ontario, Nova Scotia and British Columbia.

The respondents were located across Canada, with the following breakdown.

Survey of Senior Representatives of SPOs involved in Delivering WCI

An online survey was designed to capture the opinions of the directors and managers of SPOs that were funded by CIC to deliver any of the three settlement services of ISAP, Host, or WCI. The survey was distributed to all SPOs receiving WCI funding. The organizations were sent three reminders to participate, by email and telephone calls. The survey was available in English and French between August and December 2009. SPO directors and managers from six organizations that received WCI funding completed the online survey for the evaluation (accounting for 13% of the 45organizations that received WCI funding during the reporting period). Five also reported receiving funding for ISAP while two reported delivering other settlement services such as Enhanced Language Training (ELT), LINC, or the Resettlement Assistance Program (RAP). The average number of years that these senior representatives were involved with the settlement programs was 7.9 years (ranging from 0.25 years to 25 years). The organizations have been delivering WCI for an average of 1.8 years (minimum of one year and a maximum of three years).

Survey of SPOs

Between August 2009 and January 2010, attempts were made to contact all WCI SPOs listed in SAP by email and telephone (up to five times for each SPO) to invite them to complete an online survey based on their WCI project(s). Representatives of 12 organizations completed the survey (accounting for 27% of the 45 organizations that received WCI funding during the reporting period). Of the 12 projects associated with these SPOs, six were national in scope and six were regional. Seven projects were ongoing and five have been completed. All respondents were directly involved in the projects in capacities such as planning, coordinating the implementation, facilitating and overseeing the activities, or developing project products and resources. We refer to this group as “WCI SPOs” in this report.

Survey of Project Users/Participants

An online survey was created to capture impact data and explore the experiences of individuals who had used products of WCI projects or had participated in WCI activities. WCI SPOs were asked to distribute the survey among their project users/participants. In addition, each SPO was contacted up to five times in order to remind them about the importance of this type of feedback and encourage them to send the survey link to their project users/participants^{Footnote 19}. The survey was available online in English and French, between October 2009 and January 2010.

Twenty-four project users/participants completed the survey^{Footnote 20}. Participants had experience with eight different WCI projects. In terms of providing services to newcomers, 15 participants reported that their respective organizations do provide services to newcomers, five did not, three respondents were unsure, and one did not respond. Seven respondents worked for SPOs, 6 were with community organizations and 5 respondents were with educational institutions.

Case Studies of Funded Projects

A total of 14 projects funded under WCI were selected as case studies. Case studies were selected to represent the diversity of WCI projects, as characterized by variations in:

• Focus (outreach, awareness, training, settlement support, research);
• Target (newcomers, community organizations, Canadians); and
• Geographical location (across Canada, urban vs. rural), and reach (national vs. regional).

Each case study involved a set of interviews and a review of documents, such as reports and promotional materials. The table below summarizes the characteristics of projects selected for case study.

3.2. Secondary data sources

The secondary data sources for this evaluation include a document review, literature review, and a review of administrative and financial data. Descriptions of each secondary data source are provided below.

• Document review. A wide variety of documents were reviewed, including the WCI RMAF, program manuals, RPPs, Federal budgets, Speeches from the Throne, action plans and acts, discussion papers, previous review and evaluation reports, and WCI reports to CAPAR. A review of WCI reports and CAs was also undertaken to identify and group funded projects based on their objectives, scope and outcomes in order to develop a typology of WCI projects.
• Literature review. The literature review included online materials and WCI research reports as well as other academic articles on racism and discrimination in Canada. The literature review focused mainly on the need for programs, such as the WCI.
• Administrative Data. The Integrated Financial and Material System (also known as SAP) and CAMS (Contribution Agreement Management System) were reviewed and analyzed to prepare a profile of the Initiative and identify budget information. SAP is a financial data system that tracks all funds committed by CIC and CAMS collects information on the activities and objectives of projects receiving contribution funding.

3.3. Evaluation strengths and limitations

There were some limitations associated with the implementation of the methodology. They included:

• Difficulty in tracking outputs and outcomes – There were a wide variety of projects funded under WCI that included a diversity of objectives, activities and target audiences. Common outputs and outcomes could only be identified at a very high level, and the Initiative experienced challenges in implementing a performance measurement strategy. Consistent information on WCI projects is not captured in any system, like the Immigration-Contribution Accountability Measurement System (iCAMS). The Initiative has relied on information, largely narrative and variable in its content, collected through its annual reporting to CAPAR. Also, respondents tended to be familiar with only a small number of projects, rather than the WCI as a whole. With this in mind, the data available for the evaluation was limited, and findings could not be easily generalized, particularly with respect to impacts.
• Level of overlap with ISAP B – Due to the similarity between the WCI projects and other settlement programs, particularly ISAP B projects, it was difficult for SPOs and, in some cases, CIC representatives to distinguish the WCI projects from those funded under other CIC settlement programs.
• Lack of information from project users – The number and characteristics of the total population of project users/participants is unknown. Considering the limited number of project users and participants surveyed, the findings on impacts should be interpreted with caution.
• Difficulty tracking Contribution Agreements (CAs) and funding – There are a number of challenges in tracking WCI-funded activities. Information on WCI projects is not easily recognizable in the Contribution Accountability Monitoring System (CAMS), as it is subsumed under ISAP and Host. Information on WCI funding is not easily obtained, as it is not consistently captured in SAP^{Footnote 21}, and cannot always be disaggregated from other settlement funding. Therefore, it was difficult to identify WCI projects for the evaluation.

The evaluation used several methods to enrich the data collection, as well as mitigate the limitations and increase confidence in the overall results. The strengths of the evaluation methodology included:

• Use of multiple lines of evidence through three different modes of communication (i.e. interview, survey, and case study) helped us to triangulate findings and increase their reliability. Multiple stakeholders from four distinct groups of stakeholders (i.e. SPO, CIC, provincial representatives, and project users/participants) increased the comprehensiveness of data and their reliability. Triangulation of research results indicates consistency in findings across data sources.
• Broad regional representation was obtained through the surveys, interviews and case studies across Canada.
• In describing the projects and developing a typology, a comprehensive document review was conducted using a large number of secondary sources, such as WCI reports, RMAF, CAPAR documents, administrative data, and CAs. WCI project activities and funding amounts were further verified with program representatives.
• Case studies provided additional information and examples of particular projects and their design, implementation and impact.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6
• 7 (current)
• 8
• 9
• 10
• 11
• 12
• Next