2. Methodology

The methodology used in this evaluation seeks to determine to what extent the logic described in the technical appendices was carried out as planned. To do this, the evaluation used four lines of evidence to examine the evaluation questions below (see the Technical Appendices for the complete evaluation matrix).

Table 2: Evaluation questions

  1. Is there a continued need for the recruitment and integration of French-speaking immigrants into FMCs?
  2. Is the initiative aligned with CIC and GoC priorities?
  3. Is this initiative consistent with federal roles and responsibilities?
  4. Have the main partners undertaken coordination, collaboration and research activities to support the implementation of the initiative?
  5. Are French-speaking foreign nationals aware of opportunities to immigrate to FMCs?
  6. Do Francophone immigrants obtain strengthened settlement services in French in FMCs?
  7. Have the initiatives helped to achieve the objectives set in terms of the number of French-speaking immigrants going to FMCs?
  8. Have FMCs improved their settlement and reception services capacity to facilitate the recruitment, reception, integration and retention of French-speaking immigrants in FMCs?
  9. Is the Initiative guided by a clear mandate and specific roles, responsibilities and objectives?
  10. Are communications, relationships and information-sharing among program stakeholders effective?
  11. Is the management of the initiative coordinated and supported by the tools, resources (human and financial) and mechanisms needed to ensure effective delivery?
  12. Are performance measurement, monitoring and reporting for this initiative sufficient to ensure Initiative accountability?

This evaluation specifically focuses on the achievement of the immediate and intermediate outcomes of the Initiative, which in turn contribute to the immediate outcome of the Roadmap: “Community Development.” The horizontal evaluation of the Roadmap, coordinated by the Department of Canadian Heritage, will examine the full scope of intermediate results under the Roadmap.

2.1 Key informant interviews

Preliminary interviews

In order to adequately delineate the activities to be assessed in this evaluation, a series of preliminary interviews were carried out. In all, five semi-structured preliminary interviews were completed with CIC representatives.

Key interviews

Following the preliminary interviews, 22 key interviews were completed with 33 key stakeholders in order to obtain informed opinions and perceptions on the relevance, design and implementation, as well as the effectiveness of the Initiative. The stakeholders had the option to participate in the interview alone or as part of a group.

The key stakeholders were selected in consultation with program representatives, based on their knowledge and their participation in the Initiative. To obtain diverse perspectives on the Initiative, the key interviews sample included representatives from various key stakeholder groups:

  • CIC representatives: Integration Branch, Immigration Branch, Integration Program Management Branch, and representatives from the regional offices and the international region, Mission in Paris (n=19).
  • Members of the Steering Committee and the Implementation Committee (n=5);
  • Representatives from the regional coordination networks who were not consulted during the case studies (n=7); and
  • Other key stakeholders: Representatives from the Department of Canadian Heritage (n=2).

To prepare for the interview, each stakeholder received a guide with the questions to be discussed. The interviews were conducted in person or by telephone, in the stakeholder’s official language of choice. All data collected was analyzed using NVivo software in order to identify the themes associated with each of the evaluation questions addressed by this line of evidence. (For the interview guides, see the Technical Appendices.)

2.2 Document review

All of the documents relevant to the Initiative were analyzed. The document review helped answer all the questions in the evaluation matrix. The review provided information regarding the relevance of the Initiative, the activities undertaken as part of the Initiative, and the outputs produced and results achieved through the three components of the Initiative.

The list of documents consulted includes:

  • The Strategic Plan to Foster Immigration to FMCs
  • The Strategic Framework to Foster Immigration to FMCs
  • The Roadmap for Canada’s Linguistic Duality 2008–2013
  • The Action Plan for Official Languages 2003–2008
  • Speeches from the Throne
  • All of the Initiative planning documents, including Treasury Board submissions
  • All of the documents on the implementation of activities related to the three components of the Initiative
  • Funding records (project proposals, interim and final reports, etc.)
  • Background material on the mandate of the various committees (Implementation Committee, Steering Committee)
  • Other corporate documents and studies considered relevant

NVivo software was used to organize and analyze the considerable volume of data collected for the document review. (For more information on the documents reviewed, see the Technical Appendices.)

2.3 Administrative data analysis

The analysis of administrative data from a number of administrative systems used by CIC helped answer questions 6, 7 and 8 of the evaluation matrix. The administrative systems are:

  • SAP, CAMS and Regional Reports: The data from the Integrated Financial and Material Management System, also called SAP,Footnote 5 were analyzed to review actual expenditures (Votes 1 and 5) of the Initiative, and have been compared to the allocated budget. The information from SAP (Vote 5) was compared to the data compiled in the Contribution Agreement Management System (CAMS),Footnote 6 as well as to the information on the activities identified in the regional reports forwarded to the Integration Program Management Branch. The information was synthesized from these three data sources in order to create a complete list of projects related to the Initiative, funded by CIC throughout the course of the Roadmap. Only financial data for the 2008–2009 to 2010–2011 projects were available at the time of the evaluation. As a result, the data analysis and the list of projects are for this three-year period.
  • FOSS: Data from the Field Operations Support System (FOSS)Footnote 7 were analyzed in order to better understand the profile of newcomers settling in OLMCs, particularly French-speaking immigrants settling in FMCs. For lack of a more precise definition, FMCs refer to all communities outside Quebec where French-speaking people reside. The statistics on French-speaking newcomers were estimated based on a combination of variables, including mother tongue, official languages spoken and country of birth. The method used to derive this estimate is described in detail in section 3.2 of the Results.
  • In Quebec, the statistics on English-speaking newcomers were estimated based on the number of permanent residents who declared English to be their only official language spoken. The statistics presented on newcomers in OLMCs cover the period from 2003 to 2011.
  • iCAMS: The data from the Immigration Contribution Accountability Measurement System (iCAMS)Footnote 8 were used to calculate and compare the number of clients who received a needs assessment, referral, support, or information service (NARSI) or a community connection service (CC) in an official language or in other languages. In addition, the data on SPOs from this system were analyzed in order to better understand the distribution of services received in French in the census metropolitan areas (CMAs) across Canada, and to contribute to the capacity estimate for the delivery of settlement services in French in regions where CIC is responsible. The statistics presented on clients who received settlement services (NARSI and CC) cover the period from 2005–2006 to 2010–2011.

2.4 Case studies

Eleven case studies were conducted in the following cities: Moncton, St-Léonard, Halifax, Ottawa, Toronto, Winnipeg, Regina, Edmonton, Calgary, Brooks and Vancouver. Several factors influenced the selection of cities for the site visits:

  • The concentration of Francophone immigrants in FMCs (e.g.: a number of urban centres with a relatively high concentration of Francophone immigrants and a few rural centres with a lower concentration were selected);
  • The location of Francophone immigration networks and program representatives;
  • The distribution of funds for settlements services; and
  • The funding arrangements (e.g.: the provinces with alternative funding arrangements were included).
  • The case studies were made up of three main components:
  • An in-depth review of documents and internal systems;
  • Interviews with relevant stakeholders; and
  • Focus groups with clients who received settlement services funded by CIC.

With the exception of questions 2 and 3, the case studies helped answer all of the evaluation matrix questions.

In all, 43 interviews were completed with 59 stakeholders, including representatives from CIC’s regional and local offices, community organizations, Francophone immigration networks and provincial governments.

In addition to interviews, a total of 10 focus groups were conducted in each of the cities visited (except Brooks). These groups were made up of six to ten French-speaking newcomers (including some refugees).

The documents, interviews and notes relating to the case studies were analyzed through using NVivo software in order to facilitate organizing the information and comparing the perspectives of the various key stakeholder groups. (For tools in support of case studies, see the Technical Appendices.)

2.5 Methodological limitations

The main methodological limitations are largely related to the analysis of administrative data.

The review of data from SAP, CAMS and the information identified in the regional reports showed some inconsistencies as to what constitutes a project under the Initiative (Vote 5). As a result, the data from these three data sources were integrated in order to make a single list of projects related to the Initiative. Consequently, the level of investment for the various projects identified on the list does not directly correspond to the budget allocations for these activities, but reflects the scope of activities related to the goals of the Initiative in a more comprehensive way.

The definition adopted in the Strategic Plan clearly states the criteria to be used to define a French-speaking immigrant. However, there is currently no one single validated and accepted method to measure this definition. The evaluation had to estimate the population of French-speaking newcomers in minority communities by using approximate measurements derived from data in FOSS. FOSS does not note the official language of choice or the language used by the newcomer. Only the information on official languages spoken (based on unverified self-identification, and excluding any information on the proficiency level or usage) and mother tongue was available at the time of evaluation. The estimate of the number “French-speaking” newcomers as defined in the Strategic Plan raised an issue in this evaluation, which is explored in section 3.2 of the report on Results.

The figures in iCAMS may underestimate the level of service provided in French for NARSI and CC activities. Other evaluations on the components of the Settlement Program demonstrated the under-representation of service providers in iCAMS,Footnote 9 which could cause an under-representation of clients for certain years in the period observed in this evaluation. Moreover, in July 2010, the rules regarding the method of reporting for service providers was changed. Thus, although the results from iCAMS indicate the overall direction of trends regarding the level of service provided in French, the results must be interpreted with caution.

Features

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: