Evaluation of the Overseas Orientation Initiatives

2. Methodology

A terms of reference for the evaluation was approved by CIC’s Departmental Evaluation Committee in March 2011. The evaluation followed the scope and methodology set out in an evaluation plan developed during a planning phase prior to the commencement of the evaluation. The evaluation planning phase was undertaken from April to June, 2011 and was completed in consultation with all CIC Branches involved in the initiatives.

2.1 Evaluation issues and questions

The evaluation of the overseas orientation initiatives was designed to address three broad themes: relevance, design and implementation, and performance. In keeping with the requirements of the Directive on the Evaluation Function (Treasury Board of Canada Secretariat, 2009), program relevance was assessed in terms of: (1) continued need; (2) consistency with respect to federal roles and responsibilities; and (3) alignment with government and departmental objectives and priorities. Program performance was assessed by examining program results in terms of: (4) effectiveness; and (5) efficiency and economy (Table 2-1). See Appendix A for the logic model and Appendix B for the evaluation matrix, which includes specific indicators and methodologies for each evaluation question.

2.2 Evaluation scope

COA was previously evaluated in 2004-2005, therefore, the current evaluation included activities from 2005-2006 to 2010-2011. AEIP has not previously undergone an evaluation; therefore the evaluation included activities since the inception of the program in 2008. With respect to CIIP, the evaluation mainly on the first year of operation under CIC (2010-2011), however, because some of the participants to the FSW survey would have taken CIIP when it was the responsibility of Human Resources and Skills Development Canada (HRSDC), that line of evidence covers CIIP activities in 2009-2010.

2.3 Data collection methods

The evaluation of the overseas orientation initiatives included the use of multiple lines of evidence and complementary research methods to help ensure the strength of information and data collected. Following the completion of data collection, each line of evidence was analyzed separately using an evidence matrix, which was organized by evaluation question and indicator. A number of brainstorming sessions were then held with project team members to examine the findings from each line of evidence and to develop overall findings and conclusions. Each of the methods is described in more detail below.

Table 2-1: Summary of evaluation themes and questions

2.3.1 Interviews

A total of 72 interviews were completed for the evaluation (Table 2-2). Interviews were undertaken with six key stakeholder groups (i.e., CIC representatives, provinces/territories, delivery agents, service provider organizations, other stakeholders, and academics/experts). The list of interviewees was developed by R&E with consultation from the policy and program areas. Interviewees were selected based on their knowledge of the initiatives.

The interviews were conducted to respond to all of the evaluation questions in the evaluation matrix, covering areas of program relevance, design and implementation, and performance (see Technical appendix D for the interview guides).

Table 2-2: Summary of interviews completed

Six additional interviews were conducted with IOM program coordinators and representatives from the United States (US) and Australia to gather information on best practices for delivering pre-departure orientation to refugees.

The results of the interviews were summarized in an interview notes template and were then coded and analyzed to determine key themes. Where interview information is used in the report, it is presented using the scale shown in Table 2-3. Note that in some cases (i.e., where the number of interviewees was too small or where the question yielded more descriptive information), the responses were not coded and a summary approach to analysing the information was used.

Table 2-3: Scale for the presentation of interview results

2.3.2 Administrative data analysis

Administrative data, obtained mainly from the annual reports from the three initiatives, were reviewed to examine participant data by location of pre-departure orientation and target group. CIC landings data (by year and immigration category) were also used to examine the indicators related to whether pre-departure orientation was being offered in the appropriate locations, and the proportion of individuals taking pre-departure orientation in relation to source countries.

Estimates of FTE time spent on the initiatives were obtained from representatives of each of the initiatives and total spending by delivery agents was obtained from CIC tracking financial sheets. This information was used to establish the overall costs for each of the initiatives and to calculate cost per participant.

2.3.3 Site visits

Site visits were conducted in Manila, the Philippines; Taipei, Taiwan; and Hong Kong from November 30 to December 13, 2011. The objectives of the site visits were to obtain a better understanding of the pre-departure orientation programs’ operation, including how they work with local partners; and to collect materials and tools (e.g., promotional brochures, curriculum). The following activities were undertaken during the site visits:

• tours of the Canadian missions in Manila and Hong Kong, the S.U.C.C.E.S.S. facilities in Taipei, and the IOM and ACCC facilities in Manila;
• interviews/meetings with mission staff in Manila and Hong Kong, S.U.C.C.E.S.S. representatives in Taipei, and IOM and ACCC representatives in Manila; and
• attendance/observation at various sessions [e.g., AEIP banking workshop, AEIP pre-departure orientation session, AEIP one-on-one counselling session with LCs, a CIIP pre-departure orientation session (GO session) and a My Action Plan (MAP) session, and two COA sessions].

An observation protocol and interview guides were developed to gather information during the site visits (these tools are included in Technical appendix E).

2.3.4 Focus groups with live-in caregivers

Three focus groups were held in Oakville, Milton and Burlington with 32 LCs^{Footnote 13} (Table 2-4), with the objective of understanding how useful COA or AEIP was to them (e.g., usefulness of information provided, extent to which it helped with preparation for life in Canada). The focus groups were organized with the assistance of the Halton Multicultural Council, which screened current clients, invited participants, arranged for meeting locations, and provided translation assistance.

Table 2-4: Number of focus group participants, by location and orientation initiative

An introductory survey was administered to participants during registration to gather demographic information (e.g., country of birth, age, education levels). The introductory survey and the focus group moderator guide are included in Technical appendix F. All of the participants were Filipino and came to Canada either from the Philippines, Taiwan, Macau, or Hong Kong. Just under half of participants (15 of 32) were aged 25-35; 12 were aged 36-45; and the remainder (5) were aged 46-55. Most (29 of 32) had post secondary education (e.g., college, university).

2.3.5 Analysis of COA survey responses

IPMB administers a client survey to COA participants to gather information on the usefulness of the information provided during pre-departure orientation.^{Footnote 14} For the purposes of the evaluation, an extract of the survey responses was obtained in December 2011. The database contained responses from 915 individuals that took COA between October 2009 and September 2011 (Table 2-5). These responses were from individuals that chose to respond to the survey, therefore, it is not a random sample.

Table 2-5: Number of COA survey responses analyzed, by immigration category

Approximately 26,000 individuals took COA during that time period; therefore, the COA survey represents approximately 3.5% of the total population (margin of error of 1.9%, 19 times out of 20, or 95% of the time). See Technical appendix G for a comparison of the COA participant population and the COA survey population.

2.3.6 Federal skilled worker survey

FSWs are the only immigrants that can take any one of the three orientation initiatives. Therefore, this immigration category was surveyed to gather information on the outcomes of the initiatives to allow for a comparison between the three. The survey also included those that did not take pre-departure orientation and therefore, also allowed for a comparison between those that took pre-departure orientation and those that did not (see Technical appendix H for the FSW survey). The survey was available in English, French, Korean, Traditional Chinese, and Simplified Chinese.

An informed consent process was used to establish the sample for the FSW survey. A total of 23,450 letters were sent out and 3,034 individuals (13%) provided their consent to be surveyed (see Technical appendix I for more on this process). The survey was administered on-line beginning on September 16, 2011 and was left open until January 12, 2012. A total of 3,278^{Footnote 16} individuals were invited to participate in the survey and a total of 2,360 responses were received, for a response rate of 72.0%, or 10.0% of the total population. Table 2-6 shows the breakdown of survey responses, by orientation type. It was possible for respondents to have taken both CIIP and COA. The survey included a question to determine if they took both and if so, the survey directed them to respond to the survey with respect to the most recent pre-departure orientation they had taken.

Table 2-6: Number of FSW survey responses, by orientation type

The high survey response rate (72.0%) and a large sample size (2,360) allowed for a good level of confidence regarding the data. The margin of error was of 1.9%, 19 times out of 20 (95% of the time), which is low. In most situations, the sample allowed a broad variety of details with a sufficient number of respondents from different countries, different educational background, age, or geographical repartition. See Technical appendix J for a comparison of the FSW population and the FSW survey population.

2.3.7 Document review

Documentation was reviewed to examine program relevance, design and implementation, and best practices for delivery. The following types of documentation were reviewed:

The document review was completed using an excel template organized by evaluation question and indicator (see Technical appendix K for a list of documents reviewed for the evaluation).

2.4 Limitations and considerations

The evaluation contained a balance of qualitative and quantitative lines of evidence and allowed for the triangulation of research findings. However, there are a few methodological considerations that should be noted.

There is confidence in the FSW survey findings overall; however the level of confidence varies according to orientation initiatives.

The survey methodology included participants of all three orientation initiatives, as well as those that did not take pre-departure orientation, thus allowing for comparison between participants of the various orientation initiatives and also between those took pre-departure orientation and those that did not. There is also a good level of confidence in the survey findings, with a high survey response rate (72.0%) and a large sample size (2,360). However, it should be noted that in looking at the number of responses received for each of the orientation initiatives, there is a higher level of confidence in the responses for CIIP participants (599 responses) and COA participants (445 responses) than AEIP participants (89 responses). Therefore, caution should be used in drawing conclusion with the AEIP survey data.

There was limited information available to assess the impact of COA and AEIP on LCs.

The COA survey initially included only ten responses from LCs. To address this, program representatives worked with IOM to distribute the survey electronically to LCs, as it had recently begun to collect e-mail addresses of its participants. This was effective in increasing the number of responses to 72. The evaluation also included focus groups with LCs to supplement the survey results and gather more information on the impacts of COA and AEIP—although the participants were not selected randomly, as they were identified by a settlement organization. Therefore, results for LCs cannot be considered representative of all LCs.

The COA survey was not designed to respond to the evaluation and there were limited responses received from refugees.

The COA survey was not designed specifically to respond to the evaluation and therefore, did not provide information for all of the evaluation questions and indicators. These included, for example, indicators related to the effectiveness of promotional materials (e.g., how participants found out about the orientation sessions), the sufficiency of the time between taking orientation and departure, and how well pre-departure orientation prepared participants for employment. Only the survey questions that were aligned with the evaluation indicators were analysed.

In addition, given the size of Canada’s refugee population, the COA survey contained a limited number of responses from refugees. It was determined that conducting focus groups with refugees may not be effective in yielding additional information on results. A review of landings data concluded that it would have been difficult to locate a sufficient number of refugees in the same location, with the same cultural background and that landed in Canada within 3-9 months of taking orientation. To address this limitation, the evaluation included further research to identify additional information on best practices for delivering pre-departure orientation to refugees. This included additional interviews with representatives of pre-departure orientation programs in other countries and program coordinators with IOM, as well as additional review of literature and documentation to identify best practices for delivering pre-departure orientation to refugees.

Participants to pre-departure orientation are not tracked in a systematic way.

ACCC and S.U.C.C.E.S.S. track participants to pre-departure orientation (including name and contact information), however, CIC systems were not designed to identify which of those refugees and immigrants that arrived in Canada have taken pre-departure orientation.^{Footnote 18} This is further complicated by the fact that the time between taking orientation and arrival in Canada varies. For example, one could take orientation anywhere from one week to one year before departing for Canada. This resulted in some limitations with respect to calculating the proportion of individuals arriving in Canada that took orientation—information that was needed not only to examine program results and reach, but also to establish sample size for informed consent. Therefore, certain assumptions had to be made with respect to how much time elapsed between taking orientation and arrival in Canada. For example, based on the information from the COA and FSW survey, it was assumed that COA participants arrived in Canada anywhere between one and six months after taking orientation. It is possible that participants arrived sooner or later than that.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6 (current)
• 7
• 8
• 9
• Next

Features