Evaluation of the going to Canada Immigration Portal Initiative

2. Evaluation framework and methodology

2.1. Evaluation objectives, scope and questions

2.1.1. Objectives

The objectives of this evaluation were to assess the:

1. Relevance of this program in terms of:
   1. continued need;
   2. alignment with government objectives and priorities; and
   3. consistency with respect to federal role and responsibilities; and
2. Performance of this program in achieving results in terms of:
   1. effectiveness with respect to the intended outcomes of the programs, with a focus on their immediate and intermediate outcomes; and
   2. efficiency and economy, addressing the design and delivery approach of the GTC-IP program, as well as best practices in other jurisdictions, with a view to understanding the adequacy of these approaches and practices in meeting the information needs of potential newcomers and immigrants in order for them to work and settle in Canada.

The evaluation also examined issues of governance, collaboration and priority setting between and among GTC-IP program partners.

2.1.2. Scope

The study was undertaken in order to meet a commitment to Treasury Board to conduct a summative evaluation of the GTC-IP Initiative in 2009-10. The evaluation was delayed by one year in order to first conduct a validation of the Results-Based Management and Accountability Framework (RMAF) for the program, as well as a Data Capacity Assessment in 2009-10. As such, the evaluation, carried out from April 2010 to March 2011, covered the period from April 2005 to April 2010 and focused on the outcomes articulated in the program logic model, which stemmed from the RMAF validation exercise. The evaluation focused specifically on the GTC-IP and P/T portals and did not include municipal portals.

During the conduct of the evaluation, changes in the program have also occurred, which were not addressed in the evaluation report as they are outside the period under examination. However, these changes may influence the actions taken in response to the findings contained herein. Of note are further discussions between Portal partners regarding full integration of the GTC-IP into the CIC main site as well as further articulation of governance structures and decision-making processes across Portal partners. To the degree possible, these have been considered in the report conclusions and recommendations.

2.1.3. Evaluation questions

The evaluation questions (Table 2-1) were answered using multiple lines of evidence as follows:

• A document review;
• Key informant interviews;
• Surveys of online users of the GTC-IP;
• Survey of intermediaries;
• Focus groups with users and non-users of the GTC-IP;
• An assessment of the GTC-IP and P/T sites’ quality and consistency of information by a subject matter expert; and
• Analyses of web log files and available analytics.

A crosswalk between the evaluation questions and the methodologies used to answer them is shown in the Evaluation Framework found in Appendix A2. That framework also identifies the section of the report in which the information pertaining to each evaluation question can be located.

Table 2-1 : Evaluation issues and questions

Evaluation Questions

Relevance

1. Does the Initiative continue to be consistent with departmental and government-wide priorities?

2. Does the GTC-IP Initiative continue to address a demonstrated need?

3. Are CIC and HRSDC the most suitable delivery mechanisms for these information products and tools?

Design and Delivery

4. Is the Portal functional and user friendly?

5. Is the governance structure, internally within each department (CIC and HRSDC), between CIC and HRSDC, and between GoC and P/Ts) for the Portal effective?

6. Is the Portal URL the most appropriate, efficient and effective to meet the needs of the target population?

Performance

7. Is the target population aware of the GTC-IP? Are they using the GTC-IP? Why or why not?

8. Is collaboration and priority setting between and among the federal, provincial and territorial partners effective?

9. Do the provincial/territorial portals contain and provide links to relevant, up-to-date and understandable information on:

• immigrating, settling, living, visiting and studying in the P/Ts?
• working in the P/Ts?

10. Does the Portal contain and provide links to relevant, up-to-date and understandable information on:

• immigrating, settling, living, visiting, and studying in Canada?
• working in Canada?

11. Does the target population gain knowledge and settlement information regarding provinces, territories, and communities from across Canada?

12. Does the target population gain knowledge of immigrating to Canada and, living and working in Canada?

13. Does the Portal contribute to the target population’s ability to make informed immigration decisions, prepare for the immigration process and integrate into Canada upon arrival?

14. What have been the unintended outcomes of the GTC-IP, if any?

15. Is delivery of the GTC-IP efficient and cost-effective?

2.2. Methodologies

2.2.1. Document review

The following types of documents were reviewed during the evaluation (see Appendix D for the complete list of documents reviewed):

• CIC/HRSDC and other government department documents related to priorities and commitments;
• GTC-IP program documents such as policies, briefing notes, financial reports, statistical reports, public opinion and other research documents, usability testing reports, search engine optimization reports, web strategy documents, etc.;
• GTC-IP web-produced reports;
• GTC-IP RMAF and Data Capacity Assessment;
• Annual Service Plans of BC and MB, provincial portal contribution agreements and financial/activity reports, Letters of Understanding (LOUs) Memoranda of Understanding (MOUs), action plans and progress reports; and
• Communication and promotional materials, including frequently asked questions, press releases, fact sheets.

2.2.2. Key informant interviews

Key informant interviews were conducted with 41 interviewees in the following stakeholder groups:

• Past and present CIC officials (n=14);
• Past and present HRSDC officials (n= 8);
• Provincial/territorial portal representatives (n= 11); and
• Representatives from other government departments (OGDs) with and without web portals (n= 8).

2.2.3. Online user survey

An online user survey was conducted to collect information from those who had personally used either the GTC website or the WiC website. The survey was designed to elicit individual users’ perceptions of the websites, including the layout, quality and usefulness of the content, and the overall functionality. The online survey was also used to determine what information users accessed most frequently on the GTC-IP and whether the information provided was having its intended impact on users. The survey was completed by 299 individuals, accessed through the sites indicated in Table 2-2. The GTC-IP targets potential immigrants and newcomers to Canada but until an individual applies through one of the immigration programs, the size of the target group and their characteristics remain unknown. The program also targets friends and family of newcomers to Canada, and CIC does not have information on the size of this group or their relevant characteristics. As a result, the population for this survey is unknown.

Of those who completed the survey, 51% (n=153) did not reside in Canada, with another 4% saying they live in Canada only temporarily, suggesting an immigrant base of respondents to the survey. More males (56%) than females (44%) responded to the survey, with 80% of all respondents falling within 20-49 years of age. These characteristics also align with CIC’s past experience in immigration and respondent patterns. Additionally, 20% (n=60) of the surveys were completed in French.

The online user survey was available through a link to anyone who accessed the GTC website, WiC website, the WiC Facebook page, and WiC Twitter page. In a typical month in 2010, the WiC site received roughly 195,000 unique visitors compared to 53,000 unique visitors who went to the GTC site. This represents a GTC-to-WiC visitor ratio of 0.27, which is comparable to the ratio for respondents of the user survey (0.25).

Table 2-2 : Online user survey completions by source

* Other includes those who accessed the survey directly or from a bookmark.

2.2.4. Survey of intermediaries

A survey of intermediaries was conducted to assess the use and usefulness of the GTC-IP in providing services and information to new and prospective immigrants. The intermediary survey was sent to the following organizational groups:

• Service provider organizations (SPOs);
• Foreign missions in Canada;
• Canadian Visa offices abroad;
• National Associations; and,
• Sector Councils.

SPOs, including those not federally funded, were randomly sampled from a master list of immigrant-serving organizations across Canada that was created using a web search for potential users of the GTC-IP. Surveys were also sent to all foreign missions/consulates in Canada (list provided by the Department of Foreign Affairs) and to all CIC visa offices abroad, two groups thought to be likely users of the GTC-IP. Several sector councils and national associations, which are target groups of the GTC-IP, were also identified as potential respondents.

Table 2-3 provides a breakdown of representatives who were invited to complete the Intermediary Survey and those who completed it – an overall response rate of 33% was achieved, which was lower than expected. However, this is considered an average response rate for online surveys^{Footnote 16}.

Table 2-3 : Intermediary survey completions by organization type

2.2.5. Focus groups

Focus groups were conducted with two different groups of recent immigrants^{Footnote 19}: those who had used either the GTC website or the WiC website (user groups) and those who had not used the GTC or WiC websites but had used other internet sources when coming to Canada (non-user groups). Focus groups were designed to determine where participants found the information they needed to come to Canada or to assist others to come to Canada. More specifically, focus groups assisted in evaluating the relevance of the GTC-IP websites and the information contained on the GTC-IP. Both users and non-users were shown pages from WiC, GTC and the provincial/territorial websites in order for participants to familiarize themselves with the layout and provide comments regarding the GTC-IP and where possible, the effectiveness of the GTC-IP compared to alternative sources.

Focus group participants were recruited through the online user survey as well as through coordination and collaboration with federally funded Service Provider Organizations (SPOs) who offered to recruit participants for the groups. A total of 12 focus groups were conducted (6 with users and 6 with non-users of the GTC-IP) in SPO offices with a total of 91 recent immigrants^{Footnote 20} in the following cities:

• Vancouver (2 user groups and 2 non-user groups);
• Edmonton (1 user group and 1 non-user group);
• Winnipeg (1 user group);
• Toronto (2 user groups and 1 non-user group);
• Montreal (1 non-user group); and
• Halifax (1 non-user group).

The focus group in Montreal was conducted in French.

2.2.6. Subject matter expert assessment

A subject matter expert^{Footnote 21} on immigration and settlement content assessed the quality (relevance, currency, and clarity) of the information on the GTC, WiC, and provincial/territorial sites and also assessed the consistency of information on the provincial and territorial portals (see Appendix C for the list of websites assessed) with the information on the GTC-IP. The subject matter expert reviewed content across the following six categories including: immigrating, settling, living, visiting, studying, and working.

2.2.7. Log files and web analytics^{Footnote 22}

The web analytics reports and analyses of log files conducted by the consultants were used to gather information about referrals, website traffic, usage patterns, user behaviour, and website efficiency. The evaluation also looked at the number of errors that occur on each of the sites. Table 2-4 displays the information that was provided by Portal partners for use in the web analytics component of the evaluation.

Table 2-4 : Federal/provincial/territorial web analytics and log files provided

2.3. Limitations of the methodology

The methodology employed in this study was subject to several limitations identified below.

• Cost comparisons between the GTC-IP Initiative and OGD initiatives were not possible due to significant dissimilarities in the scope of the websites, technical considerations, and web data capture. This limited the extent to which the efficiency and cost-effectiveness of the GTC-IP (Evaluation Question 15) could be evaluated in comparison to other initiatives.
• A lower than expected number of completions (expected: 500^{Footnote 23}, obtained: 299) for the online user survey prevented some sub-group analyses such as using age group, country of birth, and length of time in Canada. The sub-group analyses would have resulted in a significant margin of error. Attempts were made to improve the number of completions – for example, continuous survey monitoring occurred and the length of time the survey was in the field was extended. Lower response rates among specific groups of intermediaries (e.g., 97 foreign missions in Canada were invited to complete the survey and only 11 responded) precluded analyses (due to a resulting high margin of error) by intermediary group and diminished the ability to make conclusions based on those data alone.
• It was not possible to know whether online survey respondents and focus group users of the GTC-IP were typical of all users of the Portal. Survey respondents and focus group users generally had positive opinions of the GTC-IP and may reflect a positively-skewed selection bias. Typically, the analysis of non-response bias consists of a comparison of variables in the survey sample with those in the sampling frame. This allows for an assessment of the difference between the true population (i.e., all users or non-users) and the survey sample (i.e., users who completed the survey, and those that attended focus groups). As no information was available regarding the survey population (users of the GTC-IP and non-users of the GTC-IP), it was impossible to assess whether the survey and focus group participants form a representative sample of all GTC-IP users and non-users. However, user survey respondent characteristics align with CIC’s past experience in immigration and respondent patterns. As well, triangulation of several lines of evidence was undertaken in order to improve confidence in the survey findings.
• Some focus group participants were unable to clearly recall experiences coming to Canada and using Internet information sources, which may have resulted in some not providing as much relevant information as others, and some of the barriers and challenges faced may not have been recalled. Fortunately, the impact of this limitation was minimized with the inclusion of over 90 focus group participants who had experience with the GTC-IP or immigrated to Canada within five years prior to focus group participation and the majority of these had clear recall.
• Inconsistencies were noted in web reporting and data collection techniques across the Initiative’s partners (WiC, GTC and provinces and territories). This resulted in limited comparable data available for analysis purposes between GTC and WiC. Comparisons were not sought between P/T partners; however, it did pose a challenge in the evaluation to have to report on a variety of metrics over various timeframes and collected at different intervals. Attempting to mitigate this, additional analytics and log files were requested but these were not available from all partners. The use of different analytics packages (which used different algorithms) to determine number of visits, unique visitors, and other key metrics among Portal partners also resulted in difficulty analyzing the effectiveness of promotional activities based on website usage. As well, only a few provinces and territories were able to provide longitudinal data which resulted in an inability to perform time trend analyses to determine the effectiveness of the provincial and territorial portals over time.

• Previous
• 1
• 2
• 3
• 4
• 5 (current)
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• Next