Evaluation of the Host Program

Appendix A: Evaluation methodology

Primary date sources

1. Interviews

Thirty-two interviews were conducted with three groups of Host Program stakeholders. The groups of interviewees are described below:

• Twelve CIC senior directors and managers were interviewed from a cross-section of positions (i.e., Director General, Directors, Managers, and Senior Advisors) within the Integration Branch, the Operational Management and Coordination Branch (OMC), or the Ontario, Prairies, or Atlantic Region. The average number of years that each respondent has worked with CIC programs was 6 years, with the least being one year and the most being 19 years.
• Fifteen CIC staff members were interviewed from various positions including Program Advisor, Senior Advisor, Supervisor, Research Officer, Policy Analyst, and Local Manager. The average number of years involved with CIC was 6 years, with the least being 1 and the most being 17 years. Although, two-thirds of these staff members were closely involved with the Host Program, the remaining one-third was very familiar with the Program.
• Provincial/territorial representatives participated in this evaluation from Ontario, Alberta, British Columbia, the Atlantic region, Yukon, and Manitoba (positions of respondents include Executive Directors, Directors, Team Leader, and Analysts). The participants were asked to provide their feedback on the settlement programs and funding mechanisms from their provincial perspective.

The regional distribution of each group is shown in Table A-1.

2. Surveys

In total, 336 surveys were completed by various Host Program stakeholders over the course of this evaluation (Table A-2) to collect a broad range of input from a variety of groups of Program stakeholders.

• Survey of SPO Directors/Managers
  An online survey was designed to capture the opinions of the directors and managers of SPOs that are funded by CIC to deliver any of the three settlement services: ISAP, Host, or WCI. This survey purpose was to collect information on Program relevance, its implementation, performance and design and management of the Program. The survey was distributed to all SPOs identified in CIC’s Integrated Financial and Material System (SAP). Out of the 65 senior managers that responded, thirty-two were directors and managers of SPOs that deliver Host completed the survey. On average, respondents had been involved with the Program for 10 years (their involvement varied from 4 months to 30 years). On average, these SPO directors and managers reported that they spend 91% of their time on work related to the Host Program (e.g., providing direction, training, support and leadership to Host Program workers).
• Survey of SPO Staff
  An online survey was designed for settlement workers delivering Host. This survey was designed to gather information on Program performance and its design and management, including the capacity of SPOs and CIC to deliver the Host Program. Additionally, data was gathered to assess the outcomes of the Host Program. This survey was distributed to all SPOs that received Host funding between 2004/05 and 2008/09. Forty-four Host SPO representatives from 31 different organizations completed the questionnaire. On average, respondents had been involved with the Program for 3 years (involvement varied from less than 1 year to 15 years). The Host SPO staff reported spending an average of 106 hours per month on activities related to the Host Program (this varied from 5 hours up to 180 hours). The respondents reported that they were involved primarily in the recruitment, matching and support of newcomer/volunteer matches (64%). In addition, about one-third (32%) reported involvement in organizing and facilitating individual and group activities. Twenty-one percent of the respondents were specifically involved with a Host Program targeting youth.
• Survey of Host Clients
  A survey was designed to collect outcome information from Host clients. Host clients were also asked questions relating to Program delivery, its ability to meet their needs as well as any existing barriers to participation. SPOs were requested to distribute the electronic link or the hardcopy of the survey to their clients. To increase the response rate, the survey was available in nine languages to accommodate the language limitations of newcomers and to increase their comfort level by communicating in their first language (see Table A-3).

The survey was administered in three ways: online, on paper, and by telephone. Ninety-two Host clients completed the survey, of which 69 were completed online, 22 on paper, and one by telephone. On average, respondents had been in Canada for an average of 4 years (n=88). More than two-thirds (69%) of clients were female and 50% were between the ages of 25 and 44 (n=89).

Host clients surveyed represented several immigration categories: 38% were from the family class, 27% from the refugee class, and 17% from the economic class. Almost half (45%) were the principal applicant, 28% were spouse/common law, and 20% were dependents. At the time of completing the questionnaire, 85% of the Host clients were permanent residents, 7% were Canadian Citizens, 1% were in the process of obtaining permanent resident status, and the remaining 7% did not provide this information (n=92).

SPOs report client statistics though the Immigration–Contribution Accountability Measurement System (iCAMS). The Host client survey sample was compared with a larger sample of Host clients obtained through iCAMS (see Table A-4). The comparison indicates that the characteristics of survey respondents were generally consistent with those presented in iCAMS. Where they are considerable differences, the survey respondents were more likely to:

• Be in the Family class (38% versus 8.5%);
• Be female (69% versus 53%); and
• Identify India as the country of origin (13% versus 3%).

Source: iCAMS, Surveys

• Survey of Host Volunteers
  A survey of Host volunteers was design to capture their experiences in participating in the Host Program. Contact information for volunteers is not available to CIC. As a result, SPOs were requested to distribute a survey to their Host volunteers. One hundred and sixty-eight Host volunteers completed the evaluation questionnaire. Volunteers were more likely to have been born in Canada (59%) than outside Canada (41%) (n=163). Those not born in Canada had lived in this country for an average of 16 years (n = 68). Almost twice as many respondents were female (68%) versus male (n =163). Volunteers were somewhat more likely to be between 25 and 34 years old (20%); the rest were evenly distributed among other age categories except for those within the age group of 65 or older (9% of volunteers). Most Host volunteers (96%) were not previous clients of the Program.

More than one-quarter (26%) of Host volunteers were students and 15% were retirees (6% were retirees from the education field and 9% retirees from other professions). Among those who were employed, 12% of volunteers were in the health care and social assistance field and had occupations such as doctors, nurses, social workers, and employment counsellors. For a list of other occupations, see Figure A-1 .

Figure A-1: Occupations of Host volunteer survey respondents

Source: Volunteer Survey (n=163)

Text version: Occupations of Host volunteer survey respondents

3. Focus Groups

Separate focus groups were conducted with CIC officers, SPO representatives (including ISAP and Host SPOs), and a combination of Host volunteers and clients. In total, 18 focus groups were conducted in six cities across Canada. One focus group with SPOs was conducted in French. The purpose of focus groups with CIC and SPO staff was to collect data on Program need, delivery and management. Focus groups with clients and volunteers on their experience participating in the Program. CIC regional directors were consulted to select locations for focus groups and regional/local offices assisted in organizing the events. CIC officers invited SPOs to the focus groups and SPOs asked their clients and volunteers to participate in these events. The breakdown of participants is shown in Table A-5.

Secondary data sources

1. Document review

A detailed review was undertaken with a particular emphasis on Program relevance including continuing need for the Host Program, consistency with the priorities of Government of Canada and CIC and federal roles and responsibilities in funding and developing such programs. The document review also provided information relating to particular issues of Program performance and design and management. A wide variety of documents including Accountability, Risk and Audit Frameworks (ARAFs) and manuals, CIC Reports on Plans and Priorities (RPPs), federal budgets, Speeches from the Throne, legislation, discussion papers, and previous review and evaluation reports were reviewed (a list of documents is provided in Appendix D). Other key documents/information sources reviewed include government and SPO websites, CIC reporting templates, training and promotional materials, statistics, guidelines and tools including SPO intake/needs assessment forms.

2. Literature review

The literature review included online and library materials and archives. The literature review provided contextual information regarding the need for Host, the role of networking in settlement of immigrants and relationship with other settlement programs, as well as comparison of similar programs in other jurisdictions. A list of references can be found in Appendix E.

3. Administrative data analysis

The iCAMS and the Integrated Financial and Material System also known as SAP, were reviewed and analyzed to prepare a profile of the service providers, program services, and clients. iCAMS is an Internet-based system through which SPOs provide CIC with information about their services and clients. iCAMS has collected information on Host since April 2004. SAP is a financial data system that tracks all funds committed by CIC and serves as a central repository of all Contribution Agreements (CAs).

• Previous
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10 (current)
• 11
• 12
• 13
• 14
• 15
• Next