ARCHIVED – Enhanced Language Training Initiative: Formative Evaluation

Appendix A: Evaluation Methodology

The methodology employed for the formative evaluation of ELT included six lines of evidence:

  • Document review. This included a review of documents pertaining to design of the program (including procedures documents), the inventory of all ELT projects, implementation of the program (including conference reports, provincial agreements, slide presentations). A number of documents were also reviewed with a view to assess the overall rationale of the program.
  • Key informant interviews. In all, 26 interviews were conducted as follows:
    • 11 interviews with CIC personnel (n=7 interviews with 8 individuals) from NHQ, the regions (n=3 interviews with 4 individuals) and local offices (n=1 interview with 1 individual);
    • 12 interviews with provincial/territorial representatives from 9 provinces/territories;
    • 2 interviews with national key stakeholder organizations, including the Centre for Canadian Language Benchmarks (n=2 individuals) and the Centre for Education and Training (n=2 individuals).
    • 1 interview with another government department (Health Canada).
      The list of all key informant interviews is provided at the end of this section.
  • Statistical analysis of the ELT database. CIC provided GGI with data from the database and GGI undertook a number of analyses to:
    • Develop a profile of projects and participants to date;
    • Assess the success of the program to data; and
    • Assess the overall quality of the data and database in order to ensure that data will be available for the summative evaluation.
  • Site visit case studies. Based on a list of funded projects in the cities chosen for a site visit, GGI identified 12 SPOs to participate in a case study. Efforts were made to get a cross section of SPOs reflecting diversity in: type of organization, development versus delivery projects (or both), completed versus ongoing projects, projects with generic curriculum versus those that are occupation specific, small versus large numbers of participants, features of the project (e.g., partnerships, bridge-to-work component, focus on certifications). Each case study site visit included an in-depth interview with one or more SPO representatives, a focus group of clients [ note 40 ], and a document review. Where possible, interviews were also undertaken with employers and/or instructors. A list of SPOs visited as part of the case studies is included at the end of this Appendix.
  • Mini case studies. This line of evidence was added near the end of the data collection phase due to challenges with the original planned line of evidence of the online survey of SPOs. In all, 20 additional SPOs were interviewed by telephone using the same interview guide as for the SPO interviews for the site visit case studies. A list of SPOs that participated in a case study is presented at the end of this Appendix.
  • Expert Assessment. Through the conduct of the site visit case studies, GGI assembled 13 tools from ten SPOs. [ note 41 ] Two experts in English as a second language and education reviewed the 13 tools using a review template.

In carrying-out the methodology, a number of data collection tools were developed. The development of the instruments was guided by the program’s existing logic model and evaluation matrix (Exhibits A1 and A2, respectively). Instruments are available at the end of this appendix and include:

  • Key informant interview guides. Six guides were developed for the following interview types: CIC managers and program officers, provincial representatives with an agreement, provincial representatives without an agreement, SPO representatives, employer representatives, other stakeholder and federal department representatives.
  • Client focus group guide.
  • Case study review template. This template, based on the key informant guides and focus group protocols, will provide the framework for the site visits, including the list of data required sought from all lines of evidence in the field. The mini case studies did not seek to answer all fields the template but only to answer the questions contained on the SPO interview guide.
  • Tool review template. In order to ensure a consistent approach to the review of the tools, a template was developed that examined issues related to the development process, content, format and design, and generalizability of the tool.
Revised Enhanced Language Training Logic Model

Exhibit A2: Evaluation Framework for Enhanced Language Training

Evaluation Issues Indicator Methodology
Program Rationale of the ELT Initiative: To what extent does ELT remain relevant to federal government priorities and to the needs of newcomers to Canada?
1. Is there a continuing need for federal government involvement?
  • At the formative evaluation, we will want the consultant to determine how we will assess this question at the summative evaluation. How can we know that labor market integration of immigrants is a problem? Is it a problem in relation to particular occupations?
  • To be developed by the Consultant
2. Are the objectives of ELT consistent with current GoC and CIC priorities and objectives?
  • GoC/CIC consider language training and labor market integration of immigrants to be important areas for continued government involvement
  • Document Review (may include information notes, contact memo, reports, annual reports, processes and procedures documents, meeting minutes, etc.)
  • Interviews with CIC and OGDs
Program Delivery: To what extent are the design and delivery of the initiative appropriate?
3. Are partnerships with provinces operating effectively?
  • Agreements/Terms and conditions have been signed
  • Program objectives, roles & responsibilities are clearly understood by CIC, provincial partners and SPOs (including for the period covering post decentralization of the initiative)
  • Necessary information is shared between parties in an open and timely manner
  • Respective responsibilities are carried out in a complete and timely manner
  • Document Review – Agreements/Terms and conditions with provinces, program documents, meeting minutes, correspondence, etc.
  • Interviews with CIC, Provinces
  • Survey of SPOs (N=156)
4. Are communications, relationships between SPOs, Provinces and CIC effective?
  • Transparency of processes
  • Consistency across jurisdictions or type of partnerships
  • Efficiency of call for proposal
  • Views/opinions of SPOs, provinces and CIC
  • Document Review – Agreements with provinces and SPOs, evaluation grid for assessing project applications, etc.
  • Interviews with CIC, provinces, and SPOs representatives
  • Survey of SPOs (N=156)
5. Recognizing process is different from one province to another, are CIC & provinces receiving good applications in a timely manner? If not, what are the problems (e.g., insufficient promotion, requirement for cost-sharing, bottlenecks in process, etc.)?
  • Number of applications is close to expected/ targeted
  • #/level of approved applications is close to expected/targeted
  • CIC and Prov. Are satisfied with number, quality, etc. of applications
  • Document Review (may include information notes, inventory list of all projects, contact memo, reports, administration/financial data analysis, annual reports, statistical reports, processes and procedures documents, meeting minutes, etc.
  • Interviews with CIC and Provinces
6. To what extent do service providers collect and report initiative data? Will there be information available to measure the impact of the initiative 2-3 years from now?
  • Data being collected is appropriate for CIC to manage, monitor and assess ELT
  • SPOs provide complete, consistent and accurate data
  • Document Review (may include information notes, inventory list of all projects, contact memo, reports, financial data, annual reports, statistical reports, processes and procedures documents, meeting minutes, etc.)
  • Interviews with CIC, provinces and SPOs representatives
  • Survey of SPOs
  • Expert review of products
7. Do the development projects result in useful tools/information that support and improve the delivery projects?
  • Development projects create new curriculum, tools, assessment processes, research, etc.
  • Stakeholders and independent academics consider tools to be appropriate and useful
  • New tools, etc. are used in delivery projects
  • Success of delivery projects linked to tools
  • Document Review (compare documents from development projects with those of delivery, Narrative Progress Report)
  • Interviews with CIC, Provinces and SPOs
  • Survey of SPOs
  • Expert review of products
8. Are there any barriers for potential ELT clients? (eg, waiting list, transportation, profession not represented and in need, etc.)
  • SPOs offer support services (child-minding, transportation, etc) as required
  • Support services are used by participants
  • Participants are satisfied with support services
  • SPOs and participants identify potential barriers that are not addressed
  • #/% of participants who do not complete project due to barriers
  • Cost to clients is not a barrier
  • Doc Review – Narrative Progress Report
  • Needs Assessment - if any was done…
  • Analysis of data from iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Focus Groups of clients (2 groups: 1 gr. with those who have completed training and 1 gr. With those who have not complete training).
  • Interviews with SPOs
  • Survey of SPO
  • Data collection + questionnaire
9. a) To what extent are delivery projects able to engage employers? b) Does this engagement make a difference on the impact of the delivery project (what is the difference between delivery projects where employers are engaged vs where employers aren’t?)?
  • #/% of delivery projects in which employers are active participant
  • Ways in which employers are engaged in projects
  • Comparison of labour market outcomes for those projects that do and do not engage employers
  • #/% of clients who have completed ELT and have found a position in the labor market at levels commensurate with their skills
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Survey and Interviews with SPOs
  • Focus group with clients
10.What are the qualifications of teachers/instructors?
  • Type of qualification
  • Improvement in CLB level by qualified teacher
  • ELT completion rate by teacher’s profile
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Survey and Interviews with SPOs
  • Focus group with clients
Success: To what extent has the initiative been successful in achieving its immediate objectives/outcomes?
11. Have clients improved their language skills?
  • Clients successfully complete ELT
  • Scores obtained at the post test of ELT are higher than those at pre-test (data not available until the summative evaluation)
  • SPOs and clients’ perception/opinion
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Interviews with SPOs
  • Interviews/Focus group with clients
12. Do clients have more experience in the Canadian work environment?
  • Clients have internship or work placement
  • Analysis of data looking at variation related to length, occupation, paid/voluntary
  • Clients/employers feel work placement is useful
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Interviews with SPOs
  • Interviews/Focus group with clients and with employers
13. Have clients increased their knowledge related to the Canadian work environment?
  • Clients receive training/info related to Canadian work/occupation-specific environment
  • Clients feel training has enhanced skills
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Interviews with SPOs
  • Interviews/Focus group with clients
14. Have clients increased their job search skills?
  • Clients have taken job search skill training (resume, interview courses, exam preparation, etc)
  • Clients feel training has enhanced skills
  • Clients have undertaken more job searches
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Interviews with SPOs
  • Interviews/Focus group with clients
15. Do clients have established networks/contacts/mentors?
  • Clients have mentors/Clients have networks/Clients have contacts
  • Clients feel mentor/network is useful
  • Clients provide example of ways in which mentors/contacts/networks have enhanced their ability to find work
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Interviews with SPOs
  • Interviews/Focus group with clients
Intermediate Outcome:
16. Do clients have better job opportunities
  • Clients find more application opportunities (where they meet essential criteria and can send their resume);
  • Clients are invited to job interviews;
  • Clients feel better equipped to compete in the Canadian labor market;
  • Interviews/Focus group with clients
  • Interviews with SPOs
  • Doc Review – Narrative Progress Report (to see if they mention any thing about permanent placement rate)
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
17. Is any model more successful than others?
(Elements to consider for the model are: type of language training; internship/work placement; Canadian work environment; job search skills; network/mentoring/contact; employers’ engaged).
  • At formative evaluation will be linked to the intermediate outcome. At summative will be linked to final outcomes.
  • Analysis of data from Q16 by a multi-dimensional typology (elements of the typology in Q17)
  • Interviews/Focus group with clients
  • Interviews with SPOs
  • Doc Review – Narrative Progress Report (to see if they mention any thing about permanent placement rate)
  • Data Analysis of iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
  • Expert Assessment
Final Outcomes: We need to make sure we will have the data in place to measure these outcomes at the summative evaluation
18. Do clients who have received and successfully completed ELT find positions in the labor market at levels commensurate with their skills and experience? Does this vary in relation to diversity of program participants?
  • Clients have completed ELT and have found a position in the labor market at levels commensurate with their skills
  • Analysis will be done in relation to occupation, #, type of intervention, provinces/federal administration, location, diversity of program participants (language, culture, immigration classes)
  • Will also assess to the extent possible # with jobs at the formative evaluation
  • Analysis of data collection, questionnaire
  • Interviews/Focus group with clients
19. Do clients remain in the labor market at levels commensurate with their skills and qualifications?
  • This question will be answered at the summative
  • Clients remain in the labor market at appropriate levels at the time of evaluation.
  • Follow-up survey of participants
  • Consultant to ensure follow-up contact data will be available for the summative evaluation
  • iCAMS or Quarterly Statistical Progress Report or Data collection questionnaire
Effectiveness and Efficiency: To what extent do job-specific language training and bridge-to-work activities provided by ELT involve the most appropriate efficient and cost-effective methods to meet their objective?
20. Is the operation of the ELT initiative cost-effective?
  • Actual achievement of outcomes and costs associated with them
  • Comparison of costs for different (existing?) delivery models
  • Cost-effectiveness study (Data Analysis of financial data and interviews)
  • Consultant ensure costing data will be available at time of summative evaluation
Alternative Programs of Workforce Integration
21. Are there alternative programs of workforce integration available to immigrants? If so, how do they compare to ELT with respect to eligibility, scope, cost, etc.?
  • This question will be answered at the summative
  • Identification of alternative programs of workforce integration available to immigrants
  • Stakeholder views re alternative programs of workforce integration
  • For the formative, we will want consultant to identify potential alternative programs.
  • Comparative review of existing alternative programs of workforce integration
    • Identification (interviews, document reviews)
    • Interviews with reps of alternative programs
    • Review of documentation from alternative programs
  • Interviews with key stakeholders

List of Key Informants

Citizenship and Immigration Canada

  • Rose Kattackal, DG, Integration Branch, CIC NHQ
  • Diane Mikaelsson, DG, Operations Branch, CIC NHQ
  • Elizabeth Ruddick, DG, Research and Evaluation Branch, CIC NHQ
  • Jean Séguin, Director, Client-Centred Program, CIC NHQ
  • Wally Boxhill, Director, Integration and Refugee Program Delivery, CIC NHQ
  • Martha Justus, A/Director, Strategic Research and Statistics, CIC NHQ
  • Lucie Béland-O’Keefe, A/Manager, ELT, CIC NHQ
  • Gail Devlin, Policy Analyst, Immigration Branch, CIC NHQ
  • Elisete Bettencourt, Regional Program Advisor, CIC Ontario Region
  • Rocky Gushuliak, Regional Program Advisor, CIC Prairie Region
  • Sylvie Tremblay, Regional Program Advisor, CIC Atlantic Region
  • Alissar Ribahi, CIC Officer, CIC Halifax Office, Atlantic Region
  • Daisy Pond, Program Advisor, Calgary Local CIC Office

Provinces/Territories

  • Brent Slobodin, Assistant Deputy Minister, Advanced Education, Yukon
  • Shelly Forrester, Manager, International Qualifications Unit, Ministry of Economic Development, British Columbia
  • Carolyn Dieleman, Manager, Language Training, Alberta HR and Employment, Alberta
  • Valerie Fulford, Project Officer, Government of Saskatchewan, Saskatchewan
  • Josh Yasinowski, Project Officer, Government of Saskatchewan, Saskatchewan
  • Jean Gabert, Government of Saskatchewan, Saskatchewan
  • Cathy Zao, Government of Saskatchewan, Saskatchewan
  • Margaret Pidlaski, Director, Labour and Immigration, Adult Language Training Branch, Manitoba
  • Judy Stanleigh, Manager, Language Training Unit, Ministry of Citizenship and Immigration, Ontario
  • Patricia Bishop, Manager, Labour Market Integration Unit, Ministry of Citizenship and Immigration, Ontario
  • Ashraf Ghanem, Program Consultant, NB Post Secondary Education Training and Labour, New Brunswick
  • Paroo McKinnon, Settlement Officer, NS Office of Immigration, Nova Scotia
  • Deborah Sheppard, Policy and Program Development Specialist, Newfoundland and Labrador

Stakeholders/OGDs

  • Carolyn Cohen, Centre for Education and Training
  • Subhadra Ramachandran, Centre for Education and Training
  • Pauline Mcnaughton, Centre for Canadian Language Benchmarks
  • Pat Meek, Centre for Canadian Language Benchmarks
  • Helga Loechel, Senior Policy Analyst, Health HR Strategies Division, Health Canada

List of Participating SPOs

Site Visit Case Studies

  • Back in Motion/MOSAIC, Vancouver, BC
  • Edmonton Mennonite Centre for Newcomers, Edmonton, AB
  • Vancouver Community College, BC
  • Bredin Institute – Centre for Learning, Edmonton, AB
  • Regina Open Door Society, Regina, SK
  • Government of Saskatchewan, Regina, SK
  • Assiniboine, Cambrian and Astra Credit Unions, Winnipeg, MB
  • Red River College, Winnipeg, MB
    Toronto District School Board, Toronto, ON
  • COSTI, Toronto, ON
  • Inter-Cultural Neighbourhood Social Services, Mississauga, ON
  • Halifax Immigrant Learning Centre (HILC) and Metropolitan Immigrant Settlement Association (MISA), Halifax, NS

Mini Case Studies

  • Association for New Canadians, St. John’s, Newfoundland and Labrador
  • New Brunswick Multicultural Council, Fredericton, New Brunswick
  • Holland College, Charlottetown, P.E.I
  • Mohawk College of Applied Arts & Tech, Hamilton, ON
  • Niagara College, Welland, ON
  • Waterloo Region District School Board, Kitchener-Waterloo, ON
  • La Cité Collégiale, Ottawa, ON
    Women’s Enterprise Skills Training of Windsor(WEST), Windsor, ON
  • Keyano College, Fort MacMurray, AB
  • Brooks Adult Community Centre, Brooks, AB
  • Yukon College, Whitehorse, YK
  • Fatima Soares and Associates, Winnipeg, MB
  • Central Vancouver Island Multicultural Society, Nanaimo, BC
  • Red River College, Winkler, MB
  • Penticton, Penticton, BC
  • Red River College, Steinbach, MB
  • Camosun College, Victoria, BC
  • Centre for Skills Development and Training, Burlington, ON
  • Chilliwack Community Services, Chilliwack, BC
  • Community Futures Corporation of North Fraser Mission, Abbotsford, BC

______________

[40] Although there were twelve case studies, there were only eleven focus groups because one of the case studies was of a development project. Also, because of inability to attract participants, one group included only one participant.

[41] Tools were not available from two SPOs.

Page details

Date modified: