ARCHIVED – Enhanced Language Training Initiative: Formative Evaluation
Appendix A: Evaluation Methodology
The methodology employed for the formative evaluation of ELT included six lines of evidence:
- Document review. This included a review of documents pertaining to design of the program (including procedures documents), the inventory of all ELT projects, implementation of the program (including conference reports, provincial agreements, slide presentations). A number of documents were also reviewed with a view to assess the overall rationale of the program.
- Key informant interviews. In all, 26 interviews were conducted as follows:
- 11 interviews with CIC personnel (n=7 interviews with 8 individuals) from NHQ, the regions (n=3 interviews with 4 individuals) and local offices (n=1 interview with 1 individual);
- 12 interviews with provincial/territorial representatives from 9 provinces/territories;
- 2 interviews with national key stakeholder organizations, including the Centre for Canadian Language Benchmarks (n=2 individuals) and the Centre for Education and Training (n=2 individuals).
- 1 interview with another government department (Health Canada).
The list of all key informant interviews is provided at the end of this section.
- Statistical analysis of the ELT database. CIC provided GGI with data from the database and GGI undertook a number of analyses to:
- Develop a profile of projects and participants to date;
- Assess the success of the program to data; and
- Assess the overall quality of the data and database in order to ensure that data will be available for the summative evaluation.
- Site visit case studies. Based on a list of funded projects in the cities chosen for a site visit, GGI identified 12 SPOs to participate in a case study. Efforts were made to get a cross section of SPOs reflecting diversity in: type of organization, development versus delivery projects (or both), completed versus ongoing projects, projects with generic curriculum versus those that are occupation specific, small versus large numbers of participants, features of the project (e.g., partnerships, bridge-to-work component, focus on certifications). Each case study site visit included an in-depth interview with one or more SPO representatives, a focus group of clients [ note 40 ], and a document review. Where possible, interviews were also undertaken with employers and/or instructors. A list of SPOs visited as part of the case studies is included at the end of this Appendix.
- Mini case studies. This line of evidence was added near the end of the data collection phase due to challenges with the original planned line of evidence of the online survey of SPOs. In all, 20 additional SPOs were interviewed by telephone using the same interview guide as for the SPO interviews for the site visit case studies. A list of SPOs that participated in a case study is presented at the end of this Appendix.
- Expert Assessment. Through the conduct of the site visit case studies, GGI assembled 13 tools from ten SPOs. [ note 41 ] Two experts in English as a second language and education reviewed the 13 tools using a review template.
In carrying-out the methodology, a number of data collection tools were developed. The development of the instruments was guided by the program’s existing logic model and evaluation matrix (Exhibits A1 and A2, respectively). Instruments are available at the end of this appendix and include:
- Key informant interview guides. Six guides were developed for the following interview types: CIC managers and program officers, provincial representatives with an agreement, provincial representatives without an agreement, SPO representatives, employer representatives, other stakeholder and federal department representatives.
- Client focus group guide.
- Case study review template. This template, based on the key informant guides and focus group protocols, will provide the framework for the site visits, including the list of data required sought from all lines of evidence in the field. The mini case studies did not seek to answer all fields the template but only to answer the questions contained on the SPO interview guide.
- Tool review template. In order to ensure a consistent approach to the review of the tools, a template was developed that examined issues related to the development process, content, format and design, and generalizability of the tool.
Exhibit A2: Evaluation Framework for Enhanced Language Training
|Program Rationale of the ELT Initiative: To what extent does ELT remain relevant to federal government priorities and to the needs of newcomers to Canada?|
|1. Is there a continuing need for federal government involvement?||
|2. Are the objectives of ELT consistent with current GoC and CIC priorities and objectives?||
|Program Delivery: To what extent are the design and delivery of the initiative appropriate?|
|3. Are partnerships with provinces operating effectively?||
|4. Are communications, relationships between SPOs, Provinces and CIC effective?||
|5. Recognizing process is different from one province to another, are CIC & provinces receiving good applications in a timely manner? If not, what are the problems (e.g., insufficient promotion, requirement for cost-sharing, bottlenecks in process, etc.)?||
|6. To what extent do service providers collect and report initiative data? Will there be information available to measure the impact of the initiative 2-3 years from now?||
|7. Do the development projects result in useful tools/information that support and improve the delivery projects?||
|8. Are there any barriers for potential ELT clients? (eg, waiting list, transportation, profession not represented and in need, etc.)||
|9. a) To what extent are delivery projects able to engage employers? b) Does this engagement make a difference on the impact of the delivery project (what is the difference between delivery projects where employers are engaged vs where employers aren’t?)?||
|10.What are the qualifications of teachers/instructors?
|Success: To what extent has the initiative been successful in achieving its immediate objectives/outcomes?|
|11. Have clients improved their language skills?||
|12. Do clients have more experience in the Canadian work environment?||
|13. Have clients increased their knowledge related to the Canadian work environment?||
|14. Have clients increased their job search skills?||
|15. Do clients have established networks/contacts/mentors?||
|16. Do clients have better job opportunities||
|17. Is any model more successful than others?
(Elements to consider for the model are: type of language training; internship/work placement; Canadian work environment; job search skills; network/mentoring/contact; employers’ engaged).
|Final Outcomes: We need to make sure we will have the data in place to measure these outcomes at the summative evaluation|
|18. Do clients who have received and successfully completed ELT find positions in the labor market at levels commensurate with their skills and experience? Does this vary in relation to diversity of program participants?||
|19. Do clients remain in the labor market at levels commensurate with their skills and qualifications?||
|Effectiveness and Efficiency: To what extent do job-specific language training and bridge-to-work activities provided by ELT involve the most appropriate efficient and cost-effective methods to meet their objective?|
|20. Is the operation of the ELT initiative cost-effective?||
|Alternative Programs of Workforce Integration|
|21. Are there alternative programs of workforce integration available to immigrants? If so, how do they compare to ELT with respect to eligibility, scope, cost, etc.?||
List of Key Informants
Citizenship and Immigration Canada
- Rose Kattackal, DG, Integration Branch, CIC NHQ
- Diane Mikaelsson, DG, Operations Branch, CIC NHQ
- Elizabeth Ruddick, DG, Research and Evaluation Branch, CIC NHQ
- Jean Séguin, Director, Client-Centred Program, CIC NHQ
- Wally Boxhill, Director, Integration and Refugee Program Delivery, CIC NHQ
- Martha Justus, A/Director, Strategic Research and Statistics, CIC NHQ
- Lucie Béland-O’Keefe, A/Manager, ELT, CIC NHQ
- Gail Devlin, Policy Analyst, Immigration Branch, CIC NHQ
- Elisete Bettencourt, Regional Program Advisor, CIC Ontario Region
- Rocky Gushuliak, Regional Program Advisor, CIC Prairie Region
- Sylvie Tremblay, Regional Program Advisor, CIC Atlantic Region
- Alissar Ribahi, CIC Officer, CIC Halifax Office, Atlantic Region
- Daisy Pond, Program Advisor, Calgary Local CIC Office
- Brent Slobodin, Assistant Deputy Minister, Advanced Education, Yukon
- Shelly Forrester, Manager, International Qualifications Unit, Ministry of Economic Development, British Columbia
- Carolyn Dieleman, Manager, Language Training, Alberta HR and Employment, Alberta
- Valerie Fulford, Project Officer, Government of Saskatchewan, Saskatchewan
- Josh Yasinowski, Project Officer, Government of Saskatchewan, Saskatchewan
- Jean Gabert, Government of Saskatchewan, Saskatchewan
- Cathy Zao, Government of Saskatchewan, Saskatchewan
- Margaret Pidlaski, Director, Labour and Immigration, Adult Language Training Branch, Manitoba
- Judy Stanleigh, Manager, Language Training Unit, Ministry of Citizenship and Immigration, Ontario
- Patricia Bishop, Manager, Labour Market Integration Unit, Ministry of Citizenship and Immigration, Ontario
- Ashraf Ghanem, Program Consultant, NB Post Secondary Education Training and Labour, New Brunswick
- Paroo McKinnon, Settlement Officer, NS Office of Immigration, Nova Scotia
- Deborah Sheppard, Policy and Program Development Specialist, Newfoundland and Labrador
- Carolyn Cohen, Centre for Education and Training
- Subhadra Ramachandran, Centre for Education and Training
- Pauline Mcnaughton, Centre for Canadian Language Benchmarks
- Pat Meek, Centre for Canadian Language Benchmarks
- Helga Loechel, Senior Policy Analyst, Health HR Strategies Division, Health Canada
List of Participating SPOs
Site Visit Case Studies
- Back in Motion/MOSAIC, Vancouver, BC
- Edmonton Mennonite Centre for Newcomers, Edmonton, AB
- Vancouver Community College, BC
- Bredin Institute – Centre for Learning, Edmonton, AB
- Regina Open Door Society, Regina, SK
- Government of Saskatchewan, Regina, SK
- Assiniboine, Cambrian and Astra Credit Unions, Winnipeg, MB
- Red River College, Winnipeg, MB
Toronto District School Board, Toronto, ON
- COSTI, Toronto, ON
- Inter-Cultural Neighbourhood Social Services, Mississauga, ON
- Halifax Immigrant Learning Centre (HILC) and Metropolitan Immigrant Settlement Association (MISA), Halifax, NS
Mini Case Studies
- Association for New Canadians, St. John’s, Newfoundland and Labrador
- New Brunswick Multicultural Council, Fredericton, New Brunswick
- Holland College, Charlottetown, P.E.I
- Mohawk College of Applied Arts & Tech, Hamilton, ON
- Niagara College, Welland, ON
- Waterloo Region District School Board, Kitchener-Waterloo, ON
- La Cité Collégiale, Ottawa, ON
Women’s Enterprise Skills Training of Windsor(WEST), Windsor, ON
- Keyano College, Fort MacMurray, AB
- Brooks Adult Community Centre, Brooks, AB
- Yukon College, Whitehorse, YK
- Fatima Soares and Associates, Winnipeg, MB
- Central Vancouver Island Multicultural Society, Nanaimo, BC
- Red River College, Winkler, MB
- Penticton, Penticton, BC
- Red River College, Steinbach, MB
- Camosun College, Victoria, BC
- Centre for Skills Development and Training, Burlington, ON
- Chilliwack Community Services, Chilliwack, BC
- Community Futures Corporation of North Fraser Mission, Abbotsford, BC
 Although there were twelve case studies, there were only eleven focus groups because one of the case studies was of a development project. Also, because of inability to attract participants, one group included only one participant.
 Tools were not available from two SPOs.
- Date modified: