ARCHIVED – Enhanced Language Training Initiative: Formative Evaluation

Appendix A: Evaluation Methodology

The methodology employed for the formative evaluation of ELT included six lines of evidence:

• Document review. This included a review of documents pertaining to design of the program (including procedures documents), the inventory of all ELT projects, implementation of the program (including conference reports, provincial agreements, slide presentations). A number of documents were also reviewed with a view to assess the overall rationale of the program.
• Key informant interviews. In all, 26 interviews were conducted as follows:
  • 11 interviews with CIC personnel (n=7 interviews with 8 individuals) from NHQ, the regions (n=3 interviews with 4 individuals) and local offices (n=1 interview with 1 individual);
  • 12 interviews with provincial/territorial representatives from 9 provinces/territories;
  • 2 interviews with national key stakeholder organizations, including the Centre for Canadian Language Benchmarks (n=2 individuals) and the Centre for Education and Training (n=2 individuals).
  • 1 interview with another government department (Health Canada).
    The list of all key informant interviews is provided at the end of this section.
• Statistical analysis of the ELT database. CIC provided GGI with data from the database and GGI undertook a number of analyses to:
  • Develop a profile of projects and participants to date;
  • Assess the success of the program to data; and
  • Assess the overall quality of the data and database in order to ensure that data will be available for the summative evaluation.
• Site visit case studies. Based on a list of funded projects in the cities chosen for a site visit, GGI identified 12 SPOs to participate in a case study. Efforts were made to get a cross section of SPOs reflecting diversity in: type of organization, development versus delivery projects (or both), completed versus ongoing projects, projects with generic curriculum versus those that are occupation specific, small versus large numbers of participants, features of the project (e.g., partnerships, bridge-to-work component, focus on certifications). Each case study site visit included an in-depth interview with one or more SPO representatives, a focus group of clients [ note 40 ], and a document review. Where possible, interviews were also undertaken with employers and/or instructors. A list of SPOs visited as part of the case studies is included at the end of this Appendix.
• Mini case studies. This line of evidence was added near the end of the data collection phase due to challenges with the original planned line of evidence of the online survey of SPOs. In all, 20 additional SPOs were interviewed by telephone using the same interview guide as for the SPO interviews for the site visit case studies. A list of SPOs that participated in a case study is presented at the end of this Appendix.
• Expert Assessment. Through the conduct of the site visit case studies, GGI assembled 13 tools from ten SPOs. [ note 41 ] Two experts in English as a second language and education reviewed the 13 tools using a review template.

In carrying-out the methodology, a number of data collection tools were developed. The development of the instruments was guided by the program’s existing logic model and evaluation matrix (Exhibits A1 and A2, respectively). Instruments are available at the end of this appendix and include:

• Key informant interview guides. Six guides were developed for the following interview types: CIC managers and program officers, provincial representatives with an agreement, provincial representatives without an agreement, SPO representatives, employer representatives, other stakeholder and federal department representatives.
• Client focus group guide.
• Case study review template. This template, based on the key informant guides and focus group protocols, will provide the framework for the site visits, including the list of data required sought from all lines of evidence in the field. The mini case studies did not seek to answer all fields the template but only to answer the questions contained on the SPO interview guide.
• Tool review template. In order to ensure a consistent approach to the review of the tools, a template was developed that examined issues related to the development process, content, format and design, and generalizability of the tool.

Exhibit A2: Evaluation Framework for Enhanced Language Training

List of Key Informants

Citizenship and Immigration Canada

• Rose Kattackal, DG, Integration Branch, CIC NHQ
• Diane Mikaelsson, DG, Operations Branch, CIC NHQ
• Elizabeth Ruddick, DG, Research and Evaluation Branch, CIC NHQ
• Jean Séguin, Director, Client-Centred Program, CIC NHQ
• Wally Boxhill, Director, Integration and Refugee Program Delivery, CIC NHQ
• Martha Justus, A/Director, Strategic Research and Statistics, CIC NHQ
• Lucie Béland-O’Keefe, A/Manager, ELT, CIC NHQ
• Gail Devlin, Policy Analyst, Immigration Branch, CIC NHQ
• Elisete Bettencourt, Regional Program Advisor, CIC Ontario Region
• Rocky Gushuliak, Regional Program Advisor, CIC Prairie Region
• Sylvie Tremblay, Regional Program Advisor, CIC Atlantic Region
• Alissar Ribahi, CIC Officer, CIC Halifax Office, Atlantic Region
• Daisy Pond, Program Advisor, Calgary Local CIC Office

Provinces/Territories

• Brent Slobodin, Assistant Deputy Minister, Advanced Education, Yukon
• Shelly Forrester, Manager, International Qualifications Unit, Ministry of Economic Development, British Columbia
• Carolyn Dieleman, Manager, Language Training, Alberta HR and Employment, Alberta
• Valerie Fulford, Project Officer, Government of Saskatchewan, Saskatchewan
• Josh Yasinowski, Project Officer, Government of Saskatchewan, Saskatchewan
• Jean Gabert, Government of Saskatchewan, Saskatchewan
• Cathy Zao, Government of Saskatchewan, Saskatchewan
• Margaret Pidlaski, Director, Labour and Immigration, Adult Language Training Branch, Manitoba
• Judy Stanleigh, Manager, Language Training Unit, Ministry of Citizenship and Immigration, Ontario
• Patricia Bishop, Manager, Labour Market Integration Unit, Ministry of Citizenship and Immigration, Ontario
• Ashraf Ghanem, Program Consultant, NB Post Secondary Education Training and Labour, New Brunswick
• Paroo McKinnon, Settlement Officer, NS Office of Immigration, Nova Scotia
• Deborah Sheppard, Policy and Program Development Specialist, Newfoundland and Labrador

Stakeholders/OGDs

• Carolyn Cohen, Centre for Education and Training
• Subhadra Ramachandran, Centre for Education and Training
• Pauline Mcnaughton, Centre for Canadian Language Benchmarks
• Pat Meek, Centre for Canadian Language Benchmarks
• Helga Loechel, Senior Policy Analyst, Health HR Strategies Division, Health Canada

List of Participating SPOs

Site Visit Case Studies

• Back in Motion/MOSAIC, Vancouver, BC
• Edmonton Mennonite Centre for Newcomers, Edmonton, AB
• Vancouver Community College, BC
• Bredin Institute – Centre for Learning, Edmonton, AB
• Regina Open Door Society, Regina, SK
• Government of Saskatchewan, Regina, SK
• Assiniboine, Cambrian and Astra Credit Unions, Winnipeg, MB
• Red River College, Winnipeg, MB
  Toronto District School Board, Toronto, ON
• COSTI, Toronto, ON
• Inter-Cultural Neighbourhood Social Services, Mississauga, ON
• Halifax Immigrant Learning Centre (HILC) and Metropolitan Immigrant Settlement Association (MISA), Halifax, NS

Mini Case Studies

• Association for New Canadians, St. John’s, Newfoundland and Labrador
• New Brunswick Multicultural Council, Fredericton, New Brunswick
• Holland College, Charlottetown, P.E.I
• Mohawk College of Applied Arts & Tech, Hamilton, ON
• Niagara College, Welland, ON
• Waterloo Region District School Board, Kitchener-Waterloo, ON
• La Cité Collégiale, Ottawa, ON
  Women’s Enterprise Skills Training of Windsor(WEST), Windsor, ON
• Keyano College, Fort MacMurray, AB
• Brooks Adult Community Centre, Brooks, AB
• Yukon College, Whitehorse, YK
• Fatima Soares and Associates, Winnipeg, MB
• Central Vancouver Island Multicultural Society, Nanaimo, BC
• Red River College, Winkler, MB
• Penticton, Penticton, BC
• Red River College, Steinbach, MB
• Camosun College, Victoria, BC
• Centre for Skills Development and Training, Burlington, ON
• Chilliwack Community Services, Chilliwack, BC
• Community Futures Corporation of North Fraser Mission, Abbotsford, BC

______________

[40] Although there were twelve case studies, there were only eleven focus groups because one of the case studies was of a development project. Also, because of inability to attract participants, one group included only one participant.

[41] Tools were not available from two SPOs.

• Previous
• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9 (current)
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• Next