Evaluation of the Sectoral Initiatives Program

Budget 2021 and program transformation

In support of the post-pandemic recovery, Budget 2021 proposed to provide $960 million over 3 years, beginning in fiscal year 2021 to 2022, to transform the SIP into the Sectoral Workforce Solutions Program (SWSP). The SWSP has an expanded scope for larger-scale projects that support a wider range of sector-focused activities including training and upskilling for workers and developing solutions to employers’ workforce challenges.

Consult Annex A for a brief description of the new program

Evaluation focus

The evaluation was carried out in compliance with Financial Administration Act and Treasury Board’s Policy on Results. This evaluation assesses SIP’s contribution to achieving its expected outcomes (as laid out in the program’s logic model, see Annex C) and covers the 5-year period from April 2017 to March 2022. It builds on the previous evaluation completed in 2018 and aims to support the program in identifying lessons learned from the delivery and considerations to inform and improve future programming. The findings and considerations of this evaluation are based on the analysis of multiple lines of evidence.

A summary of the last evaluation of the program can be found in Annex B.

Key findings

There are 7 main findings from the evaluation:

1. the uptake of SIP-supported products and tools was above the overall target of 100,000 users in fiscal year 2017 to 2018. However, consistent tracking of this indicator ceased in fiscal year 2018 to 2019
2. evidence from the survey, case studies and project documentation indicate that SIP-supported products and tools helped identify, forecast and address human resources and skills issues across various economic sectors. However, the extent to which these resources met employers’ needs is only partially assessed
3. in addressing the ongoing need for more granular Canadian labour market intelligence, SIP complements and aligns with other available labour market programming
4. GBA+ considerations are integrated in the requirements and program selection criteria. While there is evidence from interviews and case studies stating the program’s success in improving some equity considerations, these outcomes can be enhanced
5. pilot projects are useful for both employers and job seekers. The program contributes to identification, implementation, and expansion of innovative projects and practices
6. funding recipients used various mechanisms to communicate, disseminate and promote the usage of the products and tools. Though not the program’s role, funding recipients expressed their willingness for the program to engage more to facilitate and enhance the adoption and use of the products and tools
7. the reporting requirements were streamlined for funding recipients. Nevertheless, there is a need for greater clarity on the reporting expectations and guidance on how to meet the requirements

Considerations

Based on these findings, the evaluation provides the following 3 considerations to the Department:

1. a better understanding of the overall program level outcomes could be achieved by improving the collection and tracking of end-users’ data
2. greater clarity on reporting expectations and technical support and guidance to funding recipients in meeting reporting requirements have potential to improve the reporting process, and
3. the program may enhance generating equitable outcomes by encouraging funding recipients to develop more specific products and tools in addressing the needs of underrepresented groups, sub-sectors and sub-regions

Program outcomes

The program aimed to achieve the following expected outcomes as reflected in the 2019 logic model.

• Short-term outcomes mostly focus on the use and adoption of SIP-supported products and tools by employers and other end-users and the replication of innovative approaches
• Medium-term outcomes expect the use and adoption of SIP-supported products and tools to improve employers’ decision making and employees’ employability and mobility, while the innovative approaches are to be expanded to more employers and end-users
• Long-term outcomes relate to the products’ contributions to the efficiency of the labour market and adaptability of the labour force in targeted sectors

Program budget

The budget for SIP projects combines 2 sources of funds: funding from the Consolidated Revenue Fund and funding from Part II of the Employment Insurance Act.

Figure 1 illustrates the annual budget for the projects under SIP from fiscal year 2017 to 2018 to fiscal year 2021 to 2022, apportioned from each funding envelope. On average over the period covered, $28.23 million was planned annually for the program, $20.07 million from Part II of the Employment Insurance and $8.16 million on average from the Consolidated Revenue Fund. In the most recent year under the SIP, 2020 to 2021, the total amount through the 2 envelops was $25.13 million.

Source: Program Data

Notes: In fiscal year 2017 to 2018, planned program spending represents total SIP Employment Insurance and Consolidated Revenue Fund budgets combined at the beginning of the year before transferring. In fiscal year 2018 to 2019 and afterwards, the planned figures include transfers out to other programs. Available budgets change from year to year due to transfers. In fiscal year 2021 to 2022, projects are transited to the Sectoral Workforce Solutions Program.

Figure 2 illustrates the actual vs. planned program spending from fiscal year 2017 to 2018 to fiscal year 2021 to 2022. On average, the program spent $20.9 million annually over the 4-year period from fiscal year 2017 to 2018 to fiscal year 2020 to 2021. In 2021 to 2022, projects were transited to the Sectoral Workforce Solutions Program.

Source: Program Data

Note:The program reported few active projects during fiscal years 2017 to 2018 and 2021 to 2022 while new calls for proposals were launched, explaining the higher variance between actual and planned spending in those years. In fiscal year 2021 to 2022, projects are transited to the Sectoral Workforce Solutions Program.

Funded projects and outputs

From fiscal year 2017 to 2018 to fiscal year 2020 to 2021, 90 projects received funding through SIP, out of which 70 were new projects. The new projects were wide ranging, mostly covering specific sectors of the economy or supporting underrepresented groups (Figure 3).

Note: Funded projects supported multiple economic sectors and underrepresented groups. Sectoral and underrepresented group categories are not mutually exclusive, and some projects supported additional underrepresented groups (for example, women).

The scope and the number of projects selected through each call for proposals cycle depended on the evolving labour market needs. The program noted that the 2017 call for proposals focused on the changing nature of work, and gender inclusive growth. A total of 35 multi-year proposals received $89 million in funding, starting in fiscal year 2018 to 2019.

The 2021 call for proposals focused on sector-based solutions to address the workforce challenges and needs of Canadian workers and employers; and to address specific workforce needs in support of economic recovery for a specific sector or across sectors. A total of 32 multi-year proposals received $101 million in funding, starting in fiscal year 2021 to 2022.

Note: From January 22 to March 4, 2021, the SIP launched a call for proposals that funded a total of 32 projects. This included 10 projects that focus on creating employment and career-building opportunities for persons with disabilities, and 22 projects that would help the tourism and hospitality sector, one of the sectors hardest hit by the COVID-19 pandemic.

Projects funded through SIP developed and/or updated a large number of products and tools from fiscal year 2017 to 2018 to fiscal year 2020 to 2021, mostly supporting the provision of labour market information for various industries (Figure 4).

Note: These figures only present some key outputs. Numerous other products were developed each year. These included: competency profiles, detailed job descriptions, detailed interview guides, curriculum mapping tools, and human resources toolkits and videos.

In addition to these resources, 5 reports documenting best practices and lessons learned from pilot projects were produced and shared over the period.

Evaluation questions

1. In addition to funding, what is the program’s contribution to the identification, replication and/or expansion of creative workforce solutions? (Program Chain 3)
2. To what extent have employers adopted or used workforce solutions, labour market intelligence and tools, delivered as a result of sector-based program funding? (Program Chain 1 and 2)
   • 2.1. How did the program contribute to the adoption and use of these resources and tools
3. To what extent have the skills and employment needs of employers been met as a result of obtaining, adopting and/or using workforce solutions, labour market intelligence and tools?

Evaluation scope and limitations

Following are some highlighted scopes and limitations of this evaluation.

• The evaluation did not review projects funded under the Sectoral Workforce Solutions Program introduced in fiscal year 2021 to 2022
• One of the main data sources for the analysis is the Common Grants and Contributions System. Out of the 90 funded projects from fiscal year 2017 to 2018 to fiscal year 2020 to 2021, data were available for only 62 projects in the Common Grants and Contribution System during the fieldwork phase of the evaluation; among these, 44 projects concluded during the evaluation period. The project data analysis is based on this sample
• The information on the uptake of SIP-supported products and tools could not be aggregated across projects. As a result, the evaluation could not fully assess the program level outcome
• The Department (ESDC) does not have access to contact information of employers. The evaluation had to rely on funding recipients to send the survey link to their employers’ lists. At the end, the online Survey covered a limited number of employers. Given those limitations, the survey data are presented in a descriptive manner and were triangulated with other lines of evidence, as applicable

Details about the lines of evidence and their limitations can be found in Annex D: Lines of Evidence.

Finding 1: The uptake of SIP-supported products and tools was above the overall target of 100,000 users in fiscal year 2017 to 2018. However, consistent tracking of this indicator ceased in fiscal year 2018 to 2019

In fiscal years 2017 to 2018 and 2018 to 2019, output and outcome data were collected through the Recipient Survey of SIP Performance Indicators.

Survey results for the fiscal year 2017 to 2018 indicate that:

• 103,337 stakeholders used SIP-supported Labour Market Information (LMI) products and tools
• the count of users of the National Occupational Standards (NOS) is 15,429 for that fiscal year
• 462 employees were certified under SIP-supported certifications and 7 programs of study accredited

Over 4,000 large-employers and 7,350 small and medium-sized firms were reported using SIP-supported LMI products in fiscal year 2017 to 2018. A total of 624 large firms and 999 small and medium-sized firms used the NOS in the fiscal year.

The survey was discontinued in fiscal year 2018 to 2019 to ease data collection requirements on funding recipients; during that fiscal year data were not collected consistently across projects to permit aggregation.

Source: ESDC Recipient Survey of SIP Performance Indicators

In fiscal years 2019 to 2020 and 2020 to 2021, the program collected output and outcome data through quarterly activity reports submitted by funding recipients populated in the Common System for Grants and Contributions. The reporting of output and outcome data was conducted at the projects’ level with a certain flexibility to account for the variability in project objectives, deliverables and outcomes. The information collected through quarterly reports, reorganized into results trackers, was used to inform program design, implementation and related activities but led to inconsistencies in aggregating some project outcomes such as the numbers of end-users. Due to this reason, the aggregated numbers of the program uptake across different products and tools could not be calculated for fiscal years 2019 to 2020 and 2020 to 2021.

Finding 2: Evidence from the survey, case studies and project documentation indicates that SIP-supported products and tools helped identify, forecast and address human resources and skills issues across various economic sectors. However, the extent to which these resources met employers’ needs is only partially assessed

The review of project documentation indicated that the usage and adoption of SIP-supported products and tools contribute to a number of the outcomes sought by the program. In particular, the products and tools helped address employers’ needs to:

• locate, onboard, recruit and retain workers, and/or
• make informed workforce planning decisions

A majority of key informants, including funding recipients, program officials and other stakeholders, (n=18), provided anecdotal examples of ways in which SIP-supported products and tools contributed in helping employers meet their needs. The key themes from these examples can be summarized as follows:

• Labour market information (LMI) reports and other products are helping employers in human resource/workforce planning, development of job descriptions/postings and recruitment and retention strategies
• LMI data is informing human resource and compensation policy development work, and
• National occupational standards and certification programs are helping to align training and skill development programs with changing industry needs

Case studies conducted for 3 projects also showed that SIP-supported tools and products are contributing to a number of program outcomes.

Given the unique national and regional coverage and efforts in aligning the education/training with the employers’ demand, the Construction and Maintenance Industry LMI Project helped with informed decision making for construction businesses, and related unions, colleges and apprentices’ organizations. A 2020 survey of key end-users beneficiaries (n=1,600) indicated that more than a quarter of stakeholders (29%) used the LMI products to inform staffing, recruitment and human resource decisions.

Situational analyses conducted for the project Building Stronger Indigenous Labour Market Program and Services Links with Employers helped:

• identify service issues and gaps in order to improve existing labour market program service delivery
• improve understanding of Indigenous engagement in the regions and sectors targeted, and
• support development of a business case for Indigenous involvement in the targeted regions/sectors

The project contributed to improving the understanding of labour market issues.

The Recognizing Environmental Expertise project supported employers’ need to:

• retain employees and improve workforce employability through the certification process that ECO Canada promotes and subsidizes
• recruit employees with the appropriate technical and soft skills through the Professional Development Training Framework and the Post-secondary Accreditation Program, and
• play a significant part in identifying skills requirements and gaps through the accredited program and certification webinars and trainings

Overall, the Recognizing Environmental Expertise project helped inform decision making and improve labour market issues.

For the purposes of this evaluation, an online survey of employers was conducted in 2022. Surveyed employers consisted of a non-random sample (n=91). Responses suggest that the majority of employers who used SIP-supported products and curriculum/trainings were able to address their human resources needs and improved various workers’ employment outcomes. The results show that:

• most of the surveyed employers using LMI reports and/or forecasting systems (90%, n=28, out of 31 survey respondents using these products) agreed that the products helped “make more informed workforce planning decisions”
• the majority of employers using LMI reports and/or forecasting systems (n=17 or 55%) indicated that the products helped “locate and recruit workers”, while 52% reported the products helped them to “onboard and retain workers” and “recruit workers from equity-deserving groups”
• as shown in Figure 6, the majority of surveyed employers who used curriculum, training and certification programs in their organization were of the view that these programs improved their workers’ outcomes (skills, employability, productivity, adaptability and mobility)

There were, however, challenges in assessing the extent to which workforce solutions, labour market intelligence products and other tools met employers’ needs. Programs officials and funding recipients pointed to difficulties in tracking the usage data of SIP-supported products and tools by employers and the early stages of some projects. While the access to products can be tracked through the number of downloads or the number of print materials disseminated, it is more difficult to assess whether and to what extent those who have accessed the products are using them.

Finding 3: In addressing the ongoing need for more granular Canadian labour market intelligence, the SIP complements and aligns with other available labour market programming

A review of the Canadian labour market literature suggests that there is an ongoing need for improvements to Canadian labour market intelligence, specially for Canadian small and medium enterprises. The availability of sector and region specific data are also scarce (Labour Market Information Council, 2018, 2019). From a labour market intelligence perspective, the literature also points to challenges associated with skills forecasting and informing skills alignment with employers’ requirements (Bonen and Loree, 2021; Labour Market Information Council, 2018; Wilson et al., 2016).

Experts interviewed (n=5) for this evaluation indicated similar gaps and believed that SIP complements and aligns with other available labour market programming to address these challenges. Specifically, experts identified that there is a need to:

• provide more granular, real-time labour market data for rapidly evolving sectors (for example, IT and tourism)
• ensure coordination among sector councils to promote the use of consistent terminology across LMI resources and tools
• develop labour market information for public employment (for example, teachers, hospital workers, government employees)

Finding 4: GBA+ considerations are integrated in the requirements and program selection criteria. While there is evidence from interviews and case studies stating the program’s success in improving some equity considerations, these outcomes can be enhanced

GBA+ Considerations are integrated in SIP design

The 2017 call for proposals introduced a number of GBA+ considerations. With respect to the project governance, the call for proposals made several references to women and/or gender. The types of issues considered in LMI research, dissemination, and results tracking also referred to gender and/or women. Each of the projects was required to establish a steering committee, requiring governance structure with a diverse representation. LMI projects had to produce research and analytical reports that, among other things, consider labour supply demographics and the availability of workers in underrepresented groups.

The GBA+ considerations are having impact

There is evidence from multiple lines that SIP-funded projects are having success in targeting some of the equity-deserving groups.

The Building Stronger Indigenous Labour Market Program and Services Links with Employers project was successful in helping improve Indigenous engagement in the target sectors. However, more efforts are required to develop sustainable, long-term relationships. Further actions and priorities to consider include: establishing more sector-based indigenous organizations where appropriate, as well as ongoing working groups or committees with stakeholders from governments, industry and Indigenous communities.

The Recognizing Environment Expertise project contributed to increasing the number of certified women and Indigenous people to the pool of candidates. The program indicated that the project exceeded its goal of adding certified women members to the pool (53%, compared to the target of 50%). The new membership levels targeting youth and students were reported to be very successful in reaching these demographics and helping support the new professionals transitioning into the sector. There is, however, still a need to raise the number of Indigenous candidates to the pool. The proportion of self identified Indigenous candidates among new certified members was 3%, lower than the project target of 6%. The funding recipient noted that this might happen in part due to lower levels of participation of Indigenous workers in post-secondary education.

The reskilling program developed through the Upskilling Mid-Career Workers Impacted by Automation Project helped reducing the hiring bias against disadvantaged groups including racialized minorities, women and older workers. The project documentation highlighted that target interventions such as specialized coaching and Workplace Integrated Skills Experience programs were effective in mitigating some of the hiring bias for racialized minorities, women and workers over 50 years of age; almost all job-seekers who benefited from tailored interventions received full-time positions through the reskilling program. The funding recipient highlighted that further actions to consider include directing more funding towards strategies that promote inclusive program outcomes and are specifically designed to address the challenges of mid-career workers across various demographics.

Finding 5: Pilot projects are useful for both employers and job seekers. The program contributes to identification, implementation, and expansion of innovative projects and practices

Between fiscal year 2017 to 2018 and fiscal year to 2021 to 2022, the program supported 3 pilot projects. Among the few employers who responded to the 2022 online survey, some took part in these pilot projects.

• 5 employers participated in at least one of the pilot projects. Most of these employers (n=4) agreed that pilot project(s) improved the capacity and/or ability of their organization to address labour market issues. One employer noted that the pilot project allowed its company to attract and hire workers which helped to rebuild the organization’s capacity during the pandemic
• 3 employers adopted or expanded a pilot project — these employers indicated that the pilots addressed labour market issues for visible minorities

Below are some specific examples of benefits for the 3 pilot projects which ran during the evaluation period. The evidence was gathered through the review of project reports.

Examples of benefits for the Building Canada’s Work Integrated Learning Recruitment Network project:

• employers were able to post work integrated learning placement opportunities and connect with students from over 65 institutions across Canada
• increased real-time LMI provided more accurate assessment of the economic needs of employers, communities, regions and provinces

Examples of benefits for the Upskilling Mid-Career Workers Impacted by Automation project:

• a validated approach to mid-career job transition that can be replicated several times and in different regions
• 32 employers participated across 3 cohorts; 12 employers participated in more than 1 cohort
• 96% of the participating graduates received job offers

Examples of benefits for the Future of Skills in Canada's Labour Market pilot project:

• the Forecast of Canadian Occupational Growth provided growth and decline projections for each of 485 Canadian occupational unit groups
• an interactive web application enabled users to filter forecast results by occupations, skills, geography, and demographic characteristics

Multiple lines of evidence suggested that SIP-supported pilot projects were successful in exploring innovative ideas and promising practices in addition to helping employers and job seekers. A majority of program officials (n=5) also noted some instances where pilot projects evolved into ongoing programming. One of the examples includes Palette Skills pilot project (see the text box on the Upskilling mid-career workers impacted by automation projecton project).

Program officials (n=4) also highlighted other mechanisms through which the program contributed to identification, implementation, and expansion of promising or innovative practices.

These include:

• designing calls for proposal processes in a way that ensures the program is open to supporting new ideas
• maintaining ongoing interactions with labour stakeholders who keep the program informed about what is working and help program representatives identify the best practices, and
• making recommendations for funding increases when SIP-supported projects demonstrate success and opportunities for expansion

Finding 6: Funding recipients used various mechanisms to communicate, disseminate and promote the usage of the products and tools. Though not the program’s role, funding recipients expressed their willingness for the program to engage more to facilitate and enhance the adoption and use of the products and tools

The 2018 evaluation of the Sectorial Initiatives Program recommended that the program explore ways to encourage funding recipients to strengthen product outreach and dissemination. The document review, interviews and case studies showed that funding recipients used various mechanisms to communicate, disseminate and promote the products and tools developed through the program over the evaluation period. These mechanisms included:

• promoting products and tools directly to end-users (for example, through employer consultations; direct promotion of educational resources to teachers/schools; hosting of career fairs, webinars)
• including information about products and tools in organizational newsletters, “email blasts” or other communications that are sent out regularly to target audiences
• maintaining relationships with employers, industry associations, training institutions, membership lists, etc. and promoting the resources through these connections
• producing advertisements for social media and other forms of media to raise awareness of the products and tools and/or entice employers and employees to provide input (for example, by completing employee/employer surveys)
• uploading LMI reports and other products onto their organization’s website
• sharing information about the products and tools at various events including conferences, and trade shows, and
• developing personalized messages/plain-language summaries, or other tailored products for target audiences to encourage use of finalized resources and tools

The following examples of outreach activities are sourced from the review of project reports:

• the Canadian Agricultural Human Resource Council showcased and disseminated the employer tools to over 2,000 stakeholders through their winter and spring newsletters
• Energy Safety Canada used webinars to share findings from their research with a diverse group of stakeholders
• the Forest Products Association of Canada promoted their updated LMI tool at various events, overall, reached 1.2 million people over a span of 5 months through advertising
• TECHNATION Canada organized conferences to disseminate findings

Interviewed funding recipients offered a number of suggestions to the program to enhance its contribution for the adoption and use of the products and tools. Their suggestions/views are summarized as follows:

• the program could take an active role in promoting the products and tools developed through SIP-funded projects (via social media or ESDC’s website), and undertaking more frequent, active endorsement of SIP-supported products
• the program could do more to ensure sustained funding for ongoing project activities (for example, website hosting/maintenance), as well as promotional and project evaluation activities
• in order to facilitate the sharing of resources and tools, the program could do more to connect funding recipients with other relevant stakeholders, including representatives of other federal and provincial/territorial government departments with relevant expertise, human resources councils and other funding recipients

Finding 7: The reporting requirements were streamlined for funding recipients. Nevertheless, there is a need for greater clarity on the reporting expectations and guidance on how to meet the requirements

The review of the program documentation indicated that reporting requirements were simplified and streamlined for funding recipients.

The annual Recipient Survey of SIP performance indicators was discontinued in 2018 to 2019 to ease data collection requirements on funding recipients. Funding recipients previously had to collect comprehensive data and provide information on an annual basis on the users of their products.

Starting in fiscal year 2019 to 2020, the program introduced more flexibility in the reporting of output and outcome data at the projects’ level to account for the variability in project objectives, deliverables and outcomes. Funding recipients regularly report on project activities and project-specific outcomes through submission of quarterly fiscal and activity reports and a final report at the outset of the project. Program officials are responsible for rolling up the information from the activity reports in results trackers for reporting purposes.

Interviews gave insights on the perceptions of stakeholders regarding the reporting requirements. The majority of funding recipients (n=12) noted that the reporting requirements for their SIP-funded project(s) have been fair, or manageable. However, many funding recipients (n=8), including a few who reported that reporting requirements were fair or appropriate, stated that the reporting is heavy/burdensome, challenging, and/or restrictive in various ways.

A majority of funding recipients (n=10) as well as some program officials (n=3) offered suggestions for improving the reporting process for funding recipients. Key informants identified a need for greater clarity and consistency in reporting requirements. Specific suggestions included:

• providing additional guidance to funding recipients at project outset on how to meet reporting expectations
• providing feedback on reporting so that funding recipients can be confident they are meeting the reporting needs
• adopting standardized terminology and clarifying expectations for reporting related to the involvement of equity-deserving groups in SIP-funded projects, and
• ensuring that all ESDC staff involved in SIP projects have a common understanding of the reporting process and common expectations for reporting

Lessons learned: Covid context

Interviewed funding recipients noted that the program provided support in a number of instances. When needed, the program assisted the recipients in project planning and reporting, managed budgets or funding changes and responded to questions in a timely manner. Almost all funding recipients affected by the pandemic noted that the program assisted their organizations in overcoming challenges related to project delivery by:

• allowing flexibility with project timelines
• helping to negotiate funding and agreement changes, and
• responding quickly to recipients requests/questions

Below are some specific examples of support across 3 SIP-funded projects. The evidence was gathered through the case studies.

For the Recognizing Environmental Expertise project, program officials offered the right amount of flexibility, especially during COVID-19, in managing the budget to best serve the needs of the project and achieve the objectives. The support for the proposed project amendment, including associated funds and extended timeline, enabled the project to respond to emerging needs and priorities. The program also shared valued expertise and contacts (within other sectors).

For the Upskilling Mid-Career Workers Impacted by Automation project, the recipient organization was provided additional time for the third cohort of the project to find employment given pandemic disruption. The program was flexible with the submission dates of the final report

For the Construction and Maintenance Industry Labour Market Information project, the funding recipient noted that the program understood the impact that the pandemic was having on the construction sector and provided extensions to ensure the activities could be fully carried out.

Document, data and literature review

The document and data review included:

• 31 internal documents (for example, Departmental Performance Reports)
• 18 external documents (for example, organizational reports)
• administrative data collected by the program through the Recipient Survey (2017 to 2018 to 2018 to -2019), and sector-specific results trackers (2019 to 2020 to 2021 to 2022), and
• close-out reports for SIP-funded projects

The literature review gathered information primarily from grey literature published after 2017, including program evaluations and reports (N=27), as well as from peer-reviewed journals (N=1).

Key limitations

• Internal documents focused mostly on the program’s operations and desired outcomes. The documents provided little information on the actual results that funded projects achieved. As a result, few documents were relevant to address the evaluation questions
• The literature review was limited in its ability to inform some evaluation questions and indicators due to a limited number of publications emphasizing employer perspectives, especially in the Canadian context, on demand-driven approaches to overcome labour and skill shortages

Key informant interviews

A total of 32 interviews were completed. Interviewees are distributed as follows:

• 8 ESDC program officials from the Skills and Employment Branch (n = 4) and the Program Operations Branch (n = 4) who worked on the design and delivery of the program
• 17 funding recipients from projects funded under each program chain and from different sectors
• 5 experts to provide insight into the emerging theories and trends in the area of sectoral labour market information or competency-based skill development
• 2 other stakeholders to identify potential overlap or complementarity between the program and other initiatives, and to identify best practices

Challenges

The initial goal was to complete approximately 35 key informant interviews with the designated key informant groups. The target was not reached for funding recipients and other stakeholders. A number of funding recipients were unresponsive to multiple requests for an interview. Many of the other stakeholders identified were not familiar with the program and declined participation.

Analysis scale

The following scale was used for analysis of interview results:

• few is used when less than 10% of participants have responded with similar answers. The sentiment of the response was articulated by these participants but not by other participants
• several is used when fewer than 20% of the participants responded with similar answers
• some is used when more than 20% but significantly fewer than 50% of participants responded with similar answers
• many is used when nearly 50% of participants responded with similar answers
• a majority is used when more than 50% but fewer than 75% of the participants responded with similar answers
• most is used when more than 75% of the participants responded with similar answers
• vast majority is used when nearly all participants responded with similar answers, but several had differing views
• unanimous or almost all are used when all participants gave similar answers or when the vast majority of participants gave similar answers and the remaining few declined to comment on the issue in question

Case studies

Case studies were conducted for 4 selected projects active during the evaluation period and covered different program chains and sectors. The selected projects were:

• Construction and Maintenance Industry Labour Market Information Program (BuildForce Canada)
• Recognizing Environmental Expertise: Professional Certification and Program Accreditation (ECO Canada)
• Upskilling Mid-Career Workers Impacted by Automation (Palette Skills Inc.)
• Building Stronger Indigenous Labour Market Programs and Service Links with Employers (Indigenous Works)

Case studies included interviews and a review of internal and external documents. Overall, 17 individuals were interviewed across the 4 case studies and included program officials, funding recipients, end-users, and stakeholders having participated in the project.

Challenges

Other than the funding recipients, contact information of individuals that participated in the case study projects was not available. Therefore, the evaluation had to rely on funding recipients to obtain contact information of program officials, end-users and other stakeholders involved during the projects designing and implementation.

Survey of employers

The online survey of employers was developed by ESDC and PRA Inc. to gather information on the usage and uptake of SIP-supported products and tools by employers, and to address existing data gaps given that the program discontinued the annual survey of funding recipients after fiscal year 2018 to 2019. The finalized survey instrument is comprised of 38 questions, with some sub-questions, of which 22 were open-ended. The survey was administered online from November 3, 2022, to December 5, 2022, and took approximately 10 to 20 minutes for respondents to complete. Overall, 91 employers completed the online survey.

Key limitations and challenges

Not having access to employer lists/contact information

ESDC does not have access to contact information of employers. The evaluation had to rely on funding recipients to send the survey link to their employer lists.

Not having access to the employer lists posed a number of issues:

• the survey link was probably not sent to all potential employers that benefitted from the products/tools developed through SIP-funded projects or to a representative sample of these employers
• there is no sense of the response rate as the total number of employers that could have been sent the survey link is not known

Given the limited number of responses to the survey and the above limitations, the survey information was analyzed and triangulated with the other lines of evidence.

Selection bias

Due to the recruitment approach and the lack of sampling frame, the sample is not representative of the entire population of employers using the products and tools developed through SIP’s funding and support. Nevertheless, the distributions across sectors, regions and firm size indicate that the survey covered a somewhat diverse set of employers. The survey remains informative and will be analyzed qualitatively considering the number of open-ended questions (22 out of 38).

Not being able to directly identify the products/tools employers are referring to in the survey

Given the long list of products and tools developed through SIP-funded projects during the evaluation period and employers not being expected to know the exact names of the products/tools and/or project names, it was not possible on the survey to ensure that employers were referring only to the products and tools developed through SIP-funded projects. However, the request to complete the survey coming from the funding recipients may have signalled to employers which products and tools are being referred to when completing the survey. The results of the survey speak more generally to the usefulness of the types of products and tools developed by SIP-funded projects. To mitigate this limitation, the evidence from case studies and key informant interviews is used to contextualize and corroborate the results from the survey.