Evaluation of the Opportunities Fund for Persons with Disabilities – Phase I
On this page
- Executive summary
- Management response and action plan
- 1. Introduction
- 2. Methodology
- 3. Key findings
- 4. Recommendations
- Appendix A – Phase I evaluation questions
- Appendix B – Technical studies conducted in support of phase I of the evaluation
- Appendix C – Opportunities Fund logic model, 2016
Evaluation of the Opportunities Fund for Persons with Disabilities – Phase I [PDF - 944 KB]
Request other formats online or call 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105. Large print, braille, audio cassette, audio CD, e-text diskette, e-text CD and DAISY are available on demand.
This report summarizes the findings from Phase I of the evaluation of the Opportunities Fund for Persons with Disabilities. Launched in 1997, the Program was designed to fill a gap in federal supports for persons with disabilities who are not eligible for the Employment Insurance program or training support under Employment Insurance Act Part II. More specifically, the Program funds initiatives designed to assist persons with disabilities to prepare for, obtain and maintain employment or self-employment.
In order to further align the Opportunities Fund for Persons with Disabilities Program with labour market needs and demands, reforms were announced in Budgets 2012 to 2014. These included the following enhancements:
- The provision of more demand-driven training solutions
- The involvement of employers in the design and delivery of projects
- Increases in both the number of youth taking part in the Program and the number of participants acquiring work experience in small and medium enterprises
The evaluation focused on the need for the Program, its performance, and the implementation and delivery of reforms. The evaluation also looked at the extent to which Budget 2014 initiatives (CommunityWorks/Worktopia and Ready, Willing and Able) have been implemented as designed.Footnote 1 The other projects reviewed all started between April 2015 and December 2016, and resulted from the 2014 Call for Proposals. The latter reflected the Program's new terms and conditions.
This first phase of a 2-phase evaluation made use of 4 lines of evidence: data assessment; survey of participants in single year projects; document review; and key informant interviews.Footnote 2 Building on preliminary results from Phase I, Phase II of the evaluation will concentrate on participant outcomes over a longer period of time (that is, beyond the 12 month post-program period) and delve further into the employers' experience of the Program. It is scheduled for completion in March 2020.
The evaluation approach was developed and presented to the Performance Measurement and Evaluation Committee in May 2015. The evaluation is conducted in accordance with the Financial Administration Act (required by November 2019) and Treasury Board obligations. It is intended to provide timely evidence to support program management decision-making.Footnote 3
Program relevance and alignment with priorities
The Program is relevant and meets the needs of persons with disabilities by filling a gap in employment services for those who are not eligible for Employment Insurance benefits and/or cannot access training support under Employment Insurance Act Part II.
The Opportunities Fund continues to align with the priorities, roles and responsibilities of Employment and Social Development Canada and the federal government. This is notably reflected in the Government's engagement towards the elimination of barriers to employment for persons with disabilities and the recent tabling of new federal accessibility legislation.Footnote 4
There is evidence that the Opportunities Fund could be better known among service providers and potential clients, including employers.
Phase I of the evaluation confirmed that, through the Program, persons with disabilities are given the opportunity to acquire or develop skills, and/or to obtain paid employment experience or assistance in becoming self-employed. However, evidence from key informant interviews revealed that the availability of support for workplace accommodations may not be well known among employers. It also revealed that meeting the needs of participants can sometimes require additional resources that have to be obtained through other programs or organizations.
The percentage of participants who were employed, self-employed or returned to school, up to 12 months after their last program intervention, increased progressively from 42% in fiscal year 2012 to 2013 to 55% in fiscal year 2016 to 2017. On average, during this period, the Program served 4,316 persons with disabilities every year; of these, 41% (1,791) were employed or self-employed, and 6% (253) had returned to school, after the end of an intervention. On average, 53% (2,272) of participants were neither employed, self-employed nor returned to school after the Program.Footnote 5
The post program employment rate (41% based on a 5 year average) should be considered in the context of the population of persons with disabilities, whose overall employment rate is much lower than that of persons without disabilities (for example, 49% versus 79% as of 2011, among Canadians aged 25 to 64).Footnote 6 It should also be considered that the Opportunities Fund serves individuals who are far removed from the labour market and more likely to face disability related barriers to employment, such as health issues that arise in the course of the Program and the inability to commute to and from work.Footnote 7 Further, having or expecting to have an employment income that is not high enough to compensate for a loss incurred in social assistance benefitsFootnote 8 may undermine some participants' motivation to enter or remain in the labour market, throughout the Program.
Phase II of the evaluation will examine how participants' labour market outcomes evolve in the medium- to long-term, beyond the end of the reporting period. It will also elaborate on participants' results using a gender-based analysis plus lens. This includes, for example, intersections between gender, disability, Indigenous groups, and region.Footnote 9
Implementation of reforms
Evaluation findings indicate that more participants acquire employment experience in small and medium-sized businesses through the reformed Program, in which all projects involving participants must comprise a Wage Subsidy or Self-Employment activities.Footnote 10 The evaluation also noted increases in the number of youth aged 15 to 30 among participants.
However, the evaluation could not find evidence that more demand-driven training solutions are being provided through the reformed Program, nor that it is more responsive to labour market needs. As well, it was not possible to confirm that labour market information has helped funding recipients to better align training activities with the needs of the labour market. Since the Program was reformed, training activities supported through Skills for Employment interventions cannot last more than 6 months.Footnote 11 This was not viewed by a few recipient organizations as being long enough to allow for participants to acquire or solidify the technical or career specific knowledge and skills in demand in the labour market.
Further, evaluation findings indicate that employers are generally not involved in the design of projects. Instead, they contribute to the delivery of projects by covering part of participants' wages and providing on-the-job training and supervision. In fact, due to changes in the eligibility criteria for funding, it has become more difficult for employers to design and deliver stand-alone Wage Subsidy projects, as this would require hiring at least 8 participants at once.
Finally, evidence was not sufficient to determine if, and to what extent, social innovation (that is, social enterprise, social partnerships or pay-for-performance) has helped maximize the impacts of the Program.
The Ready, Willing and Able and CommunityWorks/Worktopia initiatives
While only preliminary results were available, input from key informants and the project documents reviewed indicate that these 2 budget initiatives faced implementation challenges, which required adjustments to be made.
For example, supply-side issues faced during the delivery of the Ready, Willing and Able initiative led to the introduction of autism outreach coordinators. Their task was to identify job seekers with autism spectrum disorders who did not have a co-occurring intellectual disability. Therefore, the duration of the project was extended from 3 to 4 years to help meet the targets initially set in terms of participants served.
As well, in order to help increase completion rates at certain delivery sites of the CommunityWorks/Worktopia initiative, the recipient organization decided that all youth enrolled in the Employment Work stream of the Program should be independent for transportation and have good language skills.
A limited cost analysis revealed that the Program costs per client served, and per successful outcome were comparable to those of 2 other federal labour market programs serving disadvantaged groups. For example, between fiscal years 2012 to 2013 and 2016 to 2017, the average cost per client served under the Opportunities Fund for Persons with Disabilities was $6,961.Footnote 12
In comparison, the average cost per client served under the Aboriginal Skills and Employment Training Strategy was $7,389, and the average cost per client served under the Skills Link stream of the Youth Employment Strategy was $8,149.Footnote 13
On average, in the same period, the cost per Opportunities Fund client with successful outcome (employed, self-employed or returned to school) was $14,687. In comparison, the average cost per Aboriginal Skills and Employment Training Strategy client with successful outcome (employed or self-employed) was $12,886, and the average cost per Skills Link client with successful outcome (employed or self-employed) was $11,719.
Performance measurement and administrative data quality
A new participant information form that reflects some elements of the Program's reforms was implemented starting in fiscal year 2015 to 2016. However, limited information is still collected on employers' involvement with the Program, and on the results of Employer Awareness activities. As well, the Program's impact on participants' skills and employability is not accurately assessed, and very little information is collected to track the progress made through Self-Employment activities.
Based on the evidence collected and analyzed, the following recommendations have been developed:
- continue enhancing employers' awareness of the valuable contribution that persons with disabilities can make and of the various supports available to facilitate their hiring and workplace integration;Footnote 14
- explore the possibility of extending the availability of supports and follow-ups with participants;Footnote 15
- further clarify the concept of social innovation and its application within the context of the Program, and establish specific criteria for assessment and monitoring purposes;Footnote 16
- better link the Program's logic model to the reforms and new policy directions of the Program, and continue improving indicators used for performance measurement purposes.Footnote 17
Management response and action plan
Employment and Social Development Canada's Skills and Employment Branch and Program Operations Branch thank the Evaluation Division for conducting Phase I of the evaluation of the Opportunities Fund for Persons with Disabilities. The contributions of key informants, who participated in interviews during the course of the evaluation, including external stakeholders and program participants, are also acknowledged.
The Opportunities Fund provides participants with the opportunity to acquire or develop employment readiness skills and/or to obtain employment experience or assistance in becoming self-employed. The Program's participant success rates have improved steadily from fiscal year 2012 to 2013 to fiscal year 2016 to 2017. Opportunities Fund projects also support activities that improve employers' awareness of the contributions that persons with disabilities bring to the workplace. Findings indicate that the Program continues to be relevant and meet the needs of persons with disabilities, including those who may face multiple barriers. The evaluation nonetheless points to improvements that can be made.
Employment and Social Development Canada is already addressing some of the findings identified in this Phase I report. For example, the Opportunities Fund's 2018 Call for Proposals placed a greater emphasis on working with employers to better integrate their needs into project design and prepare workplaces for the inclusion of persons with disabilities.
Recommendations and follow-up actions
The evaluation outlined four recommendations, all of which the Program area supports:
Continue enhancing employers' awareness of the valuable contribution that persons with disabilities can make and of the various supports available to facilitate their hiring and workplace integration.
The Program's 2018 Call for Proposals included additional emphasis on employer-focused activities. Program funding can now be used for an expanded range of activities such as coaching, training and human resources support to help employers develop inclusive workplaces. These activities can be combined with participant-focused activities or provided as stand-alone supports to employers with a demonstrated commitment to hiring persons with disabilities.
In addition, $18 million in new funding was announced in June 2018 to fund additional activities and enhance Opportunities Fund programming to improve employer knowledge, policies and practices to recruit, employ and retain persons with disabilities in the workplace. The development of in-house recruitment strategies in conjunction with the development of matching services will serve to close this gap.
Under a renewed performance measurement framework, employer engagement activities undertaken by service providers will be tracked. This is expected to improve the ability to measure employer engagement activities and outcomes.
|Management action plan||Completion date|
|1.1 Enhanced Employer Activities introduced in 2018 Call for Proposals||March 2018|
|1.2 Include employer awareness activities in 2018 Call for Proposals' agreements||Fall 2018|
|1.3 Implement new Performance Measurement Strategy||Winter 2019|
Explore the possibility of extending the availability of supports and follow-ups with participants.
Employment and Social Development Canada agrees with this recommendation. Following the 2018 Call for Proposals, regional agreements will last for up to 3 years rather than for 1 year, which is expected to allow for increased follow-ups with participants. Employment and Social Development Canada will also examine the availability of supports in the context of other federal programs and initiatives.
|Management action plan||Completion date|
|2.1 Set longer duration agreements for regional projects||Fall 2018|
|2.2 Develop linkages with other federal initiatives, including the Accessible Canada Initiative, to foster accessible and inclusive workplaces||Ongoing|
|2.3 Follow-up analysis, policy adjustment, and system changes, where required||Fall 2019|
Further clarify the concept of social innovation and its application within the context of the Program and establish specific criteria for assessment and monitoring purposes.
Employment and Social Development Canada agrees with this recommendation. While the 2018 Call for Proposals maintained the same criteria for social innovation (that is, pay for performance, social enterprise, and social partnership) as the 2014 Call for Proposals, the departmental definitionFootnote 18 has evolved considerably over the last few years. Employment and Social Development Canada will explore the definition in the context of the Opportunities Fund to decide how it should be redefined and implemented for future funding investments.
|Management action plan||Completion date|
|3.1 Analyze the results from the completed social innovation projects from 2014 Call for Proposals||Fall 2018|
|3.2 Consider new approaches to enhance social innovation in the Program, which could include broadening the types of innovative projects supported by the Opportunities Fund||March 2019|
Better link the program's logic model to the reforms and new policy directions of the Program, and continue improving indicators used for performance measurement purposes.
Employment and Social Development Canada supports this recommendation. Work has already been initiated to both revise the Opportunities Fund's logic model as well as strengthen the performance measurement strategy for both participant-based and employer-based projects.
In advance of the launch of the 2018 Call for Proposals, a multi-branch Performance Measurement Working Group convened to develop indicators to reflect the new policy directions, in particular for the new "Working with Employers" stream. Work will continue through this group to further revise the Opportunities Fund's logic model to reflect the 2014 reforms and the new policy directions introduced with the 2018 Call for Proposal.
As new projects begin, service providers will be required to report in greater detail on engagement activities with employers. Work is also underway to explore how to improve the measurement of "enhanced employability" and innovation projects, and how this aligns with the measures and performance indicators developed under the new Provincial and Territorial Workforce Development Agreements.
|Management action plan||Completion date|
|4.1 Review/revise logic model||Summer 2018|
|4.2 Develop enhanced performance indicators||Summer to Fall 2018|
|4.3 Manual tracking of new performance measures||2018 to 2019|
|4.4 Automate performance measures||2019 to 2020|
This report summarizes key findings from Phase I of the evaluation of the Opportunities Fund for Persons with Disabilities. It includes information on the Program background, reforms and budget initiatives.
1.1 Program background
Due to limitations inherent to their condition, and to barriers and obstacles they may encounter, many persons with disabilities have to drop out of the labour market intermittently or permanently, while others can only work on a part time basis or following a fixed schedule.Footnote 19 Such challenges not only limit their ability to obtain and maintain employment, but also to work enough hours to qualify for Employment Insurance benefits during periods of unemployment.
In 2012, around 10% of Canadians aged 15 to 64 (2,338,200 people) had a disability.Footnote 20 Approximately 45% of these individuals (1,057,100 people) were employed. Among those who were not employed, about 411,600 reported having the potential to work.Footnote 21 Of these potential workers, around 25% (103,300 people) had never worked or had not worked in the previous five years, and may have potentially benefited from the Program.Footnote 22
Launched in 1997, the Opportunities Fund was designed to fill gaps in federal support for persons with disabilities who were not eligible for Employment Insurance or training under Employment Insurance Act Part II. To ensure that these individuals can access the assistance needed when trying to integrate or re-integrate into the labour market, the Program supports a range of projects. Most of these projects are designed and delivered by organizations known as Community Coordinators (also referred to as funding or contribution recipients). These organizations are external to the Department and act as intermediaries between participants and employers.
Projects designed and delivered by Community Coordinators must include Wage Subsidies and/or Self-Employment activities, which contribute to the mandatory employment experience component, as defined by the Program's Operational Directives. More specifically:Footnote 23
- Wage Subsidy activities provide a financial incentive to encourage employers to hire persons with disabilities whom they would not normally hire, in the form of a share of the wages paid to these participants. Such activities must be used to place participants in positions that are likely to lead to permanent jobs with the host employers, and to assist participants whose likelihood of finding employment will be markedly improved through the creation of employer-employee relationships;
- Self-Employment activities provide technical and consultative expertise to participants who have developed a business concept or are interested in self-employment. These activities must be used to provide participants with a variety of services that will improve their chances of success in developing and implementing their business plans.Footnote 24
In addition, projects can include interventions that fall into these 2 categories:Footnote 25
- Enhanced Employment Assistance Services support a mixture of pre-employment activities tailored to meet the needs of persons with disabilities and facilitate their integration into employment. They may include counseling and job placement services, access to labour market information, assistance in determining appropriate skills training, and more.
- Skills for Employment interventions enable participants to undertake short-duration training and develop skills, from basic to advanced. Such skills must relate to a specific job opportunity in a stable or expanding sector, and where there is a reasonable expectation of employment.Footnote 26
The Program also funds Employer Awareness activities, which are intended to raise the profile of persons with disabilities within the employer community and to highlight the capabilities and skills of workers with disabilities. These activities can be delivered through stand-alone agreements or with other program activities.
Projects can be regional or national in scope. Regional projects are managed by Service Canada regional offices, and national projects by Employment and Social Development Canada's National Grants and Contributions Delivery Centre. Although there are more regional projects than national projects, the latter receive, on average, a larger amount of funding.Footnote 27 This is mainly due to the fact that national projects are delivered in 3 or more provinces or territories across Canada and are usually of longer duration.Footnote 28
1.2 Program reforms
A number of changes have been made to the design of the Program since the last evaluation was carried out in 2014. These changes, outlined in Budgets 2012 and 2013, include:
- providing more demand-driven training solutions to persons with disabilities
- making the Program more responsive to labour market needs
- tnvolving employers and community organizations in the design and delivery of projects
- connecting more Canadians with disabilities to work experience in small and medium-sized businesses
- further ensuring employers are aware of the contribution persons with disabilities can make
These reforms have resulted in the following operational changes,Footnote 29 beginning in fiscal year 2015 to 2016:
- mandatory employment experience is now included in all projects except for stand-alone Employer Awareness projects. Accordingly, Enhanced Employment Assistance Services and Skills for Employment interventions can no longer be delivered unless they are followed by, or combined with, Wage Subsidies or Self-Employment activities;
- through the reformed Opportunities Fund, participants can only acquire subsidized employment experience in positions that are part of the host employer's normal operations and are likely to lead to a permanent job.Footnote 30 In all cases, the host employer has to cover part of the wages paid to participants.Footnote 31 These changes are consistent with observations brought to the attention of the Standing Committee on Human Resources, Skills and Social Development and the Status of Persons with Disabilities.Footnote 32 The Program has also established that at least 50% of the duration of all project activities must relate to the employment experience component by 2017 to 2018Footnote 33;
- under the reformed Program, Skills for Employment interventions (that is, short-duration training programs) can only be supported if they help participants develop skills "related to a specific employment opportunity in a stable or expanding sector, where there is a reasonable expectation of employment."Footnote 34 Before April 2015, this criterion was only considered "optimal", not mandatory.Footnote 35 Further, while Skills for Employment interventions could last up to 52 weeks or 78 weeks under specific circumstances,Footnote 36 their maximum duration is now 6 months or 26 weeks. Consequently, participants have a shorter timeframe to prepare for the labour market through the Program.
The following enhancements were also made to the Program:Footnote 37
- in order to help more youth with disabilities gain employment experience, the Program now allows employers to hire students with disabilities through Wage Subsidy agreements on a part-time basis (for up to 15 hours per week) during the school year.Footnote 38 The Program also established the objective that youth represent a quarter of all participants.Footnote 39 These enhancements are aligned with observations made by the 2013 Panel on Labour Market Opportunities for persons with disabilities, which underscored the need to provide opportunities for early work experience to youth with disabilities;Footnote 40
- to strengthen performance measurement and assess longer-term outcomes, the Program now measures – for multi-year projects only – the number of clients employed or self-employed, and the number of clients returned to school, 6 months and 12 months post-intervention, within the timeframe of the agreement.Footnote 41 Prior to the reforms, these results were assessed, at the latest, 24 weeks following the end of the last intervention. A new participant information form is also used to collect additional information on employers hiring persons with disabilities through Wage Subsidy agreements;
- under the reformed Program, applicants who include an alternative social innovation model as a framework for project activities do not have to leverage funding from sources other than the Department for their project to be approved. In the context of the Opportunities Fund, "social innovation" is encouraged to maximize the impact of federal spending, and can take one of the following forms: social partnerships, pay-for-performance and social enterprise.Footnote 42
Finally, all projects except stand-alone Employer Awareness projects now have to include at least 8 participants. Consequently, employers can only design and deliver stand-alone Wage Subsidy projects if they hire at least 8 eligible participants simultaneously. In comparison, prior to the reforms, Wage Subsidy stand-alone projects could include only one participant.
1.3 Budget initiatives
In Budget 2014, the following investments were announced:
- CommunityWorks (now Worktopia), a national network of vocational training programs for persons with autism spectrum disorders administered by the Sinneave Family Foundation and Autism Speaks Canada, will receive $11.4 million over 4 years (2014 to 2018).
- The Ready, Willing and Able initiative, a partnership between the Canadian Association for Community Living and the Canadian Autism Spectrum Disorders Alliance, will receive $15 million over 3 years (2014 to 2017). This project aims to help employers develop their capacity and confidence in hiring people with autism spectrum disorders or developmental disabilities.
Neither of these 2 projects, both specifically designed to respond to the needs of individuals with autism spectrum disorders, intellectual or developmental disabilities, makes use of targeted wage subsidies.
CommunityWorks/Worktopia enhances the employment readiness of individuals with autism spectrum disorders, through interventions such as pre-employment programming, one-to-one peer mentoring and the facilitation of the acquisition of volunteer experience. This project serves teens and young adults aged 15 to 29.
The Ready, Willing and Able initiative promotes understanding and awareness of the benefits of hiring persons with intellectual disabilities or autism spectrum disorders, among employers and the general public. This is done in part through activities similar to those delivered through Employer Awareness projects. The project also reaches out to specific employers, providing information and encouraging them to make concrete commitments to hiring persons with disabilities. As well, participants of the Ready, Willing and Able initiative may receive employment related supports, including: job coaching; short-term training, educational books, manuals and reference guides; job site assessments, worksite accessibility accommodations and alterations; adaptive technology; attendant care and sign language interpretation; and work related transportation.
Both budget initiatives, sometimes referred to as pilot projects, were implemented in parallel to the Program's reforms. The evaluation examined the extent to which these projects were implemented as designed. Related findings are discussed in Section 3.4.
In May 2015 the Department's Performance Measurement and Evaluation Committee approved the Opportunities Fund Evaluation Strategy, including the evaluation questions and methodologies detailed in Appendix A. The Evaluation Strategy outlines a phased approach, with Phase I examining the design, implementation and delivery of the reformed Program, with a focus on projects that started between April 2015 and December 2016.
Findings from Phase I were based on the following lines of evidence:
- a data assessment
- a survey of participants of single year projects (n=358 responses; 44% response rate)
- the review of about 500 project documents (for example, applications for funding, assessment reports, activity reports and close-out summary reports) from 48 projects that started being delivered between April 2015 and December 2016. The documents reviewed included the most recent available when the work was conducted (for example, activity reports produced during fiscal year 2016 to 2017)
- interviews with 35 key informants with four groups: (1) participant employers; (2) single and multi-year projects' representatives; (3) Service Canada representatives; and (4) Employment and Social Development National Head Quarter representatives
Triangulation was used where possible to validate the evidence gathered. Additional details on data collection processes, reporting methods and limitations, can be found in Appendix B.
Phase II of the evaluation will examine how different interventions and series of interventions affected participants' labour market outcomes, in the medium- to long-term. This will be done using linked Opportunities Fund Participant Information Module, Employment Insurance and Canada Revenue Agency income tax data.
During Phase II, it will also be possible to examine participant's results with a gender-based analysis plus lens, which could not be done for Phase I. This was due to the fact that participants' results were aggregated and could not be examined by gender or other characteristics (for example, age, level of education, visible minority status or Indigenous identity).
Finally, Phase II will delve further into employers' experience of the Program. It is scheduled for completion in March 2020.
3. Key findings
3.1 Program relevance
The Program responds to the needs of persons with disabilities, and continues to fill a gap in employment services for those who are not eligible for Employment Insurance benefits or training support under Employment Insurance Act Part II (or cannot access such services). Evidence also indicates that there is an ongoing need to enhance employer awareness of the potential contribution that persons with disabilities can make, and confirms that financial incentives may be needed to encourage employers to hire more persons with disabilities.
The objectives of the Opportunities Fund for Persons with Disabilities continue to be relevant and reflect the needs of persons with disabilities
Evidence from the review of project documents and publications on the labour market outcomes for persons with disabilities confirms the need to address barriers and challenges that prevent persons with disabilities from obtaining and keeping employment or self-employment.
- Important gaps can be observed between employment outcomes for persons with disabilities and outcomes for persons without disabilities: For example, according to the 2011 National Household Survey and the 2012 Canadian Survey on Disability,Footnote 43 in 2011, 49% of Canadians with disabilities aged 25 to 64 were employed, compared with 79% of their counterparts without a disability. The unemployment rate was 11% and the participation rate was 55% for Canadians with disabilities aged 25 to 64, compared to 6% and 84% for their counterparts without a disability.Footnote 44
- Unequal labour market outcomes for persons with disabilities represent a large opportunity cost that is likely to increase in the context of an aging population: According to the latest estimates available (at the time this report was prepared), about 411,600 Canadians with disabilities aged 15 to 64 are not employed, despite having the potential to work.Footnote 45 This issue is likely to become increasingly concerning with the aging of the population, a trend that already emphasizes the need to fully utilize the country's talent pool.Footnote 46 Further, the portion of the labour force represented by older workers is expected to increase. Since the prevalence of disability increases with age (see Figure 1 below), the proportion of Canadians in the labour force with a disability will also tend to increase. As well, disability is likely to affect an increasing number of women.
Text description of Figure 1
|15 to 24||4%||4%|
|25 to 44||6%||7%|
|45 to 64||15%||17%|
Source: 2012 Canadian Survey on Disability (most recent data available when this report was prepared)
- While youth are less likely to have a disability, those who have one are more likely to see their condition negatively impact their employment outcomes. The prevalence of disability is lower among youth, but those who have one are more likely to have a mental or psychological disability, which makes them more likely to face disability related employment barriers or challenges. In fact, according to the 2011 National Household Survey and the 2012 Canadian Survey on Disability, persons with mental or psychological disabilities have a significantly lower probability of being employed than individuals who have other types of disabilities, even when taking severity and other factors into account.Footnote 47
- The majority of Canadians with severe or very severe disabilities are working-age adults. According to the 2012 Canadian Survey on Disability, nearly 16% of seniors aged 65 or more had a severe or very severe disability, representing 685,400 individuals. In comparison, around 5% of Canadians aged 15 to 64 had a severe or very severe disability, which represented 1,147,000 individuals, including 814,000 individuals aged 45 to 64. As shown in the graph below, the severity of disability has a significant impact on the likelihood of being employed.Footnote 48
Text description of Figure 2
Source: 2012 Canadian Survey on Disability (most recent data available when this report was prepared)
Findings indicate that there is a continued need to enhance employer awareness of the contributions that persons with disabilities can make
Evidence from key informant interviews and project documents underscored the continued need to educate employers to treat persons with disabilities fairly, and not to underestimate their potential and ability to perform in the workplace.
In addition, evidence from the document review showed that Program activities may help employers diversify their workforce and/or address specific labour shortages. For example, a number of projects designed and delivered by industry associations were identified in the document review. These projects were aimed at responding to the labour needs of these associations' business partners through a combination of Wage Subsidy and Employer Awareness activities, or through stand-alone Employer Awareness projects.
Financial incentives may be required for employers to hire participantsFootnote 50
A number of employers interviewed for the evaluation indicated that the persons with disabilities hired through Wage Subsidy agreements surpassed expectations and were hired on a regular basis when wage subsidies ended. These employers stated that they likely would not have hired the workers in question if they had not been first exposed to the Program.
Additional evidence from the document review and from key informant interviews indicates that employers expect to receive wage subsidies from agencies working with persons with disabilities. Key interviewees also noted that employers tend to be more responsive to Employer Awareness activities when they are with a Wage Subsidy. It should be noted, however, that evidence also shows that financial incentives (that is, wage subsidies and support for workplace accommodations) do not always guarantee that participants will secure employment through the Program, or that those who succeed in obtaining employment will be hired on a regular basis by their host employer.
In this regard, it was also reported that employers were sometimes concerned by the amount of paperwork that would be required, or by the fact that receiving wage subsidies would not be authorized by their employees' union, head offices and/or human resources policies. During key informant interviews, it was also noted that employers may prefer to hire participants directly, without requesting a subsidy, as they saw it as an opportunity to contribute to a cause or to their community. It was not possible to determine the extent to which this happened, due to data limitations.
The Opportunities Fund continues to fill gaps left by other employment programs for persons with disabilities
An array of provincial/territorial and federal programs exist to enhance the labour market participation of persons with disabilities. Some of these programs can overlap, complement or substitute each other, but gaps persist.Footnote 51
This is especially true for job seekers with disabilities who do not qualify for Employment Insurance benefits or other supports funded through Employment Insurance Act Part II, and/or who are located in rural or remote areas where the supports and services delivered through the Opportunities Fund may be the only ones available for persons with disabilities. It should be noted that the Opportunities Fund is the only employment-focused program for persons with disabilities at the federal level. Therefore, it plays a key role in supporting initiatives undertaken in this area at the national level.
Key informant interviewees also noted that persons with disabilities are more likely to be employed on a part-time basis, which makes them potentially vulnerable when downsizing occurs in an organization. This can negatively affect the ability of these individuals to qualify for Employment Insurance benefits. While the number of hours worked by participants employed is not captured by the Program, the survey of participants in single year projects done by the evaluation revealed that most respondents (64%) who were working at the time of the survey were working on a part-time basis (less than 30 hours in a typical week).Footnote 52
Finally, as pointed out by a few key informants, unlike for some other employment programs for persons with disabilities, a formal medical assessment is not needed to be eligible for Opportunities Fund services and supports.Footnote 53 This allows potential participants to avoid a process that can be lengthy, burdensome, and even costly.
The Program continues to align with priorities, roles and responsibilities of Employment and Social Development Canada and the federal government
The Program aligns with the Government of Canada's policy directions and priorities. This is reflected in the continued funding of the Program, the recent tabling of new federal accessibility legislationFootnote 54 and the Government's broader engagement towards the elimination of barriers to employment among underrepresented groups on the labour market, including persons with disabilities.
More specifically, the Program aligns with the Government of Canada's efforts to “eliminate systemic barriers and deliver equality of opportunity to all Canadians living with disabilities,”Footnote 55 and to “improve work opportunities and employment outcomes for persons with disabilities,”Footnote 56 as outlined in Budgets 2016 and 2017.
The objectives of the Opportunities Fund also reflect Employment and Social Development Canada's mission of building “a stronger and more inclusive Canada, to support Canadians in helping them live productive and rewarding lives and improving Canadians' quality of life.”Footnote 57
The Program also aligns with a number of the Department's responsibilities, such as:
- developing policies that ensure that all Canadians can use their talents, skills and resources to participate in learning, work and their community;
- fostering inclusive growth by providing opportunity and assistance to Canadians with distinct needs, such as Indigenous people, people with disabilities, homeless people and recent immigrants.Footnote 58
Finally, Employment and Social Development Canada is well positioned to deliver the Program given its leading role in delivering programs and services that affect Canadians throughout their lives. These include supports to unemployed workers who are not eligible for Employment Insurance benefits and/or who are traditionally underrepresented in the labour force.
There is potential to increase the reach of the Program
The Opportunities Fund served approximately 4,300 clients per year, on average, from fiscal year 2012 to 2013 to fiscal year 2016 to 2017. In comparison, as of 2012 (most recent estimates available at the time this report was prepared), approximately 103,300 Canadians aged 15 to 64 had a disability and, despite having the potential to work, had never worked or had not worked in the previous five years.Footnote 59 Potentially, they could have benefitted from the Program.Footnote 60
It was also noted during key informant interviews that there could be better awareness and understanding, among employers, of the Program in general and how it works, and more broadly about the resources available to help integrate persons with disabilities into the workplace.Footnote 61 This indicates that there is opportunity for greater awareness and visibility of the Program among service providers and potential clients. In this regard, it should be noted that both the 2008 Summative Evaluation and the 2014 Summative Evaluation of the Opportunities Fund concluded that there was no systematic approach to promote the Program to clients and service providers.Footnote 62
Finally, persons with disabilities may decide to remain outside the labour market despite being aware of the availability of employment services and supports such as those delivered through the Opportunities Fund. This may be due to various factors that were identified in existing studies, and were confirmed by the current evaluation. These include:
- the expectation that employment income will not be high enough to compensate for the loss incurred in terms of social assistance benefits, including related supports such as a drug plan or assisted housing;
- the inability to commute to and from work;
- the lack of access to affordable childcare;
- family responsibilities;
- the perception that jobs are not available.Footnote 63
3.2 Program performance
Evidence indicates that progress is being made towards the achievement of the Program's direct outcomes. From fiscal year 2012 to 2013 to fiscal year 2016 to 2017, the percentage of participants who were employed, self-employed or returned to school (up to 12 months after their last intervention) increased progressively from 42% to 55%. Ongoing challenges or barriers faced by persons with disabilities continue to prevent approximately half of the participants from becoming employed or self-employed through the Opportunities Fund. Finally, limited information is collected on employers' experience and satisfaction with the Program, and the progress made through Employer Awareness activities.
Various benefits for persons with disabilities can be derived from participation in the Program
Evaluation evidence confirms that, through the Program, progress is being made towards enabling participants to access labour market information, to develop skills needed to search for and find employment, to acquire employment experience and workplace accommodations or to take part in Self-Employment activities. Evidence also indicates that the Program can have the following benefits for participants and/or their family:
- the development of social skills, workplace communication and interview skills, organizational, administrative and problem solving skills, as well as time, anxiety and stress management skills;
- the improvement of overall well-being, attitudes, behaviors and motivation;
- an increase of self-esteem, pride, self-confidence and sense of self-worth;
- an increase of self-awareness and understanding of personal strengths, weaknesses and professional interests;
- an increase of networking opportunities;
- a decrease of social isolation and feelings of stigmatization;
- the acquisition or development of technical skills in areas like customer service, general maintenance and logistics, landscaping and bookkeeping.
The Program's overall success rate increased progressively over the 5 previous years
Table 1 shows that the proportion of participants served who were either employed, self-employed or returned to school after participating in the Program, improved progressively between 2012 and 2017, reaching 55% in fiscal year 2016 to 2017, up from 42% in fiscal year 2012 to 2013.Footnote 64 On average, during the same period, the Program served approximately 4,316 persons with disabilities each year. Of these, around 41% (1,791) were employed or self-employed and 6% (253) had returned to school.
These improvements could be due to numerous factors that can influence a job seeker's likelihood of finding employment, such as the evolution of economic conditions and of the types of skills that are in demand on the labour market.
As well, changes in reporting requirements have occurred along with the reform of the Program.Footnote 65 Prior to fiscal year 2014 to 2015, participants' results were assessed at the latest 24 weeks following the end of their last intervention. Since reforms were implemented, the Program updates these results:
- at 6 months and 12 months post-intervention, within the timeframe of the agreement, for multi-year projects; and
- for single year projects, up to 12 weeks after participants' last intervention, and within 60 days of the project end date.
A survey conducted as part of this evaluation to collect more recent results for participants in single year projects (at 6 and 12 months post intervention) showed that approximately 42% were employed or self-employed following participation. This figure is very close to the one indicated in Table 1 (for fiscal year 2015 to 2016). Survey results indicate that female participants were more likely to work at the time of the survey (47%) compared to male participants (38%).Footnote 66
|Results by fiscal year||2012 to 2013||2013 to 2014||2014 to 2015||2015 to 2016||2016 to 2017||Five year average|
|Clients employed or self-employed||1,563 (37%)||1,879 (37%)||1,455 (42%)||1,950 (43%)||2,108 (48%)||1,791 (41%)|
|Clients returned to school||219 (5%)||290 (6%)||192 (6%)||286 (6%)||280 (7%)||253 (6%)|
|Clients neither employed, self-employed nor returned to school||2,440 (58%)||2,843 (57%)||1,826 (53%)||2,273 (50%)||1,978 (45%)||2,272 (53%)|
|Total number of clients served||4,222 (100%)||5,012 (100%)||3,473 (100%)||4,509 (100%)||4,366 (100%)||4,316 (100%)|
Source of data: Performance Measurement, Analysis and Reporting, Common System for Grants and Contributions - Client Module and Participant Information Module, June 22, 2017.
Note: These results do not include those of participants of the CommunityWorks/Worktopia and Ready, Willing and Able budget initiatives.
In addition, survey results suggest that the percentage of participants returning to school increases with time, subsequent to their participation in the Program, and that Table 1 may underestimate this result.Footnote 67 When asked to indicate the level of education they were enrolled in (information that is not collected by the Program through participant information forms), survey respondents who had returned to school provided the responses presented in Table 2 below.
|High school or equivalent||16 (46%)||3 (16%)||19 (35%)|
|Community college||8 (23%)||9 (47%)||17 (32%)|
|University||8 (23%)||5 (26%)||13 (24%)|
|Other||3 (9%)||2 (11%)||5 (9%)|
|Total||35 (100%)||19 (100%)||54 (100%)|
Source of data: TNS Canadian Facts Ltd. (2017), Survey of Participants for the Evaluation of the Opportunities Fund for Persons with Disabilities – Phase I, prepared for Employment and Social Development Canada.
Based on the figures in this table, males were more likely than females to be enrolled at the high school or equivalent level (46% versus 16%, respectively). Females were more likely than males to be enrolled in a community college (47% versus 23%, respectively). It should be noted that only 54 respondents reported being enrolled in school, which constitutes a small sample. Therefore, this comparative analysis should be interpreted with caution.
The Program's target population is mainly composed of individuals who are initially far removed from the labour market and can face a variety of employment related challenges
Evidence indicates that participants' journey through the Program does not follow a standard path. This is consistent with the fact that persons with disabilities form a particularly diverse segment of the population. Some may have multiple disabilities that greatly limit all their daily activities, while others have only one disability that has less of an impact on their activities. Persons with disabilities also have various levels of educational attainments, which can significantly affect employment outcomes.Footnote 68
Further, the Program fills gaps in areas where the services offered to persons with disabilities are limited or nonexistent, and is delivered in different regions across Canada, where economic and labour market conditions can vary considerably. Finally, the Opportunities Fund may act as a last resort solution for individuals with disabilities who are not eligible for Employment Insurance benefits or employment programs delivered through Employment Insurance Act Part II.Footnote 69 This means that most Program participants may have never worked in Canada, may have not worked sufficient hours to qualify for the Employment Insurance program, or may have withdrawn from the labour force for five years or more.
Therefore, the Program's clientele conceivably includes individuals that have greater challenges to overcome and greater labour market vulnerabilities, as opposed to both persons without disabilities and persons with disabilities who have stronger ties to the labour market. If needed, the latter may qualify for Employment Insurance benefits and be eligible for other employment supports and services delivered at the federal, provincial or territorial level.
Many participants are successful while others may continue to face barriers or challenges in trying to secure or maintain employment
About half of participants have successful outcomes, which include finding employment or pursuing further education after taking part in the Program. At the same time, about half of participants do not reach that objective.Footnote 70 Such outcomes may be due to different factors. The most commonly identified by the evaluation were physical or mental health related issues that emerged during the course of the Program, such as injuries, illnesses or relapses requiring short-term hospitalization, social anxiety disorders, episodic depression and addiction issue. Lack of motivation or commitment among participants was also mentioned on one or more occasions in project documents.Footnote 71
Further, the document review revealed that participants may have a limited range of jobs for which they are qualified – a situation that can be attributable to adverse economic conditions at the local or regional level, and also to participants' limited formal education, abilities, skills, professional interests or employment history. Participants may not be able to perform functions that are physically demanding and this may further limit their employment opportunities.Footnote 72
Finally, participants may not obtain or maintain employment through the Program for reasons similar to those that can initially discourage persons with disabilities who are not employed from accessing employment services and supports. These reasons include lack of affordable childcare, difficulties commuting to and from work (adapted transportation is unavailable or too costly), and the inability to obtain an employment income that exceeds available social assistance benefits.Footnote 73
Overall, these findings are consistent with evidence from the 2012 Canadian Survey on Disability, from previous evaluations of the Program and from the evaluation of the Canadian Disability Savings Program.Footnote 74 Findings were also confirmed by results of the survey conducted in the context of this evaluation. Approximately 29% of survey respondents had never worked after participating in the Program.Footnote 75 When asked to identify the reasons why they had not been working:
- 24% cited illness or disability
- 13% noted that they believed no work was available
- 7% mentioned having other personal or family responsibilitiesFootnote 76
- 7% indicated that it was difficult to find a job or that they could not find a job
- 6 % mentioned that it was a personal choice or that they lacked motivation
Participants may secure jobs with employers they were not initially involved with through Wage Subsidy activities
Among respondents who had taken part in Wage Subsidy activities, about 39% were working at the time of the survey conducted for this evaluation. Of these respondents, approximately 36% (or 14% of all respondents who took part in Wage Subsidy activities) continued to work for the same employer, 6 to 18 months after the subsidy ended. The other 64% (or 25% of all respondents who took part in Wage Subsidy activities), were working with other employers than those who hired them through Wage Subsidy agreements.Footnote 77 It should be noted that data collected by the Program through participant information forms does not indicate whether participants employed are working with the employer they were involved with through Wage Subsidy activities or with another employer.
During key informant interviews, it was noted that participants of Wage Subsidy activities may not always want to occupy the jobs initially obtained through the Program, if they find better opportunities. Similar input was obtained directly from employers interviewed as key informants, some of which also reported that participants had to be dismissed for aggressive behavior or stopped showing up at work and could not be reached, while others simply decided to leave as they felt they could not work at the level required, or could not handle the stress of the workplace and the resulting anxiety.Footnote 78
Additional evidence is required to fully assess the Program's effect on the medium- to long- term labour market outcomes of participants
The majority of key informants who addressed this issue indicated that the Program had a positive effect on the employment opportunities that are available to persons with disabilities.Footnote 79 However, the performance information and evidence gathered through the document review are not robust enough to indicate that Program interventions resulted in increased employment opportunities for persons with disabilities or led to more persons with disabilities to be integrated into the labour market.Footnote 80
Additional evidence is also needed to assess the extent to which changes resulting from the reform of the ProgramFootnote 81 have contributed to improve participants' employment outcomes on the medium and long term. Such evidence will be developed and examined during Phase II.Footnote 82
Further, it should be noted that survey respondents who worked for pay at some point following participation provided mixed responses when asked how likely they would have been to obtain employment without participating in the Program. For instance, 28% of these respondents suggested that they would have likely found employment without participation, while about a quarter (22%) noted that they would have been unlikely to find work without participating in the Program.Footnote 83
These findings indicate that various factors other than Program interventions can influence participants' chances of success in the labour market. Phase II of the evaluation will examine this aspect more thoroughly.
Important investments are being made to enhance employers' awareness of the valuable contribution that can be made by persons with disabilities
Three stand-alone national Employer Awareness projects were funded through the Program for the fiscal year 2015 to 2016. These projects received combined contributions of approximately $3 million, part of which was spent over several years.Footnote 84 In addition, Employer Awareness activities are delivered through other projects, generally in combination with Wage Subsidy activities, and through the Ready, Willing and Able initiative.Footnote 85
Data collected with regards to Employer Awareness activities is limited and generally comprised of indicators such as the number of activities delivered and/or the number of employers reached through the process. While relevant, such information is not sufficient to assess the impacts of these activities, not only in terms of awareness and understanding, but also with regards to employers' human resources policies and practices. For example, the impact of employer awareness activities could be measured by increases in the number of persons with disabilities who were hired by employers after receiving awareness material or being reached through other channels.
Further, when asked how their experience with the Program affected their perception of persons with disabilities, employers interviewed as key informants provided a range of responses. Some found persons with disabilities to be as competent as those without disabilities,Footnote 86 while other employers indicated that they needed more experience with the Program to make a determination. Finally, some employers reported that they would consider renewing their involvement in the Program, but preferably with participants that do not have specific types of disabilities.
A number of measures may help to foster better results for participants and improve the Program's performance
Approximately 29% of survey respondentsFootnote 87 did not find an employment after participating in the Program and were asked what additional assistance, accommodations or supports would have helped them to find employment. Of these respondents:Footnote 88
- close to half (45%) identified the need to develop a system to connect participants with employers willing to hire persons with disabilities, through job postings for persons with disabilities and introductions to employers. This echoes the findings of Employment and Social Development Canada's latest Evaluation of Learning and Labour Market Information;Footnote 89
- approximately a third (35%) identified a need for more training and longer interventions, which was also mentioned by a few key informants. On this matter, a few key informants also specified that funding agreements should be longer (for example, 3 years) and the processing period shorter;
- about a fifth (18%) identified a need for help and support with job search (for example, from an advisor or employment recruiter). While these supports can be delivered through Enhanced Employment Assistance Services, they may not always be accessible to participants, considering that funding recipients have limited resources;
- approximately 12% expressed that more funding is needed, especially for workplace accommodations, which is consistent with input from a few key informants who noted that, funding recipients sometimes have to partner with other organizations who have access to additional resources or funding in order to ensure that the needs of each participant are fully met.
The document review, a line of evidence used in this evaluation, could not determine the extent to which support for workplace accommodations were provided to Program participants as needed. Input from key informant interviews revealed that the availability of such supports may not be well known among employers.
It was also mentioned during interviews with Service Canada representatives that coaching and other forms of psychosocial supports could help participants to stay employed or remain motivated when they do not succeed in finding employment. In particular, this could be useful for those who have mental or psychological health related disabilities. It should be noted that it was not always clear among service representatives whether such supports can be provided via the Program.
Finally, a few informants among project representatives, Service Canada regional staff and Employment and Social Development Canada National Head Quarter staff, identified options to explore. These would help to strengthen the efforts made in the area of employer awareness and/or to make securing job placements easier and quicker for participants. Their suggestions included:
- a broader national employer awareness strategy, inclusive of all disability types and delivered by the Federal Government in consultation with key stakeholders
- workshops and training on disability for specific types of employers or job sites
3.3 Implementation of reforms
More participants are acquiring employment experience in small and medium-sized businesses through the reformed Program. Also, the proportion of youth included among participants is increasing. The evaluation found no evidence that more demand-driven training activities are being provided nor that the reformed Program is more responsive to labour market needs. Further, findings indicate that employers are generally not involved in the design of projects; they mainly contribute to their delivery by covering part of participants' wages and providing on-the-job training and supervision. More evidence is needed to determine to what extent the use of social innovation has contributed to the optimization of federal spending dedicated to the Program.
Progress is being made towards all participants acquiring employment experience or taking part in Self-Employment activities through the Program
As shown in Table 3, Wage Subsidy activities represented a higher proportion of all activities undertaken in the fiscal year 2016 to 2017 (37%), than in the fiscal year 2015 to 2016, during which they accounted for 25 % of all interventions undertaken. When adding Self-Employment activities, which can also account for the mandatory employment experience component of participants' action plans, these percentages reach 29% for the fiscal year 2015 to 2016 and 41% for the fiscal year 2016 to 2017.
Following the implementation of Program reforms, the number of participants receiving Enhanced Employment Assistance Services remained higher than the total number of Wage Subsidy and Self-Employment activities undertaken.Footnote 90 Given data limitations, it was not possible to determine if specific variables explain why participants may receive Enhanced Employment Assistance Services without subsequently acquiring employment experience through Wage Subsidy activities or undertaking Self-Employment activities.Footnote 91
Finally, information from key informant interviews with project representatives and Service Canada regional staff indicate that participants may find employment without requiring the help of a wage subsidy. In this type of situation, they would still be included among the group of participants considered to have succeeded in the Program, both as participants with enhanced employability and as participants employed or self-employed.Footnote 92 The evaluation could not determine how often this occurred, due to limitations in data available.Footnote 93
|Intervention type||Fiscal year|
|2012 to 2013||2013 to 2014||2014 to 2015||2015 to 2016||2016 to 2017|
|Wage Subsidy (OF-WS)||318||717||933||1,238||2,074|
|Enhanced Employment Assistance Services (OF-EEAS)||2,833||4,125||1,823||2,948||2,704|
|Skills for Employment (OF-SFE)||643||615||278||480||610|
|Work Experience (OF-WE)||405||318||394||114||n/a|
|Services (OF-S)Footnote 94||29||11||2||1||n/a|
|OF DevelopmentalFootnote 94||47||2||3||n/a||n/a|
Source of data: Performance Measurement, Analysis and Reporting, June 22, 2017.
Note: carry-over interventions are not included (new starts only).
Note: These results do not include those of participants of the CommunityWorks/Worktopia and Ready, Willing and Able budget initiatives.
It should be noted that, before the Program was reformed, participants could also acquire subsidized employment experience through interventions referred to as Work Experience activities. Unlike positions occupied by participants of Wage Subsidy activities, those occupied by participants of Work Experience activities had to be incremental to, and not part of, the employer's normal operations. Also, there was not the same expectation that the employer would keep the participant as an employee once the subsidy would have ended.
Finally, while participants were paid through both Wage Subsidy and Work Experience activities, wages paid to participants of Work Experience activities were entirely subsidized by Employment and Social Development Canada, while those of participants of Wage Subsidy activities are subsidized only in part. Therefore, all employers hiring individuals with disabilities through the Program now have to contribute financially to the wages paid to these individuals.
More participants are gaining employment experience in small and medium-sized enterprises
The evaluation could not determine the exact proportion of Wage Subsidy activities that took place in small and medium-sized businesses, due to data limitations. However, the vast majority of key informants who addressed the question indicated that employers from this category account for the largest part of employers across Canada. Therefore, it can be inferred that more persons with disabilities are acquiring work experience in small and medium-sized enterprises as the number of participants of Wage Subsidy activities increases.
The proportion of youth among Program participants increased after the implementation of reforms
Program administrative data reviewed indicates that the share of youth aged 15 to 30 among all participants is significantly higher than 25%, and has increased progressively between the fiscal years 2014 to 2015 and 2016 to 2017, starting at 43% and reaching approximately 52%, respectively.Footnote 96
According to the regional Service Canada key informants who addressed the issue of greater youth participation, this increase mainly stems from the fact that the goal to include more youth among cohorts of participants has been communicated to funding recipients.
Finally, part of this increase can be attributed to the fact that the participants in the CommunityWorks/Worktopia project,Footnote 97 one of the two budget initiatives implemented in parallel to the Program's reform, were all youth aged 15 to 30.Footnote 98
Findings do not indicate that the reforms implemented have facilitated the provision of demand-driven training solutions
Under the reformed Program, Skills for Employment interventions (that is, short-duration training programs) can only last for a maximum of 6 months or 26 weeks. The majority of the regional Service Canada key informants who were interviewed stated that this period is not long enough to allow for participants to consolidate the skills they acquired or to develop specialized skills that are in demand on the labour market. The reduction of the maximum duration of Skills for Employment interventions was implemented within the context of the reforms. The underlying rationale behind such change, and its alignment with the objective to provide more demand-driven training solutions to participants, could not be formally identified.
Overall, within the sample used for the purposes of the document review, a few projects were designed to provide participants with training that aligns with specific needs of the labour market, and one of these projects made use of Skills for Employment interventions. This particular project enabled a group of unemployed youth with disabilities living in First Nations communities to complete the Heavy Equipment Operator training program. These individuals were subsequently hired under a Wage Subsidy agreement by their band councils to perform duties that would have otherwise been contracted out to individuals living outside their communities.
In the other projects examined, the content of activities delivered through Enhanced Employment Assistance Services was adapted to prepare participants to occupy jobs in specific sectors where there was demand for labour at the regional level (for example, wood manufacturing, gardening and catering industries); however, this could also be done before the Program was reformed, and the evaluation found no evidence that this happened more commonly under the reformed Opportunities Fund. Further, project documents indicate that not all of the participants obtained jobs in one of these sectors, for different reasons including skills mismatches and a lack of interest on the part of some participants.
When asked where their project has helped to respond to labour demand in specific industries, the majority of project representatives interviewed mentioned the retail and hospitality sectors.Footnote 99 While a few key informants from this group noted that steps have been undertaken to open new opportunities for participants in the IT, banking, green energy and energy efficiency sectors, none of them confirmed whether these initiatives have led to concrete employment opportunities for participants.
Evidence indicates that some employers are involved in the design of projects and that this aspect of the Program could be strengthened
Among the documents reviewed, the evaluation identified a few projects that were designed and delivered by organizations representing employers (industry associations) and with the objective of addressing specific shortages of workers and/or to help employers from a specific sector diversify their workforces through targeted stand-alone Employer Awareness projects (as well as through a combination of Employer Awareness and Wage Subsidy activities). During one of these projects, the funding recipient also tested a “disabilities in the workplace self-assessment tool” designed for a specific occupation, and found three employers to ‘champion' this initiative.”Footnote 100
The other examples of employer involvement in specific project activities or initiatives identified by the evaluation include:
- the creation of a network of "ambassadors" representing locally based small and medium-sized enterprises that have demonstrated an interest in hiring qualified persons with disabilities
- an "employer recognition" campaign designed to demonstrate the value of the employment relationships established through the Program
- discussions on employers' expectations with a panel of local employers that took place in the context of enhanced employment assistance servicesFootnote 101
Under the reformed Program, the vast majority of projects that comprise Wage Subsidy activities are not designed and delivered by employers looking to address specific needs in labour, but by Community Coordinators, who act as intermediaries between several participants and employers. All projects except stand-alone Employer Awareness projects now have to include at least eight participants,Footnote 102 a criterion that was described as being very difficult or even impossible to meet for employers, especially for small and medium enterprises.Footnote 103
At the same time, it was noted that, from a program delivery perspective, allowing employers to design and deliver stand-alone Wage Subsidy projects with less than eight participants would not be a cost-effective option, considering the administrative burden and overhead costs incurred for applicants and Employment and Social Development Canada, whose regional service-delivery staff has diminished over the years.
Partnerships between funding recipients and employers are usually created or formalized after projects have been approved for funding
According to the Program's operational directives, applicants should provide supporting documents from project partners with their application for funding, such as a list of potential employers for wage subsidy participants. The majority of applications for funding reviewed in the context of this evaluation did not identify actual or potential employer or business partners, and none of the employers interviewed as key informants reported having been involved in the design of a project.
These findings are consistent with Employment and Social Development Canada and Service Canada key informants who indicated that, although applicants may have informal discussions with potential employers prior to the submission of their application for funding, these first steps do not necessarily lead to the creation of formal partnerships.
Evaluation findings indicate that employment relationships created through OF Wage Subsidy activities are generally fostered by initiatives of funding recipients and by job seeking efforts made by participants, in collaboration with project staff, when funding for projects has already been approved by the Department. In turn, employers' contributions towards the delivery of projects usually consist of covering part of participants' wages and providing them with on-the-job training and supervision.Footnote 104
Efforts continue to be made to ensure the needs of employers are met during project delivery
Interviews conducted with project representatives and findings from the review of project documents indicate that, during the delivery of projects, funding recipients may conduct follow-up activities such as worksite visits to ensure that participants receive the support and training needed, to discuss issues that may arise and to provide additional job coaching if needed. This was confirmed by the majority of employers interviewed. The latter indicated that funding recipients were helpful during project delivery as they helped participants acquire/develop skills for the workplace and/or have made efforts to ensure a good fit between participants and the jobs in which they were placed.
One employer interviewed stressed that the selection of participants for Wage Subsidy activities had been entirely made by the recipient organization, while another reported having the opportunity to meet and decide (or not) to hire potential participants (during an "open house" event organized by a funding recipient).Footnote 105
Most employers interviewed for this evaluation mentioned that participants were adequately prepared before starting work. They indicated that the preparation level could vary among participants and/or that “a bit more work” could be done to ensure participants knew what employers are looking for and what is expected from them in the workplace.Footnote 106
Overall, only one of the employers interviewed reported that the individuals hired through Wage Subsidy activities were unable to perform very basic tasks. This key informant also noted that if he/she would have been consulted beforehand, these participants' needs in terms of pre-employment education and training would have been identified, which in turn would have increased their chances of staying employed after participation.Footnote 107
The majority of regional Service Canada key informants as well as a few Employment and Social Development Canada key informants mentioned that the Program was more flexible before the 2015 reforms and used to put a greater focus on developing participants' pre-employment, transferable and “soft” skills.Footnote 108 A few of these informants specified that the interventions falling under the “Work Experience” categoryFootnote 109 were a good way to prepare certain participants to integrate the labour market, notably as they could help better assess their needs in terms of skills development and workplace accommodations, which can be difficult to do in the limited timeframe allowed under the reformed Program.
Phase II of the evaluation will help determine if, and to what extent, participation in Wage Subsidy activities leads to better labour market outcomes in comparison with participation in Work Experience activities, which are no longer eligible for funding under the Opportunities Fund. On this matter, a key informant from Employment and Social Development Canada also noted that, while Work Experience activities could lead to employers “taking advantage” of persons with disabilities, the new focus on Wage Subsidy interventions may encourage funding recipients to select participants that were initially more “job ready.”Footnote 110
More evidence is needed to determine the extent to which social innovation has helped maximizing the impact of program spending
The Opportunities Fund encourages the use of social innovation (that is, social enterprise, social partnerships or pay-for-performance) in order to maximize the impact of federal spending.Footnote 111
The document review identified a few projects where the recipient organization used a social innovation model. In the majority of cases, it consisted of a social enterprise that obtains revenues from the provision of services that are related to the workplace integration of persons with disabilities. Such services are provided on a fee-for-service basis to employers. They may consist of workplace accommodations assessments and support with the development and implementation of inclusive human resources policies and practices.
In another project, the recipient organization made use of a pay-for-performance model where a fraction of departmental funding is conditional on the achievement of predetermined results. More specifically, the funding agreement specifies that an amount of $108,000, which represents about 5% of the total project costs, will be paid to the funding recipient at a rate of $1,500 per successfully self-employed participant.
The evaluation could not determine the extent to which these projects' social innovation component has helped maximizing the impact of Opportunities Fund spending. These components were either already in place when projects have started or were at too early a stage of implementation and had not produced concrete results when the evaluation was conducted.Footnote 112 It should also be noted that the value of the resources generated by recipient organizations through their social enterprise could not be determined, due to limitations in available data.
Overall, these findings were confirmed by the few regional Service Canada and Employment and Social Development Canada key informants who discussed the subject. Some of these informants further noted that the concept of social innovation was not defined in a clear and concise manner in the Program's call for proposals.Footnote 113 This resulted in administrative challenges both at the project assessment and monitoring stages, and required the Program to develop a rating scale for social innovation.Footnote 114
Finally, one of these informants also mentioned that the use of social innovation within the context of the Program was not promoted and often happened incidentally.
3.4 The Ready, Willing and Able and CommunityWorks/Worktopia initiatives
The Ready, Willing and Able initiative faced issues of collaboration between partners and difficulties with the recruitment of participants. In the case of the CommunityWorks/Worktopia project, many initial participants did not complete the project. In order to achieve their objectives, changes were made in the delivery approach of both projects.
When the evaluation work was conducted, only preliminary results were available for the Ready, Willing and Able and CommunityWorks/Worktopia initiatives. Challenges faced in the implementation of these 2 projects were identified through the review of the project documents and interviews with key informants. As well, 2 evaluation reports (a formative evaluation report and a progress report) of the Ready, Willing and Able initiative were prepared by the University of British Columbia, on behalf of the recipient organization. Some of the findings discussed in this section were drawn from these reports.
Ready, Willing and Able
Ready, Willing and Able is a national strategy delivered by the Canadian Association for Community Living, in partnership with the Canadian Autism Spectrum Disorders Alliance. This initiative aims to increase the labour force participation of people with intellectual disabilities or autism spectrum disorders. To support this goal, it generates increased demand to hire people with developmental disabilities, and works with employment service providers to ensure that effective job matching tools are in place.
More specifically, this project delivers interventions similar to Employer Awareness activities. The project also reaches out to specific employers, providing information and encouraging them to make concrete commitments to hiring persons with disabilities. As well, a range of employment supports can be provided to participants; examples include job coaching, short-term training, job site assessments, worksite accessibility accommodations and alterations, adaptive technology, tools and equipment, attendant care and sign language interpretation.
Unlike other Opportunities Fund projects, detailed information on each participant of the Ready, Willing and Able initiative is not collected by the Program. This is normally done through participant information forms, which are also used to verify participants' eligibility for the Program. Instead, results for participants of the Ready, Willing and Able initiative are self-declared by the recipient organization, in aggregate form (for example, total number of participants served who found employment), and are not validated by the Program.
The Ready, Willing and Able initiative had to address supply-side issues
There were situations where jobs were created in collaboration with employers, yet could not be filled due to supply-side issues. The causes of this situation were not clearly identified, but were perceived by key informants interviewed by the University of British Columbia (during focus groups), as being more related to structural issues rather than to an absence of potential workers. Such issues included a lack of collaboration from third party employment support agencies.Footnote 115
A progress report prepared by the University of British Columbia indicates that the introduction of autism outreach coordinators has helped to make progress in the area of participant recruitment. These coordinators' role is to identify job seekers with autism spectrum disorders who do not have a co-occurring intellectual disability.Footnote 116
According to this report, these individuals are “unlikely to qualify for traditional services offered to people with intellectual disabilities or co-occurring intellectual disabilities and autism spectrum disorders,” while generally having “employment skills and assets that are beyond those required for jobs traditionally accessed through employment services for people with intellectual disabilities”. The report also states that this situation results “in a disconnect between opportunities and ability which can lead to a lack of opportunity to meet their full employment potential.”Footnote 117
Progress was realized in addressing challenges encountered in the delivery of the project
A formative evaluation of the Ready, Willing and Able initiative found that there were tensions between national and provincial/territorial project partners, and third party employment agencies. Such tensions were said to have created a “competition-like environment” as well as a “disincentive to collaborate.”Footnote 118
According to a progress report prepared by the University of British Columbia, significant improvements have been reported in this regard, in particular between the national Ready, Willing and Able team and provincial/territorial project partners. At the same time, according to the report, collaboration issues remained, especially between Ready, Willing and Able staff and locally supported (third party) employment agencies.
Such challenges range from an absence of service providers in some regions to what was described by key informants as "philosophical differences" in the Ready, Willing and Able approach and that of traditional service providers.Footnote 119 More specifically, the first focuses on supporting people in getting jobs, while the latter focuses on training and work readiness.
Finally, it was also reported during focus groups that participants have expressed concerns with the fact that the roles and responsibilities of the wide range of partner organizations involved in the Ready, Willing and Able initiative were not clearly defined.Footnote 120
The majority of Ready, Willing and Able participants find employment in sectors that are traditionally associated with supported employment
According to the progress report prepared by the University of British Columbia, notable increases in job opportunities for participants occurred in sectors where people with intellectual disabilities haven't historically worked. At the same time, the engagement of “non-traditional” sectors (and “non-traditional roles” within traditional sectors) was said to remain a “complicated long game” that requires a “significant investment of time with a relatively limited number of outcomes.”Footnote 121 Such challenges were further impacted by job-seeker availability issues in provinces or territories across Canada.Footnote 122
The document review indicated that an emphasis was placed on national-scale employers for the work placement of Ready, Willing and Able participants. According to a few project representatives interviewed as key informants, in general, larger employers tend to be more receptive to hiring persons with disabilities, as they have more resources. Therefore, they may already have human resources policies that help better integrate these individuals into the workplace. A few other project representatives noted that small and medium enterprises tend to have more flexible and simple hiring processes. This makes them easier to work with for the purposes of Wage Subsidy activities, which are not undertaken by Ready, Willing and Able participants.
CommunityWorks/Worktopia is delivered by the Sinneave Family Foundation, in partnership with Autism Speaks Canada. This initiative focuses on improving the employment readiness and enhancing the employment skills of individuals with autism spectrum disorders, with a focus on teens and young adults. This project, originally called CommunityWorks, has evolved and now comprises 2 streams that are grouped under the Worktopia initiative. More specifically:
- stream 1 includes 2 programs called CommunityWorks and SchoolWorks. The first is a peer supported pre-employment program for teens and youth aged 15 to 21, who have autism spectrum disorders and attend high school. The second program offers social and vocational skills training to this same group, and complements existing school curricula by adding elements that are unique to the learning needs of individuals with autism spectrum disorders;
- stream 2 includes one program called EmploymentWorks, which offers employment preparedness training for individuals aged 15 to 29 who have autism spectrum disorders and are no longer attending school. These individuals are looking to develop the essential skills needed to obtain meaningful and sustainable paid employment.
One objective of Worktopia is to have individuals with autism spectrum disorders, and their peer mentors, engage in volunteer work in their community. Such experience helps developing job skills, learning to problem solve, and socializing. Finally, the CommunityWorks/Worktopia project also has the objective to enhance the capacity of the community employment services system to be “an effective bridge-builder” between small, medium and large-scale employers and individuals with autism spectrum disorders.Footnote 123
The CommunityWorks/Worktopia program is successful at helping participants develop skills, increase their employability and/or return to school, and there could be a better link to employers
According to recent data provided by the Program, 264 participants were served through this project, and among these participants, 34 (13%) found employment, and 63 (24%) returned to school.Footnote 124
The project documents reviewed indicate that some participants did not complete the Program as they found paid employment on their own, not necessarily as a result of their participation in CommunityWorks/Worktopia. In this regard, it was also noted in project documents that, when it comes to connecting with potential employers and finding work, there may be a gap between participants' expectations and how the Program can actually help them.
While the number of applications received by potential participants exceeds the number of places available, completion rates varied across delivery sites
According to project documents, at certain delivery sites where the Program's completion rates were particularly low, the Sinneave Family Foundation (recipient organization) decided to ensure that all youth enrolled in the Employment Work stream of the Program had independent transportation and good language skills.Footnote 125 This points to some barriers faced by participants. The Sinneave Family Foundation has also considered offering post-program supports to all CommunityWorks/Worktopia participants, which was not originally planned.Footnote 126
In sum, both budget initiatives faced implementation challenges that required significant changes in delivery. At the time of the evaluation, the information and data available were too limited to determine the extent to which such changes have improved results for participants.
3.5 Program costs
The average costs per client served through the Opportunities Fund for Persons with Disabilities were comparable to the average costs per client of other federal labour market programs for disadvantaged groups
A limited cost analysis of the Program was undertaken for Phase I. As shown in Table 4, average cost per client served through the Opportunities Fund for Persons with Disabilities compared favorably with average costs per client served through other federal labour market programs for disadvantaged groups.
For example, from fiscal year 2012 to 2013 to fiscal year 2016 to 2017, the average cost per client served under the Opportunities Fund was $6,961. This is lower than both the average cost per client served under the Aboriginal Skills and Employment Training Strategy (ASETS) ($7,389), and the average cost per client served under the Skills Link stream of the Youth Employment Strategy ($8,149).Footnote 127
|Fiscal year||Five year averages|
|Program||2012 to 2013||2013 to 2014||2014 to 2015||2015 to 2016||2016 to 2017|
Data sources: Chief Financial Officer Branch; Performance Measurement, Analysis and Reporting, Employment and Social Development Canada, July 2017.
As shown in Table 5, on average, from fiscal year 2012 to 2013 to fiscal year 2016 to 2017, the cost per Opportunities Fund client with successful outcome (employed, self-employed or returned to school) was $14,687. This is slightly higher than the average cost per Aboriginal Skills and Employment Training Strategy client employed or returned to school ($12,886), and higher than the average cost per Skills Link client employed or returned to school ($11,719).
Table 5: Costs per successful client, fiscal years 2012 to 2013 till 2016 to 2017
|Fiscal year||Five year averages|
|Program||2012 to 2013||2013 to 2014||2014 to 2015||2015 to 2016||2016 to 2017|
Source of data: Chief Financial Officer Branch; Performance Measurement, Analysis and Reporting, Employment and Social Development Canada, July 2017.
It should be noted that the amounts presented in Table 4 and Table 5 only include departmental contributions towards projects' costs. However, these costs do not include contributions made towards the Ready, Willing and Able and the CommunityWorks/Worktopia initiatives. Due to differences in their design, these two projects cannot be compared to the other Opportunities Fund projects.
Finally, it was not possible to determine whether operations and management costs of the Opportunities Fund (detailed in Table 6 below), were in line with the Department's standards for grants and contributions programs.
Table 6: Operations and management costs of the Opportunities Fund
|Fiscal year||2015 to 2016||2016 to 2017|
|Operations and management Costs||$4,853,883||$5,070,951|
|Program expenditures (grants & contributions)||$34,826,882||$41,232,527|
|Operations and management costs as a percentage of program expenditures||13.9%||12.3%|
Source of data: Chief Financial Officer Branch; Performance Measurement, Analysis and Reporting, Employment and Social Development Canada, July 2017.
Note: data was only available for fiscal years 2015 to 2016 and 2016 to 2017.
3.6 Performance measurement strategy and quality of administrative data
More information could be collected on employers' involvement with the program, and on the progress made in the area of employer awareness. Progress made through Self-Employment activities, as well as the impacts interventions have on participants' skills and employability, require a fuller assessment by the Program. Administrative data were found to be of good quality, with minimal missing data.
Administrative data were found to be of good quality, while gaps were identified in relation to up-to-date participant contact information and information on employers in general
The Summative Evaluation of the Opportunities Fund for Persons with Disabilities that was conducted in 2014 noted a lack of consistency in the reporting of project numbers in the Common System for Grants and Contributions. The evaluation noted that pre-April 1, 2015 data still contained missing or questionable end dates and duplicate intervention records. Such issues were not found in the data collected after April 2015.
An area that could be improved upon is the participants' contact information fields. These were found to contain out-of-date, inaccurate or missing information. In particular, the administration of the survey of participants revealed that a high proportion of mailing addresses, telephone numbers and e-mail addresses were invalid, and that approximately half of participants did not have an email address.Footnote 128
Similarly, employer contact information and information on their experiences with the Program could be collected in a more systematic way. Almost all employers hiring participants through Wage Subsidy activities under the reformed Program do so through a Community Coordinator, and do not enter directly into funding agreements with the Department. Therefore, employers' contact information is now very rarely collected directly by the Program.Footnote 129 Having access to such information would facilitate the collection of data used for evaluation and performance measurement purposes, including feedback from employers who hired program participants through wage subsidy agreements or have received funding for workplace accommodations.
Changes in post-program labour market outcomes for participants' could be more accurately reflected in the Program's performance data
According to the Program's operational directives, for single year projects, participants' results can be updated up to 12 weeks after the end of their last intervention. As well, updates cannot be done later than 60 days following the project end date. Consequently, the situation of a single year project participant who becomes employed or self-employed only after the end of the reporting period (for example, four months after completing the Program) would not be accurately reflected in the Program's performance data.
Further, the Program's operational directives indicate that participants' results can only be updated if the initial result is ‘not employed' or ‘unknown.' Therefore, as soon as a participant is deemed employed, self-employed or returned to school, the Program does not update this participant's labour market situation, even if changes happen within the reporting period.
Finally, the review of regional and national project documents indicated that participant results may vary substantially from one project to another. It was also found that the results of participants in small scale single-year projects were often equal or better than those of some multi-year projects. The causes behind this trend could not be determined.
Further evidence is needed to assess the progress made through Self-Employment activities
The results showing the number of participants employed or self-employed at the end of an intervention are reported into a single category, and cannot be disaggregated. Further, the documents reviewed for projects comprising Self-Employment activities do not indicate whether or not participants have started to implement their business plans, or if those who did eventually proved successful.
Program activities' impact on participants' skills and employability needs to be better assessed
For performance measurement purposes, participants are considered to be more employable as soon as they have completed one program activity or did not complete their first program activity because they obtained employment.Footnote 130 Therefore, as soon as participants have received Enhanced Employment Assistance Services, they are recorded as participants with “enhanced employability,” whether or not they subsequently take part in Wage Subsidy activities, or become employed or self-employed.Footnote 131
A recipient organization noted that persons with disabilities may experience many unique situations that cannot necessarily be reflected in the categories used for performance measurement purposes under the Program. This organization pointed to a need for a more comprehensive database that effectively captures participants' transitions between interventions and path through the Program.Footnote 132
Key elements of the Program's reforms have not been reflected in the Program's performance measurement strategy
In order to strengthen performance measurement and to reflect the Program's reforms, the Department implemented a new participant information form in fiscal year 2016 to 2017. This new form notably contains a field that captures the small and medium-sized enterprise status of organizations hiring participants through Wage Subsidy agreements.
However, it is still not possible to determine whether or not participants were retained by their host employer once the subsidy ended. Such an indicator would help gauge the extent to which employers' needs are met, and the extent to which Wage Subsidy activities help persons with disabilities seize “meaningful mainstream employment opportunities” and “obtain permanent jobs.”Footnote 133
Overall, employment relationships created through the Program are discussed on an ad hoc basis. This is mainly done in the success stories and lessons learned sections of activity reports prepared by funding recipients. Finally, the new performance strategy identifies “the number of employers who indicate increased awareness” as an indicator of success for Employer Awareness activities. Presently, performance data are not collected in support of this indicator.
The results for participants of the Ready, Willing and Able project appear to be strong and monitoring those results more closely may be warranted
When the evaluation work was conducted, the latest activity report produced by the Canadian Association for Community Living indicated that a total of 1,447 employment outcomes, including 301 employment opportunities at less than 15 hours per week, have been generated through the Ready, Willing and Able initiative. This activity report also indicated that approximately 95% of the individuals who were hired through the initiative were retained by their employer.Footnote 134
These results are reported at an aggregate level by the recipient organization and are not validated by the Program. In comparison, the Department collects individual information on each participant through participant information forms, for all other projects except stand-alone Employer Awareness projects, which do not include participants. This process notably allows to confirm eligibility and constitutes a key source of performance related data.
Recommendations based on the evaluation findings are provided below.
- Continue enhancing employers' awareness of the valuable contribution that persons with disabilities can make and of the various supports available to facilitate their hiring and workplace integrationFootnote 135
- Explore the possibility of extending the availability of supports and follow-ups with participantsFootnote 136
- Further clarify the concept of social innovation and its application within the context of the Program, and establish specific criteria for assessment and monitoring purposesFootnote 137
- Better link the Program's logic model to the reforms and new policy directions of the Program, and continue improving indicators used for performance measurement purposesFootnote 138
Appendix A – Phase I evaluation questions
|Issues / Questions||Indicators||Key informant interviews||Document review||Data review||Limited cost analysis||Survey|
|Employers||OF funded projects representa-tives||Service Canada regional office staff||National Head Quarter program manage-ment|
|Relevance (continued need for program )|
|Relevance (alignment with government priorities)|
|Relevance (alignment with federal roles and responsibilities)|
|Program and policy design, delivery, implementation and effectiveness|
||Key informants' perceptions and documented evidence on progress towards:
||Key informant perceptions and documented evidence on:
|Performance (Efficiency and Economy)|
|In Support of Departmental Program Monitoring Activities|
Appendix B – Technical Studies Conducted in Support of Phase I of the Evaluation
TNS Canadian Facts Ltd. (2017), "Data Assessment Report for the Evaluation of the Opportunities Fund for Persons with Disabilities – Phase I", prepared for Employment and Social Development Canada (not published; available upon request)
This document provides information related to the assessment of the Opportunities Fund for Persons with Disabilities administrative data in order to inform the survey, and evaluation work undertaken and forthcoming in Phase II. While, Program administrative data were found to be of good qualityFootnote 140, the evaluation at this time could not address some Program outcomes and reforms due to limited Program performance data and supporting performance indicators and inability to access Canada Revenue Agency data during the conduct of Phase I.
TNS Canadian Facts Ltd. (2017), "Survey of Participants for the Evaluation of the Opportunities Fund for Persons with Disabilities – Phase I", prepared for Employment and Social Development Canada (not published; available upon request)
This report presents findings from a survey of Opportunities Fund participants who completed participation in the period April 1, 2015 to September 30, 2016 (study reference period). To collect more detailed performance information on participants from single year projects, a short online and telephone survey was conducted between January and April 2017 and measured basic labour market activity via participant intervention results for 2 indicators (1) clients employed and (2) clients returned to school, at 6 and/or 12 month post intervention. The survey achieved 358 completions and achieved a response rate of 44.4% calculated using Employment and Social Development Canada’s response rate formula. Survey findings can be used to reflect the experience of Program participants broadly but, they should not be used to infer the experience of participants from British Columbia and Quebec, nor participants with last interventions involving multi-year projects, who were not included in the sample of participants.
Employment and Social Development Canada (2017), "Document Review for the Evaluation of the Opportunities Fund for Persons with Disabilities – Phase I" (not published; available upon request)
A document review was conducted to address questions related to the relevance, design, implementation and early delivery of the Opportunities Fund and its progress towards Program outcomes. This document review focuses on a stratified random sample of 48 projects from fiscal year 2015 to 2016. The sample was proportionally similar to all Opportunities Fund projects in terms of representation by province/territory and by type of project (multi-year and single year). In total, roughly 500 project documents were reviewed. These included, for example, applications for funding, assessment reports, activity reports and close-out summary reports. In addition, the document review examined:
- the Terms and Conditions of the Opportunities Fund; the March 2008 Program Operational Guidelines, the May 2015 and February 2017 Program Operational Directives;
- the most recently approved Performance Measurement strategy for the Opportunities Fund, previous evaluations of the Program and other policy documents;
- speeches from the throne, budget speeches, Employment and Social Development Canada Departmental Performance Reports and Report on Plans and Priorities, relevant Statistics Canada publications and, academic publications.
The limitations of the document review are the inability to have reviewed all 126 Opportunities Fund projects delivered in fiscal year 2015 to 2016, the uneven quality of the project documentation produced by funding recipients and a lack of administrative data to address some Program outcomes. Further, at the time the report was prepared, the most recent Statistics Canada data source available on the labour market outcomes of people with disabilities was the 2012 Canadian Survey on Disability. As well, the results available for multi-year projects were preliminary. Finally, the relatively broad scope of the Program’s objectives, the presence of a large number of potential beneficiaries and that there are many other federal, provincial/territorial and Indigenous communities serving persons with disabilities, prevented the review from determining the Program’s reach.
Employment and Social Development Canada (2017), "Key Informant Summary of Findings in support of the Evaluation of the Opportunities Fund for Persons with Disabilities – Phase I" (not published; available upon request)
This document summarizes findings from 35 key informant interviews conducted among participant employers (8), single and multi-year project representatives (14), Service Canada staff (9) and Employment and Social Development Canada NHQ staff (4). Informants were selected based on their experience and high level of knowledge of the Program and ability to speak to Program reforms, progress towards expected Program outcomes, and delivery of projects.
The majority of interviews were conducted in April and May 2017 and were designed to be approximately 30 to 40 minutes in length. The challenge of the key informant interviews is that each group were prone to bias stemming from their stake in the Program. This was mitigated by the inclusion of 4 groups with varying degrees of separation from the Program, and the use of probing questions.
Qualitative findings from the key informant interviews represent the informed perspectives of the respondents. In exceptional circumstances a critical finding from 1 or 2 highly knowledgeable key informants may be presented, but findings overall are based primarily on the trends from multiple respondents and reported using the following relative weights:
- "all/almost all" – findings reflect the views and opinions of 90% or more of the key informants in the group
- "large majority/most" – findings reflect the views and opinions of at least 75% but less than 90% of key informants in the group
- "majority" - findings reflect the views and opinions of at least 51% but less than 75% of key informants in the group
- "half" – findings reflect the views and opinions of 50% of respondents in the group
- "some" - findings reflect the views and opinions of at least 25% but less than 50% of key informants in the group
- "a few" - findings reflect the views and opinions of at least two respondents but less than 25% of key informants in the group
Appendix C – Opportunities Fund Logic Model, 2016
Annex C – Opportunities Fund Logic Model, 2016
Appendix C illustrates the logic model of the Opportunities Fund for Persons with Disabilities, which outlines, from the top to the bottom, its inputs, activities, outputs, immediate outcomes, intermediate outcomes, ultimate outcome and the strategic outcome the program intends to achieve or to which it intends to contribute.
Funding available through contribution agreements
These inputs lead to the following program activities:
- provide employment services and interventions for persons with disabilities requiring assistance, skills and experience
- raise awareness of the employment barriers faced by persons with disabilities and promote their important contribution to the workforce
These program activities lead to the following outputs:
- completed employment services and interventions
- completed employer awareness projects
These outputs lead to the following immediate outcomes:
- persons with disabilities acquired labour market information, skills, work experience and/or received workplace accommodations
- enhanced employer awareness of the abilities and availability of persons with disabilities
These immediate outcomes lead to the following intermediate outcomes:
- persons with disabilities are employed, self-employed, or have returned to school
- more employment opportunities are available to persons with disabilities
These intermediate outcomes contribute to the following ultimate outcome:
- persons with disabilities are integrated into the labour market
This ultimate outcome contributes to the following strategic outcome:
- program Activity Architecture strategic outcome for employment supports: A skilled labour force and an efficient labour market.
Source: Employment and Social Development Canada, 2016, Performance Measurement Strategy
Report a problem or mistake on this page
- Date modified: