Evaluation of the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund

Executive summary

Historically, Aboriginal Canadians have experienced higher rates of unemployment, lower rates of labour force participation and higher rates of dependence on income assistance than the non-Aboriginal population. To address these challenges, Aboriginal labour market programs are available to increase workforce participation and help First Nations, Métis and Inuit people prepare for, find and maintain jobs.

The Aboriginal Skills and Employment Training Strategy (ASETS) is a five-year (2010-2015) Federal Government program designed to increase the number of Aboriginal peoples (First Nations, Inuit, Métis as well as status and non-status Aboriginal peoples living off-reserve) employed and integrated into the labour force. The program provides financial assistance to Aboriginal organizations through multi-year contribution agreements to support the costs of human resources development programs and services.

Launched in 2010, with funding to March 2015, the Skills and Partnership Fund (SPF) is a demand-driven, partnership-based contributions program that funds short-term projects contributing to the skills development and training of Aboriginal peoples (First Nations, Métis as well as status and non-status Aboriginal peoples living off-reserve), as well as employment. SPF is a separate but complementary program to ASETS.

Both ASETS and SPF are expected to increase the number of Aboriginal people who are employed and integrated into the labour market.

The ASETS and SPF evaluation covers program activities from April 2010 to January 2014, focussing on three strategic priorities: demand-driven skills development; partnerships; and accountability for improved results. A calibrated approach was adopted for this evaluation, which emphasized areas where knowledge gaps exist, such as ASETS and SPF partnerships and the extent to which demand-driven skills development has taken place, while incorporating multiple lines of enquiry and leveraging previous similar evaluation results where appropriate and necessary.

Findings of the evaluation

Program relevance

The evaluation findings demonstrate that ASETS and SPF are relevant. They are meeting the priorities of the Government of Canada, and there continues to be a demonstrable need for labour market programming for Aboriginal Canadians. The evidence indicates that both programs are working towards achieving their intended outcomes.

Partnership and engagement

Formal and informal labour market development partnerships were being established with training institutions and private and public sector employers. Evidence also demonstrated the likelihood that these partnerships are sustainable in the long term, once the funding agreement ends:

  • It is estimated that 3,500 (2,350 ASETS; 1,150 SPF) partnerships were created.
  • Overall, 71% of the partners were employers in either the private (40%) or public (31%) sectors.
  • The majority of partnerships that were created could be characterized as formal in nature. Formal partnerships with employers were more likely to result in a worker being retained after the funding period ends.
  • A majority of AAHs expected that their partnerships would continue. A greater number of ASETS partners (90%) than SPF partners (80%) were of the belief that the partnership with their AAH would continue.

While much work has been undertaken to develop partnerships, challenges remain. Staff turnover appears to be an ongoing problem, overburdening existing Aboriginal Agreement Holder (AAH) staff and managers due to a lack of internal resource capacity. In addition, the time-consuming nature of engagement at times affected the AAH’s ability to establish and maintain partnerships.

Demand-driven labour market programming

In an effort to address the gap in sustainable economic development identified in the previous evaluation (2009) of the Aboriginal Human Resource Development Strategy (AHRDS), one of the strategic priorities of ASETS was to ensure that skills development was demand-driven. The evaluation demonstrated that AAHs are taking a demand-driven approach and aligning their labour market programming with the labour market demands of their region:

  • Approximately eighty-one percent of employer partners stated that working with AAHs reflected a demand-driven approach to meeting skills development needs in their industry, province or area.
  • Ninety-two percent of partners felt that the program that was introduced in the workplace by the AAH was demand-driven.
  • Sixty-one percent of employer partners indicated that AAHs had targeted training to their particular occupational skills needs.

While in-demand occupations have been targeted, AAHs continue to face a number of challenges including: a lack of economic opportunities in remote communities; the volatile nature of the natural resources sector; low educational attainment and/or essential skills of the target population; and poor employer perceptions regarding the hiring of Aboriginal peoples. In addition, while employers and training institutions worked with AAHs to find local demand-driven employment opportunities and training options, the lack of accurate and up-to-date local labour market information impacted their ability to accurately forecast in-demand employment opportunities.

ASETS participant outcomes

The outcome analysis measured the difference in the average annual labour market outcomes of ASETS participants in the five years preceding the start of program participation and outcomes in the year following their program start year. Results from this outcome analysis should be interpreted with caution as they pertain to a relatively short period following the start of participation. Bearing this in mind, the evidence demonstrated that:

  • There was a 17 percentage point gain (+$1,621) in participant’s average annual employment earnings.
  • The incidence of employment increased by 5 percentage points between both periods.
  • The proportion of participants in receipt of EI benefits increased by 3 percentage points while the average annual amount of EI benefits collected increased by 37 percentage points (+$266), which implies an improved attachment to the labour market after program participation.
  • Participants also had small increases in their use of SA benefits.

ASETS/SPF incremental impacts

In an effort to calibrate the level of effort of the evaluation, evaluators summarized and used the results of the previous AHRDS and Aboriginal Skills and Employment Partnership Program (ASEP) evaluations. These results serve as viable proxies for ASETS and SPF respectively, as both AHRDS and ASEP were very similar in relation to the types of core programs and services that they provided to Aboriginal clients. Results from the incremental impact analysis demonstrated that overall, both AHRDS and ASEP were effective at increasing the employment earnings of participants and also had positive impacts on the incidence of employment of participants. Based on these observations, it is expected that ASETS and SPF would produce similar impacts.

Average cost

Using the most recent participant Action Plan result outcome indicators reported by AAHs in the Standard Data File, the cost analysis revealed that the average cost per client for ASETS was $6,122 and for SPF $9,469. With 42.3% of ASETS clients experiencing positive post-program outcomes as of 2012-13 (including employment/self-employment or returning to school), the average estimated cost per successful ASETS client was $14.7K. It should be noted that the latter cost is likely overestimated, as the evaluation was limited to conducting a one year post-program analysis. As a result, the outcomes of upwards of 21% of ASETS clients have yet to be determined as they were still enrolled in the program at the time of the evaluation. In the case of SPF, it is too early to develop an estimate of the cost per successful SPF client as many (46%) were still participating in the program at the time the evaluation was conducted.

Program data requirements and collection

Overall, AAH representatives were of the view that the administrative data supporting the outcomes analysis was generally of good quality. The technical nature of data requirements and case management systems proved to be challenging in the context of staff turnover (at Service Canada, Employment and Social Development Canada [ESDC] and within the agreement holders), as well as the reporting requirements of ESDC and other departments vis-à-vis the time spent on delivery of services to clients, are straining the abilities of AAHs to meet their obligations.

Recommendations

The Evaluation Directorate proposes the following recommendations which should be considered by ESDC in advancing the relevance and outcomes of ASETS and the SPF:

Recommendation #1: The Evaluation Directorate recommends that ESDC puts measures in place to encourage the adoption of formal agreements between Aboriginal Agreement Holders and employer partners, where appropriate and possible.

Recommendation #2: The Evaluation Directorate recommends that ESDC should develop strategies with Aboriginal Agreement Holders to strengthen the establishment and maintenance of partnerships, including addressing the capacity issues (training and guidance from national headquarters) faced by Aboriginal Agreement Holders, and dealing with poor perceptions amongst employers in the context of hiring Aboriginal people.

Recommendation #3: The Evaluation Directorate recommends that ESDC develop strategies to address expected outcomes where Aboriginal Agreement Holders serve remote communities. Remoteness issues to be addressed include, but are not limited to, the scarcity of economic opportunities and industries, and sector volatility.

Recommendation #4: The Evaluation Directorate recommends that ESDC should continue to ensure that continuous training is offered to Aboriginal Agreement Holders in the area of data collection and case management.

Recommendation #5: The Evaluation Directorate recommends that ESDC should continue to streamline data collection processes for Aboriginal Agreement Holders, including the adoption of consistent and common data reporting requirements, and avoid reporting overlap and duplication where feasible.

Recommendation #6: ESDC should develop a strategy for strengthening Aboriginal Agreement Holders access to timely labour market information to support their service delivery.

Management response

Introduction

The Skills and Employment Branch (SEB) and the Program Operations Branch (POB) would like to thank all those who participated in conducting this combined formative and summative evaluation of Employment and Social Development Canada’s (ESDC) Aboriginal Skills and Employment Training Strategy (ASETS) and Skills and Partnership Fund (SPF) programs. In particular, SEB and POB (the program) acknowledges the contribution of the numerous partner organizations and funding recipients who participated as key informants. The program agrees with the evaluation findings and proposes the following Management Response.

Recommendations and planned follow-up actions

Recommendation #1

The Evaluation Directorate recommends that ESDC puts measures in place to encourage the adoption of formal agreements between Aboriginal Agreement Holders and employer partners, where appropriate and possible.

Explanation

The program agrees that formal and informal partnerships established in the delivery of both ASETS and SPF are demonstrating positive outcomes in terms of supporting the attainment of critical skills experience to enhance participant employability, and lead to employment. As a result, ESDC will continue to require that agreement holders establish and maintain formal partnerships with employers, training institutions and others.

Applicable lessons learned as set out in this and previous Aboriginal labour market evaluations will be integrated in the new post-2015 program design, delivery and monitoring/measurement activities.

Specifically, ESDC proposes to strengthen partnerships for future training-to-employment agreements by modeling them on the best practices seen under the Aboriginal Skills and Employment Partnership (ASEP) program. Under ASEP, Aboriginal organizations and project partners formed a legally incorporated non-profit organization to administer the project. This model, with enhanced participation from ESDC, would increase involvement from all parties due to greater stake in the outcomes. Where formal partnerships are not feasible, ESDC supports the use of informal partnerships as they have been identified as supporting successful employment outcomes.

Recommendation #2

The Evaluation Directorate recommends that ESDC should develop strategies with Aboriginal Agreement Holders to strengthen the establishment and maintenance of partnerships, including addressing the capacity issues (training and guidance from national headquarters) faced by Aboriginal Agreement Holders, and dealing with poor perceptions amongst employers in the context of hiring Aboriginal people.

Explanation

The program agrees with this recommendation and acknowledges that partnership support is an ongoing, value-added activity that ESDC and Service Canada can offer funding recipients. Attention will be given to further leveraging the existing support offerings including the partnership section of the Aboriginal Labour Market Bulletin that is developed by ESDC for agreement holders; obtaining feedback about and, as required, enhancing the Partnership Handbook; and developing or identifying tools and new resources to support the creation and maintenance of formal partnerships; including the monitoring of partnership effectiveness. Key partnership support priorities and related strategies will be developed and identified as part of the post-2015 Aboriginal labour market program design and implementation.

Specifically, ESDC will play a more active role in brokering partnerships between Aboriginal agreement holders and employers. The proposed non-profit organizational structure will contribute to this goal. There will also be greater ESDC coordination with employers and Aboriginal agreement holders to help market their successes, demonstrate value-added, and encourage informed decision making as part of the partnership-building process.

Recommendation #3

The Evaluation Directorate recommends that ESDC develop strategies to address expected outcomes where Aboriginal Agreement Holders serve remote communities. Remoteness issues to be addressed include, but are not limited to, the scarcity of economic opportunities and industries, and sector volatility.

Explanation

The program agrees that addressing issues identified in this recommendation are critical to facilitating the achievement of employment and increased employability of Aboriginal people participating in the Government of Canada’s Aboriginal labour market programs. These issues were the subject of stakeholder (National Aboriginal Organization’s, AAH’s, Private Sector, Province and Territories) engagements meetings and electronic surveys that took place during 2013-2014 and will be addressed as part of the post-2015 Aboriginal labour market policy and program design.

In addition, the program will, in post-2015 program development, seek to identify ways to mitigate issues with respect to the high costs of program delivery in remote communities. A study was commissioned by ESDC in March 2008 Footnote 1 to provide insights into the nature of the cost/price differentials facing agreement holders and sub-agreement holders in northern and remote communities and the impact of these differentials on their operations and service delivery. The study confirmed that there are significant cost differences among northern and remote agreement holders and southern and non-remote agreement holders. It also noted that the lack of accessibility and dispersed points of service increases costs and funding allocation should be more equitable by taking into account higher cost factors.

Consideration will be given to remoteness and transportation challenges, and the cost of doing business in the North, when employment targets are set. Initial targets set in remote areas are often tied to one or two opportunities that may be subject to delay or cancellation, lower than anticipated human resource requirements, or challenges with the length of time needed for employment readiness of the client cohort.

Specifically, ESDC is proposing to work more closely with agreement holders to identify challenges and help shift the focus to identifying alternative employment sources or re-adjust targets as necessary.

Recommendation #4

The Evaluation Directorate recommends that ESDC should continue to ensure that continuous training is offered to Aboriginal Agreement Holders in the area of data collection and case management.

Explanation

The program agrees that ongoing, relevant support to funding recipients is necessary to ensure that data collection and case management practices are effective, and to this end, client data is accurate and provided in a timely manner. This will include continuing to provide technical support directly to funding recipients as required, and developing and distributing client data management guidance to funding recipients in time for the roll out of Aboriginal labour market program post-2015.

Recommendation #5

The Evaluation Directorate recommends that ESDC should continue to streamline data collection processes for Aboriginal Agreement Holders, including the adoption of consistent and common data reporting requirements, and avoid reporting overlap and duplication where feasible.

Explanation

The program agrees that an ongoing commitment to streamlining reporting requirements of funding recipients would bring program and funding recipient closer to a mutually beneficial understanding of performance monitoring. Used judiciously to monitor the achievement of intended outcomes, reporting can be a critical factor in ensuring cost effective, high impact programming. In support of this, Skills and Employment Branch and Program Operations Branch will meet annually (Q1 of each fiscal year) with key departmental stakeholders to review the ongoing relevance of program performance measurement strategies, data collection instruments, client data management and reporting guidance, internal directives, and quality assurance practices. Adjustments and updates be made accordingly.

Recommendation #6

ESDC should develop a strategy for strengthening Aboriginal Agreement Holders access to timely labour market information to support their service delivery.

Explanation

The program agrees that timely and relevant labour market information to inform strategic and operational decision making is a valuable contribution that the department can make to Aboriginal organizations undertaking labour market programming. In 2014-2015, SEB and POB will seek formal feedback from users on the relevance and utility of the Aboriginal Labour Market Bulletin (ALMB). The ALMB is the current programming tool that provides labour market and sector-specific information to Aboriginal organizations involved in labour market programming. The ALMB will continue to be distributed directly to AAH’s when it is published to ensure they are kept aware of labour market issues and trends. ESDC will continue to encourage agreement holders, through the ALMB and meetings, to access ESDC’s Job Bank, Job Match and Job Seeker websites, which provide timely information regarding employers who are seeking to hire workers across Canada. ESDC will also explore other avenues for obtaining enhanced labour market information specifically related to Aboriginal peoples and the labour market to assist in future labour market programming.

1. Introduction

This report offers a summary of the findings and conclusions from the evaluation of the Aboriginal Skills and Employment Training Strategy (ASETS) and the Skills and Partnership Fund (SPF) programs. The evidence presented in this report is drawn from evidence collected through multiple lines of qualitative and quantitative enquiry and from previous evaluations of similar programs. The execution and the results of the evaluation are presented in six sections as follows:

  • Section 1: provides a description of ASETS and SPF.
  • Section 2: describes the scope, methodologies and limitations of the evaluation.
  • Sections 3, 4, and 5: present the main findings regarding Relevance and Performance (Effectiveness, Efficiency and Economy).
  • Section 6: summarizes the main conclusions, and key recommendations.

1.1 Profile of programs

ASETS is the successor program to Employment and Social Development Canada’s (ESDC) Aboriginal Human Resources and Development Strategy (AHRDS). Officially launched in April 2011, after a one-year transition period, ASETS is a Federal Government program designed to increase the number of Aboriginal people (First Nations, Inuit, Métis, as well as status and non-status Aboriginal peoples living off reserve) employed and integrated into the Canadian labour force. The program places an emphasis on: the development of demand-driven skills; fostering partnerships with the private sector and other levels of government; and accountability for improved results.

ASETS funding is provided through contribution agreements to Aboriginal organizations, including Modern Treaty Self-Governing organizations. The multi-year funding total for ASETS is $1.68B over five years (April 1, 2010 to March 31, 2015). Funding is provided in order to design and deliver demand-driven labour market programs, as well as related services and supports such as childcare and special supports for persons with disabilities. As ASETS emphasizes a partnership approach to addressing Aboriginal employment-related needs, stakeholders include partner organizations participating in initiatives under the strategy, including other levels of government, the private sector and/or voluntary sector organizations.

SPF is a demand-driven, partnership-based contribution program that supports government priorities (federal/provincial/territorial) and partnerships by funding short-term projects (i.e. from 1-3 years) contributing to the skills development, training as well as employment of Aboriginal peoples (First Nations, Inuit, Métis as well as status and non-status Aboriginal peoples living off-reserve). The successor to the Aboriginal Skills and Employment Partnership Program (ASEP), the SPF program was launched in 2010 with an investment of $210M over five years (its current end date is March 31, 2015).

As with ASETS, the ultimate goal of the SPF is to increase the number of Aboriginal people who are employed and integrated into the labour market. The SPF supports this goal by enabling Aboriginal organizations to experiment and be innovative by using partnerships to create labour market products and services for Aboriginal people. Innovation in this context is defined as “new” systems, supports, practices or clients that an organization (i.e., project holder) has not previously had the opportunity to “test” or use in its unique organizational environment. Funding is provided to ASETS Aboriginal agreement holders and other Aboriginal organizations, and similar to ASETS, all SPF projects must be partnership-based (including other levels of government, the private sector, educational institutions, and other Aboriginal organizations).

2. Evaluation scope and methodology

The following section details the evaluation scope, as well as the methodologies used to perform the joint SPF and ASETS evaluation fieldwork. The constraints and limitations that impacted the evaluation findings and conclusions are also presented.

2.1 Evaluation scope

This evaluation is written in accordance with Treasury Board Secretariat (TBS) Policy on Evaluation (2009) and reports on the core issues of relevance and performance (effectiveness, efficiency and economy). The evaluation covers ASETS and SPF activities from April 1, 2010 to January 2014. A calibrated approach was adopted for this evaluation, adjusting for the level of effort where knowledge gaps exist, and by leveraging previous evaluation results where necessary. The evaluation focuses on three strategic priorities: demand-driven skills development; partnerships; and, accountability for improved results. The evaluation is outcomes based and draws on findings from previous incremental impact studies on Aboriginal labour market programming to assess overall performance.

2.2 Methodology

The evaluation incorporated multiple lines of enquiry, and underwent an extensive peer review process. Data collection activities were carried out between June 2013 and January 2014. The evaluation methodology consisted of:

  • Sixty-eight interviews with Aboriginal Agreement Holders (AAHs);
  • twelve interviews with program officials;
  • a telephone survey of 250 ASETS and SPF employer/educational partners; and
  • a document and file review, including reporting and monitoring documents from 59 SPF and 30 ASETS AAHs.

In addition, client profiles were developed by linking ASETS 2010 client administrative data to Employment Insurance (EI) and Canada Revenue Agency (CRA) data, in an effort to measure program outcomes.

2.3 Constraints and limitations

Ideally, the evaluation would examine the net impact benefits of ASETS/SPF on employment, earnings, social assistance and EI receipt three to five years post-program. However, only ASETS administrative data for 2011-2012 and 2012-2013, and SPF administrative data for 2012-2013 were available. The lack of program administrative data with a sufficient post-program follow-up period, as well as the availability of EI and CRA data for 2011 only, meant that there was no possibility of conducting a rigorous cost-benefit analysis as part of this evaluation. As a result, the post-program impact analysis was limited to conducting a profile of participants and a one-year post-program outcome analysis. The latter complicates the analysis of SPF outcomes in particular, given that the majority of employment results were not expected until at least 2014 (with almost half not expected until after 2015). Future evaluations will be in a better position to examine ASETS/SPF outcomes and net impacts when additional post-program data and CRA data become available in 2015-2016 and 2016-2017. In an effort to mitigate the inability to conduct a net-impact analysis of ASETS/SPF outcomes, the findings from the previous evaluations of the AHRDS and ASEP were used as proxies, as AHRDS and ASEP are very similar programs in terms of services offered and types of clients served.

3. Relevance

The evaluation found that there continues to be a demonstrated need for labour market programming for Aboriginal Canadians and ASETS/SPF are meeting the priorities of the Government of Canada. Historically, Aboriginal Canadians have experienced higher rates of unemployment, lower rates of labour force participation and higher rates of dependence on income assistance than the non-Aboriginal population. Evidence from this evaluation points to ASETS and SPF efforts to overcome these challenges and reflect the Government of Canada priority of employing people in occupations with a high demand for labour. ASETS and SPF align with the roles and responsibilities of the Government of Canada as they fall under the mandate of the Department of Human Resources and Skills Development Act (S.C. 2005, C.34).

4. Performance – effectiveness

This section summarizes the key performance findings of ASETS and SPF, with a particular focus on the effectiveness of partnerships and demand-driven skills development. Footnote 2 It includes a discussion of the attainment of employment outcomes, and presents an explanation for attributing the success of ASETS and SPF Employment Benefit and Support Measures (EBSM) interventions. In some cases it was necessary to use previous impact evaluation findings as proxies in examining the success of ASETS and SPF.

4.1 Labour market programming, employer engagement and partnerships

Labour market programming

One of the objectives of ASETS/SPF was for AAHs to align their programs with the labour market demands in their region. This was considered a key pillar of these programs, demonstrating that a fundamental shift to a more demand-driven model has taken root. Approximately 81% of employers stated that working with AAHs reflected a demand-driven approach to meeting skills development needs in their industry, province or area. In addition, 92% of overall partners surveyed (95% of SPF partners and 89% of ASETS partners) were of the view that the workplace programs introduced by the AAH through their involvement were demand-driven. This includes 96% of all employer partners.

The evaluation found that AAHs are targeting their training programs or services to the types of occupational skills needed by employers or industry. For instance, 61% of employers surveyed indicated that their partnerships with AAHs had resulted in training that had been targeted to the employer’s particular occupational skill needs. In addition, all SPF and ASETS/SPF agreement co-holders, as well as the majority of ASETS agreement holders indicated that training and skills development were tied directly to available in-demand occupations. When programs were implemented, 72% of employer partners indicated that they met all or most of their skill needs. Almost all partners noted an increase in the partnerships’ effectiveness, such as job creation/employment of Aboriginal people, and the provision of training, when the partnership is targeted to the partner organization’s and/or industry’s needs.

While much work has been undertaken to target occupations in demand, AAHs continue to face the following employment challenges:

  • There are fewer economic opportunities or a narrow array of industries to support employment in isolated or remote communities. In some instances, there may be just a single sector to target near a community and oftentimes, Aboriginals were reluctant to leave their communities for employment opportunities. In Nunavut, for example, the lack of diversification of the economy, as well as the lack of mining sites in proximity of reserves constitute major barriers to building partnerships, training programs and employment.
  • Economic downturns, the seasonal nature of the work, market fluctuations or dealing with volatile sectors such as oil, gas, and mining can impact on employment commitments, with employers hesitant to commit to long-term, permanent employment.
  • Low educational attainment of the target population relative to the need by some employers for highly-skilled workers (engineers, for example) can make it difficult to match individuals to occupations that are in-demand. The development of such highly-skilled workers is often outside the realm of the training support provided by ASETS and SPF.
  • Lack of accurate and up-to-date local labour market information makes it difficult to accurately forecast in-demand employment opportunities.
  • The short timeframe available to set up the agreements, mobilize their office and begin the programming was also cited by a few SPF agreement holders as hindering their ability to adequately start and maintain a project.
  • Clients with multiple barriers to employment, including a lack of essential skills (life and foundational skills) required culturally appropriate holistic approaches and services, including pre-employment training (literacy, safety, interview guidance, etc.). Addressing these barriers often require multiple interventions delivered over an extended period.
  • Poor employer perceptions, including discrimination, misinformation, cultural misunderstandings, and other related workplace issues specifically in the context of hiring Aboriginal peoples still persist among some employers. According to agreement holders, employers can be unwilling to commit to providing work experience to clients if they are unfamiliar with the qualifications of First Nations people and/or have negative perceptions as to their suitability as workers. Such perceptions can be difficult to overcome, as some employers may be reluctant to become informed.

Employer engagement

Securing the participation of employers is a critical step to increasing Aboriginal peoples’ participation in demand-driven skills development programming. Success is demonstrated through employer support and by the proportion of employers participating in the program that are linked to current labour market demands and/or skills shortages.

The evaluation evidence found that private and public sector employers and training institutions are aware and engaged in the development of labour market programs and services by AAHs. Seventy-one percent of partners overall, are aware of programs and services developed by AAHs, with public (77%) rather than private (61%) sector employers being more aware of the programs and services being developed. Of the 71% of partners who reported being aware of the programs and services developed by the AAHs, the majority (51%) reported being engaged by the AAHs in their development. Early engagement of partners in the design of a project, program or service is important as it allows partners to significantly shape the structure, goals and activities of the project, thereby creating buy-in and allowing partners to link these projects to their particular occupational needs or requirements. Of those partners who reported engaging AAHs in the design and development of labour market programs or services, almost half stated these programs or services had been introduced in the workplace. The implementation of a program or service is more likely for ASETS partners (61%) compared to SPF partners (40%). It was also more likely to be the case with public sector employers (65%) than with private sector employers (29%).

The following are the most common ways in which partners were engaged:

  • Employers and to a lesser extent training institutions:
    • Worked together with AAHs to develop or modify existing training curricula or delivery mechanisms.
    • Assisted with research and labour market information to identify local demand-driven employment opportunities and training options.
  • Employers provided feedback on required soft skills and essential skills, and AAHs responded by concentrating on and developing programs and services to support these skills.

In cases where partners were not engaged in the development of labour market programs and services, it was because they had pre-established training programs in place; wanted specifically-qualified individuals to employ; or didn’t want to participate in the administration of training.

Financial and in-kind contributions leveraged from partners created effective partnerships that lead to job creation/employment of Aboriginal people and the provision of training. In fact, surveyed employer partners who made an in-kind or a financial contribution were more likely to retain a greater number of workers after funding ended, when compared to employers who did not make a contribution. In addition, AAHs believed they strengthened their partnerships by ensuring partners were invested in the program or project (in fact, the leveraging of cash or in-kind contributions is a mandatory requirement of SPF partnerships). Most employer partners supported this view, indicating that it increased the effectiveness of the partnership with the AAH.

Key findings on financial and in-kind contributions as reported by surveyed project partners are as follows:

  • Forty-four percent of employer partners contributed financially towards their AAH partnership (48% ASETS and 37% SPF) while 49% of employer partners provided in-kind contributions (48% ASETS and 50% SPF).
  • The average financial contribution was $41K ASETS and $129K SPF and the average value of in-kind contribution was estimated at $41K ASETS and $102K SPF. In-kind contributions identified by surveyed partners included: volunteer time, including administration, providing input, training, hosting meetings, etc. (52% of respondents); absorbing the costs for inputs such as a trainer, counsellor, travel costs, wages, etc. (28%); the provision of office space, training facility, etc. (21%); and the provision of equipment (19%).
  • Public sector employers (56%) were more likely to provide a financial contribution than private sector employers (42%). Similarly public sector employers (68%) were more likely to provide an in-kind contribution than private sector employers (34%).
  • Overall, surveyed partners who indicated having a written agreement with AAHs were far more likely to report that they had made a cash contribution (52.1%), when compared to those partners who indicated having no written agreement in place (18.3%).

One of the objectives of developing partnerships as a result of the funding agreements was long-term sustainability, after the funding period ended. A majority of AAHs interviewed were of the view that their partnerships would continue. This was also true of 90% the partners surveyed. A greater number of ASETS partners (96%) than SPF partners (80%) expected that the partnership with their AAH would continue.

While much work has been undertaken by AAHs to develop partnerships, challenges remain. The majority of key informants cited capacity issues, both in terms of time and resources, as hindrances to the development and maintaining of partnerships with employers. For instance, agreement holders noted that creating and supporting partnerships required ongoing time and effort, and involved continual communication, marketing, and monitoring which, combined with staff turnover, was overburdening AAH staff and managers. Informants noted that it takes time to build trust with potential employer partners, citing challenges to partnership commitments including: labour market fluctuations that change employee requirements; poor perceptions of Aboriginal workers due to past experiences or preconceived stereotypes; and previous poor relationships with First Nation communities/tribal councils. These issues were also identified as a constraint to the success of the Aboriginal Skills and Training Strategic Investment Fund Footnote 3 (ASTSIF).

Labour market development partnerships

A key activity was the establishment of active labour market development partnerships with public and private sector employers and training institutions. AAHs focused on responding to labour market opportunities by providing clients with the necessary skills and/or meeting current labour market needs. Partnerships were designed to help improve capacity, diversify the funding base for the delivery of ASETS programs and contribute to the ongoing sustainability of the program.

AAHs established partnerships with employers and training institutions. It is estimated that 3,500 (2,350 ASETS; 1,150 SPF) formal and informal partnerships were created. As no previous calculations of the number of partners created with the assistance ASETS/SPF or AHRDS/ASEP funding exist for benchmarking purposes, this estimate may serve as a point of reference for future evaluations. Overall, 71% of the partners were employers in either the private (40%) or public (31%) sectors. SPF (76%) had more employer partners than ASETS (67%). Private sector employers comprised national, regional and local or community-owned companies. About one-third of the private sector employers were in the natural resources sector, including mining, quarrying, oil and gas extraction, agriculture, forestry, fishing and hunting. An additional third were from the construction, and accommodation and food services sectors. Remaining sectors included health care and social assistance, arts, entertainment and recreation, etc.

The majority of partnerships that were created could be characterized as formal in nature. Formal partnerships have written agreements in place that define the purpose of the partnership, as well as the expected roles and responsibilities, and expected outcomes. Informal partnerships are collaborations that seek to work towards compatible goals which are not governed by official written agreements. The ESDC summative evaluation of the ASEP program, released in 2013, recognized the importance of written commitments between partners and AAHs. Overall, the majority of ASETS and SPF partnerships were formal in nature in that they had written agreements. In cases where financial support was provided by AAHs to assist employer partners hire Aboriginal workers, formal partnerships between AAHs and employers were more likely to result in a greater number of workers being retained after funding ended, when compared to informal partnerships. Evaluation evidence confirmed that having formal partnerships is a best-practice.

In some cases, informal agreements with partners were favoured as they reduce delays stemming from board/band council approval processes and requirements to seek legal counsel; for their flexible nature regarding commitments (i.e. uncertainties from employers surrounding the timing and quantity of employment opportunities); and from the perspective of smaller communities, the perception that informal agreements convey greater trust between parties.

The evaluation evidence, as presented in this report, illustrates other elements of effective partnerships such as:

  • Early and frequent communication and engagement between parties.
  • Successful partnership development is a long-term undertaking. It can take time to build relationships, trust between parties, and establish buy-in.
  • Mutually beneficial partnerships are based on collaboration and follow through to ensure results and success.

ASETS is similar to the Labour Market Development Agreements (LMDAs) and other ESDC Aboriginal Labour Market programming such as ASEP, ASTSIF and SPF. Evidence from LMDA evaluations demonstrated that programming is complementary to the extent an Aboriginal client can be assisted by more than one source related to different aspects of their return-to-work-action plan. ASETS was also found to be complementary to ASEP and ASTSIF because each program provided specialized program delivery. Case management processes were in place between AHRDS/ASETS and ASEP to avoid overlap. A good portion of ASETS AAHs who are aware of provincial Labour Market Agreements (LMAs), tend to collaborate and make use of LMA funding for research, hosting conferences and to provide joint training. A key factor positively influencing collaborations was close proximity to provincial and territorial LMA service points. In contrast, a majority of SPF AAHs noted that they do not work with provincial LMAs. Those who reported not collaborating with LMAs cited the following reasons, including: the lack of inclusion of ASETS in provincial strategies; the perceived lack of opportunity to provide input or become involved in the LMAs; and friction resulting from the duplication of services, leading to a competition for clients.

Finally, the majority of AAHs interviewed believed that their partnerships were effective. Similarly, 91% of employer and educational partners surveyed also viewed their involvement with ASETS and SPF AAHs as effective.

4.2 Participant profile

This section provides a description of the characteristics of clients who participated in ASETS and SPF.

Participant profile

The participant profile analysis is based on all individuals who started participation in ASETS or SPF between April 2010 and July 2013. This represents 67,568 ASETS participants and 3,303 SPF participants. Appendix 1 presents a more detailed breakdown of the socio-demographic characteristics of ASETS and SPF participants per principal EBSM. In summary, the profile of the socio-demographic characteristics of participants illustrates that the majority of ASETS and SPF clients were members of a First Nation, Metis or Inuit community. More males than females participated overall (SPF at 63% and ASETS at 53%). Just over half of all clients were 30 years of age or under. Sixty-eight point five percent of ASETS clients were EI non-claimants, 21.6% former EI claimants and 9.9% were active EI claimants. Comparatively, 66.5% of SPF clients were non-claimants, 22.5% former claimants and 11% active claimants. Within the last five years prior to participating in either an ASETS or SPF intervention, just over half have taken part in an intervention from the Aboriginal Human Resources Development Strategy (AHRDS) Footnote 4 and over one third have participated in an intervention under a provincial/territorial Labour Market Development Agreement (LMDA) Footnote 5 .

4.3 ASETS participant outcomes

The following (Table 1) presents an analysis of participant’s labour market outcomes 5 years prior to program participation, in the program start year and in the year following the program start year for the 2010 ASETS clients in all intervention types.

Table 1: Labour market outcomes of 2010 ASETS clients
Pre-program2 Program start year Year after program start year3 Change from pre-program to year after the program start year
Earnings1 $9,458 $9,477 $11,079 $1,621***
Proportion employed 59% 62% 64% 5pp***
Proportion on EI 12% 19% 15% 3pp***
EI benefits $713 $1,378 $979 $266***
Number of weeks on EI 2.28wk 4.07wk 2.87wk 0.59wk***
Proportion on SA 15% 18% 16% 1pp***
SA benefits4 $858 $1,106 $977 $120***
Dependence on income support5 15% 23% 18% 3pp***

1 Earnings includes individuals who reported no earnings in a given year.

2 Average annual outcome pre-program: Represents the average annual outcomes over the five year pre-participation period.

3 Annual outcomes in the year following the program start year. This period may represent the second year of participation for some participants or the first year post-program for other participants.

4 SA benefits include all Federal/Provincial/Territorial income supports.

5 The percentage dependence on income support was calculated by dividing the total EI and SA supports received by EI, SA and total income earnings.

* p> .1, **p > .05, *** p > .01.

N.B Earnings, EI benefits and SA benefits are not adjusted for inflation; pp = percentage points.

As shown in Table 1, there was a gain of $1,621 in participant’s average annual employment earnings between the pre-program period and the year following the program start year. As well, incidences of employment increased by 5 percentage points between both periods. However, the proportion of participants in receipt of EI benefits increased by 3 percentage points and the average annual amount of EI benefits collected increased by $266. This increase in EI use may be attributed to the experience of uninsured claimants Footnote 6 . Uninsured claimants are individuals who are not working and ineligible to collect EI. They would have participated in an intervention, and then found employment of long enough duration to become eligible for EI benefits. While their subsequent unemployment led to in an increased proportion on EI benefits overall, this increase implies an improved attachment to the labour market after program participation.

Participants also had small increases in their use of SA benefits. This is shown by an increase of 1 percentage point in the proportion of participants in receipt of SA benefits and by an increase of $120 in the amount of SA benefits collected. As well, the level of dependence on income support increased by 3 percentage points.

Overall, results from this outcome analysis should be interpreted with caution as they pertain to a relatively short period following the start of participation. Specifically, the year following the program start year still represents the in-program period for some participants while it likely represents a transition period for the individuals who had completed their participation. Nonetheless, the gains found in the incidence of employment and in the employment earnings provide indications that 1) an increased proportion of ASETS participants were able to find employment following their participation and 2) those who found work were able to improve their earnings by either working more hours or having a better paying job.

Survey results further support these findings suggesting that clients who had taken training or other programs provided by an AAH are employed. Forty percent of employer partners (45% for ASETS and 31% for SPF) hired participants. Employers hired on average 8.5 participants, the majority of which (73%) hired between 1 to 5 participants.

4.4 Incremental impact analysis: AHRDS and ASEP evaluation results as proxies for ASETS and SPF

Because of a lack of administrative data with a long enough follow-up period, evaluators were not able to conduct an analysis of incremental impacts for ASETS and SPF; consequently the decision was made to utilize the results of the previous evaluations of AHRDS and ASEP. These two programs were very similar in relation to the core programs and services that they provide to Aboriginal clients. They offer a similar set of EBSM/EBSM like interventions to clients with many of the same characteristics and in comparable social conditions. In fact, over 50% of the ASETS clients had previously taken part in an intervention from AHRDS. Furthermore, 78 of the 79 former AHRDS AAHs continued their service delivery under the newly formed ASETS. Within this context, it is reasonable that the results of AHRDS and ASEP serve as viable proxies for ASETS and SPF.

The key results from the AHRDS evaluation (2009) were as follows:

Skills Development (SD) — (37% of all participants).

Participation in AHRDS increased the incidence of employment and earnings for active claimants and non-EI eligible clients and reduced their use of Social Assistance (SA) and dependence on government income support. Non-EI eligible clients increased their EI use in the full post-program period, reflecting an increased EI eligibility based on post-program employment activities. Former EI claimants experienced an increased incidence of employment but lower earnings and mixed results for EI benefits and dependence on income support.

Targeted Wage Subsidies (TWS) — (4% of all participants)

Participation in AHRDS increased the incidence of employment and earnings for all clients. Former EI claimants and non-EI eligible clients increased their EI use through insurable earnings and reduced the use of SA.

Self-Employment (SE) — (1% of all participants)

Participation in AHRDS resulted in a decline in the incidence of employment and earnings for Active and Former EI claimants.

Job Creation Partnerships (JCP) — (4% of all participants)

Participation in AHRDS increased the employment earnings for Active EI claimants only. There was also an increase in the incidence of employment for all client groups. Former EI claimants and non EI eligible clients increased their EI use and reduced the use of SA.

Employment Assistance Services (EAS) — (18% of all participants)

Participation in AHRDS increased the incidence of employment in the short term and employment earnings for Active EI clients and non EI eligible clients.

The ASEP incremental impact analysis serves as a proxy for SPF clients, because the two programs share a project-based nature, dealing with both broad and specific industry partners.

Key results from the analysis of ASEP incremental impacts include:

  • ASEP increased the employment earnings of participants. The cumulative incremental gains in employment earnings over the four year period following the start of participation were in the $3,567 to $5,170 range. ASEP also had positive impacts on the incidence of employment of participants.
  • The incremental earning gains were higher for participants in projects focusing on a specific industrial development (such as mining) than for participants in projects with a broader industry focus (e.g. construction or forestry).
  • Incremental impacts were also assessed using a different reference group composed of Aboriginal individuals who received employment benefits as part of their Aboriginal Human Resource Development Agreements (AHRDAs) in order to test if the particular focus of ASEP on industry, more direct involvement of the private sector in the programming, and matching of program delivery to the labour market produced better outcomes than the delivery model used by AHRDAs, which is less demand driven. Relative to the AHRDA reference group, ASEP participants had a statistically significant cumulative gain in employment earnings of $2,939 over the four year period following the start of participation. These results provide indications that the ASEP approach was more effective than the AHRDA approach at helping the participants who worked after participation to improve their employment situation by either getting jobs with better pay or by working more hours than before their participation.

Overall, the results of the incremental impact analysis of AHRDS and ASEP demonstrated that both programs were effective at increasing the employment and earnings of participants. Given the similarity between AHRDS/ASEP and ASETS/SPF, these results are suggestive of similar findings for ASETS and SPF clients.

5. Performance – efficiency and economy

Key findings of the evaluation with respect to the cost analysis of ASETS and SPF programming are summarized in this section.

5.1 Cost-benefit analysis

Evaluators were unable to conduct a cost-effectiveness analysis for ASETS and SPF. However, it was possible to calculate the average cost per participant and refer to past evaluation conclusions representative of the cost-benefit ratio of ASETS and SPF.

Cost-benefit

As was demonstrated in Section 4.3, the AHRDS evaluation can serve as a proxy for the ASETS cost-effectiveness analysis. The AHRDS evaluation found that EAS and TWS interventions for active EI claimants, EAS for former EI claimants and EAS for non-claimants had the best cost-benefit ratio. Table 2 below describes the ratio of the cost to the department for each dollar earned by the clients. The lower the number in the table, the lower the cost is to the Department, hence the greater the cost-benefit ratio.

Table 2: AHRDS cost per earnings impact
Earnings impact by client type: Cost per impact by principal EBSM:
SD TWS SE JCP EAS
Active claimants $1.65 $0.94 - $4.58 $0.18
Former claimants - $1.56 - - $1.11
Non-claimants $1.65 $3.13 - - $0.64

Source: AHRDS Summative Evaluation Report, 2009.

The ASEP evaluation was used as the proxy for SPF. The benefit of participating in ASEP measured over the four year period totaled $3,120 (refer to the following Table 3). In comparison, the total average cost of delivering ASEP was $16,484 per participant from the social perspective and $9,705 per participant from the governmental perspective Footnote 7 . The costs and benefits of ASEP, a project-based program, were also compared to those of AHRDS, a broad-based program. From a social perspective, participating in ASEP leads to greater employment earnings compared to participating in AHRDS (incremental benefit of $2,833). However, ASEP cost $10,102 (incremental cost) more per participant to deliver than AHRDS. Therefore, from a social perspective, the incremental cost to deliver ASEP compared to AHRDS was $7,269 greater than the incremental benefit. But from a government perspective, the incremental or additional cost ($3,323) of delivering ASEP compared to AHRDS was slightly more ($490) than the incremental or additional benefit ($2,833).

Table 3: The incremental benefit and cost of delivering ASEP
Cost benefit perspective Benefit Cost Difference
Cost and benefit of ASEP
Social perspective $3,120 $16,484 -$13,364
Government perspective $9,705 -$6,585
Incremental cost and benefit of ASEP relative to AHRDS
Social perspective $2,833 $10,102 -$7,269
Government perspective $3,323 -$490

Source: ASEP Summative Evaluation Report, 2013.

Participant cost analysis: Comparisons to previous evaluation results

A simple cost analysis, based on ESDC contributions, was undertaken to calculate the average participant cost and average cost per employed client. The AHRDS evaluation findings served as the most appropriate comparison for ASETS. The average cost per client for ASETS was $5,836 in 2011-2012 and $6,231 in 2012-2013, or $6,122 over both years, which is slightly greater than AHRDS participants at $5.9K (see Table 4). These higher costs may be in part explained by the increased effort and costs associated with the demand-driven approach to labour market programming under ASETS. Also, an inflationary rate of 2% per year may also account for some of the increased costs.

The ASEP and the Aboriginal Skills and Training Strategic Investment Fund (ASTSIF) evaluation findings served as the most appropriate comparison for SPF. The average cost per client for the government under SPF at $9,469 (refer to Table 5) compared favourably at 5% less than ASEP at $10K and 8% more than ASTSIF at $8.7K.

Table 4: ASETS client and costs summary
Fiscal years Total expenditures (EI +CRF) Total clients 1 Cost per clients
2011 - 2012 $287,398,414 49,245 $5,836
2012 - 2013 $292,685,016 46,974 $6,231
Total $589,083,430 96,219 $6,122

1 In this case, clients are defined as ASETS participants with new or ongoing action plans in a given fiscal year.

Table 5: SPF clients and costs summary
Fiscal years Total expenditures (EI +CRF) Total clients Cost per clients
2012 - 2013 $44,800,096 4,731 $9,469

A rigorous analysis of the cost per participant employed will be possible when evaluators are able to measure the employment of clients over a longer time period following the end of participation using taxation records. In the interim, to obtain limited indications of what the cost per participant employed would be, the evaluators produced rough estimates based on the most recent participants’ outcome results reported by the AAHs in the ESDC Standard Datafile (SDF) and total program expenditures for the fiscal years 2011-2012 and 2012-2013.

As demonstrated in the following Table 6, a total of 48,014 ASETS clients with new or ongoing action plans were recorded in the database in 2011-2012 and 45,380 ASETS clients with new or ongoing action plans were recorded in the database in 2012-2013. Footnote 8 / Footnote 9 Of these, a total of 42.3% (n=39,470) found employment/self-employment or returned to school. Considering that the ASETS program costs for 2011-2012 and 2012-2013 totalled $589M, for all individuals who successfully became employed/self-employed or returned to school, the evaluation estimates the ASETS costs to be $14.7K per successful client. It should be noted that these costs are likely overestimated, given that a significant portion of ASETS clients in both 2011-2012 and 2012-2013 (21%) were still enrolled in the program at the time the evaluation was conducted (assigned as “pending outcome” in Table 6). In addition, an average of 8% of ASETS clients in both years could not be reached, leaving their post-program outcomes unknown.

Table 6: Action Plan outcomes for 2011-2012 and 2012-2013 ASETS clients
Outcome 2011-2012 2012-2013
Freq. Percent Freq. Percent
Unemployed but available for work 9,933 20.7 7,045 15.5
Employed 15,904 33.1 13,295 29.3
Self-employed 419 0.9 294 0.7
Returned to school 5,272 11.0 4,286 9.4
Unspecified – client could not be reached 4,630 9.6 3,103 6.8
No longer in the labour force 331 0.7 323 0.7
Pending outcome1 6,774 14.0 12,447 27.4
SD apprentice 1,162 2.0 1,087 2.4
Work experience – student employment 3,339 7.0 3,219 7.0
Referral to agencies 250 0.5 281 0.6
Total 48,014 100 45,380 100

1“Pending outcome” indicates that the outcome was not assigned by a case worker. Likely individuals who are still participating or who recently completed their participation.

In the case of SPF (refer to Table 7 below), it is too early to develop an estimate of the costs per successful SPF client, as too many were either still participating in the program (assigned as “pending outcome” in Table 7) at the time the evaluation was conducted (46%), or could not be reached (5%). The large number of clients still participating in SPF programming is not surprising given that the evaluation was limited to conducting a one year post-program outcome analysis. In the case of SPF, the majority of employment results were not expected until at least 2014 (with almost half not expected until after 2015). Future evaluators will be in a better position to examine ASETS/SPF outcomes and net impacts when additional post-program administrative data and CRA data becomes available in 2015-2016 and 2016-2017.

Table 7: Action Plan outcomes for 2012-2013 SPF clients
Outcome Freq. Percent
Unemployed but available for work 575 12.2
Employed 933 19.7
Self-employed 21 0.4
Returned to school 215 4.5
Unspecified – client could not be reached 250 5.3
No longer in the labour force 9 0.2
Stayed in school 4 0.1
Pending outcome 1 2,195 46.4
SD apprentice 43 0.9
Work experience – Student employment 18 0.34
Referral to agencies 468 10.0
Total 4,731 100

1 “Pending outcome” indicates that the outcome was not assigned by a case worker. Likely individuals who are still participating or who recently completed their participation.

5.2 Key findings on data quality and data collection

Evaluators examined and assessed the quality of the administrative data, as well as the processes used to support AAHs in the area of data collection. Overall, ASETS and SPF administrative data are of a good quality and AAHs captured most of the required data. AAHs are required to report to ESDC on 34 standard data file variables. They consistently reported on 28 of the fields that were of greatest interest to program managers and evaluators. Seven variables were underreported with 1% to 78% of the data missing in any given year. However, these seven variables had no impact on the evaluation given their incidental nature e.g. client initials, telephone number.

The majority of AAH respondents reported that ASETS and SPF program officials provided a number of tools and supports to build AAH capacity in the area of data collection, case management and reporting, in order to improve the quality and integrity of data. Such supports included manuals, templates and training, as well as individual-level support from regional Service Canada staff. Despite the availability of these supports, the high reporting burden was cited as a challenge by AAHs, especially if tight budgets placed limits on available time/staff. This was compounded by the existence of other monitoring and reporting requirements to ESDC, other departments, and the provinces. In addition, key informants reported a lack of consistency with respect to the monitoring and reporting requirements, citing discrepancies and inconsistencies between ASETS/SPF data that is uploaded, as well as changes to the reporting templates and data requirements. Also, as a result of staff turnover (at Service Canada, at ESDC and at the agreement holder level) the subsequent new staff were challenged by the technical nature of the data requirements and case management systems being used to transfer data to ESDC. Altogether, AAHs were reported to be straining to meet their reporting obligations.

6. Conclusions and recommendations

Relevance

The evaluation findings demonstrate that ASETS and SPF are relevant. ASETS and SPF form a federal government priority and their delivery is aligned with federal government responsibilities. There continues to be a demonstrable need for labour market programming for Aboriginal Canadians.

Partnership and engagement

The evaluation found that labour market development partnerships were being established with training institutions and private and public sector employers. The evaluation identified several lessons learned in the effective establishment of partnerships. For example, formalized written agreements between AAHs and employers can be viewed as a best practice, resulting in higher post-program employment outcomes. However, informal agreements were seen as preferable by some employer partners due to uncertainties about eventual employment opportunities and some AAHs in smaller communities perceived that informal agreements convey greater trust between parties.

Recommendation #1

The Evaluation Directorate recommends that ESDC puts measures in place to encourage the adoption of formal agreements between Aboriginal Agreement Holders and employer partners, where appropriate and possible.

Private and public sector employers and training institutions are aware of and engaged in the development of labour market programs and services developed by AAHs. Public sector employers tend to be more engaged than private sector ones. AAHs leveraged financial and in-kind contributions. Evidence also suggests that partnerships are sustainable in the long term. However, AAHs cited: the time consuming nature to establish and maintain partnerships as an issue, especially given their staff turnover. In addition, the time-consuming nature of engagement at times affected the AAH’s ability to establish and maintain partnerships; as well as poor perceptions amongst employers in the context of hiring Aboriginal peoples.

Recommendation #2

The Evaluation Directorate recommends that ESDC should develop strategies with Aboriginal Agreement Holders to strengthen the establishment and maintenance of partnerships, including addressing the capacity issues (training and guidance from national headquarters) faced by Aboriginal Agreement Holders, and dealing with poor perceptions amongst employers in the context of hiring Aboriginal people.

Demand-driven labour market programming

As suggested by the evidence, AAHs are taking a demand-driven approach and aligning their labour market programming with the labour market demands of their region. Overall, employers felt AAH efforts to target occupations reflected a demand-driven approach. Skills programming for occupations in demand in the labour market was developed. Overall, training programs or services targeted partner organization or industry occupation skill needs. While much effort has been made in targeting occupations in demand, a number of challenges were identified. These included the scarcity of economic opportunities or industries in remote communities, volatility within the natural resources sector, low educational attainment and/or essential skills of the target population and lack of labour market information.

Recommendation #3

The Evaluation Directorate recommends that ESDC develop strategies to address expected outcomes where Aboriginal Agreement Holders serve remote communities. Remoteness issues to be addressed include, but are not limited to, the scarcity of economic opportunities and industries, and sector volatility.

Recommendation #4

The Evaluation Directorate recommends that ESDC should continue to ensure that continuous training is offered to Aboriginal Agreement Holders in the area of data collection and case management.

Participant outcomes

A higher proportion of ASETS participants were employed in the year following the program start year (64%) compared to average annual proportion employed in the five years before participation (5 percentage points). As well, on average, ASETS participants earned $1,621 more from employment in the year following the program start year ($11,079) than before participation ($9,458). These results provide early indications that an increased proportion of ASETS participants were able to find employment following their participation and those who found work were able to improve their earnings by either working more hours or having a better paying job. The outcome analysis also showed that participants had higher use of EI and SA, as well as a higher level of dependence on income support in the year following the program start year compared to the years prior to program participation. However, the increased proportion on EI benefits implies an improved attachment to the labour market after program participation.

Program data requirements and collection

Overall, the administrative data supporting the outcomes analysis was generally of good quality. The technical nature of data requirements and case management systems in the context of AAH capacity issues (e.g. staff turnover) as well as the reporting requirements of ESDC and other departments are straining AAH abilities to meet their obligations.

Recommendation #5

The Evaluation Directorate recommends that ESDC should continue to streamline data collection processes for Aboriginal Agreement Holders, including the adoption of consistent and common data reporting requirements, and avoid reporting overlap and duplication where feasible.

Recommendation #6

ESDC should develop a strategy for strengthening Aboriginal Agreement Holders access to timely labour market information to support their service delivery.

Average cost

Using the most recent participant Action Plan result outcome indicator reported by AAHs in the Standard Data File, the cost analysis revealed that the average cost allocated per client for ASETS was $6,122 and for SPF $9,469. With 42.3% of ASETS clients experiencing positive post-program outcomes as of 2012-2013 (including employment/self-employment or returning to school), the average estimated costs allocated per successful ASETS client was $14.7K. It should be noted that the latter cost is likely overestimated, as the evaluation was limited to conducting a one year post-program analysis. As a result, the outcomes of upwards of 21% of ASETS clients have yet to be determined as they were still enrolled in the program at the time of the evaluation. In the case of SPF, it is too early to develop an estimate of the costs per successful SPF client as many (46%) were still participating in the program at the time the evaluation was conducted.

Appendix 1 – ASETS and SPF participant profile Footnote 10

Socio-demographic characteristics of ASETS and SPF participants per principle EBSM (proportions)
Overall ASETS Overall SPF
Characteristic categories SD TWS SE JCP EAS only Total SD TWS SE JCP EAS only Total
Number of observations 67,568 3,303
Proportion of participants 44% 4% 1% 5% 46% 100% 43% 4% 0% 1% 51% 100%
Gender
Male 50% 55% 49% 58% 55% 53% 59% 65% 0% 60% 66% 63%
Female 49% 44% 49% 41% 44% 46% 39% 34% 0% 40% 31% 35%
Age
Under 30 (Youth) 57% 45% 23% 57% 51% 53% 54% 54% 0% 64% 51% 52%
31-40 20% 25% 26% 20% 22% 21% 24% 29% 0% 19% 23% 24%
41-50 14% 18% 26% 14% 17% 16% 14% 13% 0% 15% 18% 16%
51 and over (Older workers) 8% 11% 25% 8% 10% 9% 8% 4% 0% 2% 8% 8%
Aboriginal status
First Nation 75% 85% 73% 89% 79% 78% 74% 74% 0% 83% 90% 82%
Metis 16% 6% 17% 2% 8% 11% 5% 21% 0% 9% 3% 5%
Inuit 5% 6% 3% 7% 7% 6% 16% 1% 0% 4% 2% 8%
Non-status Indian/unspecified 4% 4% 7% 3% 6% 5% 6% 5% 0% 4% 5% 5%
Marital status
Married or common law 20% 25% 30% 21% 19% 20% 19% 19% 0% 17% 20% 19%
Widow/ divorced or separated 5% 5% 7% 5% 5% 5% 5% 5% 0% 0% 5% 5%
Single 53% 48% 40% 51% 51% 52% 53% 57% 0% 60% 53% 53%

Note: Totals may not add up to 100% due to rounding or non-reporting.

Number and proportion of APEs per principal EBSM Type
ASETS SPF
Active claimants
Number of APEs 6,653 365
Proportion among all APEs for active claimants 9.9% 11%
Former claimants
Number of APEs 14,602 743
Proportion among all APEs for former claimants 21.6% 22.5%
Non-claimants
Number of APEs 46,313 2,195
Proportion among all APEs for non-claimants 68.5% 66.5%
Total for active, former claimants and non-claimants
Total Number of APEs 67,568 3,303
ASETS/SPF participants who participated in other ESDC labour market programming
Presence of interventions assigned under other initiatives Proportion of ASETS Participants pre-program

(all claimants N=67,568)
Proportion of SPF participants pre-program

(all claimants N= 3,303)
AHRDA intervention present 57.2% 53.6%
ASEP intervention present 4.1% 12.0%
LMDA intervention present 38.9% 38.5%
ASETS intervention present 12.9% 11.6%
SPF intervention present 0.2% 2.1%

Note: Pre-program period is up to 5 years from the participation

Appendix 2 – ASETS and SPF interventions by core EBSM type

ASETS and SPF interventions to core EBSM type and definitions for EBSMs

ASETS intervention to evaluation intervention types
ASETS interventions Core interventions for evaluation purposes
1. Career research and exploration EAS
2. Diagnostic assessment EAS
3. Employment counselling EAS
4. Skills Development- essential skills EAS
5. Skills Development – academic upgrading SD - classroom training
6. *Occupational skills training – degree SD – classroom training
7. *Occupational skills training – diploma SD - classroom training
8. *Occupational skills training – certificate SD - classroom training
9. *Occupational skills training – apprenticeship apprentices are excluded from SD sample.
10. Occupational skills training – industry recognized SD - classroom training
11. Work experience – student employment Excluded, only represented a small percentage among ASETS/SPF interventions.
12. Work experience - wage subsidy TWS
13. Work experience - job creation partnerships JCP
14. Self-Employment assistance SE
15. Job search preparation strategies EAS
16. Job starts supports EAS
17. Employer referral EAS
18. Employment retention supports EAS
19. Referral to agencies Excluded: only represented a small percentage among ASETS/SPF interventions.

For the purpose of this study, EBSMs are defined as follows:

Employment benefits include:

  • Skills Development (SD) – SD helps individuals obtain basic to advanced skills, through direct assistance to individuals. Clients are responsible for selecting, arranging, and directly paying for their training courses. ASETS and SPF AAHs provide financial support to the client to assist in the cost of taking the training course(s).
  • Targeted Wage Subsidy (TWS) – TWSs help individuals, including those facing particular disadvantages in the labour market, to find a job that will provide them with work experience. The purpose of TWS is to encourage employers to hire individuals they would not normally hire in the absence of the subsidy. The hope is that employers will keep TWS clients as employees once the subsidy period expires.
  • Self-Employment assistance (SE) – SE assists unemployed individuals to create jobs for themselves by starting a business.
  • Job Creation Partnerships (JCP) – JCP encourages employers and organizations to create meaningful, "incremental" work opportunities through which clients can gain work experience leading to on-going jobs.

Support measures include:

  • Employment Assistance Services (EAS) – EAS provides financial assistance, often to second party organizations through service delivery agreements, to assist organizations in the provision of employment services to unemployed persons. Types of activities covered under EAS include: individualized counselling, job finding clubs, job search workshops, access to labour market information; resume writing support; interview skills training; and developing self-employment options.

List of acronyms

  • AAH: Aboriginal Agreement Holders
  • AHRDA: Aboriginal Human Resources Development Agreements
  • AHRDS: Aboriginal Human Resources Development Strategy
  • APE: Action Plan Equivalent
  • ASEP: Aboriginal Skills and Employment Partnership Program
  • ASETS: Aboriginal Skills and Employment Training Strategy
  • ASTSIF: Aboriginal Skills and Training Strategic Investment Fund
  • CRA: Canada Revenue Agency
  • EAS: Employment Assistance Services
  • EBSM: Employment Benefit and Support Measures
  • EI: Employment Insurance
  • ESDC: Employment and Social Development Canada
  • GED: General Educational Development
  • JCP: Job Creation Partnership
  • LMDA: Labour Market Development Agreement
  • SA: Social Assistance
  • SD: Skills Development
  • SE: Self-Employment assistance
  • SPF: Skills and Partnership Fund
  • TWS: Targeted Wage Subsidy
Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: