Evaluation : Canada-Manitoba Labour Market Agreement for Persons with Disabilities, February 2010

List of Abbreviations

• CM-LMAPD Canada-Manitoba Labour Market Agreement for Persons with Disabilities
• CNIB Canadian National Institute for the Blind
• CPA Canadian Paraplegic Association of Manitoba
• CPP Canada Pension Plan
• EAPD Employability Assistance for Persons with Disabilities Agreement
• ECY Education, Citizenship and Youth
• EI Employment Insurance
• EIA Employment and Income Assistance (Manitoba's social assistance program)
• FSH Family Services and Housing
• HRSDC Human Resources and Skills Development Canada
• ICF International Classification of Functioning, Disability and Health
• JEC Joint Evaluation Committee
• LMAPD Labour Market Agreement for Persons with Disabilities
• OECD Organisation for Economic Co-operation and Development
• PALS Participation and Activity Limitation Survey
• SA Social Assistance
• SMD Society for Manitobans with Disabilities
• VR Vocational Rehabilitation
• VRDP Vocational Rehabilitation for Disabled Persons

Executive Summary

Canadian statistics point to the need for employment interventions for persons with disabilities. These citizens typically face levels of employment, income, and other outcomes well below that of the rest of Canadians. Although the employment barriers that contribute to these differences are many, the situations facing persons with disabilities are by no means universal. The multifaceted nature of the disability experience means unique and challenging barriers are common in the workplace, and, as such, so too have been the efforts to address them.

Government attempts to address these barriers have a long history in Canada. Motivated in part by social preference for the inclusion of persons with disabilities in all facets of Canadian society, these initiatives have often included both provincial and federal governments. The Canada-Manitoba Labour Market Agreement for Persons with Disabilities CM-LMAPD is one example of such an initiative, coordinating the efforts to address the needs of persons with disabilities in the workplace. The joint funding agreement has a long history and is rightly viewed as an evolution of previous policy initiatives, rather than a purely novel approach. As a result, many of the provincially delivered services cost-shared under the agreement are time tested and long-standing.

With the initial signing of the CM-LMAPD, so too came the opportunity for a demonstration evaluation of its success. The evaluation, which was preceded by an assessment process, was to examine if the activities cost-shared under the agreement aligned with its intent, identify possible changes to improve the agreement and its cost-shared programming, and assess the relevance of cost-shared programming to clients with disabilities. To do so, the evaluation examined four main aspects of the agreement: rationale, design and delivery, impacts, and cost-effectiveness. Overall, the evaluation largely reflected positively on the agreement and the programs it helps fund.

The evaluation found that, given the barriers facing persons with disabilities in the workplace, a strong rationale exists for programming. In addition, the programming offered aligns well with many of the intents of the CM-LMAPD. As much of the programming cost-shared under the agreement is long-standing, few design and delivery issues exist. Further, to the extent possible, the evaluation found that programs do positively impact employment outcomes for persons with disabilities. The evaluation also provided a detailed account of the agreement's costs.

That being said, areas for improvement exist. In particular, there is little coordinated effort under the agreement to develop and disseminate best practices. There is evidence clients with little prospect for employment are entering programming due to a lack of more suitable program options. Differences in service success exist across delivery organizations, suggesting that practices among some could be improved. A focus on supply-side interventions under the agreement may be limiting available program options. In addition, other observations on possible improvements also exist.

More generally, however, the evaluation has identified four critical observations that should inform employment-related disability programming. First, the target population is very diverse, with clients presenting a considerable range of need. Second, training may have important outcomes that are not necessarily reflected in increased earnings, hours of work, or others strictly associated with competitive employment. Third, and related to the last point, outcomes are usually realized over a longer period than active labour market programs for other Canadians. Finally, the demand side of employment initiatives should be acknowledged, as employers are important partners needed for employment success.

Management Response

Manitoba Family Services and Housing would like to thank all those who participated in formulating and conducting this evaluation of the Canada-Manitoba Labour Market Agreement for Persons with Disabilities (CM-LMAPD). In particular, we acknowledge the contribution of Human Resources and Skills Development Canada, partner provincial departments, key informants, service providers, employers and participants who responded to surveys and took part in focus groups.

We acknowledge the observations provided and note these findings. Actions have been taken to address many of the identified programming shortfalls.

The CM-LMAPD is a cost-sharing arrangement between the Government of Canada and the Government of Manitoba. The federal government contributes 50 per cent of the expenditures that Manitoba incurs in providing eligible programs and services under the LMAPD, up to a maximum of $8.965M annually.

The LMAPD provides for the transfer of federal funding to the provinces and territories for a range of programs and services that enhance the economic participation of working age adults with disabilities in the labour market. The LMAPD has been designed to provide provinces with flexibility in funding activities designed to address the unique labour market challenges faced by people with disabilities.

The program areas in which employment-focused services are provided to assist eligible participants with disabilities in preparing for, obtaining and maintaining employment are:

Manitoba Family Services and Housing (FSH) - Vocational Rehabilitation Program, Day Services Follow-Up Services, Employment and Income Assistance Work Incentive Program.

Manitoba Health - Community and Institutional Mental Health Programs, Doray Enterprises, Proctor Program, Employment Dimensions, SSCOPE Inc., Selkirk and Interlake Mental Health Support Centre, Program for Assertive Community Treatment, Ventures, Addictions Foundation of Manitoba and Behavioural Health Foundation.

Manitoba Education, Citizenship and Youth (ECY) - CareerOptions for Students with Disabilities Program.

This evaluation of the CM-LMAPD was conducted as a result of the provision within the LMAPD Multilateral Framework for bilateral demonstration evaluations, jointly carried out by the federal and provincial governments. The CM-LMAPD evaluation is the first of these demonstration evaluations and was completed in June 2009.

The objectives of the CM-LMAPD evaluation were to assess the rationale for the agreement, the design and delivery of programs, and the impacts and cost-effectiveness. The key findings and conclusions are outlined in these four areas.

The following outlines the Management Response and commitments to follow up on the findings and conclusions of the CM-LMAPD demonstration evaluation.

Rationale

Key Finding 1: The rationale for the CM-LMAPD is firmly established by the evaluation. Literature review, participation figures and participant interviews indicate a continuing need for programming under the CM-LMAPD.

FSH is pleased that the evaluation shows that the CM-LMAPD programming aligns with the thinking of both the provincial and federal governments with respect to programming for people with disabilities. FSH is committed to maintaining the direction CM-LMAPD programming has taken to continue to meet the needs of people with disabilities.

Key Finding 2: Feedback from service providers, program managers and clients show that the CM-LMAPD programming remains relevant to the needs of people with disabilities. That is not to say that the CM-LMAPD programming meets all client needs in all situations. Some barriers to employment, such as disability stigma and the inflexibility of the standard work week are not addressed through the CM-LMAPD. As well, the evaluation found that there are perceived work disincentives from provincial Employment and Income Assistance (EIA) structure. The evaluation also found that CM-LMAPD programming focuses on employment and does not recognize volunteer work, education and socially useful activities.

FSH launched a disability awareness campaign in 2008 to raise awareness, promote people with disabilities among employers, and address disability stigma. As well, Vocational Rehabilitation (VR) counsellors assist participants in obtaining employment that takes into account the participant's disability and ability to work, including part-time employment.

Many provincial supports have been in place prior to signing of the Employability Assistance for Persons with Disabilities to address work disincentives, including support for work clothing, work transportation, child care, extended health services and a rapid re-enrolment policy. In addition, beginning in 2007 a number of initiatives were introduced under the Rewarding Work (RW) Strategy to address poverty. Initiatives under the RW Strategy that address work disincentive from EIA include:

• Enhanced work incentive policy which allows all participants to keep $200 of earnings and 30% of earnings over $200 before benefits begin to be reduced;
• Manitoba Works is a partnership with Employment Manitoba to provide wage subsidies to employers to hire and train people on EIA with the goal of securing permanent work;
• marketAbilities Fund supports partnership building between potential employers and other community stakeholders to help increase the employment rate of persons with disabilities who live in northern and rural regions.
• The marketAbilities Team is a specialized group of staff focused on working with EIA participants with disabilities who are interested in exploring employment;
• Sara Riel Work Placement Force Program has been expanded to provide additional supports to people experiencing mental health concerns.

The following RW initiatives are available in support of volunteerism and education:

• The Get Ready! training and education policy allows people on EIA to participate in approved education and training programs for up to 2 years, or longer for single parents and persons with disabilities.
• The High School Completion policy promotes greater high school graduation rates by allowing the deferral of employment expectations for 18-21 year old EIA participants who are enrolled in and attending high school.
• The Rewarding Volunteers Benefit provides EIA participants enrolled in the disability category with approved volunteer plans with monthly financial support to assist with the costs of volunteering.

FSH will evaluate the Rewarding Volunteers Benefit in terms of the uptake of the benefit and number of EIA participants who leave volunteer positions for employment. This review will help to inform FSH of the impact volunteerism has in preparing EIA participants for future employment.

Key Finding 3: Programming strongly supports the employment and training, employment participant and employment opportunities objectives of the LMAPD. Limited emphasis is placed on the LMAPD objectives of connecting employers with people with disabilities and building knowledge.

Under the Rewarding Work initiatives, the marketAbilities Fund supports partnership building between employers and other community stakeholders to help increase the employment rate of persons with disabilities who live in northern and rural regions.

A disability awareness campaign was launched in 2008 targeting employers to create greater awareness that people with disabilities should be considered for jobs based on what they can do, not by what they cannot do, showing real Manitobans with disabilities in the work force and connecting employers to qualified people, and promoting the skills and potential of hard working Manitobans with disabilities.

A Multi-Sector Forum was developed to gather employers, service delivery managers and provincial representatives to discuss programming and will be re-convened to continue discussions.

Key Finding 4: The flexibility of the CM-LMAPD has allowed the province to tailor programming to the needs of citizens, and for diverse service delivery organizations to be included under the CM-LMAPD.

FSH is pleased that CM-LMAPD programming aligns with the priorities of the federal government, provincial government and disability service providers and will endeavour to ensure that programming continues to meet the needs of all stakeholders.

Key Finding 5: Employers are inconsistently integrated in program design and delivery. There is a need to address employer concerns regarding the cost of hiring persons with disabilities, providing on-the-job support and information about how to implement disability accommodations.

A disability awareness campaign was launched in 2008 targeting employers to create greater awareness that people with disabilities should be considered for jobs based on what they can do, not by what they cannot do.

Supported Employment Programs and the VR Program provides for on-site job coaches to assist participants with skill development, including task learning and short-term counselling to resolve problems and issues within the job setting.

Additional VR employment accommodations and support include building modifications, wage subsidies and technical aids and devices.

Design and Delivery

Key Finding 6: Under the CM-LMAPD provincial and service delivery organizations relationships are strong. Federal involvement is limited to the provision of funding.

There is little formal cross-agency or departmental consultation with regard to delivery which may impact referral of individuals to appropriate services.

The VR Program is delivered by vocational counsellors in FSH regional offices, the Regional Health Authorities and designated agencies serving specific disabilities. While there is no formal cross-agency or departmental consultation with regard to delivery, counsellors are aware of community agencies/resources and refer participants to alternate resources as appropriate. In addition, an Integrated Service Delivery initiative is underway with Family Services and Housing and the Winnipeg Regional Health Authority allowing a more integrated/coordinated approach to providing services. This includes co-location of health and social services at community Access Centres, collaborative case management between Health and FSH, as well as assigning a single case worker, in some cases, for cross program delivery.

Where multiple disabilities exist, agencies and the department communicate to ensure appropriate services are available for the participant. In addition, the Service Delivery Support branch of FSH serves a central liaison function to assist in service coordination.

Key Finding 7: Services are tailored to individuals with specific disabilities which is an effective means of providing services. However, individuals with mental illness, cognitive impairments, those with recently recognized disabilities such as Asperger's syndrome and FASD, and individuals with multiple disabilities are less well-served and lack dedicated service delivery organizations.

The VR program will continue to focus on existing client services delivered through Canadian National Institute for the Blind (CNIB), Canadian Paraplegic Association (CPA), Society for Manitobans with Disabilities (SMD) and FSH. The Department is looking at other opportunities for provide appropriate services for persons with disabilities not currently served by existing programs, including those with autism, Asperger's syndrome and FASD. For example, Spectrum Connections FASD Services assists transitional youth and adults with FASD who are ineligible for other Manitoba support programs to function more interdependently in the community and increase the functioning and quality of life. Services provided include clinical case management, transition planning, outreach mentorship, clinical services, consultation and training services.

Under the Rewarding Work initiative additional resources have been targeted to support individuals experiencing mental health concerns.

Key Finding 8: Little evidence exists to suggest overlap among cost-shared programs. Some participants with little prospect of employment participate in CM-LMAPD programming because no alternative exists.

FSH is pleased that the evaluation has shown that there is little evidence of overlap among programs and services offered for people with disabilities. FSH will endeavour to ensure services and programming continues to meet the needs of people with disabilities, including those with limited potential for employment. For example, the Department has recently increased funding to Opportunities for Employment Inc. for its Stages of Change program. This program helps persons with disabilities begin the process of preparing for work. It helps build self-esteem, addresses attitude barriers and starts to teach the skills people need to find work.

Key Finding 9: While eligible participants are able to access support, there is some variability in timely access to services. In addition, a barrier that prevents individuals from accessing programming is a lack of knowledge about available services.

Under the Rewarding Work Strategy, additional resources have been provided to address service waitlist, including additional staff resources. Information about FSH services, including the Vocational Rehabilitation (VR) Program, is available on-line at Manitoba Family Service and Consumers Affairs Website.

FSH is developing an on-line tool designed to help staff and citizens learn about FSH programs that may be available to them. ServiceLink is still in development but is a tool that will take users through a series of basic demographic and eligibility questions. Based on the information provided, a results page will be presented containing links to FSH services that may be applicable to their situation. A description of the service, additional considerations and contact information will be provided under each service listed.

Key Finding 10: There is no formal mechanism for developing and disseminating best practices and lessons learned among CM-LMAPD programming.

FSH collects comprehensive outcome information, including information on the services provided and employment outcomes of participants. This data is shared with FSH staff and designated agencies. It is anticipated that this will be one measure to identify best practices and FSH will continue to look at how best to communicate these practices to the field.

Key Finding 11: Expanding the scope of services beyond standard employment to include pre-employment skills i.e. social and workplace behaviours would benefit many program participants.

As mentioned, the Department recently increased funding to Opportunities for Employment Inc. to provide pre-employment skills for persons with disabilities through the Stages of Change program. The Department will look for other opportunities to expand programming as resources permit.

More employer involvement is needed in programming, design of interventions and in the policy development process.

In 2005, a Multi-Sector Committee was established that brings together government, business, disability agencies and consumers. The purpose of the Multi-Sector Committee is to engage stakeholders, as well as generate discussion and feedback on programming and interventions for people with disabilities. FSH is currently planning to re-engage this committee and will be reviewing the recommendations from the previous Committee and determining the priorities for the upcoming year.

Impacts

Key Finding 12: Participant employment situations, education levels and quality of life typically improved.

FSH is pleased that the evaluation has shown that the LMAPD programming has had positive impacts on participant employment, education levels and quality of life. FSH will strive to ensure that funding continues to focus on, and be available to support, employment and education.

Key Finding 13: Factors that impact successful completion of programs and success in the labour market include the type of programming provided, the designated agency or organization providing service, and severity of participant disability.

In April 2007, FSH developed enhanced outcome measures to assist the department in identifying successful services and contribute to the identification and dissemination of best practices. These outcomes will be monitored to determine what services are successful and ensure that these best practices are shared with those involved in program design and delivery.

Key Finding 14: In terms of Employment and Income Assistance (EIA) use, participants for whom more vocational rehabilitation dollars were spent, were also associated with reduced EIA dependency.

For those participants who were working, average employment income increased. Participants noted their quality of life improved, including a positive impact on self-esteem, confidence, and income. In addition, job satisfaction increased as a result of LMAPD programming.

FSH is encouraged by the positive impacts LMAPD programming has shown on reducing participants' dependency on EIA, increasing employment levels, improving quality of life and job satisfaction.

Building on the vision and principles of Full Citizenship: A Manitoba Provincial Strategy on Disability, a discussion on a long-term disability strategy Opening Doors: Manitoba's Commitment to Person with Disabilities has recently been released. Opening Doors is a discussion paper that sets out Manitoba's ideas for moving forward in such areas as housing, employment, rights, accessibility, education and health.

As mentioned above, many Rewarding Work initiatives have been implemented to help people with disabilities move from EIA to employment, including increased education and training opportunities, helping low-income working families with the costs of raising their children, offering support to help people find work and keep people in jobs by providing ongoing support to encourage them to stay employed.

Key Finding 15: Participants and employers rated their satisfaction with programming highly.

FSH is encouraged by the satisfaction levels expressed by employers and participants. The department will continue to monitor programming to make sure that those involved in LMAPD programming remain satisfied with the programs and services offered.

Key Finding 16: Generally, employers indicated that many individuals placed with their organization worked out well: 39% indicated that the individual placed with their organization had integrated well all or most of the time, while another 28% stated that the individual had integrated well about half the time. The majority of employers indicated that service provider organizations offered sufficient support to individuals.

FSH will continue to seek ways to follow up with employers to ensure supports are in place for employing people with disabilities. In 2005, a Multi-Sector Committee was established that brings together government, business, disability agencies and consumers. The purpose of the Multi-Sector Committee is to engage stakeholders, as well as generate discussion and feedback on programming and interventions for people with disabilities. FSH is currently planning to re-engage this committee and will be reviewing the recommendations from the previous Committee and determining the priorities for the upcoming year.

Key Finding 17: Overall, the evaluation did not identify any general unintended impacts.

The evaluation reinforced the validity of Manitoba's understanding of CM-LMAPD programming and should additional evaluations be possible in future it will further add to Manitoba's understanding of programming for people with disabilities.

Key Finding 18: Employment at service provider organizations and in the provincial government represented a beneficial impact on the community. Successful employment of individuals results in provincial savings to EIA. Working with persons with disabilities in the community raised awareness about disability.

FSH is encouraged with the positive impacts identified. FSH is committed to maintaining the direction CM-LMAPD programming has taken to continue to meet the needs of persons with disabilities, including support for employment at service provider organizations and in the provincial government.

Cost Effectiveness

Key Finding 19: CM-LMAPD expenditures in 2004/05 was $20,174,400 and in 2005/06 it was $21,747,500. Of this, the federal government contributed the maximum $8,965,000 in both fiscal years. Manitoba typically pays about 60% of annual program costs.

The March 2004 federal budget included an additional $30 million nationally for LMAPD. This new funding provided an additional $1,051.0 million for Manitoba, resulting in an increase in the federal contribution level from $7,914.0 million to $8,965.0 million annually. Manitoba appreciates the contribution the federal government provides in funding programs for persons with disabilities and improving the lives of Manitobans living with disabilities.

Key Finding 20: The majority of service providers believe that programs are as efficient as possible. Suggestions to reduce costs include moving towards non-governmental service delivery, block-funding service delivery organizations, implementing an outcome tracking system to identify success and failures to improve efficiency, and more education or communication to the employer community about employment services.

In 2008, the VR Program moved from a per diem to a grant funding model. This change to grant funding provides greater stability in funding, more streamlined administrative procedures, and improved outcome tracking.

As mentioned above, FSH outcome measures have been developed and will assist the department in identifying successful services and contribute to best practices dissemination.

A disability awareness campaign was launched in 2008 targeting employers to create greater awareness that people with disabilities should be considered for jobs based on what they can do, not by what they cannot do.

FSH will continue to work with the community to develop effective methods for communicating with employers about employment services.

Key Finding 21: Overall net impact analysis of programming was not possible and savings generated by LMAPD programming cannot be assessed.

Multiple lines of inquiry were examined and many questions were answered. Given the complexity of the programs, the diverse needs of participants and the availability the data, it was not possible to perform a quantitative net impact analysis of the program and, as a result, specific savings generated by CM-LMAPD programming could not be assessed. However, other outcome data as outlined in this document point to the success of the program.

1. Introduction

The Canada-Manitoba Labour Market Agreement for Persons with Disabilities (CM-LMAPD), which replaced the former Employability Assistance for People with Disabilities (EAPD) Agreement on April 1, 2004, is a bilateral agreement between the Government of Canada and the Province of Manitoba. The agreement outlines federal-provincial cost-sharing provisions for eligible, provincially designed and delivered programming meant to increase the economic participation of persons with disabilities. It is based on a Multilateral Framework that was approved by the federal and provincial governments in December 2003.

This report presents a summary of findings from the evaluation of the CM-LMAPD conducted between January 2007 and September 2008. The findings are based on multiple lines of evidence as documented in various evaluation technical reports. These individual reports are discussed in Section 4.1.

1.1 Organization of the report

This report includes five additional sections. Sections 2 through 4 provide background information important to the evaluation. Of these, section 2 discusses disability and government policy, while section 3 summarizes the policy evolution leading to the CM-LMAPD. Section 4 provides information about the evaluation. Section 5 details summary findings from the CM-LMAPD evaluation, and finally section 6 provides concluding remarks.

2. Disability and Government Policy

Canadian disability, employment, income, and education statistics point to the need for employment interventions for persons with disabilities. According to the 2001 Participation and Activity Limitation Survey (PALS), 14.6% of people over 15 in Canada report having a disability. As with other countries, this varies with age, as almost four times as many seniors (65 and older) have a disability compared to working-age people (Statistics Canada, 2002, p. 17). In Manitoba, prevalence is similar but slightly higher than the national average, with 16.9% of individuals aged 15 and older reporting a disability.

In Canada, among people with disabilities who are of working age, 37% report that they have not completed their high school education compared to only 25% of the working-age population in Canada without disabilities. By contrast, only 11% of working-age Canadians with a disability report having a university-level education, compared to 20% of the Canadian working-age population without disabilities. Levels of educational attainment in Manitoba are similar. While 38% of working-age people with disabilities and 32% of working-aged people without disabilities report that they have not completed high school, 10% of working-age people with disabilities and 17% of working-age persons without disabilities report having a university level education (Calculated from Statistics Canada, 2002, p. 7, p. 21, p. 29, and p. 36).

Compared to people who do not have disabilities, the working-age population in Canada reporting a disability has a lower employment rate-43% compared to 74%. In Manitoba, employment figures are similar, with an employment rate of 77% among the working-age population of people without disabilities, and 52% among working-age people with disabilities.

Working-age people with disabilities in Canada who earned an income in 2000 earned $22,451 on average, while the rest of the working-age Canadian population who earned an income in 2000 earned $31,509 on average. In Manitoba, these figures showed slightly less income disparity, with people with disabilities earning $24,423 on average, and people without disabilities earning $28,499 on average. However, it is important to note that while 3% of people with disabilities across Canada reported no income, 4% reported no income in Manitoba.

2.1 Defining disability

Defining disability is a difficult task, complicated by a variety of perspectives on the phenomena. While recognizing that each involves a considerable amount of nuance, these perspectives may be located on a spectrum with two conceptual extremes. At one end is the individual definition of disability. This sees an individual with a disability as impaired relative to the general population. At the other end is the social definition. This definition is based on the argument that disability results from exclusion and from a failure to accommodate persons with disabilities in a social setting (Office of Disability Issues, 2003, p. 5).

The individual or medical perspective grew out of the "biomedical paradigm of healthcare" (Jongbloed & Crichton, 1990, p. 26), which was dominant in the late 1800s and early 1900s. At this time, the absence of infectious disease among individuals constituted health, so definitions of disability became centered on the individual (Lunt & Thornton, 1994, p. 226). Individual disability was seen as a condition to overcome, with or without community support, and the responsibility for change was placed solely on persons with disabilities (Jongbloed & Crichton, 1990, p. 26). Although the medical approach has evolved, it is still considered restrictive and runs the risk of omitting many potentially significant health conditions, which may emerge or may not yet appear on a generally recognized or legislated list of impairments (Bernell, 2003, p. 37).

By contrast, the social or socio-political perspective ^Footnote1 , which developed throughout the 1970s, explicitly acknowledges that health is influenced by a variety of factors that are beyond the control of the individual, including social and environmental settings (Lunt & Thornton, 1994, pp. 226-227). The socio-political approach seeks to shift the focus away from the physical or mental limitations of the individual and "direct it toward the larger social, cultural, economic, and political environments" (Bernell, 2003, p. 38).

Most current perspectives on disability now incorporate aspects of both the individual and social perspectives. One such example is the functional limitations perspective on disability, developed in the late 1900s (Schalock, 2004, p.204). In one such example, disability stems from a state of active pathology, where individuals may suffer from any number of medical and clinical conditions. The key to the definition is that these conditions will, to varying degrees, lead to functional limitations among individuals. Another example is the biopsychosocial perspective used in the International Classification of Functioning, Disability and Health (ICF) developed by the World Health Organization in 2001. Based on the International Classification of Impairments, Disabilities and Handicaps implemented in 1980, the ICF is a scaled, multi-purpose framework used to examine various aspects of health (World Health Organization, 2002, p. 2).

The literature and the CM-LMAPD evaluation lines of evidence underscore the multidimensionality of disability and support the contention that attempts to establish a single definition, applicable in all contexts, is neither feasible nor relevant. Developing a framework for understanding aspects of disability, and then defining it along various dimensions, is more helpful. Such a view links directly to clinical characteristics as well as to functional and contextual aspects resulting from daily life. This is applicable in a labour market or other context where, for example, someone with complete hearing loss may have a radically different experience than someone suffering from a mental illness.

2.2 The case for government action

In the Canadian context, governments have intervened in the market for two broad reasons: market failure and social preference. Market failure typically encourages governments to tax, subsidize, produce goods and services, or regulate in an effort to create optimal outcomes.

Market failures in a labour market setting may result from many situations including:

• the presence of externalities in the labour market that do not allow employers to capture the full benefit of their training investment in employees. In this situation, private firms may not sufficiently invest in training, resulting in a low supply of training, especially for persons with disabilities and socially disadvantaged groups;
• asymmetrical information when consumers or firms have specialized knowledge, leading to distortions in price. In a labour market situation, job applicants always have better information regarding their true ability than the recruiter. By contrast, the employer may have preconceived notions about the applicant and may undervalue their productivity. In either case, a price distortion with regard to the applicant's wage is likely to result;
• transaction costs that represent the costs of using the market and often emerge as high information costs. An example of labour market transaction costs may include the cost of testing and rejecting an applicant during the hiring process. If these costs are too high, an employer may not start the process, believing that these costs outweigh the benefits associated with hiring the employee.

Although market failures do justify government intervention in many contexts, in the context of disability programming, social preference represents the stronger justification for government intervention. Social preference, despite being primarily political, relates to economic and social outcomes in society. Governments respond to voters' demands for social and economic change, and when voters question social and economic outcomes, governments intervene to produce outcomes that are more desirable.

In the context of disability programming, government may, for example, pursue a policy of affirmative action towards persons with disabilities as a social goal supported by the voting population. Intervention on the part of the government may then include programs to train persons with disabilities, cash incentives to hire workers, and incentives to invest in workplace modifications to accommodate workers with disabilities. Canada has developed a strong social and political environment supporting inclusive policies for persons with disabilities.

The long-term employment of individuals with disabilities also serves an obvious economic purpose, that is, economic self-sufficiency, which has a number of subsequent benefits for the broader society by decreasing dependence on social assistance. However, employment can also greatly improve the quality of life of people with disabilities by providing structure, increasing their self-esteem, and giving them a new sense of identity.

2.3 Employment programming

Because employment often acts as a sign of inclusion, people with disabilities are often marginalized within the broader society as a result of their lack of work. To counter this exclusion, many governments and health professionals provide intervention services for persons with disabilities to facilitate employment for those who are potentially able.

The majority of current policies, including the CM-LMAPD, remain supply-side driven and consist of interventions mainly directed towards persons with disabilities. Most mirror the training programming offered to social assistance or employment insurance clients. This supply-side perspective views a person with a disability as needing remediation. By contrast, the demand-side perspective views employers and the place of employment as important for assisting people with disabilities to perform in the workplace.

Regardless of the programming approach, the transition to employment can be difficult. Interventions designed to assist people with disabilities can take a great deal of time and effort, and numerous barriers to employment exist - barriers as multi-dimensional as the disability experience itself. As noted above, in North America and other OECD countries, interventions have historically focused on supply-side adjustments (i.e., training and education) to increase the market "competitiveness" of persons with disabilities in the job market. Although many programs have recently increased their support for workplace accommodations and have encouraged employers to adapt their workplaces, programming meant to initiate large-scale shifts in public attitude toward disability rarely exists.

Traditional methods of vocational rehabilitation (VR) involve vocational assessment, followed by pre-placement training. This pre-employment training often takes place in sheltered environments where competitive employment is not an immediate expectation or goal. However, with regard to mental disability, a large body of work highlights the ineffectiveness of these approaches. Furthermore, traditional approaches often result in negative effects for VR clients, including:

• prolonged segregation from competitive employment markets
• reduced expectations of competitive employment
• perpetuation of the self-perception of disability among clients (Moll et al., 2003, p. 299)

By contrast, models of individual placement and support or other supported placement models, instead of emphasizing front-end vocational assessment and pre-employment training, immediately emphasize competitive employment and then provide ongoing support services in the work environment (Moll et al., 2003, p. 299). The interests of rehabilitating clients guide the initial job search, and employment is supported after clients are placed. Assessment and improvement takes place on the job in order to maintain both the clients' interests in the employment and their focus on the centrality of competitive employment.

3. The Canada-Manitoba Policy Evolution

The 1867 British North America Act gave the federal government the responsibility for "peace, order and good government in the country." The Act allocated responsibility for health, education, and social welfare to the provinces. While the Act clearly established areas of sole responsibility-including federal jurisdiction over defence and provincial jurisdiction over education-it did not eliminate jurisdictional ambiguity in all areas. Differences in taxing powers at the federal and provincial levels resulted in further jurisdictional overlap as the federal government contributed to programming in provincial jurisdictions including health and social services.

As the federal government's policy on provincial transfers for health and social programming evolved, so too did the federal-provincial-territorial relationship with regard to disability. The post-World War II period saw considerable change in the perspective toward supporting persons with disabilities. In particular, The Constitution Act of 1982 and The Charter of Rights and Freedoms defined an important new statement of rights, which, in addition to the customary rights, articulated rights for persons with disabilities to participate fully in Canadian society through the specification of equality and mobility rights (See Torjman, 2001).

Every individual is equal before and under the law and has the right to equal protection and equal benefit of the law without discrimination and, in particular, without discrimination based on race, national or ethnic origin, colour, religion, sex, age or physical or mental disability.

The statement, which entrenched the social importance of the inclusion of persons with disabilities in Canadian society, identified the federal government as the central defender of the rights of persons with disabilities. All jurisdictions were to support the provisions of the Charter, and, by extension, the federal government's involvement in programming targeted at persons with disabilities. These changes provided strong moral justification for federal involvement in disability issues.

3.1 Antecedents to the CM-LMAPD

The joint approach to disability programming underpins the CM-LMAPD. However, two important antecedent agreements exist. The Vocational Rehabilitation for Disabled Persons (VRDP) Program Agreements began in 1962 and represented the first federal-provincial cost-sharing initiative to support programs and services for people with disabilities. The Agreements continued for over 35 years (F/P/T Ministers Responsible for Social Services, 1997). In fiscal year 1996-1997, the VRDP bilateral agreements were extended for one final transitional year. The federal, provincial, and territorial governments sought to improve on the VRDP agreements by creating the Employability Assistance for People with Disabilities Agreement (EAPD). The transition from the VRDP to the EAPD narrowed the programming focus from broad spectrum rehabilitation-related supports to those most closely linked to employability.

In December 2003, the Ministers Responsible for Social Services approved a new Multilateral Framework for the Labour Market Agreements for Persons with Disabilities (LMAPD). As a result of the Multilateral Framework, the CM-LMAPD replaced the former EAPD in Manitoba on April 1, 2004. It built on the employability-centered approach to cost-sharing for persons with disabilities established under the EAPD. However, the CM-LMAPD also recognizes the importance of a range of stakeholders in the success of programming. The following principles govern the CM-LMAPD and the Multilateral Framework for Labour Market Agreements for Persons with Disabilities:

• "persons with disabilities should be fully included in Canada's social and economic mainstream, a key element of which is successful participation in the labour market. Fostering greater self-reliance through obtaining and maintaining meaningful employment will help persons with disabilities achieve inclusion";
• "a diverse set of approaches is required to support both persons with disabilities and employers in order to improve the employment situation of persons with disabilities";
• "persons with disabilities should have access to mainstream and targeted employment programming to ensure their successful participation in the labour market";
• "employment-related and workplace supports are critical to the success of persons with disabilities in the labour market";
• "supports and services should be individualized, holistic, linked to other needed support systems, easy to access, portable across life transitions, timely and inclusive";
• "co-operation and partnership among governments, persons with disabilities, community-based organizations, business, labour, Aboriginal, Métis and Inuit peoples and other stakeholders is key to the success of a comprehensive labour market strategy";
• "accountability and reporting to citizens in order to demonstrate results and inform program and policy development is a foundation of this framework." (Province of Manitoba, 2006, p. 5).

3.2 The CM-LMAPD

As noted above, the CM-LMAPD is a bilateral agreement between the Government of Canada and the Province of Manitoba. The agreement outlines federal-provincial cost-sharing provisions for eligible, provincially designed and delivered programming, meant to increase the economic participation of persons with disabilities. Initially covering the period of April 1, 2004 to March 31, 2006, the CM-LMAPD has had a series of extensions, with the most recent extension being to March 31, 2011 (Province of Manitoba, 2009, p. 3). Under the agreement, the federal government agrees to cover 50% of the costs incurred by the province for eligible programs and services, up to a pre-defined maximum amount.

In its more recent years, CM-LMAPD cost-shared programming was offered by three main departments of the Government of Manitoba: Family Services and Housing (FSH); Manitoba Health; and Education, Citizenship and Youth (ECY). Earlier in the agreement, Competitiveness, Training and Trade also delivered programming. During the life of the agreement, programs cost-shared under the CM-LMAPD fell under five employment-related priority areas:

• Education and training - to improve the level of basic and post-secondary education and work-related skills for people with disabilities.
• Employment participation - to improve the labour market situation and independence of people with disabilities through employment-related activities.
• Employment opportunities - to expand the availability, accessibility and quality of employment opportunities for people with disabilities in partnership with business and labour.
• Connecting employers with persons with disabilities - to enhance awareness of the abilities and availability of people with disabilities and strengthen people with disabilities' knowledge of labour market opportunities.
• Building knowledge - to enhance the knowledge base, which contributes to continuous improvement of labour market policies and programs for people with disabilities.

Appendix C illustrates the structure of cost-shared programming delivered under the CM-LMAPD at the time of the evaluation. While some programming is delivered directly through the previously-mentioned provincial departments, many services come from non-governmental organizations. These organizations generally enter into agreements with the provincial departments under the auspices of CM-LMAPD cost shared programming.

Because total annual eligible expenditures under the CM-LMAPD normally exceed twice the federal maximum contribution, in practice the agreement results in a fixed annual federal contribution of $8,965,000 and a variable provincial contribution in excess of this amount. For example, in the 2004-2005 fiscal year, total eligible program costs of $20,174,400 resulted in a federal contribution of $8,965,000 and a provincial contribution of $11,209,400 (Province of Manitoba, 2005, p.10). In the 2005-2006 fiscal year, eligible costs rose to $21,747,500, resulting in a federal contribution of $8,965,000 and a provincial contribution of $12,782,500 (Province of Manitoba, 2006, p.10). These levels of provincial and federal contribution represented a 45:55 federal-provincial cost-sharing ratio in 2004-2005 and a 41:59 federal-provincial cost-sharing ratio in 2005-2006.

Table 1 outlines CM-LMAPD total expenditures according to provincial department. It also details the provincial and federal contributions to this total expenditure.

• Source from the above table Source: Province of Manitoba, Annual LMAPD Reports, 2004-2005 and 2005-2006

4. The Canada-Manitoba Labour Market Agreement for Persons with Disabilities Evaluation

To ensure that the Canada-Manitoba Labour Market Agreement for Persons with Disabilities (CM-LMAPD) is achieving its goal of increasing the employment of persons with disabilities, a demonstration evaluation was conducted. It was overseen by a Joint Evaluation Committee (JEC) consisting of representatives from the Province of Manitoba and the Government of Canada. Provincial representatives included individuals from the departments of Family Services and Housing and Health. Federal representatives included individuals from HRSDC and Service Canada. While the Government of Canada directed the evaluation, the province provided logistical support, extensively reviewed documentation for program accuracy, and gave contextual information relevant to the evaluation.

This evaluation was designed to:

• determine if CM-LMAPD programming aligns with the Multilateral Framework and is being implemented as intended;
• identify possible changes to CM-LMAPD design and delivery to improve program success;
• assess if CM-LMAPD services are relevant to persons with disabilities and if they reflect best practices.

As a demonstration evaluation, the scope of the CM-LMAPD evaluation was to remain broad and comprehensive within the limits of the data availability. It was also to provide insight into potential evaluation approaches for similar disability programming.

4.1 Background

An Evaluability Assessment, pre-dating the start of the CM-LMAPD evaluation, laid the groundwork for the evaluation. Completed in 2005, the Evaluability Assessment included a review of CM-LMAPD documentation, a summary literature review, an examination of a VR case file sample, a review of a VR-Employment and Income Assistance (EIA) participant dataset, and approximately 15 key informant interviews. The assessment also included illustrative impact estimates using statistical matching; a methodology that the peer reviewers at the time advised not be implemented during the eventual evaluation.

The assessment resulted in the evaluation framework, and outlined a preliminary methodology for the CM-LMAPD evaluation. This framework included 21 questions that were grouped into four main categories: rationale, design and delivery, impacts, and cost-effectiveness. Appendix A includes these 21 questions, grouped according to their respective categories. To address the questions outlined in the framework, the evaluation used 11 lines of evidence ^Footnote2 :

• A literature review of recent disability works. This review examined the definition of disability, employment programming for persons with disabilities, government disability initiatives, and program evaluations in Canada and abroad. The review gathered information from over 200 sources.
• A program profile for the CM-LMAPD. This profile detailed the cost-shared initiatives under the agreement at the time of its writing. The information presented in the profile was based on available program documentation and information gathered during two group interviews with provincial representatives.
• Key informant interviews. Interviews with 39 individuals from the federal and provincial governments as well as CM-LMAPD service delivery organizations provided a variety of information on the agreement and its programming.
• A data development and review activity. This activity involved collecting and constructing a single administrative dataset from the various data available for the evaluation. It also involved reviewing the final dataset and suggesting various data development activities to support the evaluation. The administrative data review report documented the over 450 variables associated with the 4,038 individual VR client sample.
• A survey of service providers. This survey gathered feedback from 187 CM-LMAPD service providers associated with Manitoba Family Services and Housing and Manitoba Health. The survey fielded from September to November 2007.
• A survey of employers. This survey solicited feedback from employers associated with CM-LMAPD programming. These employers either hired a client of CM-LMAPD cost-shared programming or had direct contact with a services provider; 112 employers completed surveys between September and November 2007.
• A survey of VR clients. A total of 1,341 clients of the VR program participated in a client survey. The survey, conducted between September and December 2007, gathered information on various evaluation questions and provided additional data to support the evaluation's econometric and costs analyses. The survey gathered information about clients in 2007 and asked them to reflect on their situations in 2003 for comparison purposes.
• Focus groups with clients, service providers, and employers. These groups gathered additional feedback from individuals associated with the VR program cost-shared under the CM-LMAPD. Conducted between November 2007 and January 2008, the 10 separate focus groups included between 4 and 10 individuals each.
• Dyad case studies of clients. These case studies resulted from guided discussions with individual VR clients and a close friend or family member. The discussion provided two perspectives on each client's disability and employment experience. Six case studies were conducted between January and February 2008.
• An econometric analysis. The econometric analysis involved a limited treatment impact assessment of various VR components relative to a baseline client counselling activity. The analysis used an overall sample of 1,062 clients and examined six employment-related outcomes.
• A cost analysis. This cost analysis examined the overall costs of providing VR services and presented these costs relative to the impacts of various program components.

Overall, research work for the evaluation took place between January 2007 and October 2008. All documentation related to the evaluation was extensively reviewed. Three academic peer reviewers examined and commented on all evaluation documents prior to their finalization. They also provided feedback on technical aspects of the evaluation methodology. The JEC later provided comments and suggested changes on the evaluation documents. The extensive review process ensured that the evaluation was of the highest possible quality within the limits of the program context and data availability.

4.2 Interpreting findings

Some general points should be considered when interpreting the evaluation findings. Although the CM-LMAPD outlines cost-sharing for three provincial departments, participation in the evaluation was not universal. Only Family Services and Housing participated in all aspects of the evaluation. Manitoba Health was actively involved in the program profile, key informant interviews, and the survey of service providers, but did not participate in other aspects of the evaluation. Education, Citizenship and Youth had no involvement in the evaluation. Although Education, Citizenship and Youth's programming represents only a small percentage of the overall CM-LMAPD spending, programming offered through Manitoba Health represents approximately half the spending and activity.

This uneven participation meant that many of the lines of evidence involved only Family Services and Housing and its clients, and, as a result, it is only possible to make evaluation statements about its programming. Many evaluation findings cannot be generalized to cost-shared programming outside of Family Services and Housing and to the CM-LMAPD as a whole.

It is also important to recognize that the CM-LMAPD is only a funding agreement. It did not result in new programming, but rather supported existing provincial programming in Manitoba. Findings from the evaluation relate first and foremost to the functioning of this programming as designed and delivered by the province. Inferences about the success of the CM-LMAPD are made in relation to these program offerings. If the provincial programs show success, then it implies that the federal contribution provided through the CM-LMAPD is an effective use of government funding.

5. Summary of findings

The discussion that follows presents a summary of findings from the CM-LMAPD evaluation. In many cases, the discussion includes general statements without extensive discussion of the methodology used to generate the supporting evidence. This is meant to provide readers with an overall sense of the CM-LMAPD's success without extensive discussion of all aspects of the evaluation, which is well beyond the scope of this report.

The discussion adheres to the order of the evaluation issues and questions noted in Appendix A. Findings related to the rationale and design and delivery of the CM-LMAPD usually apply to all programming supported under the agreement. Insights on impacts and cost-effectiveness draw heavily on evidence gathered from the survey of VR clients, focus groups, dyad case studies, econometric analysis, and cost analysis and pertain only to programs offered through Manitoba Family Services and Housing.

5.1 Rationale

What need exists for programs and services to support the employment goals of persons with disabilities?

The rationale for the CM-LMAPD is firmly established by the evaluation. The literature review, for example, highlighted the prevalence of disability in Canada, noting a considerably lower employment rate among people with disabilities-43% versus 74% for those without a disability. In Manitoba, the employment rate was only slightly higher at 52%. In addition, the evaluation found that working-age persons with disabilities in Canada, who earned an income in 2000, earned an average of $22,451 while the rest of the working-age Canadian population earned an average of $31,509. In Manitoba, these figures showed slightly less income disparity, with persons with disabilities earning $24,423 on average, and the remainder of the population, earning $28,499 on average.

Client participation figures and comments during key informant interviews also suggest a high demand for CM-LMAPD cost-shared programming. For example, the program profile indicates that the CM-LMAPD cost-shared programming served over 14,000 clients in the 2004-2005 fiscal year, and over 15,000 in the 2005-2006 fiscal year (see Table 2 below). Despite the sharpest increase among Family Services and Housing programs, Health programming for persons with disabilities represented more than half of all clients in both years. The number and distribution of clients across a range of program types also demonstrates considerable and diverse client need.

• Source from the above table Source: Province of Manitoba, Annual LMAPD Reports, 2004/2005 and 2005/2006
• Source from the above table ¹ Figures double count individuals receiving more than one type of funded expense
• Source from the above table ² Referred to as Advanced Education and Training in the Annual LMAPD Reports
• Source from the above table ³ Figure for 2003/2004 fiscal year

Overall there is a strong indication that a continuing need exists for the programming offered under the CM-LMAPD.

Is the LMAPD relevant to the employment needs of persons with disabilities?

Feedback from service providers, program managers, and clients shows that the CM-LMAPD programming remains relevant to the needs of persons with disabilities. Many of the barriers to employment identified through the evaluation, including low educational attainment, a lack of essential workplace skills, and inadequate transportation to and from work, are addressed to some extent through CM-LMAPD programming.

That is not to say that CM-LMAPD programming meets all client needs in all situations. In fact, service providers suggest that for some, additional client supports and services are needed. Among the service providers who were surveyed, slightly less than one third (32%) agreed that CM-LMAPD-funded disability programming meets the needs of all groups of employable persons with disabilities. Although it is likely that resource constraints help drive these results, some barriers to employment are not addressed under the CM-LMAPD. For example, the agreement does not address disability stigma, or the inflexibility of the standard work week.

Key informants also remarked on what they perceived to be a work disincentive from the provincial EIA structure. This perception exists despite the presence of provincial supports to address work disincentives. For example, work incentives including support for work clothing, work transportation and child care have been available to employed EIA participants since prior to the signing of the EAPD. Since 1993, individuals with a disability who have found employment may be eligible for extended health services. Additionally, since 2001, EIA has implemented a rapid re-enrollment policy where participants enrolled in the disability category who leave EIA for work can re-enroll without having to repeat the medical panel process until the date that their previous medical assessment expires.

That being said, key informants stressed that the cost-sharing flexibility built into the CM-LMAPD is critical to creating programming that meets the needs of persons with disabilities. Without the ability to cost-share programming of various types, the agreement would not succeed in addressing the significant range of client needs. The dyad case studies conducted for the evaluation provide a tangible sense of this variation, charting the successes and failures of clients across a number of disability types. By way of contrast, one case study highlighted a two-year period of retraining leading to full time employment, while another documented a lifetime of employment difficulties. Even among people with similar disability types, feedback from service providers and clients suggests considerable variation.

As such, many individuals involved in service delivery, for example, argued that successful programming requires even greater flexibility. Many of the key stakeholders expressed concern about the CM-LMAPD's emphasis on employment programming. Some noted that employment programming alone serves certain client groups poorly. For example, several mentioned that a broader-based Supported Living program could increase the quality of life for people with little prospect of paid employment. These comments align with an important theme that emerged in the literature review and among stakeholders interviewed that employment and employability should be viewed as a continuum including paid work, but also volunteer work, education, and other socially-useful activities. The validity of this idea is underscored by the fact that 30% of respondent service providers reported an increase in clients with little prospect of paid employment. Some clients also stated in the survey and focus groups that their goal was simply to find a useful daily activity, not necessarily paid employment.

Do the programs and services provided under the LMAPD support the objectives of the agreement? Are there objectives that are not supported?

Many service providers involved with CM-LMAPD cost-shared programming remain unaware of the agreement itself. In fact, 36% of service providers surveyed during the evaluation indicated no familiarity with the agreement. Key informants at the managerial level were more familiar with the CM-LMAPD. They suggested that only three of the agreement's five employment-related priority areas were strongly supported through cost-shared programming: employment and training, employment participation, and employment opportunities. They also provided reasons why connecting persons with disabilities with employers and coordinating efforts to disseminate best practices and lessons learned were less readily addressed.

Many note that employers have a poor understanding of disability programming and persons with disabilities. In some cases, this makes finding employment placements for persons with disabilities difficult. At the same time, although some service providers note that educating employers is beneficial, others state that there is little they can offer to employers to make employment and training placements "worth their while."

Key informants noted that, given the longstanding nature of much disability programming, shifting toward an emphasis on connecting persons with disabilities to employers may be challenging. One of the informants explained this well, suggesting that many staff members are not experienced or skilled at connecting to employers or promoting the idea of placements for persons with disabilities. A shift in practice will necessitate a shift in staff thinking, and will likely require considerable operational and strategic adjustment.

Provincial key informants reported that the province does not have an established forum for developing and sharing best practices among all CM-LMAPD cost-shared delivery bodies. Many service delivery organizations make individual efforts at improving practice, but only recently have broader initiatives developed. The Multi-Sector Forum and other provincial groups are examples of attempts to gather employers, service delivery managers, and provincial representatives to discuss programming.

Yet, as a few provincial respondents reported, getting representatives to participate and form an effective forum is difficult. Key informants provided a salient example of this generally acknowledged difficulty. One stated that, while the Multi-Sector Forum previously established in Manitoba brought employers and VR management together to discuss programming, it did not include representation from field staff. Another service provider also noted that their own efforts at collective work were limited because discussions often involved people with little ability to implement or affect policy.

Despite the limited provincial emphasis on connecting persons with disabilities to employers or developing best practices, under the agreement, the province is not obligated to address all LMAPD priority areas. As such, despite the lack of emphasis on these two priorities, provincial programming remains within the scope of the CM-LMAPD.

Do the LMAPD-funded programs and services align with economic independence policies and programming for persons with disabilities?

The flexibility of the CM-LMAPD is perhaps also the reason for the close alignment of supported programming with the priorities of the federal government, provincial government, and disability service providers. With the underlying assumption that employment represents an integral part of full citizenship, the programming cost-shared under the CM-LMAPD clearly aligns with the federal perspective on initiatives for persons with disabilities. The flexibility of program design provided by the agreement also allows the provincial government to tailor on-the-ground programming to the needs of its citizens. Further, the delivery approach developed in Manitoba has allowed a number of service delivery organizations with diverse goals to be included under the umbrella of the CM-LMAPD.

As evidence of the alignment, provincial key informants noted that the provincial government is currently emphasizing employment programming for persons with disabilities, independent of additional federal funding, which is capped at its pre-determined CM-LMAPD level. Examples such as strong ministerial backing and the implementation of the Rewarding Work program, a provincial initiative, both point to this increased provincial emphasis on employment programming. Provincial key informants also stated that in 2007 nearly half a million dollars - annualized to nearly a million - was added to the VR program, cost-shared under the CM-LMAPD, despite the lack of matching federal dollars.

Do all policies and programs needed to enhance the employability of persons with disabilities exist within Manitoba?

A final point about the rationale of CM-LMAPD relates to the need for additional policy. No individuals strongly suggested the need for additional legislation or legal frameworks for the integration of persons with disabilities in the workplace. Discussions with service providers revealed that, while some client groups are not equally served, only one important group is inconsistently integrated during program design and delivery-employers. Although employers represent the demand-side of the employment equation, cost-shared activities do not typically involve them. Service providers said that much more was needed than merely mandating hiring and providing financial incentives. It also must involve, for example, addressing employer concerns about the costs of hiring persons with disabilities, and providing on-the-job support and information about how to implement disability accommodations.

5.2 Design and delivery

Does an effective relationship exist among federal government, provincial government, program service providers, community based organizations, etc.?

Key informants interviewed for the CM-LMAPD evaluation stated that effective relationships support the delivery of CM-LMAPD cost-shared programming. Federal involvement is limited to the provision of funding and oversight, but this is not unusual. Some key informants, when reflecting on the previous EAPD agreement, contended that the implementation of the CM-LMAPD saw an overall reduction in federal involvement and a weakening of the federal connection to provincial or other service delivery organizations. By contrast, provincial and other service delivery organizations heavily involved in program delivery generally characterize these working relationships as strong. Further, no individual involved in the evaluation suggested that these intra-provincial relationships have changed as a result of the implementation of the CM-LMAPD.

That being said, some key informants and service providers suggested that improved referrals between organizations would benefit clients. As many service delivery organizations cater to specific disability groups, referring individuals to appropriate services is an integral part of client success. Although referrals among organizations associated with a single provincial department work well, some suggested that cross-departmental movements are more difficult. This may reflect less than optimum coordination among provincial departments. There appear to be independent paths to service for persons with disabilities and little formal cross-agency or departmental consultation with regard to delivery.

To what extent is the mix of programs and delivery processes effective in terms of coverage of participant needs and the capacity to support all potential participants?

Tailoring services to individuals appears to be central to the service delivery philosophy behind cost-shared programming. With that in mind, funding for organizations with extensive experience serving people with specific disabilities supports the effective provision of services. The Canadian National Institute for the Blind (CNIB) is a good example of one such organization. However, service providers surveyed during the evaluation noted that some groups require improved services. For example, up to a third of respondents to the service provider survey suggested that individuals with mental illness (30%) and cognitive impairments (22%) required improved services. They also stated that those with recently recognized disabilities are often less well-served. These comments were echoed by key informants who also noted that certain disability groups currently have weaker support networks than others.

For example, individuals with Asperger's syndrome, acquired brain injuries, fetal alcohol spectrum disorder, and learning disabilities lack dedicated service delivery organizations, and as a result specialized services are not available. Key informants also mention that the limited resources available provincially make it difficult to provide appropriate service to individuals with autism. Finally, providing support to individuals with multiple disabilities is particularly problematic as service delivery bodies may adequately address a primary disability but have difficulty addressing secondary or tertiary conditions.

These problems are exacerbated by aspects of some provincial programming. A few key informants, for example, noted that the disability categories used in some provincial programs are outdated and reflect an outdated understanding of disability. These programs fail to adequately recognize many of the under-served disability populations in the province. Those with recently identified disabilities do not fit neatly into the existing disability taxonomy.

Is there unnecessary overlap among the programs and services offered? If so, can and/or should it be reduced?

The majority of service providers surveyed (63%) agreed that their organization offers complementary programs and services to those provided by provincial government departments. Many (27% to 30%), however, were uncertain about how other programs available to persons with disabilities complemented their work. This was true of federal government initiatives including the Canada Pension Plan Disability program and services offered through Workers Compensation. This suggests a reasonable amount of service coordination, if primarily at the provincial level.

Overall, little evidence exists to suggest overlap among cost-shared programs. That being said, service providers stated that some clients with little long or short-term prospect of employment participate in CM-LMAPD cost-shared programming simply because no alternative exists. Key informants suggested that, for these individuals, more appropriate services could involve supported living or volunteering activities.

Key informants also highlighted a number of issues surrounding the inclusion of these individuals in CM-LMAPD cost-shared programming. First, it impacts on the overall perceived success of the cost-shared programming, which is closely focused on employment. It may also contribute to client frustration when they cannot meet the cost-shared programming's employment goals. Finally, it suggests that a more efficient use of resources to meet these clients' needs may exist.

Do LMAPD-funded programs have effective identification, selection, assessment, and placement procedures to match participants to programs?

While some individuals with little prospect of paid employment received services under CM-LMAPD programming, little evidence indicates that eligible clients were denied or could not access support. Discussions with VR clients did point to some variability in timely access to services and success in education, training, and employment. However, access to programming was largely universal if not immediate for all clients. In fact, the universal nature of the programming made the identification of a comparison group for impact analysis impossible.

It appears that one of the main barriers that prevent clients from accessing programming is a lack of knowledge about available services. During focus group discussions, VR clients, regardless of their type of disability, identified an initial lack of awareness of the VR program as one of their main difficulties. Although all clients participating in the focus groups were either current or former VR clients, many noted that prior to their entry into the program they had little information about VR. Many entered into the program after time at a medical facility or after contacting the Society for Manitobans with Disabilities (SMD), the Canadian Paraplegic Association of Manitoba (CPA), or CNIB for general disability support. The example of spending years in university not knowing about VR, but finding out about VR soon after going to a designated agency for disability-related support, resonated with participants.

What best practices or lessons learned have been or can be learned about the design and delivery of LMAPD-funded programs and services provided under the agreement?

As mentioned above, there is no formal mechanism for developing and disseminating best practices and lessons learned among CM-LMAPD cost-shared programs. Some best practices appear to develop within organizations and service delivery bodies based on their own work. In fact, their development of best practices does not appear to be directly influenced by the CM-LMAPD in any significant way. Many, if not all, of the service delivery organizations under the umbrella of the CM-LMAPD predate the agreement, and the development of best practice activities would have likely continued in its absence. As such, the development of these best practices cannot be attributed to the CM-LMAPD.

How could/should Manitoba improve the programs and services it offers under the LMAPD?

Suggestions for the improvement of programming elicited varied responses. A summary of the suggestions noted in the client survey, focus group, survey of service providers, survey of employers, and key informant interviews revealed consistent ideas. Among service providers, for example, the desire for additional funding to meet a growing demand for services was evident. Many also reported that cases were becoming more complex and require additional and more intensive services.

Others also noted that expanding the scope of services beyond standard employment programming would benefit many clients. These individuals noted that while many clients come to CM-LMAPD programming with few job skills, some do not understand basic social and workplace behaviours. They must acquire such skills so they can transition to more formal job training and then to employment, which is a major reason why this group requires sustained training and support.

The need for greater involvement of employers in programming was also a consistent theme during the evaluation's interviews and discussions. The literature also provided support for greater employer involvement. Along similar lines, it highlighted the importance of targeted programming that recognizes not only the role of clients, but also employers, community, and non-governmental organizations. This view suggests an approach that focuses not only on client-centered remediation but also on addressing barriers to employment beyond the control of the individual, including the failure or inability to accommodate individuals in workplaces.

However, service providers of CM-LMAPD cost-shared programming have limited direct contact with employers. Service providers may contact employers as clients reach training milestones or finish their programming and typically cultivate relationships individually. The employer community is not substantively involved in the design of interventions or the policy development process. Many interviewees mentioned that employers must be more engaged in the CM-LMAPD and believe this change alone would have far-reaching and beneficial impacts on program success.

Despite the call for increased employer participation, focus groups and dyad discussions with VR clients offered two important cautionary points. The attitudes of not only managers/employers but also co-workers affect the success of persons with disabilities in the workplace. As an example, mental illness remains a persistent stigma in society. People with hearing impairments are often cut out of the informal dialogue that is an essential part of a modern office through no overt discrimination. Improving the perception of persons with disabilities, held by employees, complements direct employer participation in programming. In addition, the costs employers incur to accommodate employees with disabilities can be much greater than the costs to employ other people. Failure to find ways to offset these costs, invariably limits the pool of available jobs for persons with disabilities.

5.3 Impacts

What have been the differential gross impacts of LMAPD programming on groups of participants? What factors predict successful completion of programs? What factors predict success in the labour market? To what extent have participants reduced dependency on EI/SA, increased employment and or wage, increased their quality of life, and increased job or career satisfaction?

Outcomes associated with programming

Surveyed service providers from Family Services and Housing and Manitoba Health reported that their clients' employment situations and quality of life typically improved during the CM-LMAPD period. They note that approximately half of their clients completed programming as planned. They also indicated that although client participation in the programming improved employment outcomes over the evaluation program period of 2004 to 2006, their perceived quality of life and educational levels showed even greater improvements.

Service providers who were surveyed identified many client benefits from programming. While commenting on the effectiveness of their programming during the CM-LMAPD period:

• 77% agreed that the quality of life of their clients had improved;
• 73% agreed that their clients had greater independence;
• 65% agreed that their clients had increased their level of education or training;
• 64% agreed that their clients were able to better maintain employment;
• 63% agreed that their clients had successfully secured employment; and
• 54% agreed that their clients had greater employment opportunities.

By contrast:

• 14% disagreed or were neutral that the quality of life of their clients had improved;
• 18% disagreed or were neutral that their clients had greater independence;
• 22% disagreed or were neutral that their clients had increased their level of education or training;
• 22% disagreed or were neutral that their clients were able to better maintain employment;
• 20% disagreed or were neutral that their clients had successfully secured employment; and
• 32% disagreed or were neutral that their clients had greater employment opportunities.

Results from the survey of VR clients corroborated this, showing increases in clients' educational levels over the program period. This suggests that programming supports intermediate outcomes that may eventually lead to employment. Of the VR clients surveyed, more had completed post-secondary education in 2007 than in 2003. This increase in post-secondary educational attainment was most striking for clients with learning (25%) and physical (19%) disabilities. Table 3 provides additional details on these changes.

• Source from the above table Source: Client survey

In terms of direct employment outcomes, the number of VR clients without a job increased slightly between 2003 and 2007. In addition, the average number of hours worked decreased slightly between 2003 and 2007. Table 4 presents these and other changes to VR client employment outcomes.

• Source from the above table Source: Client survey
  Note: Totals do not sum to 100% due to rounding.

The most obvious explanation for these decreases was that increased participation in employment services precluded employment. For those clients who were working in both 2003 and 2007, the average number of hours worked increased. The average weekly employment income for those clients who were working in both 2003 and 2007 also increased. Although relatively small, these changes suggest a positive program impact over the evaluation program period.

• Source from the above table Source: Client survey

Client job satisfaction among VR clients also increased somewhat between 2003 and 2007. In 2003, the split between satisfied and unsatisfied was an even 41%, whereas in 2007 the split had changed to 49% satisfied and 33% unsatisfied. More clients also reported that their quality of life had improved from 2003 to 2007-an increase from 35% of clients in 2003 to 43% in 2007 ^Footnote3 .

When asked how their job situation affected their quality of life, respondents most often mentioned a positive impact on self-esteem or confidence, greater income, increased time spent with others, and a better feeling about how others perceived them. Some individuals also mentioned feeling better because their independence level had improved between 2003 and 2007. Table 6 notes these and other reasons for job satisfaction noted during the VR client survey.

• Source from the above table Source: Client survey
  Note: Respondents could choose more than one answer. Totals may sum to more than 100%.

Impacts of programming

Because of the lack of a comparison group, the evaluation only conducted a limited treatment analysis of the VR programming. This analysis involved assessing the net impact of program components, provided in different combinations to each client, relative to a baseline level of treatment received by all. Conceptually, this meant assessing the impact of training, educational, or other VR components relative to the client counselling services received by all.

The regression analysis examined six outcomes including the change in weekly hours worked, annual employment earnings, hourly wage, educational attainment, EIA use, and Employment Insurance (EI) use over the program period. Appendix D includes tables of results from these regressions. Readers will find that these tables present a considerable amount of information about the factors associated with these outcomes. However, some key observations bear mention. These observations are based on the results presented in Appendix D, which readers should consult for their substantiation.

The first observation involves the type of programming offered under the VR program. The analysis found that the programming offered through the CM-LMAPD has positive impacts on outcomes. For example, educational programming was associated with an improvement in educational attainment as one would expect and hope. Similarly, employment-related interventions were associated with improvements in the more direct employment outcomes such as earnings and hours worked. These relationships were complex, often showing different marginal impacts depending on the number of program activities attempted for a single client. Overall, this appears to suggest that VR program components are successful at influencing their expected client outcomes.

However, many of the estimated program impacts were quite small. This may be partly a result of the short program period established for the evaluation. Experts, service providers, and program managers stated that it might take five to seven years of programming for persons with disabilities to receive considerable benefits. However, the short-term impacts may suggest additional influences in the long term.

Second, the service delivery organization providing programming appears to influence client outcomes. This is true even when controlling for other factors including disability type and severity. Under the VR program, provincial offices and three designated agencies provide services to clients with disabilities. The analysis noted that some of the designated agencies appear to result in a more positive impact on client outcomes than others.

Finally, disability type and self-report disability severity appear to influence program success. Impact estimates for disability types varied considerably across outcome types. However, the influence of disability severity appeared consistent. Generally, the more severe an individual's disability, the less likely he or she is to have a positive program outcome estimated by the current analysis. Bearing the possibility of endogeneity associated with any self-report measure of disability severity, this appears to align with the accepted association between disability severity and work-related outcomes voiced throughout the evaluation and noted in the literature.

These disability-related findings also appear to align with statements from VR clients about their own sources of income and disability severity. Table 7 presents changes in sources of financial support among VR clients over the evaluation program period by disability severity. As the table shows, greater disability severity is associated with a decrease in full-time employment over the program period. Also, mild disability severity is associated with a far greater reduction in reliance on friends and family for financial support.

• Source from the above table Source: Client survey.

In sum, the CM-LMAPD cost-shared programming appears to provide a diversity of positive employment impacts to clients. In addition, it is likely a contributing factor to increased client satisfaction through its effects on employment outcomes.

How satisfied are participants with LMAPD-funded programs and services provided under the agreement?

During the evaluation, both clients of the VR program and employers associated with various CM-LMAPD service providers rated their satisfaction with cost-shared programming highly (Table 8). Clients commented on seven kinds of VR services that they had received since 2003: counselling, work assessments, tuition funding, work training or unpaid work experience, transportation funding, technical aids and adaptive devices, and follow-up services for supported employment. Only a small percentage of respondents indicated that they were very unsatisfied with any one service (3% to 7%).

However, it is important to note that not all clients responding to the survey had received all services. This is because VR counsellors actively coordinate services to meet specific client needs. As many as 74% of survey respondents had not received a particular service.

• Source from the above table Note: Totals sum horizontally and may not sum to 100% due to rounding.

When probed further about their reasons for dissatisfaction, clients' were primarily concerned with unsuitable work training placements, a lack of communication with counsellors or support staff, or a lack of funding or financial support to obtain the services the client wanted to receive.

Satisfaction levels of employers with the type/level of skills training received by program participants. Level of employer satisfaction with the type/level of assistance provided to support workers with disabilities.

Generally, employers indicated that many individuals placed with their organization worked out well: 39% indicated that the individual placed with their organization had integrated well all or most of the time, while another 28% stated that the individual had integrated well about half the time. Fewer individuals met the work skills requirements the employers expected: 25% said the individual placed with their organization had job-related skills that met or exceeded their expectations all or most of the time; and 31% indicated that this occurred rarely or never. The majority of employers (71%) were satisfied with service providers and could not identify changes that would better meet their needs. The majority of employers (79%) also indicated that service provider organizations offered sufficient support to clients to allow them to succeed.

Have LMAPD-funded programs and services resulted in any unintended impacts? What general benefits have LMAPD-funded programs and services had on the community in terms of increased employment and overall economic growth?

Overall, the evaluation did not identify any general unintended impacts of the CM-LMAPD. This may partly be a result of the long-standing nature of many of the programs cost-shared under the CM-LMAPD. In addition, there were few indications of considerable community-level economic impact including growth. Some key informants note that the employment of staff at service provider organizations and in the provincial government represented a beneficial impact on the community. Key informants also noted that the successful employment of clients results in provincial savings to EIA and related support programs. Finally, some key informants indicated that simply working with persons with disabilities in the community raised awareness about disability and may eventually improve employment success for clients and others with disabilities.

5.4 Cost-effectiveness

What are the costs of delivering LMAPD-funded programs and services?

As noted above, the total CM-LMAPD expenditure for cost-shared programming was $20,174,400 in 2004-2005 and $21,747,500 in 2005-2006. Of this, the federal government contributed $8,965,000 in both fiscal years. Manitoba has typically paid about 60% of annual program costs.

The individual client records used for the evaluation included only the cost of direct funded services to VR participants. As such, they excluded many costs of programming. The most important of these were those associated with staff time used for client counselling. Counselling costs are embedded in the budgets and salaries of the Family Services and Housing and designated agency offices providing these services.

The evaluation estimated these costs by having 281 client counsellors complete a short questionnaire identifying the number of hours they spent with randomly selected clients. A regression technique allowed for the imputation of counselling time and, by extension costs, to the full VR client sample used for the evaluation. This produced an average estimated counselling cost per client of $3,095 from January 1, 2004 to December 31, 2006. When combined with each client's funded services, the average cost per client was $4,887 per client for the same period. Table 9 presents the average cost of VR services for a number of client sub-groups.

• Source from the above table ¹Sub-categories are not mutually exclusive and, as such, individual n values will sum to more than 1,062.

Are there more cost-effective ways of delivering programs and services to achieve the same objectives?

It is difficult to determine whether there are more cost-effective ways to provide CM LMAPD cost-shared programming. Best practices identified in the literature review suggest certain ways to increase effectiveness, but some are often very expensive. For example, while there is a great deal of evidence to suggest that individualized programming is the most effective kind of programming for persons with disabilities, this approach is very labour intensive, and its value for money has not yet been studied.

Many service providers claimed that programs are as efficient as possible. However, a few did suggest ways to reduce costs while maintaining success, such as moving towards non-governmental service delivery. Another involved block-funding service delivery organizations to allow greater flexibility and a way to transition to individualized service. Some suggested that implementing an outcome tracking system to identify successes and failures in training would improve efficiency. A few employers also provided suggestions to improve cost-effectiveness, such as providing better pre- and post-employment services and more education or communication to the employer community about employment services. VR clients did not provide much feedback concerning more cost-effective ways to deliver the program.

Table 10 presents the average gross changes in the five outcomes measured in the econometric analysis per $1,000 of program expenditure. The figures are based on data from both the client survey and the administrative data available to the evaluation.

• Source from the above table Source: Constructed from client survey and administrative data

Although the table above does suggest some return on VR spending, the increases in EIA payments for clients appear counter intuitive and are traditionally viewed as a negative result. However, the evaluation's program period aligned only with the period of the CM-LMAPD; many clients receive programming for longer periods of time. Many service providers and experts stressed that clients often need far more than three years to complete employment-related activities successfully. Therefore, the three-year program period established for the evaluation would fail to capture the success of many clients. In fact, most of the individuals who had a substantial decrease in EIA use from 2003 to 2007 had been in the program twice as long (80 months) as those who experienced a substantial increase in their EIA use (44 months). The number of clients falling into the second group far exceeds the number in the first group, which skews the overall results.

What are the potential savings generated by the programs?

It is not possible to assess savings generated by the CM-LMAPD clearly, as no overall net impact analysis of the cost-shared programming was possible. Figures highlighted and discussed above only include savings that may have resulted over the program period.

6. Conclusions

As the first evaluation of a Labour Market Development Agreement for Persons with Disabilities, the CM-LMAPD evaluation attempted to provide policy and program insights for federal and provincial governments. Although it was not a summative evaluation, it did include detailed quantitative and qualitative analyses of the employment services for persons with disabilities. This report provided an overview of the evaluation findings, including many details about the success of the CM-LMAPD.

Overall, there is strong rationale to support the services and programming offered under the CM-LMAPD. Substantial differences exist between people with disabilities and the broader population with regard to employment outcomes. The employment of individuals with disabilities not only contributes to individual well-being, it also reduces the need for ongoing income support from government and families. In addition, for those who chose to work, employment is an important aspect of full citizenship-a goal of the government of Canada. The CM-LMAPD funds programming intended to support persons with disabilities to gain increased financial independence through market-based employment.

The CM-LMAPD is the latest in a series of funding arrangements supporting programming for persons with disabilities. The current programs are the result of years of incremental refinement by the provincial government-the CM-LMAPD simply changes the funding arrangements while additionally recognizing the importance of many stakeholders in the employment process. The refinements focused on the supply-side approach to programming, adapting best practices from the literature, and involving broad stakeholder and delivery groups. However, far less has been done to address the demand side, and employers are less engaged under this agreement.

Based on an examination of the VR program cost-shared under the CM-LMAPD, the evaluation revealed positive employment-related changes over the program period. Certain client groups showed relatively more progress in key outcomes, such as earnings and decreased use of EIA. These included, for example, people with learning disabilities and people who have received services over an extended period. Disability severity appears to be important: clients who report greater disability severity experience less growth in earnings and hours of work. More frequent client contact and intensity of services, as well as sustained follow-up, seem to be important for realizing employment success. In addition, satisfaction with quality of life among clients improved. Finally, compared to government-run agencies, certain designated agencies appear to have a more positive impact on client outcomes.

In terms of costs, Manitoba typically pays for approximately 60% of CM-LMAPD program costs. Since an overall net impact assessment was not feasible, the evaluation could not measure the cost-effectiveness of programming. However, many involved in programming believe that services are provided in the least costly and most effective manner possible. Again designated agencies seem to have an edge compared to government-run programming.

The evaluation presents only a partial picture of the CM-LMAPD. A fuller picture requires additional program data and the full participation of all departments involved in program delivery. The multilateral agreement's commitment to continual improvement requires acknowledgement of the following four dimensions of the disability experience. First, the population is very diverse, with clients presenting a diverse range and depth of need. Second, training may have a range of outcomes, some of which are not necessarily reflected in increased earnings or hours of work. Some clients may simply increase their capacity to undertake further training and not reveal changes in financial independence. Third, and related to the last point, outcomes for this group are usually realized over a longer period than active labour market programs with other client populations. Fourth, the demand side must be acknowledged; employers are important partners and can be more engaged in programming supported by the agreement.

References ^Footnote4

Bernell, S. L. (2003). Theoretical and Applied Issues in Defining Disability in Labour Market Research. Journal of Disability Policy Studies, 14(1), p. 36-45.

Federal/Provincial/Territorial Ministers Responsible for Social Services. (1997). In Unison: A Canadian Approach to Disability Issues: A Vision Paper. Hull, Quebec: Human Resources Development Canada.

Jongbloed, L. & Crichton, A. (1990). Difficulties in shifting from individualistic to socio-political policy regarding disability in Canada. Disability, Handicap & Society, 5(1), p. 25-36.

Lunt, N. & Thornton, P. (1994). Disability and Employment: Towards an understanding of discourse and policy. Disability & Society, 9(2), p. 223-238.

Moll, S., Huff, J., & Detwiler, L. (2003). Supported Employment: Evidence for a Best Practice Model in Psychosocial Rehabilitation. The Canadian Journal of Occupational Therapy, 70(5), p. 298-310.

Office of Disability Issues, Human Resources Development Canada. (2003). Defining Disability: A Complex Issue. Gatineau: Government of Canada.

Province of Manitoba. (2005). Labour Market Agreement for Persons with Disabilities (LMAPD - LMAPD Report 2004-2005. Winnipeg: Province of Manitoba.

Province of Manitoba. (2006). Labour Market Agreement for Persons with Disabilities (LMAPD - LMAPD Report 2005-2006. Winnipeg: Province of Manitoba.

Province of Manitoba. (2009). Labour Market Agreement for Persons with Disabilities (LMAPD - LMAPD Report 2008-2009. Winnipeg: Province of Manitoba.

Schalock, R. L. (2004). The Emerging Disability Paradigm and its Implications for Policy and Practice. Journal of Disability Policy Studies, 14(4), p. 204-215.

Social Development Canada. (2005). Request for Proposal - Formative Evaluation of the Manitoba Labour Market Agreement for Persons with Disabilities. Ottawa: Service Canada.

Social Development Canada. (2008). Introduction to Multilateral Framework for Labour Market Agreements for Persons with Disabilities. Retrieved November 19, 2008, from Benefits and Services for Persons with Disabilities

Stapleton, D. C., & Burkhauser, R. V. (2003). The Decline in Employment of People with Disabilities: A Policy Puzzle. Unpublished Manuscript, W. E. Upjohn Institute for Employment Research.

Statistics Canada. (2002, December). 2001 Participation and Activity Limitation Survey: A profile of disability in Canada, 2001 - tables. Ottawa: Statistics Canada.

The Government of Canada & The Government of Manitoba. (1997, March 21). Vocational Rehabilitation of Disabled Persons Agreement between Canada and Manitoba.

The Government of Canada & The Government of Manitoba. (1998, April 14). Canada-Manitoba Agreement on Employability Assistance for People with Disabilities.

Torjman, S. (2001).Canada's federal regime and persons with disabilities. In D. Cameron & F. Valentine (Eds.), Disability and federalism: Comparing different approaches to full participation, (pp. 151-196). Montreal & Kingston: McGill-Queen's University Press for the Institute of Intergovernmental Relations, School of Policy Studies, Queen's University.

World Health Organization. (2002). Toward a common language for functioning, disability and health. Geneva: World Health Organization.

Appendix A

CM-LMAPD Evaluation Issues and Questions

Rationale

1. What need exists for programs and services to support the employment goals of persons with disabilities?

2. Is the LMAPD relevant to the employment needs of persons with disabilities?

3. Do the programs and services provided under the LMAPD support the objectives of the agreement? Are there objectives that are not supported?

4. Do the LMAPD-funded programs and services align with the economic independence policies and programming for persons with disabilities of:

• the province of Manitoba?
• the federal government?
• Program/service providers?

5. Do all policies and programs needed to enhance the employability of persons with disabilities exist within Manitoba? Are there specific measures in the legal framework (Department of Labour and/or Human Rights) aimed at integrating persons with disabilities into the workplace? Are all groups that should be, included in the LMAPD (e.g., employers, unions, secondary and post-secondary institutions, etc.)?

Design and delivery

6. Does an effective relationship exist among federal government, provincial government, program/service providers, community based organizations, etc.? What if any changes are needed? Has the change in agreements Fed-Prov from the EAPD to LMAPD generated any changes in program delivery and/or administration? Does the implementation and operation of the EAPD/LMAPD reflect the basic principles underpinning the federal and provincial governments' objectives, and what changes would be desirable, if any?

7. To what extent is the mix of programs and delivery processes (government, NGO, private firm) effective in terms of coverage of participant needs and the capacity to support all potential participants?

8. Is there unnecessary overlap among the programs and services offered? If so, can it be/should it be reduced?

9. Do LMAPD-funded programs have effective identification, selection, assessment, and placement procedures to match participants to programs? Is the process inclusive? What, if any, types of persons with disabilities are excluded?

10. What best practices/lessons learned have been/can be learned about the design and delivery of LMAPD-funded programs and services provided under the agreement? Has this information been applied to programs and services? Is it shared among the program/service providers and federal and provincial governments?

11. How could/should Manitoba improve the programs and services it offers under the LMAPD?

Impacts

12. What have been the differential gross impacts of LMAPD programming on groups of participants, specifically with respect to:

• Numbers of participants successfully trained/educated?
• Labour force participation rate? Earnings and total income?
• Quality of Life?
• Overall level of independence?

13. What factors predict successful completion of programs? What factors predict success in the labour market?

14. To what extent have participants:

• Reduced dependency on EI/SA?
• Increased employment/wages?
• Increased their quality of life?
• Increased job/career satisfaction?

15. How satisfied are participants with LMAPD-funded programs and services provided under the agreement?

16. How satisfied are employers with the type/level of skills training participants receive? Are they satisfied with the type/level of assistance provided to support workers with disabilities?

17. Have LMAPD-funded programs and services resulted in any unintended impacts such as:

• Increased use of EI or SA in the [short/medium/long]-term?
• Unrealistic expectations placed on participants?

18. What general benefits have LMAPD-funded programs and services had on the community in terms of increased employment and overall economic growth?

Cost-effectiveness

19. What are the costs of delivering LMAPD-funded programs and services?

20. Are there more cost-effective ways of delivering programs and services to achieve the same objectives?

21. What are the potential savings generated by the programs? i.e., in Health care, Social assistance and other government services.

Appendix B

• Source from the above table * Clients could provide more than one response
  ** Disability types are not mutually exclusive

Appendix C

Appendix D

Econometric Analysis: Tables of Results

• Source from the above table Note: M.E.-mutually exclusive; N.M.-non-mutually exclusive; PP-program period

• Source from the above table Note: M.E.-mutually exclusive; N.M.-non-mutually exclusive; PP-program period

• Source from the above table Note: M.E.-mutually exclusive; N.M.-non-mutually exclusive; PP-program period

• Source from the above table Note: M.E.-mutually exclusive; N.M.-non-mutually exclusive; PP-program period

• Source from the above table Note: M.E.-mutually exclusive; N.M.-non-mutually exclusive; PP-program period

• Source from the above table Note: M.E.-mutually exclusive; N.M.-non-mutually exclusive; PP-program period