Evaluation of the First Nations and Inuit Mental Wellness Programs 2010-2011 to 2014-2015

Prepared by
Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

July 2016

Executive Summary
Management Response and Action Plan
Appendix 1 - Logic Model
Appendix 2 - Summary of Findings
Appendix 3 - Evaluation Methodology
Appendix 4 - References

List of Tables

Table 1 - Programs' Expenditures from 2010-2011 to 2014-2015 ($ millions)
Table 2 - Limitations and Mitigation Strategies
Table 3 - Client Satisfaction with IRS RHSP Services
Table 4 - Summary of MW Programs Expenditures from 2010-2011 to 2014-2015

Executive Summary

The evaluation covered the First Nations and Inuit Health Branch Mental Wellness (MW) Programs for the period from April 2010 to March 2015. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance of the MW Programs (including the Brighter Futures (BF) program, Building Healthy Communities (BHC) program, National Aboriginal Youth Suicide Prevention Strategy (NAYSPS), National Native Alcohol and Drug Abuse Program (NNADAP), National Youth Solvent Abuse Program (NYSAP), and the Indian Residential Schools Resolution Health Support Program (IRS RHSP)) as well as the activities of the Mental Wellness Teams. Note that other strategies/funding sources form part of the FNIHB's MW Programs, but are excluded from the scope of this evaluation. For the purposes of this report, the term "MW Programs" refers only to those programs within scope for this evaluation. The evaluation scope included all activities of these MW Programs, with the exception of those undertaken in British Columbia and in northern Canada (north of the 60th parallel).

The methodologies used in the evaluation included a document and administrative data review, a literature review, 40 key informant interviews including Health Canada staff, Mental Wellness Team (MWT) coordinators, community leads and representatives of other government and First Nations and Inuit organizations, an online survey of 157 health directors/staff and 13 community leaders, and site visits to 14 communities representing six regions of Canada.

Program Description

The MW Programs fund community-based programs and services that aim to provide treatment, reduce risk factors, promote protective factors and improve health outcomes associated with the mental wellness of First Nations people and Inuit. The MW Programs provide a range of culturally-relevant mental health and addictions programs and services which are guided by priorities established by First Nations and Inuit communities.

There are six programs which together form the MW Programs, namely Brighter Futures (BF) program, Building Healthy Communities (BHC) program, National Aboriginal Youth Suicide Prevention Strategy (NAYSPS), National Native Alcohol and Drug Abuse Program (NNADAP), National Youth Solvent Abuse Program (NYSAP), and the Indian Residential Schools Resolution Health Support Program (IRS RHSP). The total budget for these MW Programs was $1.4 billion over the five-year period covered under the scope of this evaluation. Of this total, a combined $407 million was spent on NNADAP, $266 million on BF, $208 million on BHC, $96 million on the NYSAP, $58 million for the NAYSPS, and $285 million on IRS RHSP. As per the IRS Settlement Agreement, the IRS RHSP is set to sunset in 2016 and it is unclear if additional programming will be made available to address the specific needs of IRS survivors.

The total budget, which includes all related strategies and additional funding sources which support the funding of the Mental Wellness Teams (MWTs), was $1.5 billion over the five-year period covered under the scope of this evaluation.

Conclusions - Relevance

Continued Need for the Programs

In Canada, there are a number of risk factors associated with mental wellness issues, including substance abuse and suicide. First Nations people and Inuit experience a disproportionately high prevalence of suicide, drug and alcohol addiction, substance use and other mental health problems compared to the overall Canadian population. For example, between 2005-2007, the suicide rate among youth under the age of 19 living in areas with a high First Nations population was 11 times higher for males and 21 times higher for females, as compared to youth living in areas with a low First Nations population (Peters, Oliver, & Kohen, 2013). Similar trends exist for Inuit populations, with suicide rates among youth under the age of 19 living in areas with a high Inuit population (Inuit Nunagat) being 35 times higher for males and 28 times higher for females compared to suicide rates among the Canadian population overall between 2004 to 2008.

Many First Nations and Inuit communities face crises related to suicide, alcohol and drug use, communicable diseases and child welfare apprehensions. The combined influence of historical and current traumas and stressors on some First Nations and Inuit individuals, families and communities are contributing factors to a high level of crises at the individual, family and community level. The activities implemented by the MW Programs aim to address the needs of First Nations people and Inuit and combat the high rates of substance abuse, suicide and other mental health issues.

The findings of this evaluation are aligned with results of the evaluation of MW programs conducted in northern Canada (north of the 60th parallel) covering the period from 2005 to 2010. Both evaluations concluded that there is an ongoing need to address the high rates of suicide, alcohol and drug abuse, and other mental wellness issues, among First Nations and Inuit communities.

Alignment with Government Priorities

The objectives and activities of the MW Programs, which include the promotion of positive mental wellness, as well as alcohol and substance abuse treatment, are aligned with the priorities of the federal government and the strategic outcomes of the Department.

Alignment with Federal Roles and Responsibilities

The federal government has a role in the provision of health care for First Nations and Inuit populations. The MW Programs appear to complement, rather than duplicate, other federal programs which are designed to address mental health and addiction issues in First Nations and Inuit communities. The MW Programs are unique in terms of their broader scope than other federal programs, and focus on ensuring the services reflect First Nations and Inuit culture, traditions, and way of life.

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

While it was difficult to accurately measure the programs' performance due to limited performance measurement data, the available performance data, combined with the field work findings, indicates that the MW Programs have made progress towards their intended outcomes. The evaluation found that recent efforts by the MW Programs to integrate culture and traditions into mental wellness programming has resulted in an increase in the practice of healthy behaviors due to increased acceptance and participation in MW programs and services. The most frequently cited examples of positive changes in behaviour by First Nations and Inuit individuals were reduced substance abuse, increased self-esteem and confidence, improved sense of positive cultural identity, strengthened family and community relations, and increased help-seeking behavior. The integration of culture into MW Programs has also resulted in increased community ownership to address mental health issues. The Mental Wellness Teams have been particularly effective in integrating culture into mental wellness programs and have been able to achieve significant results including greater access to specialized services and increased ownership among community members. The First Nations Mental Wellness Continuum Framework has been instrumental in providing guidance regarding the integration of culture and traditions into mental wellness programming, as well as outlining the continuum of essential services necessary to address the needs of First Nations people.

The MW Programs have made progress in improving the continuum of the programs and services in First Nations and Inuit communities through greater focus on coordination of services among federal, provincial and community service providers; increased use of case management services; the introduction of new initiatives such as Mental Wellness Teams; and the development of a number of frameworks, tools and resources. However, there continue to be gaps associated with early identification and intervention, crisis planning and response, trauma-informed treatment, access to detox, access to specialized/professional services and aftercare services. Another ongoing challenge is the ability to reach those who are most vulnerable and who need the services the most. Insufficient capacity to deliver MW Programs remains one of the most pressing issues in many communities. While MWTs have increased capacity in the communities they serve, the 11 Mental Wellness Teams throughout the country currently provide services to only approximately 12% of the total number of First Nations and Inuit communities covered by the evaluation.

Demonstration of Economy and Efficiency

The MW Programs have undertaken a number of resource maximization measures to enhance their economy and efficiency. However, there are several factors that have constrained the efficiency and economy of the MW Programs, including resource challenges related to funding amounts remaining largely unchanged over many years; short term funding cycles; high turnover of program staff in the communities; lack of integration among MW Programs; insufficient funding flexibility at the community level; and insufficient data to assess the performance of MW Programs and allocate resources to the most effective MW programming.

Recommendations

Recommendation #1:

Focus should be placed on addressing gaps in the continuum of essential services in collaboration with First Nations people as implementation of the First Nations Mental Wellness Continuum Framework moves forward.

Gaps currently exist in the continuum of services available at the community level. These include gaps associated with early identification and intervention, crisis planning and response, trauma-informed treatment, access to detox, access to specialized/professional services and aftercare services. Consistent with the First Nations Mental Wellness Continuum Framework, potential means by which these gaps could be addressed include collaborating with partners at the regional level, such as provincial service providers, communities and Tribal Councils, to address issues related to wait times, access, and cultural competency of specialized service providers. Building the capacity of communities to provide specialized care has been effective in some communities; however, due to confidentiality reasons, service availability or urgent situations, some community members have been required to access such services outside of their community. Better coordination, such as referral systems, case management and discharge planning, including to P/T services as well as at NNADAP/NYSAP, treatment centers will help community health services to better connect with those returning to communities, and provide improved aftercare services

Recommendation #2:

Enhance efforts in support of building community capacity to deliver effective, quality services to address substance abuse, suicide and other mental health issues and increase community resilience.

The lack of capacity of communities to deliver effective services is one of the most critical areas affecting the success of the Mental Wellness Programs. A number of factors currently contribute to this issue including: high turnover of community program staff due in part to wage disparities; limited clinical and cultural supervision for existing staff members; and, communication challenges with staff in the communities related to program updates and sharing of best practices. In addition, there is a mixed level of awareness at the community level of these training materials and events. As well, the degree to which current training materials are aligned with community needs is unclear.

Moving forward, greater effort should be placed on sharing program related information (including best practices, benchmarks and lessons learned in the field of mental wellness) with communities to enhance capacity building efforts. Similarly, there is a need to ensure that existing training resources are tailored to address the most pressing issues and that they focus on critical gap areas, such as trauma informed care. Helping communities to enhance internal policies and procedures and staff capacities to ensure confidentiality and protect private information would help to enhance overall effectiveness of the services.

In order to enhance access to quality MW programs and services, it is also necessary to increase efforts to incorporate culture and traditions into mental wellness program design and delivery by implementing First Nations approaches to mental wellness along with western style services (e.g., involving traditional healers, Elders, and other cultural practitioners). As specified in the First Nations Mental Wellness Continuum Framework, integrating cultural knowledge into programs will act as a catalyst for healing for First Nations individuals and will improve the effectiveness of MW programs. The evaluation found that increased participation in MW Programs has occurred primarily as a result of the incorporation of First Nations culture and traditions into mental wellness programming. The community-driven approaches used by communities to develop and implement MWTs should be applied more broadly because they have been particularly effective in integrating culture into mental wellness programs as well as building community capacity.

Recommendation #3:

Continue to integrate First Nations and Inuit Mental Wellness programming at the national and regional levels within Health Canada, and formalize mechanisms to ensure integration across the individual programs.

Despite improvements in integration over recent years, FNIHB MW Programs are still largely implemented in silos within Health Canada at the national and regional levels, with insufficient interaction among staff in different programs. The evaluation identified variation regarding the extent to which staff across programs collaborate and integrate the activities of individual programs, at both the regional and national level. There also exist variations with regard to mechanisms for ensuring regular communication and the sharing of best practices by staff at the community, regional and national level.

Greater integration of the MW Programs at all levels would facilitate better cooperation and interaction among staff members at the national and regional offices and would improve the coordination of program activities at the community, sub-regional and regional level.

Recommendation #4:

Develop a comprehensive Performance Measurement Strategy to guide the collection and usage of performance measurement data.

The evaluation findings demonstrate significant improvements in the availability and quality of performance data for MW Programs. However, the consistency of performance data across activities and years would be improved by creating a comprehensive Performance Measurement Strategy that provides detailed information with regards to sources of performance data, identifies those responsible for collecting and reporting the data, and describes the frequency and timing of the data collection. Because the MW Programs include complex programming and cover a large number of the communities, special studies (e.g., Mental Wellness Team evaluations and NAYSPS case studies) are an effective way of providing more in-depth assessments of specific areas and identifying best practices.

Management Response and Action Plan Evaluation of the First Nations and Inuit Mental Wellness Programs
Recommendations	Response	Action Plan	Deliverables	Expected Completion Date	Accountability	Resources
Focus should be placed on addressing gaps in the continuum of essential services in collaboration with First Nations as implementation of the First Nations Mental Wellness Continuum Framework (FNMWCF) moves forward.	Management agrees with the recommendation and is working toward addressing gaps in the continuum of essential services with First Nations and other Partners.	Building on the work to develop the FNMWCF, complete a gap analysis of MW programs and services which can inform future program and policy decisions.	1. Completed gap analysis of federal MW programs and services	1. May 2016	1. Executive Director, PHPCD, FNIHB	No additional resources required
	The Framework outlines a holistic approach to mental wellness services with First Nations culture as the foundation, based on care along a continuum of services, away from siloed programs, delivery mechanisms and systemic gaps. Key to applying the MWCF is collaboration. Management is working with partners at various levels including those at the regional level such as provincial service providers, communities and Tribal Councils to implement the MWCF and find collaborative ways to address issues related to access and to increase the cultural competency of specialized services.	Work through regional partnership tables to develop work plans which support addressing gaps in the continuum of essential services.	2. Completed 4 regional work plans to address gaps in the continuum of essential services	2. September 2016	2. Regional ADM, FNIHB
		Support the implementation of FNMWCF demonstration projects focused on culture as foundation and increasing access to essential services. The projects will demonstrate the impacts of what communities are doing to implement key components of the FNMWCF and facilitate knowledge exchange between communities on innovations and promising practices to address gaps in the continuum of services.	3. Completed 5 knowledge exchange reports and present findings to First Nations partners at FNMWCF Implementation Team meeting.	3. March 2017	3. Executive Director, PHPCD, FNIHB
		To support implementation of the FNMWCF, work with First Nations partners to develop a tool kit which describes how the FNMWCF model can be used by stakeholders to address gaps in the continuum of essential services.	4. Completed FNMWCF model toolkit	4. March 2017	5. Executive Director, PHPCD, FNIHB
Enhance efforts in support of building community capacity to deliver effective and quality services to address substance abuse, suicide and other mental health issues and increase community resilience.	Management agrees with the recommendation. Community development, ownership and capacity building is one of 5 key themes in the FNMWCF and are significant factors that must be taken into account when enhancing mental wellness in First Nations communities. To support the ongoing knowledge and skill development of service providers and community in addressing addictions and mental health issues, practice based evidence alongside with indigenous based evidence in culture and knowledge is needed.	Concerted efforts are underway across MW to improve training opportunities and increase community resilience:				No additional resources required
		Provide training to communities on the Native Wellness Assessment Tool, developed in 2015/16, to support the ongoing knowledge and skill development of service providers in addressing addictions and mental health issues. This knowledge translation exercise will ensure the evidence base for cultural interventions gains broader application in community based services as well as residential treatment programs.	1. Provided training on Native Wellness Assessment Tool (provided to a minimum of 150 First Nations in 5 communities)	1. March 2017	1. Executive Director, PHPCD, FNIHB
		Update the 2003 resource "Promising Strategies. Aboriginal Youth: A Manual of Promising Suicide Prevention Strategies" to better reflect current knowledge related to suicide, its prevention and overall mental health needs, including new and emerging best practices. The resource will be available to communities and will include "wise practices" in First Nations community-based suicide prevention.	2. Draft of "Wise Practices" resource	2. March 2018	2. Executive Director, PHPCD, FNIHB
		Complete the development of First Nations adaptation of Mental Health First Aid (MHFA) training module for First Nations individuals, families and communities. The MW Division is funding the Mental Health Commission of Canada (MHCC) to produce a Mental Health First Aid First Nations curriculum, adapted for First Nations peoples, which recognizes their unique history and context. This training will help build community capacity to better manage potential or developing mental health issues in First Nations communities.	3. Completed a First Nations MHFA training module	3. March 2017	3. Executive Director, PHPCD, FNIHB
Continue to integrate First Nations and Inuit Mental Wellness programming at the national and regional levels and formalize mechanisms to ensure integration across the individual programs.	Management agrees with the recommendation and will determine the feasibility of merging several existing programs under one single Mental Wellness Program as a formal mechanism to ensure integration	Explore the possibility to amalgamate existing programs such as Brighter Futures and Building Healthy Communities, Mental Wellness Teams and the National Native Drug and Alcohol Abuse Program under one Mental Wellness Program.	1. Feasibility assessment for SMC review	1. September 2016	1. Executive Director, OPPH, FNIHB, Regional Executives, FNIHB	No additional resources required
			2. Engage partners to develop recommendations for EC	2. October 2016	2. Executive Director, OPPH, FNIHB, Regional Executives, FNIHB
			3. Present recommendations to EC	3. December 2016	3. Executive Director, OPPH, FNIHB, Regional Executives, FNIHB
Develop a comprehensive Performance Measurement Strategy to guide the collection and usage of performance measurement data.	Management agrees with the recommendation and is working towards improving the availability of and access to high quality data to support evidence-based decision making in policy, expenditure management and program improvements. Performance measurement is recognized as a necessary supportive element to support implementation of the FNMWCF. Work on a comprehensive PMS is underway in collaboration with the Office of Audit and Evaluation. The MW PMS format is consistent with other PMS in the Branch. The strategy will identify sources of performance data, who is responsible for collecting and reporting data, and describe the frequency and timing of the data collections.	SPPI in partnership with MW will refine the existing performance measurement strategy.	1. Completed Performance Measurement Strategy	1. September 2016	1. ADM, FNIHB	No additional resources required

1. Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the Mental Wellness (MW) Programs, for the period from April 2010 to March 2015. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Evaluation (2009).

2. Program Description

2.1 Program Context

The First Nations and Inuit Health Branch (FNIHB) of Health Canada supports a range of mental wellness initiatives and services in First Nations and Inuit communities. These initiatives and services were previously identified as the Mental Health and Additions Cluster. From 2005-2006 to 2009-2010, the actual program expenditures for these programs was approximately $894 million. Since 2011, these initiatives and services have been described as the First Nations and Inuit MW Programs.

A First Nations Mental Wellness Continuum (FNMWC) Framework was developed on a collaborative basis and launched in 2015 by FNIHB, the Assembly of First Nations (AFN) and Indigenous mental health leaders from various First Nations non-governmental organizations. The Framework presents a shared vision for the future of First Nations MW programs and services and practical steps towards achieving that vision. It outlines opportunities to build on community strengths and control resources in order to improve existing MW programming for First Nations communities. The foundation of the framework is culture, emphasizing First Nations strengths and capacities to build community resilience and address MW issues and problems. The Framework attests that a full spectrum of culturally competent supports and services is necessary to address mental wellness needs in First Nations communities. The Framework outlines the following continuum of seven essential services as necessary to address First Nations health care needs:

Health Promotion, Prevention, Community Development and Education;
Early Identification and Intervention;
Crisis Response;
Coordination of Care and Care Planning;
Detox;
Trauma-Informed Treatment; and
Support and Aftercare.

The Framework also stipulates that First Nations communities require access to a range of integrated services, meaning that health care services should be managed and delivered in a manner to ensure that clients can receive a continuum of services from prevention to treatment and aftercare, according to their needs over time and across different levels of the health system. For the client, an integrated continuum means that their care is easy to navigate, that health workers are aware of their health as a whole, and that health workers from different systems communicate well. Health Canada is currently supporting Inuit Tapiriit Kanatami on the development of an Inuit mental wellness continuum framework.

An evaluation of the Mental Health and Addictions Cluster (2005-2006 to 2009-2010) was last completed in 2013, and its findings and recommendations are referenced in this evaluation of MW Programs.

2.2 Program Profile

The MW Programs fund community-based programs and services that aim to provide treatment, reduce risk factors, promote protective factors and improve health outcomes associated with the mental wellness of First Nations people and Inuit. The goal of the MW Programs is to provide First Nations and Inuit communities, families and individuals with culturally-relevant mental wellness services and supports, such as suicide prevention and substance abuse treatment, that are responsive to their needs. Guided by the FNMWC Framework, and priorities established by communities, the Programs seek to support a continuum of care that includes primary, secondary and tertiary prevention activities and knowledge development. The Programs are delivered across Canada through involvement of staff at the regional and community level, as well as in conjunction with First Nations and Inuit organizations and provincial partners. The activities and programming implemented through the MW Programs are expected to improve collaboration and networking at all levels, increase participation of First Nations people and Inuit in program activities, enhance the continuum of programs and services in the communities, increase awareness and practice of healthy behaviours, enhance community capacity and ownership of the programs, and improve the quality and accessibility of available services.

The FNIHB suite of MW Programs includes the following six programs^{Footnote i}, and activities of the Mental Wellness Teams:

The Brighter Futures (BF) program is designed to improve the quality of, and access to, culturally appropriate, holistic and community-directed mental health, child development, and injury prevention services at the community level. The program is delivered by First Nations and Inuit communities with funding provided through FNIHB contribution agreements. In 2009-2010, contribution agreements were signed by FNIHB with 299 communities to deliver the BF program. Total expenditures from 2010-2011 to 2014-15 were approximately $266 million.
The Building Healthy Communities (BHC) program assists First Nations and Inuit communities in developing community-based approaches to mental health crisis management and youth solvent abuse. The programming includes two major activity areas: Solvent Abuse, which assists communities to develop local programs aimed at preventing the abuse of solvents and the treatment of solvent abuse; and Mental Health Crisis Management, which focuses on enabling communities to respond to and heal from crises such as suicide. In 2009-2010, contribution agreements were signed by FNIHB with 299 First Nations and Inuit communities to deliver BHC programming. Total expenditures from 2010-2011 to 2014-15 were approximately $208 million.
The National Aboriginal Youth Suicide Prevention Strategy (NAYSPS) aims to increase protective factors (e.g., youth leadership) and decrease risk factors (e.g., loss of traditional culture) related to youth suicide in First Nations and Inuit communities. Program objectives include increasing community capacity to prevent and deal with youth suicide, enhancing community understanding of effective suicide prevention strategies, and supporting community efforts to reach youth. The NAYSPS supports activities related to primary prevention, including positive mental health promotion activities; secondary prevention or early intervention, which includes assistance for potentially suicidal individuals either prior to self-injury or during a suicidal crisis; and tertiary prevention, which includes efforts to improve and increase crisis response efforts to intervene more effectively in the prevention of suicides and post-crisis suicide clusters (i.e., when one suicide in the community contributes or triggers more suicides or suicide attempts within a short period of time in the same community or region). NAYSPS supports between 115 to 150 youth suicide prevention projects annually. These projects are selected based on input from First Nations and Inuit advisory groups and community members. The types of projects undertaken include suicide awareness raising events, leadership and life skills development courses, cultural activities that incorporate traditional practices, Elder mentorship opportunities for youth, and postvention activities such as grief and loss workshops. Contribution agreements are signed by FNIHB with First Nations and Inuit communities to deliver the NAYSPS. Total expenditures from 2010-2011 to 2014-15 were approximately $58 million.
The National Native Alcohol and Drug Abuse Program (NNADAP)is designed to assist First Nations and Inuit communities to operate initiatives to reduce alcohol, drug and substance abuse. The program is comprised of two major components: a community based component and a residential treatment component. Community-based NNADAP provides prevention, intervention, aftercare and follow-up services in First Nations and Inuit communities across Canada. In most communities, this component is delivered by community-based workers who are responsible for identifying community needs and delivering program activities. The residential treatment component consists of a national network of 43 NNADAP/NYSAP treatment centres. Through these national programs, First Nations people and Inuit have access to inpatient, outpatient, and day treatment services, as well as specialized services for people with unique service needs (e.g., programming for families, youth, solvent abusers, women, and people with concurrent disorders). Total expenditures from 2010-2011 to 2014-15 were approximately $407 million.
The National Youth Solvent Abuse Program (NYSAP) is a national residential in-patient treatment program. It is paired with community-level activities aimed at preventing youth solvent abuse. The goal of NYSAP is to improve the quality of life and the functional abilities of persons addicted to solvents by minimizing the effects and risks associated with solvent use. The program provides culturally appropriate, specialized treatment and recovery programs for First Nations and Inuit youth with chronic solvent abuse problems. Currently, there are eight NYSAP and one NNADAP-NYSAP treatment centres committed to empowering Aboriginal youth through the provision of a holistic healing program influenced by traditional Aboriginal values, beliefs and practices. Each centre follows a continuum of care approach that encompasses pre-treatment, treatment, and post-treatment care. Families of the youth are involved in the entire process. These centres are operated by First Nations and Inuit organizations and/or communities. Total expenditures from 2010-2011 to 2014-15 were approximately $96 million.
The Indian Residential Schools Resolution Health Support Program (IRS RHSP) was developed in accordance with the commitment made by the Government of Canada in the 2007 Indian Residential School Settlement Agreement to provide health support services for all former Indian residential school students. The program provides mental health and emotional supports to eligible former Indian Residential School students and their families before, during and after their participation in Settlement Agreement processes, including: Common Experience Payments, the Independent Assessment Process, Truth and Reconciliation Commission events and Commemoration activities. The objective of the program is to ensure that eligible former students of Indian Residential Schools (IRS) and their families have access to professional counselling, cultural support, resolution health support, and transportation assistance in order to address mental health issues related to their experience with Indian Residential Schools. The IRS RHSP program is administered by FNIHB regional offices who sign contribution agreements with First Nations and Inuit communities and service providers to deliver program activities. Transportation assistance is provided to allow professional counsellors and other service providers to travel to communities and visit clients. Total expenditures from 2010-2011 to 2014-15 were approximately $285 million.
Mental Wellness Teams (MWTs)are community-based, client-centred, multi-disciplinary teams that provide a variety of culturally safe mental health and addictions services and supports to First Nations and Inuit communities. The MWTs are owned, defined and driven by the community and include a mix of First Nations and Inuit traditional, cultural and mainstream clinical approaches to mental wellness services, which span the continuum of care from prevention to aftercare. The MWTs were initiated in 2007 on a pilot basis and were later expanded to all regions of Canada. Currently, there are 11 MWTs serving 72 First Nations and Inuit communities in Alberta, Saskatchewan, Manitoba, Ontario, Québec, Atlantic, and Northern Region. In Ontario, there are also Community Wellness Development Teams (CWDTs) that provide mental health and addictions expertise and support to First Nations communities. The main objective of the MWTs is to support existing efforts to address mental health and substance abuse by increasing access to mental health and addictions services that are comprehensive, client-centered, culturally safe, community based and fill gaps in the continuum of care. The members of MWTs can include qualified coordinators, Elders, cultural workers, psychiatrists, psychologists, social workers and therapists. The MWTs are delivered either by First Nations and Inuit communities or tribal councils with funding provided by contribution agreements with FNIHB.

2.3 Program Logic Model and Narrative

The primary objective of the MW Programs is to fund and support the development, implementation, monitoring and evaluation of community-based and culturally relevant programs and policies that promote mental wellness. The activities undertaken by the MW Programs to achieve this objective are:

Collaborating and developing partnerships with First Nations, Inuit, Federal, Provincial and Territorial authorities and organizations through the signing of agreements, implementation of joint projects, participation in various committees and the building of strategic alliances;
Delivering mental health programs and services for First Nations and Inuit individuals, families and communities by engaging First Nations and Inuit community leadership and Provincial/Territorial governments;
Building the capacity of community workers and health professionals at the community level by organizing and delivering culturally appropriate training, continuing education, professional development opportunities and information and knowledge materials;
Organizing ongoing data collection research and surveillance to support service delivery, policy and program development; and,
Developing and sharing policies, procedures, guidance and frameworks in partnership with stakeholders at community, regional and national levels.

In the immediate term, these activities are expected to result in:

Increased and improved collaboration and networking;
Increased continuum of programs and services in First Nations and Inuit communities;
Increased participation of First Nations and Inuit individuals, families and communities in the programs and services; and,
Increased awareness of healthy behaviours.

In the intermediate term, the activities of MW Programs are expected to result in:

Increased practice of healthy behaviours;
Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide and other mental health issues; and,
Improved access to quality, well-coordinated programs and services for First Nations and Inuit individuals, families and communities.

In the long term, FNIHB MW Programs are expected to contribute to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services. The connection between these activity areas and the expected outcomes is depicted in the logic model provided in Appendix 1.

2.4 Program Alignment and Resources

FNIHB MW Programs fall under Program Authority 3.1.1 First Nations and Inuit Health Promotion and Disease Prevention and 3.1.1.2 First Nations and Inuit Mental Wellness of the Health Canada Program Alignment Architecture (PAA).

During the five year period from 2010-2011 to 2014-2015, actual expenditures (excluding Employee Benefits Plan (EBP)) on MW Programs within scope for this evaluation were approximately $1.3 billion and are presented below (Table 1). Of this total, $407 million was spent on NNADAP (31%), $285 million on IRS RHSP (22%), $266 million on BF (20%), and $208 million on BHC (16%), $96 million on NYSAP (7%), and $58 million for the National Aboriginal Youth Suicide Prevention Strategy (4%).

Table 1 - Programs' Expenditures from 2010-2011 to 2014-2015 ($ millions)
Programs	2010-2011	2011-2012	2012-2013	2013-2014	2014-2015	Total
Mental Wellness Programs
Indian Residential School Resolutions Health Support Program	51.3	60.8	57.4	59.6	55.6	284.7
Brighter Futures	40.8	57.1	58.0	56.5	53.6	266.0
NNADAP - Community Based Program	35.3	48.5	52.8	52.2	47.7	236.5
NNADAP - Treatment Centres Program	30.0	36.0	37.4	35.1	31.6	170.1
Building Healthy Communities	34.7	44.7	45.6	43.2	39.5	207.7
National Youth Solvent Abuse Program	18.1	20.0	20.5	19.5	18.3	96.4
National Aboriginal Youth Suicide Prevention Strategy	11.9	12.8	11.7	10.7	10.4	57.5
Sub-total for MW Programs	222.1	279.9	283.4	276.8	256.7	1,318.9
Related Strategies/Funding Sources^{Footnote *}
National Anti-Drug Strategy	8.0	10.7	10.1	8.7	11.9	49.4
Labrador Innu Comprehensive Healing Strategy	4.5	4.6	4.4	4.7	4.6	22.8
Canada Drug Strategy	0.3	0.7	0.3	0.3	0.8	2.4
Mental Health & Addiction - Transfer mode	46.6	0	0	0	0	46.6
Mental Wellness - Public Health Nursing services & Community Health Representatives	0	0	0	0	17.5	17.5
Mental Wellness - Policy Development & Program Oversight	0	0	0	0	5.4	5.4
Sub-total for Related Programs/Strategies	59.4	16.0	14.8	13.7	40.2	144.1
Total	281.5	295.9	298.2	290.5	296.9	1,463.0
Source: Financial data provided by the Office of the Chief Financial Officer, Health Canada Table 1 Footnotes Table 1 footnote * These strategies/funding sources do not fall within scope for this evaluation. They are included in the table as they contribute funding to support some of the activities and programming delivered through the MW programs (e.g., Mental Wellness Teams). Return to table 1 footnote * referrer

3. Evaluation Description

3.1 Evaluation Scope, Approach and Design

The evaluation included all activities of the MW Programs during the period from April 1, 2010 to March 31, 2015.

The scope of the evaluation did not include activities undertaken in British Columbia, where program responsibilities have been transferred to the First Nations Health Authority under the British Columbia Tripartite Framework Agreement. It also did not include MW program activities undertaken in northern Canada (north of the 60th parallel), which are part of the block funding agreements reached in the Northern Wellness Approach and require that evaluations are undertaken by the respective territorial governments. Of note, however, is that the activities undertaken by the IRS RHSP, NNADAP treatment centres and MWTs in northern Canada are not included in the territorial block funding agreements, and therefore the perceptions of First Nations people and Inuit residing in northern Canada may have been captured in the current evaluation. Only a small sample size of Inuit participants may have been reached, therefore limiting the ability of the evaluation to report separately regarding program impact on the Inuit population.

As part of this evaluation, a review was conducted to compare current findings with results of evaluations of MW programs activities implemented in northern Canada (i.e., north of the 60th parallel). While it was appropriate to identify similarities and differences with respect to need, comparisons regarding performance were not made due to program variations and data availability.

The evaluation issues, design and data collection methods were aligned with the Treasury Board of Canada's Policy on Evaluation (2009). As shown in Appendix 3, the evaluation methodology considered five core issues under the two themes of relevance and performance. Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process. In accordance with the Treasury Board's Policy on Evaluation (2009), the evaluation design employed a non-experimental outcome-based evaluation approach based on the Evaluation Framework/Evaluation Matrix document. The evaluation relied on multiple lines of evidence including:

A document and literature review consisting of over 200 documents;
Forty key informant interviews with Health Canada national and regional staff (n=16), representatives of other government departments and First Nations/Inuit organizations (n=2), MWT coordinators (n=6) and community leads (n=16) who are employed by health centres, communities, tribal councils to undertake MW program funded projects and initiatives;
An online survey of 158 community health directors/staff and 11 community leaders; and
Site visits to 14 First Nations communities in six regions of Canada (i.e., Atlantic, Québec, Ontario, Manitoba, Saskatchewan, and Alberta) that included focus groups, consultations and surveys with community-based program staff and clients.

Culturally relevant research and evaluation approaches were employed including contacting community leaders to obtain their consent prior to approaching community-based staff members; providing respect for privacy and confidentiality to all participants; utilizing First Nations evaluation team members; and developing culturally appropriate data collection tools in conjunction with departmental program staff and representatives of the Assembly of First Nations (AFN) and Inuit Tapiriit Kanatami (ITK). In addition, the results of the focus group discussions and interviews were validated by participants to mitigate effects of possible respondent biases and increase reliability of the results.

As a component of this evaluation, a Social Return on Investment (SROI) analysis was piloted on a NNADAP adult treatment centre. SROI examines social, environmental and economic outcomes and represents them with monetary values. This enables a ratio of benefits to costs to be calculated.

Evaluation data were analyzed by triangulating information gathered from the different evaluation methodologies. The use of multiple lines of evidence and triangulation were employed to increase the reliability and validity of the evaluation findings and conclusions. A detailed description of the evaluation scope, approach and design is provided in Appendix 3.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications on the validity and reliability of the evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation, as well as the mitigation strategies that were employed to ensure that the evaluation findings can be used with confidence to guide program planning and decision making.

Table 2 - Limitations and Mitigation Strategies
Limitation	Impact	Mitigation Strategy
There is limited publicly available surveillance data on First Nations and Inuit health outcomes and health status (e.g., rates of suicide, addictions, substance abuse and other mental issues and problems).	While the available health outcome data were analyzed, the limited availability of the information did not allow for a comprehensive assessment of the trends and achievement of long term outcomes (improved health status).	To assess the achievements of the MW Programs with respect to achievement of outcomes, evaluation data from other lines of evidence (e.g., focus groups with community-based program staff and clients, surveys of community health staff and community leaders, and literature review) were triangulated.
Due to the nature of the programming, most performance data collected by the MW Programs is focused on program outputs, including descriptions of activities.	Much of the information collected through reporting templates was focused on the production of outputs and provided limited information about the achievement of program outcomes.	Focus groups, surveys and interviews were utilized to assess the extent to which program outcomes were achieved.
The use of key informant interviews and surveys with community health representatives (respondents who have a vested interest in the MW Programs) can create the potential for respondent biases.	Respondent biases may affect the reliability and validity of the findings.	Several measures were implemented to reduce the effect of respondent biases: (i) the purpose of the evaluation and confidentiality of responses were clearly communicated to respondents; (ii) the interviews were conducted by skilled interviewers; (iii) answers from each sample of respondents were cross-checked with those of other groups for consistency and validation; and (iv) the findings of the key informant interviews and surveys were validated with the results from other lines of evidence.
The sample of 14 communities included in site visits represents a small percentage of all First Nation and Inuit communities across Canada.	The data collected from the communities visited cannot be generalized to represent all First Nations and Inuit communities served by the program.	The sample of 14 communities was selected to include most regions of Canada covered under the evaluation (Atlantic, Québec, Ontario, Manitoba, Saskatchewan and Alberta) and included a wide range of backgrounds such as communities that are urban, rural, large, small, remote/isolated and easily accessible. The findings of the site visits were triangulated with results of the other lines of evidence to assess consistency.
The programs which conduct activities aimed at prevention had limited data on program impact.	The evaluation is limited in its ability to report on the outcome achievement of prevention work conducted as part of the MW Programs. In contrast, significant information was available on the treatment component of the MW Programs.	Focus groups, surveys and interviews were utilized to assess the extent to which program outcomes were achieved.
There is the potential for non-response error among the survey of community health representatives as well as communities that agreed to participate in site visits.	Given the self-selected nature of the survey, the characteristics of community health representatives who completed the survey may be different from those who did not. Also communities that have higher capacity were more likely to agree to participate in site visits.	Equal opportunity for participation was provided to all communities and survey participants. Multiple efforts were made to contact each community representative/participant and request their participation. The findings of the surveys and site visits were triangulated with results of the other lines of evidence to assess consistency.
There was a low response rate to web-based survey of community leaders and health directors/staff.	Given the low response rate, the data collected from the web-based survey may not be generalizable to represent all First Nations and Inuit communities served by the program.	The findings of the survey were triangulated with results of the other lines of evidence, in particular site visits, to assess consistency.

4. Findings

4.1 Relevance: Issue #1 - Continued Need for the Programs

There is a continued need for the MW Programs as First Nations people and Inuit consistently experience higher rates of mental health issues (e.g., psychological distress and mood disorder), suicide, illicit drug and alcohol use than the general population in Canada.

First Nations people and Inuit in Canada experience a disproportionately high prevalence of suicide, drug and alcohol addiction, substance use and other mental health issues compared to the overall Canadian population. For example, in the 2008-2010 First Nations Regional Health Survey (RHS), approximately one-half (51%) of all First Nations adults living on-reserve reported either moderate or high levels of psychological distress as compared to 34% of the general population. Among First Nations youth, approximately one-third of females and one-fifth of males reported feeling depressed, sad or blue for two or more weeks in a row during the previous 12 months.^{Footnote 1} According to data from the Aboriginal Peoples Survey, from 2007 to 2012, First Nations people (15 years and older) living off-reserve were twice as likely to report having been diagnosed with a mood disorder (13% of First Nations people aged 15-24, 17% aged 25-54 and 11% aged 55 and older^{Footnote 2}), compared to Inuit and non-Aboriginal people (7% of Inuit^{Footnote 3} and 6% of non-Aboriginal people). First Nations people living off-reserve and Inuit were more likely to rate their mental health status as poor or fair (14% of First Nations^{Footnote 4} and 6% of Inuit) as compared to 5% for the non-Aboriginal population.^{Footnote 5}

The links between substance use and mental health issues are complex. It is generally known that someone with a mental health issue is more likely to use substances to self-medicate, just as a person with a substance use issue is more likely to have or develop a mental health issue. Likewise, it is generally recognized that people with co-occurring mental health conditions have poorer treatment outcomes; are at a higher risk for harm; and have the most unmet needs.

The mental health issues of First Nations people and Inuit are exacerbated by higher rates of substance abuse and suicides than the general population in Canada as indicated below:

Suicide

Suicide rates among First Nations people and Inuit are much higher than those of the general population and youth suicide rates are increasing.
Both First Nations and Inuit youth and adults experience higher suicide rates than the Canadian population overall.

Between 2005-2007, the suicide rate among youth under the age of 19 living in areas with a high First Nations population was 11 times higher for males and 21 times higher for females as compared to youth living in areas with a low First Nations population.^{Footnote 6} Similar trends exist for Inuit populations. Between 2004 to 2008, suicide rates among youth under the age of 19 living in areas with a high Inuit population (Inuit Nunagat) was 35 times higher for males and 28 times higher for females compared to suicide rates among the Canadian population overall. In 2001, the national Inuit suicide rate was 135 per 100,000 population which is more than 11 times higher than the overall Canadian rate of 12 per 100,000 population.^{Footnote 7}

During the period from 2000-2002 to 2005-2007, the suicide rates for youth under the age of 19 living in areas with a high First Nations population increased by 13% for males and 43% for females, while suicide rates in low First Nations populated areas decreased by 24% for males and did not change for females (.09%).^{Footnote 8} In addition, from 1994-1998 to 2004-2008, the suicide rates for youth under the age of 19 years living in areas with a high Inuit population (Inuit Nunagat) increased by 32% for males while the suicide rate for the same age category of the general Canadian population decreased by 42%.^{Footnote 9}

First Nations people and Inuit are also more likely to experience suicidal thoughts and attempted suicides compared to the non-Aboriginal population in Canada. Among adults living on-reserve, the percentage of First Nations people who reported having attempted suicide at some point in their lifetime (13%) was greater than the proportion of adults in the general Canadian population (9%).^{Footnote 10} According to results of the Aboriginal Peoples Survey 2012, both First Nations (26%) and Inuit (24%) women living off-reserve were more likely than non-Aboriginal women to report suicidal thoughts (14%); and First Nations (21%) and Inuit (23%) men living off-reserve were more likely than non-Aboriginal men to report suicidal thoughts (11%).^{Footnote 11}

Substance Abuse

Although First Nations people and Inuit are less likely than the general population to consume alcohol, those who do are more likely to engage in heavy drinking. According to the 2008-2010 RHS, almost two-thirds (64%) of First Nations people who drink reported heavy drinking as compared to 18% of the Canadian population. Whereas the rates of heavy drinking^{Footnote ii} tend to decline after young adulthood among the general population, heavy drinking remained prevalent among First Nations adults in their 30s, 40s and 50s (FNIGC, 2012).^{Footnote 12} In 2012, 26% of Inuit aged 15 and older reported heavy drinking, as compared to 18% of the Canadian population.^{Footnote 13}
A significantly higher percentage of First Nations and Inuit adults and youth use illicit drugs than the general population. The 2011 Canadian Alcohol and Drug Use Monitoring Survey indicates that the two most commonly used illicit drugs among the Canadian population (aged 15 years and older) were cannabis and crack cocaine, with 9% and 0.9% of the general population, respectively, reporting past-year use.^{Footnote 14} In comparison, in the 2008-2010 RHS, 32% of First Nations adults reported cannabis use and 8% reported cocaine/crack use.^{Footnote 15} While there is limited up-to-date data on illicit drug use among the Inuit population, one study undertaken in 25 communities in Nunavut in 2007 and 2008 demonstrated that 62% of respondents over the age of 18 reported that they had experimented with a substance at least once in their lives in order to get high, 43% reported using recreational drugs such as marijuana or hashish in the previous 12 months, 4% reported using over the counter or prescription drugs to get high, and 5% reported using hard drugs such as cocaine and crystal methamphetamine.^{Footnote 16}
Data suggests that the misuse of prescription drugs, particularly opioids, occurs at disproportionately high levels in First Nations communities in Canada. According to the 2008-2010 First Nations RHS, 5% of First Nations people aged 18 and older living on-reserve or in northern First Nations communities reported past year use of illegal or prescription opioids (including morphine, methadone and codeine) without a prescription, and 6% reported non-prescribed use of sedatives or sleeping pills, including Valium, Serepax and Rohypnol. Similar trends are also seen in First Nations youth (aged 12-17), with 1.3% reporting using illegal or prescription opioids without a prescription during the previous 12 months and 2.2% reporting non-prescribed use of sedatives or sleeping pills.^{Footnote 17} Some First Nations communities have reported epidemics, with as many as 43% to 85% of the communities' population addicted to opiates (National Advisory Council on Prescription Drug Misuse.^{Footnote 18}

In comparison, in 2012, only approximately 0.5% of the Canadian adult population reported abusing opioid pain relievers, 0.6% reported abusing stimulants, and abuse of sedatives was not reportable due to the low numbers of users and small sample size.^{Footnote 19} Recent data from Health Canada's FNIHB Non-Insured Health Benefits Program show that 898 opioid prescriptions were dispensed per 1,000 First Nations individuals aged 15 and older in Ontario in 2007, with 119 prescriptions for oxycodone formulations alone.^{Footnote 20} Easy access to prescription drugs and issues associated with the surveillance and diversion of prescription drugs from pharmacies has contributed to the prescription drug abuse problem.

Many First Nations and Inuit communities face crises related to suicide, alcohol and drug use, communicable diseases and child welfare apprehensions. In addition many First Nations communities face crises related to prescription drug abuse. While there are no exact statistics on the number of crises occurring in First Nations and Inuit communities, a review of the literature indicates that the combined influence of historical and current traumas and stressors on some First Nations and Inuit individuals, families and communities are contributing factors to crises. The nature and severity of these crises vary depending upon the unique circumstances of each community. For instance, in some communities, it is manifested in an epidemic of suicides and addictions, whereas in others it may be significant rates of violence or widespread apprehension of children by child welfare services.

According to the literature, the disproportionate prevalence of substance abuse, suicides and other mental health problems among First Nations people and Inuit can be attributed, at least in part, to their historical experiences of trauma and stressors associated with colonization, cultural disruption, oppression, marginalization and intergenerational trauma.^{Footnote 21} Historically, suicide was a very rare occurrence amongst First Nations and Inuit. It was only after contact with Europeans and the subsequent effects of colonialism that suicide became prevalent.^{Footnote 22} Forced relocation and settlement, the restriction of hunting and gathering practices, and the transformation of social and family structures through the introduction of European and Christian norms and values and the advent of residential schools have all contributed to contemporary disparities in the social determinants of health and health outcomes of Canadian First Nations people and Inuit.^{Footnote 23} According to results of the 2008-2010 RHS, First Nations people who attended residential schools were less likely to report feeling mentally balanced (44%) and were more likely to report psychological distress than First Nations adults who did not attend residential school (50%). The residential schools did not just impact the students, but also, the next generations, the children and families of the survivors. The 2008-2010 RHS stated that First Nations youth whose mother or father had attended residential school were 1.4 times more likely to have thought about suicide as compared to youth whose mothers or fathers had not attended residential school.^{Footnote 24}

Other factors identified in the literature as contributing to the high prevalence of mental health and addictions issues among First Nations people and Inuit and which can be attributed, at least in part, to the above, include income and employment disparities, lower levels of education and literacy, discrimination and racism, poor prenatal care and drinking and smoking during pregnancy. These factors are linked to poor physical, emotional, and intellectual development among some First Nations and Inuit children.

Additional factors contributing to a high prevalence of mental health and addictions issues among First Nations people and Inuit include challenges with respect to access to health care services due to the remote location of many First Nations and Inuit communities, lack of transportation infrastructure, inadequate number of health professionals and language barriers. The communities served by MW Programs vary widely in terms of geography, culture, language and political structures. Program documents reveal that 35% of First Nations communities are geographically isolated (i.e., have limited or no road access); the majority (56%) are small communities with a population of fewer than 500; and the communities speak over 50 distinct languages. Due to factors such as geographic isolation and language barriers, a significant proportion of First Nations adults reported that, compared to other Canadians, they have less access to quality health care. According to results of the 2008-2010 RHS, 39% of First Nations adults reported having less access to health services than the general Canadian population while about half (49%) of all First Nations adults indicated that they had the same level of access to health services as the general Canadian population.^{Footnote 25}

The findings of this evaluation are aligned with results of the evaluation of MW programs conducted in northern Canada (north of the 60th parallel) covering the period from 2005 to 2010. Both evaluations demonstrated high rates of suicide, alcohol and drug abuse, and other mental wellness issues across the region, and found that the higher rates of MW issues experienced by First Nations people and Inuit in the north were affected by isolation and remote locations and historical trauma, including the experience of residential schools.^{Footnote 26}

4.2 Relevance: Issue #2 - Alignment with Government Priorities

4.2.1 Are the Mental Wellness Programs aligned with Federal government priorities and departmental strategic outcomes?

The objectives and activities of FNIHB's MW Programs, which include the promotion of positive mental wellness as well as alcohol and substance abuse treatment, are aligned with the priorities of the federal government and the strategic outcomes of the Department.

Alignment with Government Priorities

Investing in mental health services has been highlighted as a priority in recent federal budgets. Over the last five years, the federal government has made a number of commitments relevant to mental wellness, with a strong emphasis on First Nations and Inuit specific services.

Budget 2013 made an additional investment of $52 million over two years to strengthen health services for First Nations people and Inuit, of which $4 million was dedicated for mental health services to support four multi-disciplinary mental wellness teams.^{Footnote 27} Ongoing annual funding for these teams was later confirmed in Budget 2015. Budget 2014 allocated $44.9 million over five years to expand the focus of the National Anti-Drug Strategy to address prescription drug abuse in Canada, which included $13.5 million to enhance prevention and treatment services in First Nations communities.^{Footnote 28}

Canada's first mental health strategy Changing Directions, Changing Lives: The Mental Health Strategy for Canada, produced by the Mental Health Commission of Canada, proposes six strategic directions as a means to improve the mental health system to meet the needs of all Canadians. These recommendations focus on the promotion of mental health and wellbeing, the prevention of mental illness and suicide where possible, as well as supporting the treatment and recovery of individuals experiencing mental health challenges. The Strategy calls for collaboration across all levels of government in order to ensure access to supports/services and treatment across the lifespan. The fifth strategic direction specifically calls for a continuum of culturally relevant mental health services for Aboriginal people as a means to "contribute to truth, reconciliation, and healing from intergenerational trauma."

The MW Programs build on the federal government's apology for Indian Residential Schools and were part of the commitment made to provide mental and emotional support to individuals involved in the settlement process through the Indian Residential Schools Resolution Health Support Program.

Alignment with Health Canada Strategic Outcomes

The MW Programs are aligned with Health Canada's departmental priorities and strategic outcomes. In particular, the objectives of the MW Programs are aligned with Health Canada's stated departmental priority to "strengthen First Nations and Inuit health programming". The objectives of the MW Programs are also aligned with the following Health Canada's strategic outcomes: "First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status".

Health Canada's Report on Plans and Priorities 2014/2015 (as well as the previous 2013/2014 version of the report) indicates the federal government's commitment to:

Continue to improve service delivery models in remote and isolated First Nations communities to support modern, sustainable, high quality health care for First Nations communities;
Implement the First Nations Mental Wellness Continuum Framework and continue working with Inuit partners to develop a specific Inuit Mental Wellness Continuum Framework;
Continue the implementation of Honouring our Strengths Addictions Framework: A Renewed Framework to Address Substance Use Issues Among First Nations People in Canada including the development of tools to support uptake of a strengths-based systems approach to addressing substance use issues among First Nations people in Canada;
Continue to provide health support services so that eligible former Indian Residential School students and their families can safely address a broad spectrum of wellness issues related to the impacts of these schools; and
Continue to support First Nations and Inuit communities in addressing Aboriginal youth suicide prevention.

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

4.3.1 Are the Mental Wellness Programs aligned with federal roles and responsibilities?

The federal government has a role in the provision of health care for First Nations and Inuit populations.

In Canada, health care is a shared undertaking between the federal, provincial and territorial governments. Provinces and territories provide universal insured health services (physician and hospital services) to all residents, including First Nations, Inuit, and Métis.

Federal policies, such as the Indian Health Policy and the Indian Health Transfer Policy, outline the goals of the federal government with respect to Aboriginal health. The Indian Health Policy (1979) provides authority for the delivery of First Nations and Inuit health programs, of which mental wellness programming is a component. It also encourages the improvement of First Nations and Inuit health status, "through mechanisms generated and maintained by the communities themselves".

4.3.2 Do the Mental Wellness Programs duplicate, complement or overlap with other programs (e.g., federal, provincial, territorial)?

The MW Programs appear to complement, rather than duplicate, other programs which are designed to address mental health and addiction issues in First Nations and Inuit communities. The MW Programs are unique in terms of their broader scope, and focus on ensuring the services reflect First Nations and Inuit culture, traditions, and way of life.

There are other federal government programs that have components similar to those of the MW Programs. For example, Indigenous and Northern Affairs Canada (INAC) delivers five social support programs for Aboriginal peoples, all of which include mental wellness components. The document review indicated that some of these programs (e.g., Child and Family Services Program and the Family Violence Prevention Program) have similar objectives to that of the MW Programs (e.g., the women's shelters include mental health related components, such as crisis counselling for victims of violence), however the objectives of the MW Programs are broader and more comprehensive, whereas other programs focus on a specific area of well-being. Similar programming and services are also offered by provincial service providers, but a large proportion of First Nations people and Inuit experience difficulties in accessing those services due to jurisdictional, cultural and geographic barriers. A key aspect of the MW Programs, highlighted by key informants, is that MW Programs incorporate First Nations and Inuit culture, traditions and way of life into the programming to a greater extent than other programs.

Among key informants, approximately one half (54%) reported no overlap or duplication between the Mental Wellness Programs and the services provided by other federal or provincial/territorial programs, one quarter (25%) did not offer an opinion, and approximately one-fifth (21%) reported some overlap or duplication.

In addition, according to key informants, First Nations and Inuit communities face such significant MW problems, that resources from one program alone are not sufficient to address them. Consequently, they felt that activities and programming implemented by similar programs mostly complement rather than duplicate the activities of MW Programs. Nevertheless, some key informants stated the importance of and the need for greater collaboration and partnership at the national and regional levels among all stakeholders to ensure continuous coordination and alignment of MW programming with other similar programs and the services.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the Mental Wellness Programs made progress towards the achievement of their intended outcomes?

Immediate Outcome #1: Increased and improved collaboration and networking

Collaboration and networking activities have improved at all system levels: at the national level, a new approach to policy development and engagement was adopted to involve First Nations and Inuit stakeholders in the early stages of the process; at the regional level, co-management structures were created involving provincial and community partners in the decision making process; and at the community level, MWTs played a critical role in improving networking and coordination between service providers. However, the individual Mental Wellness Programs are still largely implemented in silos at the national and regional levels at Health Canada, with insufficient interaction among staff in different programs.

Collaboration and Networking at the National and Regional Level

The MW Programs have made progress in facilitating collaboration, partnerships and networking at the national level. In recent years, FNIHB has changed its approach towards policy development at the national level by recognizing the importance of involving First Nations and Inuit organizations and communities as equal partners at the early stages of policy and program development. The most significant achievement has been the new and innovative processes employed in developing Honouring Our Strength: A Renewed Framework to Address Substance Use Issues Among First Nations People in Canada and the First Nations Mental Wellness Continuum Framework. The development of both frameworks is perceived as a highly successful process involving all stakeholders (e.g., First Nations, mental health experts and funding agencies) and instrumental in fostering meaningful and collaborative policy development. The process of developing these documents involved strong and authentic partnerships with First Nations stakeholders, expert-driven advisory panels and a collaborative approach. Both frameworks were drafted to reflect the discussions and feedback received from stakeholders involved in the process. This level of collaborative policy development has received strong support from First Nations stakeholders.

At the national level, staff from the MW Programs collaborate to some extent with representatives of programs implemented by other federal government departments, particularly INAC. According to key informants, current coordination efforts between INAC and FNIHB are not systematic and depend on the initiative of individual regional and national staff members. However, the recent establishment of First Nations Mental Wellness Continuum Framework Implementation Team was noted as having been helpful in bringing various partners together at the national level and may support collaboration with INAC moving forward. In general, key informants consider the MW Programs to be somewhat successful in networking at the national level (an average rating of 3.6 out of 5 on a scale of 1 to 5 where 1 is not at all successful, 3 is somewhat successful and 5 is very successful).

At the regional level, the development of co-management structures in Alberta and the Atlantic regions has facilitated partnerships with provincial and community partners. The voting members of these co-management structures often include FNIHB and community leaders, and a dedicated Mental Wellness Committee appointed by the co-management structure oversees the mental wellness programs undertaken by all stakeholders. These Mental Wellness Committees have contributed to increased linkages and greater coordination of activities with all stakeholders, including provincial service providers.

The focus of partnerships and linkages and the extent to which they have been developed varies across regions based on regional circumstances, needs and delivery models. For example, it is more difficult to build linkages in the Atlantic region, due to the need to coordinate with four provincial governments. In Québec, delivery of FNIHB MW Programs is more complex and integrated with the provincial government due to the establishment of health boards that include representatives of First Nations and Inuit communities. The health boards deliver provincially funded MW programming in First Nations and Inuit communities as well as ensure alignment and communication with the MW programming funded by FNIHB. In Saskatchewan, the focus of the collaboration and partnership has been on developing and delivering joint projects and initiatives such the preparation of the provincial mental wellness strategic plan and the All Nations' Healing Hospital, which is built on First Nations land, managed by the province and supported by federal funds. In Alberta, the regional partnerships have focused on crisis planning and management and the FNIHB office works with Alberta Health to address crises in communities and leverage provincial support to plan and address crisis situations.

Despite improvements in recent years, there are variations at the national and regional level within FNIHB regarding the extent to which staff across programs collaborate and the activities of the various programs are integrated. A key factor that has constrained the level of collaboration and partnership is the lack of a formalized mechanism, such as national strategy/system for collaboration and the lack of integrated program delivery. Furthermore, according to program staff, a table or committee to bring all program stakeholders together to discuss and identify opportunities for collaboration does not exist, however, the First Nations Mental Wellness Continuum Framework Implementation Team offers some promise in this regard.

Collaboration and Networking at the Community Level

Over the period covered under the evaluation, the frequency and degree of collaboration and networking activities at the community level has improved through introduction of MWTs, increased use of referrals and case management, and, according to key informants, the use of more flexible funding models. During the First Nations community site visits, community staff members (n=45) noted that they consider the MW Programs as having been somewhat successful in improving collaboration and networking (average rating of 3.4^{Footnote iii}). Most collaboration and partnerships at the community level occur with regard to client referrals. About half (49%) of health directors/staff surveyed reported that the use of case management services for client referrals helped build linkages with other service providers both in and outside of their communities. Most community health directors and staff indicated that they use child and family service providers and other health care services in their communities as well as service providers outside of their communities to provide care for their clients. The most frequent linkages between community health staff and organizations outside the community are with provincial health authorities, service providers, educational organizations and the justice system.

Mental Wellness Teams, where they exist, have played an important role in improving collaboration and partnerships at the community level. The 2012 evaluation of the MWTs stated that all MWTs reviewed have managed to build better interactions, partnerships and coordination between the service providers in the communities, which improved the continuum of care for community representatives. The MWTs also played a significant role in improving networking and coordination between service providers (e.g., schools, justice system, clinics, health boards, band leaders, hospitals) and organizing cultural events, workshops and capacity building activities to bring community members together. In addition, most teams have leveraged additional resources and funding from other sources to increase community capacity (e.g., in New Brunswick the province has contributed psychiatric nursing services to the MWT).

The level of integration of the different MW programs varies considerably and impacts the extent of collaboration between these programs. The degree of integration and collaboration is also impacted by the community funding models, as key informants (i.e., community health staff members) note that integration and collaboration can be undertaken more easily in communities that are funded under more flexible funding models.. In addition, remoteness and isolation affects the extent to which communities are able to build collaborations and partnerships. The levels of interaction, referrals, linkages and partnerships are much greater in the communities that are located in proximity of large urban centers compared to those that are in remote and isolated areas.

According to one quarter of the key informants and the results of the community consultations, there is an inadequate level of communication between FNIHB and communities. Several community health staff members stated that they feel disconnected from FNIHB and are unaware of the developments in the field of mental wellness and the changes happening to MW Programs. Some key informants noted that they have also been limited in their ability to visit communities and participate in networking and capacity building activities. Community health staff requested regular communications and updates from FNIHB related to their work in areas such as changes in program structure and delivery; new programs and funding opportunities; best practices in delivering mental wellness programs; and national and/or regional benchmarks. In Alberta, the FNIHB regional office has recently developed a newsletter that provides regular updates to communities and shares news, information and best practices. Community staff members find this newsletter very useful in their work.

Immediate Outcome #2: Improved continuum of programs and services in First Nations and Inuit communities

MW Programs have made progress in improving the continuum of the programs and services in First Nations and Inuit communities through introduction of traditional healers and Elders to deliver mental wellness support, a greater focus on the coordination of services among federal, provincial and community service providers, and the introduction of new initiatives such as Mental Wellness Teams. Existing gaps in the continuum include lack of capacity to proactively identify and address early signs of mental health and addiction problems, plan for and address crises, deliver aftercare for those returning from treatment and provide access to detoxification and other specialized services.

The evaluation identified that progress has been made in the number of communities that offer MW programs and services. According to results of the Community-Based Reporting Template (CBRT) reports, in 2013-2014, 345 First Nations and Inuit communities reported delivering MW programming, which is 71% of the 485 communities included in the scope of the evaluation. This represents a 25% increase from 2011-2012, where 277 communities reported delivering MW programs.

Key informants stated that the MW Programs have made some progress in improving the continuum of programs and services available in the communities (average rating of 3.3^{Footnote iv} provided by Health Canada staff, average rating of 3.7 by Mental Wellness Team coordinators and average rating of 3.6 by community leads). Currently, the types of MW programming delivered in the communities includes traditional activities; sport, recreational and other activities; suicide prevention-related community awareness activities; life skills development programming (e.g., leadership, problem solving); training workshops in suicide monitoring and response and crisis intervention; presentations and workshops on substance addiction and mental health activities; cultural events; addictions recovery support groups; and school-based programs to support awareness of substance abuse and addictions. According to key informants, including First Nations community representatives, there has been an increased focus and support for promoting community culture and traditions. The communities have been able to recruit and use traditional healers and Elders, and organize cultural activities and events, which has increased the range and continuum of services available. The continuum of services has also been improved through greater focus on coordination of services among federal, provincial and community service providers, increased use of case management services, the introduction of new initiatives such as Mental Wellness Teams, and the development of a number of frameworks, tools and resources.

To assess if there are any gaps in existing services, the activities of the MW Programs have been compared to the continuum of seven essential services specified in the First Nations Mental Wellness Continuum (FNMWC) Framework as necessary to address the needs of First Nations. Because the Framework was launched in 2015, it is too early to expect alignment of MW Programs with the Framework; however, the exercise was useful to determine progress made in aligning the existing programming with elements and principles stipulated in the document and identify existing gaps. The key findings of the mapping are summarized as follows:

Health Promotion, Prevention, Community Development and Education

Of the seven essential services specified in the Framework, MW Programs have made the most significant achievements in the area of health promotion, prevention and education. Results of the community site visits demonstrated that most activities and programming delivered by the MW Programs have some focus on health promotion, prevention and education. In all 14 First Nations communities visited, the programs deliver a large number of mental wellness workshops, training sessions, information sessions, cultural events and ceremonies and distribute information materials. Results of the CBRT reports also demonstrated that most communities are engaged in promotion, prevention and education activities. For example, 88% of First Nations and Inuit communities who submitted a CBRT report in 2013-2014 described delivering suicide prevention-related community awareness activities, and 97% reported delivering wellness activities promoting mental health.

Crisis Response

Significant progress has been made in terms of crisis preparedness. Key informants and First Nations focus group participants in the communities visited as part of the evaluation reported that, over the period covered under the evaluation, significant progress has been made in the development of community crisis plans and the organization of community crisis response teams. Limited capacity in the communities to adequately address crises has increased focus on crisis preparedness and an increased number of communities are developing crisis response plans and crisis response teams. However, there continues to be a major gap in community capacity to effectively handle crises such as suicide clusters.

Coordination of Care and Care Planning

Improvements have been made towards better coordination of care and care planning. The introduction of the MWTs and Community Wellness Development Teams (CWDTs) in Ontario has played a critical role in improving coordination of care and planning at the community level. The CBRT reports show that most First Nations and Inuit communities engage in partnerships with regional health authorities, non-profit organizations, the police, educational organizations and provincial government departments delivering services to coordinate their services and plan care for their clients. However, the results of the focus groups in First Nations communities reveal that the planning and coordination of mental wellness services in the 14 communities visited remains ad hoc and mostly consists of referrals. Community health directors and staff suggest that the primary challenge associated with coordination of care and care planning is the lack of case management. According to the online survey of community health directors/staff and community leaders, the use of case management has increased during the period covered under the evaluation; however, over half of communities (51%) still do not have case management systems in place. This negatively impacts the ability of communities to coordinate services, leading to reduced quality of services and inadequate services for clients who fall between the cracks in the system.

Conversely, limited progress has been made in the following program areas stipulated in the Framework:

Early Identification and Intervention

Little progress has been made in terms of early identification and intervention of mental health and addiction issues. Although CBRT reports show that a large number of communities report providing screenings and basic assessments, the results of the site visits and key informants interviews suggest that communities largely lack capacity to proactively identify and address early signs of mental health and addiction problems. Screenings and assessments for signs of mental health issues and substance abuse most commonly occur only when community members request help, often after the situation has escalated into a crisis. These challenges are aggravated by complicated or disjointed referral processes.

Trauma-Informed Treatment

Trauma-informed treatment in the communities remains inadequate, primarily due to a lack of culturally appropriate specialized services. The IRS RHSP has improved the continuum of programs by focusing on trauma-informed treatment, which is one of the seven essential services specified in the FNMWC Framework as necessary to address the needs of First Nations communities. According to some key informants and focus group participants, prior to IRS RHSP, there was little understanding of the trauma that IRS caused in the lives of those affected and the survivors had very little support to go through the settlement process. The program has helped survivors to go through a healing process and provided support during the settlement process. The program has also brought to light some of the effects of the IRS system and allowed people to come forward and talk about their experiences and trauma. As noted by key informants and community stakeholders, the major gap in IRS RHSP services is related to the extent of the support made available through the program, which is not considered to be adequate to address the level of trauma that the survivors have experienced. The IRS RHSP is set to sunset at the completion of the IRS Settlement Agreement as determined by the Court and it is unclear if additional programming will be made available to address the specific needs of IRS survivors. The community leaders and health directors/staff surveyed indicated that there was ongoing need for the following types of services: traditional healers and Elders to help survivors heal from the legacy of residential schools (95% of respondents), emotional services provided by community-based Aboriginal support workers ( 87%); mental wellness promotion and support services (87%); crisis intervention such as suicide risk assessment and intervention (86%); trauma treatment such as sexual, physical and emotional abuse (86%); and addictions treatment and related resources (86%).

As most First Nations and Inuit communities do not have access to specialized services (e.g., therapists, psychologists, psychiatrists, registered social workers) for those with complex needs, clients are referred outside of the community, often to provincial service providers who may be perceived as not being culturally safe. While FNIHB has provided communities with some training to better understand trauma and provide services to trauma-affected clients, the training is not adequate to compensate for the lack of specialized services. For example, in 8 of the 14 First Nations communities visited, mental wellness staff members reported having received some trauma-related training including Red Road training for dealing with violence and addiction, Post-Traumatic Stress Disorder counseling, Seeking Safety treatment, and culturally relevant women and addictions focused therapy. However, several staff members stated that that the amount of training is not sufficient to make a significant improvement in the overall capacity of the MW Programs.

Detox

Limited progress has been made in terms of providing access to quality detox services. As medically-based detoxification services are the responsibility of the provincial health authorities, Health Canada does not provide funding for these services. Evidence from focus group discussions in the 14 First Nations communities visited indicates gaps in access to detox services including transportation barriers, long wait lists and a lack of culturally appropriate services, as most detox services provided by the provincial governments lack cultural safety. For example, in most regions, the wait list for a provincially run detox program is 6 to 8 weeks, during which time the clients receive little support and care.

Support and Aftercare

Limited progress has been made in providing aftercare services. Limited aftercare services are provided to clients returning from treatment, and no formal processes are in place for sharing client treatment records with NNADAP community health workers. When clients are discharged from a NNADAP treatment centre and return to their communities, they often return to the same pre-treatment environment, resulting in relapse. In most of the First Nations communities visited, NNADAP workers have limited capacity, guidance or resources to follow up with clients and provide support services. There is a need for better recognition of determinants of health in discharge planning and aftercare and greater collaboration with social and community services. Program data was not available regarding the number of treatment centres that provide aftercare support or the clients who accessed these services.

In summary, the MW Programs have made progress in improving the continuum of the programs and services in First Nations and Inuit communities. However, some gaps in the continuum do remain, including the lack of capacity to proactively identify and address early signs of mental health and addiction problems, plan for and address crises, deliver aftercare for those returning from treatment and provide access to detoxification and other specialized services.

The coverage and capacity of the MW Programs remains a serious issue. Many of the 485 communities located in the regions covered under this evaluation are not delivering a full range of mental wellness programming. While MWTs have demonstrated success in filling gaps, the services provided by the MWTs covered only 12% of the total number of First Nations and Inuit communities included in the scope of the evaluation. As a new initiative, the coverage of the MWT programming has been growing slowly and is yet to be expanded to most communities across Canada.

Immediate Outcome #3: Increased participation of First Nations and Inuit individuals, families and communities in programs and services

Although comparative data on program participation rates is not available, there is a perception amongst community health staff and community members that participation in mental wellness activities has increased over the past five years. An ongoing challenge is the ability to reach those considered to be most vulnerable.

The performance data available is not sufficient to determine the extent to which participation of the First Nations and Inuit individuals, families and communities in MW Programs activities and services has increased over the period covered by the evaluation. Nevertheless, a review of the program documents and files indicate that a large number of First Nations and Inuit individuals and communities are participating in the MW Programs:

A total of 34,981 First Nation and Inuit individuals received community-based supports, 30,502 received direct counselling, 28,918 received cultural supports, 25,529 individuals received brief interventions, 8,407 individuals received screening and basic assessments, 8,009 individuals received comprehensive assessments, and 4,128 received referrals to specialized supports in the 345 communities that reported delivering MW Programs in 2013-2014 (CBRT reports);
NAYSPS provided funding for 132 projects that involved 34,974 First Nations and Inuit individuals in 2012-2013;
A total of 4,827 survivors of the residential schools received 26,733 counselling sessions through IRS RHSP In 2009-2010;
NNADAP/NYSAP treatment centres served 3,736 clients, of which 71% completed the program in 2014-2015^{Footnote v};
The MWTs provided services to 57 communities with a total population of 52,393 in the regions covered under the evaluation (excluding British Columbia and Northern Canada).

The results of key informant interviews and First Nations community site visits suggest that there has been an increase in participation of community members in the activities and programming delivered by the MW Programs. One of the most frequent reasons provided by respondents for increased participation in MW Programs is the incorporation of First Nations and Inuit culture and traditions into mental wellness programming. In 12 of the 14 communities visited, key informants indicated that the integration of cultural practices into mental wellness programming has significantly increased participation and attendance and reduced the stigma associated with mental wellness services. Some community members are very reluctant to attend western style mental wellness services, which are often associated with stigma, but are more willing to participate in cultural events and activities, which focus on strengthening community ties, healthy behaviours and healing. The cultural activities have had a positive impact on program participants' sense of community belonging, particularly among youth, and have helped to develop stronger intergenerational relationships between youth and Elders.

Nevertheless, according to results of the key informant interviews and community site visits, the MW programs have challenges in reaching out to certain groups in the communities. Those suffering from addictions, substance abuse, suicidal thoughts and/or trauma fear being judged for their mental health issues and often fail to seek the services they need.

In the communities without MWTs, the services are mostly reactive in nature and when crises happen or community members need help, the services are then made available to handle the issues. Mental health and addiction workers do not always reach out (e.g., knock on their door) to offer help, citing security concerns as a reason. In contrast, in communities where MWTs operate, the services are more proactive and attempt to address issues before they escalate into a crisis. In most communities that have MWTs, team members approach the most vulnerable, engage them in program activities and reach out to them with other innovative methods (e.g., organizing cultural events and activities, using social media, participating in radio programs).

Immediate outcome #4: Increased awareness of healthy behaviours by First Nations and Inuit individuals

The Mental Wellness Programs have been successful in increasing awareness of healthy behaviours. Children and youth have benefited the most from increased awareness of First Nations and Inuit culture and traditions and the importance of a strong cultural identity and attendance at land-based activities.

Of the 144 community leaders and health directors/staff surveyed, 72% indicated that, as a result of involvement in mental wellness programming, community members learned new skills and knowledge on how to stay healthy and 67% indicated that community members have increased knowledge on how to keep their families healthy. As the programs are implemented in a mostly integrated manner at the community level, it is difficult to determine the contribution of individual MW programs to increased awareness of healthy behaviors.

Health Canada staff and MWT coordinators consider the MW Programs as successful in increasing awareness of healthy behaviours. When asked to rate the success of the MW Programs in increasing awareness of healthy behaviours among First Nations and Inuit individuals, representatives of Health Canada provided an average rating of 3.7^{Footnote vi} and MWT coordinators provided an average rating of 4.2. According to key informants, the most significant progress in increasing awareness of healthy behaviours was made with children, youth and young adults as they are the primary target of most MW programing at the community levels and demonstrate greater interest to learn about their culture and traditions and participate in program activities. Key informants stated that the greatest increase in awareness of healthy behaviours is linked to the importance of a positive cultural identity; attendance at cultural activities or spiritual practices; and participation in land-based activities.

4.4.2 To what extent have the intermediate outcomes been achieved?

Intermediate outcome #1: Increased practice of healthy behaviours by First Nations and Inuit individuals

The MW Programs have resulted in increased practice of healthy behaviors^{Footnote vii} (e.g., reduced substance abuse, increased self-esteem and confidence, improved sense of positive cultural identity, strengthened family and community relations, and increased help-seeking behavior) by First Nations and Inuit individuals.

During the First Nations community site visits, participants in MW Programs (a total of 110 participants representing all 14 communities visited across six regions) were asked to rate the impact of MW programming on their attitudes and behaviour. These program participants noted significant impacts in terms of strengthened relationships with their families (average rating of 4.2), increased knowledge and mindfulness of healthy behaviours (4.2), increased motivation to practice healthy behaviours (4.2), improved mental wellness (4.2), better ability to manage their own health (4.1), increased connection with other supportive people (4.1), improved self-esteem and self-confidence (4.1), increased willingness to seek and accept help from others (4.0), help on their healing journey (3.9) and strengthened connections with the community (3.9). Some examples of specific program impacts reported by clients of MW Programs during the community site visits are as follows:

Reduced alcohol abuse among youth in one community due to use of traditional healing in counselling and treatment;
Reduced addiction due to introduction of methadone treatment in the community and referrals to treatment facilities outside of the community;
Better ties between young people and Elders through participation in joint cultural activities such as First Nations and Inuit language classes; and
Increased healing from Indian Residential School trauma due to help from traditional healers.

The survey of community leaders and health directors/staff also found that there has been increased practice of key healthy behaviours by First Nations and Inuit individuals. According to survey participants, as a result of involvement in mental wellness programming, community members are better able to ask for help when needed (69% of respondents); to contribute to their communities (63%); to feel support from the community (57%); to face the challenges in their lives (55%); to have better relationships with the people around them (51%); to have improved their overall physical health (46%) and to have more control over their lives (44%). According to the survey respondents, increased education and awareness have reduced the stigma of mental wellness issues in communities and have contributed to healing in the communities.

Lastly, program staff agreed that mental wellness programming has been successful in improving the practice of healthy behaviours among representatives of the target group. The average rating of the extent to which MW programming has increased the practice of healthy behaviors was 3.5 by Health Canada representatives, 4.0 by MWT coordinators, and 3.5 by community leads. According to key informants, the most significant progress was made among children, youth and young adults. According to program staff, the MW Programs were seen as successful in contributing towards healthier relationships with family (78%), healing (76%), increased attendance at cultural activities or spiritual practices (78%), increased ability to manage one's own health (71%), increased self-esteem and self-confidence (71%), a more positive cultural identity (67%) and positive parenting and family communication (62%). In addition, key informants stated that, of the different MW Programs, NNADAP (including NYSAP), Brighter Futures and Building Healthy Communities had the most significant impact as these programs have the largest coverage of First Nations and Inuit communities across Canada and provide the flexibility needed by communities to use program funding to address the most pressing priorities.

While it is not possible to assess the contribution of all individual MW programs to the increased practice of healthy behaviors, the following paragraphs provide a summary of the information available.

NNADAP treatment centers have been successful in helping participants reduce substance use. Outcome studies have demonstrated that interventions implemented at NNADAP treatment centres have led to decreased quantity and frequency of use. In 2011, after completing the program, over 90% of clients who participated in the outcome study had since reduced their use of at least one substance. All participants indicated that they reduced the use of inhalants, 86% reduced the use of alcohol, 85% reduced the use of sedatives, 83% reduced cannabis and cocaine use, 80% reduced amphetamines, 78% reduced hallucinogens and 76% reduced the use of opioids. Overall, 71% of all survey participants who completed the program indicated that they use fewer substances compared to before treatment, while 15% indicated the amount of use is the same as before. Similarly, in 2012, 90% of NNADAP treatment center clients surveyed felt less like using substances compared to before treatment. For those clients who did use substances after treatment, the vast majority (95%) reported they used a smaller quantity than before treatment.
Overall, of the NNADAP Treatment Centre clients surveyed for the 2012 Outcome Study, 89.5% felt like using less, as compared with before treatment. When clients did feel like using, 59.3% never used after treatment and an additional 36% used less than before treatment. 94% of clients reduced their use of at least one substance (such as alcohol, cannabis, cocaine, inhalants, or opioids) since treatment. When clients were asked how long after their completed treatment their first use occurred: 60% responded they never used after treatment, 3.5% first used six months after treatment, 10.5% first used between four and six months after treatment, 10.5% first used less than three months after treatment 3.5% first used less than a week after treatment 1.2% responded that their first use was the same day they completed treatment. For those clients who did use after treatment, the vast majority (94.9%) reported they used a smaller quantity than before treatment.
Projects supported by NAYSPS have reported success in increasing suicide protective factors among community members such as a sense of identity and belonging and an increase in help seeking behaviour, self-esteem and confidence. During 2011-2012 and 2012-2013, NAYSPS supported a total of 272 projects in eight regions of Canada, of which 187 projects provided some outcome data. Approximately 48% of these projects reported an increased sense of identity and community belonging among project participants; 11% reported a decrease in the number of suicide related incidents, and 6% reported impacts in terms of increased help seeking behaviour among young people in the communities. Case studies of nine NAYSPS projects conducted in 2011-2012 demonstrated the positive effects of the programming in helping youth to develop suicide protective factors. In particular, youth were better able to build relationships and support networks; had increased levels of confidence, optimism and hope; could identify and increase protective factors and reduce surrounding risk factors; could understand and participate in culture; were more open and willing to talk about suicide; and engaged more in their community. Some projects led to improved school performance, and overall increases in pride, discipline and confidence.
The IRS RHSP has significantly contributed to the increased practice of healthy behaviours by First Nations and Inuit individuals across all regions and communities served by the program. For example, almost all (93%) of community leaders and health directors/staff surveyed indicated that the program supported the community in its efforts to heal from the legacy of residential schools, 84% indicated that, as a result of the IRS RHSP, community members speak more openly about the legacy of the schools; 78% indicated that community members seek help from counselors, community workers and Elders; 68% reported an increase in attendance at community cultural events; and 50% reported an increase in the number of community members who seek help to address their alcohol or drug use.

Intermediate outcome #2: Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide and other mental health issues

While the MW Programs have increased community ownership and capacity to address mental health issues, insufficient capacity remains one of the most pressing issues in many communities.

A review of similar programs in other countries demonstrates that strong community capacity and ownership of the programs is often linked to better mental wellness outcomes. Communities that have more control over programming decisions are usually better able to build community resilience and address mental wellness problems and issues, as they have the ability to tailor programming to the specific needs of their community.

In an effort to increase community capacity to combat mental health and addiction issues, the MW Programs have delivered a large number of training sessions to increase the knowledge and skills of staff members working in the communities. For example, in 2013-2014 and 2014-2015, the MW Programs delivered a total 139 workshops and training sessions to 5,537 program staff and community workers in seven regions of Canada. Of the 139 sessions delivered, 81% taught staff about interventions to address various mental health issues and crisis, 53% focused on post intervention follow up, 47% focused on teaching skills and building capacities to prevent mental health issues and crises, 14% focused on helping MW program workers develop coping skills and resilience, and 13% focused on teaching traditional approaches to mental wellness. Staff members from all programs attended training. More specifically, 78% of capacity building sessions were attended by NNADAP staff, 48% by NAYSPS workers, 38% by BF staff, 38% by BHC workers and 6% by IRS RHSP staff. In addition to training of staff, some key informants stated that the support workers recruited and trained by IRS RHSP (e.g., social workers, psychologists) in the communities have become a good resource to communities and improve community capacity in the long term. Other key informants noted that introduction of MWTs in some regions has increased community ownership and capacity to combat mental health issues. Similar findings were reported in the 2012 evaluation of the MWTs, which stated that the ability of MWTs to create cultural safety for participants by integrating First Nations and Inuit culture and traditions with western approaches to mental wellness (e.g., Elders working with psychologists and mental health workers) has contributed to increased ownership among community members. MWTs are involved in building capacities of the communities through delivering training and workshops (e.g., one team trained over 130 service delivery people in four communities on grief and trauma and 25 community program staff on addictions and aftercare). MWTs also facilitated access to resources and tools by local service delivery staff, helped communities access additional funding and provided direct mental wellness services (e.g., traditional healing, therapy) to community members. The 2012 evaluation stated that compared to First Nations and Inuit communities without MWTs, communities with MWTs have a much stronger capacity to deal with mental wellness issues, handle crises and contribute to stronger resilience and mental wellness of community members.

In general, the evaluation found that MW Programs have increased First Nations and Inuit community capacity to combat substance abuse, suicide and other mental health issues. As an illustration, the MW Programs have had some success in increasing First Nations and Inuit community ownership and capacity (average rating of 3.9^{Footnote viii} by Health Canada representatives and 3.4 by MWT coordinators). Key informants stated that the approach recently employed by Health Canada to gradually transfer more control of health programming to First Nations and Inuit communities has resulted in an increasing number of communities demonstrating leadership and taking ownership of their mental wellness programming. Feedback obtained during community consultations also confirms that the MW Programs have made progress in increasing community capacity to deliver mental wellness services and programming. During the community site visits, community staff members reported that the MW Programs were somewhat successful in building the capacity of community workers and health professionals (average rating of 3.8). Over one half (59%) of the community health directors/staff surveyed confirmed that the capacity of communities to manage, enhance and deliver community-based mental wellness services has increased. According to these respondents, community capacity was strengthened due to increased education and awareness activities delivered by the programs (31% of respondents), improved integration and coordination of the programs (25%), increased efforts to deliver capacity building activities for community health delivery staff (19%), an increase in the amount and types of programming and services offered to community representatives (17%) and an increased focus on culturally appropriate services (12%). According to survey participants, as a result of involvement in mental wellness programming, the communities are better able to deal with substance abuse, suicide, the legacy of Indian Residential Schools and other mental health issues. In addition, community members feel support from their community.

While improvements have been made, key informants stated that the efforts to raise capacities in the communities are not sufficient to meet the need for such services. As an illustration, in 2013-2014, 30 communities in the Ontario region received funding to address prescription drug abuse problems, raising their capacity to provide some detox treatment and aftercare. However, these 30 communities accounted for less than one quarter of the 133 First Nations communities in Ontario. According to the focus groups in First Nations communities and key informant interviews, insufficient community capacity is one of the most pressing issues in many First Nations and Inuit communities. Due to short funding cycles, low wages, and resource challenges, communities are not able to hire and retain qualified and experienced staff members and some communities end up hiring unqualified staff. Low salaries, isolated and remote locations, difficult and stressful working conditions and many other structural factors contribute to high staff turnover among community staff members (staff turnover was the most significant program issue in 8 of the 14 communities included in site visits). Respondents indicated that identification of training needs and the development and implementation of capacity building activities for community staff are inconsistent, ad-hoc and, in many regions, reliant on activities delivered by partners.

Intermediate outcome #3: Improved access to quality programs and services for First Nations and Inuit individuals, families and communities

While the MW Programs have improved access to quality programs and services due to factors such as the integration of culture into mental health and addictions programming, there exist a number of constraints affecting the quality of MW programs and services in First Nations and Inuit communities.

Key informants considered the MW Programs as somewhat successful in improving access to quality programs and services for First Nations and Inuit individuals, families and communities (average rating of 3.3 by Health Canada staff, 3.6 by MWT coordinators, and 3.9 by community leads). The following paragraphs summarize the evaluation findings regarding how MW Programs have improved access to quality mental health programs and services:

Integration of community culture and traditions in the delivery of MW Programs has contributed to increased quality and appropriateness of the services for communities. To promote culture and traditions, the MW Programs use the services of traditional healers and Elders. In addition, regular community cultural events and activities are undertaken such as moon ceremonies, sweats, rite of passage ceremonies for youth and adults, craft and basket making workshops, drumming workshops, cedar baths, discussions around creation stories, teachings about traditional medicines, drumming and singing groups and language classes. The integration of cultural practices into mental wellness programming has significantly removed stigma from mental wellness services in most of the communities visited. Program participants indicated they feel safe and empowered when attending cultural and spiritual activities and heal faster. According to both program participants and staff members that participated in focus group discussions, when community members learn about their culture and understand their background, they build self-identity and self-esteem as well as an increase in resilience. The impact is greatest on youth who demonstrate a growing interest in their culture and traditions.
Improved quality of care for Indian Residential School survivors. According to some key informants, the IRS RHSP program has played a major role in addressing the root causes (intergenerational trauma) of many mental wellness issues and challenges faced byfamilies of survivors.The results of client satisfaction surveys conducted by the IRS RHSP in 2014 demonstrate that most clients of the IRS RHSP services are satisfied with the quality of the care. As indicated in the following table, they were satisfied with the extent to which their cultural values and beliefs, as well as privacy, were respected. In addition, they felt they were offered the services in the language of their choice, felt safe to talk about sensitive issues and were provided with adequate transportation.

Table 3: Client Satisfaction with IRS RHSP Services
Areas of Satisfaction	Cultural Support Providers	Resolution Health Support Workers	Counsellors
Felt cultural values and beliefs were respected	97%	98%	96%
Felt privacy was respected	95%	99%	98%
Meeting was in the language participant wanted	91%	95%	93%
Felt safe to talk about sensitive issues	86%	90%	91%
Meeting was easy to get to and from	89%	94%	87%
Will use the services in future	66%	57%	64%
(n=306) Source: IRS Client Survey Summary

Improved quality of care for those suffering from addiction and substance abuse. FNIHB has increased emphasis on the accreditation of NNADAP treatment centres and health centers across Canada and the certification of staff members. Accreditation is one of the most effective ways for health and social service organizations to improve the quality of their services, and to demonstrate that they meet or exceed national standards of excellence. A review of program documents indicates that most NNADAP/NYSAP treatment centres have been accredited and a majority of the staff members working in the NNADAP/NYSAP treatment centres are certified. In 2014-2015, of the 44 NNADAP/NYSAP treatment centres operating in the regions included in the evaluation, 84% (37) were accredited. In 2014-2015, NNADAP/NYSAP treatment centres employed a total of 278 full-time counsellors of which 79% were certified. In 2013-2014, a total of 84 community health centres went through an accreditation process and 52 (62%) were successful in receiving their accreditations.
Increased partnerships with provincial service providers have enabled MW Programs to provide better specialized services for communities. For example, in some regions where co-management structures have been established, the increased level of interaction between the communities and provincial service providers has helped communities to recruit visiting specialized care providers (e.g., therapists, psychologists, psychiatrists, registered social workers).
Introduction of MWTs has increased the quality of care and services offered by MW Programs. The teams display teamwork, ensure coordination of care and increase the overall capacity of the communities to provide better services. An example of improved quality of the services provided by MWTs was the introduction of an emergency phone number in one community. Using the number, community members were able to request help when the health centre was closed. The 2012 evaluation of MWTs stated that MWTs have been able to improve access to specialized services through inclusion of specialists in the team (most MWTs include clinical social workers, psychiatrists, psychologists, therapists, Elders and traditional healers).
Increased focus on crisis planning and preparedness has improved the quality of services that communities receive during a crisis. An increasing number of communities are developing a crisis plan, creating crisis response teams and organizing and delivering crisis training.

Despite improvements in the quality of the mental wellness services provided in communities, there still exist many gaps and challenges with the services available. The constraints affecting the quality of MW programming and services in the communities most frequently mentioned by key informants were funding limitations (44% of respondents), isolation and remoteness, (44%), issues related to transportation (28%), lack of capacity in the communities (22%) and limited flexibility over funding decisions (22%).

The focus group discussions and community consultations undertaken during the site visits with 14 First Nations communities also revealed that confidentiality was a concern. Community members often stated that they do not trust that their problems and issues will be kept confidential by community program staff members. In 8 of the 14 First Nations communities, focus group participants stated that program staff members lacked the capacity to ensure adequate confidentiality for participants and build mutual trust with clients. In these communities, some focus group participants indicated that they prefer to receive counselling and therapy from professionals outside of their communities because they are more trustworthy and provide better confidentiality.

Another quality constraint mentioned by focus group participants was the lack of service delivery measures to regularly assess the quality of the services provided by specialized service providers (e.g., therapists, psychologists, psychiatrists, registered social workers). For example, the lack of service delivery measures has resulted in problems in delisting incompetent service providers for residential school survivors in some regions.

Some key informants indicated challenges associated with integrating culture and traditions into mental wellness programming such as the cost and availability of the resources (e.g., traditional healers). Cultural activities are expensive and require additional preparation, materials and costs (e.g., wages for traditional healers, travelling expenses). Some other constraints mentioned by key informants and focus group participants were difficulties in accessing specialized services because they are not available in most First Nations and Inuit communities. In addition, there are problems in accessing specialized services outside the community due to long waiting lists, lack of transportation and cultural safety issues associated with provincial service providers.

Notwithstanding the success of the MWTs, this initiative was initiated in 2007 on a pilot basis and was later expanded to all regions in Canada; however, most First Nations and Inuit communities in Canada do not have access to their support and services. MWTs face administrative challenges at all levels, and have also been submit to community politics which has at times reduced the effectiveness of their activities. Some MWT coordinators stated that the three-year funding commitment for MWTs is too short for some of the long-term interventions that the teams are trying to implement. In addition, some teams are experiencing staff turnover due to lack of job security. MWT coordinators noted that lack of community capacity is a serious challenge for their activities. Often new initiatives undertaken by MWTs cannot be successfully implemented because the health centre in the community is understaffed and community staff members are overwhelmed with existing work.

4.4.3 To what extent has the longer term outcome been achieved?

Longer term outcome: Contribution to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services.

There is limited performance data to assess the extent to which MW Programs have contributed to the improved health status of First Nations and Inuit individuals, families and communities.

Section 4.4.2 demonstrated that the MW Programs have resulted in increased practice of healthy behaviors. Because health status is linked to healthy behaviors, this provides an early indication that the MW Programs may have contributed to the improved health status of some First Nations and Inuit individuals. However, there is limited performance data to accurately assess the extent to which MW Programs have contributed to improved health status and it is only available for some MW programs.

During the First Nations community site visits, MW program participants were asked to rate the extent to which MW programming helped to improve their overall mental wellness. The average rating of these program participants was 4.2^{Footnote ix}. The NNADAP outcome studies also demonstrate that in 2011, 85% of the NNADAP Treatment Centre clients who participated in the study indicated that their overall physical health has improved as a result of participating in the program. IRS RHSP clients also reported a range of improvements in their health and wellbeing. According to the IRS RHSP client survey, 95% (n=306) of clients felt the program services contributed to their overall wellness; 85% felt the services contributed to improved wellness of their families; and 83% felt the services contributed to the wellness of their communities.

4.4.4 Are the Continuum Framework elements driving change at the community level?

The First Nations Mental Wellness Continuum Framework has demonstrated preliminary progress in identifying programming gaps and integrating culture and traditions into mental wellness programming.

Given its recent launch (2015), it is too early to assess the impact of the First Nations MWCF in driving change at the community level. However, there are some promising signs that the Framework has already had an impact. Key informants stated that the Framework had some success in driving change at the community level (average rating of 4.2 by MWT coordinators and 3.2 by Health Canada staff) and that it has highlighted the importance of integrating cultural knowledge in mental wellness programming. The continuum of care described in the Framework is used by many communities to identify gaps in their own programming and determine areas for improvement. All MWT team coordinators noted that they are implementing the elements and principles highlighted in the Framework in communities. However, many key informants indicated that the Framework is new and needs time and additional resources to have a substantial impact at the community level.

According to key informants and case study participants, most communities across Canada are aware of the Framework, however staff members noted that significant work and resources are required to implement the principles and approaches described in the Framework at the community level. Similarly, most (78%) of the community leaders and health directors/staff surveyed stated that the MW Programs have helped connect community members to traditional/cultural supports and the traditional/cultural supports provided were helpful to community members. However, almost all community leaders and health directors/staff (91%) noted that there was a need for additional traditional/cultural supports to improve mental wellness programming. According to survey participants, the integration of cultural practices in the design and delivery of the MW Programs is the key factor to their success but there exist resource challenges to enhance the cultural components of MW Programs.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

4.5.1 Has the Mental Wellness program undertaken its activities in the most economical and efficient manner?

The MW Programs have undertaken a number of resource maximization measures to enhance their economy and efficiency. However, there are several factors that have constrained the efficiency and economy of the MW Programs, including resource challenges given that funding amounts have remained largely unchanged over many years, short term funding cycles, high turnover of program staff in the communities, lack of integration among MW Programs, insufficient funding flexibility in community programming and insufficient data to assess the performance of MW Programs.

The Treasury Board of Canada's Policy on Evaluation (2009) and guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.

The financial systems and program budgeting structure that the MW Programs have had in place during the years covered by this evaluation did not allow for a comprehensive operational assessment of economy and efficiency. While summary financial information on the MW Programs was reviewed, data on actual and planned expenditures regarding the programs operating at the community level were not available and financial information contained in contribution agreements were not provided at an individual program level. The analysis of efficiency and economy was therefore focused largely on 'allocative efficiency' using a utility based approach. The evaluation relied on a mixed approach (qualitative and quantitative data) to arrive at findings and conclusions related to the links between program outcomes and resources; relationships between program design and delivery methods and the achievement of outcomes; factors that contributed to the efficiency and economy of the MW Programs; and alternatives to program delivery.

The MW Programs have undertaken a number of resource utilization measures to achieve the intended outcomes. As indicated in Table 4, total expenditures of the MW Programs from 2010-2011 to 2014-2015 averaged $293.8 million annually. Of the total of $1.5 billion spent on all MW Programs (including related strategies/funding sources) over the last five years, 91% was spent on grants and contributions, 3% on salaries and 6% on other management and administrative. Consequently, the majority of MW Program funding was spent directly on the provision of mental wellness programs and only 9% was spent on management and administration expenses.

Table 4: Summary of MW Programs Expenditures from 2010-2011 to 2014-2015
Fiscal Year	Actual Spending
Fiscal Year	FTE	Salary^{Table 1 footnote *}	EBP @ 20%	Total salary	G&C	O&M	Capital	Total
2010-2011	N/A	6,543,886	1,308,777	7,852,663	256,731,822	18,119,770	-	282,704,255
2011-2012	N/A	7,219,389	1,443,878	8,663,267	266,442,749	21,394,551	-	296,500,567
2012-2013	N/A	6,965,512	1,393,102	8,358,614	272,387,201	18,574,216	-	299,320,031
2013-2014	N/A	6,436,494	1,287,299	7,723,793	267,566,606	16,448,394	-	291,738,793
2014-2015	N/A	10,492,512	2,098,502	12,591,014	271,900,049	14,367,374	20,222	298,878,659
TOTAL	N/A	37,657,793	7,531,559	45,189,352	1,335,028,427	88,904,305	20,222	1,469,142,306
Source: Financial data provided by the Office of the Chief Financial Officer, Health Canada Table 4 Foot notes Table 4 footnote * Note that salary figures include uncontrollable salaries. Return to table 1 footnote * referrer

In some regions, FNIHB staff members undertook specific initiatives to work with communities to improve the utilization of available resources. For example, in the Atlantic region, staff assessed the funding utilization levels in communities and identified ways to increase efficiencies. Community program staff reported that they use creative methods to leverage resources from other sources to organize events and deliver program activities. Some examples of the methods used by communities to increase the efficiency of the resources include combining resources from different program areas (e.g., representatives of different programs working together); acquiring funds from provincial governments to deliver complementary services or programming; using alternative methods for transporting patients for treatment (e.g., community members); engaging local Elders to deliver traditional ceremonies and spiritual practices for minimum payment or for free; and using holistic delivery approaches to program delivery to maximize results.

A review of one NNADAP treatment center demonstrates the social return on investment of programming invested for the treatment of addictions and substance use. The study results indicated that for every $1 invested in year 1, $3.85 of social value will be created over a 3 year period in terms of reduced social and protective services and health care costs. This value is specific to the programming and outcomes for the treatment centre that participated in the pilot study. It should not be seen as representative for all treatment centres or the broader Mental Wellness Program. For additional details on the calculations and analysis please see Appendix 3 - Evaluation Methodology.

The evaluation found that there are several factors that have constrained the efficiency and economy of the MW Programs, including:

Fixed funding amounts. Key informants and community stakeholders indicated that resource challenges, due to funding amounts remaining largely unchanged over many years, have been a critical constraint for the success and sustainability of the programming and the services at the community level.
Funding provided for a short period of time. Most funding programs/streams provided by the MW Programs are short-term and expire in a few years.^{Footnote ix} The short funding cycles and lack of consistent programming creates problems for community health centres because it does not allow for long term planning. In addition, it is difficult to build partnerships and leverage resources when programs have a short funding cycle.
Low wages and high staff turnover in the communities. Due to resource challenges, communities are not able to pay competitive salaries which make it difficult to attract qualified staff and some communities end up hiring unqualified staff. When communities spend resources to train staff, they often leave for higher paid positions with provincial service providers. Restricted timelines and short funding cycles also contribute to high staff turnover due to lack of job security. Some other factors that contribute to high turnover are the trauma, crises and other stressful situations that staff members deal with on a regular basis as well as the isolated location of work. Some staff members reported that they are overwhelmed with the amount of work and do not get an adequate level of supervision and support. High staff turnover reduces the efficiency and quality of the mental wellness services because less experienced staff are not as efficient (due to the learning curve associated with a new position) and an excessive amount of time and cost is required to hire and train new staff.
Lack of integration among MW Programs. The management and reporting of the individual MW Programs is separate. This has traditionally resulted in individual MW Programs being implemented in silos with little integration at the national and regional levels. Changes were introduced several years ago that promoted more interaction and better coordination of MW program activities at all levels. However, at the national level, the MW Programs are still mostly delivered in silos, as the reporting mechanisms and administrative requirements for each program are very different. Some linkages happen between program coordinators and staff who share information, identify commonalities between the programs and work together towards common goals. Despite these efforts, 42% of the key informants stated that there is a lack of sufficient systems to ensure integrated service delivery and that the level of integration, cooperation and information sharing is inadequate.

At the community level, the MW Programs are implemented in a more integrated manner, which involves conducting of joint initiatives, sharing of staff and co-locating of workplaces. One quarter of the community health directors and community leaders surveyed stated that the MW Programs are well integrated, 57% of community health directors/leaders indicated the MW Programs are somewhat integrated while 18% reported that the programs are not integrated at all. The most frequent reasons provided by key informants who stated that the MW Programs are only somewhat integrated is because some programs are implemented in silos (e.g., communities have several contribution agreements signed with FNIHB) which increases the level of administrative work required to manage the programs. According to key informants, greater connections, linkages and partnership within MW Programs as well as with other federal and provincial partners would maximize the use of resources and minimize administrative requirements. Key informants stated that increased integration across MW Programs at all levels - national, regional and community - would improve efficiency through better coordination of activities and result in an improved continuum of care. Among the MW Programs, the most interaction and linkages happen between the Brighter Futures and Building Healthy Communities at the community level. IRS RHSP is unique and has less interaction and linkages with other programs, partially because a portion of the IRS RHSP is delivered by regional staff.
Insufficient funding flexibility. Funding for specific programs is 'set" in approximately 23% of communities which results in limited opportunities to shift resources to address the most pressing community priorities. The advantage of more flexible funding models (e.g., 'flexible' and 'block' models) as compared to 'set" funding models is that they can take into account the fact that the communities are very diverse and their needs for mental wellness support vary greatly and change quickly. In addition, more flexible funding models enable greater integration of programming at the community level because it is much easier to conduct joint initiatives and share resources.

Performance Measurement Data

A review of the program documents and files demonstrated that the MW Programs do not have a comprehensive Performance Measurement Strategy (PMS). Instead, a document called the Consolidated Mental Wellness Indicators is used to monitor program activities and measure performance. The Consolidated Mental Wellness Indicators document was developed by the FNIHB Performance Measurement Unit (PMU) in collaboration with the MW Programs in response to a recommendation from a previous evaluation. The document lists performance indicators that the programs use to monitor activities, outputs and outcomes. However, the document does not identify when/how the data will be collected and which organization/department will be responsible for data collection.

While there is not a comprehensive PMS in place, a review of the program documents and files, combined with the findings of the interviews with Health Canada staff members, indicated significant improvements of the MW program PM system and the availability and quality of the collected performance data over the period covered under this evaluation. These improvements included:

Implementation of the revised 2011-2012 version of the CBRT which is completed by communities and submitted to regional coordinators;
NNADAP/NYSAP treatment centre report cards which provide program, client, and staff data, including completion rates;
A new Addictions Management Information System (AMIS) to improve the data collected from NNADAP and NYSAP treatment centres;
NNADAP treatment center outcome studies in 2011 and 2012, which provided comprehensive data on client outcomes who participate in NNADAP treatment centres;
Special studies to assess effectiveness of specific program activities and projects such as an evaluation of the seven original Mental Wellness Teams in 2011-2012, IRS RHSP client surveys, and NAYSPS case studies;
Training Inventory and Reporting Template to collect and report data on capacity building training sessions delivered in the communities;
IRS RHSP database of participants and services (e.g., referrals and transportation); and
Youth Solvent Addiction Committee Annual Reports, which provide information about NYSAP centers, the programming delivered and detailed data on participants.

During the interviews, Health Canada staff members noted that they are increasingly using the performance data for monitoring the program activities and making decisions. Of the 14 Health Canada staff members interviewed, 10 (71%) indicated that they regularly use the performance data for decision making purposes.

However, there are still many challenges and gaps associated with quality and availability of the data, especially with respect to consistency of the performance data across all program years. Although the MW Programs collect and report performance data on program activities and outputs and some data on program outcomes, the data was only available for one or two program years covered under the evaluation, and was not available for all regions, which made it difficult to assess the level of changes and improvements over time. A review of the program documents combined with the results of the interviews with Health Canada staff members indicated that among the MW Programs, the NNADAP residential component reports satisfactory performance data such as client background, participation in treatment and client outcomes. The performance data collected for Mental Wellness Teams, NAYSPS and IRS RHSP was also useful while the information collected for the Brighter Futures and Building Healthy Communities was insufficient (only some input and output information for 2009-2010 fiscal year). The nature (e.g., promotion and prevention activities, workshops, info session) and design (i.e., delivered through contribution agreements to a large number of communities) of programming delivered through Brighter Futures, Building Healthy Communities, and NAYSPS makes it very difficult to collect and report consistent performance data.

As the capacity of the communities to collect and report performance data varies greatly, it is very difficult to create a comprehensive reporting system and collect outcome data from the communities. Therefore, CBRT reporting focuses on collecting output and client participation data. During the community consultations, 90% of the community leads and 86% of the community program staff that participated in community focus group sessions indicated that CBRT reporting is not adequate to convey the problems that mental wellness staff are facing to accurately describe the activities implemented or to indicate the impact of the programming. The communities would like the CBRT reporting template to include a section where they can provide information on the achievements of their program and the problems and the challenges that they face. In addition, community stakeholders requested feedback from Health Canada about their CBRT reports, success of their programming, and implementation benchmarks so they could compare the progress of their programming with other communities or regions or best practices in the field. Along with feedback on CBRT reporting, the communities would like to receive a regular update (e.g., online newsletter) from Health Canada about the overall implementation of MW programs, best practices, learned lessons, funding opportunities and other developments in the field of MW. For example, a newsletter initiated in the Alberta region, which provides communities regular updates on the First Nations health, is perceived as being very effective.

Alternative Design Options

A review of mental wellness programming for Indigenous communities in the United States, Australia and New Zealand indicated that a wide variety of programs are undertaken to meet the diverse needs of the communities in each country. This diversity in programming does not allow exact comparisons between countries. Nevertheless, the review identified a number of lessons learned and best practices including the following:

Designating a specific Indigenous organization to be responsible for workforce development - The Māori organization called Te Rau Matatini in New Zealand provides a strategic focus for Māori workforce training, education and capability building for the advancement of Indigenous health and wellbeing;
Opening lines of communication, encouraging participation and building trust with Indigenous communities and organizations by involving them as equal partners;
Creating joint decision making structures and consulting with Indigenous stakeholders at the early stages of planning and policy development;
Incorporating Indigenous approaches to mental wellness programming and including Indigenous cultures and traditions that go beyond the narrow definition of health and incorporate all elements of wellbeing;
Employing a holistic multi-disciplinary (mental, physical, emotional and spiritual as well as focusing on family, community, environmental and spiritual connectedness), positive and strength-based approach to mental wellness;
Engaging communities to develop their own approaches and whole-of-community plans for addressing mental health issues; and
Developing comprehensive performance measurement and surveillance systems to monitor and report on progress. For example, in New Zealand, the Ministry of Health produces a quarterly report on improving Māori health outcomes and providing culturally-appropriate health programs and services. This helps all stakeholders to benchmark the progress made and understand the success of the programming and needs of the communities.

5. Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need for the Programs

There is a continued need for the MW Programs as First Nations people and Inuit consistently experience higher rates of mental health problems (e.g., psychological distress and mood disorder), suicide, illicit drug and alcohol use than the general population in Canada.

In Canada, there are a number of risk factors associated with mental wellness issues, including substance abuse and suicide. First Nations people and Inuit experience a disproportionately high prevalence of suicide, drug and alcohol addiction, substance use and other mental health problems compared to the overall Canadian population. For example, between 2005-2007, the suicide rate among youth under the age of 19 living in areas with a high First Nations population was 11 times higher for males and 21 times higher for females, as compared to youth living in areas with a low First Nations population.^{Footnote 29} Similar trends exist for Inuit populations, with suicide rates among youth under the age of 19 living in areas with a high Inuit population (Inuit Nunagat) being 35 times higher for males and 28 times higher for females compared to suicide rates among the Canadian population overall between 2004 to 2008.

Many First Nations and Inuit communities face crises related to suicide, alcohol and drug use, communicable diseases and child welfare apprehensions. The combined influence of historical and current traumas and stressors on some First Nations and Inuit individuals, families and communities are contributing factors to a high level of crises at the individual, family and community level. The activities implemented by the MW Programs are aligned with addressing the needs of First Nations people and Inuit and combating the high rates of substance abuse, suicide and other mental health issues.

5.1.2 Alignment with Government Priorities

The objectives and activities of FNIHB's MW Programs, which include the promotion of positive mental wellness as well as alcohol and substance abuse treatment, align well with priorities of the federal government as stated in recent federal budgets; Canada's first mental health strategy, Changing Directions, Changing Lives: The Mental Health Strategy for Canada; federal government's apology for Indian Residential Schools; and two strategic outcomes of Health Canada.

5.1.3 Alignment with Federal Roles and Responsibilities

The federal government has a role in the provision of health care for First Nations and Inuit populations. The MW Programs appear to complement, rather than duplicate, other programs which are designed to address mental health and addiction issues in First Nations and Inuit communities. The MW Programs are unique in terms of their broader scope, and focus on ensuring the services reflect First Nations and Inuit culture, traditions, and way of life.

Federal policies, such as the Indian Health Policy and the Indian Health Transfer Policy, outline the goals of the federal government with respect to Aboriginal health. The Indian Health Policy (1979) provides authority for the delivery of First Nations and Inuit health programs, of which mental wellness programming and alcohol/substance abuse treatment are a component. It also encourages the improvement of First Nations and Inuit health status, "through mechanisms generated and maintained by the communities themselves".

The MW Programs appear to complement, rather than duplicate, other federal programs which are designed to address mental health and addiction issues in First Nations and Inuit communities. The MW Programs are unique in terms of their broader scope than other federal programs, and focus on ensuring the services reflect First Nations and Inuit culture, traditions, and way of life.

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

The MW Programs have made progress in terms of achieving expected immediate, intermediate and long-term outcomes. However, greater effort is required in some areas to accelerate progress towards the desired outcomes.

While it was difficult to accurately measure the programs' performance due to limited performance measurement data, the available performance data, combined with the field work findings, indicates that the MW Programs have made progress towards their intended outcomes. The evaluation found that recent efforts by the MW Programs to integrate culture and traditions into mental wellness programming has resulted in an increase in the practice of healthy behaviors due to increased acceptance and participation in MW programs and services. The most frequent increases in healthy behaviors have been achieved in the reduction of substance abuse, increased self-esteem and confidence, improved sense of positive cultural identity, strengthened family and community relations, and increased help-seeking behavior by First Nations and Inuit individuals. The integration of culture into MW Programs has also resulted in increased community ownership to address mental health issues. The Mental Wellness Teams have been particularly effective in integrating culture into mental wellness programs and have been able to achieve significant results including greater access to specialized services and increased ownership among community members. The First Nations Mental Wellness Continuum Framework has been instrumental in providing guidance regarding the integration of culture and traditions into mental wellness programming, as well as outlining the continuum of essential services necessary to address the needs of First Nations people.

5.2.2 Demonstration of Economy and Efficiency

The MW Programs have undertaken a number of resource maximization measures to enhance their economy and efficiency. However, there are several factors that have constrained the efficiency and economy of the MW Programs, including resource challenges as funding amounts remaining largely unchanged over many years, short term funding cycles, high turnover of program staff in the communities, lack of integration among MW Programs, insufficient funding flexibility in community programming and insufficient data to assess the performance of MW Programs and allocate resources to the most effective MW programming.

Another constraint is the lack of integration among MW Programs. While some changes were introduced several years ago to promote more interaction and better coordination, the individual MW Programs are still mostly delivered in silos at the national level. At the community level, the MW Programs are somewhat integrated and this integration results from the conducting of joint initiatives, sharing of staff and co-locating of workplaces. Increased integration across MW Programs at all levels - national, regional and community - would improve efficiency through better coordination of activities and result in an improved continuum of care.

Over the period covered under the evaluation, the availability and quality of the collected performance data have significantly improved. Opportunities exist to improve the consistency of the data collected across program years.

6. Recommendations

Recommendation #1:

Focus should be placed on addressing gaps in the continuum of essential services in collaboration with First Nations as implementation of the First Nations Mental Wellness Continuum Framework moves forward.

Gaps currently exist in the continuum of services available at the community level. These include limited access to detox, aftercare and specialized/professional services. Potential means by which these gaps could be addressed include collaborating with partners at the regional level such as provincial service providers, communities and Tribal Councils to address issues related to wait times, access, the cultural competency of specialized service, and referral systems. Building the capacity of communities to provide specialized care has been effective in some communities; however, due to confidentiality reasons, service availability or urgent situations, some community members have been required to access such services outside of their community. Therefore, it is important to ensure such services are available through multiple avenues. Better coordination, such as referral systems, case management and discharge planning, including to P/T services as well as at NNADAP/NYSAP treatment centers will help community health services to better connect with those returning to communities, and provide improved aftercare services. In addition, shifting the focus towards a strength-based community development approach would enhance the continuum of services in communities.

Recommendation #2:

Moving forward, greater effort should be placed on sharing program related information (including best practices, benchmarks and lessons learned in the field of mental wellness) with communities to enhance capacity-building efforts. Similarly, there is a need to ensure that existing training resources are tailored to address the most pressing issues and that they focus on critical gap areas, such as trauma-informed care. Helping communities to enhance internal policies and procedures and staff capacities to ensure confidentiality and protect private information would help to enhance overall effectiveness of the services.

Recommendation #3:

Continue to integrate Mental Wellness programming at the national and regional levels within Health Canada, and formalize mechanisms to ensure integration across the individual programs.

Despite improvements in integration over recent years, FNIHB MW Programs are still largely implemented in silos within Health Canada at the national and regional levels, with insufficient interaction among staff in different programs. The evaluation identified variations from region to region and at the national level, regarding the extent to which staff across programs collaborate and integrate the activities of individual programs, at both the regional and national level. There also exist variations with regard to mechanisms for ensuring regular communication and the sharing of best practices by staff at the community, regional and national level.

Develop a comprehensive Performance Measurement Strategy to guide the collection and usage of performance measurement data.

Appendix 1 - Logic Model

Logic Model
Theme	Stakeholder Engagement and Collaboration	Service Provision	Capacity Building	Data Collection, Research and Surveillance	Policy Development and Knowledge Sharing
Target Group	First Nations on-reserve, Inuit in Inuit communities, eligible former Indian Residential School students and their families, and some support for knowledge development and primary prevention related to suicide for all Aboriginal people
Objectives	To fund and support the development, implementation, monitoring, and evaluation of community-based and culturally relevant programs and policies that promote mental wellness
Outputs	Collaboration in groups/committees/ consultations/projects Agreements Joint projects Working groups/ Councils/Advisory groups Committees/ Strategic alliances	Referrals/clients treated in accredited facilities/ MW projects Projects/activities Referrals Clients treated Counselling sessions Accredited facilities Travel reimbursement Resolution Health Support Workers	Culturally appropriate training, continuing education, professional development opportunities Culturally appropriate training material Training sessions Trained workers	Ongoing data collection to support service delivery, policy and program development Policies, procedures, guidelines, frameworks, reports, conferences/ workshops	Policies, procedures, guidance, frameworks developed Processes to share best practices Education/ awareness material Awareness campaigns Websites
Reach	National, regional and community partners and stakeholders, both internal and external	Recipients of funding and/or services: First Nations and Inuit individuals, families and communities and Territorial governments	Community workers and health professionals	Community representatives (e.g., workers, and Health Directors), regions and partners/ stakeholders at all levels	Partners and stakeholders at community, regional and national levels
Immediate Outcomes	Increased and improved collaboration and networking		Increased continuum of programs and services in First Nations and Inuit communities Increased participation of First Nations and Inuit individuals, families and communities in programs and services Increased awareness of healthy behaviours
Intermediate Outcomes	Increased practice of healthy behaviours Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide and other mental health issues Improved access to quality well-coordinated programs and services for First Nations and Inuit individuals, families and communities
Longer Term Outcomes	Contribution to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services

Logic Model Narrative for the First Nations and Inuit Mental Wellness Program

The primary objective of the First Nations and Inuit Mental Wellness Program is to fund and support the development, implementation, monitoring, and evaluation of community-based and culturally relevant programs and policies that promotes mental wellness. To achieve this objective and the expected outcomes stated on the logic model, the Program implements a number of activities, including:

Developing collaboration and partnerships with First Nations, Inuit, Federal, Provincial, and Territorial authorities and organizations through the signing of agreements, implementation of joint projects, participation in various committees and building of strategic alliances.
Delivering mental health programs and services for First Nations and Inuit individuals, families and communities by engaging community leadership and Territorial governments.
Building the capacity of community workers and health professionals at the community levels by organizing and delivering culturally appropriate training, continuing education, professional development opportunities and information and knowledge materials.
Organizing ongoing data collection research and surveillance to support service delivery, policy and program development.
Developing and sharing policies, procedures, guidance, and frameworks in partnership with stakeholders at community, regional and national levels.

In the immediate term, these activities are expected to result in:

Increased and improved collaboration and networking;
Increased continuum of programs and services in First Nations and Inuit communities;
Increased participation of First Nations and Inuit individuals, families and communities in programs and services; and
Increased awareness of healthy behaviours.

In the intermediate term, the Program activities are expected to result in:

Increased practice of healthy behaviours;
Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide and other mental health issues; and
Improved access to quality well-coordinated programs and services for First Nations and Inuit individuals, families and communities.

In the long term, the MW programs are expected to contribute to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services.

Appendix 2 - Summary of Findings

Rating of Findings
Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance:
A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Table 1: Relevance Rating Symbols and Significance
Evaluation Issue	Indicators	Overall Rating	Summary
Continued need for the programs
Does the Mental Wellness Program continue to address a demonstrable need?	The incidence, and prevalence of suicides, addictions, alcohol and substance use and other mental health issues among First Nations people and Inuit and crises among communities	High	There is a continued need for MW programming as First Nations people and Inuit experience higher rates of suicide and illicit drug and alcohol use and report experiencing mental health problems (e.g., psychological distress and mood disorder) at higher rates than the general population. There are also high rates of crises in First Nations and Inuit communities related to prescription drug abuse, addictions and suicide.
Are program activities responsive to current needs?	Responsiveness of programs to current needs.	High	The MW Programs are somewhat responsive to the needs of First Nations and Inuit communities as the program delivery is aligned with addressing the needs.
Alignment with Government Priorities
Is the Mental Wellness Program aligned with Federal government priorities and departmental strategic outcomes?	Evidence demonstrating mental wellness remains a priority of the federal government. Alignment of the MW program activities with departmental strategic priorities and outcomes.	High	Investing in mental health services has been highlighted as a priority in recent federal budgets and the activities of the MW programs are aligned with Canada's first mental health strategy Changing Directions, Changing Lives: The Mental Health Strategy for Canada; the federal government's apology for Indian Residential Schools and helps fulfill the commitment made to provide mental and emotional support to individuals involved in the settlement process; and two strategic outcomes of the Health Canada.
Alignment with Federal Roles and Responsibilities
Is the Mental Wellness Program aligned with federal roles and responsibilities?	Evidence that the Mental Wellness Program objectives and activities align with federal jurisdiction/role and evidence that program objectives align with departmental mandate and roles	High	Various policies and authorities provide a clear role for the federal government to provide health services for First Nations and Inuit.
Does the Mental Wellness Program duplicate, complement or overlap with other programs (e.g., federal, provincial, territorial)?	Evidence of Program differentiation/complementarity from/with other programs and stakeholder perception of duplication and linkages with provincial/territorial services	High	There are other programs and services delivered by FNIHB, and provincial and federal governments that share objectives similar to those of Mental Wellness programs. However, Mental Wellness programs mostly complement rather than overlap or duplicate other federal and provincial government programs.
Legend - Relevance Rating Symbols and Significance: High There is a demonstrable need for program activities; there is a demonstrated link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are clear. Partial There is a partial need for program activities; there is some direct or indirect link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program are partially clear. Low There is no demonstrable need for program activities; there is no clear link between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; role and responsibilities for the federal government in delivering the program have not clearly been articulated.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.

Table 2: Performance Rating Symbols and Significance
Issues	Indicators	Overall Rating	Summary
Achievement of Expected Outcomes (Effectiveness)
Increased and improved collaboration and networking	Stakeholder perception of increased and improved collaboration and networking in Mental Wellness programming and change, by year, in the number and type of collaborative arrangements, agreements and partnerships	Progress Made; Further Work Warranted	Collaboration and networking activities have improved at all system levels. Nationally, this has been facilitated through the collaborative development of the MW Continuum Framework and NNADAP Renewal Framework. Regionally, development of co-management structures in most provinces has facilitated partnerships with provincial and community partners. At the community level, MWTs/CWDTs have been critical in improving linkages and partnerships among stakeholders. Despite improvements in collaboration and networking, further progress in this area is required.
Improved continuum of programs and services in First Nations and Inuit communities	Stakeholder and community perceptions of continuum of services available and accessed and number and type of programs and services available to the communities, by year	Progress Made; Further Work Warranted	While the Mental Wellness Programs have made progress in improving the continuum of the programs and services in First Nations and Inuit communities, there continues to be gaps associated with early identification and intervention, crisis planning and response, trauma-informed treatment, detox and aftercare services.
Increased participation of First Nations and Inuit individuals, families and communities in programs and services	Stakeholder and community perceptions on increased participation in program activities and program documents showing profile of participants in different programming	Progress Made; Further Work Warranted	Although data is not available, there is a strong perception among community health staff and community members that participation in MW activities has increased. They attribute this to the incorporation of community culture and traditions into the programming, which has reduced the stigma associated with participation. However, an ongoing challenge is the ability to reach those who are most vulnerable and who need the services the most.
Increased awareness of healthy behaviours by First Nations and Inuit individuals	Stakeholder and community perceptions on increased awareness of the healthy behaviours and program documents demonstrating increased awareness.	Progress Made; Further Work Warranted	The MW programs have been successful in increasing awareness of healthy behaviours among representatives of the target groups. Children and youth have benefited the most by increasing their awareness on First Nations and Inuit culture and traditions and the importance of the strong cultural identity, attendance to land based activities and help seeking behaviour, self-esteem and self-confidence.
Increased practice of healthy behaviours by First Nations and Inuit individuals	Stakeholder and community perceptions on increased practice of healthy behaviours and program documents demonstrating change in behaviours	Progress Made; Further Work Warranted	Although available data is limited, there is a strong indication that the programs have increased practice of healthy behaviour among representatives of the target groups. Participants of MW programming report an increase in self-esteem and confidence, an improved sense of positive cultural identity, and strengthened family relations and community ties. More specifically, NAYSPS projects have contributed to an increase in suicide protective factors (e.g., confidence, hope, sense of belonging and help-seeking behaviour) and a reduction in risk factors, while NNADAP treatment centres report a reduction in substance use rates for participants.
Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide, and other mental health issues	Perception of stakeholders and community representatives with regards to capacity and community ownership of programs; profile of capacity building efforts delivered; and a review of the program documents and files demonstrating change in capacity and ownership and existing gaps	Little Progress: Priority for Attention	While efforts to improve community capacity have occurred, insufficient capacity remains one of the most pressing issues across many communities. The identification of training needs and implementation of capacity building activities are inconsistent, ad-hoc and, in many regions, reliant on activities delivered by partners. Staff burnout and high turnover represent significant constraints to community capacity and need to be addressed.
Improved access to quality programs and services for First Nations and Inuit individuals, families and communities	Levels of accreditation and certification of the existing treatment and health centers and staff members; perception of the increased quality among stakeholders; and incorporation of the culturally relevant approaches	Progress Made; Further Work Warranted	Progress has been made to increase quality and accessibility of the programs and services through the increased focus on accreditation of NNADAP treatment centres and accreditation of community health centres, the incorporation of cultural and spiritual practices into programming, introduction of MWTs, better coordination of the services and crisis planning. Funding inconsistencies, issues related to protecting confidentiality of participants, high staff turnover and remote and isolated location of some communities are negatively affecting community access to quality services.
Contribution to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services	Program documents demonstrating measurable improvements in health status in areas related to mental wellness; and perception of the stakeholders and program participants in improved health status	Progress Made; Further Work Warranted	There is limited performance data to effectively assess the extent to which MW programs have contributed to improved health status of First Nations and Inuit individuals, families and communities. However, given that the MW Programs have resulted in the increased practice of healthy behaviors, there exists early indications that that the health status of First Nations and Inuit individuals has improved.
Demonstration of Efficiency and Economy
Has the program undertaken its activities in the most efficient and economical manner?	Stakeholder perception of economical and efficient use of the program resources; breakdown of the program budget and expenditures; changes made to improve efficiently and economy; and social return on investment analysis of a treatment center.	Progress Made; Further Work Warranted	The MW Programs have undertaken a number of resource maximization measures to enhance their economy and efficiency. However, there are several factors that have constrained the efficiency and economy of the MW Programs, including unchanged funding, short term funding cycles, high turnover of program staff in the communities, lack of integration among MW Programs, insufficient funding flexibility in community programming and insufficient data to assess the performance of MW Programs.
Is there appropriate performance measurement in place? If so, is the information being used to inform senior management decision-makers?	A review of the performance measurement system and performance data collected for quality and consistency; perception of the stakeholders with regards to the appropriateness and use of the performance data and opportunities for improvement	Progress Made; Further Work Warranted	Over the period covered under this evaluation, MW programs have made some improvements in the quality and adequacy of the performance data collected and reported. Further improvements are needed to increase the usefulness of the performance data for ongoing program monitoring, making policy and program decisions and setting priorities.

Table 3: Summary of Relevance and Performance Ratings
Evaluation Issue	High	Partial	Low
Relevance:
Issue: Continued need for the programs
Does the Mental Wellness Program continue to address a demonstrable need?	√	N/A	N/A
Issue: Alignment with Government Priorities
Is the Mental Wellness Program aligned with Federal government priorities and departmental strategic outcomes?	√	N/A	N/A
Issue: Alignment with Federal Roles and Responsibilities
Is the Mental Wellness Program aligned with federal roles and responsibilities?	√	N/A	N/A
Does the Mental Wellness Program duplicate, complement or overlap with other programs (e.g., federal, provincial, territorial)?	√	N/A	N/A
	Achieved	Progress Made; Further Work Warranted	Little Progress; Priority for Attention
Performance:
Issue: Achievement of Expected Outcomes (effectiveness)
Increased and improved collaboration and networking	N/A	√	N/A
Improved continuum of programs and services in First Nations and Inuit communities	N/A	√	N/A
Increased participation of First Nations and Inuit individuals, families and communities in programs and services	N/A	√	N/A
Increased awareness of healthy behaviours by First Nations and Inuit individuals	N/A	√	N/A
Increased practice of healthy behaviours by First Nations and Inuit individuals	N/A	√	N/A
Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide, and other mental health issues	N/A		√
Improved access to quality programs and services for First Nations and Inuit individuals, families and communities	N/A	√	N/A
Contribution to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services	N/A	√	N/A
Issue: Demonstration of Efficiency and Economy
Has the program undertaken its activities in the most efficient and economical manner?	N/A	√	N/A
Is there appropriate performance measurement in place? If so, is the information being used to inform senior management decision-makers?	N/A	√	N/A

Appendix 3 - Evaluation Methodology

Evaluation Scope

The purpose of this project was to conduct an evaluation of the First Nations and Inuit Mental Wellness Program. The evaluation was conducted from 2014/15 to 2015/16 and assessed the relevance and performance (effectiveness, efficiency and economy) of the MW Programs.

The most recent evaluation of the First Nations and Inuit Mental Wellness Programs (previously known as the Mental Health and Addictions Cluster) was conducted in 2012/2013 and covered the period of program implementation from April 1, 2005 to March 31, 2010. Consequently, the scope of this evaluation covered the period from April 1, 2010 to March 31, 2015 to ensure full coverage of the program funding and compliance with the Financial Administration Act. Information and data (assessments, performance reports and previous evaluations of related programs) prior to April 2010 was also used to provide context and inform trend analysis.

The scope of the evaluation did not include activities undertaken in British Columbia, where program responsibilities have been transferred to the First Nations Health Authority under the British Columbia Tripartite Framework Agreement. The scope of the evaluation also did not include MW program activities undertaken in northern Canada (north of the 60th parallel). Funding for MW programs in Nunavut, Yukon and the Northwest Territories are part of block funding agreements reached in the Northern Wellness Approach. These block agreements require that evaluations are undertaken by the respective territorial governments. Because the activities undertaken by the IRS RHSP, NNADAP treatment centres and MWTs in northern Canada are not included in the territorial block funding agreements, the services provided by these programs to First Nations people and Inuit living north of 60 degrees latitude are within the scope of this evaluation. However, the small sample size of Inuit participants reached has resulted in minimal evidence being collected and has limited the ability of the evaluation to report separately regarding program impact on the Inuit population.

Evaluation Issues

The Treasury Board Secretariat Directive on the Evaluation Function requires that evaluations address five core areas of coverage including continued need for the program, alignment with government priorities, alignment with federal roles and responsibilities, achievement of expected outcomes, and demonstration of efficiency and economy. The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Table 1: Core Evaluation Issues and Questions

Evaluation Issues and Questions
- Relevance
  - Issue 1: Continued Need for the Programs: Assessment of the extent to which the programs continue to address a demonstrable need and are responsive to the needs of Canadians.
    - 1.1 Does the Mental Wellness Program continue to address a demonstrable need?
    - 1.2 Is the Mental Wellness Program responsive to the needs of the client population or are there gaps in existing services?^{Footnote st12}
  - Issue 2: Alignment with Government Priorities: Assessment of the linkages between the programs' objectives and (i) federal government priorities and (ii) departmental strategic outcomes
    - 2.1 Is the Mental Wellness Program aligned with Federal government priorities?
    - 2.2 Is the Mental Wellness Program aligned with departmental strategic outcomes?
  - Issue 3: Alignment with Federal Roles and Responsibilities: Assessment of the role and responsibilities for the federal government in delivering the programs.
    - 3.1 Is the Mental Wellness Program aligned with federal roles and responsibilities?
    - 3.2 Does the Mental Wellness Program duplicate or overlap with other programs (e.g., federal, provincial, territorial)?
- Performance
  - Issue 4: Effectiveness: To what extent has the Mental Wellness Program made progress towards the achievement of its immediate, intermediate, and long-term outcomes?
    - 4.1 To what extent has the Mental Wellness Program made progress towards the achievement of its immediate outcomes?
      - 4.1.1 Increased and improved collaboration and networking
      - 4.1.2 Improved continuum of programs and services in First Nations and Inuit communities
        
        4.1.2.1 How has the implementation of the Mental Wellness Teams impacted community wellness?
        
        4.1.2.2 How has the implementation of the IRS RHSP impacted individual and community wellness?
      - 4.1.3 Increased participation of First Nations and Inuit individuals, families and communities in programs and services
      - 4.1.4 Increased awareness of healthy behaviours by First Nations and Inuit individuals
    - 4.2 To what extent has the Mental Wellness Program made progress towards the achievement of its intermediate outcomes?
      - 4.2.1 Increased practice of healthy behaviours by First Nations and Inuit individuals
      - 4.2.2 Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide, and other mental health issues
      - 4.2.3 Improved access to quality programs and services for First Nations and Inuit individuals, families and communities
    - 4.3 To what extent has the Mental Wellness Program made progress towards the achievement of its long-term outcomes?
      - 4.3.1 Contribution to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services
    - 4.4 What are the factors that have contributed to the successes of the mental wellness programs?
    - 4.5 Have there been any challenges/barriers to the successes of the mental wellness programs?
    - 4.6 Are the Continuum Framework elements driving change at the community level?
  - Issue 5: Efficiency and Economy: Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes.
    - 5.1 Has the Mental Wellness program undertaken its activities in the most efficient manner?
    - 5.2 Has the Mental Wellness program undertaken its activities in the most economical manner?
    - 5.3 Are Mental Wellness services linked to other social services?
    - 5.4 Is there appropriate performance measurement in place? Is it possible to compare with existing benchmarks or similar programs?
    - 5.5 Using SROI methodology, what is the social value created by a treatment program?

Footnote st12

Return to footnote st12 Referrer

For the sake of brevity, the finding for this evaluation issue was merged with 4.1.2 in the evaluation report.

The evaluation focused largely on the achievements of the Mental Wellness Program and lessons learned as well as challenges that were experienced during program delivery. The evaluation included:

The use of culturally relevant research and evaluation methods;
Recognition of cultural protocols relevant to the First Nation and Inuit communities (i.e., community leaders should be consulted to obtain their consent prior to approaching community-based staff/project members individually);
Respect for privacy and confidentiality;
Inclusion of First Nations members in the evaluation team;
Involvement of departmental program staff, experts and key external stakeholders (i.e., AFN, ITK) in the development of tools and review of key documents; and
An SROI assessment of one NNADAP adult treatment centre.

Data Collection and Analysis Methods

The evaluation collected and analyzed data from multiple sources, summarized as follows:

An extensive review of the program documents and files. The types of files and documents reviewed included internal government documents such program authorities, public opinion research reports, previous performance, evaluation and audit reports, policy documents, budget and expenditure information, performance measurement data reports, annual progress reports, grants and contributions files, and administrative records specific to the evaluation questions posed; other evaluation reports, their accompanying Management Response and Action Plans and associated progress reports, and previous relevant reports of the Office of the Auditor General; documents produced by the FNIHB headquarters, regional offices and projects such as regional progress reports and work plans, Community Based Reporting Template (CBRT) data, and financial information and policies. While the evaluation focused on the period from April 2010 to March 2015, we also reviewed information and data (assessments, performance reports and previous evaluations of related programs) on activities prior to April 1, 2010 to provide context and inform trend analysis. In addition, as part of this methodology, a review was conducted to compare the findings of this evaluation with results of the similar evaluations of Mental Wellness programs activities implemented in northern Canada (north of the 60th parallel). The purpose of this review was to identify similarities and differences in need for and performance of the MW programs across different regions of Canada. The results of the document and data review were summarized and presented in this report.
Literature Review and Comparative Analysis, including sources external to the FNIHB Program, both peer-reviewed (scientific and academic) and grey literature (newspapers and websites) on topics related to mental health, substance abuse, suicide, child development and the effects of residential schools. As part of the literature review, a comparative analysis of the MW Programs was undertaken with similar programs in the United States, New Zealand, and Australia. A profile of each program that identified the delivery agency, delivery model, outputs, target groups, reach, budget, outcomes, best practices and lessons learned was produced based on the review of relevant websites, journals and grey literature articles. The findings of the literature review were summarized and presented in this report.
Web-based survey of community leaders and health directors/staff. Three web surveys were conducted (in English and French) between March 23 and May 1, 2015 to gather input about the impact of the programs from community leaders (Chiefs, Inuit Mayors, Band Councillors, Band staff), Health Directors and community health staff, and where possible, program participants. Chiefs, Health Directors and health staff directly were contacted with an invitation to respond to the web surveys. Chiefs were asked to forward the invitation to Band Council and staff, and Health Directors and health staff were asked to forward invitations for the web survey to program participants. A web survey invitation (and weekly reminders throughout the time the web survey was running) was sent directly to 286 community leaders and to 478 Health Directors and health/program staff. Of the 286 community leaders contacted, 11 participated in the survey yielding 3.5% response rate, and of the 478 Health Directors contacted, 158 participated in the survey yielding a 33% response rate. The findings of the survey with community stakeholders were summarized and presented in this report.
Site visits First Nations and Inuit communities and locations where MW Programs have been implemented. A preliminary list of 28 communities across six regions of Canada was identified by program staff members. The evaluation contacted all communities and was able to complete site visits in 14 communities. Site visits involved a review of relevant files and documents related to each community and the mental wellness programming delivered in the community, observation of the service delivery locations, 14 focus group discussions with staff members, a survey of 68 community program staff and leaders, 12 focus group discussions with programming participants, a survey of 110 programming participants and in-person consultations with 10 community leaders and stakeholders. The findings of the site visits were summarized and presented in this report. Results of the focus group discussions and community consultations were validated by participants to mitigate effects of possible respondent biases and increase reliability of the results. The participants were provided an opportunity to comment, revise, and validate summary results of the discussions and consultations.
Phone interviews with 40 key informants, including 7 Health Canada national office program staff, 9 regional staff, 2 representatives of other government departments and First Nations/Inuit organizations, 6 Mental Wellness Team Coordinators, and 16 community leads representing various regions of Canada who were closely involved in program delivery at the community level. A list of key informants was provided by Health Canada. Key informants included those who were directly involved in MW Programs at the national and regional levels, as well as program partners and experts working in one or more areas of MW programming. An email was sent to each interviewee that described the purpose of the evaluation and solicited their participation in a telephone interview. A relevant interview guide was attached to each email. The findings of the key informant interviews were summarized and presented in this report.
Social Return on Investment assessment. This included the review of documents provided by the treatment centre regarding the overall treatment process, activities, and descriptive details of the clients. A literature search was conducted to obtain statistics on relapse and generally accepted standards. A site visit was undertaken. During the site visits, interviews were held with staff from all aspects of the treatment centre (pre-treatment, day treatment, cultural staff, post-treatment, as well as centre management). A representative sample of clients was also interviewed. The purpose of the interviews with clients was to gain an understanding of what was the most important result of participating in the treatment program, as well as where they thought they would be without the treatment centre. Given the staff members' experiences with the treatment centre, they were asked about the goals, format and trends related to client recovery and relapse. These questions helped to articulate the theory of change for the centre. The outcomes chosen align with the individual outcomes from the overall Mental Wellness Program. Once the key outcomes were determined and validated by the staff at the treatment centre, indicators were selected to reflect these outcomes. For instance, the client interviews identified reuniting with children as a main driver for undergoing treatment. This aligned with the overall outcome of increased coping skills. It was decided to limit the outcomes to three, to focus on the main outcomes. Indicators were also limited to the three most relevant. Each of the proxy indicators for financial costs were chosen to reflect the cost if the indicator was not achieved. The actual dollar values assigned to the proxy costs were based on available Canadian or provincial data, and many had been validated by the SROI Canada Proxy Database.

Data Analysis

The information gathered from each line of evidence was summarized and presented in the progressive evaluation report outline and data collection and analysis database. SPSS software was used for cleaning and analysis of both qualitative and quantitative data collected via interviews and surveys. Data analysis techniques included statistical analysis of quantitative data from databases and surveys and key informant interviews and thematic analysis of qualitative data. Preliminary findings were presented in PowerPoint form to the Evaluation Directorate, Evaluation Committee and Assembly of First Nations/Inuit Tapiriit Kanatami partners. Triangulation of multiple lines of evidence was used to arrive at conclusions and test the strengths and limitations of each line of inquiry when writing final report. Conclusions that were supported by more than one line of evidence were presented in the final report.

Additional Detail on Social Return on Investment calculations and analysis

The outcomes for the treatment centre were based on the centre's theory of change, and align with the overall individual outcomes for the Mental Wellness Program. The theory of change states that: If First Nations and Inuit adults with alcoholism and/or drug dependency participate in a culturally relevant, safe, client-centered residential treatment program with a strengths- based approach, including a pre-treatment and aftercare component, then they will have better self-connection, better coping skills and a greater ability to reduce their dependency on alcohol and/or drugs leading to increased balance and wellness. The theory of change outcomes selected were increased coping skills, increased wellness and reduced dependency on alcohol and/or drugs.

As there was limited follow-up after treatment, relapse rates and the experience of the staff was relied on to calculate the number of people who would achieve each outcome. Based on data from the NNADAP outcome study and the treatment centre aftercare program, it was decided to select a value of 50% success for reduced addiction. The quantity of other outcomes was selected at occurring for roughly 10% of the clients. In the instance of children in care, this could be an under representation as clients in the program may have had more than one child. The instances of reduced hospital visits and interactions with police were based on client interviews regarding the number of occurrences in the year prior to entering the treatment.

As the treatment centre is not the only possible program clients are accessing, a 25% attribution rate was used to account for the successes attributable to other programs in the communities that clients may have benefited from. This attribution rate was 50% for the decreased homelessness, as the treatment centre did not provide housing. There was also a reduction in benefits over time, as research has shown that success rates for alcohol and substance abuse treatment decreases over time. With that knowledge, the Social Return on Investments was assessed over 3 years.

The input for the SROI calculation includes the funding provided from all sources. Over 75% of the program operational funding is from the federal government through NNADAP funding. Capital funding was not included.

Table 2 provides detail on the financial values calculated and the financial proxies. Table 3 describes the summary values.

Table 2: Value associated with each indicator
Theory of Change Outcomes	Indicator	Financial proxy indicator	Financial Proxy Value	Quantity^{Footnote a}	Impact				Total Value over 3 years	Total Present Value
Theory of Change Outcomes	Indicator	Financial proxy indicator	Financial Proxy Value	Quantity^{Footnote a}	Deadweight^{Footnote b}	Displacement^{Footnote c}	Attribution^{Footnote d}	Drop-off^{Footnote e} (each year)	Total Value over 3 years	Total Present Value
increased coping skills	increased ability to parent	cost for a child in foster care	$32 000	14	10%	0%	25%	20%	$737,856	$641,043
increased coping skills	increased employment or education	yearly disability support payments	$13 176	14	10%	0%	25%	20%	$303,812	$263,949
increased wellness	decrease in number of ER visits	cost of ER visit and ambulance	$388	20	10%	0%	25%	20%	$12,781	$11,104
	decrease in hospital admissions related to alcohol or drug abuse	cost of hospital stay related to alcohol or drugs	$6 368	10	10%	0%	25%	20%	$104,881	$91,120
	decrease in use of medical transport	average cost of transport	$301	10	10%	0%	25%	20%	$4,957	$4,307
reduced dependency on alcohol and/or drugs	reduced addiction	cost of addiction	$55 400	69	10%	0%	25%	20%	$6,295,822	$5,469,753
	decreased homelessness	shelter costs	$6 752.50	5	10%	0%	50%	20%	$37,071	$32,207
	decreased interactions with police	policing costs	$342	14	10%	0%	25%	20%	$7,886	$6,851
Table 2 Footnotes Table 2 footnote a The quantity refers to the number of times this indicator was achieved for one year of treatment center clients. Return to table 2 footnote a Referrer Table 2 footnote b Deadweight is a measure of the amount of outcome that would have happened even if the activity had not taken place. Return to table 2 footnote b Referrer Table 2 footnote c Displacement is an assessment of how much of the outcome displaced other outcomes. This was not seen as an issue with substance abuse programs. Return to table 2 footnote c Referrer Table 2 footnote d Attribution assesses how much of the outcome was caused by the contribution of another organization or program. Return to table 2 footnote d referrer Table 2 footnote e Drop-off accounts for the fact that the amount of outcome in future years will be less. Return to table 2 footnote e referrer

Table 3: SROI Ratio
Total Present Value (TPV)	$6 520 333
Input	$ 1 692 548 ($ 1 271 417- federal $ 421 131- provincial)
SROI Ratio (TPV/Input)	3.85:1

With those calculations and assumptions, we are able to say that for every dollar of funding in a year, the social return is $3.85 over 3 years.

Additional Detail on Evaluation Challenges and Mitigation Strategies

There was a low level of participation from health directors/staff, and community leaders in the web-based survey. Because of the need to protect anonymity of respondents, people were not asked to identify their communities. Without this information, it is not possible to assess how many communities the respondents from each region represent (several respondents could have come from the same community). It was also unclear extent to which Canadian regions were evenly represented in the survey. To mitigate this challenge, the findings of the web-based survey were triangulated with the results of the site visits and key informant interviews.

Appendix 4 - References

Allan, B. & Smylie, J. (2015). First Peoples, second class treatment: The role of racism in the health and well-being of Indigenous peoples in Canada. Toronto, ON: The Wellesley Institute.
Assembly of First Nations, Health Canada. The First Nations Mental Wellness Continuum Framework. (2015). Accessed at: http://health.afn.ca/uploads/files/24-14-1273-fn-mental-wellness-framework-en05_low.pdf
Boyer, Yvonne and McCaslin, Wanda D. First Nations Communities at Risk and in Crisis:
Justice and Security. Journal of Aboriginal Health, November 2009. http://www.naho.ca/documents/journal/jah05_02/05_02_03_Communities.pdf
Canada (2014). Economic Action Plan 2014, the Road to Balance: Creating Jobs and Opportunities. Retrieved, Jan 28, 2016: http://www.budget.canada.ca/2014/docs/plan/pdf/budget2014-eng.pdf
Centre for Suicide Prevention (2013) Suicide Prevention Resource Toolkit.
Cousins, J., Descent, D., Kinney, M., Moore, M., Pruden, J., Sanderson, K., and Wood, I. (2010). National Aboriginal Youth Suicide Prevention Strategy: Multiple Case Study of Community Initiatives.
First Nations Information Governance Centre. (2014). Youth Resilience and Protective Factors Associated with Suicide in First Nations Communities.
First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.
FNIHB (2014) NNADAP Report Cards
Galloway T. & Saudny H. (2012). Inuit Health Survey 2007-2008 Nunavut Community and Personal Wellness
Health Canada (2005). Fetal Alcohol Syndrome. Ottawa, ON: First Nations Inuit Health, Health Canada.
Health Canada. (2010). NIHB Ontario Region prescription drug trends: A ten-year analysis.
Health Canada, Canadian Alcohol and Drug Use Monitoring Survey (2011). http://www.hc-sc.gc.ca/hc-ps/drugs-drogues/stat/_2011/summary-sommaire-eng.php
Health Canada. (2012). Transition Binder Template First Nations and Inuit Health Branch.
Health Canada. (2012). National Native Alcohol and Drug Abuse Program Treatment Centre Outcome Study, Summary Report.
Health Canada. (2013). National Native Alcohol and Drug Abuse Program Treatment Centre Outcome Study, Phase II, 2012 Summary Report, Final Draft.
Health Canada (2014). Canadian Alcohol and Drug Use Monitoring Survey: Summary of Results for 2012.
Health Canada. (2015). First Nations Mental Wellness Continuum Framework.
Health Canada (2014) Report on Plans and Priorities 2014-2015.
Health Canada (2015) Report on Plans and Priorities 2015-2016.
IRS RHSP (2011) Gender-Based Analysis of the Indian Residential Schools Resolution Health Support Program.
Khan, Saman (2008). Aboriginal Mental Health: The statistical reality. "Aboriginal People" issue of Visions Journal, 2008, 5 (1), pp. 6-7.
Kielland, Norah & Simeone, Tonina (2014). Current Issues in Mental Health in Canada: The Mental Health of First Nations and Inuit Communities. Library of Parliament, Publication No. 2014-02-E.
Mental Health Commission of Canada (2012). Changing Directions, Changing Lives: The Mental Health Strategy for Canada.
National Aboriginal Health Organization (2009). Community Resilience: Models, Metaphors and Measures.
National Advisory Committee on Prescription Drug Misuse (NACPDM) (2013). First do no harm: Responding to Canada's prescription drug crisis. Ottawa: Canadian Centre on Substance Abuse.
National Collaborating Centre for Aboriginal Health (2010). Fact Sheet: Aboriginal and Non-Aboriginal Children in Child Protective Services.
National Collaborating Centre for Aboriginal Health (2011). Looking for Aboriginal Health in Legislation and Policies, 1970 to 2008: The Policy Synthesis Project. 2011.
Oliver, L., Peters, P.A., & Kohen, D.E.. (2012) Mortality Rates among Children and Teenagers Living in Inuit Nunangat from 1994 to 2008. http://www.statcan.gc.ca/pub/82-003-x/2012003/article/11695-eng.htm
Peters, P.A., Oliver, L.N., & Kohen, D.E. (2013). Mortality among children and youth in high-percentage First Nations identity areas, 2000-2002 and 2005-2007. http://www.rrh.org.au/publishedarticles/article_print_2424.pdf
Polson-Lahache, W. (2012). National Native Alcohol and Drug Abuse Program (NNADAP) Renewal Engagement Review, Final Report.
Public Health Agency of Canada. (2014). Developing a Federal Framework for Suicide Prevention: Report on Stakeholder Consultation Process.
Rhymes, J. (2013). First Nations Mental Wellness Continuum Framework: Atlantic Regional Submission.
Statistics Canada (2012). Aboriginal Peoples Survey, 2012: Lifetime suicidal thoughts among First Nations living off reserve, Métis and Inuit aged 26 to 59: Prevalence and associated characteristics. Retrieved March 2016: http://www.statcan.gc.ca/pub/89-653-x/89-653-x2016008-eng.htm
Statistics Canada (2012). Inuit Health: Selected findings from the 2012 Aboriginal Peoples Survey. http://www.statcan.gc.ca/pub/89-653-x/89-653-x2014003-eng.htm
Statistics Canada (2015). Table 105-0512 - Health indicator profile, by Aboriginal identity, age group and sex, four year estimates, Canada, provinces and territories, occasional (rate), CANSIM (database). (accessed: 2015-04-16) http://www5.statcan.gc.ca/cansim/a26?lang=eng&id=1050512
Statistics Canada. (2014). Inuit health: Selected findings from the 2012 Aboriginal Peoples Survey; 89-653-X -- No. 003; Retrieved on March 2016 http://www.statcan.gc.ca/pub/89-653-x/89-653-x2014003-eng.pdf
Wallace, Susan (2014). Inuit Health: Selected findings from the 2012 Aboriginal Peoples Survey. Statistics Canada.

List of Acronyms

AFN: Assembly of First Nations
AMIS: Addictions Management Information System
BF: Brighter Futures
BHC: Building Healthy Communities
CBRT: Community-Based Reporting Template
CCHS: Canadian Community Health Survey
CWDT: Community Wellness Development Teams
FNHA: First Nations Health Authority
FNIGC: First Nations Inuit Governance Centre
FNIHB: First Nations and Inuit Health Branch
FNMWC: First Nations Mental Wellness Continuum Framework
INAC: Indigenous and Northern Affairs Canada
IRS RHSP: Indian Residential Schools Resolution Health Support Program
IRS: Indian Residential School
ITK: Inuit Tapiriit Kanatami
MW: Mental Wellness
MWT: Mental Wellness Team
NACPDM: National Advisory Council on Prescription Drug Misuse
NAYSPS: National Aboriginal Youth Suicide Prevention Strategy
NNADAP: National Native Alcohol and Drug Abuse Program
NYSAP: National Youth Solvent Abuse Program
PAA: Program Alignment Architecture
PMS: Performance Measurement Strategy
PMU: Performance Measurement Unit
RHS: Regional Health Survey
SROI: Social Return on Investment

Footnotes

Footnote i

Return to footnote i Referrer

Note that other strategies/funding sources form part of the FNIHB's MW Programs, but are excluded from the scope of this evaluation. For the purposes of this evaluation, the term "MW Programs" refers only to those programs within scope for this evaluation.

Footnote ii

Return to footnote ii Referrer

Heavy drinking defined as 5 or more drinks per sitting at least once a month in the past 12 months

Footnote iii

Return to footnote iii Referrer

Scale of 1 to 5 where 1 is not at all successful, 3 is somewhat successful and 5 is very successful.

Footnote iv

Return to footnote iv Referrer

Rating scale from 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent.

Footnote v

Return to footnote v Referrer

Based on available data from 38 or the 44 NNADAP/NYSAP treatment centres. Three regions listed limitations with the data.

Footnote vi

Return to footnote vi Referrer

Scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent.

Footnote vii

Return to footnote vii Referrer

All of the indicators of healthy behavior are based on a review of program activities and expected outcomes, as well as other indicators for Indigenous mental wellness.

Footnote viii

Return to footnote viii Referrer

Scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent.

Footnote ix

Return to footnote ix Referrer

The funding cycles varies based on the program and ranges from one year to three years.

Footnote 1

Return to footnote 1 Referrer

First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.

Footnote 2

Return to footnote 2 Referrer

Statistics Canada. Table 2 - Prevalence of selected chronic conditions diagnosed by health professionals, by population characteristics, off-reserve First Nations population and total population of Canada, aged 15 years and older, 2012. www.statcan.gc.ca/pub/89-653-x/2016010/tbl/tbl02-eng.htm

Footnote 3

Return to footnote 3 Referrer

Statistics Canada. Study: Inuit health: Selected findings from the 2012 Aboriginal Peoples Survey. www.statcan.gc.ca/daily-quotidien/140826/dq140826b-eng.htm

Footnote 4

Return to footnote 4 Referrer

Statistics Canada. Aboriginal Peoples Survey, 2012. Social determinants of health for the off-reserve First Nations population, 15 years of age and older, 2012. www.statcan.gc.ca/pub/89-653-x/89-653-x2016010-eng.htm

Footnote 5

Return to footnote 5 Referrer

Statistics Canada. Table 105-0512 - Health indicator profile, by Aboriginal identity, age group and sex, four year estimates, Canada, provinces and territories, occasional (rate), CANSIM (database). (accessed: 2015-04-16) http://www5.statcan.gc.ca/cansim/ a26?lang=eng&id=1050512

Footnote 6

Return to footnote 6 Referrer

Peters, P.A., Oliver, L.N., & Kohen, D.E. (2013). Mortality among children and youth in high-percentage First Nations identity areas, 2000-2002 and 2005-2007. http://www.rrh.org.au/publishedarticles/article_print_2424.pdf

Footnote 7

Return to footnote 7 Referrer

Oliver, L., Peters, P.A., & Kohen, D.E.. (2012) Mortality Rates among Children and Teenagers Living in Inuit Nunangat from 1994 to 2008. Statistics Canada. Health Reports (vol. 23, no. 3) http://www.statcan.gc.ca/pub/82-003-x/2012003/article/11695-eng.htm Uses a geographic approach and data from the Vital Statistics Death Database. Standardized around the 1996, 2001 and 2006 Census.

Footnote 8

Return to footnote 8 Referrer

Footnote 9

Return to footnote 9 Referrer

Footnote 10

Return to footnote 10 Referrer

First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.

Footnote 11

Return to footnote 11 Referrer

Statistics Canada. 2012. Aboriginal Peoples Survey, 2012: Lifetime suicidal thoughts among First Nations living off reserve, Métis and Inuit aged 26 to 59: Prevalence and associated characteristics. Retrieved March 2016: http://www.statcan.gc.ca/pub/89-653-x/89-653-x2016008-eng.htm

Footnote 12

Return to footnote 12 Referrer

First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.

Footnote 13

Return to footnote 13 Referrer

Statistics Canada. 2014. Inuit health: Selected findings from the 2012 Aboriginal Peoples Survey; 89-653-X -- No. 003; Retrieved on March 2016 http://www.statcan.gc.ca/pub/89-653-x/89-653-x2014003-eng.pdf

Footnote 14

Return to footnote 14 Referrer

Health Canada, Canadian Alcohol and Drug Use Monitoring Survey. 2011. http://www.hc-sc.gc.ca/hc-ps/drugs-drogues/stat/_2011/summary-sommaire-eng.php

Footnote 15

Return to footnote 15 Referrer

First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.

Footnote 16

Return to footnote 16 Referrer

Galloway T. & Saudny H. Inuit Health Survey 2007-2008 Nunavut Community and Personal Wellness

Footnote 17

Return to footnote 17 Referrer

First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.

Footnote 18

Return to footnote 18 Referrer

National Advisory Committee on Prescription Drug Misuse (NACPDM). 2013. First do no harm: Responding to Canada's prescription drug crisis. Ottawa: Canadian Centre on Substance Abuse.

Footnote 19

Return to footnote 19 Referrer

Health Canada (2014). Canadian Alcohol and Drug Use Monitoring Survey: Summary of Results for 2012

Footnote 20

Return to footnote 20 Referrer

Health Canada. (2010). NIHB Ontario Region prescription drug trends: A ten-year analysis. http://www.chiefs-of-ontario.org/sites/default/files/files/NIHB%20Ontario%20Region% 20Prescription%20Drug%20Trends%20A%20Ten-Year%20Analysis_0.pdf

Footnote 21

Return to footnote 21 Referrer

Kielland, Norah & Simeone, Tonina. 2014. Current Issues in Mental Health in Canada: The Mental Health of First Nations and Inuit Communities. Library of Parliament, Publication No. 2014-02-E.

Footnote 22

Return to footnote 22 Referrer

Centre for Suicide Prevention (2013) Suicide Prevention Resource Toolkit

Footnote 23

Return to footnote 23 Referrer

Allan, B. & Smylie, J. (2015). First Peoples, second class treatment: The role of racism in the health and well-being of Indigenous peoples in Canada. Toronto, ON: The Wellesley Institute.

Footnote 24

Return to footnote 24 Referrer

First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.

Footnote 25

Return to footnote 25 Referrer

First Nations Information Governance Centre. (2012). First Nations Regional Health Survey Phase 2 2008-2010. National Report on the Adults, Youth and Children Living in First Nations Communities.

Footnote 26

Return to footnote 26 Referrer

Northern Region 2012. Mental Health and Addictions Evaluation 2005-2010

Footnote 27

Return to footnote 27 Referrer

Department of Finance Canada. 2015. Budget Plan: Strong Leadership, A Balanced-Budget, Low-Tax Plan For Jobs, Growth And Security

Footnote 28

Return to footnote 28 Referrer

House of Commons (2014) The Road to Balance: Creating Jobs and Opportunities. Retrieved, Jan 28, 2016: http://www.budget.canada.ca/2014/docs/plan/pdf/budget2014-eng.pdf

Footnote 29

Return to footnote 29 Referrer

Page details

Date modified:: 2020-09-14

Evaluation of the First Nations and Inuit Mental Wellness Programs 2010-2011 to 2014-2015

Table of Contents

List of Tables

Executive Summary

Evaluation Purpose and Scope

Program Description

Conclusions - Relevance

Continued Need for the Programs

Alignment with Government Priorities

Alignment with Federal Roles and Responsibilities

Conclusions - Performance

Achievement of Expected Outcomes (Effectiveness)

Demonstration of Economy and Efficiency

Recommendations

1. Evaluation Purpose

2. Program Description

2.1 Program Context

2.2 Program Profile

2.3 Program Logic Model and Narrative

2.4 Program Alignment and Resources

3. Evaluation Description

3.1 Evaluation Scope, Approach and Design

3.2 Limitations and Mitigation Strategies

4. Findings

4.1 Relevance: Issue #1 - Continued Need for the Programs

Suicide

Substance Abuse

4.2 Relevance: Issue #2 - Alignment with Government Priorities

4.2.1 Are the Mental Wellness Programs aligned with Federal government priorities and departmental strategic outcomes?

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

4.3.1 Are the Mental Wellness Programs aligned with federal roles and responsibilities?

4.3.2 Do the Mental Wellness Programs duplicate, complement or overlap with other programs (e.g., federal, provincial, territorial)?

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent have the Mental Wellness Programs made progress towards the achievement of their intended outcomes?

Immediate Outcome #1: Increased and improved collaboration and networking

Immediate Outcome #2: Improved continuum of programs and services in First Nations and Inuit communities

Immediate Outcome #3: Increased participation of First Nations and Inuit individuals, families and communities in programs and services

Immediate outcome #4: Increased awareness of healthy behaviours by First Nations and Inuit individuals

4.4.2 To what extent have the intermediate outcomes been achieved?

Intermediate outcome #1: Increased practice of healthy behaviours by First Nations and Inuit individuals

Intermediate outcome #2: Increased First Nations and Inuit community ownership and capacity to combat substance abuse, suicide and other mental health issues

Intermediate outcome #3: Improved access to quality programs and services for First Nations and Inuit individuals, families and communities

4.4.3 To what extent has the longer term outcome been achieved?

Longer term outcome: Contribution to the improved health status of First Nations and Inuit individuals, families and communities through a strengthened continuum of mental health and addictions programs and services.

4.4.4 Are the Continuum Framework elements driving change at the community level?

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

4.5.1 Has the Mental Wellness program undertaken its activities in the most economical and efficient manner?

5. Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need for the Programs

5.1.2 Alignment with Government Priorities

5.1.3 Alignment with Federal Roles and Responsibilities

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

5.2.2 Demonstration of Economy and Efficiency

6. Recommendations

Appendix 1 - Logic Model

Appendix 2 - Summary of Findings

Appendix 3 - Evaluation Methodology

Evaluation Scope

Evaluation Issues

Data Collection and Analysis Methods

Data Analysis

Additional Detail on Social Return on Investment calculations and analysis

Additional Detail on Evaluation Challenges and Mitigation Strategies

Appendix 4 - References

List of Acronyms

Footnotes

Page details