Evaluation of the First Nations and Inuit Health Branch’s Health Planning and Quality Management Activities - 2010-2011 to 2014-2015

Table of Contents

• Executive Summary
• Management Response and Action Plan (or Management Response)
• 1.0 Evaluation Purpose
• 2.0 Program Description
  • 2.1 Program Context
  • 2.2 Program Profile
  • 2.3 Program Narrative
  • 2.4 Program Alignment and Resources
• 3.0 Evaluation Description
  • 3.1 Evaluation Scope, Approach and Design
  • 3.2 Limitations and Mitigation Strategies
• 4.0 Findings
  • 4.1 Relevance: Issue #1 - Continued Need for the Program
  • 4.2 Relevance: Issue #2 - Alignment with Government Priorities
  • 4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities
  • 4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)
  • 4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency
• 5.0 Conclusions
  • 5.1 Relevance Conclusions
  • 5.2 Performance Conclusions
• 6.0 Recommendations
• Appendix 1 - Funding Models
• Appendix 2 - Logic Model
• Appendix 3 - Evaluation Description
• Appendix 4 - Actual Spending by Program Component
• Appendix 5 - Summary of Findings

List of Tables

• Table 1: Program Resources - Actuals ($)
• Table 2: Limitations and Mitigation Strategies
• Table 3: Funding Models for First Nations Communities 2007-2015

Executive Summary

This Health Planning and Quality Management (HPQM) Activities evaluation reviewed program activities for the period from April 2010 to March 2015. The evaluation was undertaken in fulfillment of the requirements of the Financial Administration Act and the Treasury Board of Canada's Policy on Results (2016).

Evaluation Purpose and Scope

The purpose of the evaluation was to assess the relevance and performance (effectiveness, economy and efficiency) of the HPQM cluster of activities across four components: health planning (governance), accreditation, health consultation/liaison, and health research. The evaluation findings will support decision-making for policy and program improvements.

A number of other program areas in the First Nations and Inuit Health Branch (FNIHB) contribute to increasing health programs and services capacity in First Nations and Inuit communities; however these were out of scope of this evaluation as they will be or have recently been evaluated separately. As well, British Columbia projects were excluded due to the British Columbia Tripartite Framework Agreement on First Nation Health Governance. Further, the funding for the Northern Wellness Agreement self-governance and land claims were excluded. The evaluation scope included Inuit under the health consultation/liaison component only.

The methodology used in the evaluation included key informant interviews with FNIHB national and regional staff, representatives from national Aboriginal organizations, staff from federal departments or agencies with linked mandates, and academic issue experts. Surveys were conducted with FNIHB regional liaison staff and First Nations community health leaders. Case studies examined health planning activities in five First Nations on-reserve communities. Data collection also included document and literature reviews, and financial data and performance data reviews.

The Assembly of First Nations (AFN) and Inuit Tapiriit Kanatami (ITK) were consulted during the scoping phase and for the development of the evaluation methodology and tools, including the survey questionnaires and key informant interview guides. They were also consulted on preliminary findings and the draft final report.

Program Description

The Health Planning and Quality Management program administers contribution agreements and direct departmental spending to support capacity development for First Nations and Inuit communities. Key services supporting program delivery include: the development and delivery of health programs and services through program planning and management; on-going health system improvement via accreditation; the evaluation of health programs; and, support for community development activities.

The program objective is to increase the capacity of First Nations and Inuit to design, manage, evaluate, and deliver health programs and services.

The HPQM cluster includes the following four components:

• Health planning (governance) - supports on-reserve First Nations communities in the planning and development of health services and program delivery models
• Accreditation - promotes and supports the accreditation of on-reserve health services for ongoing health system improvement
• Health consultation/liaison - increases partnerships of national First Nations and Inuit organizations and governments to improve health outcomes
• Health research - enhances knowledge about First Nations health through funding research projects.

The only activities that addressed capacity for the Inuit were under the health consultation/liaison component though the Inuit have access to programming tools.

During the period between April 2010 and March 2015, HPQM expenditures totalled approximately $642 million. Of this amount, approximately 80% was expended through contribution agreements for the health planning (governance) in on-reserve First Nations communities, 4% was for contribution agreements for accreditation activities in on-reserve First Nations communities, and about 13% was the combined amount for contracts for health research and a contribution agreement each with AFN and ITK for health consultation/liaison.

Conclusions - relevance

Continued Need

There is a continued need for effective capacity building to reduce health inequalities in First Nations and Inuit communities. Greater ownership and control of health programs and services by First Nations and Inuit organizations and communities can lead to better health outcomes. Increased capacity and control can have a positive impact on First Nations and Inuit community health programs and services in four key areas, in turn leading to enhanced health outcomes. These four areas are: social determinants of health, cultural competency and safety, decentralization, and quality of services.

Alignment with Government Priorities

The HPQM priorities align with the Government of Canada and departmental current priorities and strategic outcomes. Various Government of Canada commitments support health programs and services by collaborating with First Nations and Inuit communities to ensure that health programs and services are accessible, reliable and of quality. Health Canada's departmental objectives and the FNIHB strategic plan support First Nations and Inuit self-determination, community-level action and capacity building, and building stronger stakeholder connections.

Alignment with Federal Roles and Responsibilities

Work in the area of capacity building for First Nations and Inuit organizations and communities is consistent with federal and Health Canada responsibilities outlined in various policies (e.g., Indian Health Policy, Indian Health Transfer Policy) aimed to increase First Nations and Inuit community control and responsibility for health service delivery. There is complementarity across federal-level departments with programs that support capacity building; however, there are opportunities for enhanced coordination of community planning activities.

Conclusions - performance

Achievement of Expected Outcomes (Effectiveness)

Health consultation/liaison and health research activities have led to enhanced ownership and control for First Nations and Inuit organizations. Through contribution funding and engagement agreements with both AFN and ITK, there have been enhanced opportunities for national and regional First Nations and Inuit organizations to participate in the design and development of health programs and policies. FNIHB has developed strategic partnerships with federal and First Nations organizations to support various First Nations health related research and surveillance initiatives. The availability and use of evidence-based information has been enhanced. This information has been used to support decision-making both within FNIHB and by First Nations organizations. FNIHB national office and regional staff key informants indicated that further knowledge translation within FNIHB may enhance the use of evidence-based information in policy and program decision-making.

HPQM activities have positively impacted First Nation community capacity, which in turn has enhanced the quality of health programs and services in First Nations communities. There have been enhanced opportunities to exercise and build community ownership and control. The development of health plans and the accreditation process have effectively built capacity in those communities with existing leadership and organizational abilities. First Nations communities with varying levels of capacity require different approaches to maximize their opportunities to exercise ownership and control. Strengthened processes and tools in communities with lower initial levels of capacity are needed - in particular remote and isolated communities, smaller communities, as well as communities in crisis.

Demonstration of Economy and Efficiency

Some HPQM component activities have effectively built upon the expertise and capacity of First Nations organizations to enhance community ownership and control, including leveraging existing First Nations community capacity to undertake the accreditation process and working through the First Nations Information Governance Centre (FNIGC) to support Regional Health Survey activities. A number of program efficiencies have been demonstrated through the use of funding formulas and work to enhance FNIHB cultural competency.

Efficiencies could be gained by further enhancing collaboration internally across FNIHB First Nation community-based programs and to some degree with other federal government departments. There are opportunities to enhance working relationships through sharing best practices, better integrating processes and tools, and supporting cultural competency.

A refined performance data collection approach may enhance how FNIHB is able to measure increased capacity and control and the quality of service being delivered.

Recommendations

Recommendation 1

Strengthen processes and tools to improve opportunities for all First Nations communities to develop the capacity to exercise ownership and control, particularly with those First Nations communities with lower initial levels of capacity.

First Nations communities with varying levels of capacity require different approaches to maximize their opportunities to exercise ownership and control. Challenges in terms of capacity are more likely to be found in remote and isolated communities, smaller communities, as well as communities in crisis. These challenges include: limited strategic planning underway, limited access to health planning funding, and difficulty attracting and retaining human resources in many remote and isolated communities. Tailored support to meet these unique capacity needs might include: additional training or tools, leveraging networks (mentoring) and best practices from other communities, partnering with other regional stakeholders, and ongoing community engagement with FNIHB regional liaison staff. A health planning guide was recently launched by FNIHB in order to assist those communities that would benefit most from greater capacity.

Recommendation 2

Enhance information exchange and collaboration on health planning and quality management activities across FNIHB programs and regional offices.

Enhanced collaboration across FNIHB would lead to a more integrated and culturally competent approach to First Nations and Inuit health service delivery across all regions and all community-based programs. For examples, there are opportunities to: integrate accreditation standards into other program guidance documents that support capacity in community programs, such as mental wellness and healthy living; undertake sharing of best practices on capacity building across the Branch, in particular at the working level and across regional offices; and continue to enhance the cultural competency of staff in various capacity building roles.

Recommendation 3

Update HPQM performance measurement strategy to establish methods to measure increased capacity and control and the quality of service being delivered.

An updated performance measurement strategy is needed to establish methods to measure increased capacity and control as well as the quality of services being delivered. Strategies to track and analyse immediate and intermediate outcome indicators (in particular related to ownership and control and the quality of health programs and services) are required.

Management Response and Action Plan (or Management Response)

Evaluation of the First Nations and Inuit Health Branch's Health Planning and Quality Management 2010-2011 to 2014-2015

1.0 Evaluation Purpose

The purpose of the evaluation was to assess the relevance and performance of the First Nations and Inuit Health Branch (FNIHB) health planning and quality management (HPQM) activities, including the following components of this cluster: health planning (governance), accreditation, health consultation/liaison, and health research. The period covered by the evaluation was April 2010 to March 2015.

The evaluation was undertaken in fulfillment of the Financial Administration Act (FAA) and the Treasury Board of Canada's Policy on Results (2016), and was conducted by the Health Canada and the Public Health Agency of Canada's (the Agency) Office of Audit and Evaluation in accordance with the Five-Year Evaluation Plan 2015-2016 to 2019-2020. The evaluation will support Health Canada's Deputy Minister and senior management in decision-making for policy and program improvements.

2.0 Program Description

2.1 Program Context

Over the last three decades, federal government policy and programs have supported increasing the responsibility and control of First Nations and Inuit communities over their own health care. In 1979, the Government of Canada introduced its Indian Health Policy with a stated goal "…to achieve an increasing level of health in Indian communities, generated and maintained by the Indian communities themselves." Almost decade later, in 1988, the Health Services Transfer Authority "…facilitated the transfer of resources of Indian programs south of the 60^th parallel to Indian control."

FNIHB funding authorities were further consolidated in 2005 and again in 2011 to reflect program complementarities in achieving common objectives (as also reflected in FNIHB's revised Program Alignment Architecture or PAA) and to obtain new flexibilities to better accommodate the delivery of health programs and services in First Nations and Inuit context. These renewed funding authorities included the introduction of new funding models to allow FNIHB to reduce the reporting burden on recipients, and to align contribution agreements with the capacity of the recipient to design, manage and monitor their programs and services to address the needs of their population. (See Appendix 1 for more detail on funding models.)

An evaluation was completed in March 2012 covering health planning (governance) activities conducted between April 2005 and March 2010. The synthesis evaluation found that generally the capacity of First Nations communities in managing and delivering health programs and services had been supported by the program activities.

2.2 Program Profile

The Health Planning and Quality Management program administers contribution agreements and direct departmental spending to support capacity development for First Nations and Inuit communities. Key services supporting program delivery include: the development and delivery of health programs and services through program planning and management; on-going health system improvement via accreditation; the evaluation of health programs; and, support for community development activities. The program objective is to increase the capacity of First Nations and Inuit to design, manage, evaluate, and deliver health programs and services.

FNIHB's health planning and quality management activities are managed by three Directorates as well as the office of the Senior Assistant Deputy Minister (health consultation/liaison). Directorates include: Capacity, Infrastructure and Accountability under the Regional Operations Assistant Deputy Minister (health planning), Primary Health Care under the Senior Assistant Deputy Minister (accreditation), and Strategic Planning, and Policy and Information under the Senior Assistant Deputy Minister (health research).

In close collaboration with Regional Operations, the National Office in Ottawa oversees all HPQM program components. The National Office solely manages the delivery of the health consultation/ liaison and health research components. There are six Regional Offices (Atlantic, Quebec, Ontario, Manitoba, Saskatchewan, and Alberta) responsible for delivering the health planning and accreditation funding through contribution agreements with First Nations community organizations. Each regional office tailors its approach to delivering these components to reflect unique regional contexts.

A brief overview of the four program components and the target audiences for capacity building activities is provided below.

Health Planning (Governance)

The health planning (governance) component supports on-reserve First Nations communities in the planning and development of health services and program delivery models. This component supports recipients in the establishment of a strong, effective and sustainable health planning, administration (governance) and delivery infrastructure at the community level. An established health plan and health infrastructure are two critical conditions for First Nations communities to access the more flexible Block funding model.

Accreditation

The accreditation component promotes and supports the accreditation of on-reserve First Nations health programs and services for ongoing health system improvement. Contribution funding assists First Nations organizations to engage in the accreditation process and uses standards of excellence related to sustainable governance, effective organization, service excellence and positive client experience. The accreditation process supports the full involvement of First Nations health services organizations with community leadership, educational services, provincial health services, medical professionals and community members. FNIHB does not accredit health services but rather supports organizations that are engaged in the process with an accrediting body.

Health Consultation/Liaison

The health consultation/liaison component supports consultation and partnerships between National Aboriginal Organizations (NAOs) and Health Canada to improve health outcomes. Through contribution agreements, with both the Assembly of First Nations (AFN) and Inuit Tapiriit Kanatami (ITK), and engagement protocols, the intent is to establish and maintain productive lines of communication and exchanges of policy, research, evaluation and program delivery information between these national partners and regional stakeholders.

Health Research

Terminated in 2011-2012, the former Aboriginal Health Research and Coordination Projects program enhanced knowledge about First Nations health through funding research projects. Since then, through partnerships with other stakeholders, other research activities of the current Health Information and Policy Coordination Unit have continued to be implemented. These remaining activities support: the improvement of the quality and quantity of First Nations health data, research, and information; the development, advancement, distribution and knowledge translation of health information; and the capacity of First Nations and Inuit to generate and access Indigenous health information.

2.3 Program Narrative

According to Health Canada's 2014-2015 Performance Measurement Framework (PMF), the expected result of the health planning and quality management activities is "to increase the capacity of First Nations and Inuit to design, manage, evaluate and deliver health programs and services."

The connection between the activity areas and the expected outcomes is depicted in the logic model (see Appendix 2). The activity areas, outputs, and immediate and intermediate outcomes for achieving this long-term outcome are organized around five theme areas of: service provision; capacity building; stakeholder engagement and collaboration; data collection, analysis and surveillance; and policy development and knowledge sharing. Each theme area has related outputs and an identified audience that each output is expected to reach, which are then expected to lead to immediate outcomes. For the purposes of this evaluation, some similar outcomes were collapsed to optimize the cogency of the evaluation questions for the evaluation matrix.

Expected immediate outcomes include the following:

• Supporting sustained harmonized and collaborative policy approaches
• Increased collaboration in program and services planning
• Enhanced First Nation and Inuit opportunity to participate in, and influence design and development of, programs and policies
• Increased awareness by health leaders and care providers of quality planning and management activities

The following intermediate outcomes should then flow from immediate outcomes:

• Evidence-based information to support program and policy decisions
• Increased First Nation and Inuit ownership of health planning and quality improvement strategies based on culturally relevant standards
• Improved quality and delivery of programs and services

The long term expected outcome for the program was improved First Nations and Inuit capacity to influence and/or control (design, deliver, and manage) programs and services.

The evaluation assessed the achievement of the expected outcomes and whether there were any challenges and/or barriers to achieving the expected outcomes over the evaluation time frame.

2.4 Program Alignment and Resources

FNIHB's HPQM activities are part of the PAA 3.3: Health Infrastructure Support for First Nations and Inuit, sub-activity 3.3.1 First Nations and Inuit Health Systems Capacity, sub-sub-activity 3.3.1.1 Health Planning and Quality Management.

The program components' financial data for the fiscal years 2010-2011 through 2014-2015 are presented below (Table 1). Overall, the four program components had combined actual expenditures of approximately $641,817,566 over the five years.

3.0 Evaluation Description

3.1 Evaluation Scope, Approach and Design

The scope of the evaluation covered the period from April 2010 to March 2015 and included health planning and quality management activities within FNIHB. Considering the long-standing nature and materiality of the First Nations community-based programs in particular, the focus was on the health planning (governance) component and its related capacity building activities. The evaluation also included the following related capacity building activities: accreditation, health consultation/liaison, and health research.

A number of other program areas in the Branch contribute to increasing health programs and services capacity in First Nations and Inuit communities; however these were out of scope of this evaluation as they will be or have recently been evaluated separately. These programs include: Health Services Integration Fund, Human Resources and e-Health. As well, British Columbia projects were excluded due to the British Columbia Tripartite Framework Agreement on First Nation Health Governance (October 1, 2013), which will be evaluated through a separate initiative.

The evaluation scope included Inuit under the health consultation/liaison component only. This component includes contribution funding to ITK and the ITK Health Approach agreement with FNIHB. The other components (including health planning, accreditation, and health research) did not provide funding to Inuit organizations or communities between 2010 and 2015. Further, the funding for the Northern Wellness Agreement, self-governance and land claims were excluded because teasing out the federal role is challenging given the leadership role of the territorial and Indigenous governments in managing resources.

The evaluation issues covered were aligned with the Treasury Board of Canada's Policy on Evaluation (2009) and considered the five core issues under the two themes of relevance and performance, as shown in Appendix 3.^{Footnote i} Corresponding to each of the core issues, specific questions were developed based on program considerations and these guided the evaluation process.

An outcome-based evaluation approach was used for the conduct of the evaluation to assess the progress made towards the achievement of the expected outcomes, whether there were any unintended consequences and what lessons were learned. The Treasury Board's Policy on Evaluation (2009) also guided the identification of the evaluation design and data collection methods so that the evaluation would meet the objectives and requirements of the policy. A non-experimental design was used based on the evaluation plan, which detailed the evaluation strategy for this program and provided consistency in the collection of data to support the evaluation. As a non-experimental design, the evaluation relied on correlation to demonstrate effect, without implying causation. As such, the evaluation has been designed to demonstrate the likely contributions of the programs to the expected outcomes, rather than demonstrate direct causal links between the programs and outcomes.

The Assembly of First Nations and Inuit Tapiriit Kanatami were consulted during the scoping phase and for the development of the evaluation methodology and tools, including the survey questionnaires and key informant interview guides. They were also consulted on preliminary findings and the draft final report.

Data collection started in April 2015 and concluded in November 2015. Data for the evaluation was collected using various methods, which were: key internal and external informant interviews, surveys, case studies, document review, literature review, financial data review and performance data review. Performance measurement data collected by the program was analyzed. More specific detail on the data collection and analysis methods is provided in Appendix 3. In addition, data were analyzed by triangulating information gathered from the different methods listed above. The use of multiple lines of evidence and triangulation were intended to increase the reliability and credibility of the evaluation findings and conclusions.

3.2 Limitations and Mitigation Strategies

Most evaluations face constraints that may have implications for the validity and reliability of evaluation findings and conclusions. The following table outlines the limitations encountered during the implementation of the selected methods for this evaluation. Also noted are the mitigation strategies put in place to ensure that the evaluation findings can be used with confidence to guide program planning and decision-making.

4.0 Findings

4.1 Relevance: Issue #1 - Continued Need for the Program

With a view to improving health outcomes, there is a continuing need to increase capacity and control for First Nations and Inuit to design and deliver health programs and services in their communities.

The health status of many First Nations and Inuit peoples in Canada falls below that of the general Canadian population. Obesity in Canada remains higher in Indigenous populations compared with non-Indigenous populations.^{Footnote 1} In 2011, the incidence rate of active tuberculosis disease reported for the Canadian-born Indigenous population was 34 times higher than for the Canadian-born non-Indigenous population.^{Footnote 2} First Nations populations' living on-reserve experience between 2% and 14% higher rates of chronic conditions (asthma, arthritis, diabetes, high blood pressure, or multi chronic conditions) than the general Canadian population.^{Footnote 3}

Health disparities are linked to First Nations and Inuit capacity to lead and deliver health programs and services. Literature suggests that there is the potential for capacity building to improve health programs and services and that there is need for support in this area.^{Footnote 4} Research has demonstrated that initiatives that are developed, led, and managed by First Nations and Inuit have the greatest potential for success in bringing about quality improvements in health care systems.^{Footnote 5} As First Nations and Inuit assume responsibility for their health programs and services, there continues to be a need for governments to play a role in facilitating and strengthening the First Nations and Inuit capacity to build and sustain the quality of their health programs and services.^{Footnote 6}

The relationship between improving health outcomes through increased First Nations and Inuit capacity and control is complex. There are a number of considerations which may influence First Nations and Inuit capacity to deliver quality health care programs and services, in turn impacting health outcomes for First Nations and Inuit. Geographic, cultural and social considerations present unique challenges for these health programs and services which require unique, culturally appropriate solutions that are developed and implemented by First Nations and Inuit.^{Footnote 7}

A number of lines of evidence, including a review of the literature on ownership and control leading to better health outcomes, suggest that increased First Nations and Inuit capacity and control can have a positive impact on community health programs and services in four key intermediary areas (or conditions), in turn leading to enhanced health outcomes. As highlighted in Figure 1, the intermediate areas that are necessary to improve health outcomes include: social determinants of health, cultural competency and safety, decentralization, and quality of services. These are the areas much of the literature assumes are impacted as a result of increased self-determination in First Nations and Inuit health programs and services. To varying degrees, the HPQM components reviewed for this evaluation strive to improve health outcomes by increasing the capacity of First Nations and Inuit to have more ownership and control of health programs and services across each of these four intermediary areas.

Social Determinants of Health

Health research shows that poorer health outcomes are due in part to disparities between First Nations and Inuit and the general population in terms of poorer social determinants of health within communities. Health disparities are associated with social, economic, cultural and political inequities that affect individuals, communities and populations.^{Footnote 8} Among a myriad of factors, communities range in size, level of remoteness, socioeconomic stability, infrastructure, overcrowded and substandard housing, issues with waste water disposal and water pot-ability, internet availability, food shortages, and lack of employment opportunities, all of which may influence the quality of health care services. As stated in the joint Health Canada and Indigenous and Northern Affairs Canada (INAC) Community Development and Capacity Building Framework, for communities in crisis, there needs to be a focus on working with other partners to stabilize essential services and build individual capacities, without over burdening with paperwork.^{Footnote 9}

The health planning and accreditation components of the HPQM cluster, in particular, administer their activities with a view to addressing social determinants of health. For example, with the range of flexibility in the funding models and requirements for health planning contributions, there is a differentiated capacity building approach aligned with First Nations community needs. Both health planning and accreditation components are seeking to develop approaches to better address First Nations community needs across the continuum of capacity, with a focus on building on the strengths of communities with lower initial levels of capacity.

Cultural Competency and Safety

In Canada, definitions of culturally competent and safe health care will vary depending on unique individual community characteristics and the specific First Nations and Inuit group. In general, the literature has shown that traditional language knowledge is considered a strong proxy of cultural continuity and capacity. For example, increased language knowledge in Canadian First Nations and Inuit communities was associated with decreased rates of suicide,^{Footnote 10} as well as lowered prevalence of diabetes.^{Footnote 11} Past negative experience with discrimination or racism, unfamiliar language and culture, as well as geographic, financial and other barriers can decrease First Nations and Inuit utilization of health programs and services. Therefore, cultural competency and safety is also related to improving utilization rates^{Footnote 12} and effectiveness of health programs and services.^{Footnote 13}

In terms of HPQM component activities, cultural competency is built into the program design of various program components. For example, the FNIHB health research program supports the First Nations Regional Health Survey (RHS). This national survey is grounded in First Nations Principles of OCAP (ownership, control, access, and possession) which means that First Nations control data collection processes in their communities. The RHS Cultural Framework assists "…in achieving a culturally informed interpretation process that can be presented back to communities in a way that is usable and that reinforces their ways of seeing, relating, knowing and being."^{Footnote 14}

Decentralization

Decentralization is the concept of shifting health care control to the community, along with divesting more flexibility and control to Health Canada's regional offices.^{Footnote 15} Decentralization allows First Nations and Inuit organizations and communities to ensure their priorities are reflected in the health care decision they make.^{Footnote 16} This concept in integral to the objectives of all HPQM components. Key informant interviews with FNIHB national office and regional staff have indicated that federal health transfer policies have facilitated the ability of some First Nations and Inuit organizations and communities to design health programs, and establish services and allocate funds according to community health priorities. Key informant interviews with FNIHB staff and evaluation surveys (FNIHB regional liaison staff, First Nations community health leaders) also indicate that there is variety in the level of First Nations and Inuit community control exhibited across Canada, depending on the level of community capacity.

The quality of health care in First Nations and Inuit communities may be affected by the capacity and the stability of the health leadership and staff within communities. One of the principles of the Community Development and Capacity Building Framework is leadership in the community - it states that "leaders build hope for the future, and are essential to lead community development."^{Footnote 17} Key informant interviews and surveys conducted for this evaluation (with FNIHB regional liaison staff and First Nations community leaders) suggest that stable and capable First Nations and Inuit community health leadership is a critical factor for community development of health services and program capacity. It was also suggested that high turnover rates for both leadership and staff are common and the isolation of First Nations and Inuit communities is sometimes an issue in attracting and maintaining capable staff.

Quality of Services

In remote First Nations and Inuit communities, access to primary care is a major issue impacting the health of residents, particularly in rural or isolated communities.^{Footnote 18} For example, travel costs to access basic services and shipping of medical supplies still present hurdles to providing quality health care in these communities. Key informant interviews with FNIHB national office staff indicated that one of the ways health transfer policies and programs have tried to address these issues is by building First Nations and Inuit community capacity to hire health workers from within the community, as well as increasing salary equity for health care workers to improve retention and recruitment of staff.

Another health care standard important to First Nations and Inuit health is continuity of care. Continuity of care is the cogency with which health service providers at different levels (primary, secondary, tertiary) communicate with one another about shared patients. The remote nature of many First Nations and Inuit communities results in poor continuity of care due to a lack of a consistently accessible doctor, paired with limitations in nearby specialists. This situation creates issues with follow-up between care providers and with accessibility. Although electronic and tele medicine in Canada have helped in improving continuity of care in rural locations, issues still remain. The association between a lack of access to care providers, and continuity between those primary and tertiary health care systems, can lead to poor chronic care outcomes. ^{Footnote 19}

Linked to continuity of care is the better integration of First Nations and Inuit health programs and services with the provincial/territorial health care systems which will improve the access and continuity of care received by First Nations and Inuit individuals. ^{Footnote 20} Internal and external key informant interviewees and evaluation surveys (FNIHB regional liaison staff, First Nations community health leaders) suggest that the level of involvement and coordination of other First Nations partners varies across the country and has an effect on the capacity of communities and the quality of health program and services. Regions across Canada vary in terms of the level of involvement and capacity of regional partners. The number of First Nations organizations varies among regions and their varying capacities impact the level and nature of their involvement in communities in terms of building health capacity, therefore improving health outcomes.

Enhancing the quality of health programs and services is a desired outcome of all HPQM components. Impacting the quality of First Nations community health programs and services is a direct objective of community health planning and accreditation activities by embedding quality improvement activities into health programs and services. The quality of health programs and services is also indirectly linked through national and regional consultation/liaison activities (Inuit and First Nations) and enhancing the availability of evidence-based information through health research (First Nations).

4.2 Relevance: Issue #2 - Alignment with Government Priorities

Health planning and quality management activities align with Government of Canada and departmental priorities and strategic outcomes of increasing the capacity of First Nations and Inuit to manage health programs and services and to address health status inequalities affecting communities.

Numerous Speeches from the Throne over the last five years have highlighted the Government of Canada's commitment to recognizing the rights of First Nations and Inuit and the continued development and improvement of communities. The 2011 Speech recognizes that First Nations and Inuit are central to Canada's history and contains a call to action to address barriers to social and economic participation. The 2013 Speech committed to continue dialogue on treaty relationships/land claims and to continue to work with First Nations and Inuit to create healthy, prosperous, and self-sufficient communities. Going forward, the current Government through both the Mandate Letter to the Minister of Health and the 2015 Speech from the Throne has committed to renewed, nation-to-nation relationship with the First Nations and Inuit based on rights, respect, cooperation and partnership.

Through various Budget speeches, the Government of Canada has also made financial commitments to supporting health programs and services to First Nations and Inuit communities. A financial commitment to capacity development at the community level was also established. The 2010 Budget committed to assist First Nations and Inuit communities, including the support of child and family services. The 2011 Budget committed to working through First Nations Land Management Act regimes to allow First Nations to become more self-sufficient and have increased decision-making and management capacity over reserve lands. The 2013 Budget committed to assistance to First Nations and Inuit communities become more self-sufficient and ensures that health care services are accessible, reliable, and of quality for all communities and families across Canada. It also committed to helping these communities prosper. The 2016 Budget highlighted that strong families and communities are fundamental to the success of Indigenous peoples. The Government will work in partnership with First Nations and Inuit to break down the barriers that have for too long held back individuals and communities from reaching their full potential.

The outcomes for this program cluster of activities align with Health Canada's Strategic Outcome #3 as detailed in the 2015-2016 Health Canada Program Alignment Architecture: "First Nations and Inuit communities and individuals receive health services and benefits that are responsive to their needs so as to improve their health status."

Capacity building for health programs and services aligns with the Health Canada strategic outcomes as stated in recent departmental reports on plans and priorities. The 2014-2015 Report on Plans and Priorities states, "…The program objective is to improve the delivery of health programs and services to First Nations and Inuit by enhancing First Nations and Inuit capacity to plan and manage their programs and infrastructure." The 2011-2012 Report on Plans and Priorities states that this cluster of activities "…underpins the long-term vision of an integrated health system with greater First Nations and Inuit control by enhancing the capacity of First Nations and Inuit to design, manage, deliver and evaluate quality health programs and services."

At the Branch level, many of the objectives outlined in the First Nations and Inuit Health Strategic Plan (2012) are reflected in the activities of this health planning and quality management program cluster.

• Health planning and accreditation activities are reflected in the objective to "support First Nations and Inuit in their aim to influence, manage and/or control health programs and services that affect them." These last activities specifically address the need to "…advance aggregated models of health services governance led by First Nations with appropriate capacity to support groups of communities in the management and delivery of programs and services, …and support First Nations and Inuit to achieve strong governance models for health programs and services."
• Health consultation/liaison activities are linked to the stated objective to "…strengthen mechanisms for First Nations and Inuit representatives to influence decision-making at the national and regional senior management levels."
• Health research activities are directly linked to FNIHB objective to "…improve availability of, and access to, high quality data for better decisions from planning to point of care."

4.3 Relevance: Issue #3 - Alignment with Federal Roles and Responsibilities

The federal government's role in First Nations and Inuit community capacity building has been established through multiple policies, including the federal Indian Health Policy and the Indian Health Transfer Policy.

In Canada, the delivery and support of health care is a shared responsibility between federal and provincial governments. Provinces and territories provide universal insured health programs and services (physician and hospital services) to all residents, including Indigenous peoples. The federal government's role, with respect to the provision and support of health care services for First Nations and Inuit, is based on government policy and involves supplementing programs and services provided by provincial and territorial governments.

Although there is no statutory framework for the provision of health care programs and services to First Nations and Inuit by the federal government, the following policies do outline the goals of the federal government with respect to Indigenous health.

• Consistent with the Indian Health Policy (1979), Health Canada through FNIHB provides or funds health programs and services to First Nations and Inuit. This policy outlines a federal responsibility for the fostering capacity leading to ownership and control for First Nations and Inuit communities with the stated goal "…to achieve an increasing level of health in Indian communities, generated and maintained by Indian communities themselves." This goal is consistent with this program cluster's desired long-term outcome of improved First Nations and Inuit capacity to influence and control (design, deliver and manage) quality health programs and services.
• The Indian Health Transfer Policy (1988) has "…facilitated the transfer of resources of Indian programs south of the 60^th parallel to Indian control." It provides a framework for the assumption of control of health programs and services by First Nations and Inuit and set forth a developmental approach to health funding transfer centred on the concept of self-determination in health. It reflects the federal responsibility to build community capacity to support First Nations and Inuit communities in all aspects of health service programming including planning, decision-making, delivery, and potential control of health programs and services. Subsequent funding authorities (2005, 2011) have included the introduction of new funding models to allow FNIHB to reduce reporting burden on recipients, and align contribution agreements with the capacity of recipients to design, manage and monitor their programs and services to address the needs of their own communities.

While there is no duplication, there are opportunities for greater coordination and integration at the federal level.

At the federal level, there is complementarity among the roles of Government of Canada departments that support capacity building in First Nations and Inuit communities.

• INAC addresses social determinants of health on and off reserve as they relate to their broader scope for community development including: land management for economic development, development of community plans, supporting Aboriginal Financial Institutions (AFIs) for business development, community planning, and infrastructure improvement. INAC also leads the Aboriginal Peoples Survey (APS) in collaboration with its partners. There are agreements and Memoranda of Understandings (MoUs) between Health Canada and INAC at the Ministerial, Assistant Deputy Minister (ADM) and Director General (DG) levels that reflects collaboration across these two departments.
• Public Safety Canada supports emergency planning to maintain a strong and safe community so that the other health promotion elements (e.g., health planning, health service delivery/quality improvements) can be supported. The Royal Canadian Mounted Police's (RCMP's) public safety role is similar in its provision of direct policing services, particularly in rural and remote First Nations and Inuit communities, and crisis response functions (including suicide monitoring).
• The Canadian Institutes of Health Research (CIHR) have a mandate to support culturally relevant research on Indigenous peoples health. Health Canada supports the CIHR, in particular by providing advice and direction on the Pathways to Health Equity for Aboriginal Peoples project. This project aims at understanding how to implement multilevel and scalable interventions to reduce health inequities for Indigenous peoples.
• Other departments that have complementary roles to that of Health Canada include: the Canadian Mortgage and Housing Canada (housing), Employment and Social Development Canada (employment and training), and Statistics Canada (survey data and analyses, specifically through the Social and Aboriginal Statistics Division).

While it appears that the mandates of federal departments and their programs are distinct, there is still a need for a more holistic approach across departments to support capacity building in First Nations and Inuit communities. Key informant interviewees with the federal government and NAOs, and survey respondents (FNIHB regional liaison staff, First Nations community health leaders), indicated that there is a need for clearer communication, coordination and collaboration among federal departments.

4.4 Performance: Issue #4 - Achievement of Expected Outcomes (Effectiveness)

4.4.1 To what extent has evidence-based information to support program and policy decisions been made more accessible?

The FNIHB Surveillance Health Information Policy and Coordination Unit has contributed to the availability and use of evidence-based information on First Nations health.

The Surveillance Health Information Policy and Coordination Unit (SHIPCU) is the focal point for the development and dissemination of evidence-based information within FNIHB. While the Unit does not undertake primary research activities in-house, in particular since the termination of the Aboriginal Health Research and Coordination Projects funding in 2011-2012, it facilitates the generation of evidence-based information through effectively leveraging and influencing the research agendas and initiatives of other federal government organizations and partnering with First Nations organizations.

• In terms of other federal collaborations, the Unit works closely with the CIHR, in particular the Institute for Aboriginal Peoples Health. It participates in advisory meetings for the "Pathways to Health Equity for Aboriginal Peoples" initiative which is designed to increase the understanding of the implementation of multilevel and scalable interventions for Indigenous peoples. These interventions help to address issues such as tuberculosis, diabetes, mental health and suicide, and children's oral health. It collaborates with Public Health Agency of Canada to support their work on surveillance plans for communicable diseases on-reserve. The Unit also liaises with and supports INAC and Statistics Canada with respect to the health supplement of the Aboriginal Peoples Survey and on-reserve data collection.
• A significant research partnership over the last number of years has been with the First Nations Information Governance Centre (FNIGC). Through this partnership, the Surveillance Health Information Policy and Coordination Unit has been a primary financial contributor, advisor and promoter for the First Nations Regional Health Survey (RHS) - the only First Nations governed national health survey in Canada. Incorporated in 2012, the mission of the FNIGC is to build capacity and provide credible and relevant information on First Nations using the highest standards of data research practices, while respecting the rights of First Nations self-determination for research and information management. The RHS collects information about on-reserve and northern First Nations communities based on both Western and traditional understandings of health and well-being.

Evidence-based research and analyses that has been leveraged and generated by the Unit helps to support management decision-making and reporting within FNIHB. Internal FNIHB key informant interviews indicated that the Unit provides evidence-based information for senior management briefings and presentations within FNIHB, including RHS information, to support: program management (including some outcome tracking), corporate planning and reporting, and program funding renewal.

Through FNIHB, support to the Regional Health Survey has been enhanced.

The Assembly of First Nations Chiefs in Assembly, the Chiefs Committee on Health (CCOH) and First Nations Regions across the country have mandated the FNIGC to provide oversight and governance over the RHS at the national level. Activities include preparing reports, serving as the data steward, and engaging in partnerships. Ten independent regional partners coordinate the RHS in their respective regions. The FNIGC and regional partners collaborate on collective issues as well as share ideas and knowledge. The FNIGC is mandated and authorized to report on national level statistics while each regional partner is completely independent and responsible for their own respective regional databases and reporting. The FNIGC and regional partners have completed and published national and regional reports for two first two RHS phases (2002-2003, 2008-2010). The data collection for the third phase began in March 2015.

The RHS process has been a factor in the creation of the First Nations Principles of OCAP (Ownership, Control, Access and Possession). This approach is unlike other Canadian survey and research approach as the data produced through the RHS is the property of First Nations peoples. The RHS has led the way for First Nations to exercise jurisdiction over their research information and is the first national survey implemented explicitly to respect the First Nations Principles of OCAP. The RHS also follows a special code of research ethics and cultural framework which allow First Nations perspectives to be respected in the process. Internal and external key informants indicated that these principles have helped to build trust with First Nations communities and create greater buy-in to research and data collection processes as a whole.

The RHS information is used at both the national and regional levels. External key informants indicated that at the national level the Assembly of First Nations uses RHS information to make resolutions, which are important in setting the agenda of the organization. The FNIGC liaises regularly with the AFN about common data and research needs. The RHS information is used to complement regional data collection efforts to create a more fulsome picture of First Nation health within regions. It has helped to support First Nations managed regional data hubs where this data is stored and reports are published. For example, in the Quebec region, La Commission de la Santé et des Services Sociaux des Première Nations du Québec et du Labrador houses the Quebec RHS information and publishes the Quebec RHS report. According to key informant interviewees with FNIHB Quebec regional staff, these efforts have helped to develop capacity for data analysis and dissemination at the regional level for First Nations organizations in Quebec.

Further knowledge translation would increase use within FNIHB and externally at the regional and community levels.

FNIHB national office and regional staff key informants indicated that further knowledge translation within FNIHB may enhance the use of evidence-based information in policy and program decision-making.^{Footnote ii} Further uses for knowledge translation that were suggested included: corporate reporting, informing and targeting programming, informing and validating interventions, and publications. It was suggested that the level and quality of use of data could be improved with more dedicated resources to analyze data for policy purposes.

FNIHB national office and regional staff, and representatives from NAOs including the FNIGC, related that the regional capacity for information collection, analysis and dissemination vary greatly between regional hubs. Results from both surveys (FNIHB regional liaison staff and First Nations community health leaders) suggest that, at both the regional and community levels, there are gaps in knowledge about the availability and accessibility of data and information that has been generated. The survey respondents also suggested that the majority of First Nations communities are unaware that there are published studies available or data and statistics collected that provides insight on health needs. A variety of key informants suggested that when provided with this material, even fewer First Nations communities have the capacity to use the information in a practical application for their health plans or within the health programs themselves.

4.4.2 Have opportunities for First Nation and Inuit to participate in, and influence design and development of, programs and policies been enhanced? In what ways?

Progress has been made in strengthening the mechanisms available to First Nations and Inuit representatives to influence decision-making at the national and regional senior management levels.

Contribution agreements with both the AFN and ITK were in place to increase opportunities for First Nations and Inuit to participate in and influence national health policy, health systems and programs/services development in all health related areas. Annual reports and evaluation reports completed by both national Aboriginal organizations reflect that these agreements led to an increased capacity to plan and conduct work around First Nations and Inuit specific health priorities over the last five years. The flexible and multi-year nature of these Contribution Agreements was associated with many positive process improvements, including: the ability to respond to region-driven priorities, the ability to engage in long-term planning, the development of long-term partnerships, and the ability to address emerging priorities. For example, ITK Annual Report 2014-15 states that representatives of the FNIHB have done extensive work outlining health data priorities and have consistently communicated and coordinated these priorities to align with ITK's interests to the extent possible.

A significant development took place in 2014 when FNIHB senior management and two national First Nations and Inuit representative organizations formalized engagement protocols.

• The first of these agreements is the Assembly of First Nations Engagement Protocol. The Engagement Protocol was created to strengthen the collaborative relationship between AFN and FNIHB and is the result of years working together to establish a culture of transparency and reciprocal accountability. The Engagement Protocol outlines how AFN and FNIHB will engage at national and regional levels and how the two groups will work together to ensure First Nations regional representatives and communities are engaged in the advancement of the First Nations and Inuit Health Strategic Plan. Additional objectives of the Protocol include to: map out how the First Nations and Inuit Health Strategic Plan and the AFN First Nations Health Foundational Plan complement each other; respect existing and ongoing national, regional and community-level collaboration; and provide a map for a process of FNIHB engagement with the AFN, as well as engagement with other partners of mutual interest.
• The second agreement is the Inuit Tapirit Kanatami Health Approach. The Health Approach is a commitment for FNIHB to work with ITK to develop an approach to Inuit health that informs planning within Health Canada. It seeks compatibility between the First Nations and Inuit Health Strategic Plan and the National Inuit Committee on Health (NICoH) Strategic Framework. The specific objectives of the Health Approach are to: promote awareness among all FNIHB employees of the unique features of Inuit health, and contexts of the four Inuit regions and ensure that the Inuit Health Approach is a cornerstone of the Branch's policy and program frameworks; clarify the appropriate engagement process of FNIHB with Inuit at the national and regional levels; and ensure that FNIHB respects the principles and processes established in the Land Claim Agreements in its relationship with Inuit.

In terms of providing opportunities to influence decision-making at the national level, both the Engagement Protocol and Health Approach have ensured that First Nations and Inuit representatives have a seat at FNIHB's National Senior Management Committee (SMC) Table. The mandate of the SMC is to: provide overall policy direction for the Branch, consistent with the goals and principles of the First Nations and Inuit Health Strategic Plan; monitor and provide direction for the Plan's implementation, including collaboration and partnership initiatives; review, approve and provide direction on policy issues and approaches; and review and approve proposals for modernizing policies and programs. In addition, the Engagement Protocol and Health Approach have ensured that First Nations and Inuit representatives have a seat on FNIHB's SMC Policy and Planning Subcommittee. The mandate of the SMC Policy and Planning Subcommittee, among other things, is to develop an engagement approach including all relevant partners, and to define the initial scope of a new proposal or terms of reference for policy/program development or modernization.

Representation on internal FNIHB senior management committees is intended to develop quality engagement and participation and increase opportunities for these organizations to influence decision-making at the national level. Key informant interviewees, FNIHB national office and regional staff and representatives from NAOs, indicated that both AFN and ITK are engaged at the national level as a result of the Engagement Protocol and the Health Approach. Although it is too early to determine whether First Nations and Inuit representatives have been able to influence decision-making in a meaningful way through these engagement protocols, the development of these two agreements demonstrates progress in strengthening the mechanisms available to First Nations and Inuit representatives to influence national-level decisions moving forward.

4.4.3 Has First Nation and Inuit collaboration (consultation and partnerships) with partners and other stakeholders in program and services planning increased? In what ways?

Opportunities have increased for First Nation and Inuit representatives to collaborate and participate in the design and development of programs and policies at the national, regional and community levels.

Several lines of evidence (i.e., key informant interviews, evaluation surveys, case studies and document review) identified that through health planning and quality management activities, there were increased opportunities for consultation and partnerships at various levels for First Nations and Inuit representatives, resulting in increased opportunities to participate in the design and development of programs and services.

As mentioned previously, the AFN Engagement Protocol and ITK Health Approach are formal mechanisms that have increased opportunities for First Nations and Inuit representatives to influence decision-making at the national level. The Engagement Protocol specifies that FNIHB regional offices will maintain direct engagement with First Nations communities on matters that affect them, and engagement with other partners of mutual interest. Similarly, the Health Approach specifies that FNIHB will "work with Inuit, territorial and provincial governments and other federal partners to develop an approach to Inuit health that informs planning within Health Canada." The document review and key informant interviews (FNIHB national office and regional staff, representatives of NAOs) indicated that the Engagement Protocol and Health Approach have increased opportunities for First Nations and Inuit representatives to collaborate with representatives at different levels (national, regional, community) to influence program design and services development. Regionally, FNIHB regional offices engage directly with First Nations organizations on Regional Health Plans and other regional matters.

Beyond these formal mechanisms, these key informants also indicated that there is a trend toward greater FNIHB regional collaboration with First Nations and Inuit representatives. FNIHB Regional Executives and other FNIHB representatives: participated in a number of trilateral tables in which First Nation representatives participate/influence in the design of programs or policies along with provincial governments; liaised with First Nations umbrella organizations; and collaborated with Chiefs and participated in the development of working groups with Tribal Councils. The FNIHB regional liaison staff and First Nations community health leaders surveys indicated that consultation is occurring more often between FNIHB and provincial government representatives on issues related to First Nations and Inuit health programs and services. These examples of increased engagement have the potential to lead to opportunities for First Nations and Inuit representatives to influence the design and development of programs and services.

In terms of health planning and accreditation activities in First Nations communities, respondents to the FNIHB regional liaison staff and First Nations community health leaders surveys, and the case studies, indicated that these activities have supported consultation and collaboration at regional and community levels - leading to a direct impact on program decision making by First Nations communities. Health planning and accreditation have facilitated a greater awareness by First Nations communities of the importance of partnerships and collaborations at all levels. Survey respondents also indicated that longer term contribution agreements have increased the opportunity for First Nations representatives to collaborate with partners in programs and services planning. Longer term contribution agreements, accompanied with their relative flexibility, allow more time to form meaningful partnerships at all levels.

4.4.4 What types of supports or activities are effective in supporting transition to more flexible funding models? Are there any barriers or challenges?

Funding models broadly reflect levels of community capacity to take ownership and control over delivery and management of health services on-reserve.

The distribution of First Nations communities' contribution agreements in Set, Flexible, and Block funding models has not significantly changed since 2010 (see Table 3). Data from the Management Contract and Contribution System (MCCS) indicates that as of March 31, 2015 a majority of communities have signed contribution agreements in a Flexible (37%) or Block (40%) funding model. There were also 23% in the Set funding model.^{Footnote iii}

Overall interviews with FNIHB national office staff, and surveys with FNIHB regional liaison staff and First Nations community health leaders, indicated that the Set funding model, which involves lower flexibility and control, is mostly associated with lower recipient community capacity to manage and deliver health programs and services. These key informants indicated that the Flexible and Block funding models, with room for greater community control and responsibility, are generally applied to contribution agreements with recipient First Nations communities with higher capacity to manage and deliver health programs and services.

However, FNIHB national office (interviews) and regional staff (interviews and survey) also indicated that the funding model level does not always match the community's capacity to manage and deliver health programs and services. These key informants indicated that some high capacity recipient communities currently receive funding through contribution agreements with less flexible (Set or Flexible) funding models, while some lower capacity communities have contribution agreements with more flexible (Block) funding models.

Sometimes recipient First Nations communities opt to exercise their ownership over health service delivery by deciding to maintain a contribution agreement in a Set or Flexible funding model, rather than moving to a Block funding model.

• Community-level governance challenges may contribute to health administrator decisions to remain in a stable contribution funding model.
• The Treaty 6 Medicine Chest Clause is a factor that has contributed to some community decisions to remain in less flexible funding models, despite having the capacity to meet the requirements of a more flexible funding model. This clause has been linked to an understanding in some First Nations communities that there is federal government responsibility to provide all forms of health
• Care to First Nations people on an ongoing basis.
• The social determinants of health, including environmental, social and health factors that impact First Nations' community capacity to take ownership over health service delivery are inter-related and community-specific. In some cases, factors outside the scope of health contribute to community decisions to not transition to more flexible contribution funding models, despite having the organizational capacity to do so.
• Some First Nations communities decide not to transition through funding models but rather focus on capacity building and self-determination in other areas outside of health service delivery.

Some of the case studies demonstrated that community capacity also varies over time, and does not necessarily evolve in one direction. For example, in one case study of a Block contribution agreement, a dip in capacity related to community governance challenges, was then followed by a period of more governance stability and a return to a capacity to deliver health programs and services. This change occurred over several years and within the span of one contribution funding agreement.

FNIHB national office staff indicated that there may be regional synchronicity in the administration of funding models related to regional administrators' definitions and measurements of capacity to manage and deliver health programs and services. FNIHB regional liaison staff (survey) indicated that they use a variety of tools to assess community capacity to deliver and manage health programs and services. These tools include risk management activities, general assessments and capacity readiness assessments. The level of capacity required to enter into, and maintain, a Flexible or Block contribution funding agreement in one region may not match that required by a different region.

Key facilitators to funding model transitions include ongoing FNIHB regional liaison staff engagement with First Nations communities and effective community health leadership.

Transitions between funding models usually take several years, and involve strategic health planning with support from FNIHB regional liaison staff to develop a multi-year work plan (for a Flexible agreement) or a health plan (for a Block agreement). Key informants, including through the case studies, highlighted several key facilitators that contribute to effective community funding model transitions.

• FNIHB national office (interviews) and regional staff (interviews and survey), and First Nations community health leaders (survey), emphasized that early and on-going engagement by FNIHB regional staff with First Nations communities is crucial to building trust with communities and foundational to effective relationship building. FNIHB regional liaison staff also emphasized that a key relationship to facilitating funding model transitions is between First Nations community health leaders (often Health Services Managers or Health Directors) and their FNIHB regional staff counterparts, often identified as community liaison officers.
• FNIHB regional offices provide some training opportunities to support capacity building in First Nations communities, including health planning workshops for Health Directors and individual training opportunities in communities.
• First Nations community health leadership is another key facilitator for community transitions to more flexible funding models. FNIHB national office and regional liaison staff, external stakeholders and First Nations community health leaders, indicated that a key form of community support for funding model transitions is that of effective community Health Directors or other health leaders to support and sustain transitions.
• Health planning funding, which is received by First Nations communities in Flexible or Block funding models, was cited by both FNIHB regional liaison staff and First Nations community health leaders as an important contributing factor to building community capacity to conduct health planning, by enabling the community to employ a full-time Health Director.
• FNIHB national office and regional staff indicated that participating in the accreditation process contributes greatly to a community's readiness for transition to more flexible funding models, and significantly contributes to the ease with which such transitions may take place.
• Additionally, First Nations community health leaders identified collaboration with other First Nations communities to share best practices as another source of support for community transitions through funding models.

4.4.5 Has First Nations and Inuit ownership and control of health planning and quality improvement strategies increased?

There has been an increase in First Nations community uptake of health planning and accreditation activities.

FNIHB national office (interviews) and regional staff (interviews and surveys) reported that there has been an increase in First Nations community interest and commitment to health planning over the last five years. Some of these key informants indicated that the majority of First Nations communities can see the value in health planning. They suggested that First Nations communities welcome the health planning process and the flexibilities it provides to address their changing priorities. These communities see the results of previous planning initiatives. Some communities take it very seriously and work hard to address the needs identified by their community. Some key informants indicated that there is real value in knowing that there is a health plan that can be used as a basis for their work.

The uptake of accreditation by First Nations health services organizations has increased significantly since 2010. Program performance measurement data reflects that the number of community health centres and nursing stations engaged in the accreditation process, with FNIHB support, increased by 61% from 2010 to 2015. As of 2015, a total of 75 community health centres and nursing stations, serving 152 First Nations communities, were in the accreditation process or were accredited. As of 2015, 60 community health centres serving 132 communities, and one nursing station, serving one community, were accredited. (In light of the 2015 Office of the Auditor General report on the need for greater access to health services for remote First Nations communities,^{Footnote 21} a focus on the accreditation of more nursing stations has been established as a priority by Health Canada moving forward.)

FNIHB does not accredit health services but rather supports organizations that are engaged in the process with an accrediting body. Accreditation is an ongoing quality management process that community health organizations use to assess and improve the quality of their health services. Community health centres volunteer to opt into taking on the accreditation process. They must: establish and maintain a working relationship with the accrediting body throughout the accreditation process; engage in accreditation with an accreditation body; develop and implement the accreditation plan to meet the processes required of accreditation; and maintain standards to ensure accreditation. Simply going through the accreditation process increases capacity.

Health planning and accreditation activities have built community capacity to take ownership and control over health service delivery and quality improvement strategies.

Undertaking the processes required to develop health plans and meet accreditation requirements contributes to First Nations community capacity to exercise ownership and control over health programs and services by strengthening community-level organizational capacities to manage and deliver health programs and services.

FNIHB national office (interviews) and regional staff (interviews and survey), and First Nations community health leaders (survey), reported that health planning enhanced community health organizational leadership capacity to plan, deliver and evaluate health programs, and to demonstrate accountability to its funders. Health planning contributed to a community's capacity to engage members in the articulation of a vision of health for the future, and encouraged organizations to focus on measurable outcomes. It helped to define the roles and responsibilities of health workers at all levels in the organization. Additionally, some First Nations community leaders reported that health planning served as a team building exercise, leading both to enhanced individual staff members' capacities, as well as improved health organization staff cohesion.

There is strong evidence (from key informant interviews, the surveys and case studies) that the accreditation process enhanced community-level capacity by enhancing the health organization's capacity to: conduct strategic health planning, priority setting, and financial planning; demonstrate accountability; and identify gaps in service delivery processes and planning. A 2015 program-led survey of organizations involved in accreditation found that a large majority (86%) of organizations attribute an increase in their health management abilities to being involved in accreditation activities.

Health planning and accreditation also led to enhanced partnerships. FNIHB regional liaison staff reported that enhanced First Nations community health organization capacity to engage in partnerships was an unexpected outcome of health planning activities. In several case studies, health planning contributed to community abilities to collaborate with Tribal Councils. There is strong evidence through key informant interviews, surveys and case studies that accreditation has built community health organizational capacity to engage in strategic partnerships with similar organizations in other First Nations communities by facilitating and encouraging the sharing of best practices.

Health planning and accreditation led to pride and ownership of community health programs and services. FNIHB regional liaison staff and First Nations community health leaders reported that, in some cases, health planning contributed to a sense of ownership over community health programming and greater community mobilization, by actively engaging community members in the planning process. Key informant interviews and survey results indicated that accreditation was perceived as a 'gold standard' of achievement for First Nations communities for their health programs and services, and that the accreditation process contributed to an increased sense of community pride and ownership over health programs and services.

Health planning and accreditation activities have built on First Nations community strengths.

Both health planning and accreditation have built on, and enhanced, community capacity to take ownership and control and organizational strengths that are already present in First Nations communities. Health planning and accreditation investments have successfully leveraged the strengths of communities with a strong existing capacity to deliver health programs and services.

Multiple lines of evidence (key informant interviews, the surveys, case studies) indicated that health planning and accreditation are more likely to be successful in organizations with effective and stable health leadership and staff contingents. FNIHB regional liaison staff and First Nations community health leaders (surveys) indicated that health planning and accreditation contributed to stable governance in the community, by providing structure and guidance as well as clarity regarding the roles of political actors in terms of the management of community health programs and services. The effectiveness of Health Directors was cited by FNIHB national office and regional liaison staff, and external stakeholders, as a key factor in the success of health planning and accreditation. Case studies in particular also supported this finding. First Nation communities with stable governance practices, including clarity regarding the roles of political and health leaders, were more likely to succeed in health planning and accreditation.

Key informants indicated that community ownership is essential for the success of health planning process and is a pre-requisite for the accreditation process (which is a voluntary process that community health centres must opt in to). FNIHB national office (interviews) and regional staff (interviews and survey) highlighted that health plans are more useful (and therefore effective) when completed by community members and Health Directors themselves, rather than by external consultants. A program-led survey of organizations involved in accreditation found that a majority (77%) of respondents indicated that accreditation has enhanced their organization's commitment to quality improvement.

Some First Nations communities experience challenges to participating in health planning and accreditation activities.

The calibration of support to community needs is a foundational principle of community engagement and management. The FNIHB approach to community capacity building is most effective when it is adaptable based on the capacity of the community.

FNIHB national office (interviews) and regional staff (interviews and survey), external stakeholders including representatives from NAOs (interviews), and First Nations community health leaders (surveys) reported that some First Nations communities experience challenges to participating in health planning and accreditation activities, and therefore experience challenges building and exercising the ownership and control that flow from these activities. These key informants reported that these challenges are more likely to be found in remote and isolated communities, smaller communities, as well as communities in crisis.

Further, unique needs related to the exercise of ownership and control through health planning and accreditation are often related to organizational and community capacity. These unique needs may include: limited strategic planning underway, higher rates of governance turnover, limited access to health planning funding in First Nation communities in less flexible funding models (e.g., Set funding models), and difficulty attracting and retaining human resources in many remote and isolated communities. It should be noted that a health planning guide was recently launched by FNIHB in order to assist those communities that would benefit most from greater capacity.

First Nations communities with varying levels of capacity require different approaches to maximize their opportunities to exercise ownership and control.

Most FNIHB national office and regional staff reported that First Nations communities with lower levels of capacity require tailored forms of support and opportunities to meet their unique capacity needs. For example, communities in which a Health Director has not yet been identified, or for which health planning management funds have not yet been allocated (this is the case in many communities in Set funding models) may require extra support in the form of additional resources (training or tools) specific to their needs. Other suggestions from internal and external key informants (including through the case studies) to support First Nations communities in enhancing their capacity include: leveraging networks (mentoring) and best practices from other communities; partnering with other regional stakeholders; and additional time and resource commitments for FNIHB regional liaison staff for community engagement.

Some FNIHB regional liaison staff (survey) indicated that internal FNIHB workload can be a barrier to assist all First Nations communities with these activities and help them build on their strengths. They suggested that there is a need for increased capacity within the FNIHB community liaison teams as well as standardization within and amongst regional offices to ensure measurable increased capacity for recipients. Some FNIHB regional liaison staff also highlighted a need for more flexibility for staff to engage with First Nations communities at a lower level of capacity - with more opportunities to monitor these communities through reporting and program updates. Other communities, with well-established organizational capacity to manage and deliver health programs and services, may require less direct capacity building, and benefit from the opportunity to focus on program management with limited FNIHB regional liaison staff involvement and reporting.

4.4.6 Has the quality and delivery of programs and services improved?

Health planning and accreditation have had a positive impact on the quality of programs and services delivered in First Nations communities.

The 2012 FNIHB Quality Improvement (QI) Policy Framework is a Branch-wide initiative that applies to all functions within FNIHB and supports the Branch's vision of healthy First Nations and Inuit individuals, families and communities. The Framework defines quality along six dimensions: accessibility, client centred, culturally competent, effective, efficient, and safe.

There is evidence from the surveys (FNIHB regional liaison staff and First Nations community health leaders), as well as from the 2015 program-led survey of organizations involved in accreditation, that the quality of health programs and services delivered on-reserve has improved along all six of the Framework's quality dimensions. These quality improvements were noted in particular in First Nations communities that have implemented health planning and/or accreditation activities.

Accessibility

Improved coordination across programs and services (integration) is one of the principal elements of accessibility in the Framework. Respondents to the FNIHB regional liaison staff survey indicated that health planning contributes to the integration of health services by encouraging practices that improve health integration, such as holistic approaches to health, decreasing of silos, and prioritization of partnerships within the community and outside the community. They reported that health planning has increased the ability of First Nations communities to support ongoing relationship building with other First Nations organizations, the provincial government and Regional Health Authorities. Survey respondents indicated that health planning has contributed to improved willingness to work with other organizations to access health care, midwifery services, mental health supports, and victims' assistance programs.

Another element of accessibility in the Framework is the provision of appropriate health programs and services. Through the program-led survey of organizations involved in accreditation, the large majority (92%) indicated that accreditation has led to more appropriate care provided to clients and the large majority (86%) reported that accreditation has enhanced understanding related to the coordination of care.

Client Centred

Client centred services, according to the Framework, actively involve and respect the needs and preferences of individuals, families and communities. Case studies highlighted that health planning leads to increased involvement of community members which in turn is linked to the quality of health programs and services. The majority of First Nations community health leaders surveyed indicated that health planning has helped to make care more client-centred in community. Likewise, FNIHB regional liaison staff surveyed indicated that health planning contributed to an increased awareness of client needs by health care staff during the planning and delivery of community health programs and services. They also underscored that this observation applies mostly to First Nations communities in which health plans were not created by consultants. A program survey of organizations involved in accreditation found that a large majority of respondents (88%) agreed that accreditation has helped to make care more client-centered.

Culturally Competent

The Framework indicates that culturally competent care integrates knowledge about First Nations and Inuit individuals, families and communities into specific health program and service standards, policies, practices and attitudes. In their survey responses, FNIHB regional liaison staff highlighted improvements in the cultural relevance of health programs, noting that these improvements were a result of the consultation with and involvement of community members in health program planning. Some surveyed First Nations community health leaders reported that health planning has contributed to more culturally relevant community health work, citing as an example the incorporation of First Nations languages in health programming. The majority (69%) of surveyed First Nations community health leaders from organizations that have undertaken accreditation indicated that accreditation has helped to provide culturally consistent care.

Effective

As outlined in the Framework, effectiveness is about applying current evidence to achieve the best possible health outcomes. It requires continuous monitoring of processes and programs. Some FNIHB regional liaison staff reported that health planning's focus on measurable outcomes contributed to improved effectiveness of health programs and services. Some FNIHB regional liaison staff (survey) indicated that many First Nations communities focus on measurable outcomes to show success through healthier community outcomes. Respondents to the First Nations community health leaders survey indicated that health planning facilitates the collection of continuous data and information which helps benefit staff and employees stay informed and on track when delivering health programs in the community. It facilitates working on long term goals and objectives. Evaluation of the programs helps define direction and set common goals as a team from the needs of the population. A large majority (97%) of surveyed organizations that receive FNIHB funding to implement accreditation indicated that policies and processes have been implemented or created that improve operational effectiveness as a result of accreditation.

Efficient

Efficiency, as defined in the Framework, is about optimizing resources wisely and value for money. Some FNIHB regional liaison staff reported that health planning helps to increase the good governance of health programs and services through the development of a team at the health centre with increased efficiency of staff performance under guidance of good policies and ability to more efficiently meet its mission and goals in a sustainable way. Several FNIHB regional liaison staff reported that annual health plans are reviewed using accreditation standards. The large majority (90%) of surveyed organizations involved in accreditation indicated that they use human resources more efficiently as a result of participating in the accreditation process. The majority (77%) of organizations involved in accreditation also indicated that accreditation has improved their ability to manage finances, coordinate human resources and conduct strategic planning.

Safe

Safety is characterized by minimizing risk and avoiding harm as per the Framework. It means that health care programs and services promote safe practices and reduce adverse outcomes. Some FNIHB regional liaison staff reported that health planning has contributed to making health centres safer. Many (73%) First Nations community health leaders reported that health planning has contributed to improved community abilities to ensure client safety. All (100%) of the surveyed organizations involved in accreditation indicated that accreditation has enhanced staff safety; a large majority (97%) indicated that accreditation has helped to improve client safety, and a majority (80%) of respondents indicated that accreditation has helped to reduce adverse events.

There remain challenges to quality improvement in the delivery of health programs and services in First Nations communities.

All lines of evidence indicated that there remain challenges to quality improvement in First Nations communities. Challenges to the implementation of health planning and accreditation that were cited by surveyed FNIHB regional liaison staff and First Nations community health leaders include: limited human resources in communities (a need for more health professionals on the ground, especially mental health workers, dental care and pharmacy care) and limited community resources to address immediate needs. A more fulsome outline of challenges can be found in Section 4.4.5 - Ownership and Control.

4.5 Performance: Issue #5 - Demonstration of Economy and Efficiency

The Treasury Board of Canada's Policy on Evaluation (2009) and guidance document, Assessing Program Resource Utilization When Evaluating Federal Programs (2013), defines the demonstration of economy and efficiency as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. This assessment is based on the assumption that departments have standardized performance measurement systems and that financial systems link information about program costs to specific inputs, activities, outputs and expected results.

The data structure of the detailed financial information provided for the program did not facilitate the assessment of whether program outputs were produced efficiently, or whether expected outcomes were produced economically. Specifically, the lack of output/outcome-specific costing data limited the ability to use cost-comparative approaches. In terms of assessing economy, challenges in tracking funding within the broader program envelope limited the assessment. Considering these issues, the evaluation provided observations on economy and efficiency based on findings from the literature review, key informant interviews and available relevant financial data. In addition, the findings below provide observations on the adequacy and use of performance measurement information to support economical and efficient program delivery and evaluation.

Observations on Efficiency

Efficiencies have been demonstrated with respect to both program design and delivery. Further efficiencies could be achieved by improving internal FNIHB cultural competency, reducing organizational silos (at both FNIHB and federal levels) and reducing reporting burden on recipients.

Accreditation Program - Funding Formula

Efficiencies were demonstrated in the design of the accreditation component. This component employs a funding formula to determine the amount of funding provided to a community organization to support the initiation and implementation of their accreditation activities. This funding formula has been refined over time and is the result of considerable engagement with all stakeholders. Internal and external key informants indicated that the funding formula for the accreditation component is systematic, objective and useful in creating equity in funding determination.

FNIHB - Internal Integration and Cultural Competency

While there is collaboration occurring at various levels and across programs within FNIHB, survey respondents (FNIHB regional liaison staff and First Nations community health leaders), articulated the need for a more integrated and culturally competent approach to First Nations and Inuit health service delivery across all regions and all community-based programs.

• Some key informants suggested that further efficiencies could be gained if standards from the accreditation program were further integrated into FNIHB program guidance. They highlighted that that the Health Canada First Nations and Inuit Home and Community Care Quality Resource Kit (2012), which encourages quality improvement and use of risk assessments, has been designed to be a practical, relevant, and useful resource that improves the support provided to home and community care clients. Internal and external key informants noted opportunities to integrate accreditation standards into other program guidance documents that support community programs, such as mental wellness and healthy living.
• FNIHB key informants indicated that considerable variability across regional offices in terms of the approaches taken to support health planning activities within First Nations communities, including: the roles of FNIHB regional liaison staff (focus on administration versus liaison and relationship building), the tools used to track health planning activities, engagement with other FNIHB community-based programs, and liaison with other regional health stakeholders. They indicated that there are opportunities at the senior management level for dialogue across regional offices and with the national office, including: Regional Executive tables, regional transformation teams, FNIHB planning and reporting tables, and the senior management (SMC) table. Many internal FNIHB key informants from the national and regional offices noted that there are opportunities for sharing of best practices across the Branch, in particular at the working level and across regional offices.
• Cultural competency supports capacity. Internal and external informants indicated that skillful engagement, meaningful dialogue, and relationship building with First Nations communities has been crucial to developing strong health plans and building ownership of them as well as supporting accreditation activities. Key informant interviews, surveys and case studies all indicated that FNIHB has improved and can continue to enhance the cultural competency of staff in various roles.

FNIHB - Working with Other Government Departments

Many internal FNIHB and other government department key informant interviewees indicated that there are opportunities to enhance working relationships and processes among Health Canada, Indigenous and Northern Affairs Canada, Public Safety Canada, Employment and Social Development Canada (ESDC), and other government departments that provide complementary capacity building services and support to First Nations and Inuit organizations and communities.

• Health Canada supports capacity building with community health leaders and staff on- reserve.
• INAC provides a wide range of community capacity building supports, related to: governance, business development, education, water/waste water systems, and emergency preparedness.
• Public Safety Canada oversees community capacity building to reduce violence against women/girls and develop community safety plans.
• ESDC aims to increase participation of Indigenous peoples in the Canadian labour market and supports Indigenous community organizations delivering employment/training services.

First Nations and Inuit organizations and communities are required to complete a variety of reporting requirements from a mix of federal funding partners who provide community-based funding support. There are opportunities to explore efficiencies in reporting across federal departments. For example, some key informants mentioned an opportunity to further explore a reporting model recently piloted in the Quebec region with a multi-departmental streamlined effort to help reduce the reporting burden on these community organizations.

Observations on Economy

As highlighted in Table 1: Program Resources in Section 2.4 - Program Alignment and Resources, over the past five years funding for the mix of health planning and quality management activities has remained relatively stable with annual spending between $113 million and $146 million. It was noted that spending has slowly decreased over time; in particular there was an $18 million decrease in spending in 2012-2013.

About 72%-83% of the total spending amount was coded to the health planning (governance) component. The smallest proportion was coded to the accreditation component (3%-5%). The health research component comprised the balance of program spending in this area with the biggest fluctuations over the last five years - more monies from 2010-2011 to 2011-2012 (20%-24%) and a relatively smaller proportion between 2012-2013 and 2014-2015 (13%). Detailed financial tables by program component are provided in Appendix 4.

To support economic program delivery, some HPQM components have built upon the expertise and capacity of First Nations organizations and communities.

Health Research - First Nations Information Governance Centre

As detailed in Section 4.4.1, the First Nations Information Governance Centre is a First Nations entity. The Centre builds capacity and provides credible and relevant information on First Nations using high standards of data research practices, while respecting the rights of First Nations self-determination for research and information management. The First Nations Regional Health Survey is housed with the FNIGC - reflecting and ensuring First Nations ownership of their health data. The RHS led the way for First Nations to exercise jurisdiction over their information.

Internal and external interviews, and document reviews, indicated how relatively small FNIHB investments over the last 20 years to support the development and implementation of RHS activities have helped to build momentum related to First Nations owned and managed health research in Canada. Key informants indicated that the RHS is a recognizable/trusted name in First Nations communities. They highlighted how investments in the Survey have led to regionalized hubs that serve the strategic First Nations information and research needs as determined by each of the participating regions. They also highlighted that many good practices have been developed and implemented including: cultural framework, First Nations research ethics, and principles of OCAP (ownership, control, access, and possession).

In the past, First Nations on-reserve communities were often excluded from national population surveys. Internal and external key informants indicated that the success of the RHS, and the collaboration between Health Canada and the FNIGC, led to an opportunity for INAC to collect on-reserve data. Starting in 2011, INAC has worked with the FNIGC to undertake data collection on-reserve for the Aboriginal Peoples Survey, leveraging Health Canada's lessons learned from the RHS process. (Statistics Canada collects APS data for the Indigenous off-reserve population.) This novel arrangement has allowed INAC to collect on-reserve data using the already established processes and structures through the RHS.

Accreditation - Community Health Centres

FNIHB supports the voluntary accreditation of health services provided by First Nations community health centres, as well as FNIHB nursing stations. FNIHB provides funding for achieving and maintaining health services accreditation, either through targeted accreditation funding or organizational operating budgets. Costs covered include the accrediting body invoice which covers the standard fees charged by the accrediting body to pursue accreditation such as application fees, annual fees, survey/review fees, required manuals and reports, certificates and mandatory education/training sessions. It also supports the costs of the accreditation process such as staffing, meetings, honorariums, interim/follow up visits, additional training and the development of policies.

Both internal and external key informant interviews, as well as a document review, indicated that the accreditation component, with relatively small investments, has successfully leveraged the existing community expertise to enhance health programs and services. FNIHB maintains an arm's length relationship with the accrediting body and accreditation process to ensure the independence of the accreditation body and First Nations and Inuit organizations.

Observations on the Adequacy and Use of Performance Measurement Data

For most of the HPQM components, some performance measurement data has been collected.

The Branch has demonstrated a commitment to performance measurement. FNIHB national office staff (key informant interviews) and regional liaison staff (survey), as well as a review of existing performance measurement tools, indicated that steps have been taken to have performance measurement in place to track project-level activities, expenditures and outputs of funded projects health planning and accreditation projects. There are a number of tools that collect project-level performance measurement data. These include:

• grants and contributions databases track program activities and expenditures (including the former MCCS and current GCIMS);
• the Community-based Reporting Tool (CBRT) collects data on outputs by program area;
• the Community Planning and Management System (CPMS) maintains all community population and profile data for all First Nations, Innu and Inuit communities south of the 60^th parallel; and
• project level data is also collected through annual Contribution Agreement (CA) reports submitted by funding recipients.

In addition, the accreditation component had conducted a survey in the Spring 2015 that asked the funded organizations questions about the outcomes linked to their accreditation activities, including: quality of care, resource use and capacity building, benefits of accreditation, and the accreditation experience. Data from this survey has been analyzed and insights have been gained about a number of program outcomes: enhancements in quality of care; improvements in governance, leadership, and the management of finances and human resources; and the creation and implementation of policies and processes that improve people's health.

A refined data collection strategy may enhance how FNIHB and funded organizations collect and use performance measurement data in the future.

There are opportunities to refine and augment accountability measures for program components within the HPQM cluster of activities. As stated in Section 2.3, the expected result for the HPQM cluster of activities of the Health Canada PAA (3.3.1.1) is: "The capacity to deliver health programs and services is increased." For a cluster of programs like health planning and quality management, with the stated result linked to capacity building, it will be important to update the HPQM performance measurement strategy to establish methods to measure increased capacity and control and the quality of service being delivered. Examples of the types of outcome-based indicators linked to capacity building for health planning and quality management activities might include: community cohesion, governance structures, partnerships, and sustainable human resources and administration

5.0 Conclusions

5.1 Relevance Conclusions

5.1.1 Continued Need

There continues to be a need for governments to play a role in facilitating and strengthening the First Nations and Inuit capacity to build and sustain the quality of their health programs and services. A review of literature on health outcomes indicates that the health status of First Nations and Inuit in Canada falls below that of the general Canadian population. The literature on community development in First Nations and Inuit communities suggests that increased First Nations and Inuit capacity and control can have a positive impact on community health programs and services in four key areas, in turn leading to enhanced health outcomes. These four areas are: social determinants of health, cultural competency and safety, decentralization, and quality of services. The FNIHB health planning and quality management capacity building activities are designed to impact these four areas.

5.1.2 Alignment with Government Priorities

Priorities for the health planning and quality management activities align with the Government of Canada and departmental current priorities and strategic outcomes. Various Government of Canada commitments support health programs and services by collaborating with First Nations and Inuit organizations and communities to ensure they are accessible, reliable and of quality. Recent Government of Canada Speeches from the Throne committed to addressing barriers to economic/social participation for the First Nations and Inuit and Budgets have focussed on providing better services and investments in community infrastructure. Health Canada's departmental objectives and the FNIHB strategic plan support First Nations and Inuit self-determination, community-level action and capacity building, and building stronger stakeholder connections.

5.1.3 Alignment with Federal Roles and Responsibilities

There is a key federal role established in various policies (e.g., Indian Health Policy, Indian Health Transfer Policy) aimed to increase First Nations and Inuit community control and responsibility for health service delivery.

There is some complementarity across federal-level departments with programs that support capacity building; however, there are opportunities for enhanced coordination of community planning activities (e.g. emergency preparedness, health planning). In terms of health service provision, there are distinct roles between the federal and provincial/territorial governments and there are also opportunities for improved integration (across these roles) to address gaps across continuity of care (e.g. clarity in roles for funding medical transportation).

5.2 Performance Conclusions

5.2.1 Achievement of Expected Outcomes (Effectiveness)

The availability and use of evidence-based information has been enhanced. Collaborative research and surveillance activities have been undertaken with other government departments including the CIHR, Public Health Agency of Canada, and Statistics Canada. A research partnership has been established with the FNIGC to support the First Nations Regional Health Survey - the only First Nations governed national health survey that collects information about on-reserve and northern First Nations health. There is however an increased need and demand for evidence-based information as it is not always readily available in a form easily used for policy changes or improvements.

In addition, there has been enhanced opportunities for national and regional First Nations and Inuit organizations to participate in the design and development of FNIHB programs and policies. Finalized in 2014, the Assembly of First Nations Engagement Protocol and the Inuit Tapirit Kanatami Health Approach have been designed to increase opportunities for collaboration and for First Nations and Inuit representatives to influence decision-making at the FNIHB national and regional senior management levels. At the national level, it ensures full participation at senior management tables within FNIHB. At a regional level, these formalized agreements have increased opportunities for First Nations and Inuit representatives to collaborate and influence program design and development by supporting FNIHB Regional offices to engage directly with First Nations organizations on regional health plans and other regional matters through partnership tables.

Some HPQM components have positively impacted community capacity, which in turn have enhanced the quality of health programs and services in First Nations communities. The quality of services has improved in communities that have taken ownership over health service delivery and management through health planning and/or accreditation. The benefits of health planning have included: improved accessibility of health programs services, increased awareness of client needs, improved cultural relevance of health programs and services, increased effectiveness of programming via a focus on measurable outcomes, improved efficiency of health programs and services via improved governance, and increased safety of health centres. Accreditation has led to: more appropriate care being provided to patients; care to be provided in a more timely manner; more client centred and culturally consistent care; improved patient safety; and reduced numbers of adverse events. However, it was noted that lower capacity communities in Set funding models do not have ready access to health planning and accreditation guidance or funding.

Moreover, funding models broadly reflect the levels of First Nations community capacity to take ownership and control over delivery and management of health programs on-reserve however there are exceptions. Sometimes recipient First Nations communities opt to exercise their ownership over health service delivery by deciding to maintain a contribution agreement in a less flexible funding model (e.g., remaining under a Set or Flexible funding model, rather than moving to a Block funding model). Key facilitators to funding model transitions include effective First Nations community health leadership and ongoing FNIHB regional staff liaison with First Nations communities.

5.2.2 Demonstration of Economy and Efficiency

Some health planning and quality management activities have effectively built upon the expertise and capacity of First Nations organizations and communities to enhance community ownership and control. This includes relatively small investments in the FNIGC to support RHS activities has helped to build momentum related to First Nations owned and managed health research in Canada. As well, the accreditation component, with relatively small investments, successfully built upon the existing community resources and expertise to enhance health programs and services.

Further efficiencies could be gained by enhancing information exchange and coloration on HPQM activities across FNIHB programs and regional offices. While there is some collaboration occurring at various levels, there is the need for a more integrated approach to First Nations and Inuit health service delivery across community-based programs.

6.0 Recommendations

Recommendation 1

Strengthen processes and tools to improve opportunities for all First Nations communities to develop the capacity to exercise ownership and control, particularly with those First Nations communities with lower initial levels of capacity.

First Nations communities with varying levels of capacity require different approaches to maximize their opportunities to exercise ownership and control. Challenges in terms of capacity are more likely to be found in remote and isolated communities, smaller communities, as well as communities in crisis. These challenges include: limited strategic planning underway, limited access to health planning funding, and difficulty attracting and retaining human resources in many remote and isolated communities. Tailored support to meet these unique capacity needs might include: additional training or tools, leveraging networks (mentoring) and best practices from other communities, partnering with other regional stakeholders, and ongoing community engagement with FNIHB regional liaison staff. A health planning guide was recently launched by FNIHB in order to assist those communities that would benefit most from greater capacity.

Recommendation 2

Enhance information exchange and collaboration on health planning and quality management activities across FNIHB programs and regional offices.

Enhanced collaboration across FNIHB would lead to a more integrated and culturally competent approach to First Nations and Inuit health service delivery across all regions and all community-based programs. For examples, there are opportunities to: integrate accreditation standards into other program guidance documents that support capacity in community programs, such as mental wellness and healthy living; undertake sharing of best practices on capacity building across the Branch, in particular at the working level and across regional offices; and continue to enhance the cultural competency of staff in various capacity building roles.

Recommendation 3

Updated HPQM performance measurement strategy needed to better measure results.

An updated performance measurement strategy is needed to establish methods to measure increased capacity and control as well as the quality of services being delivered. Examples of the types of outcome-based indicators linked to capacity building for health planning and quality management activities might include: community cohesion, governance structures, partnerships, and sustainable human resources and administration.

Appendix 1 - Funding Models

• The First Nations and Inuit Health Branch (FNIHB) has a variety of contribution agreements which serve as vehicles for the administration and management of First Nations and Inuit community health programs and services. This is part of FNIHB's strategic direction to transfer autonomy and control of programs to First Nations and Inuit within a time-frame to be determined in consultation with them.
• These agreements vary in terms of level of control, flexibility, authority, reporting requirements and accountability and are categorized by funding model. First Nations and Inuit communities interested in having more control of their health services can decide among the different funding models based on their eligibility, interests, needs and capacity.
• In 2011, renewed funding authorities (Consolidation of First Nations and Inuit Health Contribution Program Authorities) included the introduction of new funding models to allow FNIHB to reduce the reporting burden on recipients, and align contribution agreements with the capacity of the recipient to design, manage, and monitor their programs and services to address the needs of their population.

The funding models are outlined as follows:

1. Set Funding Model
   • FNIHB designs the programs.
   • Recipients are able to redirect resources within the same sub-sub activity (with the written approval of the Minister).
   • Interim and year-end reports are required.
   • Duration of the agreements is up to three (3) years.
2. Flexible Funding Model
   • Recipients must establish a Multi-Year Work Plan, including a health management structure.
   • Recipients have the flexibility to reallocate funds within the same Program Authority and are allowed to carry forward program funding (with written approval from the Minister) for reinvestment in the following fiscal year within the same Program Authority.
   • Annual reports, including year-end audit reports, are mandatory.
   • Duration of the agreements is two (2) to five (5) years.
3. Block Funding Model
   • Recipients determine their health priorities, prepare a Health Plan (HP) accordingly, and establish their health management structure.
   • Recipients are able to reallocate funds across all authorities and are allowed to retain surpluses for reinvestment in priorities (listed in the approved HP).
   • Annual reports and year-end audit reports are mandatory as well as an evaluation report every five (5) years.
   • Duration of agreements is five (5) to ten (10) years.

Appendix 2 - Logic Model

Appendix 3 - Evaluation Description

Evaluation Scope

The scope of the evaluation covered the period from April 2010 to March 2015. It included four components of activities within FNIHB including: health planning (governance), accreditation, health consultation/liaison, and health research.

It excluded other activities within the Branch, including: Health Services Integration Fund, Human Resources and e-Health. The Northern Wellness Agreement self-governance and land claims were excluded because teasing out the federal role is challenging here given the leadership role of territorial governments in managing resources. As well, Inuit were included under health consultation/liaison component only.

Evaluation Issues

The specific evaluation questions used in this evaluation were based on the five core issues prescribed in the Treasury Board of Canada's Policy on Evaluation (2009). These are noted in the table below. Corresponding to each of the core issues, evaluation questions were tailored to the program and guided the evaluation process.

Data Collection and Analysis Methods

Evaluators collected and analyzed data from multiple sources. Sources of information used in this evaluation included

Literature review

• A search for Canadian and international literature from the past five years using search terms: "self-determination," "flexibility," "local control," "capacity building," "health services," "health programs," "health outcomes," "local needs," "health funding," "funding models," "First Nations," "Inuit," "Aboriginal," and "Indigenous."
• After examining documents to ensure relevance, 35 articles were reviewed.

Document and file review, including previous evaluations

• The main purpose of conducting a document review is to obtain a comprehensive understanding of the underlying theory of health planning and quality management activities within FNIHB, delivery of the activities, and results over the five year period covered by the evaluation.
• Approximately 180 documents pertinent to health planning and quality management activities held by FNIHB as well as external documentation.
• There was a review of one prior evaluation that covered the period April 2005 to March 2010.

Financial data review

• An analysis of financial data helps respond to questions of effectiveness, efficiency and economy.
• A review of financial data from fiscal years 2010-2011 to 2014-2015 was undertaken, which focussed on actual expenditures.

Key informant interviews

• Interviews were conducted with 43 stakeholders: Health Canada, FNIHB management and staff (n=28); other federal government departments or agencies (n=6); other non-governmental representatives (n=7); and external issue experts (n=2).
• Interview questionnaires were developed and slightly modified and tailored for each specific stakeholder group. Guides were based on the evaluation issues and questions identified in the evaluation matrix. They were developed using a semi-structured format, including probes where helpful. These semi-structured interviews based on several key questions help to define the areas to be explored, and also allow the interviewer or key informant to diverge in order to pursue an idea or response in more detail. The flexibility of this approach, particularly when compared to structured interviews or focus groups, also allows for the discovery or elaboration of information that is important to participants but may not have been previously thought of as pertinent by the evaluation team.
• Interviews were conducted in person (when possible) or by telephone.
• They were recorded, with participant's consent, and transcribed as necessary. Data was coded and analyzed with the aid of NVIVO software.

Surveys

• A web survey was conducted by the Office of Audit and Evaluation with FNIHB regional liaison staff. It provided these front line regional liaison staff with an opportunity to share their knowledge of and views on the implementation of FNIHB health planning (governance) activities. The feedback collected, specifically on the role of FNIHB and the implementation/impact of these activities, helped assess their relevance and performance. The survey was launched on September 14, 2015 and closed October 16, 2015. A reminder was sent after two weeks and a final reminder was sent after three weeks. It was sent to 83contacts provided by the management staff in six regional offices within FNIHB (excluding British Columbia and Northern region). There were 44 fully or partially completed surveys for a response rate of 53%. Average completion time was 51 minutes.
• A web survey was conducted by the Office of Audit and Evaluation with First Nations community health leaders. It provided these community stakeholders an opportunity to share their knowledge of and views on the implementation of FNIHB health planning (governance) activities. The feedback collected, specifically on the role of FNIHB and the implementation/impact of these community-based activities, helped assess the relevance and performance of these investments. The survey was launched on October, 23 2015 and closed November 13, 2015. A reminder was sent after two weeks and a final reminder was sent after three weeks. It was sent to 171contacts provided by the regional liaison staff in the regional offices within FNIHB. There were 50 fully or partially completed surveys for a response rate of 27%. The average completion time was 75 minutes.

Case studies

• The intent of the case studies was to collect site-specific information about implementation and impact.
• Five health planning (governance) case studies were completed and each included a document review, and a written questionnaire and follow-up interviews with FNIHB regional liaison staff..
• Case studies were stratified by: region - across Canada (east, west, north); funding model - Block or Flexible; amounts of funding - highest to less funding; length of funding relationship - long standing to recent; and remoteness - very isolated to close to urban centre. Also, using a self-determination lens, the evaluation looked at projects across: have always been using a Block funding model; have migrated from Flexible to Block funding model; and choose to stay at a Flexible funding model.
• The following First Nations organizations/communities were identified by FNIHB regional liaison staff as communities where there are best practices and lessons learned to be explored: Waycobah First Nation (Atlantic), Bande des Montagnais de Pakua Shipi (Quebec), Kingfisher Lake (Ontario), Sandy Bay Ojibway First Nation (Manitoba) and Piapot First Nation (Saskatchewan).

Performance measurement data (project-level performance)

• Analyses included a review of data on performance of program activities collected by the FNIHB between 2010-2011 and 2014-2015, including: Grants and contributions databases track program financial administration and activities (former MCCS and current GCIMS); the CBRT collects data on outputs by program; and the CPMS maintains all community population and profile data for all First Nations, Innu and Inuit communities south of the 60^th parallel.
• Project data is also collected through the annual Contribution Agreement (CA) reports.

Data were analyzed by triangulating information gathered from the different sources and methods listed above. This included: systematic compilation, review and summarization of data to illustrate key findings; thematic analysis of qualitative data; and comparative analysis of data from disparate sources to validate summary findings.

Appendix 4 - Actual Spending by Program Component

Appendix 5 - Summary of Findings

Rating of Findings

Ratings have been provided to indicate the degree to which each evaluation issue and question have been addressed.

Relevance Rating Symbols and Significance:

A summary of Relevance ratings is presented in Table 1 below. A description of the Relevance Ratings Symbols and Significance can be found in the Legend.

Performance Rating Symbols and Significance:

A summary of Performance Ratings is presented in Table 2 below. A description of the Performance Ratings Symbols and Significance can be found in the Legend.