First Nations and Inuit Health Facilities and Capital Program - Cluster Evaluation

Final Report

March, 2012

Table of Contents

• Management Action Plan
• Acronyms
• Executive Summary
• Background and Context
  • Program Description
  • Program Funding
  • Governance Structure
  • Project Prioritization
  • Capital Allocation Process
  • Other Program Activities
  • Evaluation Context
  • Limitations
• Performance - Relevance
  • Does the HFCP address current Government of Canada (GoC) priorities?
  • Does the HFCP address departmental strategic outcomes?
  • Does the HFCP align with federal roles and responsibilities?
  • Is there a need for the HFCP?
• Performance - Effectiveness
  • To what extent has the project management of health facilities been supported by HFCP?
  • Are resources allocated to communities exhibiting the greatest need?
  • To what extent have environmental management processes improved?
  • To what extent have security issues been identified and addressed?
  • Have HFCP planning/implementation of physical upgrades of health facilities met the needs of communities?
  • To what extent has life-cycle planning been supported by evidence-based data?
  • Have the O&M functions been adequate?
• Performance - Efficiency and Economy
  • Do the resources used for delivery approximate the minimum required to achieve the expected results?
  • Have facilities been completed in time and on budget?
  • Are there ways to improve the efficiency and economy of the program delivery?
• Conclusions and Recommendations
  • Conclusions
  • Recommendations
• Appendix A -- HFCP Logic Model

List of Tables

• Table 1 - Number of Buildings by Building Type and Region
• Table 2 - FNIHB Criteria for health facility classification (2005/06 - 2009/10)
• Table 3 - A-Base Funding
• Table 4 - Sustainability Funding
• Table 5 - Core Issues and Evaluation Questions
• Table 6 - Indicators Relating to HFCP-Specific Outcomes
• Table 7 - Number of survey participants per category
• Table 8 - Number of interview respondents by category
• Table 9 - Number of facilities per region that have one or more environmental issue identified
• Table 10 - Number of facilities with average effective age of 20+ years by region and the regional percentage of facilities with an average age of 20+ years
• Table 11 - Progress in addressing environmental priority 3 and 4 * recommendations contained in ECAFM

Management Action Plan

Note: Group all recommendations according to the core evaluation issues:

• Relevance: Continued Need for Program; Alignment with Government Priorities; Alignment with Federal Roles and Responsibilities
• Performance: Achievement of Expected Outcomes [effectiveness]; Demonstration of Efficiency and Economy

First Nations and Inuit - Health Facilities and Capital Program - Cluster Evaluation

Recommendations	Management Response	Deliverables	Accountability	Anticipated Completion Date
1. HFCP should explore ways to either restructure or replace RPMIS and make its use mandatory. This should be done as a way of improving its utility as a cost-effective source of information for program management at HQ and regional levels. The change should support the future evaluation of HFCP, including the capacity to assess the economy and efficiency of the Program.	Agree. A centralized information system that supports the Health Facilities and Capital Program (HFCP) and its management is a priority for the Business Support and Capital Division (BSCD).
	Restructure or replace RPMIS This recommendation is consistent with findings and recommendations associated with the Management Practices Review of the FNIHB Capital Contribution Agreement Process (Raymond Chabot Grant Thornton Consulting Inc, 2008) and the Audit of Capital Contribution Agreements (Audit and Accountability Bureau, 2010). Since that time HFCP has continually worked to find a solution for the Real Property Management Information System (RPMIS) within the constraints of a centralized Information Management System Division (IMSD) and the departmental Investment Plan (IP) process. To date, RPMIS has not been identified as a priority on the IP despite lobbying on the part of HFCP. In 2011-12, in an effort to assist the process and be prepared should RPMIS be identified as a priority, a Business Requirements Document was developed by HFCP with the expertise of an IT consultant. As a follow-up to this effort and to further emphasize the importance of the System: a Business Case will be developed, and, RPMIS will continue to be identified as a priority for action to IMSD and the request will be made that it be placed on the departmental IP and be funded at the earliest time possible.	RPMIS Business Case document (in-house, no cost).	Director, Business Support and Capital Division (BSCD), Business Planning and Management Directorate (BPMD), First Nations and Inuit Health Branch, (FNIHB), Health Canada, HC	March 2013
		RPMIS identified to IMSD as a priority for action and request for its inclusion on the departmental IP (cost to be determined).	Director, BSCD, BPMD, FNIHB, HC	March 2013
	Mandatory use of RPMIS Due to changing Program needs, RPMIS now has functionality limitations resulting in the use of the System being negatively impacted. Regions will continue to be trained on RPMIS and the importance of the System and its use by all regions will continue to be communicated, including through the recently revised HFCP Planning and Management Manual. In particular, the following future action will be taken: Contingent on RPMIS being restructured or an alternative system being developed, all RPMIS stakeholders/users will receive a training that includes a discussion on the roles and responsibilities of the system, including its mandatory use.	7 regional training sessions and training material to re-launch RPMIS or to introduce the new system, including a discussion on the roles and responsibilities of the system and its mandatory use (cost ~$15K). This is contingent on either RPMIS being restructured or an alternative system being developed	Director, BSCD, BPMD, FNIHB, HC	March 2014 This is contingent on either RPMIS being restructured or an alternative system being developed.
2. HFCP should explore ways to improve the consistency of regional capital project prioritization in an effort to better inform the LTCP process. These should be consistent with the new Capital Program Modernisation Framework.	Agree. Since the evaluation period (2005/06-2009/10), a significant policy change has occurred within the Program which has altered how capital projects are prioritized. April 2011, HFCP began implementing the Capital Program Modernization Framework (CPMF), an approach to managing health facility capital assets funded through HFCP. The primary objective of the CPMF is to promote more efficient, systematic and sustainable management of Health Canada funded health facility capital assets. Under this framework, priority for new construction, replacement and recapitalization is given to primary health needs, health facilities with the oldest effective age and the most vulnerable communities. As such, consistency of regional capital project prioritization has been improved. In addition, effective as of 2011-12, regional capital budgets were allocated based on a fix amount that reflects the region's previous five year allocation average. As a result, each region must work within their predetermined budget and prioritize their projects to respect this budget. It is anticipated that the CPMF and the new method for allocating regional capital budgets will continue to be implemented into the foreseeable future.	Full regional implementation of the CPMF principles (no additional cost).	Director, BSCD, BPMD (in association with Regional Directors, Capital Assets and Security [CAAS], Regions and Programs Branch [RAPB]), HC	March 2014
3. HFCP, in conjunction with FNIHB health programs and FN/I communities, should improve the processes with respect to O&M, including clarification of the roles and responsibilities and tracking of resources.	Agree. Improving the Operations and Maintenance (O&M) process, including communication and tracking is a priority for BSCD and regional HFCP stakeholders. Since the review period (2005/06-2009/10), the following measures have been taken to improve the O&M process: An O&M Guide was developed (currently in approval stages) to provide industry best practices and guidance on core building systems and equipment essential to facility operation to FN/I contribution agreement (CA) recipients. The Guide can assist recipients in meeting their obligation to develop health facility O&M management plans, which will help in clarifying the process and roles and responsibilities around O&M. Health facility O&M requirements and recipient roles and responsibilities were further clarified in the Set funding model CA template in 2010. Discussions between HFCP and the Health Funding Arrangements Division (HFAD) were held to address O&M financial coding and the need for it to appear as a budget line item in CAs. HFAD agreed to verify all CAs going forward for the inclusion of O&M and to discuss this issue with recipients if not included. This initiative provides information on current O&M practices and their cost, which is essential for benchmarking purposes, and clarifies roles and responsibilities. The recently revised HFCP Planning and Management Manual includes a section that discusses O&M, including providing clarification around the roles and responsibilities.	Approved O&M Guide and principles (no cost).	Director, BSCD, BPMD, FNIHB, HC	September 2012
		O&M business case for Senior Management Meeting consideration/ approval (no cost).	Director, BSCD, BPMD, FNIHB, HC	March 2013
		O&M Guide communication/ training material for CA recipients (cost ~$15K).	Director, BSCD, BPMD, FNIHB, HC	March 2013
		CA recipient communication/ training schedule (no cost).	Director, BSCD, BPMD, FNIHB, (in association with Regional Directors, CAAS, RAPB, HC)	March 2013
		Flexible and block CAs (multi-year work plan and health plan) that include clarification around O&M requirements and recipient roles and responsibilities (no cost).	Director, BSCD, BPMD, FNIHB (in association with Director, Health Funding Arrangement Division (HFAD), BPMD, FNIHB, HC	September 2013
	As a follow-up to previous measures and to further emphasize the importance of a well-defined O&M process, the following will occur: Approval of the O&M Guide and principles; An O&M business case will be developed, including review and analysis of current O&M funding levels; O&M Guide communication/training material to be developed for recipients; and, Refinement to CAs to clarify health facility O&M requirements and recipient roles and responsibilities.

Acronyms

ADM

Assistant Deputy Minister

BEC

Branch Executive Committee

BSCD

Business Support and Capital Division

CARC

Capital Allocation and Review Committee

CCAs

Capital Contribution Agreements

CPRC

Capital Program Review Committee

ECA

Environmental Compliance Audit

ECAFM

Environmental Compliance Audit Follow-Up Module

ESA

Environmental Site Assessments

FNIHB

First Nations and Inuit Health Branch

FN

First Nations

FN/I

First Nation and Inuit

GoC

Government of Canada

HFCP

Health Facilities and Capital Program

HQ

Headquarters

IFA

Integrated Facility Audits

LTCP

Long Term Capital Plan

MCCS

Management Contract and Contribution System

NNADAP

National Native Alcohol and Drug Abuse Program

O&M

Operating & Maintenance

RAPB

Regions and Programs Branch

R- LTCP

Regional Long Term Capital Plan

RPMIS

Real Property Management Information System

RSM

Regional Security Managers

TRA

Threat and Risk Assessment

Executive Summary

This evaluation covers the First Nations and Inuit Health Branch (FNIHB) Health Facilities and Capital Program (HFCP). This evaluation was undertaken in fulfillment of a Treasury Board commitment to evaluate the program and to meet the requirements of the Financial Administration Act and the Treasury Board's Policy on Evaluation (2009).

Evaluation Purpose, Scope and Design

The purpose of the evaluation was to assess the relevance and performance (effectiveness, efficiency and economy) of the HFCP. The evaluation covered HFCP activities related to health facility design, construction, renovation, operation and maintenance, as well as support for capital planning and management, for the fiscal years 2005-2006 to 2009-2010.

The methodologies used in the evaluation included: a document review, a review of databases, a financial data review, a survey of Health Canada staff involved with the HFCP, and in-depth interviews with key Regional and Headquarters (HQ) representatives.

Program Description

The mandate of the HFCP is to empower First Nation and Inuit community recipients in the area of capital investment and management to support their health infrastructure. HFCP objectives are:

• Funding capital investments to support the delivery of health programs and services on-reserve;
• Maintaining facilities and moveable assets to ensure that they are at an acceptable level of operation (in terms of both functionality and safety);
• Identifying risks caused by or affecting the HFCP and developing measures to reduce the probability and impacts of these risks; and,
• Increasing the capacity of community workers and health staff to manage facilities security and environmental issues.

The HFCP provides funds to eligible recipients for capital investments in First Nation and Inuit health facilities and associated lands through Capital Contribution Agreements (CCAs). Funding includes approximately $23.4M per year in A-base, plus sustainability funding of $22M per year. The funding supports a portfolio of 989 buildings, including more than 550 health facilities valued at approximately $1.2B. These health facilities provide the physical space and environment to enable over 600 First Nation and Inuit communities to deliver a variety of health programs and services ranging from primary care to health promotion and prevention.

Health Canada has no ownership or other legal interest in any capital assets on-reserve funded through the HFCP. However, the portfolio does include four off-reserve hospitals which the HFCP has funded through capital investments that are held by the Department.

Evaluation Conclusions and Recommendations

Relevance

The evaluation illustrated that the HFCP remains relevant to current Government of Canada priorities, Health Canada strategic outcomes, and federal roles and responsibilities for First Nation and Inuit health. It also concluded that the HFCP meets a demonstrable need among First Nation and Inuit communities by providing the space for a range of health programs.

Performance

Support for Project Management of Health Facilities

HFCP has adequately supported project management related to the construction and renovation of facilities. The program contributed to the knowledge and local capacity for capital project management of health facilities in recipient communities. However, there were some gaps in the communication of related guidelines and processes.

Allocation of Resources to Community Needs

The evaluation concluded that HFCP resources are being allocated to communities exhibiting health facility-related needs, the program is generally responsive to the most urgent needs, and physical upgrades to health facilities are improving. However, there are a number of factors that may be impacting the program's ability to allocate resources to the areas of greatest need, including: inconsistent incorporation of community and health programming needs into capital planning processes; differing capital project prioritization processes and interpretation of criteria across regions; and, the use of project selection partially based on the capacity of the First Nation/Inuit recipients to be able to manage the project.

Several factors make it challenging to assess whether the greatest needs are being met at the national level. For instance, regional differences and priorities result in inconsistent performance data and reporting regarding need, partially due to the inconsistent use of the national information system, Real Property Management Information System (RPMIS), which is seen as limited and not well suited to regional needs.

Adequacy of Operating and Maintenance (O&M) Functions

It was demonstrated in the evaluation that there are inadequacies related to O&M functions (clarity of roles and responsibilities, amount of money allocated and provision of on-going maintenance). There is not enough O&M activities being carried out in the communities to ensure sustainable operations, which may be accelerating the depreciation of facility components and increasing the need for major renovations and replacements over time. This is compounded by a lack of knowledge around O&M needs and processes both in programs and communities. These factors impact the degree to which communities have been able to effectively manage O&M.

Assessment of Economy and Efficiency

The evaluation concluded that cost-minimization policies and processes are in place. This along with the requirement to have an identified need for a capital investment illustrates that the program has implemented some sound program principles to help ensure economy and efficiency.

Recommendations

• Recommendation 1 -- HFCP should explore ways to either restructure or replace RPMIS and make its use mandatory. This should be done as a way of improving its utility as a cost-effective source of information for program management at HQ and Regional levels. The change should support the future evaluation of the HFCP, including the capacity to assess the economy and efficiency of the program.
• Recommendation 2 -- HFCP should explore ways to improve the consistency of regional capital project prioritization in an effort to better inform the national long term capital planning process.
• Recommendation 3 -- HFCP, in conjunction with FNIHB health programs and First Nation and Inuit communities, should improve the processes with respect to O&M, including clarification of the roles and responsibilities and tracking of resources.

Background and Context

This report presents a summary of findings, conclusions and recommendations from the Health Facilities and Capital Program (HFCP) cluster evaluation. The evaluation assessed program relevance and performance (effectiveness, efficiency, and economy) between 2005/06 and 2009/10 in accordance with the 2009 Government of Canada Policy on Evaluation, highlighting attainment of outcomes, lessons learned and best practices as well as challenges.

The evaluation report is organized into the following sections:

• Program description and evaluation context;
• Methodology used in the evaluation;
• Findings and conclusions; and
• Recommendations.

Program Description

HFCP provides funds to eligible recipients for capital investments towards FN/I health facilities and associated lands through Capital Contribution Agreements (CCAs). These health facilities provide the physical space and environment to enable FN/I communities to deliver a variety of health programs and services.

The mandate of HFCP is to empower FN/I recipients in the area of capital investment and management to support their health infrastructure. HFCP supports the construction, acquisition, leasing, expansion and/or renovation of nursing stations, health centres, health stations, health offices, treatment centres, staff residences and operational buildings. These activities provide First Nations, Inuit and FNIHB staff with the space required to safely and efficiently deliver health care services in First Nations and Inuit communities.

HFCP aims to provide First Nations and Inuit with access to a level and quality of health facilities similar to other Canadians living in similar locations, including those in isolated and remote communities.

In association with FN recipients, HFCP objectives include:

• Funding capital investments to support the delivery of health programs and services on-reserve;
• maintaining facilities and moveable assets to ensure that they are at an acceptable level of operation (in terms of both functionality and safety);
• identifying risks caused by or affecting the HFCP and to develop measures to reduce the probability and impacts of these risks; and,
• increasing the capacity of community workers and health staff to manage facilities security and environmental issues.

HFCP maintains facilities by funding repairs and renovations of these facilities to maximize operational life and minimize health and safety risks. Preventative and corrective measures are carried out to improve the working conditions and ensure compliance with building codes, environmental legislation, and occupation health and safety standards.

HFCP supports capital management and planning activities for the acquisition and operation and maintenance of facilities through the provision of guidance materials and direct assistance in designing, developing and implementing capital projects that are necessary to support the delivery of health programs and services.

HFCP supports a portfolio of 989 buildings including more than 550 health facilities valued at approximately $1.2 billion dollars. These facilities, along with moveable assets, support programs and services ranging from primary care to health promotion and prevention in over 600 First Nation communities. This includes front-line medical services such as community health nursing and dental therapy, as well as maternal and child health, mental wellness and healthy living programs.

Health Canada has no ownership or other legal interest in any capital assets (health facilities) on-reserve funded through HFCP. The capital contribution approach is beneficial to First Nations and Inuit communities in several ways including:

• Allowing First Nations to build the technical and administrative capacity to coordinate and sustain facility management activities that were previously managed by outside parties.
• Supporting the Branch mission statement: First Nations and Inuit people will have autonomy and control of their health programs and resources within a time frame to be determined in consultation with First Nations and Inuit People.
• Creating employment opportunities for members of local communities.

HFCP has funded a total of 989 facilities across the seven Health Canada regions (see Table 1 below).

Table 1 shows the numbers of facilities by facility type and region.

Table 1 - Number of Buildings by Building Type and Region ^{Footnote 1}

Region	Hospitals	Nursing Stations	Health Centres	Health Stations	Health Offices	Res	Op Bldgs	NNADAP Prevention Centres	NNADAP Treatment Centres	Total
AB	0	3	44	3	0	28	25	12	5	120
ATL	0	0	25	1	8	0	1	0	9	44
MB	2	21	10	0	28	56	53	0	3	173
ONT	2	20	39	43	24	95	35	0	7	265
PAC	0	10	23	87	12	29	1	0	5	167
QC	0	11	16	0	0	26	4	0	5	62
SK	0	12	66	1	0	56	14	0	9	158
Total	4	77	223	137	72	290	133	12	43	989

HFCP classifies facility types according to: population of community, availability of health services, transportation and infrastructure (see Table 2 below).

Table 2 - FNIHB Criteria for health facility classification (2005/06 - 2009/10)

Facility Type	Population	Isolation	Health Services	Transportation	Infrastructure
Nursing Station	Over 500	Remote/ isolated: Over 350 km to service centre	Nearest hospital more than 2 hours away, limited ambulance and first response services	No year round road access to other health care facilities	Limited community services
Health Centre (with Treatment)	Over 500	Non-isolated/ semi isolated: between 50 - 350 km from service centre	Nearest hospital by road in less than 2 hours; occasional unavailability of ambulance and first response services	All weather road/ air access; poor road conditions	Limited community services
Health Centre (without treatment)	Over 400	Non-isolated/ semi isolated: less than 350 km from service centre	Nearest hospital by road in less than 2 hours; occasional unavailability of ambulance and first response services.	All weather road/ air access; poor road conditions	Limited community services
Health Station	0 - 1000	Remote/ isolated or semi-isolated: over 350 km from service centre but within 50 km of health centre	Nearest hospital more than 2 hours away; limited ambulance and first response services	Accessible by air or road from FNIHB facility; poor road conditions	Limited community services
Health Office	0- 750	Non-isolated/ semi-isolated	Other health services available in nearby communities, adequate ambulance and first response.	All-weather road/ air access	Adequate community services

Program Funding

The estimated funding allocation details for the fiscal years 2005/06 to 2009/10 for HFCP are provided in the tables below:

Table 3 - A-Base Funding

Fiscal Year	Funding
2005/2006	$23,362,500
2006/2007	$23,362,501
2007/2008	$23,362,501
2008/2009	$24,250,797
2009/2010	$22,934,947
Total	$117,273,246

HFCP receives A-base (continuous funding) of approximately $23.4M per annum. In 2003, HFCP also began receiving sustainability funding at approximately $20.3M per annum mainly for:

1. assessment and remediation of contaminated sites and other environmental issues,
2. construction and remediation of nursing accommodations in remote and isolated communities, and
3. identification and correction of major health facility issues requiring repair or renovation.

However, a portion of that funding also goes to capital projects. It should be noted that only the A-Base portion of program funding receives a 3% annual index increase.

Table 4 - Sustainability Funding

Fiscal Year	Funding
2005/2006	$20,401,559
2006/2007	$20,430,291
2007/2008	$20,290,180
2008/2009	$20,399,700
2009/2010	$20,326,122
Total	$101,847,852

In addition, HFCP received pressure funding in 2006/2007 ($12,557,600), 2008/2009 ($19,714,125) and 2009/2010 ($24,635,553). HFCP also received specific funds during 2006/07 and 2008/09 fiscal years from the Federal Contaminated Sites Accelerated Action Plan (FCSAAP) - for the environmental assessment and remediation of contaminated sites.

Governance Structure

HFCP is managed at the national level through the Business Support and Capital Division (BSCD) which defines the main objectives of HFCP and strategic directions, plays a strong role in allocating funds and monitoring expenditures, and establishes national standards, policies and guidelines for regional and community partners. The BSCD also acts as the secretariat of the Capital Program Review Committee (CPRC).

The CPRC monitors HFCP project implementation and is composed of representatives from each region, as well as from the Office of Nursing Services, the Community Programs Directorate, the Primary Health Care and Public Health Directorate, FNIHB Financial Services Directorate and Corporate Services Branch Asset Management. The key responsibility areas of the CPRC include:

1. Recommending appropriate Branch input to the Departmental Long Term Capital Plan.
2. Reviewing and making recommendations to the ADM and Branch Executive Committee on capital and facility management proposals that require significant resources or that may have an impact on Branch activities and its relationship with First Nations and Inuit clients.
3. Reviewing Memoranda to Cabinet, Treasury Board submissions and other documents being developed.
4. Establishing new programs to determine the potential impact on the branch asset base as well as the capital and facility O&M resources.
5. Reviewing resource requirements and allocations for capital and O&M funding as part of the annual planning cycle.
6. Making recommendations to the ADM and BEC Sub-Committee on Finance, establishing a Branch Long Term Capital Plan Framework and recommending changes where necessary.
7. Identifying requirements for new policies and procedures and/or adjustments to existing policies and procedures for the effective, efficient and timely management of the Branch Long Term Capital Plan.
8. Review major capital construction projects that will require Treasury Board approval.
9. Establish methods and procedures for monitoring capital projects.

Capital Allocation and Review Committees (CARC)s are regional committees typically composed of a Facilities Management representative (Chairperson), a Program Medical Officer, Regional Nursing Officer, Resource Management representative and Zone Directors (if applicable). It is the responsibility of each region to develop their own regional models that include the community priorities of First Nations.

CARC is responsible for prioritizing, approving and monitoring regional FN/I capital projects, assessing risks and taking preventive and corrective measures where appropriate. Other responsibilities include:

1. Recommending the annual capital priorities and the Regional Long Term Capital Plan in accordance with the established reference level (envelope).
2. Developing criteria for the approval and prioritization of capital projects.
3. Reviewing and recommending approval of specific equipment purchases.
4. Reviewing the Long Term Capital Planning framework on an annual basis, including policies, standards, procedures, documentation requirements and criteria.
5. Identifying gaps in proposals and obtaining the necessary missing information.

FNIH Zone and Operation Directors are highly involved in the management and delivery of the HFCP and are ultimately responsible for the program being delivered in an effective, efficient manner that supports FN/I Programming.

Regional and Zone Facility Managers and Facility Officers, Capital Assets, Security and Administration are the focal points for the regional delivery of the HFCP. They are responsible for implementing capital contribution agreements, undertaking recipient risk assessments, providing technical advice to recipients, monitoring capital projects and managing capital contributions.

It is important to distinguish between the range of regional program partners that are involved in HFCP delivery, and the HQ component of the program titled HFCP. This evaluation uses the term HFCP to refer to all Health Canada program partners and activities, including those outside of the HQ Program and employed in the regional offices.

Project Prioritization

The CPRC and HFCP identifies a national set of criteria annually based on priorities such as facility age, type of programs delivered in each community, community isolation, target audience, etc. Regions then use these criteria to prioritize capital projects in their regions. The proposed regional projects from all regions are then discussed at an annual CPRC meeting whereby a national list of prioritized projects is developed using the same criteria for prioritization.

The prioritization of projects is a regionally driven process. It requires a holistic approach that balances the physical condition of a facility with an assessment of the community and program criteria and as such, relies on consultation with FNIH programming staff and FN recipients. As FN capital needs exceed available funding allocations, it is imperative that regions utilize fair and consistent approaches for the prioritization of projects.

The identification of projects, especially minor projects, is based partially on the prioritized recommendations stemming from the Environmental Compliance Audits (ECA) and the Integrated Facility Audits (IFA). The priority ratings used for the audit recommendations are based on potential harm to human health and safety - and/or the health of the environment, and/or regulatory non-compliance. The assigned priority given to each audit recommendation can also help prioritize the project should it be included on the regional list of capital projects.

The need for major renovations is assessed for the most part through direct assessment of facility condition, such as demonstrated threat to human health and safety, extensive deterioration that cannot be resolved through major recapitalization, or a significant lack of space that cannot accommodate program growth.

The preparation of a national Long Term Capital Plan Refresh (LTCP) is an annual Departmental requirement. Each region submits a prioritized regional Long-term Capital Plan (R-LTCP) which details all new construction, replacement, expansion, renovation/ recapitalization and environmental projects. All projects include a brief description and rationale that identifies the scope and magnitude of the project. The associated estimated project costs are spread over the number of years required for the project to be fully implemented and completed (design and construction).

HFCP consolidates all regional plans into a national LTCP roll-up which is used for the CPRC Recommended Funding Allocations Strategy. The LTCP takes into consideration all regional priorities and estimated project costs and assesses the aggregate requirements and priorities from a global perspective.

Capital Allocation Process

HFCP funding each year is generally based on historical expenditures and forecasts for requirements to meet ongoing program objectives. Capital allocations are determined by the national Long Term Capital Plan (LTCP), which is composed of two parts. The first part includes an outline of the operating environment and program requirements for a 5-year planning horizon that includes departmental mandates, corporate priorities, organization structure, strategic direction, capital asset inventory, capital funding sources and funding priorities. In general, the national LTCP demonstrates that Health Canada is aware of its capital assets, can associate these assets to approved programs, has identified the levels of capital funding required to sustain approved programs, and has established a mechanism to assign priorities, assess risks and ensure the viability capital assets for the duration of the programs they support.

The second part of the LTCP information on specific capital projects for each business and service line over a 5 year period, along with funding sources and information on capital expenditures valued at less than $500K. This includes a prioritized list of construction, recapitalization and environmental management projects by region, as compiled by Regional Facility Managers from input by program managers.

Other Program Activities

HFCP also received $65.9M in 2009/10 and $67.4M in 2010/11 from Canada's Economic Action Plan, which will assure the construction of over 40 new facilities, and the renovation of approximately 230 additional facilities by March 31, 2011. In the case of funding from Canada's Economic Action Plan, projects were selected from the HFCP Long Term Capital Plan by the CPRC. The selections were assessed by the Regions and Programs Branch (RAPB) and FNIHB according to risk and human resource needs prior to final approval. Economic Action Plan funding is planned and monitored at the project-level. National status reports on Economic Action Plan projects are compiled by HFCP monthly and quarterly.

Evaluation Context

Evaluation Objectives

The evaluation meets the program commitment to conduct an assessment of the relevance and performance of HFCP under the 5-year Departmental Evaluation Plan, and as required by the 2009 GOC Policy on Evaluation. The evaluation is in accordance with the requirements and standards set out in the 2009 Policy on Evaluation and its accompanying Standard on Evaluation - as well as the Health Canada Evaluation Policy. The evaluation covers HFCP activities related to health facility design, construction, renovation, operation and maintenance, as well as support for capital planning and management, for the fiscal years 2005/06 - 2009/10.

Evaluation Framework

Evaluation questions are based on the 2009 Policy on Evaluation core issues of relevance and performance. Relevance issues cover

1. linkages between HFCP objectives and federal government priorities and departmental strategic outcomes,
2. the roles and responsibilities of the federal government in delivering the program and
3. the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians.

Performance is defined as effectiveness as well as efficiency and economy of HFCP. Effectiveness questions assess the progress towards expected outcomes, as defined by the program logic model. Efficiency and economy refers to the assessment of resource utilization in relation to production of outputs and progress toward expected outcomes. Evaluation questions are specified as:

Table 5 - Core Issues and Evaluation Questions

Core issues	Description	HFCP Evaluation Question
Relevance	linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes	Does the HFCP address current Government of Canada (GoC) priorities?
		Does the HFCP address departmental strategic outcomes?
	roles and responsibilities of the federal government in delivering the program	Does the HFCP align with federal roles and responsibilities?
	assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians	Is there a need for the HFCP?
Performance: Effectiveness	progress toward expected immediate, intermediate and ultimate outcomes	To what extent has the project management of health facilities been supported by HFCP?
		Are resources allocated to communities with the greatest need?
		To what extent have environmental management processes improved?
		To what extent have security issues been identified and addressed?
		Have HFCP planning/implementation of physical upgrades of health facilities met the needs of communities?
		To what extent has life-cycle planning been supported by evidence-based data?
		Have the O&M functions been adequate?
Performance: efficiency/ economy	Assessment of resource utilization in relation to production of outputs and progress toward expected outcomes	Do the resources used for delivery approximate the minimum required to achieve the expected results?
		Have facilities been completed in time and on budget?
		Are there ways to improve the efficiency and economy of the program delivery?

Evaluation Methodology

The evaluation used multiples lines of evidence and the evaluation findings are based on the triangulation of evidence gathered from: document review, database review, financial review, survey and key informant interviews. A description of each data collection method is described below with its limitations.

Data Collection Methods

Document Review

The document review included approximately 300 Health Canada and GoC documents related to HFCP. These documents were identified by program area contacts, and well as through searches of departmental and GoC sources. They provided evidence of program activities and outcomes, as well as linkages to departmental and GoC priorities and mandates in support of program relevance. Examples include: budgets and expenditure information, relevant acts and policies, administrative records including meeting minutes, records of decisions, departmental program audits, HFCP capital project files, environmental assessments, Long Term Capital Plans (LTCPs), Departmental Performance Reports (DPRs) and Reports on Plans and Priorities (RPPs). External documents were also reviewed to provide program comparisons and benchmarking.

Database Review

The database review entailed the examination of performance information that was drawn from three databases:

1. Real Property Management Information System (RPMIS),
2. the Environmental Compliance Audit Follow-up Module for RPMIS (ECAFM) of the RPMIS, and
3. the Management Contract and Contribution System (MCCS).

Table 6 - Indicators Relating to HFCP-Specific Outcomes

Database	Indicators
Real Property Management Information System (RPMIS)	Number and type of health facilities characterized by building life-cycle status Percent increase or decrease in health facilities that meet physical upgrade requirements/standards Evidence that communities in greatest need of upgrades/ capital projects are beneficiaries of HFCP capital funding Number of unusable health facilities as a result of degradation
Environmental Compliance Audit Follow-Up Module (ECAFM) of the RPMIS	Percentage and type of remedial actions, based on environmental non-compliance, planned by year Percentage and type of remedial actions, based on environmental non-compliance, addressed by year Number of unusable health facilities as a result of degradation Percentage increase/ decrease in remedial actions addressed to satisfy environmental requirements
Management Contract and Contribution System (MCCS)	Number and type of health facilities by agreement type (set/transitional) that have implemented physical upgrades by year Number and type of health facilities completed within expected time-frames

The Real Property Management Information System (RPMIS) contains the following elements and information found within each respective element:

• Facility Profile Database: basic information and O&M information for 989 health facilities
• Facility Inspection and Condition Reports Database: building condition reports that describe the state of health facilities, maintenance needs and any non-compliance issues. This is used to prioritize actions, using degree of degradation, as well as percentage and type of remedial actions planned and carried out for 466 health facilities; and
• Recap Allocation Model Update list: information on the effective age of 792 health facilities and residences.

The ECAFM (which is a module of the RPMIS) provides a priority rating based on potential harm to human health and safety - and/or the health of the environment, and/or regulatory non-compliance.

The Management Contract and Contribution System (MCCS) provided indicators of time and budget amendments by type of project - i.e. construction, renovation, expansion, security upgrade and environmental remediation.

Financial Review

The financial review included a review of budget information, allocation of funds, expenditures, and reasons for variances. The analysis focused on identifying areas where efficiencies have occurred and where additional efficiencies could be found. Data were collected and analyzed based on budgeted resources (funds and FTEs) and actual resources in relation to HFCP outputs. The Facility Profile Database was used to collect information on facility-level O&M financial information.

Survey

A survey was conducted to gather the opinions, perspectives and experiences of a large and diverse sample of individuals on the operation of HFCP. These individuals included regional director generals, regional directors, and facility officers from the region as well as several staff from the National Capital Region. Contact information for potential survey respondents were constructed based on a convenient sample of those with specific knowledge or experience about specific program components. The list was created by the evaluators using the Government Wide Employee Directory and Health Canada organization charts and verified by the Evaluation Working Group. The survey was initiated with an invitation email detailing the purpose of the evaluation and the information to be captured by the survey. French and English surveys were emailed to 66 HFCP employees, and 28 completed surveys were returned, for an overall response rate of 42%. Below are the number of survey participants per category.

Table 7 - Number of survey participants per category

Category	Number surveyed	Response
National Headquarters/FNIHB
Staff	6	3
Regions/RAPB
Regional Director Generals	6	1
Regional Directors	12	5
Facilities Managers/Officers	44	19
Total	66	28

Interviews

Interviews were intended to explore and investigate issues and questions that came from survey results as well as address any outstanding issues that are not answered completely by other methods. The interview list was constructed in a similar fashion as the survey participant list: individuals were first identified for their anticipated level of knowledge and experience. After an initial analysis regarding the regional breakdown of the inventory, it was decided that a regional approach, for strategic and logistical purposes, would be required for the interview portion of the evaluation. A regional approach was used in deriving the interview list, due in part to the substantial size of the facility inventory (with a total of 790 buildings represented); and the diversity of the inventory in terms of the degree of isolation (Remote Isolated, Isolated, Semi-Isolated and Non-Isolated); and the type of facilities represented (Hospitals, Health Centres, Health Offices, Health Stations, Nursing Stations, Nurses Residences, NNADAP Treatment and Prevention Centres).

A total of 36 interviewees were contacted, and 27 (75%) of these were administered a 30-60 minute semi-structured interview in French or English. Additional details are in the table below which outlines the number of interview respondents by category.

Table 8 - Number of interview respondents by category

Category	Number of Respondents	Response
National Headquarters/FNIHB
FNIHB Senior Management	2	2
Staff	6	6
Regions/RAPB
Regional Director Generals	3	1
Regional Directors	6	4
Facilities Managers/Officers	10	8
Office of Nursing and Primary Care
Staff	2	2
Indian and Northern Affairs
Staff	1	0
Community Programming
Staff	1 per program area, 6 total	4

To assess specific questions regarding security issues, Regional Security Managers (RSMs) were also interviewed. Five out of the 6 participated in semi-structured interviews regarding the status and maintenance of security and safety at the health facilities. Information gathered from RSMs include number of threat and risk assessments identified by year, percent and type of threats and risks resolved by year, percent change in numbers of health facilities that meet security requirements and evidence that minimum security requirements are integrated into the construction of new facilities.

Limitations

The main limitation was the information gaps in the available performance data. In particular, the RPMIS and the MCCS, when used, were used inconsistently across regions and therefore were unable to provide firm indicators of performance nationally. This challenge was mitigated through reliance on interviews with key informants who reported on anecdotal evidence and recent projects to provide another line of evidence.

The evaluation was also limited by the reliance on a relatively small number of respondents who worked closely with the program. While there was a potential for response bias from interviewees with a vested interest in the program, probes and follow-up questions during interviews were used to establish concrete examples, and to assess the validity of responses. Overall, it was noted that respondents tended to provide information that appeared complete and balanced. In addition, document review data was used to triangulate and corroborate interview responses where possible.

Performance - Relevance

Does the HFCP address current Government of Canada (GoC) priorities?

HFCP is aligned with the universality and accessibility principles of the Canada Health Act - which Health Canada has the direct mandate to implement. The federal government supports health services in First Nations and Inuit communities where provincial services are not available. HFCP supports this through the provision of capital contribution agreements to eligible First Nations and Inuit communities for the construction, renovation and operation of a portfolio of facilities and moveable assets necessary to provide health services.

HFCP has continued to remain a government priority, as evidenced by the influx of money mentioned in recent budgets. Budget 2008 included $10.1M to ensure basic maintenance of existing health facilities, allow required environmental assessments and remedial measures to be employed, pay for minor capital expenditures and provide additional facilities management resources in First Nations and Inuit communities. Budget 2009, under Canada's Economic Action Plan, included $135M to accelerate infrastructure investments by supporting over 40 new major projects and 131 minor renovation projects that were previously identified as part of the HFCP Long Term Capital Plan. The release of funds supports HFCP's long-term outcome of modern and well maintained health care facilities and residences that support health program delivery for First Nations and Inuit communities.

Does the HFCP address departmental strategic outcomes?

HFCP provides support for the Health Canada strategic outcome of "Accessible and Sustainable Health System Responsive to the Health Needs of Canadians". The program aims to increase the local accessibility of health services through the provision of adequate space and equipment to allow health programming to take place in local communities. By reducing the travel required to access health services, HFCP is also contributing to the sustainability and responsiveness aspect of this strategic outcome.

The departmental strategic outcome of "Better health outcomes and reduction of health inequalities between First Nations and Inuit and other Canadians" is aligned with HFCP outcomes in that the provision of funding, through capital contribution agreements, for capital projects provides the physical platform through which FNIHB programs such as community-based health promotion and protection, primary care and emergency services are delivered. Also, Health Canada's Departmental Performance Reports (DPRs) and Reports on Plans and Priorities (RPPs) have consistently included HFCP performance indicators in order to support this strategic outcome. The two performance indicators are: "Number of constructed, expanded, and completed recapitalized health facilities", and the "Amount of funds invested in environmental compliance and assessment of health facilities."

Does the HFCP align with federal roles and responsibilities?

HFCP is aligned with federal roles and responsibilities for health programming as defined by the Canada Health Act, which specifies that that maintaining and improving health requires continued access to quality health care. The Act also indicates that the improvement of access to health care for FN/I communities remains a mandate of the federal government, and is consistent with the roles and responsibilities of the federal government.

Is there a need for the HFCP?

The provision of capital planning support and services through HFCP addresses the need for local health care services, and meets Branch and departmental mandates and objectives of enhanced community capacity for governance and self-determination in the area of health in FN/I communities. In addition, in order to assess whether there continues to be a need for new or modified health facilities and equipment, the evaluation considered whether HFCP facilities are meeting the needs of Health Canada programs, as well as the number of HFCP facilities that are identified in need of remediation.

While the Department has increased programs in the areas of health promotion and disease prevention in recent years, most HFCP facilities were constructed for other purposes. As a result, community health promotion and related programs are typically housed in nursing stations. According to recent assessments of national space standards, the average facility is 332 sq metres, which is undersized by about 30% in relation to program space requirements^{Footnote 2}.

Similarly, the number of facilities in need of remediation, based on annual audits and assessments, is identified and tracked through two sources. The first is the Environmental Compliance Audit Follow-up Module (EACFM) of RPMIS, which lists buildings that have environmental issues ranging from those that pose the least risk (priority 4) to those that pose the most risk (priority 1). In total, there were 102 facilities across Canada that had a priority 4 environmental issue. Sixty-eight of these also had a priority 3 issue, 54 had a priority 2 issue, and 9 also had a priority 1 issue (see table below for regional breakdowns).

Table 9 - Number of facilities per region that have one or more environmental issue identified

Region	Type of Environmental Issue
	Priority 1	Priority 2	Priority 3	Priority 4
British Columbia	0	5	6	12
Alberta	2	5	7	7
Saskatchewan	4	8	6	8
Manitoba	0	7	4	22
Ontario	2	8	8	8
Quebec	1	11	22	30
Atlantic	0	10	15	15
Total	9	54	68	102
(ECAFM 2005 - Sept 2010, retrieved on Aug 27, 2010)

The second system is the Real Property Management Information System (RPMIS) which contains data on effective age and Facility Inspection and Condition Reports. Facility Inspection and Condition Reports data provides information related to the condition of the facilities through inspection reports. This informs the state of health facilities, given the condition of the facility and the maintenance needs. The following table demonstrates that on average 20% of the inventory has an age in excess of 20 years (see table below).

Table 10 - Number of facilities with average effective age of 20+ years by region and the regional percentage of facilities with an average age of 20+ years

Region	Number of facilities with an effective age >20 years (n = 161)	Percentage of regional inventory with an effective age >20 years
British Columbia	36	22%
Alberta	18	19%
Saskatchewan	19	13%
Manitoba	25	24%
Ontario	44	26%
Quebec	10	16%
Atlantic	9	19%
Average	23	20%
(RPMIS on Oct, 2010)

Therefore, the data suggests that the current facilities complement is in need of updating in order to continue to provide the necessary space for community health programming.

Overall, HFCP continues to be aligned with both Government of Canada priorities and the Departmental strategic outcomes. Evidence indicates that the program addresses a demonstrable and on-going need in the communities to provide space for health programs and services. There is also a demonstrable need for the program to provide minor capital funding in order to maintain the functioning of the buildings and to address issues related to degradation.

Performance - Effectiveness

The effectiveness part of the performance section of this report will be presented according to the evaluation questions in order to guide the reader.

To what extent has the project management of health facilities been supported by HFCP?

The evaluation found that the support provided by HFCP as per its role with respect to the construction and renovation of facilities has been adequately implemented. The direct involvement of community members, and the support provided by regional Health Canada employees has likely increased the knowledge and capacity for capital project management in recipient communities. The CPRC plays a key role in providing communications, and in determining the appropriate content and specific audience for these communications within the communities. However, it was also found that the regional representatives involved in capital projects and renovations were not always fully aware of the content of guidelines and documents, or of the overall requirements that are in place.

Are resources allocated to communities exhibiting the greatest need?

As a way to guide the assessment of needs based on level and type of programming in the communities, FNIHB has developed the Community Health Needs Assessment Guide in 2000. In theory, information obtained through these assessments feeds into Community Health Plans, which are then referred to by program areas to determine programming needs. Programming needs identified through a Community Health Plan may result in the identification of a capital investment need (and facility space) which will then be included in the regional long term capital plan. However, interviews revealed that not all communities develop a needs assessment or community health plans either due to lack of interest or lack of capacity in the community, making the "needs" data that feeds into the regional capital plan inconsistent.

The process regions use to prioritize projects varies across regions. Respondents indicated that several different allocation mechanisms were used which varied by region. For example, Saskatchewan region assigns points to projects, and the Manitoba region solicits the various program stakeholders (Nursing, Community Programs, etc) to identify projects; and in Alberta, projects over $500K are managed through a co-management committee. Although these mechanisms vary across regions, these regionally defined processes are an important for making the programs community-based, and for advancing the Branch objectives of self-determination. The alternative of having a nationally prescribed or uniform process may detract from local aspirations and preferences for locally defined priorities through the current planning process.

However, interviewees noted numerous communication challenges and disconnects regarding the contents of community health plans and facility needs. Data from interviews revealed that identification of 'health facility needs' occurs at community, regional and national levels but doesn't always include those involved with capital planning. This suggests that the identification of health programming needs is inadequately linked to the capital planning processes and that the availability of health facilities may not be aligned with programming needs.

Proper alignment of property holdings with program needs is an ongoing area concern. The 2007/08 HFCP Management Action Plan notes the importance of "Finalising and implementing a real property management framework including internal policies and processes, accountabilities and performance measurement, which should provide a foundation to align health facilities with program needs". The document review also revealed concern among CPRC members that the analysis of projects needs to be more precise, and more consideration needs to be given to specific location relevant factors.

In addition, according to recently published audit documentation from Audit and Accountability Bureau, health facility projects are also selected and approved based on the relative capacity of FN/I recipients; rather than their long term health programming and facility goals. Audit documentation revealed that a significant aspect in support of project selection and approval (into LTCP) included recipient capacity and readiness assessments, which are comprised of recipient strengths and challenges in financial, administrative and health programming areas as well as signatures by individuals holding appropriate delegated signing authorities. Community and regional health needs, demographic indicators or extent of building degradation are not explicitly referenced. Recipient capacity and readiness as selection criteria may favour communities with adequate capacity, and exclude those with lower capacity, even though they may have a greater need.

Survey results indicated that the HFCP is usually responsive to the most urgent needs. Interview results substantiated this finding. When asked to list issues or challenges encountered with the facilities, a large majority of interviewees cited limited resources as the major challenge.

In general, evidence suggests that HFCP resources are being allocated to communities exhibiting health facility-related needs. However, there is no indication of whether these needs are the greatest of all FN/I health facility needs across the country due to the lack of inventory data and the inconsistent way in which communities and regions identify "need" and determine allocation priorities. Overall, the current process allocates funds to those with greatest need and works to support capacity for those interested in receiving them.

To what extent have environmental management processes improved?

HFCP has sought to address environmental management, including changes to environmental regulations and standards on reserve through improvements in tracking the implementation of environmental audit recommendations, program documentation, and awareness building.

It is important to note, that while Health Canada has worked to address changes to environmental regulations and standards, the system of environmental regulation on reserve, is notably weak across all programs. A performance audit of Land Management and Environmental Protection on Reserves revealed that Indian and Northern Affairs Canada and Environment Canada had "not established a regulatory regime that protects the environment of First Nation reserves that is comparable to the regime provided by provincial laws"^{Footnote 3}.

Health Canada's roles and responsibilities regarding environmental issues stem from its role as funding authority, which allows the Department to assist bands in the clean-up of their health facility related lands and to provide advice and guidance related the prevention and remediation of environmental issues when requested. The FNIHB environmental audit program was designed to also conduct Phase 1 Environmental Site Assessments (ESAs) of buildings funded by the Department.

As mentioned previously, environmental issues, identified through the annual audits, are rated as priority one if they are serious and the least serious are rated as a priority four. The Environmental Compliance Audit Follow-up Module (ECAFM) indicates that all priority one issues have been dealt with. However, it is not known whether all priority two issues have been addressed. It is indicated in ECAFM that total funding required to effectively deal with the 105 priority twos, is $1,439,600.00. Progress in dealing with priority threes and fours is detailed in the table below, only 1.5% of priority 3 and 4 actions have been completed to date. As this table illustrates, according to the ECAFM database, there are very few addressed/completed and in progress actions, (157/2874 = 5 %) in comparison to the total number of identified priority threes and fours.

Table 11 - Progress in addressing environmental priority 3 and 4 ^{Table 11 footnote 1} recommendations contained in ECAFM

Status	Priority 3	Priority 4s	Total
Addressed through ongoing actions	0	2	2
Completed	38	4	42
In progress	96	17	113
Proposed	1991	726	2717
Total	2125	749	2874
Table 11 footnotes Table 11 footnote 1 Note: These counts include all issues for each building, and so data may count buildings more than once. Return to table 11 footnote 1 referrer

Other program documentation indicates that environmental issues related to repairs and maintenance in nursing stations and accommodations are monitored^{Footnote 4}. Issues highlighted include: repairs and maintenance in nursing stations and accommodations; and inadequacy of training and supervision of custodial staff at some nursing stations. Risks include potential violation of the Canada Labour Code, health risks due to infection control, and issues identified as conditions in Contribution Agreements. However, it should be cautioned that many environmental compliance issues with respect to legislations are not enforceable by the federal government on reserve, and need to be monitored and carried out by the owners of the buildings in the communities.

CPRC tracks the follow-up actions related to larger environmental issues such as mould and asbestos issues. As such, this excludes the majority of recommendations stemming from the audits as many are addressed through simple changes in practice or require very limited O&M funding to implement, for instance, changes in storage practices for hazardous materials or installation of a light fixture at the entrance to a building.

Minor environmental issues costing less that $1500 are addressed through regular facility operations and maintenance provided by programs, while more extensive issues costing more than $1500 are addressed as a minor project funded through a Capital Contribution Agreement. The evaluation noted that some environmental problems were caused by a lack of operations and maintenance activities.

HFCP has also supported the work of Environmental Officer/Specialist positions in the regions, which assist mainly with environmental matters, largely related to audits and other assessments or studies, such as on-going regional follow-up and implementation of remedial actions, as well as internal auditing and preventative programs.

Program documentation reveals that additional capital funding was sought from Cabinet during the period 2005-2010, to address environmental issues at HFCP funded facilities, and to restore compliance with legislation and codes applicable to these facilities. As mentioned previously, while environmental compliance audits have been completed on a good portion of the infrastructure in each region, several interviewees noted that there are insufficient resources to both track and address follow-up actions and remedial work.

In summary, environmental management processes associated with the HFCP have been able to address the more serious environmental issues (priority one issues) but priority 2, 3 and 4 issues are less clear.

To what extent have security issues been identified and addressed?

The role of HFCP at HQ in the area of security includes:

1. providing security guidelines during design and construction phases of a project, as well as during major renovations and security-related upgrades;
2. funding security services at a limited number of "legacy" facilities;
3. working with Regional Security Managers (RSMs), who are funded through a different program.

This work is conducted in conjunction with the Capital Assets Security Division of the Regions and Program Branch of Health Canada.

The creation of the HFCP Safety and Security Policy in 2005 has functioned to provide the platform from which to more effectively address security issues in FNIHB-funded health facilities. Notably, this Policy is addressed to all FNIHB employees who manage or work in FNIHB funded health facilities, and is intended as a resource for ensuring that these employees have the appropriate tools and knowledge to identify threats and risks and to put in place effective safeguards^{Footnote 5}.

When asked to assess the effectiveness of HFCP in follow up actions in health facilities, the majority of RSMs found the program to be somewhat effective. RSM interviewees indicated that contribution recipients tend to experience high staff turnover, particularly of security guards, making it difficult to maintain capacity and training. Newly contracted nurses, are also sometimes unfamiliar with their roles and responsibilities in safety and security activities. RSMs indicated overall that resources to implement Threat and Risk Assessment (TRA) recommendations are limited, and this has impacted the ability to change or improve the security situation over time. Again, it is the responsibility (for the most part) of the programs using facilities to provide security-related resources through their O&M expenditures.

Have HFCP planning/implementation of physical upgrades of health facilities met the needs of communities?

The need for renovations to health facilities is defined by the programs that are delivered in each community, and this forms part of the basis for project approval. However, given that projects are also allocated funding on the basis of community capacity to deliver and manage a project to completion, and there is limited funding for major projects overall, some specific needs are likely not being met. On the other hand, there is no indication from any source that either major or minor renovation projects are being provided in cases where there is no need.

Staff interviewed during the evaluation conduct expressed the opinion that they are doing as much as they can with available resources. Other regions noted that a main issue for the large number of renovations requested is the lack of O&M expenditures by the programs using the facilities. An accepted standard for O&M expenditures is 2.5% of asset value per year, which would amount to approximately $22M per year for facilities funded by Health Canada. However, the level of O&M expenditures by program occupants of facilities is unknown, and the condition of buildings as impacted by lack of these expenditures is not assessed consistently. As a result, it is difficult to provide accurate assessments of the adequacy of O&M.

The general perception among HFCP personnel is that upgrades have been improving, and that staff has been dedicating increasing amounts of attention, time and funding to improve the situation.

To what extent has life-cycle planning been supported by evidence-based data?

Long-term investment planning needs have recently increased the need for a database system that keeps accurate records of the state of the health facility inventory and related assets. The evaluation found ongoing issues with the Real Property Management Information System (RPMIS). RPMIS was originally intended to store all facility-related information regarding O&M and real property data, to assist in decision-making at the level of the regions, and headquarters. It was noted by the CPRC in the 2004 formative evaluation of HFCP the system was not well-suited to regional needs, and that alternate systems were sometimes being used instead^{Footnote 6}.

Upgrades made to RPMIS between 2005-2008 sought to improve tracking of facility conditions and of inspection reports. However, a review of the system in 2008 found that the quality and completeness of data in the facility profile portion of RPMIS varied regionally and in some cases was out of date.

In addition, the formative evaluation of HFCP noted that the absence of more up-to-date information "negatively impacts on the ability of managers" to determine the adequacy of program funds to meet current and future demand. In addition, the recent Management Practices review criticised the system for not incorporating information on demographic changes or local circumstances. The 2010 Health Canada Audit of Capital Contribution Agreements similarly concluded that RPMIS was too limited, and contributed to unrealistic cost estimates. The survey of regional and HQ staff conducted for the current evaluation found that 15 out of 27 did not use RPMIS on a regular basis, and 12 out of 20 reported that the least useful component of the system was the Facilities Cost Estimates System.

Therefore, the evidence base provided by the RPMIS and other databases is incomplete and incomprehensive to provide the necessary information for decision-making.

Have the O&M functions been adequate?

The evaluation found that, while the total square metres of HFCP funded facilities continues to grow annually, and the cost of construction increased by 3-5% between 1997 and 2008, the likely expenditures on O&M had not increased significantly between 2004 and 2008^{Footnote 7}. A lack of O&M funding was cited by a focus group as far back 2004 as an impediment to ensuring sustainability of operations in many health facilities across the country. While the issue of insufficient O&M has been cited over time as a significant problem, O&M expenses have additionally been impacted more recently by increases in construction and energy costs.

Respondents in the current evaluation similarly indicated that O&M plays a key role in sustaining buildings and ensuring that they are operational over time. They agreed that O&M for HFCP funded facilities could be better managed to more effectively sustain modernized space, and indicated that current levels are not adequate. They gave various estimates of the gap. For example, while current O&M is approximately 3% of capital expenditures, they noted that the need is closer to 6 to 8%. Some estimated that there is a $10 to $15 million shortage for utilities, a $10 million shortage for minor repairs and a $25 million shortage in O&M overall. One individual noted that is nearly impossible to determine what funding is allocated to O&M due to a severe lack of breakdown in financial coding.

Program documents indicated an ongoing deficit for O&M, which in 2010 was estimated at $10 million at sub-optimal levels (i.e. funding required to just meet core facility O&M expenses) and $25 million at optimal levels (i.e. funding required to meet core facility O&M expenses, while also funding preventative facility programs requirements and implementing an optimal audit cycle). As a consequence, the deficit may accelerate depreciation of facility components and the need for major renovations and replacement over time. This produces other risks of operation, such as health risks due to infection control and building security. However, it should be noted that these estimates only capture the O&M that is tracked. The extent of actual O&M provided by programs is largely unknown.

Community members indicated that most were unaware of how O&M funds are allocated, and program representatives in the communities reported that they were unaware of the levels of community involvement in the O&M process. They either indicated that O&M funds are not included in Contribution Agreement requirements, or were unaware of recent changes to these requirements.

The interviews also indicated a need to clarify the expectations around O&M management plans, O&M schedules, and the funding formula that includes the roles and responsibilities of programs and communities for O&M activities. Interviews also indicated a need to improve band council governance and accountability with respect to HFCP funds. Some suggested that communication between communities, regional offices, FNIHB and HQ could be improved.

The overall functioning of O&M within HFCP has been impacted by two main factors: the levels of O&M expenditures, and the degree to which communities have been able to effectively manage O&M overall, especially given the lack of clarification around who is responsible for both funding and the provision of O&M activities. There is also some indication that some communities lack an adequate awareness of roles and responsibilities, knowledge, human resources and long-term commitment to carry through with O&M functions.

Performance - Efficiency and Economy

The evaluation was able to assess whether the overall processes in place are designed to minimize costs of outputs where possible, and whether the findings identified are indicative of practices that support the efficiency and economy of HFCP. Therefore, the assessment of efficiency and economy focussed on the overall resource utilization for the production of outputs in relation to the progress towards outcomes. Significant gaps in the available data presented a barrier to assessing the efficiency and economy in a quantitative sense, and qualitative reports and documents showing processes were used instead.

Do the resources used for delivery approximate the minimum required to achieve the expected results?

The document review showed that a number of formal mechanisms are in place to control expenditures for building construction. The process for needs assessment requires that project expenditures are aligned with main program outputs and outcomes. Successful capital proposals require that recipients demonstrate that the project will be managed economically, and that it will serve to meet an identifiable need in the community. Each capital project is subject to a competitive process that awards construction contracts according to the least costly proposal meeting project requirements (best value for money principle).

A number of policies and processes work to assure that costs are minimized. At the project implementation level, procedures are in place to promote completing capital projects on time and within budget. However, the current time and budget estimates that are available may not be reliable enough to assess whether these cost-minimizing procedures are effective.

At the policy and planning level, HFCP works to improve the efficiency of expenditures by allocating funds to communities in need. This follows the health policy-based assumption that the greatest uptake of a service will be among those with the greatest need, and that the greatest health impacts will be realized when interventions are provided for those with the greatest need.

In addition, the document review showed that projects are managed to completion according to a series of cost-minimizing practices, such as making dispersals contingent on meeting target dates, setting requirements, audits and approvals for cost over-runs, and conducting quality control assessments that work to assure adequate goods and services are delivered by vendors. In addition, contribution agreements prohibit stacking of funding, which works to assure that HFCP funds are not used when other funds are available. Final expenditure reports identify budget surpluses that must be returned to the program, and all budget deficits must be covered by the recipient. Finally, all funds, including those from other sources, are held in a separate account from which they can be dispersed according to schedule, and where interest on balances are retained for the purposes of legitimate project expenses.

While the indicators for minimizing capital cost could include ratios of dollar per square foot of new or renovated space - either at the inventory or individual health facility level - the data for this were not complete, and would only provide crude measures that could not be separated from factors such as the higher costs of construction in remote locations, and the lack of competitive pricing due to low numbers of contractors. Moreover, Government of Canada funded capital construction procurement processes are limited to competitive processes, and the need to prioritize First Nation bidders. Due to these constraints around contracting policy and the remoteness of many communities, the cost per square foot is not a reasonable comparison measure.

The perspective of economic analysis in the case of HFCP design follows life-cycle planning assumptions. In other words, on-going investments in O&M of buildings are needed in order to minimize degradation as a barrier to proper use of the facility, or to keep them operational for the full duration of their life-cycle. As a key indicator of the long-term efficiency of capital infrastructure, and a main element of the program logic for HFCP, this dimension of efficiency was a focus of the evaluation. Some interviewees indicated that planning and implementation of O&M activities by communities may be less than required to assure that upkeep and degradation do not shorten the life-cycle of some facilities potentially jeopardizing efficiencies related to longevity of buildings.

Have facilities been completed in time and on budget?

The databases for recording performance of contribution agreements related to time and budget are incomplete, and do not support specific quantitative indicators. Therefore, a non-representative sample of Contribution Agreements between 2005-06 and 2009-10 was drawn and linked to Departmental financial data for the purposes of making rough estimates of time and budget performance. The agreement activities listed in this sample included:

1. repair, renovations or upgrades to capital,
2. minor capital projects,
3. security services,
4. operations and maintenance,
5. equipment or assets,
6. design and construction of capital.

Of the 128 agreements sampled, 96 (75%) had a budget amendment, and 25 (20%) had a time amendment. 22 of the 128 agreements (17%) had neither a budget nor a time amendment.

It would appear that many capital projects have had amendments to time and budget. However, without adequate benchmarking, it is difficult to assess whether these rates and delays are comparable to other capital projects, or whether these delays or modification in budgets are within the norm of construction processes in general.

Are there ways to improve the efficiency and economy of the program delivery?

A number of key informants reported that additional efficiencies could be gained through improved inspection and maintenance, including potentially dedicated building inspectors, who could be engaged through partnerships with other organizations. They also indicated that improved communication and adherence to roles and responsibilities in various areas of the HFCP would increase the efficiency of the program overall. It was suggested that this could be done initially through the processes and procedures outlined in the general Contribution Agreement.

The lack of financial data that could be linked to outputs limited the capacity of the evaluation to assess the efficiency and economy of program delivery. It is notable that the Department of Aboriginal Affairs and Northern Development 2010 evaluation of its Capital Facilities and Maintenance Program found similar gaps in program data to show efficiency and economy, particularly to provide inter-regional comparisons that would support benchmarking. On the other hand, this evaluation also worked to include benchmark information from municipalities similar to First Nations communities.

It is noteworthy that municipalities also showed significant gaps in reporting and data for assessing efficiency and economy. As a result, the evaluation relied on the same types of policy and qualitative assessments as the HFCP evaluation to conclude that cost control measures appear to be in place. Overall, the evaluation found sufficient evidence to demonstrate the efficiency and economy of HFCP with respect to resource utilization and the production of outputs and progress toward expected outcomes. However, the evidence also indicated strongly that gaps in O&M threaten both the health policy outcomes and long-term efficiency of infrastructure capital investments.

Conclusions and Recommendations

Conclusions

Overall HFCP remains relevant to government and HC priorities, to federal roles and responsibilities in health, and to the needs of FN/I communities. HFCP contributes to FNIHBs strategic outcome "Better health outcomes and reduction of health inequalities between First Nations and Inuit and other Canadians" by funding the construction and upgrades of health facilities in FNI communities. By funding capital investments to support the delivery of health programs and services on-reserve, HFCP improves the access of these populations to the health system, which supports Health Canada strategic outcome: "Accessible and Sustainable Health System Responsive to the Health Needs of Canadians". The objectives of the program to address local needs for delivery of health services supports the relevance of the program, and the prioritization process for capital investment and upgrades suggests that the program is responsive to local needs. The focus on environmental issues is also supported by an indication of need, as shown through environmental assessments.

The assessment of performance was limited due to the lack of available data. This impacted the evaluations ability to assess HFCP in accordance with the logic model outcomes. Data systems are not comprehensive, incomplete and not used consistently. Overall, the current database does not meet current needs, including its limitations in assessing the true state of the inventory. While there is a clear need for a centralized inventory system that identifies issues such as the state of facilities, the evaluation found consistently that the current RPMIS data system is not functioning effectively.

On balance, evidence suggests that HFCP is challenged in addressing community priority needs for infrastructure and/or renovations. There is indication that the prioritization of need is challenged by a lack of common definition of the set criteria. As a result, there are situations where the selection of communities to receive HFCP funding is influenced by the capacity of the community to manage the project rather than solely on a demonstrated need for infrastructure.

There is some indication that the O&M process is not adequate to maintain the current inventory. While the approach to project funding works to minimize costs and ensures efficient allocation of O&M resources, there is inadequate benchmarking to conclude this with certainty. The potential lack of adequate O&M suggests that the life-cycle efficiency of facilities may be compromised. However, this assessment is limited by the fact that the depreciation of the inventory is largely unknown. The O&M issue is further compounded by the influx of stimulus funding, given that no subsequent funding has been allocated to maintain those facilities. The main barrier around the issue of O&M funding has been a lack of clarity among key partners with respect to the specific roles and responsibilities for O&M.

The evaluation found inconsistencies in the understanding of some HFCP procedures, policies and priorities among key partners. This included differing awareness regarding roles and responsibilities for O&M, varying degrees to which the national set of criteria was used in project prioritization as well as the need to maintain consistent, standardized information systems.

Recommendations

To address the broader evaluation findings, the evaluation recommends that the HFCP do the following:

1. HFCP should explore ways to either restructure or replace RPMIS and make its use mandatory. This should be done as a way of improving its utility as a cost-effective source of information for program management at HQ and regional levels. The change should support the future evaluation of HFCP, including the capacity to assess the economy and efficiency of the Program.
2. HFCP should explore ways to improve the consistency of regional capital project prioritization in an effort to better inform the LTCP process.
3. HFCP, in conjunction with FNIHB health programs and FN/I communities, should improve the processes with respect to O&M, including clarification of the roles and responsibilities and tracking of resources.

Appendix A -- HFCP Logic Model

Health Facilities and Capital Program Logic Model

	Stream 1	Stream 2	Stream 3	Stream 4
Program components	Promotion of program best management practices Coordination of the assurance framework Strategic Investment Planning and Allocation
Inputs	FTEs (Vote 1), O&M (Vote 5), Contribution funds (Vote 10)
Activities	Continual improvement of guidelines, manuals and policies for FNs and HC regional organizations to promote best management practices	Continual improvement of the Audit Assurance Framework	Continual improvement of the funding allocation strategy and strategic investment planning (short and long term)	Continual improvement of the comprehensive data management system to track and monitor all aspects of facility life-cycle
Outputs	Updated Guidelines/ Manuals CCA best management practices to ensure adherence to HC obligations O&M Plans & Schedules	Integrated Audits Identification and prioritization of preventative/corrective measures and remedial actions	Trend forecasting model Recapitalization model O&M cost/gap analysis Annual Long Term Capital Plan Sustainability analysis	Models for comprehensive allocation strategy Tracking of core facility data Audit follow-up modules
Immediate outcomes	Increased availability of health facilities, equipment & other movable assets in FN/I communities that support the provision of their health services	Improved risk management Improved efficient, productive and functional facilities & equipment Improved planning and cost analysis		Increased availability of evidence -based data to support efficient facility life-cycle planning
Intermediate outcomes	Effective planning & management of the construction & maintenance of health facilities undertaken by FN/I communities Increased Knowledge & understanding of facilities & risk management Sustained & modernized space, equipment & other movable assets Improved compliance with building & labour codes, environmental regulations & other applicable policies, guidelines & standards Improved access to quality well coordinated programs for FN/I individuals, families, and communities through increased health facility space
Final outcomes	Modern & well maintained health care facilities & residences that support effective health program delivery