First Nations and Inuit eHealth Infostructure Program (eHIP) - Evaluation

(PDF Version - 249 K)

Management Action Plan - First Nations and Inuit eHealth Infostructure Program (eHIP) - Evaluation

First Nations and Inuit eHealth Infostructure Program (eHIP) - Evaluation

Final Report

February 28, 2012

Table of Contents

Acronyms

AANDC
Aboriginal Affairs and Northern Development Canada
AFN
Assembly of First Nations
AHTF
Aboriginal Health Transition Fund
BPMD
Business Planning and Management Directorate
CFOB
Chief Financial Officer Branch
DEP
Departmental Evaluation Plan
DPMED
Departmental Performance Measurement and Evaluation Directorate
EAC
Evaluation Advisory Committee
eHIAC
eHealth Infostructure Advisory Committee
eHIP
eHealth Infostructure Program
EHR
Electronic Health Record
EMR
Electronic Medical Record
F/P/T
Federal / Provincial / Territorial
FAA
Financial Administration Act
FEDAA
Federal Accountability Act
FN
First Nations
FNIH
First Nations and Inuit Health (Regions)
FNIHB
First Nations and Inuit Health Branch
FN/I
First Nation and Inuit
HC
Health Canada
HISAP
Health Infostructure Strategic Action Plan
HQ
Headquarters
HSIF
Health Services Integration Fund
ICTs
Information and Communication Technologies
IT
Information Technology
ITK
Inuit Tapiriit Kanatami
MAP
Management Action Plan
PAA
Program Activity Architecture
PMF
Performance Measurement Framework
PHCD
Primary Health Care Division
PHCPHD
Primary Health Care and Public Health Directorate
P/T
Provincial/Territorial
PTA
Project Technical Authority
RAPB
Regions and Programs Branch
RPP
Report on Plans and Priorities
RMAF
Results-Based Management and Accountability Framework

Executive Summary

The objective of the First Nations and Inuit Health Branch (FNIHB) eHealth Infostructure Program (eHIP) evaluation was to assess the relevance and performance of the program for the five-year period from 2006/07 to 2010/11. The results of the evaluation will provide the Health Canada Deputy Minister and senior management with a comprehensive and reliable base of evidence to support decisions regarding the eHIP's present and future initiatives. The evaluation is required by the Financial Administration Act and the Government of Canada Policy on Evaluation (2009). This evaluation was conducted in accordance with Government of Canada (2009) and departmental (2010) policies for evaluations. Further guidance was provided through a variety of templates (including a data collection matrix) and a National Evaluation Working Group (EWG).

The eHIP supports the use of health technology to enable First Nations and Inuit (FN/I) community front line healthcare providers to improve people's health through innovative partnerships, technologies, tools and services. It focuses on the adoption of modern information technology (IT) for the purpose of defining, collecting, communicating, managing, disseminating and using data to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities.

The eHIP is comprised of one national office and seven regional offices. In total, the eHIP's funding allocation (including full-time employees, operations and maintenance, and grants and contributions) for the five fiscal years included in the evaluation was $131,747,810.00. This amount, provided by CFOB, includes both A-Base funding as well as funding provided by other program areas outside the eHIP National office.

The eHIP is responsible for the following six program components which served to guide this evaluation:

  1. Broadband Connectivity
  2. Public Health Surveillance Systems
  3. Telehealth
  4. Electronic Medical Record (EMR) and Electronic Health Record (EHR)
  5. IT Technical Support, Maintenance and Capacity Development
  6. Information Management.

This evaluation sought to assess the relevance and performance of the eHIP against the following five Evaluation Core Issues:

Relevance:

Performance:

Key Findings and Conclusions

Overall, this evaluation has found that the eHIP is a highly relevant program that has demonstrated progress towards its stated outcomes, but it must improve its forward-looking strategic planning and implementation to ensure the achievement of the eHIP stated outcomes.

Relevance

The eHIP continues to address a significant and demonstrable need, and has been responsive to the needs of FN/I, though opportunities to become more appropriate and responsive to FN communities exist. In particular, eHIP helps meet a variety of specific FN/I health needs and eHIP's major business activities are relevant and appropriate. Recent disease outbreaks illustrate the importance of eHealth technologies (i.e., public health surveillance systems) that support and are responsive to FN/I communities.

The eHIP's objectives are well aligned and consistent with federal government priorities and departmental strategic objectives. Evidence includes "innovation and keeping pace with technology" as a federal government priority noted in the 2010 Speech from the Throne while Electronic Medical Records (EMRs) and Telehealth were identified as key national healthcare priorities by Canada Health Infoway. Various First Nations and Inuit Health Branch (FNIHB) documents outline objectives such as access to health services and communicable disease control, and ensuring the eHIP's historical and current outcomes are aligned with key stakeholders' expectations.

The eHIP is in alignment with federal roles and responsibilities, and it is appropriate for the federal government to be delivering this program. The vast majority of stakeholders believe the federal government should be involved in funding eHealth initiatives in First Nations and Inuit (FN/I) communities. This is consistent with the 2010 Assembly of First Nations Annual General Report which identifies partnerships with various Federal/Provincial/Territorial organizations and various eHealth projects as key priorities.

Opportunities have been identified to ensure the Program continues to meet the needs identified by FN/I communities.

Performance

There has been progress toward the achievement of most of the immediate outcomes, though none have yet been fully achieved. Many communities still lack access to many eHealth services. The most measurable progress has been made in the areas of improved access to eHealth infostructure services such as broadband connectivity and Telehealth. Significant improvement is required in the areas of a public health surveillance system and EMRs/EHRs. Although some training is being provided to increase the use of IT as part of service delivery, improved effort is required. Some communication strategies were identified to increase FN/I awareness of eHealth infostructure but there has been marginal increase in awareness of eHealth in FN/I communities over the past five years. There is evidence of the increased use of evidence-based information to inform eHealth planning and implementation. However, data was insufficient to analyze two of the eHIP's immediate outcomes: improved ongoing integrated planning and implementation of complex eHealth systems; and, greater use of policies, standards and guidelines for IT implementation and use.

Overall, stakeholders expressed the need for greater collaboration on how limited resources are allocated, based on strategic priorities and community needs. The lack of predictable funding impacts the sustainability of needed eHealth systems, services and tools. Greater collaboration is needed with all stakeholders to ensure funding priorities align with FN/I community needs and contribute to overall eHealth objectives, including interoperability with provincial/territorial system requirements. Assessment of expenditure allocations demonstrates a need for improved financial performance data and tracking. Theoretical modeling demonstrates the importance of system and tool up-take (utilization) to ensure cost effectiveness of eHealth strategies ensuring the achievement of overall value-for-money.

There has been some progress towards the achievement of intermediate outcomes. This is illustrated by the establishment of valuable partnerships to ensure stakeholders in FN/I health are engaged in the integration of eHealth services. The eHIP has demonstrated measurable progress in ensuring access to health information in the areas of broadband connectivity and Telehealth with less progress in the areas of a public health surveillance system and EMRs/EHRs. Data was not sufficient to analyze two of the eHIP's intermediate outcomes: increased First Nations and Inuit management of eHealth Infostructure; and, increasingly integrated information for continuous improvement in eHealth Infostructure.

There has been varied progress toward the achievement of most of the long-term outcomes however, it should be noted that these results are not to be fully expected until 2020. Progress has been made in such areas as FN/I capacity, capability and seamless integration with provincial EHR systems. There is no performance data on EMR implementation, little data on system integration performance, and low levels of satisfaction reported by survey respondents with respect to EMR/EHR availability. Significant levels of dissatisfaction were reported from community-level stakeholders with the governance of the program demonstrating a need to improve FN capacity to influence and/or control eHealth programs and services.

The results of this evaluation also indicate that community-level service providers do not believe many of the proposed key benefits of the eHIP have yet occurred; however, there have been demonstrated benefits such as increased access to educational opportunities and skills development, and making service delivery more efficient and effective. There are many opportunities to ensure long-term success.

Recommendations

Based on the findings of this evaluation, a number of recommendations have been provided to assist the eHIP in continuing to be relevant and improving on performance through the full achievement of expected results. The recommendations resulting from this evaluation are to:

Create a single comprehensive and detailed Business Case.
The eHIP should consider preparing a comprehensive and detailed business case, with a focus on increasing adoption of modern systems, change management strategies, and utilization within communities to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities. Findings suggest more detailed information regarding the eHIP's highest-level organizational goals (and how it will achieve them) is required. The development of a business case would provide this level of detail and assist the eHIP with its "improved ongoing integrated planning of complex eHealth systems" immediate outcome, as well as its "use evidence-based information to inform eHealth planning" immediate outcome. Consideration of best practices and building on past successes must also be incorporated.
Prioritize investments and further clarify funding policies and practices.
The eHIP needs to ensure funding is adequate for both implementation projects and ongoing operations through a review and identification of funding priorities within the eHIP component areas. Since eHealth priorities and progress vary significantly between communities, the eHIP should be flexible in working with communities to understand their unique needs and develop solutions. Investments should be based on the program components, regional needs and identified funding priorities.
Build eHealth capacity of FN communities.
Ongoing support to external stakeholders will identify areas for investing and sustain First Nations and Health Canada governance, planning and strategy development as well as First Nations human and organizational capacity development. Readiness assessments will facilitate each FN/I community to assess gaps in eHealth knowledge, identify training needs of service providers in their community, and develop a strategy that is tailored to meet their specific needs.
Implement a comprehensive Communications Approach.
The development and implementation of a Program-wide, multi-pronged, communications approach will increase awareness of eHIP's activities across all Regions and communities (one of the eHIP's immediate outcomes). It will also facilitate dialogue between stakeholders at various levels of government, as well as external stakeholders, and will provide greater opportunity for stakeholders at the community level to be involved in eHealth program planning and decision-making.
Implement a refined Performance Measurement Strategy.
The refined strategy will ensure: the collection of better quality performance data (including financial data); improved consistency of data collected across jurisdictions; better measures of success against established targets; support for informed decision making; and facilitate future evaluations.

1. Background

The objective of the FN&I eHealth Infostructure Program (eHIP)Footnote 1 evaluation was to assess the relevance and performance of the program for the period of 2006/07- 2010/11. The evaluation will provide the Health Canada Deputy Minister and senior management with a comprehensive and reliable base of evidence to support decisions regarding the continued implementation of the program's present and future initiatives. The evaluation will also identify any gaps, barriers to success, areas of concern, and success stories related to eHealth at the community, regional and national levels.

This evaluation is required by the Financial Administration Act and the Government of Canada (GoC) Policy on Evaluation (2009). As per Health Canada's 5-Year Departmental Evaluation Plan (DEP), the evaluation of the eHIP is required to be completed in 2011/12.

2. Program Description

The eHIP supports the use of eHealth technology to enable First Nations and Inuit community front line healthcare providers to improve people's health through innovative eHealth partnerships, technologies, tools and services. It focuses on the development and adoption of modern IT systems for the purpose of defining, collecting, communicating, managing, disseminating and using data to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities.

The program evolved out of the need to align with First Nations' ehealth strategies, health plans and policy directions, as well as the movement by provinces and territories and the health industry towards increased use of information and communications technologies (ICTs) to support health service delivery and public health surveillance. A brief history of the evolution of the eHIP from 1999 to present is outlined below:

1999 - 2002: First Nations and Inuit Health Information System (FNIHIS)
The original FNIHIS application traces back to the 1980s, when the FNIH Ontario Region developed a health information computer system at a time when there was no existing electronic means for collecting health data of First Nations living on-reserve. The system was jointly owned by Health Canada's FNIHB and the Chiefs of Ontario. By 2002, FNIHIS was running in health facilities in 65% of the communities served by FNIHB as a case management and planning tool for public health nurses and other health professionals.
2002 - 2010: e-Health Solutions Unit (e-HSU)
The FNIHIS Project Team became the core of a newly created organizational unit in FNIHB called the e-Health Solutions Unit. The e-HSU was created in 2002 and evolved out of the need for FNIHB to align with First Nations' eHealth strategies, health plans and policy directions, as well as the movement by provinces/territories and the health industry towards increased use of IT to support health service delivery and public health surveillance.
2010 - Present: eHealth Infostructure Program (eHIP)
Following the Health Infostructure Strategic Action Plan (2010) and under the new Program Authorities, the e-Health Solutions Unit transitioned into the eHealth Infostructure Program. The emphasis of the new program is on: Service Provision Capacity Building; Stakeholder Engagement and Collaboration; Data Collection, Surveillance, and Research; and Policy Development and Knowledge Sharing. This will provide support for Electronic Health Record (EHR) capacity and capability for First Nations in the future.

Health Canada is committed to achieving a fully integrated, sustainable health service for First Nations and Inuit community members that gradually adds more community-level eHealth services, and enables front-line health care providers to use innovative technologies and services to improve health care. Guided by FNIHB's Health Infostructure Strategic Action Plan (HISAP), work towards this vision is continuing in close partnership with other federal departments such as Aboriginal Affairs and Northern Development Canada (AANDC) (formerly Indian and Northern Affairs Canada), Canada Health Infoway, provincial governments, private sector organizations and First Nations and Inuit leadership. FNIHB and FNIH Regions support the delivery of public health and health promotion services to First Nations on-reserve and in Inuit communities.

The eHIP logic model (see Figure 1) is derived from the HISAP, outlining the main activities, outputs and expected outcomes of the eHIP. The eHIP also plays an important role in FNIHB's public health plans, evidenced by the Five-Year Strategic Framework for FNIHB's Public Health Role in First Nations Communities. The eHIP's portfolio of program components, described in Section 2.1, are derived from these various strategies and plans.

Figure 1: FN&I eHealth Infostructure Sub-Sub-Activity Logic Model

2.1 Program Components

The eHealth Program is responsible for the following six components:

  1. Broadband Connectivity
    Sustainable Broadband Connectivity is the key basic element for modernizing community level health service delivery in First Nations communities. Working in partnership with First Nations and Inuit, AANDC leads a Government of Canada approach to community broadband connectivity with Infrastructure Canada, other federal departments and provincial governments. FNIHB works on achieving "last mile" solutions to First Nations/Inuit health facilities and supporting the sustainability of broadband for health business. FNIHB works with AANDC on capacity development strategies to maintain the connectivity infrastructure for the benefit of all public services at the community level.
  2. Public Health Surveillance and Related Systems (i.e., Panorama, or equivalent)
    Public Health Surveillance is understood to be the on-going, systematic use of routinely collected health data to guide public health action in a timely fashion. The goal of FNIHB and FNIH Regions is to work with the provinces towards a system that ensures adequate public health surveillance for First Nations and Inuit peoples across the country that is comparable to public health surveillance in place for other Canadians.
  3. Telehealth
    Telehealth is the use of information and communications technologies (ICTs) to deliver health services and transmit health information over both long and short distances. FNIHB and FNIH Regions work with First Nations and Inuit regionally to access key services within established provincial and territorial Telehealth Networks, forming partnerships for integrated service delivery and cost support. Telehealth services aim to be responsive, flexible, accessible, and cost effective for communities. This includes providing supports and tools for professionals and community members to establish long-term sustainable funding for telehealth infrastructures and service delivery programs.
  4. Electronic Medical Record and Electronic Health Record (EMR/EHR)
    FNIHB works with FN/I to address privacy concerns and seeks to meet the principles of ownership, control, access and possession (OCAP) where applicable, while ensuring that all provincial and federal legislative requirements are satisfied. At the community level, the use of the EMR/EHR will be based on the community's health needs assessment, and local health management priorities.
  5. Information Technology (IT) Technical Support, Maintenance and Capacity Development
    FNIHB works with First Nations/Inuit to harmonize community IT policy, support strategies, and leverage approaches and resources for efficient community level IT public services support. Capacity development and support around ICT management at the community level is a longer term investment in both time and resources and this will be coordinated with other key federal departments, especially AANDC, who shares similar community ICT support challenges.
  6. Information Management (IM)
    Quality health information is essential for: planning and implementing health policy and programs;informed decision making; providing information on health events and their causes and impacts; integration of information that can be passed on to individuals and communities to help them protect, improve and maintain their health;identifying research hypotheses and assistance in research studies; and detection of unusual events, epidemics, or other health changes. IM tools, procedures and legislation (federal/provincial/territorial) apply to all health information, regardless of whether it is in paper or electronic form.

2.2 Program Resources

The eHIP is comprised of one national office (FNIHB-HQ) and seven regional offices, all with varying resources. Table 1 outlines eHIP allocations for the fiscal years included in the evaluation.

Table 1 - Spending and Fund Transfers by Major Group Table 1 footnote *
2006/07 2007/08 2008/09 2009/10 2010/11 5 Year Total
Full-Time Employees $3,408,701 $3,174,834 $4,256,142 $4,499,676 $4,944,304 $20,283,657.00
Operations & Maintenance $4,834,615 $2,938,646 $5,980,966 $3,831,520 $3,580,191 $21,165,938.00
Minor Capital $708,822 $573,985 $0 $0 $103,077 $1,385,884.00
Grants & Contributions $19,391,720 $21,672,803 $14,603,349 $15,238,578 $18,005,881 $88,912,331.00
Total $28,343,858 $28,360,268 $24,840,457 $23,569,774 $26,633,453 $131,747,810.00

Table 1 footnotes

Table 1 footnote *

Note: Total expenditures include both A-Base funding as well as funding provided by other Program areas outside of the eHealth Program's National Office. Financial information provided by CFOB.

Return to table 1 footnote * referrer

Sources of Funding

The table above illustrates total investments of $131M which includes regional allocations provided by other federal program sources. Some of these other sources may include the Aboriginal Health Transition Fund, AANDC, etc. Expenditures may also include one-time investments by the department.

FNIHB National Office primarily funds FNIH regional offices that, in turn, fund First Nations and Inuit communities and regional organizations using various funding models. In addition, some FNIHB funds are used to support targeted projects with a national scope designed to examine innovations for possible application to national programming.

2.3 Key Stakeholders and Partners

The following are some of the eHIP's key stakeholders and partners:

3. The FN&I eHealth Infostructure Program Evaluation

3.1 Evaluation Objective and Context

The objective of the FNIHB eHIP evaluation was to systematically collect and analyze evidence of the eHIP's results to assess the relevance and performance of the program for the period of 2006/07- 2010/11. The evaluation will provide Health Canada's Deputy Minister and senior management with a comprehensive and reliable evidence base to support decisions regarding the continued implementation of the program's present and future initiatives. The most recent program review was done in 2006 covering the period between September 2002 and December 2005.

This evaluation was conducted between June and December 2011.

3.2 Evaluation Scope

All six eHIP components listed in Section 2.1 are in scope for this evaluation.

The eHIP has planned to undertake two separate phased evaluation activities as shown in the eHIP Evaluation Plan (Figure 2). Phase One of this process is this departmental evaluation, and Phase Two, which is not in scope for this report, will be conducted in the near future. Phase Two will examine international best practices, potential business models for eHIP, and the costs to operate the program at both the national and international levels. This evaluation will support Phase 2 as a line of evidence in order to recommend best practices and potential future business models.

Figure 2: FN&I eHealth Evaluation Plan

Figure 2

Text description

The eHIP has planned to undertake two separate phased evaluation activities as shown in the eHIP Evaluation Plan (Figure 2). Phase One of this process is this departmental evaluation, and Phase Two, which is not in scope for this report, will be conducted in the near future. Phase Two will examine international best practices, potential business models for eHIP, and the costs to operate the program at both the national and international levels. This evaluation will support Phase 2 as a line of evidence in order to recommend best practices and potential future business models.

For this evaluation, the eHIP evaluation falls under 3.3 Health Infrastructure Support, 3.3.2 Health System Transformation, and 3.3.2.2 Health Infostructure of the Program Activity Architecture (PAA), and includes performance indicators relevant to the eHIP as described therein.

As specified in the eHIP Evaluation Framework, this evaluation assesses the relevance and performance of the eHIP against five core issues.

Table 2 - Evaluation Core Issues
Evaluation Core Issues Description
Relevance
Issue #1 Continued Need for Program

(Maps to Relevance Question R1)
Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians
Issue #2 Alignment with Government Priorities

(Maps to Relevance Question R2)
Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes
Issue #3 Alignment with Federal Roles and Responsibilities

(Maps to Relevance Question R3)
Assessment of the roles and responsibilities for the federal government in delivering the program
Performance (effectiveness, efficiency and economy)
Issue #4 Achievement of Expected Outcomes

(Maps to Performance Questions P1, P2, P3)
Assessment of progress toward expected outcomes (including immediate, intermediate and long-term outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes
Issue #5 Demonstration of Efficiency and Economy

(Maps to Performance Questions P4, P5, P6, P7)
Assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes

The following evaluation questions were created to gather data about the Evaluation Core Issues. The eHIP Data Collection Matrix identifies questions for each of the specific indicators for Relevance, and outcomes and indicators for Performance.

Relevance Questions R1-R3

Performance Questions P1-P7

Core Issue #4:

Core Issue #5

3.3 Departmental Assessment of Evaluation Risk

Health Canada assesses the evaluation risk to determine an evaluation approach and the level of effort required to complete the evaluation. The overall risk ranking level for the eHIP, as determined in the HC Departmental Evaluation Plan 2011/2012, was "Low". The low ranking was considered in the design of this evaluation.

3.4 Evaluation Approach, Design and Methodology

3.4.1 Evaluation Approach

The evaluation approach for this assessment was to examine the achievement of expected outcomes, that is, the results achieved by the program based on its logic model (see Figure 1). This summative evaluation focus was on immediate outcomes given the program's long-term outcomes are not expected to be achieved until 2020.

The evaluation also included a participatory approach, that is, the inclusion of internal and external stakeholders in the development of the evaluation framework to ensure the relevancy of the evaluation. This included an Evaluation Working Group (EWG) which was co-chaired by the eHIP Program Liaison and a Departmental Performance Measurement and Evaluation Directorate (DPMED) Senior Evaluator. Membership of the EWG consisted of the eHIP Program Liaison, DPMED, eHIP program coordinator, FNIH regional representatives, an Assembly of First Nations representative, and a consultant.

3.4.2 Evaluation Design

The Government of Canada Policy on Evaluation (2009) and the HC Evaluation Policy (2010) were reviewed to receive guidance on evaluation design and data collection best practices. An Evaluation Framework was developed to guide the evaluation including an examination of the logic model, its context and position within the department's Program Activity Architecture (PAA); an assessment of the logic model's validity in this context and the expected results chain; the use of comparison data; and baseline data when available. This was a non-experimental evaluation. However, a theory-based approach and cost utilization analysis were included in the design to support the assessment of economy and efficiency.

3.4.3 Data Collection Methods

A Data Collection Matrix was developed as part of the Evaluation Framework to guide the development of the evaluation data collection strategy. The Core Issues outlined in the GoC Policy on Evaluation (2009) include the integration of program performance and evaluation measures, methodologies and other elements that strengthened this evaluationFootnote 2.

The methods used in this evaluation included the use of surveys (2) and an extensive document and literature review. Multiple lines of evidence were gathered from different sources and through the methods described below to allow for data comparison, and to support evidence-based conclusions.

3.4.3.1 Documentation and Literature Review

Program documents were obtained from FNIHB eHIP. These included annual reports, work plans, program files and other relevant material that both described and documented the eHIP's progress over time.

Internet and literature searches were conducted to identify other relevant Canadian eHealth reports, strategies, and evaluations, including non-First Nations initiatives. Documents and literature were reviewed by the consultant, and data relevant to the indicators was extracted.

A total of 184 documents from 2002 to present were systematically reviewed to identify relevance and performance data using data collection templates provided by DPMED. These standardized grids documented and mapped the relevant data to specific performance indicators and assisted in the overall analysis based on the evaluation questions.

3.4.3.2 Stakeholder Surveys

Two separate surveys were developed and administered: one for community-level service providers (herein referred to as "Community survey"), the other for Federal government management (National office and FNIH regional offices), herein referred to as "Management survey". The decision to create two separate surveys was based on input from the EWG, as it was felt that survey questions needed to be tailored to specific recipient groups. Surveys were piloted within the target audience communities.

Survey Question Development

Survey questions were developed based on indicators and designed to be an additional data source and line of evidence, as well as to capture information not expected to be found during the document review, such as stakeholder opinions. Survey questions were reviewed and revised extensively by members of the Evaluation Working Group (EWG). The surveys were adjusted to an appropriate readability level, and were translated into French.

Survey Administration

Names and contact information for community-level service provider recipients were solicited from the members of the EWG. Examples of recipients at the community-level are Health Directors and eHealth Coordinators of community health facilities. The names and contact information for 302 community-level recipients, and 112 internal management recipients were provided to the consultant. All recipients identified were sent the respective survey.

A Canadian-based online tool, Fluid Surveys, was used to administer the surveys. Recipients were given five weeks to complete the survey.

Survey Response Rates

Overall, survey response rates were lower than expected, but on par with average response rates of other Health Canada surveys within First Nations and Inuit communities. The response rate for the Management survey was 32% (N=112), with 36 recipients completing all questions in the survey. Of the 302 Community survey recipients, 40 completed all questions in the survey, resulting in a 13% response rate. However, 13 recipients notified the consultant that they had no involvement in eHealth activities and thus opted-out of the survey. The adjusted response rate is 14% (N=289).

It was determined that of those who responded, despite response rate variances from region to region, respondents reflected a fair cross-representation of both management and community participants. The implications of overall low survey response rates to data quality and general evaluation results and strategies for improving response rates in future evaluations are discussed in the Limitations section (Section 3.5).

Table 3a - Regional Distribution of Community Respondents
Response National Percentage Count/Total Survey Recipients
Pacific 20% 8/44
Alberta 0% 0/14
Saskatchewan 22% 9/94
Manitoba 20% 8/47
Ontario 10% 4/16
QuebecTable 1 footnote * 2% 1/43
Atlantic 25% 10/44
Total Responses 40/302

Table 3a footnotes

Table 3a footnote *

The First Nations of Quebec and Labrador Health and Social Services Commission compiled their responses into one survey. (Source: AFN)

Return to table 3a footnote * referrer

Response rates may also have impacted the data in that, Regions with more community-level service providers currently active in eHealth activities and projects may be overrepresented in the sample. Recipients who felt their communities and Regions were not currently engaged in eHealth projects fully, may have opted-out of the survey skewing the overall general interpretations of findings.


Table 3b - Regional Distribution of Management Respondents
Response National Percentage Count/Total Survey Recipients
Pacific 3% 1/44
Alberta 3% 1/7
Saskatchewan 14% 5/22
Manitoba 28% 10/17
Ontario 17% 6/17
Quebec 6% 2/7
Atlantic 14% 5/19
National Capital Region 17% 6/19
Total Responses 36/112

3.4.4 Data Analysis Methods

The data collected was analyzed using the following methods:

3.4.5 Multiple Lines of Evidence

Gathering multiple pieces of corroborating evidence helps improve the quality of certain data. As described above, the evaluation methods relied on more than one line of evidence. The majority of evaluation questions were addressed through multiple lines of evidence, as determined through a cross-walk and data collection template.

3.4.6 Ethical/Human Subject Protection Issues and Protocol

Ethical and human subject protection principles were upheld in survey administration, data management, and reporting processes. A Canadian-hosted online tool (Fluid Surveys) was used to administer the surveys as per standard HC evaluation guidelines. Participation in the survey was voluntary.

Metadata was only provided to the contractor, and all personal or identifying information was kept confidential. Responses are presented in summary form within this Evaluation Report. The information collected was not disclosed to external third parties, as specified by the Privacy Act.

3.5 Limitations and Mitigation Strategies

The following limitations were observed during the data collection process. Potential impact on how the findings and conclusions are interpreted, and risk mitigation strategies that were used in this evaluation are discussed below.

3.5.1 Low Survey Response Rates

Several strategies were used to solicit the highest response rate possible, including:

Low survey response rates may be attributed to a statement in the survey introduction asking participants with no eHealth involvement to not complete the survey and notify the consultant to be removed from the recipient list. Participants may have self-assessed their involvement in eHealth activities and decided to opt-out because they felt they lacked eHealth knowledge and experience. In addition, recipients may not have been comfortable completing an online survey due to computer literacy, privacy, and/or accessibility issues. Furthermore, surveys were sent via email from the consulting company and recipients may not have recognized the sender, deleting or ignoring the email.

3.5.2 Data Limitations

Low survey response rates can negatively impact data quality, as a smaller sample size is less likely to represent the overall population. However, a low response rate does not guarantee lower survey accuracy; it simply indicates the risk of lower accuracy. The distribution of responses is more critical for data interpretation, as data trends are more difficult to identify with smaller sample sizes, and the risk of misinterpretation increases. As such, results of this survey must be interpreted carefully, with special care in generalizing the findings. In an analysis of the data by region, no meaningful trends were identified likely due to low response counts by Region. Drawing conclusions and comparing findings across Regions is not appropriate considering some Regions had very few or no respondents.

Gaps in Regional performance data existed in documentation provided to the contractor. These gaps included: limited or inconsistent availability of performance data over the full period of the evaluation; inconsistent availability of data between program areas; and, inconsistent reporting practices between Regions. As such, the performance data extracted from program documentation was insufficient to support nine indicators related to Evaluation Core Issue #4.

The strategies used to mitigate the risk of poor data quality in this evaluation included:

With the use of several of these mitigation strategies, data was analyzed and interpreted using methods to increase the credibility and reliability of the findings presented in this report.

3.5.3 Assessing Economy and Efficiency

Setting the context of the efficiency and economy aspects of this evaluation in relation to the Government of Canada (GoC) Evaluation Policy (2009) is important. The five years of the eHealth Program evaluated were implemented prior to the recent GoC Policy on Evaluation. Thus, specific requirements for defining and operationalizing efficiency and economy analysis were not set out for the Program's performance measurement strategy - it did not define efficiency and economy performance measures, definitions or indicators.

As such, there is a lack of "object costing", consistent financial performance data tracking as it relates to program activities and outputs. Similar to other programs within Health Canada, financial performance data was not consistently collected across the Program's national office and regional counterparts, nor was there consistency of program reporting in terms of components or specific component-related activities. Changes in the program's overall structure, financial allocations, operational priorities, and expected results have also complicated the assessment of economy and efficiency.

Shifts in financial reporting over the past number of years, including redefining the Program Activity Architecture (PAA), the 'clustering' model within FNIHB, and how the eHealth Program results align with overall FNIH Branch expected results, all contributed to an inability to explicitly measure economy and efficiency for this evaluation.

Although these limitations present a challenge in providing an overall assessment of the economy and efficiency of the eHIP, the evaluation did attempt to provide a general sense of resource utilization by comparing resource allocation/expenditure data with program results. As well, stakeholder opinions provided an additional line of information on the appropriateness of resource utilization. This was then supported through an economic modeling exercise that examined cost effectiveness and cost utilization as elements of ensuring eHealth effectiveness by demonstrating the importance of tool and service utilization as a key factor for ensuring economy and efficiency within the program's components.

Combined, these assessments indicate where programmatic successes (in terms of financial effectiveness) have been made, where greater successes can be made in the future, and whether the Program has provided economy and efficiency in addressing the eHealth needs of FN/I communities in Canada.

4. Key Findings

4.1 Relevance

4.1.1 Core Issue #1: Continued need for program

To what extent does the eHIP continue to address a demonstrable need, and is responsive to the needs of FN/I communities?

Generally, findings from both the survey and document review indicated that the eHIP continues to address a significant demonstrable need, and has been responsive to the needs of FN/I communities. Some opportunities for improvement were observed.

"Bottom line, this program is the only opportunity for eHealth initiatives to grow within First Nation territories. I encourage Health Canada and the Treasury Board to fund these current initiatives."
Community-Level Service Provider Survey Respondent

A number of documents indicated FN/I health needs that eHIP helps meet, such as access to care through Telehealth programs, and other needs to improve health and save lives through its various other eHealth programs and infostructure services. Once a public health surveillance system/tool is integrated with its provincial system, stakeholders feel that communicable disease control, administration and tracking of immunizations will improve.

The majority of survey respondents agreed that the eHIP's major business activities - which comprise a broad spectrum of the eHIP's activities - were relevant, appropriate and responsive (see Tables 4a and 4b).

Low agreement that certain activities were relevant and useful (appropriate and responsive) were found (e.g., expanding the number of tripartite data sharing and storage agreements, website hosting, lab information systems, drug information systems, diagnostic imaging, and registries). This indicates areas the eHIP should further analyze for possible refinement.

Mixed results were found regarding program-funded services and projects and their relevance, usefulness and appropriateness to addressing the needs of the First Nations and Inuit.

Table 4a - Relevant, Useful and Appropriate Agreement Rates of Community-level Service Provider Respondents (n=40)
Program Component Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree Don't Know N/A
Broadband Connectivity 2% 20% 10% 30% 18% 2% 18%
Public Health Surveillance Tool (i.e., Panorama, or equivalent) 5% 20% 25% 8% 12% 12% 18%
Telehealth 2% 12% 2% 38% 22% 8% 15%
EMR/EHR 5% 28% 5% 20% 15% 10% 18%
IT Technical Support, Maintenance and Capacity Development 5% 20% 10% 22% 25% 2% 15%
Information Management 8% 18% 15% 25% 18% 5% 12%
Table 4b - Relevant, Useful and Appropriate Agreement Rates of Management-level Respondents (n=36)
Program Component Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree Don't Know N/A
Broadband Connectivity 0% 11% 0% 28% 50% 0% 11%
Public Health Surveillance Tool (i.e., Panorama, or equivalent) 0% 6% 8% 28% 44% 3% 11%
Telehealth 0% 0% 8% 19% 58% 0% 14%
EMR/EHR 6% 6% 8% 11% 47% 8% 14%
IT Technical Support, Maintenance and Capacity Development 3% 8% 0% 25% 42% 8% 14%
Information Management 0% 8% 8% 28% 39% 3% 14%

Low agreement scores above may indicate a variety of issues, such as flawed strategy to poor data quality. For public health surveillance in Table 4a, the neutral response rate may be attributable to a lack of awareness of current activities or little-to-no implementation and/or participation at the community level. Further study of the areas is required.

Management-level stakeholder survey results indicated that there is a continued need for investment in eHealth since it helps provide more efficient and effective service delivery (63% agree, N=36), supports innovation in the delivery of health services (69% agree, N=36), offers improved access to primary care (55% agree, N=36), reduces patient travel times and costs (55% agree, N=36), and increase access to educational opportunities and skills development for providers (63% agree, N=36). In particular, Telehealth was found to increase FN/I access to specialized health services, reduce transportation costs, and provide a number of other positive benefits.

Survey results indicated that community-level service providers do not believe that many of the proposed key benefits of the eHealth Program occurred during the evaluation period. In addition, discrepancies emerged between the management and community groups related to satisfaction with the progress of the eHealth Program.

Key Findings

4.1.2 Core Issue #2: Alignment with government priorities

Assessment of the linkages between program objectives and (i) federal government priorities, and (ii) departmental strategic outcomes

Generally, findingsFootnote 4 indicated that the program's outcomes are well aligned and consistent with federal government priorities and departmental strategic objectives.

Federal government priorities include:

Departmental strategic objectives include:

The majority of management-level survey respondents (52-78%, N=36) agreed or strongly agreed that the eHIP is well-aligned with its vertical and horizontal stakeholders and partners, including Health Canada's FNIHB and Regions and Programs Branch (RAPB), Aboriginal Affairs and Northern Development Canada (AANDC), and Canada Health Infoway.

Findings indicated the majority of management-level surveyFootnote 5 respondents (93-95%, N=36) agreed that the eHIP's historical and current goals and priorities were also consistent with their goals and priorities including:

4.1.3 Core Issue #3: Alignment with federal roles and responsibilities

Assessment of the roles and responsibilities for the federal government in delivering the program

Generally, findings indicated that the program is in alignment with federal roles and responsibilities, and it is appropriate for the federal government to be delivering the program.

Management-level survey respondents (97%, N=36) indicated that the federal government should be involved in funding eHealth initiatives in FN/I communities. In addition, as stated in the findings for Core Issue #2, survey respondents agreed there was strong consistency between the eHIP's components and the priorities of a number of other federal government organizations, and that the eHIP's stated current and historical goals align with the priorities of many of their federal-level stakeholders.

For example, the 1979 Federal Indian Health Policy identifies that one of the pillars of the policy is the responsibility of the Federal Government to maintain a health system that supports public health activities on reserves, health promotion, and the detection and mitigation of hazards in the health environment. And further, in March 2010, the Assembly of First Nations identified collaboration and partnership between First Nations, Canada Health Infoway and provincial governments on Telehealth and public health surveillance projects.

Some differences were noted by management-level survey respondents in the alignment of federal government roles and responsibilities (i.e., F/P/T partners' goals and priorities in relation to the eHIP outcomes).

4.2 Performance

4.2.1 Core Issue #4: To what extent have immediate outcomes (short-term goals) been achieved?

Overall, findings from the document review and surveys indicate that there has been progress towards the achievement of most of the immediate outcomes.

Immediate Outcome 1: Improve Access to eHealth Infostructure Services

As implementation of eHealth tools into FN communities relies heavily on a community's readiness (connectivity, capacity), evaluation results note that there has been varied progress in increasing and/or improving access to eHealth infostructure services across the country. Findings indicate the most significant measurable progress has been made in the implementation of broadband connectivity in health facilities in FN communities; Telehealth; videoconferencing implementations; and e-SDRT (electronic service delivery reporting template). In contrast, low or zero implementation progress data has been reported for EMR, Public Health Surveillance ToolsFootnote 6 (i.e., Panorama, or equivalent) and Drug Information Systems.

There was a lack of performance data contained in the FNIH Regional Year-End Reports to assess the availability of eHealth infostructure services in FN communities, with the exception of Broadband Connectivity and Telehealth.

Key Findings

Broadband Connectivity:

Telehealth:

Electronic Medical Records (EMR):

Community-level service provider respondents were asked to indicate the current stage of implementation of various eHealth tools. Although not all of the tools identified below are managed by the eHealth Program, they each have an electronic health component. Reported implementation progress for various eHealth tools by community-level service providers can be seen in Table 5. The high number of "Don't Know" responses may indicate a lack of awareness of tool status or direct involvement in some areas at the community level.

Table 5 - Reported Implementation Progress for eHealth Tools (n=40)
Planning in Progress Implemented and Still Using Implemented but Not Using Not ImplementedTable 1 footnote * Don't Know
a. Public Health Surveillance Tool (i.e., Panorama, or equivalent) 35% 0% 0% 38% 28%
b. Electronic Medical Record (ex. EMR in a public health clinic) 12% 22% 0% 48% 18%
c. Drug Information Systems 5% 0% 0% 57% 38%
d. Diagnostic Imaging 5% 18% 0% 50% 28%
e. National Native Addictions Information Management System (NNAIMS) 10% 8% 2% 35% 45%
f. Home and Community Care/Aboriginal Diabetes Initiative (HCC/ADI) 10% 45% 0% 15% 30%
g. Medical Transportation Records System (MTRS) 8% 18% 0% 40% 35%
h. First Nations and Inuit Health Information System (FNIHIS) 5% 15% 12% 38% 30%
i. Community Reporting 22% 25% 0% 20% 32%
j. Electronic Service Delivery Reporting Template (e-SDRT) 0% 68% 2% 8% 22%
k. Email 0% 79% 0% 5% 16%
l. Internet Access 0% 32% 0% 5% 15%
m. Videoconferencing
i. Administrative purposes 15% 52% 0% 15% 18%
ii. Family encounters 15% 15% 2% 30% 38%
n. Telehealth
i. Health Promotion (client education) 5% 42% 0% 32% 20%
ii. Clinical consultations 12% 38% 0% 32% 17%
iii. Education sessions/training (professional development) 8% 60% 0% 15% 18%

Table 5 footnotes

Table 1 footnote *

Some respondents may not be aware of "Planning in Progress".

Return to table 5 footnote * referrer

Immediate Outcome 2: A workforce that is increasingly comfortable using IT as part of service delivery

Progress has been made towards this outcome. Some training is being provided to promote a workforce that is increasingly comfortable using IT. The same goal will be accomplished as more eHealth services are implemented and used by the workforce over the course of time.

Key Findings
Immediate Outcome 3: Increased First Nation awareness of eHealth Infostructure

Progress has been made towards this outcome. From FNIH Regional Program documentation, it appeared that a formal communications strategy is lacking. Overall, a need was identified to increase frequency and level of communication with FN communities and organizations as well as other partners including provincial jurisdictions. Respondents indicated a need to develop an effective communications approach that keeps information flowing to and from the eHIP to First Nations communities and FNIH Regions.

Key Findings
Immediate Outcome 4: Improved ongoing integrated planning and implementation of complex eHealth systems

There were no specific performance indicators or data collected to analyze this outcome.

Immediate Outcome 5: Increased use of evidence-based information to inform eHealth planning and implementation

Progress has been demonstrated, particularly related to planning at the national level; however there is a lack of evidence that demonstrates the use of evidence-based information to inform planning and implementation at the community and regional levels.

Key Findings
Immediate Outcome 6: Greater use of policies, standards and guidelines for IT implementation and use

There were no specific performance indicators or data collected to analyze this outcome.

4.2.2 Core Issue #4: To what extent have intermediate outcomes (medium-term goals) been achieved?

Findings indicate there has been some progress towards the achievement of most of the intermediate outcomes. Significant gaps existed in the data sources for the performance indicators in this section.

Intermediate Outcome 1: Key stakeholders in First Nation and Inuit health are engaged in the integration of health services

Overall, it was found that progress has been made in the eHealth Program, FNIH Regions and FN communities in the development of partnerships and collaboration between stakeholders in implementing eHealth strategies and projects. From the data, it is apparent that the level of integration between FN/I eHealth stakeholders is high. Additionally, engagement in partnerships with federal and provincial governments, FN organizations, eHealth agencies, and regional health authorities emerged as the most frequently cited alternative method or approach communities explored to make their eHealth Program or projects successful.

Key Findings
Intermediate Outcome 2: Increased First Nation management of eHealth Infostructure

There were no specific performance indicators or data collected to analyze this outcome.

Intermediate Outcome 3: Increasingly integrated information for continuous improvement in eHealth Infostructure

There were no specific performance indicators or data collected to analyze this outcome.

Intermediate Outcome 4: Access to health information

There is overlap between the indicators in this outcome and those of outcome P1.1 "Improved access to eHealth Infostructure services".Please refer to Section 4.2.1 for findings related to the number and type of eHealth information tools implemented across FN communities.

4.2.3 Core Issue #4: To what extent have long-term outcomes (long-term goals) been achieved?

There has been varied progress towards the achievement of most of the long-term outcomes. However, it should be noted that these results are not to be fully expected until 2020.

Long-Term Outcome 1: An EHR capacity and capability for First Nation and Inuit by 2020 and seamless integration with provincial EHR systems

An EHR capacity and capability for all First Nations and Inuit with seamless integration with provincial electronic health systems has achieved moderate progress. As previously discussed in Section 4.2.1, there has been some progress towards achieving this vision in terms of EMR planning, however satisfaction levels are low regarding the availability of EMR/EHR and results are not fully expected until 2020.

Long-Term Outcome 2: Promote innovative integrated health governance relationships

Good progress towards this outcome has been demonstrated. Findings indicate that innovative and integrated health governance relationships are being built between the Regions, communities, provincial governments, the federal government and other relevant stakeholders.

Key Findings
Long-Term Outcome 3: Improved FN capacity to influence and/or control (design, deliver, and manage) eHealth Programs and services

Little progress has been made for this outcome as demonstrated by high levels of dissatisfaction reported among community respondents with the governance of the program components. Stakeholders want to be more involved in the planning and/or control of the major eHealth Program components. Many stakeholders are unclear about their current role in the planning or control of the eHIP components. This needs to be addressed in order to improve overall FN capacity for influencing and/or controlling programs and services within the context of eHealth.

There were gaps in the program documentation for the performance indicators in this section.

Key Findings
Table 6 - Community Respondent Satisfaction with their Current Role in the Planning and Implementation of eHealth Program Components (N=40)
  Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied Don't Know N/A
a. Broadband Connectivity 2% 20% 25% 20% 8% 5% 20%
b. Public Health Surveillance Tool (i.e., Panorama, or equivalent) 2% 22% 32% 20% 2% 5% 15%
c. Telehealth 5% 15% 28% 30% 2% 2% 18%
d. EMR/EHR 2% 15% 25% 25% 0% 8% 25%
e. IT Technical Support, Maintenance, and Capacity Development 0% 25% 30% 22% 2% 2% 18%
f. Information Management 2% 25% 25% 25% 0% 2% 20%

4.2.4 Core Issue #5: Assessment of Economy and Efficiency

The assessment of economy and efficiency demonstrates that further emphasis on increased utilization of eHealth services is required to ensure overall eHIP cost effectiveness.

The evaluation attempts to assess efficiency and economy under Core Issue #5: Demonstration of Efficiency and Economy as outlined in the 2009 TB Policy Directive on Evaluation. That is, an assessment of resource allocation and utilization in relation to the production of outputs and progress toward expected outcomes.

The Government of Canada (GoC) Policy on Evaluation (2009) defines the demonstration of efficiency and economy as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Within the realm of program activities and FNIHB activities in general, there is considerable difficulty in measuring economy and efficiency in terms of comparison, alternative approaches, and attribution of the outcomes.

Specifically, the evaluation framework's matrix for economy and efficiency outlined four questions and set out to measure economy and efficiency using a standard set of performance indicators.

Those questions are:

However, due to a lack of financial performance data, the following approach was taken to assess resource allocation and utilization using four methods outlined below. They included:

  1. addressing questions about economy and efficiency at the level of program implementation and delivery stage (activities, outputs) through an assessment of available financial data (expenditure allocation review);
  2. obtaining clarification of data expenditure trends through interviews with key program staff (HQ);
  3. obtaining and reviewing qualitative data through key stakeholder surveys (community and management-level) on opinions regarding the factors affecting and/or influencing the achievement of long-term outcomes as they relate to resource availability and/or resource allocations; and
  4. theoretical modeling to examine whether telehealth services aimed at First Nations and Inuit communities in Canada are cost effective; whether electronic health and medical records (EHR/EMR) improve service efficiency; and whether public health surveillance tools (ie., Panorama, or equivalent) are cost effective.
4.2.4.1 Resource Allocation and Utilization

Financial data and related analysis provides an overall assessment of the impact of expenditure allocations in the context of resource utilization. This analysis included an examination of program delivery costs (direct and indirect salary, operating/maintenance costs and grants and contributions), including cost drivers, resource allocations (by program component) as well as cost/output ratios. In most cases, trend data was reviewed to understand how expenditure allocations affected program delivery and potentially, expected outcomes.

Total resources of $131,747,810 include the combined eHIP funds received by FNIHB National Office, FNIH Regional offices and distributed to FN/I communities. Contributions to the total funding may include resources made by other FNIH programs, regional funds re-allocated within given fiscal years and one-time special investments made by the Branch.

On average, the overall program expenditures per year were $ 26,349,561.38 across the 5 years reviewed (2006/07 - 2010/11). There was little variance from year to year of total expenditures (average 2%) as illustrated in Figure 3.

Figure 3 - Total Expenditures by Fiscal Year

Figure 3

Text description

Figure 3 illustrates that, on average, the overall program expenditures per year were $ 26,349,561.38 across the 5 years reviewed (2006/07 - 2010/11). There was little variance from year to year of total expenditures (average 2%).

For the five fiscal years evaluated, expenditures by FNIHB and FNIH Regional offices, varied greatly, with the greatest percentage of expenditures in Ontario, Alberta and Pacific regions, as indicated in Figure 4.

Figure 4 - Total Expenditures as a Percentage by Region, 2006/07 - 2010/11

Figure 4

Text description

Figure 4 indicates that, for the five fiscal years evaluated, expenditures by FNIHB and FNIH Regional offices, varied greatly, with the greatest percentage of expenditures in Ontario, Alberta and Pacific regions.

For most FNIH Regional offices, as well as the NCR national office, year-over-year expenditures varied by only <10% (on average) with the exception of Ontario and Pacific where the greatest fluctuations in expenditures were in 2008/09 and 2009/10 varying as much as a decrease of between 50 - 80%. Greatest fluctuations were in expenditures associated with contribution agreements. Fluctuations exist given that funding allocations for community-based projects are based on a number of criteria that impacts the fiscal allocations for each FNIH Region. These criteria include: allocations provided based on number and size of proposals approved; previous year funding allotments and priorities for longer-term project funding; and impact of future strategic priorities and funding cycles. Population size within each region also impacts on funding requests and allocations. The Northern Region received funding only in 2007/08.

Across the five years evaluated, total expenditures were tracked by salary, operating and maintenance, capital (minor) and contributions (agreements through FNIH Regional offices to FN/I communities and/or NAOs). Contributions comprised the majority of total expenditures (67.5%) while operating and maintenance costs comprised 16% and salaries 15.4% of total expenditures (see Figure 5).

Capital expenditures were made only in the first two years of funding (2006/07 and 2007/08) in the Atlantic, Manitoba, Alberta and Pacific Regions. The capital costs represent less than 1.1% of the total expenditures across the five years examined. Capital costs were initially covered by HQ but subsequently through contribution agreements within FN/I communities.

Figure 5 - $131,747,810 Expenditure Breakdown by Type, 2006/07 - 2010/11

Figure 5

Text description

Figure 5 illustrates that, across the five years evaluated, total expenditures were tracked by salary, operating and maintenance, capital (minor) and contributions (agreements through FNIH Regional offices to FN/I communities and/or NAOs). Contributions comprised the majority of total expenditures (67.5%) while operating and maintenance costs comprised 16% and salaries 15.4% of total expenditures.

Despite the limitations in data availability to examine expenditures by current eHIP components, some financial breakdown was provided for total expenditures (Figure 6 - Actual Costs and Figure 7 - Actual Costs as a Percentage) for comparative purposes. Nearly half of the investments in eHealth over the past 5 years were made in Broadband Connectivity and IT Technical Support, Maintenance and Capacity Building.

A quarter of investments were made in Information Management (Business Strategy, Management and Service Manager and Policy and Program Development). The remaining 25% of investments were split between Public Health Surveillance, Telehealth and Emerging Technologies (including EMR/EHR and emergency mobile technology).

Figure 6 - Actual Costs, 2006/07 - 2010/11

Figure 6

Text description

Figure 6 illustrates that, despite the limitations in data availability to examine expenditures by current eHIP components, some financial breakdown was provided for total expenditures, in particular, the actual costs to the eHIP program. Nearly half of the investments in eHealth over the past 5 years were made in Broadband Connectivity and IT Technical Support, Maintenance and Capacity Building.

Figure 7 - Actual Costs as a Percentage, 2006/07 - 2010/11

Figure 7

Text description

Figure 7 illustrates that, despite the limitations in data availability to examine expenditures by current eHIP components, some financial breakdown was provided for total expeditures, in particular, the actual costs as a percentage of overall costs for Broadband Connectivity and IT Technical Support, Maintenance and Capacity Building. A quarter of investments were made in Information Management (Business Strategy, Management and Service Manager and Policy and Program Development). The remaining 25% of investments were split between Public Health Surveillance, Telehealth and Emerging Technologies (including EMR/EHR and emergency mobile technology).

4.2.4.2 Review of Stakeholder Opinions
Key Findings

Through the key stakeholder's survey (community and management-level), questions regarding the likelihood of achieving long-term outcomes raised a number of issues specifically as they relate to funding availability and/or funding allocation. These issues were identified in open-ended questions regarding challenges and/or barriers to the Program's success.

For community respondents funding was the most frequently discussed topic with a majority of respondents (N=40) indicating that funding is insufficient, and this lack of funding is slowing progress on outcomes.

Funding sustainability was consistently identified by community level stakeholders and was evident throughout the program documentation as the primary barrier to implementing and maintaining eHealth tools in FN communities. Funding needs to be adequate for both implementation projects and ongoing operations. Without adequate funding to sustain the ongoing operation of eHealth infrastructure and services over the long-term, they will deteriorate.

Stakeholders indicated that funding needs to be appropriate to build required capacity, and, that more clarity is required regarding what initiatives will be funded, how they will be funded, when funding will be received, and stipulations and requirements around funding.

4.2.4.3 Theoretical Modeling
Key Findings

Several eHealth services were included in theoretical modeling to determine their cost effectiveness. These include telehealth, electronic health records and electronic medical records, and public health surveillance systems (i.e., Panorama, or equivalent). For each component, the process required a review of current literature to derive a cost effectiveness model, and outlined the implications for the Canadian healthcare system. It is shown that it is possible for these services to be cost effective, but that the cost effectiveness is strongly dependent on the utilization of the service.

Telehealth
Many telehealth programs have been implemented worldwide to provide service to remote populations. Medical benefits to the service users, as well as cost effectiveness have been shown in many instances. A model developed to gauge cost effectiveness showed that a telehealth system can be cost effective if the product of the fraction of patients who use the services and the fraction of users who avoid travelling is greater than 0.5. The breakeven point is slightly greater than 0.5 by a value determined by the cost of running the service, the money spent on medical travel, and the size of the population that uses the service. Analysing several past projects, it is found that their cost effectiveness can be explained by this model, and that as a whole, telehealth projects in Canada can be cost effective.
Electronic Health Records and Electronic Medical Records (EHR/EMR)
Electronic Health Records and Electronic Medical Records (EHRs and EMRs) have the potential to improve the efficiency of the Canadian healthcare system. Implementation in other countries has been shown to be financially viable. A model to determine the cost effectiveness has been derived, but data is deficient from Canadian EMR and EHR systems. An EMR/EHR system can be cost effective if utilization is high and if the costs of running the system are less than the money saved by using electronic rather than paper records. It is recommended that data be gathered in order to better determine the cost effectiveness.
Public-Health Surveillance and Related Systems (i.e., Panorama, or equivalent)
Public Health Surveillance is understood to be the ongoing, systematic use of routinely collected health data to guide public health action in a timely fashion. Surveillance is descriptive in nature, reporting on occurrence of injury or disease and their treatments as well as broader factors that determine health or factors that put some people at increased risk of injury or disease. Such systems have been shown to be cost effective in other countries. Using the recent SARS outbreak as an example, the potential savings are in the hundreds of millions of dollars.

The outlook of cost effectiveness of a Public Health Surveillance system is favourable. Similar projects in other countries have been shown to be economically favourable and simple economic models used to predict cost effectiveness show that it can be achieved with realistic parameters.

5. Conclusions

5.1 Relevance

Findings from this evaluation of the FN&I eHIP suggest that it is highly relevant, but improvement is required particularly in EMR/EHR, and IT capacity building program components to facilitate further progress and the achievement of expected results. FNIHB and FNIH Regions continue to make progress towards integration. As a result, there is strong evidence that supports continued investment in eHealth.

Evaluation findings suggest that the eHIP continues to address a demonstrable need and is responding, albeit with varying success across FN/I communities, to the healthcare and health technology needs of FN/I. The priority areas of the eHealth Program are well aligned with the needs of FN/I communities, and the projects and program components are useful and relevant to the needs of the FN/I communities.

The eHealth Program's outcomes are well aligned and consistent with federal government priorities, roles and responsibilities, and departmental strategic outcomes. The federal government considers eHealth a priority, and the evaluation findings illustrate that the eHealth Program and its business activities and outcomes are in alignment with various federal organizations including RAPB and AANDC.

Some areas for improvement were noted such as addressing concerns about lack of sustainable funding mechanisms to support projects and, ensuring that the Program supports increased community-based governance of eHealth projects, and eHealth capacity building in FN communities.

5.2 Performance

There has been varied progress towards the achievement of the eHIP's immediate, intermediate and long-term outcomes. However, inadequate and inconsistent Program data for many performance indicators makes it difficult to confidently and accurately draw comprehensive conclusions regarding the achievement of outcomes. Long-term outcomes are not expected to be fully realized until 2020.

Immediate (Short-term) Outcomes

There is evidence indicating progress towards improving access to some eHealth Infostructure services (e.g. high-speed connectivity, Telehealth and e-SDRT), while more effort is required in other services such as public health surveillance tools and EMR/EHR.

Integration of a Public Health Surveillance System (i.e., Panorama, or equivalent) is dependent on provincial selection and implementation of their respective tool.

Continued effort to ensure effective connectivity, support, maintenance, capacity development and information management are all components of the core infrastructure required to provide effective eHealth services. Findings suggest that Telehealth has been the most successful component. Further effort is required to ensure that a public health surveillance tool (that is compatible with its respective provincial system) will improve the management of public health information. EMR/EHR is a leading F/P/T priority.

Progress has been demonstrated in the use of evidence-based information to inform eHealth planning and implementation, particularly related to planning at the national level (i.e., in the development of strategic plans such as the HISAP and interim directives), however there is a lack of evidence that demonstrates the use of evidence-based information at the community and regional levels, particularly in the areas of improved capacity and capability of the work force to use IT as part of service delivery.

Intermediate Outcomes

One of the major strengths of the eHIP is progress made towards the development of partnerships and collaboration between stakeholders in implementing eHealth strategies and projects. From the data, it is apparent that the level of collaboration between FN/I eHealth stakeholders is high. Additionally, engagement in partnerships with federal and provincial governments, FN organizations, eHealth agencies, and regional health authorities emerged as the most frequently cited alternative method or approach communities explored to make their eHealth Program or projects successful.

There is progress being made towards improving access to health information, as assessed by the number and type of eHealth information tools implemented across FN communities. However, this progress varies across each eHealth tool and from FN community to community.

Long-Term Outcomes

Progress has been demonstrated towards building innovative and integrated health governance relationships between FNIH Regions, communities, provincial governments, the federal government and other relevant stakeholders through the establishment of key effective partnerships.

Although the focus of this evaluation was not on the long-term outcomes, there is evidence to show that some progress has been made in many areas including developing partnerships, planning for EHRs, and integrating public health surveillance systems with the provinces. There remains a need to focus on assessment planning and full implementation of the eHealth program components in order to achieve the desired long term outcomes.

Overall, these conclusions highlight one of the main issues to emerge from this evaluation - the eHIP is highly relevant to addressing the needs of the FN communities, and aligns well with federal priorities, however continued progress will be achieved by addressing:

Funding sustainability was consistently identified by stakeholders and was apparent throughout program documentation as the primary barrier to implementing and maintaining eHealth tools in FN communities. Funding needs to be adequate for both implementation projects and ongoing operations. Without adequate funding to sustain the ongoing operation of eHealth infrastructure and services over the long-term, they will deteriorate. More clarity is required regarding what initiatives will be funded, how they will be funded, when funding will be received, and stipulations and requirements around funding.

Based on an analysis of expenditure investments across the six program components, effort should be made to re-examine where future investments are made to ensure appropriate resource allocations that maximize the achievement of program outcomes. For example, interoperability with provincial/territorial systems requires significant investment, and so a possible re-examination of both when and how resources are allocated should be incorporated into a long-term business case.

Theoretical modeling has shown that connectivity can improve both the efficacy of a health care service as well as the cost effectiveness. Cost effectiveness depends strongly on utilization. It is paramount that any technology-supported health information system that is implemented be used as much as possible (use maximization), or else it may not be cost effective.

6. Recommendations

Recommendation #1 - Create a Single Comprehensive and Detailed Business Case

The eHIP should consider preparing a comprehensive and detailed business case, with a focus on increasing adoption of modern systems, change management strategies, and utilization within communities to enable better access, quality and productivity in the health and health care of First Nations and Inuit communities. Findings suggest more detailed information regarding the eHIP's highest-level organizational goals (and how it will achieve them) is required, and such a plan would provide this level of detail.

A comprehensive and detailed business case would: allow eHIP to align and tie together important concepts and issues; form a complete picture of their Program (components, partners/stakeholders); help the Program anticipate financial requirements well into the future; ensure the Program's roles and responsibilities are clearly identified; provide an opportunity to track and closely study the national and international eHealth landscapes; and provide an opportunity to track and closely study the continuously evolving needs of the program's end users.

The business case would help the eHIP achieve the "improved ongoing integrated planning of complex eHealth systems" immediate outcome, as well as the "use evidence-based information to inform eHealth planning" immediate outcome. Consideration of best practices and building on past successes must also be incorporated into the business case.

eHealth and the integration of eHealth services is a very complex undertaking. A single comprehensive strategy and planning document will help to bring cohesion and alignment to this complex subject.

Recommendation #2 - Prioritize Investments and Further Clarify Funding Policies and Practices

Due to the high levels of relevance and alignment of the eHIP with stakeholder needs and federal government priorities, it is recommended that there is a continued investment of resources in all six eHIP components as identified in the HISAP in order to facilitate the achievement of long-term outcomes.

eHealth projects and services are capital and HR intensive to execute, implement and operate on an ongoing basis. Without adequate funding to sustain the ongoing operation of eHealth infrastructure and services over the long-term, they will deteriorate. The eHIP needs to ensure funding is adequate for both implementation projects and ongoing operations.

The eHIP should continue to monitor the readiness of FNIH Regions, communities and provinces in each program component to determine annual investments. In order to distribute funding accordingly, each FNIH Region should continue to provide an individualized annual workplan to identify the status of each program component and the planned goals/activities for the following fiscal year. Analysis should be performed on an ongoing basis as priorities shift and provincial public health surveillance implementation plans are advanced.

Funding should be aligned with needs and priorities of FN communities through a readiness assessment. Since eHealth priorities and progress vary significantly between communities, the eHIP should be flexible in working with communities to understand their unique needs and develop solutions.

Findings indicate more clarity is required by stakeholders regarding what initiatives will be funded, how they will be funded, when funding will be received, and other stipulations and requirements around funding.

Recommendation #3 - Build eHealth Capacity of FN Communities

The development and implementation of a community-based training strategy will improve the performance satisfaction levels of stakeholders with eHealth tool training available, and furthermore will support the FN&I eHealth Program's outcome of improving the capacity and proficiency in the use of eHealth applications in FN communities.

A community-based strategy allows for each FN/I community to assess gaps in eHealth knowledge, to identify training needs of service providers in their community, and to develop a strategy that is tailored to meet their specific needs. FN/I communities should be provided with sufficient guidance and support in the development and execution of their eHealth tool training strategy from Regional and/or Federal bodies to ensure consistency in the quality and availability of training across communities. A community-based training approach will also support greater involvement of FN/I communities in the planning and governance of eHealth.

Innovative methods of training delivery should be explored to improve availability of training opportunities, such as partnerships between FN/I communities in sharing training resources, and leveraging videoconferencing systems to support remote training.

Recommendation #4 - Implement a Comprehensive Communications Approach

The development and implementation of a Program-wide, multi-pronged, communications approach will increase awareness of eHIP's activities across all Regions and communities (one of the eHIP's immediate outcomes). It will also facilitate dialogue between stakeholders at various levels of government, as well as external stakeholders, and will provide greater opportunity for stakeholders at the community level to be involved in eHealth program planning and decision-making.

Keeping community level service providers and other stakeholders informed of the status of projects, upcoming initiatives, staffing changes, etc. is important to facilitate collaboration and build a positive work environment. Stakeholders have indicated their support for increased frequency and level of communication with FN communities and organizations as well as other partners including provincial partners.

Messaging in the communications should also seek to provide information about the benefits that have been realized from implementing eHealth services, about training and education opportunities, employment opportunities, eHealth progress related to goal achievement, and progress of eHealth tool implementations and services (e.g. lab information systems, public health surveillance systems, Telehealth, etc.).

Recommendation #5 - Implement a Refined Performance Measurement Strategy

The eHIP should refine and implement a performance measurement strategy to:

Components of this strategy could include:

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: