Evaluation of the Named Grant to the Institute for Environmental Monitoring and Research
Assistant Deputy Minister (Review Services)
Reviewed by CRS in accordance with the Access to Information Act (AIA). Information UNCLASSIFIED.
Table of Contents
Acronyms and Abbreviations
2.0 Findings and Recommendations
Annex A—Management Action Plan
Annex B—Evaluation Methodology and Limitations
Annex C—Logic Model
Annex D—Evaluation Matrix
- Asst CAS
Assistant Chief of the Air Staff
Aboriginal Traditional Knowledge
Canadian Armed Forces
Chief of the Defence Staff
Chief Review Services
- D Air CFG
Director Air Contracted Force Generation
- D Air Sp
Director Air Support
Defence Environmental Strategy
- Asst Comd RCAF
Assistant Commander of the Royal Canadian Air Force
Institute for Environmental Monitoring and Research
Department of National Defence
Environmental Impact Statement
Federal Sustainable Development Strategy
German Air Force
Low-level Training Area
Minister of National Defence
Operations and Maintenance
Royal Canadian Air Force
Species at Risk Act
Treasury Board Secretariat
Federal Environmental Assessment and Review Office
Multinational Memorandum of Understanding
The Named Grant to the Institute for Environmental Monitoring and Research (IEMR) is effective until March 31, 2015. In accordance with the Treasury Board Secretariat (TBS) Policy on Transfer Payments and the IEMR Named Grant Agreement, this evaluation was conducted to assess the relevance and performance of the IEMR.
- The IEMR has effectively achieved the expected outcomes related to research and monitoring activities; increased knowledge about the people, the flora and the fauna; and increased trust amongst Aboriginal peoples and other stakeholders.
- Primarily due to the very limited low-level Allied flight activities at 5 Wing Goose Bay, the Named Grant to the IEMR is no longer required in its current form.
The IEMR is an independent, provincially incorporated, not-for-profit organization. The IEMR was created in May 1995 as a necessary condition of approval for the provision of low-level Allied flight training activities at 5 Wing Goose Bay.1 Aboriginal groups living in and near the training areas were concerned that the noise caused by significant increases in low-level flight training activities would endanger their way of life. As an arm’s length organization, the IEMR performs monitoring and research on the environmental impacts and socio-economic effects of low-level Allied flight training activities in the low-level training area (LLTA). It also acts as catalyst for engagement and consultations with Aboriginal peoples and other stakeholder groups concerning those issues.
The IEMR is headed by a Chairperson who reports to both the Minister of the Environment and the Minister of National Defence (MND). The IEMR Board is comprised of nine voting members from Aboriginal groups and local communities, and five non-voting members from provincial and federal governments, which as a group set research priorities and ensure that relevant criteria and policies are followed. A Scientific Review Committee has also been established to provide scientific credibility and oversight for the IEMR. For DND/CAF, the Royal Canadian Air Force (RCAF)/Director Air Support (D Air Sp) is responsible for the management of all aspects of the Named Grant.
IEMR’s activities contribute to DND/CAF’s support to the federal government’s roles and responsibilities for appropriate environmental stewardship. The activities align with the federal government’s Federal Sustainability Development Strategy (FSDS) and DND/CAF’s Defence Environmental Strategy (DES), which strive to minimize environmental impacts.
The IEMR continues to be instrumental in bringing Aboriginal groups, municipalities, provincial governments, federal departments and environmental experts together to discuss and consult on issues related to low-level Allied flight training activities. However, with the current limited number of low-level Allied flights occurring at 5 Wing Goose Bay, the established environmental 5 Wing Goose Bay Mitigation Program, and the research conducted during the last 17 years, there is no longer a need for the IEMR to continue to research and monitor the effects of low-level Allied flight training activities in the LLTA.
Multiple lines of evidence indicate that the IEMR has performed its activities successfully. The IEMR has provided DND/CAF with a significant amount of monitoring and research data, which has contributed to the development of the 5 Wing Goose Bay Mitigation Program. The studies, consultations and socio-economic research have led to a substantial increase in knowledge of the people, the flora, the fauna and the environmental risks to northeastern Québec and Labrador. The IEMR has been successful in developing a strong level of cooperation and trust between disparate stakeholders. Nevertheless, it is difficult to assess the extent to which the IEMR has contributed to minimizing the environmental impacts, specifically from low-level Allied flight training activities due to the significant decrease of low-level Allied flight training, the substantial increase of civilian air traffic in the Goose Bay area, and the wide range of new economic development activities in the region.
The Evaluation examined the extent to which the IEMR has used appropriate, efficient, and economical means to achieve expected outcomes. The IEMR’s indirect costs were found to be in line with comparable research centres. However, with the significant reduction of low-level Allied flight activities at 5 Wing Goose Bay, the Named Grant to the IEMR is no longer providing value for money.
Key Findings and Recommendation
Finding 1: IEMR has fulfilled the requirements for an independent research and monitoring body to study the effects of low-level Allied flight training activities. Based on 17 years of completed environmental research, current low-flight rates, and established mitigation measures, continued research and monitoring to support an low-level Allied flight program in the LLTA is no longer required.
Finding 2: IEMR has become a successful forum, bringing together representatives from Aboriginal groups, municipal, provincial and the federal government representatives, as well as environmental experts in both informal and established working partnerships to discuss and consult on issues and impacts related to low-level Allied flight training activities.
Finding 3: As long as any type of military training activity continues in the Labrador and northeastern Québec areas, Aboriginal groups, provinces and local communities expect:
- The continuation of an environmental research and monitoring program.
- The continuation of informal discussions and consultations regarding the effects of military activities on the environment.
Finding 4: The delivery of IEMR activities aligns with the federal government’s and DND/CAF’s roles and responsibilities.
Finding 5: IEMR activities align with federal government priorities and the DND/CAF strategic outcomes.
Finding 6: IEMR has provided DND/CAF with a significant quantity of research and monitoring data, as well as mitigation recommendations, to the satisfaction of all stakeholders.
Finding 7: IEMR studies, consultations and socio-economic research have led to a substantial increase in knowledge about the people, the flora, the fauna and the environmental risks to northeastern Québec and Labrador.
Finding 8: The level of trust between IEMR and Aboriginal peoples and other stakeholder groups has remained high during the past five years.
Finding 9: The relationship between Aboriginal groups and DND/CAF remains fragile. The relationship between DND/CAF and the other stakeholders remains good.
Finding 10: IEMR has contributed to well-informed decisions by 5 Wing Goose Bay to mitigate the environmental effects of low-level Allied flight training activities.
Finding 11: The extent to which IEMR contributes to minimizing the environmental impact of low-level Allied flight training is difficult to assess due to limited military low-level flying and increasing socio-economic activities in the region.
Finding 12: IEMR’s indirect costs are in line with other comparable research centres.
Finding 13: Given the current significant reduction in the number of low-level Allied flight activities, the Named Grant to IEMR is no longer providing value for money.
Recommendation: As the Evaluation found that there is no longer a need for the Named Grant to IEMR in its current form, the Evaluation recommends the funding of the Named Grant to IEMR be ceased when it expires on March 31, 2015.
Note: Please refer to Annex A—Management Action Plan for a complete list of recommendations and management responses.
1.1 Profile of the Institute for Environmental Monitoring and Research
The Evaluation of the Named Grant to the IEMR was conducted by Chief Review Services (CRS) from September to November 2013, as a component of the DND Five-Year Evaluation Plan (2012/13 to 2016/17). The Evaluation was conducted in accordance with the TBS Policy on Evaluation (2009)2 and the 2008 TBS Policy on Transfer Payments.3 The Evaluation examines the relevance and the performance of the Named Grant to the IEMR.
The IEMR was established in 1995 in response to a recommendation by an independent Environmental Assessment Panel,4 which was appointed by the Federal Environmental Assessment and Review Office (FEARO) to review Allied military flight activities in Labrador and northeastern Québec. In this context, low-level Allied flights refers to flights conducted under 1,000 feet.
The FEARO review was sought in response to Aboriginal groups living in and near the LLTA who felt that the noise caused by low-level flight training activities was negatively impacting the land, the flora and the fauna, endangering their way of life. Several demonstrations and protests by the Aboriginal groups received global media attention, which, in turn, threatened to undermine the viability of a North Atlantic Treaty Organization flight program at 5 Wing Goose Bay. FEARO’s 1995 Report contained 58 recommendations, and proposed the establishment of the IEMR as a necessary condition to the approval of a Multinational Memorandum of Understanding (MMOU)5 between Canada and various North Atlantic Treaty Organization countries for the provision of low-level flight training activities in the LLTA.
At its peak in the mid-1980s, approximately 7,000 flights per year6 were taking place in the LLTA, yet since early 2000 the Allied requirement for low-level flight training activities has diminished significantly. In 2006, the MMOU ended, and arrangements that were previously covered under it were replaced with separate pricing agreements with individual nations; this arrangement remains in place today.
In 2009, the CRS conducted an evaluation of the IEMR. The Evaluation recommended the continuation of the Named Grant to the IEMR, but highlighted areas of improvement. The Evaluation made five recommendations, four of which have been completed. Recommendation #1, which was not fully implemented, was developed due to the range of ongoing military activities in the LLTA, and to clarify some inconsistent terminology7 across key documentation. This is the recommendation:
Conduct a joint review of the Mission statement, Mandate and Grant Agreement of the IEMR to ensure all military-related training activities in the low-level training area (LLTA) are included.8
This recommendation received a Management Action Plan response stating that:
There is potential for developing new military training opportunities in the LLTA (other than low-level flying activities) such as Close Air Support/Forward Air Controller (CAS/FAC), supersonic flight and ground forces training. In consultation with the IEMR, the present Grant Agreement [6 September, 2005] is being reviewed and the new Grant Agreement will incorporate the potential additional military training activities in the LLTA.
However, the new Grant Agreement, dated November 24, 2009, maintained the same parameters and ambiguous terminology as the 2005 Grant Agreement, such that the IEMR’s mandate continued to involve only research and monitoring related to low-level flights. During recent interviews, several IEMR Board members explained that the ambiguous wording remains an issue of contention for them.
1.1.2 Program Description
The IEMR is an independent, provincially incorporated, not-for-profit organization that operates at arm’s length from DND/CAF. According to its mission statement,9 the IEMR conducts scientific research and monitoring initiatives related to low-level Allied flight activities on the Labrador and northeastern Québec ecosystems within the LLTA, which is administered by 5 Wing Goose Bay. Research is also conducted on the socio-economic effects of military training activities in the area.
The IEMR is headed by a Chairperson who reports to both the Minister of the Environment and the MND. The IEMR Board consists of nine voting members, (representing Aboriginal communities and local communities and organizations), the Chair, five non-voting representatives from provincial and federal wildlife agencies, and DND/CAF.10 The Board is responsible for setting research priorities and ensuring that relevant criteria and policies are followed in soliciting and awarding funding for research activities. The Scientific Review Committee is comprised of recognized experts selected for their expertise and knowledge about issues related to the northern ecosystem and its people. The Scientific Review Committee provides scientific direction and support; conducts a peer review process for all scientific publications from the IEMR; ensures the relevance of proposed monitoring and research activities to the mandate of the IEMR; and advises the Board on the monitoring and research priorities.
The 2009 Named Grant Agreement identifies the Director Air Contracted Force Generation (D Air CFG) as the Minister’s representative, for administrative purposes, to approve, sign and amend the Grant Agreements with the IEMR. The D Air CFG is now referred to as the D Air Sp.11
1.1.3 Program Objectives
As per the documentation and files reviewed, the mandate12 of the IEMR is to:
- focus on the protection of the environment and, within the concept of sustainable development, support the viability of the military flight-training program;
- provide independent verification of environmental effects as well as expertise and advice on structuring adequate monitoring and mitigation measures; and
- foster a level of trust amongst all groups affected by the military training program.
The Objectives13 of the IEMR are to:
- initiate, coordinate, support and conduct environmental research;
- monitor effects and propose mitigation measures associated with low-level flight training activities over northeastern Québec and Labrador;
- foster inclusion or incorporation of Aboriginal environmental knowledge and cooperation in research and monitoring activities amongst DND/CAF, scientific establishments, research institutions, consultants and universities interested in the mandate and objectives of IEMR;
- promote a solid understanding and flow of information with the public on the work of IEMR; and
- provide advice and information to stakeholders to ensure that the environmental impact from military training activities is minimized to the fullest extent possible.
The activities, outputs and outcomes of IEMR are illustrated in the program Logic Model at Annex C.
For the purpose of this Evaluation, the term “stakeholder” refers to individuals, groups or organizations responsible for, or that may be affected by, low-level Allied flight training in the LLTA. Stakeholders also include one other federal department, provincial governments and affected communities. The DND/CAF stakeholders for this Evaluation are the Deputy Minister (DM), the Chief of the Defence Staff (CDS) and the Commander of the RCAF. The Program Manager is the RCAF/D Air Sp, and the main Program stakeholder is the IEMR. The primary recipient stakeholders are: 5 Wing Goose Bay; the Innu Nation; the people of Nunatsiavut; the Makivik Corporation; the NunatuKavut Community Council; the Naskapi Nation of Kawawachikamach; the Mamit Innuat; the people of the Municipalité régionale de comté de Minganie; the people of the Town of Churchill Falls; the people of Happy Valley-Goose Bay; and the Central Labrador Economic Development Board, Inc. The secondary recipient stakeholders are: Environment Canada; the Government of Newfoundland and Labrador; and the Government of Québec.
1.2.1 Coverage and Responsibilities
The Evaluation covers the grant-financed activities of IEMR since 2009. The focus is on the assessment of the continued relevance and performance of the IEMR. In terms of performance, the Evaluation focuses on research, information distribution and consultation, as described in Section 1.1.3 (Program Objectives).
The work of IEMR assists DND/CAF to accomplish activities under Program Activity Architecture 4.3.1—Sub-Program “Environment,” Program “Environmental Protection and Stewardship.”14
IEMR receives $1.5 million per year for a total funding of $7.5 million over five years.
IEMR’s staff consists of four full-time employees and the Chairperson, who works on a per diem basis. The staff work either from the IEMR offices in Moncton or Happy Valley-Goose Bay, as indicated in Table 1.
Table 1. Positions and Locations of IEMR Offices.
This table presents IEMR employee positions and their office locations. The left-hand column lists the employee positions and the right-hand column lists the location associated with the position.
|Chair||Moncton, New Brunswick|
|Executive Assistant to the Chair||Moncton, New Brunswick|
|Administrative Manager||Happy Valley-Goose Bay, Newfoundland|
|Geographic Information Systems Specialist||Happy Valley-Goose Bay, Newfoundland|
|Secretary||Happy Valley-Goose Bay, Newfoundland|
1.2.3 Issues and Questions
In accordance with the TBS Directive on the Evaluation Function (2009),15 the Evaluation addressed the five core issues related to relevance and performance. An Evaluation matrix listing each of the evaluation questions, with associated indicators and potential data sources, is provided at Annex D. The methodology used to gather evidence in support of the evaluation questions is located at Annex B.
Question 1: To what extent does the IEMR continue to address a demonstrable need?
Question 2: Does the federal government (and DND/CAF specifically) continue to have roles and responsibilities in the delivery of IEMR activities?
Question 3: Do IEMR activities align with current federal government priorities and DND strategic outcomes?
Performance – Effectiveness
Question 4: To what extent does IEMR meet the expected outcomes?
Question 5: Is the funding of IEMR the most efficient or economical means of achieving the expected outputs and outcomes for DND/CAF?
2.0 Findings and Recommendations
Evaluation findings and recommendations are outlined in Sections 2.1 through 2.6.
2.1 Continued Need
This section examines the extent to which IEMR activities continue to address a demonstrable need. To make this determination, two key indicators were assessed:
- analysis of empirical data (including research results, monitoring activities and flight activity data) from 2009-2013 to determine if there was a demonstrable need for the program; and
- assessment of qualitative evidence provided by the opinions of program managers and primary and secondary program recipients (stakeholders) regarding the need for the program.
The findings are based on evidence from document reviews and key informant interviews.
Key Finding 1: IEMR has fulfilled the requirements for an independent research and monitoring body to study the effects of low-level Allied flight training activities. Based on 17 years of completed environmental research, current low flight rates, and established mitigation measures, continued research and monitoring to support a low-level Allied flight program in the LLTA is no longer required.
Key Finding 2: IEMR has become a successful forum, bringing together representatives from Aboriginal groups, municipal, provincial and federal government representatives, as well as environmental experts, in both informal and established working partnerships to discuss and consult on issues and impacts related to low-level Allied flight training activities.
Key Finding 3: As long as any type of military training activity continues in the Labrador and the northeastern Québec areas, Aboriginal groups, provinces and local communities expect:
- The continuation of an environmental research and monitoring program.
- The continuation of informal discussions and consultations regarding the effects of military activities on the environment.
Empirical data and qualitative evidence revealed varying interpretations and opinions as to whether the IEMR continues to address the need for continued research and monitoring activities. These differences of opinion are a result of stakeholders having different interpretations of the IEMR’s raison d’être. The Aboriginal groups and other stakeholders expect that IEMR will continue to conduct environmental research and monitoring on the effects of all military activities in the LLTA. DND/CAF (represented by D Air Sp and 5 Wing Goose Bay representatives), on the other hand, emphasizes that the IEMR was established as a condition of approval for low-level Allied flight training activities.
Between the years 2009 and 2013, less than 200 Allied flights in total took place in the LLTA, despite the efforts of DND to attract Allies to use the facility. Due to the lack of Allied interest in using the LLTA for low-level flight training, DND/CAF recently reviewed the role of 5 Wing Goose Bay. In March 2013, the CDS provided guidance16 on new military activities planned in the 5 Wing Goose Bay areas, which will include: doubling the Canadian Ranger presence in Labrador to 600 positions, as well as establishing a Canadian Rangers Training Centre at 5 Wing Goose Bay; improving infrastructure at 5 Wing Goose Bay in order to increase capacity to host Canadian and international sea, land and air training events; and enhancing sovereignty through the expansion of the contracted maritime/coastal surveillance patrols of the eastern maritime approaches to Canada’s Arctic.
The last season of significant low-level military jet training was in 2005, with a total of 1,150 sorties conducted.17 During the past five years, with the exception of 2011 when the RCAF conducted Close Air Support Training with contracted Alpha jets, only the German Air Force (GAF) and the United States Air Force (USAF) have used the LLTA for flight training. The GAF has conducted ▓▓▓▓▓▓▓▓▓▓▓▓▓ low-level flight training with Transall C-160 two-engine turboprop transport aircraft, and the USAF has conducted flight training with Hercules C-130 four-engine turboprop transport aircraft. As illustrated in Table 2, the number of flights conducted by the GAF range from ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓, and the USAF ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
Table 2. Flights in the LLTA from 2009 – 201318
This table illustrates the number of flights conducted between the years 2009 – 2013. The furthest column to the left lists the year, the next column lists the type of training, followed by the number of flights and in the last column are listed the organization and type of aircraft used.
|Year||Type of Training||Number of Flights||Organization and Type of Aircraft|
|2009||Low-Level Flight Training||▓▓▓||GAF C160|
|2010||Low-Level Flight Training||▓▓▓||GAF C160|
|2011||Low-Level Flight Training||▓▓▓||GAF C160|
|2011||Close Air Support Training||13||CAF Contracted Alpha-Jet|
|2012||Low-Level Flight Training||▓▓▓||GAF C160|
|2012||Low-Level Flight Training||▓▓▓||USAF C130|
|2013||Low-Level Flight Training||▓▓▓||GAF C160|
Both military and civilian aircraft use the Goose Bay Airport, although civilian aircraft activity has continually increased since 1993. Table 3 demonstrates that in 1993 usage rates between military and civilian aircraft were comparable, with 52 percent being military and 48 percent being civilian. In 2011, civilian aircraft activity at the Goose Bay Airport totalled 95 percent of all flights, while the number of military flights decreased to 5 percent.19
Table 3. Percentage of Military and Civilian Plane Usage at Goose Bay Airport
This table compares the usage rates between military and civilian aircraft at the Goose Bay Airport, for 1993, 1999, 2007 and 2011. On the left-hand column is the Usage, including military and civilian. Following across are the years and associated percentage of usage.
Potential noise disturbance from civilian aircraft activity now far outweighs any effect from military aircraft activity. An IEMR report20 noted that “most low-level training is conducted by Transall or Hercules transport aircraft, which are both quieter than and do not fly as low as fighter aircraft.” Based on an assessment of the IEMR research data collected over the past 17 years, the IEMR’s Scientific Review Committee concluded that for many of the wildlife species being studied, the flight restrictions or closures should be lifted given the limited number of low-level Allied flights currently being conducted, “particularly in regards to jet fighter aircrafts.” For other species, the Scientific Review Committee recommended minor modifications to the established mitigation measures or maintaining the status quo. However, the Scientific Review Committee recommended that a re-evaluation of the mitigation measures be conducted should significant changes21 in military activity occur at 5 Wing Goose Bay in the future.
As interviews revealed, RCAF stakeholders believe that IEMR environmental monitoring and research is no longer required. They state that the IEMR was created as a pre-condition for the Allied military flight-training program, which is also reflected in the IEMR’s Constitution.22 Moreover, there have only been limited low-level Allied flights in recent years, with no anticipated increase. The RCAF (through D Air Sp and 5 Wing Goose Bay) is confident that recommended mitigation measures for low-level flight activities are sufficient.
Conversely, the Aboriginal peoples and other stakeholders stated in interviews that they believe that all military activities—not just low-level Allied flight training—can negatively affect the environment and, therefore, should be reflected in an expanded IEMR mandate. This interpretation is further reinforced by non-specific terminology in the 2009 Named Grant Agreement, which refers to “low-level and other flying training,” as well as “military training activities.” This issue arose in the 2009 CRS Evaluation (refer to Section 1.1.1 Background).
The expansion of the IEMR mandate to include ground-based military activities, and the continuation of environmental monitoring, is further supported by the Government of Québec, the Government of Nunatsiatvut and the Government of Newfoundland and Labrador, in their letters to the IEMR responding to the Scientific Review Committee’s review of the mitigation measures.23 The letter from the Government of Québec24 to the IEMR stated that: “IEMR should expand its mandate and …should also monitor and protect against military ground operations.”
The letter from the Government of Newfoundland and Labrador25 stated:
The Environmental Impact Statement (EIS)26 pertains to all types of military training and hence the obligation under the original EIS to monitor wildlife and mitigate for activities still appl[ies], inclusive of all foreign military training—both aerial and ground-based activities.
The Government of Nunatsiatvut letter to the IEMR,27 stated that:
While the current mandate is effective for monitoring noise disturbances caused by low-level flying military aircrafts, the Nunatsiatvut Government recognizes that the military training exercises now occurring at 5 Wing Goose Bay are primarily ground based […] IEMR should modernize its mandate to mitigate effects of military exercises in a changing training environment.
The IEMR has also become a forum for informal consultations28 and informed discussions between disparate groups affected by the low-level flight activities. It has brought together representatives from Aboriginal groups, environmental experts, and municipal, provincial and federal government representatives, in both informal and established working partnerships to study and monitor environmental effects of low-level flight activities in the LLTA. All stakeholders agreed that the IEMR has promoted and continues to successfully cultivate an effective level of communication, trust and understanding.
2.2 Alignment with Federal Roles and Responsibilities
This section examines the extent to which IEMR activities are aligned with the roles and responsibilities of the federal government. The findings in this section are based on evidence from document reviews.
Key Finding 4: The delivery of IEMR activities align with the federal government’s and DND/CAF’s roles and responsibilities.
Environment Canada, on behalf of the federal government, is responsible for the coordination of the policies and programs respecting the preservation and enhancement of the quality of the natural environment, including migratory birds and other flora and fauna.29 However, Environment Canada does not have the mandate to oversee the environmental aspects of Defence activities.
In support of the federal government’s environmental responsibilities, military activities at all CAF Bases and Wings are conducted in a manner that maintains environmental stewardship while achieving operational readiness. All military activities must comply with Defence Administrative Orders and Directives 4003-2 (Environmental Assessment), the Canadian Environmental Assessment Act, and the Environmental Assessment Regulations under the Newfoundland and Labrador Environmental Protection Act.
However, the situation at 5 Wing Goose Bay was unique due to the introduction of the low-level Allied flight training program in the 1980s. Additional measures were deemed necessary to ensure the continuation of this program with appropriate environmental stewardship. Prior to the establishment of IEMR, protests and demonstrations against Allied flight training activities were frequent in the Goose Bay area. As a result, the Canadian Environmental Assessment Agency’s Environmental Assessment Panel reviewed the low-level flying activities in Labrador and Québec. The Panel recommended the establishment of IEMR, which was accepted by the federal government.
The environment is a shared responsibility between the federal government30 and the provinces. This responsibility correlates with IEMR’s objective to initiate, coordinate, support and conduct environmental research to support the viability of the low-level Allied flight training in the LLTA.
2.3 Alignment with Government Priorities
This section examines the extent to which IEMR activities align with the federal government’s priorities and DND/CAF strategic outcomes. The findings in this section are based on evidence from documents reviewed for the Evaluation.
Key Finding 5: IEMR activities align with federal government priorities and the DND/CAF strategic outcomes.
Aboriginal consultation and environmental stewardship continue to be priorities for the federal government. The 2013 Speech from the Throne31 announced the federal government’s continued commitment to work in partnership with Aboriginal peoples to create healthy, prosperous, self-sufficient communities. The government’s 2013-2016 FSDS, released November 2013,32 emphasizes the importance of the federal government, the provinces, the territories and Aboriginal groups working together to protect Canadian landscapes, seascapes, ecosystems and species at risk.
In response to the 2010-2013 FSDS, DND/CAF developed the DES 2013.33 The vision of the Strategy is to manage Defence establishments, and ensure that training and operating areas are such as to minimize any effects on the environment while ensuring military readiness. The DES provides direction to DND/CAF on how to accomplish this.
IEMR assists DND/CAF to accomplish the Strategic Outcome of “Care and Support to the Canadian Armed Forces and Contribution to Canadian Society.” This program activity promotes public health and safety, and supports sustainable development on Defence lands with protection and stewardship that complies with applicable legislation and federal policies.
2.4 Achievement of Expected Outcomes (Effectiveness)
To determine the overall effectiveness of IEMR, a logic model of the program was developed. The logic model grouped the program activities by common outputs, and then linked outputs to intended outcomes (see Annex C).
The effectiveness of IEMR was assessed by applying appropriate performance measures and/or key performance indicators against each outcome. Data for the performance measures was obtained from reports, documents, studies, and financial reports provided by IEMR, as well as documents provided by D Air Sp staff, and from several interviews with the program manager and primary and secondary recipient stakeholders. Based upon an analysis of this information, the overall effectiveness of IEMR was determined. Accordingly, an assessment was made of the following outcome areas:
- Research and monitoring activities, as well as provision of mitigation program recommendations to DND/CAF;
- increased knowledge of the stakeholders in the people, the flora, the fauna and the environmental risks to northeastern Québec and Labrador; and
- increased trust between IEMR and Aboriginal peoples and other stakeholder groups.
- Well-informed decisions are made by DND/CAF to minimize the environmental impact of low-level Allied flight activities; and
- strengthened relationships between DND/CAF and Aboriginal peoples and other stakeholder groups.
Minimized environmental impact from low-level Allied flight activities.
2.4.1 Immediate outcome: Research and monitoring activities, as well as provision of mitigation program recommendations to DND/CAF.
2.4.2 Immediate outcome: Increased knowledge of the people, flora, fauna, and the environmental risks to northeastern Québec and Labrador.
The provision of IEMR research, monitoring and mitigation recommendations was assessed in the Evaluation by examining the comprehensiveness of the research and monitoring studies and the degree of stakeholder satisfaction with the quality of the studies and the mitigation recommendations. The Evaluation determined the degree of increased stakeholders’ knowledge of the people, the flora, the fauna and the environmental risks to northeastern Québec and Labrador through a document review and key informant interviews.
Key Finding 6: IEMR has provided DND/CAF with a significant quantity of research and monitoring data, as well as mitigation recommendations, to the satisfaction of all stakeholders.
Key Finding 7: IEMR studies, consultations and socio-economic research have led to a substantial increase in knowledge about the people, the flora, the fauna and the environmental risks to northeastern Québec and Labrador.
A review of relevant documentation shows that IEMR’s work during the last five years has consisted of: data collection; compilation and analysis of data; animal population surveys; acquisition of high-resolution imagery data; coordinating and attending workshops and training sessions; community consultation; collecting data from unmanned aerial vehicles,34 mitigation reviews and reviews of avoidance criteria; study of Forward Air Control Training; validation of the BoomCast model35; and updating the Economic Trend Analysis.36 As a result of these and other IEMR monitoring and research initiatives, 31 reports were produced37 either for or by IEMR between February 2009 and October 2012. These reports are accessible to the public through IEMR’s website.38
To promote a general interest in the environment, and to establish close contact with the local communities and youth, IEMR also hosted annual photo contests that engaged students from schools in the region.39
To increase knowledge of specific wildlife species, IEMR formed Technical Committees, specifically designed to study noise impacts on species such as caribou, osprey and other waterfowl. The members of the various IEMR Technical Committees include local and international scientific experts, resource managers from affected governments, and Aboriginal experts.40
Table 4 charts comparisons between various wildlife species studied between 2005 – 2008 and 2009 – 2013. All the species studied in 2005 – 2008 continued to be studied throughout 2009 – 2013, although, given the changing military training activities at 5 Wing Goose Bay, the Scientific Review Committee recommended the study of additional species. Boreal owls and bats, porcupines and beavers, and land and shore birds, were added to the list of species being studied as of 2009.
Table 4. Comparison of Species Studies in 2005 – 2008 and 2009 – 2013
The table compares various wildlife species studied between 2005 – 2008 and 2009 – 2013. The left-hand column has the names of the species studied by the IEMR, which include: Woodland Caribou; George River Caribou Herd; Habitat Data, Harlequin Duck; Osprey; Small Mammals; Barrow’s Goldeneye; Sea Ducks; Golden Eagles and Bald Eagles; Moose; Boreal Owls and Bats; Porcupines and Beavers; and Land Birds and Shore Birds. The next two columns are the years of study and they identify if the species were studied during those years.
|Species Studied by the IEMR||2005–2008||2009–2013|
|George River Caribou Herd||Yes||Yes|
|Golden Eagles and Bald Eagles||Yes||Yes|
|Boreal Owls and Bats||No||Yes|
|Porcupines and Beavers||No||Yes|
|Land Birds and Shore Birds||No||Yes|
The Scientific Review Committee ensures credibility of IEMR monitoring and research processes and activities. Once the annual priorities are set, the IEMR Board makes sure that relevant policies and criteria are followed when soliciting contractors and funding awards. The Scientific Review Committee stays engaged with contractors, provides scientific direction and support for approved monitoring and environmental research activities, and then concludes with peer reviews of the completed studies.
In addition to conventional research and monitoring techniques, the IEMR also incorporates Aboriginal traditional knowledge (ATK) through:
- Aboriginal peoples’ recommendations, at IEMR Board meetings, for wildlife species to be studied;
- Aboriginal peoples’ companies conducting wildlife species research;
- Aboriginal peoples’ knowledge being incorporated at the workshop and technical committee levels;
- IEMR Board members (voting members) being Aboriginal peoples community representatives;
- IEMR consultations with community “Committees of Elders”; and
- inviting Elders from the communities represented on the Board to share their life experiences with the IEMR Board members.
While the opportunity for inclusion of Aboriginal peoples’ knowledge is encouraged at the IEMR Board meetings in support of IEMR research activities, some IEMR Board members indicated that there is still room for improvement.41 As noted during the interviews with IEMR Board members, there is a consensus that the incorporation of ATK in IEMR research and monitoring activities is improving; however, the level of satisfaction by the Aboriginal members of the Board ranged from 5 through 7 on a 7-point scale. Almost all the Board’s non-Aboriginal members rated their satisfaction from 6 through 7, except one member who rated it as a 4 on the 7-point scale. This last assessment does not reflect how well IEMR is doing in this regard, but how much further ATK needs to be incorporated into all environmental decision-making.
In 2012, the IEMR contracted AMEC, an engineering, project management and consultancy company,42 to produce its eighth economic impact and trend analysis report, to continue socio-economic research for the Happy Valley-Goose Bay area.43 The 2012 report highlights the impacts that 5 Wing Goose Bay’s changing activities have had on the local economy. The reduction of the Allied flight training activities at 5 Wing Goose Bay initially had a major effect on the local and provincial economies, although the new focus on environmental cleanup and capital expenditures at 5 Wing Goose Bay has helped to improve the economies. The region has also experienced economic diversification, with substantial increases in business and natural resources development.
All IEMR Board members agreed that the significant amount of monitoring and research conducted by the IEMR has led to an increased knowledge of the people, the flora, the fauna and the environmental risks to northeastern Québec and Labrador. On this issue, both the Aboriginal members and non-Aboriginal members provided a rating between 6 and 7 on a 7-point scale.44 As an overall assessment, most IEMR Board members felt that the Named Grant to the IEMR has been money well spent. The Environment Canada representative noted45 that collaboration with IEMR has provided assistance in reaching areas that are very difficult to access and work in; and, in turn, it contributed to increased knowledge that would otherwise not have been generated. An RCAF representative noted that the areas in and surrounding 5 Wing Goose Bay are the most studied areas in Canada, in terms of the impact of military operations on the environment.46
Despite the comprehensiveness of the monitoring and research on the effects of low-level flight activities in the LLTA, the Aboriginal members of the IEMR Board are concerned that other military activities do have a cumulative effect on wildlife, particularly on the caribou. As the cumulative effects of all military activities have not been studied, due to the scope of the current IEMR mandate, Aboriginal members47 strongly believe that there is a need for an expanded IEMR mandate to continue research and monitoring of the potential environmental effects from all military activities.
2.4.3 Immediate outcome: Increased Trust between the IEMR and Aboriginal Peoples and Other Stakeholder Groups.
The Evaluation assessed the perceived level of trust between the IEMR and the Aboriginal peoples and other stakeholder groups through key informant interviews.
Key Finding 8: The level of trust between IEMR and Aboriginal peoples and other stakeholder groups has remained high during the past five years.
The Aboriginal members of the IEMR Board spoke at length about the mistrust between the various Aboriginal groups and DND/CAF prior to the creation of the IEMR.48 According to the Aboriginal members, the years of work that IEMR has put towards building trust with the Aboriginal groups has been successful. It is important to the Aboriginal members to have consistency in order to create and maintain trust. It was reported that there is a strong level of trust and a positive relationship between IEMR and the Aboriginal members, facilitating dialogue amongst all the various stakeholder groups. A representative from the Labrador North Chamber of Commerce stated that:
Despite political differences of the day, despite differences in outlook on given issues, this is the one forum where business, science and ATK come together and are discussed in a calm and straightforward manner without any political overtones or echoes.49
The Program Manager’s staff explained that IEMR was able to build trust by ensuring that the stakeholders were involved in the decision-making of research and monitoring activities.50 Stakeholders could, therefore, see that the data and conclusions were not just provided by DND/CAF, but were based on independent scientific research.
There was consensus with all IEMR Board members that the level of trust between IEMR and the Aboriginal members and non-Aboriginal members was high. The non-Aboriginal members of the IEMR Board felt that the level of trust has remained consistently high. The rating by the Aboriginal members was a unanimous 7, while the non-Aboriginal members rated it between 6 and 7 out of 7.
2.4.4 Intermediate Outcome: Strengthened Relationship between DND/CAF and Aboriginal Peoples and other Stakeholder Groups.
The Evaluation used key informant interviews to determine the perceived extent to which the relationship between DND/CAF and Aboriginal peoples and other stakeholder groups has been strengthened.
Key Finding 9: The relationship between Aboriginal groups and DND/CAF remains fragile. The relationship between DND/CAF and the other stakeholders remains good.
Interviews with the Aboriginal members of the IEMR Board51 revealed that the quality of the relationship between DND/CAF and the Aboriginal groups has not improved significantly during the last five years. The Aboriginal members believe that 5 Wing Goose Bay staff have not tried to understand the Aboriginal peoples’ perspectives. One major source of frustration for the Aboriginal representatives is the fact that the senior military members at 5 Wing Goose Bay change, on average, every two years. The Aboriginal members expressed some frustration with the fact that they must re-start relationship-building each time a new Wing Commander is in place. The Aboriginal members rated their satisfaction of the relationship with DND/CAF between 3 and 4 out of 7.
Non-Aboriginal members of the IEMR Board, on the other hand, indicated that they have a good relationship with DND/CAF. Some did mention that they would like more transparency and openness from 5 Wing Goose Bay staff, especially in regards to military activities in the LLTA.52 Every non-Aboriginal member of the IEMR Board, except one, rated their satisfaction as 6 out of 7. One Board member rated it as 4 because she felt that the relationship with DND/CAF would be the same regardless of IEMR efforts.
2.4.5 Intermediate outcome: Well-informed Decisions to Mitigate the Environmental Effects of Low-Level Allied Flying Activities.
In order to measure this intermediate outcome, the Evaluation undertook key informant interviews to determine the perceived extent to which the activities of the IEMR led to more informed decision-making by DND/CAF.
Key Finding 10: IEMR has contributed to well-informed decisions by 5 Wing Goose Bay to mitigate the environmental effects of low-level Allied flight training activities.
The 5 Wing Goose Bay is responsible and accountable for the design and implementation of a Mitigation Program. This Mitigation Program assists in determining the airspace that can be open for low-level Allied flight in the LLTA, with the objective of minimizing potential adverse effects on wildlife associated with noise originating from the low-level flights.
The data that contributes to the development of the 5 Wing Goose Bay Mitigation Program comes from multiple sources: the IEMR, the province of Québec and the province of Newfoundland and Labrador. Information concerning the location of sensitive wildlife is provided daily to the 5 Wing Goose Bay Mitigation Officer by IEMR. Twice weekly, the two provinces53 provide 5 Wing Goose Bay with information on the locations of caribou.54 Since 2009, the IEMR also provides their annual Monitoring and Research Workplan to DND/CAF, which summarizes the field work completed during the past year and identifies the areas of interest for the following year.
Avoidance of sensitive wildlife in the LLTA is implemented through sensitive area closures, which are based on criteria that have evolved since the creation of the IEMR. The 5 Wing Goose Bay Mitigation Officer and the Wing Community Liaison Officer are responsible for the compilation and assessment of Wildlife Avoidance restrictions and Human Avoidance restrictions, respectively. Their closure recommendations55 are presented to the 5 Wing Operations Officer. Taking the closure recommendations into account, the Operations Officer determines any implications for the scheduled training program, and coordinates the low-level Allied flight activities within the LLTA airspace.56
Not only does IEMR contribute to the DND/CAF’s ability to make well-informed decisions to minimize the environmental effects of low-level Allied flight activities, but if studies show that those flight activities lead to significant negative effects on the local people, flora or fauna, IEMR may recommend that appropriate measures be taken. IEMR can also recommend to the MND and the Minister of the Environment that military flight activities be limited or phased out.57
2.4.6 Ultimate outcome: Minimized Environmental Impact from Allied Low-Level Flight Training Activities.
The Evaluation examined program documents and reports to identify the extent to which the IEMR has contributed to minimizing the environmental impact of low-level Allied flight training activities. Key informant interviews were also used to assess the perceptions of the IEMR contributions.
Key Finding 11: The extent to which IEMR contributes to minimizing the environmental impact of low-level Allied flight training is difficult to assess due to limited military low-level flying and increasing socio-economic activities in the region.
IEMR has undertaken several monitoring and research activities since 2009. The data from these activities has contributed to the 5 Wing Mitigation Program. It is difficult, however, to determine the extent to which IEMR has contributed to minimizing the environmental impact of low-level Allied flight activities, simply because there have been so few low-level Allied flights during the last five years. Additionally, there are other human, economic development activities and environmental factors that may also impede the determination of causality.58 As noted by the Aboriginal members of the IEMR Board, the caribou herds continue to decline, even with minimal low-level Allied flight training activities. As one IEMR Board member observed:
The mitigation measures may be stellar and cutting-edge, but with so few flights, it is hard to determine how effective they really are. It is a question of making sure the efforts are properly placed to get the maximum benefit.59
There is currently no information on the cumulative effects of other military training activities, and the wider range of activities stemming from the economic diversification in the region.
2.5 Demonstration of Efficiency and Economy
The following section examines the extent to which IEMR provides value for money60 by using the most appropriate, efficient, and economical means to achieve its expected outcomes. The Evaluation considered processes and mechanisms in place for monitoring efficiency and economy, expenditures related to IEMR in comparison to other comparable research centres, and alternative means used in other RCAF Wings. This information was gathered through program documents and reports review, literature review and key informant interviews.
Key Finding 12: IEMR’s indirect costs are in line with other comparable research centres.
Key Finding 13: Given the current significant reduction in the number of low-level Allied flight activities, the Named Grant to IEMR is no longer providing value for money.
IEMR budgets for the past five years are shown in Table 5. Within the Administration expenditures, the two main cost drivers were salaries and travel costs. For Research expenditures, the main cost drivers were Contractuals in 2010 and 2011, data acquisition in 2012 and 2013, and professional fees and honorariums (2010-2013).
Table 5. 2009 - 2013 IEMR Expenditures
This table shows the expenditures by categories for the years 2009 - 2013. The left-hand column lists the various expenses under Administration and Research, and provides a Total. The years are listed across the top row, and amounts are listed under each year.
Revenue + Interest
Salaries and fringe benefits
Interest and bank charges
Repairs and maintenance
Stationary and office supplies
Training and education
Amortization - tangible capital assets
Amortization - intangible capital assets
Professional fees and honorariums
Minor equipment and project supplies
Research services - Transportation
|Revenue less Expenditures||($445,174)||($161,765)||$107,254||$131,587||($118,441)|
Figure 1. Expenditures of the IEMR, in Dollars, 2009 – 2013.
Text description for Figure 1.
This figure illustrates the range in expenditures for years 2009 – 2013 for each of the categories: Administration salaries, Administration Operations and Maintenance (O&M), Research salaries, and Research O&M. The data shown in Figure 1 is summarized in Table 6.
Table 6. Expenditures of the IEMR, in Dollars, 2009 – 2013
This table compares the expenditures for salaries and O&M, for each of the administration and research categories, from 2009 – 2013. The left-hand column lists the expenditure categories, whereas years are listed across the top row. The amounts are listed under each year.
|IEMR Expenditure Categories||2009||2010||2011||2012||2013|
Table 7 compares IEMR expenditures over the past five years attributed to administration costs and research activity costs (salaries and O&M). Since 2009, administration expenditures increased from 38 to 45 percent, with the remainder attributed to decreases in research activities (62 to 55 percent). Over the past five years, IEMR’s administration expenditures or indirect costs61 represented an average of 45 percent of the grant contribution received from DND/CAF, which is comparable to Canadian university and college research centres, which average 40 percent.62
Table 7. Percentage of IEMR Administration and Research Expenditures, 2009 – 2013
This table compares the percent IEMR spent on research and administration between the years 2009 – 2013. The left-hand column lists both expenditure categories, namely Administration and Research. The top row lists years 2009 – 2013, with associated percentages for each category listed below.
The Chair of IEMR informed the Evaluation that IEMR has begun taking steps to reduce expenditures during the past two years. The number of IEMR Board meetings has decreased from four to two per year, and they are held in Halifax, which is the least expensive location for all members to travel to due to its central location. The Chair believes that greater efficiencies can be found, and foresees being able to cut the budget by $500,000 to $600,000 in the first year, with additional budget cuts in the following years. IEMR is currently working on a proposal based on an expanded mandate and a decreased budget.63
An aspect of IEMR’s value for money was assessed by comparing the total annual funding allocated to IEMR in relation to the total number of low-level Allied flights per year. The original funding of the IEMR anticipated 18,000 flights per year, which averages $83 per flight based on the current budget of $1.5 million. Using this same calculation, the average funding for the years 2009 to 2013 was over $50,000 per flight, as noted in Table 8. With the reduced numbers of low-level flight activities at 5 Wing Goose Bay, IEMR’s present funding level is no longer economical.
Table 8. Average Funding per Flight, 2009-2013
This table compares the total annual funding allocated to IEMR for 2009 – 2013, in relation to the total number of low-level Allied flights per year. The left-hand column lists the years 2009 – 2013. The next column identifies the number of flights in that year. The following column states the annual budget, which remained constant for all five years. The last column shows the average funding per flight.
|Year||Number of Flights||Annual Budget||Funding per Flight|
To understand how other RCAF Wings manage environmental responsibilities and to consider more economical means to achieve desired outcomes, the Evaluation interviewed Environmental Officers from CFB Kingston, 4 Wing Cold Lake, and 8 Wing Trenton,64 and the Wing Operations Officer from 4 Wing Cold Lake. Each Wing utilizes a blend of staff in positions such as Environmental Officer, Liaison Officer, and Legal Officer, and these may deal with both environmental issues and Aboriginal peoples’ concerns. The specific roles and responsibilities of these positions are different for each RCAF Wing, according to their specific issues and concerns.
The 5 Wing Goose Bay site is unique because there are six Aboriginal groups that derive much of their sustenance from the land, which includes the LLTA. One of these Aboriginal groups remains involved in land claims negotiations with the federal government.65 To add to this complexity, the LLTA is situated in both the province of Newfoundland and Labrador and the province of Québec. Additionally, no other RCAF Wing has the requirement to monitor and research environmental effects for one particular military activity, such as the low-level Allied flight training at 5 Wing Goose Bay. The functions that IEMR provides for 5 Wing Goose Bay are not replicated at any other RCAF Wing.
As established in the Relevance Section, continued research and monitoring by IEMR to support a low-level Allied flight program is no longer required. This assessment is based on the evidence provided by a recent comprehensive review of 17 years of completed environmental IEMR research; the current limited number of low-level Allied flights conducted annually; and the mitigation measures now established for low-level flight activities in the LLTA.
As established in the Performance Section, IEMR achieved all expected immediate and intermediate outcomes during the evaluation period. Of note, IEMR has fostered a high level of trust between disparate groups, which has benefitted all stakeholders. There remains a great deal of value in maintaining these relationships. The mandate and the funding level of the Named Grant to the IEMR, however, no longer reflects the reality of very limited low-level Allied flight activities at 5 Wing Goose Bay, and is no longer providing value for money.
Board members and other stakeholders have suggested expanding the mandate of IEMR to include the research and monitoring of environmental effects derived from all other military activities in the LLTA. However, given increased economic development and associated activities throughout the northeastern Québec and Labrador areas, the military activities in the Goose Bay training areas are no longer the only significant variables interacting with the people, flora and fauna.
1. As the Evaluation found that there is no longer a need for the Named Grant to IEMR in its current form, the Evaluation recommends the funding of the Named Grant to IEMR be ceased when it expires on March 31, 2015.
OPI: Comd RCAF
1. As the Evaluation found that there is no longer a need for the Named Grant to IEMR in its current form, the Evaluation recommends the funding of the Named Grant to IEMR be ceased when it expires on 31 March 2015.
Management Action Plan
The RCAF agrees with the CRS recommendation to cease funding the grant in its current form as of March 31, 2015. This will need to be communicated to the Chairperson of IEMR one year in advance; therefore, not later than March 31, 2014. This will permit IEMR to deal with administrative and management issues, such as lease cancellations and IEMR employees’ annuity payments. A communication plan will be developed following final direction from the CDS/DM in order to explain the rationale for this decision.
In accordance with the 2009 Named Grant Agreement (paragraph 14), “Changes to or termination of the Grant payment are subject to the changing of Departmental priorities as directed by the Minister. Consequently, the Minister reserves the right to cancel, reduce or terminate this Grant, in consultation with the Minister of Environment, as may be required.”
As such, to effect the cancellation of the Named Grant to the IEMR by March 31, 2015, the Comd RCAF will prepare a notification letter for MND signature following approval of this Management Action Plan by the CDS/DM. This letter must be sent to the Chairperson of the IEMR before March 31, 2014.
OPI: Comd RCAF
Target Date: March 31, 2014
Annex B—Evaluation Methodology and Limitations
The Evaluation used multiple lines of evidence and complementary qualitative and quantitative research methods as means to help ensure the reliability of information and data to support evaluation findings. The methodology established a consistent approach in the collection and analysis of data to support the evaluation findings, conclusions and recommendations. Based on the evidence from available sources, the Evaluation reviewed the achievement of expected outcomes, and the program’s efficiency and economy, to develop a balanced picture of the relevance and performance of IEMR. Information and data were correlated to each evaluation question and corresponding indicators. To ensure the validity of the data captured, the Evaluation used a data triangulation approach.
1.1 Overview of Data Collection Methods
Data collection methods were selected based on the data required to address performance indicators. The following data collection methods were used to gather qualitative and quantitative data for each type of operation in the Evaluation:
- literature and document reviews;
- key informant interviews; and
- administrative and financial data reviews.
1.2 Documents and Files Review
A preliminary document review was conducted as part of the planning phase of the Evaluation, to garner a foundational understanding of IEMR. A comprehensive document review was undertaken as part of the conduct phase of the Evaluation, focusing on the relevance and the performance of IEMR activities.
The following documents were reviewed during the conduct phase of the Evaluation:
- Program documents: The Treasury Board Submission, IEMR Annual Reports, IEMR Monitoring and Effects Workplans, Scientific Review Committee briefing books, Wildlife Avoidance Criteria, mitigation measures review, IEMR Economic Assessments 2009 and 2012, and other research and activities reports.
- Administrative documents: IEMR annual financial reports, Named Grant Agreements 2005 and 2009, IEMR Publication Policy, Memorandum of Understanding between DND and IEMR, Research Proposals Protocol Review Process, and Implementation Agreements for Data Provision and Correspondence Exchange.
- Accountability documents: DND/CAF Reports on Plans and Priorities, DND/CAF Departmental Performance Reports, Canada First Defence Strategy and Speech from the Throne.
The document review was conducted using a customized template organized according to the evaluation questions and indicators.
1.3 Key Informant Interviews
Key informant interviews and information sessions scheduled with IEMR stakeholders, who were directly or indirectly involved in program delivery, served as an important source of qualitative information.
Individual interviews were conducted in person66 with the program manager (D Air SP and 5 Wing Goose Bay) and the Chair of the IEMR. Individual interviewees were provided with the interview guide in advance. Interviews were recorded with the authorization of the interviewees. The evaluators transcribed the notes taken during the interviews and compared them with the recorded documents.
Two group interviews were conducted with IEMR Board members, one for the Aboriginal members and one for the non-Aboriginal members, including voting and non-voting members.67 Both group interviews followed the same procedures. The bilingual (English and French) questionnaire was provided and explained to the members before beginning the interview, encouraging them to be open and candid in their responses to the questions. For each question, a 7-point scale was used, anchored at the end-points with the terms “strongly agree” for 7 and “strongly disagree” for 1, and a “not applicable” (N/A) point outside the scale. A section for comments was also provided immediately following each question. On a few occasions, some participants challenged the answer provided by a colleague. The answers were aggregated in order to ensure anonymity. Interviews were recorded with the authorization of the interviewees. The evaluators transcribed the notes taken during the group interviews and compared them with the recorded documents.
Staff from CFB Kingston, 4 Wing Cold Lake, and 8 Wing Trenton were also interviewed, in an effort to understand how other RCAF Wings compare to 5 Wing Goose Bay, and how they deal with environmental and Aboriginal peoples’ concerns. Interviews were conducted by phone, with transcripts verified by interviewees for approval.
Table B-1: Number of Interviewees by Organizations
The interview groups are listed in the left-most column and include D Air Sp Staff, 5 Wing Goose Bay (email), IEMR Chair, Aboriginal Board Members, Other Board Members, 4 Wing Cold Lake, 8 Wing Trenton, and CFB Kingston. For each interview group, read across the row to determine the number of interviews and the number of interviewees. The last row provides the total number of interviews and interviewees.
|D Air Sp Staff||3||3|
|5 Wing Goose Bay (email)||2||2|
|Aboriginal Board Members||1||6|
|Other Board Members||1||10|
|4 Wing Cold Lake||2||2|
|8 Wing Trenton||1||1|
1.4 Review of Financial and Administrative Data
The IEMR financial data was reviewed in order to determine the degree of efficiency and economy of the activities and outputs. The data covered five calendar years, from 2009 to 2013, which was extracted from the IEMR financial reports.
The following table shows the limitations related to the sources and the mitigation strategies applied to address them.
Table B-2: Evaluation Limitations and Mitigation Strategies
The limitations are listed in the left-most column, while mitigation strategies appear in the right column. Read across the row to identify the limitation identified and the corresponding mitigation strategy.
Attribution of activities and outputs of the program to ultimate outcome was difficult due to the high level of influence of external factors and lack of data.
More focus was placed on measuring the immediate and intermediate outcomes, and, from these findings, deductions were made to assess the achievement of the ultimate outcome.
Possibility of the interviewees to provide biased information and only positive stories about their program.
A comparison was made between interviewees with other people from the same organization or group, and information from other sources (documents and files).
Different terminologies used for the mandate of the IEMR
The evaluators referred to the 2009 Grant Agreement to determine the specific terminology.
Different perspectives from the Aboriginal and non-Aboriginal members of the Board regarding the impact of the military training activities on the environment
Board members were interviewed in two separate groups: Aboriginal groups and non-Aboriginal groups.
Figure C-1. Logic Model for the IEMR
Text description for Figure C-1.
The logic model shows the program’s four main activities, leading to outputs, immediate and intermediate outcomes, and to the ultimate outcome of minimized environmental impact with respect to sustainable development and viability of low-level Allied flight training activities.
The Logic Model for the IEMR is described as a series of inputs and activities that lead to four main outputs. The four outputs lead to three immediate, two intermediate, and one ultimate outcomes associated with the program, and one departmental strategic outcome supported by the program. The Logic Model components are broken down as follows:
- Named Grant Agreement 2009;
- Species at Risk Act; and
- Canadian Environmental Assessment Act.
- initiate, coordinate, support and conduct research;
- monitor effects of low-level Allied flight training activities;
- foster the inclusion of ATK and cooperation in research and monitoring activities; and
- promote a solid understanding of flow of information with the stakeholders on the work of the IEMR.
- comprehensive environmental and socio-economic research;
- real-time monitoring studies and effects-monitoring studies;
- incorporation of ATK into all research and monitoring activities; and
- solid engagement and communication with stakeholders.
- Immediate Outcomes
- research and monitoring activities, as well as provision of mitigation program recommendations to DND/CAF;
- increased knowledge of the people, the flora, fauna and the environmental risks to northeastern Québec and Labrador; and
- increased trust amongst various Aboriginal peoples and other stakeholder groups.
- Intermediate Outcomes
- well-informed decision making regarding the environmental impact of low-level Allied flight training activities; and
- relationships and partnerships are established and strengthened with Aboriginal peoples and other stakeholder groups.
- Ultimate Outcome
- minimized environmental impact with respect to sustainable development and viability of low-level Allied flight training activities.
- Strategic Outcome
- Defence remains continually prepared to deliver care and support to the CAF and to contribute to Canadian society (supporting sustainable development on Defence lands with protection and stewardship, in compliance with legislation and federal policies).
The three evaluation questions related to relevance are listed in the left-hand column. Read across the rows to determine the indicators used to assess each evaluation question, and the lines of evidence used for the indicators and questions. The columns for the lines of evidence indicate whether that particular line of evidence was used for each indicator.
|Evaluation Issues/ Question||Indicators||Details (info sources/data requirements)|
1.1 To what extent does the IEMR continue to address a demonstrable need?
Evidence of continuous need for environmental monitoring and research for DND/CAF
Evidence of continuous need for environmental monitoring and research for Aboriginal communities
2.1 Does the federal government (and DND specifically) continue to have roles and responsibilities in the delivery of the IEMR activities?
Degree of alignment with federal jurisdiction and with environmental policies
Evidence of complementarities or duplicative initiatives within DND/CAF and with OGDs initiatives
3.1 Do the IEMR activities align with the current federal government’s priorities and DND strategic outcomes?
Degree of alignment with the federal government’s priorities
Degree of alignment with DND strategic outcomes
Table D-2. Evaluation Matrix—Performance (Effectiveness)
The left-hand column lists the six evaluation questions related to performance effectiveness of the program’s outcomes. Read across the rows to determine the indicators used to assess each evaluation question, and the lines of evidence used for the indicators and questions. The columns for the lines of evidence indicate whether that particular line of evidence was used for each indicator.
|Evaluation Issues/ Question||Indicators||Details (info sources/data requirements)|
4.1 To what extent does IEMR provide DND/CAF with research and monitoring, as well as mitigation program recommendations?
Comprehensiveness of research and monitoring studies provided
Degree of satisfaction of stakeholders with the quality of the studies, as well as with the mitigation program recommendations
4.2 To what extent does IEMR increase knowledge of the people, flora, fauna and the environmental risks to northeastern Québec and Labrador?
Perceived degree of increased knowledge of the stakeholders on these issues
4.3 To what extent has the trust between IEMR and Aboriginal peoples and other stakeholder groups increased?
Perceived degree of increased trust between the IEMR and Aboriginal peoples and other stakeholder groups
4.4 To what extent does IEMR establish and strengthen relationships and partnerships with Aboriginal groups?
Perceived degree of strengthened relationships and partnerships established
4.5 To what extent does IEMR contribute to well-informed decision-making regarding the environmental impact of low-level Allied flight training activities?
Perceived extent to which IEMR led to more informed decision-making
4.6 To what extent does IEMR contribute to minimizing the environmental impact of low-level Allied flight training activities?
Evidence of changes implemented by DND/CAF regarding the environment
Perceived extent to which IEMR contributes to these changes
Table D-3. Evaluation Matrix—Performance (Efficiency and Economy)
The single evaluation question related to performance efficiency and economy is listed in the left-hand column. Read across the rows to determine the indicators used to assess this evaluation question, and the lines of evidence used for the indicators and questions. The columns for the lines of evidence indicate whether that particular line of evidence was used for each indicator.
|Evaluation Issues/ Question||Indicators||Details(info sources/data requirements)|
|PERFORMANCE (Resource Utilization: in relation to outputs and in relation to outcomes) (Efficiency and Economy)|
5.1 Is the funding of the IEMR the most efficient or economical means of achieving the expected outputs and outcomes for DND/CAF?
Processes and mechanisms in place for monitoring, research activities and report recommendations
Expenditures related to the IEMR in comparison to other comparable research centres
Comparison to other RCAF Wings
Footnote 1 This pertains to foreign low-level flying and does not include any Canadian Armed Forces (CAF) or North American Aerospace Defense Command activity not related to foreign military training. http://www.iemr.org/about_con01.html.
Footnote 2 TBS, Policy on Evaluation, April 1, 2009. Retrieved December 2, 2013, from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024§ion=text.
Footnote 3 TBS, Policy on Transfer Payments, October 1, 2008. Retrieved December 2, 2013, from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=13525§ion=text.
Footnote 4 Canadian Environmental Assessment Agency, Military Flying Activities in Labrador and Québec. Report of the Environmental Assessment Panel, Minister of Supply and Services Canada, February 1995, page 71–73.
Footnote 5 This arrangement applied to low-level flight training by Allied air forces, not the CAF.
Footnote 6 AMEC Environment & Infrastructure in association with Gardner Pinfold Consulting Economists Limited, Economic Impact and Trend Analysis of the Canadian Forces Base – 5 Wing Goose Bay (1999-2012). Report Submitted to the IEMR, St. John, NL, October 25, 2012, page 10. Retrieved December 2, 2013, from http://www.iemr.org/pdfs/R_Economic/IEMR_Final_Report_Oct_25_2012.pdf.
Footnote 7 The following are examples of key documents utilizing inconsistent terminology: The Named Grant to the IEMR, September 6, 2005, refers to “low-level flying and other training” and “military training activities,” whereas “low-level flight training” is used in the Memorandum of Understanding between DND and the IEMR Concerning Cooperation in the Environmental Management Program Related to Low-Level Flight Training at 5 Wing Goose Bay, May 2011.
Footnote 8 CRS, Evaluation of the Grant Program for the IEMR, DND, October 2009, page 6. Retrieved December 2, 2013, from http://www.crs-csex.forces.gc.ca/reports-rapports/pdf/2009/135P0893-eng.pdf.
Footnote 9 IEMR, Mission Statement. Retrieved December 2, 2013 from http://www.iemr.org/about_mission.html.
Footnote 10 Voting Aboriginal Members of the IEMR Board: Innu Nation; Nunatsiavut Government; Makivik Corporation/Naskapi Nation of Kawawachikamach (rotational members); NunatuKavut Community Council and Mamit Innuat. Voting non-Aboriginal Members of the IEMR Board: Central Labrador Economic Development Board, Inc.; Town of Happy Valley-Goose Bay; Municipalité régionale de comté de Minganie (Québec); and Churchill Falls Town Office. Non-voting members: Directorate Air Support; 5 Wing Goose Bay; Environment Canada; Government of Newfoundland and Labrador; and Government of Quebec. IEMR, 16th Annual Report 2012, Happy Valley-Goose Bay and Moncton, IEMR 2012, page 6.
Footnote 11 DND, Grant Agreement between the Minister of National Defence and the IEMR, November 24, 2009, page 2.
Footnote 12 IEMR, Mandate, DND, Grant Agreement, op. cit. Retrieved December 2, 2013, from http://www.iemr.org/about_mandate.html.
Footnote 13 IEMR, Objectives, DND, Grant Agreement, op. cit. Retrieved on December 2, 2013, from http://www.iemr.org/about_objectives.html.
Footnote 14 DND, Report on Plans and Priorities 2013-2014. Retrieved December 2, 2013, from http://www.forces.gc.ca/en/about-reports-pubs-report-plan-priorities/2013-other-section-ii-analysis-programs-strategic-outcome.page
Footnote 15 TBS, Directive on the Evaluation Function, April 1, 2009. Retrieved December 5, 2013, from http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15681§ion=text.
Footnote 16 National Defence Headquarters, CDS, CDS Tasking Order for Improvement to the Operational Mandate of 5 Wing Goose Bay, National Defence internal document, Ottawa, March 4, 2013, 6 pages. Retrieved December 2, 2013, from http://sjs.mil.ca/dgoperations/_docs/CDS_Task_Os/CDS%20Tasking%20Order%20for%20Improvements%20to%20the%20Operational%20Mandate%20of%205%20Wing%20Goose%20Bay.pdf.
Footnote 17 IEMR, Wildlife Avoidance Criteria, Recommendations for Modifications to the Wildlife Avoidance Criteria of the Department of National Defence Mitigation Program for Low-Level Military Jet Training Activities at CFB-Goose Bay, October 2013, page 22.
Footnote 18 Data provided by 5 Wing Goose Bay via email, September 13, 2013.
Footnote 19 AMEC Environment & Infrastructure, in association with Gardner Pinfold Consulting Economists Limited, Economic Impact and Trend Analysis of the Canadian Forces Base – 5 Wing Goose Bay (1999-2012), op. cit, Table 5.2, page 48. Retrieved December 2, 2013, from http://www.iemr.org/pdfs/R_Economic/IEMR_Final_Report_Oct_25_2012.pdf.
Footnote 20 In order to assess the scope and comprehensiveness of the IEMR environmental research to date, the Scientific Review Committee undertook a review of the IEMR monitoring and research data collected over the past 17 years, IEMR, Wildlife Avoidance Criteria, op. cit., 64 pages.
Footnote 21 An increase of more than 5,000 low-level flight training activities per year would constitute a type of significant change, as indicated in the IEMR, Wildlife Avoidance Criteria, op. cit.
Footnote 22 IEMR Constitution. Retrieved December 2, 2013 from http://www.iemr.org/about_con.html.
Footnote 23 IEMR, Wildlife Avoidance Criteria, op. cit.
Footnote 24 Letter to the IEMR from Francis Bouchard, Director, Government of Quebec, in his professional capacity, July 11, 2013.
Footnote 25 Letter to the IEMR from John Bake, Director of Wildlife, Government of Newfoundland and Labrador, September 18, 2013.
Footnote 26 The EIS was undertaken in 1987 – 1988 by a group of companies headed by Fenco Newfoundland Ltd. The work involved extensive field work and community consultation in an effort to ensure that the effects of low-level flight training activities could be identified, and that changes to the training area could be made if required. DND, EIS: Military Flight Training. An Environmental Impact Statement on Military Flying Activities in Labrador and Québec, Ottawa, DND, 1989.
Footnote 27 Letter to the IEMR from Glen Sheppard, Minister, Government of Nunatsiatvut, September 12, 2013.
Footnote 28 In addition to practicing good governance, the federal government has statutory, contractual and common law obligations to consult with affected Aboriginal groups. The IEMR does not formally consult with Aboriginal groups on behalf of the federal government or DND/CAF. See: Aboriginal Affairs and Northern Development Canada, Aboriginal Consultation and Accommodation, Updated Guidelines for Federal Officials to Fulfill the Duty to Consult. Retrieved December 2, 2013, from http://www.aadnc-aandc.gc.ca/DAM/DAM-INTER-HQ/STAGING/texte-text/intgui_1100100014665_eng.pdf.
Footnote 29 Department of Justice Canada, R.S.C., 1985, c. E-10. Current to November 12, 2013. Retrieved December 2, 2013, from http://laws-lois.justice.gc.ca/PDF/E-10.pdf.
Footnote 30 One part of a federal government strategy for the protection of wildlife species at risk is the Species at Risk Act (SARA). SARA provides federal legislation to prevent wildlife species from becoming extinct and to provide for their recovery. Two species in the LLTA identified by SARA have been studied and monitored by the IEMR: the Barrow’s Goldeneye and the Woodland Caribou.
Footnote 31 Government of Canada, Speech from the Throne, To open the 2nd session of the 41st Canadian Parliament, Wednesday, October 16, 2013. Retrieved December 5, 2013, from http://speech.gc.ca/eng/full-speech.
Footnote 32 Environment Canada, Planning for a Sustainable Future: A Federal Sustainable Development Strategy for Canada 2013 – 2016. Retrieved December 2, 2013, from http://www.ec.gc.ca/dd-sd/A22718BA-0107-4B32-BE17-A438616C4F7A/FSDS%202013-2016%20Final%20E.pdf.
Footnote 33 DND/CAF, Investing in Environment. Retrieved December 5, 2013, from http://www.forces.gc.ca/en/business-environment/index.page.
Footnote 34 In 2012, IEMR funded an unmanned aerial vehicle project that demonstrated the capabilities of a small unmanned aircraft for detecting wildlife.
Footnote 35 The BoomCast model is a predictive tool to assess noise projection from military training activities, including supersonic flights. IEMR, 16th Annual Report, 2012, op. cit, page 16.
Footnote 36 AMEC Environment & Infrastructure in association with Gardner Pinfold Consulting Economists Limited, Economic Impact and Trend Analysis of the Canadian Forces Base – 5 Wing Goose Bay (1999-2012), op. cit. This report focuses on the economic impact of military flight training in Labrador and northeastern Quebec.
Footnote 37 Compilation by the Evaluation, November 29, 2012.
Footnote 38 IEMR web site, http://www.iemr.org/research.html.
Footnote 39 In 2012, students from Labrador and Quebec’s North Shore were invited to participate. IEMR Annual Report, 2012. Visit http://www.iemr.org/new.html to view the photos.
Footnote 40 IEMR, Wildlife Avoidance Criteria, op. cit.
Footnote 41 Interviews with IEMR Board members, in Halifax, Nova Scotia, October 16, 2013. (Henceforth “Halifax interviews.”)
Footnote 42 AMEC, www.amec.com/aboutus/at_a_glance.htm.
Footnote 43 AMEC Environment & Infrastructure, in association with Gardner Pinfold Consulting Economists Limited, Economic Impact and Trend Analysis of the Canadian Forces Base – 5 Wing Goose Bay (1999 – 2012), op. cit., covered the years 2011-2012, while the trend analysis covered years 1999 – 2000, 2001 – 2002, 2002 – 2003, 2006 – 2007 and 2011– 2012. Retrieved December 2, 2013, from http://www.iemr.org/research_Economic.html.
Footnote 44 Halifax interviews.
Footnote 45 Halifax interviews.
Footnote 46 Interview with former D Air Sp staff on September 30, 2013.
Footnote 47 Halifax interviews.
Footnote 48 Halifax interviews.
Footnote 49 Halifax interviews.
Footnote 50 Interview with Program Manager staff October 3, 2013.
Footnote 51 Halifax interviews.
Footnote 52 Halifax interviews.
Footnote 53 The Department of Labrador and Aboriginal Affairs and le ministère des Ressources naturelles et de la Faune du Québec provide 5 Wing Goose Bay with the data.
Footnote 54 IEMR has contributed funds to the provinces for the placement of the tracking collars on the caribou. IEMR, 16th Annual Report, 2012, op. cit.
Footnote 55 For example, in the case of eagle nests, flights are requested to stay 2.5 nautical miles away from the nests. For caribou, flights are requested to stay 8 nautical miles away.
Footnote 56 Information received via email from 5 Wing Goose Bay Mitigation Officer, September 13, 2013.
Footnote 57 IEMR, 16th Annual Report, 2012, op. cit.
Footnote 58 As noted in the Economic Trend, op, cit., the regions surrounding 5 Wing Goose Bay have experienced economic diversification. The Happy Valley-Goose Bay area serves as the regional hub for natural resource development, including the Lower Churchill Hydro project. The mining sector for example, has expanded with the Voisey's Bay mine, the Grand River Ironsands, and the potential for uranium development and mining. The completion of the Trans-Labrador Highway from Cartwright Junction to Happy Valley-Goose Bay has also positively affected the region’s economy.
Footnote 59 Halifax interviews.
Footnote 60 The TBS Policy on Transfer Payments, 2008, op, cit., defines “value for money” as the extent to which a program demonstrates relevance and performance. Relevance is achieved by addressing a demonstrable need that is appropriate for the federal government and is responsive to Canadians. Performance is achieved by using taxpayer resources well, producing program outputs in an affordable manner and achieving outcomes consistent with program objectives.
Footnote 61 As per the federal government’s Indirect Costs Program Web site: “The term ‘indirect costs’ refers to the central and departmental administrative costs that institutions incur to support research but [which] are not attributable to specific research projects.” There are five eligible expenditure categories: research facilities; research resources; management and administration of an institution’s research enterprise; regulatory requirements and accreditation; and intellectual property. Retrieved November 18, 2013, from http://www.indirectcosts.gc.ca/using-utilisation/expenditures-depenses-eng.aspx.
Footnote 62 Association of Universities and Colleges of Canada, Funding the Institutional Costs of Research: An International Perspective, Ottawa, May 2009, page 2. Retrieved November 18, 2013 from http://www.aucc.ca/wp-content/uploads/2011/07/institutional-costs-international-2009-05-e.pdf.
Footnote 63 Interview with Chair of IEMR, October 7, 2013.
Footnote 64 Interviews were conducted by phone, with transcripts verified by interviewees for approval. Interviews took place October 18–21, 2013.
Footnote 65 In 2011, the Innu Nation’s land claims in Labrador received an Agreement-in-Principle, although the final agreement remains unsettled. Jenny Higgins, Innu Organizations and Land Claims, Newfoundland and Labrador Heritage Web Site. Retrieved November 18, 2013, from http://www.heritage.nf.ca/aboriginal/innu_claims.html
Footnote 66 With the exception of one interview questionnaire that was completed and returned by email.
Footnote 67 One Board member did not attend the meeting and sent his answers by email.
Report a problem or mistake on this page
- Date modified: