Evaluation of the Participant Funding Program
January 4, 2016
Submitted to CEAA Project Authorities:
Steve Chapman
Director, National Programs
and
Lisa St-Amour
Director, Finance and Administration
Submitted by:
Alan Winberg Consulting Inc.
3090 Barlow Crescent
Dunrobin, Ontario, K0A 1T0
Telephone: 613-859-9827
Email: alanrwinberg@gmail.com
Contents
- List of Acronyms
- Evaluation of the CEAA Participant Funding Program
- Introduction
- Limitations and Mitigation Strategies
- Program Profile
- Summary of Findings
- Relevance
- Performance
- Cost-effectiveness
- PFP Client Service Standards
- PFP Program Awareness
- Timing of expenditures under signed contribution agreements are difficult to forecast
- A proponent may change a proposed project significantly during the process
- PFP is focussed on providing modest support during the EA process
- Core Control Audit Follow-up
- Conclusions
- Recommendations
- Appendix A: Evaluation Matrix
- Appendix B: Interview Guide
- Annex C: Individuals and organizations invited to participate in an interview
- Appendix D: Documents Reviewed
- Appendix E: File Review Sample
List of Acronyms
- AANDC
- Aboriginal Affairs and Northern Development Canada
- CEAA
- Canadian Environmental Assessment Agency
- CEAA 2012
- Canadian Environmental Assessment Act, 2012
- CNSC
- Canadian Nuclear Safety Commission
- DPR
- Departmental Performance Report
- EA
- Environmental Assessment
- EAs
- Environmental Assessments
- FTE
- Full-time equivalent
- G&Cs
- Grants and Contributions
- GoC
- Government of Canada
- NEB
- National Energy Board
- O&M
- Operations and Maintenance [funding]
- OCG
- Office of the Comptroller General
- PA
- Project Authority (or Authorities)
- PAA
- Program Alignment Architecture
- PFP
- Participant Funding Program
- RPP
- Report on Plans and Priorities
- SCC
- Supreme Court of Canada
- TB
- Treasury Board of Canada
- TBS
- Treasury Board of Canada Secretariat
Evaluation of the CEAA Participant Funding Program
Executive Summary
The Canadian Environmental Assessment Agency (CEAA) engaged Alan Winberg Consulting Inc. to conduct an independent review of the CEAA Participant Funding Program (PFP) pursuant to the requirements of the Treasury Board Policy on Transfer Payments and section 42.1 of the Financial Administration Act.
The evaluation was designed to assess: the program’s continued relevance; the performance of the program with regard to the achievement of results and the program’s cost-effectiveness. In addition, the evaluation was required to conduct a follow-up examination from the Core Control Audit, to consider Recommendation 12B (with regard to program files) and Recommendation 13B (with regard to the stacking limit set out in the program terms and conditions).
The evaluation used a file review, documents and website review, and key informant interviews to collect data for each of the evaluation issues.
The program provides modest support for participation by the public, such as eligible individuals, incorporated not-for-profit organizations, and Aboriginal groups at key points during the Environmental Assessment (EA) process. The program contributes to high quality EAs, and contributes to meeting the Crown’s duty to consult Aboriginal groups.
The evaluation found that the PFP is well-aligned with Federal and Agency priorities. The PFP is relevant and meeting demonstrated needs of the public and Aboriginal groups. The program is delivering its intended results. Interviewees stated that PFP support has increased participation in the EA process and thereby, has increased the information and the range of perspectives provided to the Agency and Panels for EAs. The evaluation found that the PFP has operated within its budget. It is cost-effective and delivering value-for-money.
The follow-up to the Core Control Audit included reviewing a sample of active files, and conducting interviews with program management and staff. With regard to program files, the follow-up found that the files are complete and that the expected documents are readily available in well-organized paper files, in electronic files, and on the CEAA Public Registry. With regard to due diligence with respect to the stacking limits, the follow up found that some important additional steps had been taken since the Core Control Audit with regard to additional reporting required from recipients. The evaluation report describes the additional reporting, and concludes that with these new measures in place, there is low risk of exceeding the stacking limits; however, due diligence regarding stacking limits can be improved.
The evaluation concludes that the program is relevant and performing as expected. It is cost-effective and should be continued.
The evaluation makes recommendations regarding continuing the program; tracking and reporting potential financial pressures for delayed projects; considering additional support when a proponent makes significant modifications to a proposed project during the EA process; publishing the details regarding ‘stacking limits’ in program documentation; and, completing the PFP Performance Measurement Strategy.
Introduction
Evaluation Purpose, and Methods: Context
This report presents the results of an evaluation of the CEAA Participant Funding Program (PFP). Recommendations have been made regarding improvements to the program and to inform future decision-making.
Purpose and Scope
The Canadian Environmental Assessment Agency (CEAA) engaged Alan Winberg Consulting Inc. to conduct a program evaluation of the PFP pursuant to the requirements of the Treasury Board’s Policy on Transfer Payments and section 42.1 of the Financial Administration Act which require a review every five years of the relevance and effectiveness of each ongoing program.
Using standard evaluation methods, the evaluation was designed to assess: the program’s continued relevance (including the continuing need for the program and alignment with government priorities and federal roles and responsibilities); the performance of the program regarding the achievement of results and the program’s cost-effectiveness. In addition, the evaluation was required to conduct a follow-up examination from the Core Control Audit [1], to consider Recommendation 12B (with regard to program files) and Recommendation 13B (with regard to the stacking limit set out in the program terms and conditions).
The evaluation has covered the activities, outputs, outcomes and expenditures over the period from fiscal year 2010-11 to 2014-15, with emphasis on the period since the Canadian Environmental Assessment Act was amended in 2012; of the PFP.
Evaluation Approach
This evaluation was conducted using a phased approach over the period from April to October, 2015. The three phases of this evaluation study comprised: Phase 1: Planning (including development of an evaluation matrix); Phase 2: Conduct of the Evaluation (data collection and analysis); and Phase 3: Reporting. We maintained close consultation with CEAA’s Project Authority for each phase of the evaluation. The key analytical approach used for this summative evaluation was outcome-based analysis, which considered the extent to which the program had achieved its intended results.
Evaluation Methodology and Constraints
The evaluation relied on multiple lines of evidence to gather qualitative and quantitative data for this evaluation, including a documents and literature review, a review of a sample of program files, and interviews.
The documents and literature review covered program files and materials regarding the EA process at CEAA and regarding the PFP at CEAA, National Energy Board (NEB) and Canadian Nuclear Safety Commission (CNSC), as well as relevant information from other federal government sources.
We conducted brief website and literature scans on the EA process and CEAA 2012. There is some readily available documentation on the EA process in other countries. However, using standard Internet search techniques, we did not identify any similar PFP programs for participation in an EA process in other countries, such as the United Kingdom, the United States, or Australia.
The evaluation used financial and administrative data on the PFP that was provided by the Project Authority.
Interviews, structured with a short interview guide, were completed with 42 key informants. Key informants included: 26 CEAA officials, 4 officials from other federal departments and agencies, and 12 recipients of contribution agreements. The majority of interviews in the NCR were conducted in person. Telephone interviews were conducted with regional CEAA officials and officials of other federal departments and agencies. A few individuals provided written comments to the interview questions instead of a person-to-person interview. To encourage frank discussion of the issues, an undertaking was made to interviewees that comments would be reported only in the aggregate and would not be attributed to specific individuals or specific organizations.
Interviewees | Number of Completed Interviews |
---|---|
CEAA officials | 26 |
Officials from other federal departments and agencies | 4 |
Recipients of Contribution Agreements | 12 |
Total | 42 |
Considering the aggregate of the data provided by the documents, file review, and interviews, these methods contributed to a complete set of qualitative and quantitative information regarding the relevance, performance and cost-effectiveness of CEAA’s PFP. To facilitate understanding of the methodologies used in this evaluation and to facilitate performance measurement work by the program and future evaluations, in the appendices, we have provided additional details regarding the data collections and tools. Appendices include an example of the interview guides, additional details regarding the sample of files reviewed, the documents reviewed, and the titles of the interviewees invited to participate. These lines of evidence, including the set of interviews, the file review, and the documents and literature review, are sufficient and appropriate in order to draw conclusions as presented in this report.
Limitations and Mitigation Strategies
Input from Recipients of Contribution Agreements
Recipients of contribution agreements are key stakeholders regarding the program. A relatively small number of recipients provided input to the evaluation through telephone interviews. As a supplement to the interview data, the results of a questionnaire that is sent to recipients by PFP following final payment was reviewed as part of the documents review. Since the group of recipients who completed the questionnaire or who participated in an interview is not necessarily representative of all recipients, the questionnaire and interview results cannot be generalized to the entire population of recipient stakeholders, but they nevertheless provided some key insights and informed the evaluation in combination with the other lines of evidence, including the file review, and the documents and literature review.
Performance Data
The PFP program published service standards in mid-June 2015. Performance against these standards is being measured and is expected to be available for the next evaluation study. This study relied on qualitative information regarding performance, based on the files and documents reviewed and the interviews conducted. Given the consistent high performance noted in all lines of evidence, the information reported is considered valid and reliable.
Program Profile
Program Description
Responsible federal authorities involved in an environmental assessment process are required to establish participant funding programs under the Canadian Environmental Assessment Act, 2012 (CEAA 2012).[2] Three responsible federal authorities, the CEAA, the Canadian Nuclear Safety Commission (CNSC) and the National Energy Board (NEB), have established participant funding programs.
CEAA’s PFP provides financial assistance to individuals, incorporated not-for-profit organisations, and Aboriginal groups to participate in environmental assessments conducted by the CEAA, by review panels and joint review panels pursuant to the CEAA 2012.
Participating in federal environmental assessments helps to ensure that concerns from the public and Aboriginal groups, regarding the potential effects of a project on the environment, on the public, on Aboriginal groups and on existing or potential Aboriginal or Treaty rights, are taken into consideration during the environmental assessment process. Participant funding can cover eligible costs such as expert advice, travel expenses and reporting costs (pursuant to section 8 of the program terms and conditions).
Information on potential and current environmental assessments of projects subject to the federal environmental assessment process and related participation opportunities are published on the CEAA Registry [3] and on the Consulting with Canadians portal [4].
To be eligible to apply, an applicant must demonstrate the value the applicant will add by participating in the environmental assessment. In addition, applicants must meet at least one of the following criteria:
- Have a direct, local interest in the project, such as living or owning property in the project area;
- Have community knowledge or Aboriginal traditional knowledge relevant to the environmental assessment;
- Plan to provide expert information relevant to the anticipated environmental effects of the project; and/or
- Have an interest in the potential impacts of the project on treaty lands, settlement lands or traditional territories and/or related claims and rights.
Aboriginal Consultations
For Aboriginal consultations in particular, a common law “duty to consult” is based on judicial interpretation of the obligations of the Crown (federal, provincial and territorial governments) in relation to potential or established Aboriginal or Treaty rights of the Aboriginal peoples of Canada, recognized and affirmed in section 35 of the Constitution Act, 1982.
In several decisions of the Supreme Court of Canada (SCC), it has been held that the Crown has a duty to consult and, where appropriate, accommodate when the Crown contemplates conduct that might adversely impact potential or established Aboriginal or Treaty rights. This duty has been applied to a number of Crown actions and in relation to a variety of potential or established Aboriginal or Treaty rights.
For these reasons, the CEAA is involved in Aboriginal consultations and plays the role of the Crown Consultation Coordinator to integrate the Whole-of-Government approach to Aboriginal consultation activities into the EA process where possible. This applies to all EAs for which the Agency is the responsible authority, including review panels. PFP provides funding to assist eligible Aboriginal groups in preparing for and participating in consultation activities.
PFP was recently included in a Core Control Audit
The program was reviewed in a Core Control Audit, completed by the Office of the Comptroller General (OCG) in June 2014, for transactions during fiscal year 2013-2014. This evaluation reviewed aspects of two recommendations from that Audit:
- Recommendation #12. Supporting documents are on file for all contribution payments made, in accordance with the documentation requirements of the funding agreement; and,
- Recommendation #13. Proper due diligence is exercised when confirming that transfer payment amounts do not exceed the stacking limit.
Program Budget
There are two funding components:
- Regular Funding is available to assist in preparation for, and participation in EAs conducted by the Agency or by review panels (and is available to public groups, or to Aboriginal Groups rated as “low” in terms of potential impacts).
- Aboriginal Funding is available to assist in Aboriginal consultations (with regard to potential impacts on Aboriginal rights and titles) that occur during the EAs conducted by the Agency or by review panels (This funding is intended for Aboriginal Groups rated as “high” or “medium” in terms of potential impacts).
PFP Funds “aggregates” in addition to individual aboriginal communities. Providing PFP funding for “aggregates” (made up of two or more individual Aboriginal communities) is fully consistent with federal priorities and practices. CEAA PFP policies and practices have been developed to ensure PFP’s rigorous funding approach is effectively adapted to facilitate supporting these “aggregates”.
Program annual planned spending is $4.469 million [5], of which up to 20% may be used for O&M ($0.8938 million). Actual expenditure on contributions was less than planned expenditure in each of the last five fiscal years. O&M and salary expenditures were under the maximum allowed in each of the last five fiscal years.
Fiscal year | Contributions | O&M | Salaries |
---|---|---|---|
2010 -2011 | $1,665,083 | $150,198 | $255,050 |
2011 -2012 | $2,842,203 | $152,204 | $260,068 |
2012 -2013 | $1,993,668 | $46,744 | $375,878 |
2013 -2014 | $2,518,323 | $81,327 | $455,114 |
2014 -2015 | $2,069,320 | $41,809 | $360,509 |
Previous evaluations of the PFP
An evaluation of the PFP was completed in 2009.
The 2009 evaluation aimed to inform the renewal of the Terms and Conditions of the Program, which at that time were valid until December 31, 2009. The objective of the study was to evaluate the PFP in terms of relevance, program impacts (success), and cost-effectiveness/alternatives, covering the five fiscal years from 2003-2004 to 2007-2008.
With regard to “Relevance”, the evaluation found that there was a continuing need for the PFP, primarily as a result of the Canadian Environmental Assessment Act which required the establishment of a participant funding program. As well, the evaluation found that PFP funding facilitated participation in the environmental assessment process and without this funding this participation may not have taken place.
With regard to “Success”, the evaluation found that the PFP had enabled participation in the EA process; especially for those organizations that were believed to have the highest need for funding in order to participate (i.e., Aboriginal organizations and smaller, local incorporated not-for-profit organizations).
With regard to “Cost-effectiveness and Alternatives”, the evaluation found that PFP spending was in line with targeted spending ratios. As well, the evaluation stated that according to stakeholders, PFP is cost-effective although a few stakeholders indicated that cost-effectiveness could be increased by reducing the administrative burden associated with the Program.
The 2009 evaluation made recommendations related to: improved performance measurement; enhanced outreach by CEAA to improve awareness levels of potential clients; enhanced outreach by program proponents to improve awareness of the PFP; improved feedback from selected panel members following the completion of the EA process; and reduction in the administrative burden of the Program.
Program Logic Model
The model below was prepared by the evaluator, solely for the purpose of planning and conducting this evaluation of the PFP. The draft logic model is intended to illustrate the objectives of the PFP and its associated key intended outcomes. The sources of information for the preparation of this Logic Model are: the PFP National Program Guidelines, presented at http://www.ceaa-acee.gc.ca/default.asp?lang=En&n=9772442E-1&offset=3&toc= show; the CEAA Report on Plans and Priorities 2015-16; and an unpublished, draft logic model, under development by the program as part of a Performance Measurement Strategy. The draft logic model presented below was shared with CEAA during the evaluation study. This draft logic model may not reflect all aspects of the CEAA 2012, or PFP Guidelines.
Whole-of- government | Economic Affairs Outcome Area: Strong Economic Growth | |||
---|---|---|---|---|
CEAA Strategic Outcome [and associated organizational priorities] | High-quality and timely environmental assessments of major projects to protect the environment and support economic growth [CEAA Priority: Deliver high-quality environmental assessments of major projects CEAA Priority: Build effective relationships with Aboriginal peoples] |
|||
Ultimate Outcomes | High quality and timely EAs are delivered Effective public and Aboriginal participation occurs Concerns with respect to the potential effects of a project on the environment, Aboriginal groups and on existing or potential Aboriginal or Treaty rights are taken into consideration during the EA process |
|||
Intermediate Outcomes | Comments from the Public and Aboriginal groups are taken into account in a timely manner | The EA process is conducted in a balanced manner | The quality and credibility of a project’s environmental review is strengthened | |
Immediate Outcomes | Local and traditional knowledge about a project's physical location helps to identify and address potential environmental effects at the early stage of an EA | Individuals and organizations have an opportunity to contribute to the planning of projects that may affect them | Proponents and the Agency better understand and address public interests and concerns regarding the potential environmental effects of a project | Potential adverse environmental effects are identified and are prevented or mitigated in the EA by the application of community knowledge and Aboriginal traditional knowledge |
Outputs | Information about the project is provided from the perspective of recipients | Considerations regarding the potential environmental effects of a project are expressed | Considerations regarding potential Aboriginal or Treaty rights are expressed | Community knowledge and Aboriginal traditional knowledge is made available for consideration |
Activities | Recipients submit comments at key points in an EA process Recipients participate in information sessions and public hearings associated with a review panel |
|||
Input | PFP provides human resources, funding, transfer payment expertise, and contribution agreements, according to the legal and policy framework, to provide funding for participation in EAs |
Please note: For evaluation purposes, there is a diminishing level of potential attribution of outcomes to PFP as one moves from the bottom of the logic model to the top.
Summary of Findings
Relevance
The PFP is well-aligned with the roles, responsibilities and priorities of the Federal government.
It is a requirement of the CEAA 2012, at sections. 57 & 58, that CEAA “must” establish a PFP.
The PFP continuing, permanent (A-Base) budget is $1.469 million. The time-limited portion of PFP funding ($3 million annually for the three-year period from April 1, 2012 to March 31, 2015) was renewed in Budget 2015 until 2020. Renewal of sunsetting funding in Budget 2015 is a clear indication that PFP is an indicator that the program is aligned with the priorities of the Federal government.
PFP is well-aligned with CEAA’s mandate, Strategic Outcome & Priorities. The program is consistently reported in CEAA’s Estimates documents, that is, CEAA’s Report on Plans and Priorities (RPP) and its Departmental Performance Report (DPR).
Based on the files and documents reviewed, and all interviews conducted for this evaluation, PFP addresses a demonstrated need of the public and Aboriginal groups. Documents and interviews confirmed that the participants which had received support from PFP could not participate, or could not participate to the same extent, without support from PFP.
As evidenced by interviews and the observed degree of responsiveness to opportunities to participate in the EA process, the evaluation concludes that PFP continues to address demonstrated needs of individuals, incorporated not-for-profit organizations and Aboriginal groups.
Performance
The outputs and outcomes noted in the draft PFP logic model were assessed using multiple lines of evidence to determine the extent to which PFP was delivering the expected results. The qualitative and quantitative data collected during this evaluation confirms that PFP has delivered the intended expected outputs and outcomes.
The evidence from interviews confirms that the PFP has increased participation by the public and Aboriginal groups in the EA process. Participants use PFP to augment existing capacity to participate, such as:
- preparation of written comments at key milestones in the process; and
- participation at hearings of a review panel.
The evidence from interviews confirms that the quality of the information considered within the EA process is improved by PFP support. Support helps participants to contract for technical expertise and legal assistance, to hold community meetings, or to travel to hearings.
As a result of the increased participation of PFP recipients:
- considerations regarding the potential environmental effects of a project are expressed from the perspective of PFP recipients;
- considerations regarding potential Aboriginal or Treaty rights are expressed from the perspective of PFP recipients; and
- community knowledge and Aboriginal traditional knowledge is made available for consideration from the perspective of PFP recipients
Participants are using the program in the manner expected.
Cost-effectiveness
Over the full period examined, the program has operated within the allocated budget limits.
There is a rigorous process in place, including a funding review committee, which makes recommendations to the President of CEAA, to allocate the program’s limited budget pursuant to the program terms and conditions. The process is well-documented. Section 4 of the terms and conditions set out a detailed list of criteria [7] for funding decisions. Funding decisions are based on a systematic analysis against these explicit criteria.
Funds allocated to public and Aboriginal groups are modest compared with the expected expenditures to be incurred. Interviewees noted that while the support is most appreciated and valued, especially by the smaller incorporated not-for-profit organizations and Aboriginal groups with limited capacity, due to the amount of funding available for each EA project in the CEAA budget, the amount of support provided is significantly below the potential cost of participation in the EA process.
Since the evaluation evidence confirms that the program is operating within budget, rigorously following explicit criteria to allocate limited funding, and providing modest support for participation in the EA process, the evaluation concludes that the program operates in a cost-effective manner.
The previous evaluation noted that improvements and streamlining in the administrative process would be beneficial. With regard to the PFP process over the period under examination for the current evaluation, interviewees noted that there have been significant improvements in program delivery since 2012. In particular, interviewees noted that the application process has been streamlined and improved. Interviewees from within CEAA and from organizations that had received PFP funding commented that recipients especially appreciated the flexibility that is allowed in the use of the approved contribution agreement budget. The program has made effective use of information technology to streamline and automate the application and claims processes. Interviews confirmed that the renewed application process is straightforward; the contribution agreement is easy-to-understand; and, the expense claim process works well. Based on this qualitative evidence, the evaluation concludes that program delivery has improved significantly over the period under examination.
With regard to the application of risk-management principles, the program is using a risk assessment process for decision-making regarding the management of program funds. First Nations with “Default Management under way” (list published at AANDC) are recognized as higher risk. Monitoring plans reflect the track record of recipients and whether the recipient has received support for participation in an EA in past. Recognizing the risks associated with advance payments, interviewees from CEAA confirmed that advance payments are used only in exceptional circumstances. Interviews confirmed that there are very few cases where the program has had to establish an overpayment. Avoidance of overpayments is an additional indicator that the risk management approach used by the program is effective.
The program published service standards in June 2015, setting out expectations for timeliness in delivering the program. There was no quantitative data available for examination during the evaluation regarding performance against these standards. The program plans to collect data regarding performance against these standards, and to have this data available for the next evaluation. Although there is no quantitative data, the evaluation has considered qualitative data collected for this evaluation from interviewees and a summary of the results from a voluntary, anonymous questionnaire that is sent to recipients after the file is closed. Interviewees noted the timely and responsive service provided by the PFP unit. Results from the anonymous questionnaire were consistent with the interviews.
Interviewees from within CEAA noted that the government policy to achieve efficiencies by making all its payments by direct deposit by April 1, 2016, was an irritant for some Aboriginal groups, and in particular, Aboriginal groups which may already have provided the paperwork for this initiative to the Department of Aboriginal Affairs and Northern Development Canada (AANDC). Interviews suggested that current government policy with regard to this transition does not consider whether a group may already have provided the necessary paperwork and authority to another department. This is a short-term transition issue about a government-wide financial policy, not specific to PFP.
Apart from the direct deposit irritant, there were no issues raised with regard to the program. The administrative streamlining, and increased use of the Internet-based fillable forms that has occurred was noted with high satisfaction. With regard to the program process and administrative tools, the evaluation finds that the tools and forms have been significantly improved and, based on qualitative data, that program delivery is timely and responsive.
PFP Client Service Standards
Treasury Board Transfer Payment Policy [8] requires that contribution programs such as PFP have in place reasonable and practical service standards. Based on TB Guidelines for service standards [9], a service standard is a public commitment to a measurable level of performance that clients can expect under normal circumstances. According to the TB Guidelines, service standards serve two key purposes: to provide staff with performance targets; and, to inform clients what to expect. PFP published service standards June 17, 2015 [10] for the key steps in the contribution agreement process. Thus, the program is in compliance with this requirement of the TB Transfer Payment policy.
This is the first time service standards have been published for this program. The service standards present timeliness standards, based on estimates of historical performance. The standards present a range rather than an outside limit with regard to timeliness. The standards include a calculation of a total of the time that is measured by the individual service standards, including both calendar days and working days. The use of a range and provision of a total of this nature adds a degree of complexity for measurement and use of the service standards. CEAA intends to monitor performance against these standards and program management may modify the service standards after an initial trial period. The goal is to have practical and measurable service standards in place for program management purposes, to facilitate continuous improvement and to conduct periodic evaluations.
Process Phases | Standard | Notes/Comments |
---|---|---|
*In working days, except for the Application period | ||
Application period | 30 days | In calendar days. May vary in some exceptional circumstances |
Review Applications | 1 to 3 days | |
Conducting Funding Review Committee and obtaining client revisions/additional documents as required | 1 to 5 days | May vary depending on client’s response time |
Allocations Decision process | 5 to 15 days | |
Signing of Agreement by the Agency once received from recipient | 1 to 3 days | |
Payments processing (once complete expense claim is received) | 10 to 20 days | |
TOTAL | 48 to 76 days* |
PFP Program Awareness
Awareness of PFP funding opportunities is achieved using the CEAA Register, notices in local newspapers, and announcements on local radio stations as well as local cable television. Aboriginal groups which may be impacted by a project are identified and sent an email or letter to make them aware of the funding available.
Based on the numbers of applications received for opportunities and the responsiveness of Aboriginal groups to the emails and letters, and based on the interviews with CEAA officials and recipients, the current approach works well. In addition, interviews noted that work is underway to continue to improve awareness (of the EA process) by publishing to the Internet all documentation provided to CEAA under the EA process.
The evaluation concludes that program awareness activities have proven to be effective.
Timing of expenditures under signed contribution agreements are difficult to forecast
Looking at the past five fiscal years, the actual expenditure has been less than was planned at the start of the fiscal year. It is mainly the actions taken and the decisions made by a project proponent that determine the timing of a proposed project and therefore the EA process. A project proponent may decide to delay starting the next step in the process, depending on economic conditions which change over time. For example, commodity prices fluctuate, and this may result in modified plans by a proponent to proceed. Neither the CEAA nor the recipient of a PFP contribution agreement has control of the precise timing of a specific project, and therefore, the speed at which the EA process will unfold.
Although these potential delays in the EA process are outside the control of CEAA or the recipient of a contribution agreement, the support for participation that is set out in the contribution agreement remains in place over time. For example, the contribution agreement represents a commitment by CEAA to support public comment on an environmental impact statement or a draft EA Report. This commitment by CEAA to the recipient remains in place even when a project is delayed (section 15 of the contribution agreement). However, CEAA’s transfer payment budget is not carried forward to future fiscal years. As of October 15, 2015, CEAA’s PFP had in place 92 contribution agreements (which together represent future potential expenditures of an estimated $1,581,978), all of which are related to 15 dormant projects [12] (projects delayed by the proponents with no specific end dates).
As long as economic conditions remain challenging, based on the experience of the past five years, it is likely, but not certain, that CEAA could meet most near-term claims from contribution agreement recipients with its existing PFP budget.
CEAA does discuss the variances that have occurred in past between planned and actual spending each year with regard to transfer payments in its Departmental Performance Report (DPR). For example, the 2013-14 DPR states that “the number of projects entering the environmental assessment process is a function of external factors, such as the state of the economy and commodity prices (e.g., gold, copper). In addition, the pace of progress through the process may be affected by a proponent's level of effort at a given time and the completeness of its submissions. These factors, which are beyond the control of the Agency, affect when opportunities for public participation and Aboriginal consultation will occur and when reimbursement of expenses in accordance with contribution agreements will be requested. As a result, it is difficult to accurately forecast when resources will be required for allocation and reimbursement. This often results in funds that are earmarked for a given year being required in the subsequent year, and therefore requires an approach to budget allocation that reflects the inherent variability in the timing of demands on the Participant Funding Program.”[13] Although the uncertainty is well described in the DPR, and although future contract and transfer payment obligations are noted in the CEAA financial statements [14], CEAA could consider reporting an additional level of detail regarding the specific amount of contribution agreement funding that has been delayed due to projects which a proponent has put on hold. Although contribution agreements contain a standard condition (the Crown’s legal obligation is subject to appropriations [section 3.03]), carrying forward the obligation to reimburse expenses pursuant to a contribution agreement, without carrying forward the required funding, creates a “reputational risk”.
The evaluation has concluded that the rate of expenditure has been less than expected due to the economic cycle and decisions made by proponents. The program has used its budget with due regard to economy, the program terms and conditions, and the expected pace of development of projects by proponents. When economic conditions improve, EA processes now on hold could become active again, even though the funding that was planned for these specific EAs has lapsed. While there would be adequate time to explain a potential budget shortfall to Central Agencies, the CEAA may wish to consider the current wording of the text used in the DPR (quoted in the preceding paragraph), or the text used at note 9 of the CEAA Financial Statements and whether additional detail would assist a reader of these documents to understand the magnitude of the risk by knowing the precise amount of signed commitments in place related specifically to delayed projects. The additional detail with regard to the future transfer payments of CEAA would allow a reader to distinguish between future commitments where the project is proceeding as planned and those future transfer payment commitments related to a project that has been put on hold.
A proponent may change a proposed project significantly during the process
A proponent may change a proposed project significantly to reflect changing economic conditions over time or to address comments received after the EA process is underway. The changes to a proposed project may be significant. An issue regarding the level of support when a proponent modifies a proposed project was raised during the interviews. PFP recipients may lack capacity to review the potential impacts of the changed project. However the modest support received from PFP may have been expended to assess the original project. Interviews revealed that this can present a significant issue and that recipients can be expected to seek additional modest support from PFP in order to assess the changes. Consideration of this issue would be linked to the program’s capacity to make a provision for such funding within the program’s current limited budget, as well as whether an amendment to program terms and conditions would be required.
PFP is focussed on providing modest support during the EA process
Interviewees noted the potential value of CEAA providing support before and after the EA process. PFP is designed to provide modest support during the EA process only. The evaluation evidence confirms the relevance and performance of PFP in delivering what it is designed to achieve. However there is a perceived gap in necessary support for participation by PFP recipients pre- and post- the EA process. In addition, interviews revealed that providing support to build a basic level of capacity for participation in the EA process in those Aboriginal communities where capacity is low has a high potential value. Addressing this perceived gap would require an expansion of the PFP terms and conditions or a new program, and is outside the scope of this evaluation.
Core Control Audit Follow-up
As a follow-up to the Core Control Audit, the evaluation assessed: to what extent necessary PFP supporting records are filed effectively; and, to what extent the PFP is exercising due diligence with respect to stacking limits.
PFP supporting records and files
Based on a review of a sample of electronic files and paper files containing records from fiscal years 2012-13 to 2015-16, PFP files contain the expected supporting records. The filing system is effective and records are accessible in a timely manner. Key documents are maintained in both an electronic and a paper file. Files reviewed were organized in a professional and consistent manner. Paper files used a system of tabs and contained a checklist, noting the key documents that staff are to be certain to include in the file before it is closed. In all the files reviewed, this checklist was present and was signed by the PFP Manager. In all paper files examined, the documents listed on the checklist were in fact included in the paper file.
Stacking limit
Pursuant to section 7 of the PFP Terms and Conditions, the maximum level of Total Government Assistance (federal, provincial, territorial, or municipal assistance for the same eligible expenditures) is not to exceed 100% of eligible expenses.[15] In the event this limit is exceeded, CEAA must adjust its level of contribution (and seek reimbursement, if necessary) so that the stacking limit is not exceeded.
The 100% stacking limit is noted at Section 7 or the program terms and conditions. The terms and conditions are accessible to program staff with access to the internal CEAA network, but are not published or accessible to the public.
Following the completion of the Core Control Audit, the program made modifications to application forms, contribution agreement forms, and expense claims forms to request information from recipients regarding other sources of funding. Applications (section 6bi), contribution agreements (section 2.05; section 3.03) and expenditure claims (section 5) require a recipient to indicate whether they have another source of funding. Having made these changes after the Core Control Audit demonstrates that CEAA is visibly exercising a higher degree of due diligence with regard to ensuring the stacking limit is respected. However, these additions to the forms and the contribution agreement do not reflect completely paragraph 7 of the program terms and conditions.
The 100% Total Government Support stacking limit presented in section 7 of the program terms and conditions is not published on the website; nor is it published in the National Program Guidelines; nor is it noted in the Contribution Agreement template.
The levels of funding provided by PFP are quite modest compared with the expected levels of expenditure required to participate in an EA process. Although the risk of exceeding the stacking limit is low, additional clarity and precision would reduce any risk that a Recipient would exceed the stacking limit.
To improve due diligence with respect to the stacking limit, and to facilitate understanding of the stacking limit by a recipient, the information requested from a recipient regarding other sources of funding could be focused explicitly on the stacking limit, that is, the maximum level of Total Government Assistance (federal, provincial, territorial, and municipal assistance for the same eligible expenditures). Modified text in the program documentation [16] could provide three additional points:
- The maximum level ('stacking limit') of Total Government Assistance for the same eligible expenditures is not to exceed 100%.
- CEAA will use the attestation provided by the recipient to ensure that the stacking limit is not exceeded.
- In the event that the stacking limit is exceeded, CEAA will adjust its level of contribution, and if necessary, seek reimbursement so that the stacking limit is not exceeded.
Conclusions
With regard to continued relevance, the evaluation found that the PFP is well-aligned with Federal and Agency priorities. The PFP is relevant and meeting demonstrated needs of the public and Aboriginal groups.
The program is delivering its intended results. PFP support has increased participation in the EA process and thereby, has increased the information and the range of perspectives provided to the Agency and Panels for EAs. With regard to actual expenditures for contributions and for operating and maintenance expenses, the evaluation found that the PFP has operated within its budget. It is cost-effective and delivering value-for-money. Some projects may be delayed by a proponent for reasons related to the economic cycle (and beyond the control of CEAA). Contribution Agreements to support participation in the EA remain in place, even if the project has been delayed. In these cases, the budget which was set aside to support participation in the EA will lapse at the end of the fiscal year and in general, cannot be carried forward to future fiscal years. Although the signed contribution agreements remain in effect, and are noted as future obligations in the CEAA financial statements, future year transfer payment obligations for projects that are temporarily on hold for an unspecified period are not distinguished from future year transfer payment obligations for projects that are proceeding as planned.
The follow-up to the Core Control Audit included reviewing a sample of active files, and conducting interviews with program management and Senior Funding Officers. With regard to program files, the follow-up found that the program files are complete and that the expected documents are readily available. With regard to due diligence with respect to the stacking limits, the follow up found that some important additional steps had been taken since the Core Control Audit with regard to additional reporting required from recipients. The evaluation concluded that with these new measures in place, there is low risk of exceeding the stacking limits; however, due diligence regarding stacking limits can be improved by making a more precise and explicit reference to the details in program documentation.
Recommendations
Recommendation 1: As PFP is relevant, is delivering expected results, and is cost-effective, PFP should be continued.
Recommendation 2: CEAA should consider whether to publish in its public reporting additional detail regarding its future obligations to fund PFP contribution agreements for projects where contribution agreements are in place, but where the precise timing of the future year expenditure is difficult to forecast since projects have been put on hold for an unspecified period by a proponent.
Recommendation 3: In order to provide support for unexpected additional costs during the EA process, CEAA should consider providing additional support to existing contribution agreement recipients in the event that a project proponent makes a significant change to a proposed project during the EA process.
Recommendation 4: To improve due diligence and facilitate understanding of the stacking limit by applicants and recipients, the PFP stacking limits, as described in the program terms and conditions, should be noted in the standard text of PFP Contribution Agreements, and other relevant program documents and forms.
Recommendation 5: To demonstrate performance in an ongoing manner, to provide information for management decision-making and to facilitate program evaluations, the PFP should complete work on its Performance Measurement Strategy, and track performance against its published service standards.
Appendix A: Evaluation Matrix
The Evaluation Matrix below was used for the planning and conduct of this evaluation. The detailed analysis and findings for each of the evaluation issues, and for the sub-issues noted below has been presented in the main text of the evaluation report, using the structure of this matrix. The high level analysis and findings for each issue and sub-issue are summarized in column 6 of the matrix.
Evaluation Issue | Documents Review | File Review | Interviews | Performance indicator | Analysis and Findings |
Relevance: Assessment of the role and responsibilities for the federal government in delivering the program; Assessment of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes; Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians |
|||||
---|---|---|---|---|---|
#1 Is the PFP aligned with the roles, responsibilities and priorities of the federal government? |
X | Consistency with Constitutional roles, Legislation | Documents indicate full alignment with the roles, responsibilities and priorities of the federal government. | ||
#2 Is the PFP aligned with the CEAA mandate, strategic outcome and priorities? | X | Consistency with PAA; discussion in RPP and DPR; Consistency with legislation | Documents and interviews indicated full alignment with the CEAA mandate, strategic outcome and priorities. | ||
#3 Does the PFP continue to address a demonstrated need? Has the PFP been responsive to the needs of target groups? |
X | X | Statements by interviewees (program officers; CEAA senior officials; NEB and CNSC officials; recipients); results of questionnaires completed by recipients on completion of CAs; numbers of contribution agreements from public and Aboriginal groups | Documents and interviews confirmed that the PFP continued to address a demonstrative need of the public and Aboriginal groups. PFP has been responsive to the needs of target groups for support during the EA process. Interviewees noted an additional need for support during the EA process in cases where the proponent makes significant changes to a proposed project during the EA process. Interviewees noted the potential value in terms of assisting participation of receiving support pre- and post- the EA process. Interviewees noted potential value of building capacity in Aboriginal groups with low current capacity. | |
Performance (effectiveness, efficiency and economy) Effectiveness: Assessment of progress towards expected outcomes with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes |
|||||
#4 Has the PFP progressed towards the expected outcomes? | X | Statements by interviewees (program officers) |
A draft logic model was prepared. It presents the expected outcomes to be measured in this evaluation. PFP has operated as expected and progressed towards the expected outcomes noted in the draft logic model. Sub-questions related to question 4 were addressed in the evaluation from the qualitative data that was collected and are reported in more detail in the evaluation report. Highlights related to the sub-questions are summarized in this matrix. |
||
4. a. Is the PFP monitoring and reporting effective? | X | X | Results published in the RPP and DPR; Statements by interviewees (program officers; recipients) |
Overall findings, are reported in the evaluation report with regard to performance: Based on the documents and interviews, the program has operated as expected and delivered the expected outcomes. The program operated in a cost-effective manner. Administrative improvements have been put into place since the last evaluation. DPR and RPP reported planned and actual results achieved including the program budget and expenditure and the number of contribution agreements in place. The PFP process is transparent and unbiased. Opportunities are published on the Registry and made known in local media. An internal funding review committee prepares a memo to the President, based on the criteria noted in the program terms of reference. Potentially impacted Aboriginal groups receive a letter inviting them to apply. Risks are effectively managed. There are very few overpayments. The AANDC listing of First Nations in 3rd party and co-management is used by the program with regard to identifying higher risk participants. The PFP process purpose and process are well-understood. |
|
4. b. Is the PFP process transparent and unbiased? | X | Statements by interviewees (program officers; recipients) | |||
4. c. Have the risks (related to the management of the Contribution Agreements) been managed effectively? (e.g. mitigation strategies are implemented) | X | X | Risk identification and mitigation strategies from documentation; management of risks indicator: statements from interviewees (program staff) | ||
4. d. Is the PFP purpose and process understood? | X | X | X | Statements of interviewees (Program Staff; Recipients). Error rates in applications and claims; number of overpayments. | |
4. e. Does the PFP process adhere to outlined service standards? | X | X | Published Service Standards; Results of Service Standards measurement; Statements by interviewees (Program staff; Recipients) | Service Standards were published in June 2015. No quantitative data was available for the evaluation. The program has confirmed plans to measure performance against published service standards. Interviews and documents indicated that the application review process was timely. Interviewees commented positively regarding the efficiency and timeliness of advice and assistance provided by the PFP officials. |
|
4. f. Is the application review process timely? | X | X | Service standards for data regarding timely review of complete applications; statements by interviewees (program staff; recipients) | ||
4. h. Are the communication tools clear and effective? (related to 4.j) |
X | X | # of Applications received for CAs; Statements by interviewees (program staff) | Communication tools examined included the CEAA website, CEAA registry, emails, and public notices. Communication tools are clear and effective. The PFP purpose and process appear to be understood. | |
4. i. Are staff and stakeholders aware of and understand the PFP? | X | X | # of Applications received for CAs; Statements by interviewees (program staff; recipients); error rates | Based on qualitative data, staff and stakeholders are aware of the PFP and understand the program and the contribution agreement requirements. | |
4. j. Are the marketing products effective? (related to 4.h.) |
X | X | # of Applications received for CAs; Statements by interviewees (program staff) | Marketing products include the website, CEAA Registry, and the public notices. Marketing tools have been effective and participants have been responsive to PFP funding opportunities. | |
4. k. Is the engagement with potential participants effective? | X | Statements by interviewees (CEAA senior officials; Program Staff; Recipients) | Based on interviews, the engagement with potential participants was effective. | ||
#5. Has the PFP progressed towards the longer term Outcomes? | X | Statements by interviewees (Program Staff; Recipients) | The evaluation considered the outcomes noted in the draft logic model. The PFP has performed as expected and progressed towards the longer term outcomes noted in the draft logic model prepared for the purposes of this evaluation. | ||
5. a. To what extent is the PFP managed effectively? | X | X | Documented manual; plans; program delivered within approved budget; performance reports. Results of Service Standards measurements maintained by program; Statements by interviewees (Program Staff; Recipients) | Service Standards were published in June 2015. The program has confirmed plans to measure performance against the published standards. Data collected during interviews confirms that the program is well-managed. The program operates within budgets that have been allocated. Reporting is consistent and accurate. Files are well-organized and complete. Interviewees noted streamlined processes and effective use of the Internet. Program documentation and website content is clear and well-written and describes well the steps in the process. | |
5. b. To what extent have the public’s, Aboriginal groups’ and other stakeholders’ engagement added value to the EA process? | X | Statements by interviewees (Program Staff; Recipients; CEAA senior officials) | Qualitative data indicated that the participation of the public and Aboriginal groups and their engagement has brought valuable information and traditional knowledge to CEAA and Review Panels and thereby, this engagement has added value to the EA process. | ||
Performance – Efficiency and Economy: Assessment of resource utilization in relation to the production of outputs and progress towards expected outcomes | |||||
#6 (Efficiency) The extent to which timely decisions are made to provide participant funding after applications for funding are received? | X | X | Measurements against service standards maintained by program; time between receipt of complete application and funding decision in files assessed for file review | A draft performance measurement strategy is under development by the program. Service Standards were published in June 2015. No quantitative data is currently collected by PMP. Interviews provided qualitative data in regard to this question. Evidence collected indicates that the decisions are made in a timely manner. A funding review committee prepares a memo to the President with recommendations, based on application of the criteria noted in the program terms and conditions. | |
#7 (Efficiency) The extent to which recipients received approved funding in a timely manner either based on claims for eligible expenses that have been properly completed, or by exception, as advance payments that allows them to prepare for participation? |
X | Measurements against service standards maintained by program; potential comments by interviewees (program staff; recipients) | No quantitative data is currently collected by PMP. Service standards are under development. Interviews provided qualitative data in regard to this question. Interviews indicated that claims are processed in a timely manner. There were no complaints or issues raised by interviewees. A risk management framework for identification of low, medium, or high risk payments is in preparation, and being used informally. Advance payments are used by exception only. One interviewee had received an advance payment and there were no issues. The AANDC published listing of First Nations in Default management is consulted. First time recipients are noted as higher risk than experienced participants with a good track record. |
||
#8 (Economy) Does the program operate within the approved budget and is the level of operational expenditure reasonable when compared with the level of program (transfer payment) expenditures? |
X | X | Program operates within program budget each fy; evidence in files, or statements of interviewees, regarding areas for improvements in efficiency or economy, sound stewardship (lack of evidence of waste or duplication); Ratio of internal expenditure by fy to total number of CAs administered during fy (data, if maintained by the program will be assessed); ratio of internal expenditure to total value of CA expenditure (data, if maintained by program will be assessed) | The evaluation was guided in analysis of the data available for this question by the TB Transfer Payment Policy definition of Value for Money [the extent to which a program demonstrates relevance and performance. Relevance is achieved by addressing a demonstrable need that is appropriate for the federal government and is responsive to the needs of Canadians. Performance is achieved by using taxpayer resources well, producing program outputs in an affordable manner, and achieving outcomes consistent with program objectives].[17] PLP operates within the approved budget for both transfer payments and for O&M expenditures. Actual transfer payment expenditures have been less than expected in each of the past five fiscal years for reasons outside the control of CEAA. Operational expenditure has been less than the 20% allowed and is reasonable compared with the level of program expenditures. |
|
Follow-up of Recommendations from Core Control Audit (Management Information Needs) | |||||
Core Control Audit Recommendation 12B follow up To what extent are necessary PFP supporting records filed effectively? |
X | X | Completeness of files in File review; statements by interviewees (PFP officials) | Files were complete, well-organized, filed effectively and readily accessible. File review included a random sample of files from a list of active files, provided by the program. Additional files were reviewed related to the projects where PFP support was provided and where interviews with recipients were planned. File review was conducted on site at CEAA. | |
Core Control Audit Recommendation 13B follow up To what extent is the PFP exercising due diligence with respect to stacking limits? |
X | X | References in manual, application, CA, claim forms regarding stacking (other [federal] sources of funding for any one activity, initiative or project of a recipient); statements by interviewees (PFP officials) | CEAA is exercising due diligence with respect to stacking limits. Following the Core Control Audit, applicants and recipients were required to disclose additional sources of funding. The evaluation noted that there are no explicit references in the PFP contribution agreement or forms to the precise details of the stacking limit. The evaluation conclusion is that the exercise of due diligence has improved, but that this can be improved further by making more explicit reference to the details regarding the stacking limit in the contribution agreement and program forms and documentation. This would allow recipients to know how CEAA uses the information that is requested, and would facilitate understanding by recipients of the stacking limit. |
Appendix B: Interview Guide
Interview guides were prepared for each set of interviewees. An example of the interview guides that were used for this evaluation is provided in this appendix to facilitate future evaluation and performance measurement work.
The sample interview guides that are presented in this Appendix were used with (1) CEAA Officials, and with (2) Recipients of Funding from the CEAA Participant Funding Program.
- Interview Guide for CEAA Officials
- Interview Guide for use with Recipients of Funding from the CEAA Participant Funding Program
Additional versions of this interview guide were tailored to the specific knowledge of each set of interviewees. The following interview guides were prepared but are not included in the Appendix:
- Interview Guide for NEB and CNSC Officials
- Interview Guide for Officials from Environment Canada, and Aboriginal Affairs and Northern Development Canada
- Interview Guide for Recipients of Contribution Agreements (Regular and Aboriginal Funding)
Interview Guide for CEAA Officials
Project Title: Evaluation of the Participant Funding Program
Date:
Attendees:
Introduction
Description of the Evaluation:
The Canadian Environmental Assessment Agency (CEAA) has engaged Alan Winberg Consulting Inc. to conduct an evaluation of the Participant Funding Program (PFP).
In order to help us with this evaluation, we would like to discuss with you the continued relevance and performance of this program and its implementation.
Overview of the PFP:
Legislative requirement: Responsible federal authorities involved in an environmental assessment process are required to establish participant funding programs under the Canadian Environmental Assessment Act, 2012.
Support provided: The CEAA administers the Participant Funding Program, which supports individuals, non-profit organizations and Aboriginal groups interested in participating in federal environmental assessments.
Objectives: Participating in federal environmental assessments helps to ensure that concerns from the public and Aboriginal groups, regarding the potential effects of a project on the environment, on the public, on Aboriginal groups and on existing or potential Aboriginal or Treaty rights, are taken into consideration.
Eligible costs: During the environmental assessment process, Participant funding can cover eligible costs such as expert advice, travel expenses and reporting costs.
To be eligible to apply, an applicant must demonstrate the value the applicant will add by participating in the environmental assessment. In addition, applicants must meet at least one of the following criteria:
- Have a direct, local interest in the project, such as living or owning property in the project area;
- Have community knowledge or Aboriginal traditional knowledge relevant to the environmental assessment;
- Plan to provide expert information relevant to the anticipated environmental effects of the project; and/or
- Have an interest in the potential impacts of the project on treaty lands, settlement lands or traditional territories and/or related claims and rights.
Confidentiality and Information Management:
We would like to assure you that the information gathered in individual interviews will be reported only in the aggregate. Your name will not be revealed or associated with your responses. The information collected from this interview will be kept in a secure location and will only be viewed by CEAA staff and members of our study team working on the evaluation.
Do you have any questions in this regard?
Background Questions
1. To start our discussion, can you please describe your responsibilities as they relate to CEAA’s Participant Funding Program?
PFP Program Evaluation Questions
During the evaluation we will be considering the continued relevance of the PFP and its performance (achievement of results and cost effectiveness). We have organized the questions for our discussion in that manner.
Looking at the relevance of the PFP:
2. On a scale of 1 to 7, where 1 is “not at all”, 4 is “somewhat”, and 7 is “completely”, to what extent is the PFP program well-aligned with the federal government’s stated priorities and the strategic outcomes of the Canadian Environmental Assessment Agency?
1 2 3 4 5 6 7
Please explain the rationale for your response:
3. To the best of your knowledge, to what extent does the Program continue to address a demonstrated public need or a demonstrated need of Aboriginal communities?
1 2 3 4 5 6 7
Please explain what you see as the key needs of the public or the needs of Aboriginal communities for this Program:
The next question looks at the performance of the PFP in terms of progress towards expected outcomes (i.e. results):
4. In your experience what have been the most significant outcomes achieved to date by the PFP?
Now we would like to consider cost-effectiveness and value for money.
5. What are the key challenges to the delivery of the PFP?
6. What are the key benefits (value) delivered by the PFP?
Over what timeframe would you expect these benefits to flow?
7. Do you have specific suggestions regarding how the PFP can be improved to better achieve the desired results at the same or a reduced cost?
Conclusion
8. Do you have any additional comments or questions?
Thank you for taking the time to participate in this interview.
If you have any further questions or comments to add, please contact [deleted] at email address [deleted] or telephone number [deleted].
Interview Guide for use with Recipients of Funding from the CEAA Participant Funding Program
Project Title: Evaluation of the Participant Funding Program
Date:
Attendees:
Introduction
Description of the Evaluation:
The Canadian Environmental Assessment Agency (CEAA) has engaged Alan Winberg Consulting Inc. to conduct an evaluation of its Participant Funding Program (PFP). We understand that you have received funding from this program.
In order to help us with this evaluation, we would like to ask you a few questions to gather facts related to your participation in the program. This will help us make improvements in order to better serve future participants.
Overview of the PFP:
Legislative requirement: Responsible federal authorities involved in an environmental assessment process are required to establish participant funding programs under the Canadian Environmental Assessment Act, 2012.
Support provided: The CEAA administers the Participant Funding Program, which supports individuals, non-profit organizations and Aboriginal groups interested in participating in federal environmental assessments.
Objectives: Participating in federal environmental assessments helps to ensure that concerns from the public and Aboriginal groups, regarding the potential effects of a project on the environment, on the public, on Aboriginal groups and on existing or potential Aboriginal or Treaty rights, are taken into consideration.
Eligible costs: During the environmental assessment process, Participant funding can cover eligible costs such as expert advice, travel expenses and reporting costs.
To be eligible to apply, an applicant must demonstrate the value the applicant will add by participating in the environmental assessment. In addition, applicants must meet at least one of the following criteria:
- Have a direct, local interest in the project, such as living or owning property in the project area;
- Have community knowledge or Aboriginal traditional knowledge relevant to the environmental assessment;
- Plan to provide expert information relevant to the anticipated environmental effects of the project; and/or
- Have an interest in the potential impacts of the project on treaty lands, settlement lands or traditional territories and/or related claims and rights.
Confidentiality and Information Management:
We would like to assure you that the information gathered in individual interviews will be reported only in the aggregate. Your name will not be revealed or associated with your responses. The information collected from this interview will be kept in a secure location and will only be accessed by CEAA staff and members of our team working on the evaluation.
Background Questions
1. To begin, can you please describe your job title and responsibilities, in particular, those that are related to the Participant Funding Program?
2. How long have you worked in this area?
PFP Evaluation Questions
We will be considering the continued relevance of the program and its performance (in terms of achievement of results and cost effectiveness). We have organized the questions for our discussion in that manner.
Relevance of CEAA’s Participant Funding Program - (Is this Program responding to a demonstrated need?)
Does your organization / community need a program like this one in order to participate in an environmental assessment process?
3. On a scale of 1 to 7, where 1 is “not at all”, 4 is “somewhat”, and 7 is “completely” or “very much”, to what extent is this Program responding to a demonstrated need in your community or organization?
1 2 3 4 5 6 7
4. What needs are addressed by this program?
Results achieved from the support received from the CEAA’s Participant Funding Program –
5. On a scale of 1 to 7, where 1 is “not at all”, 4 is “somewhat”, and 7 is “completely” or “very much”, to what extent did the Program help your community or organization to participate in the EA process?
1 2 3 4 5 6 7
6. What activities did your organization complete with the support received from the Participant Funding Program?
Efficient program delivery
7. How did your organization become aware of the Program?
8. Was the program application process clear and easy to understand?
9. Did you contact the program officials at Canadian Environmental Assessment Agency for help to complete the application?
If you contacted them to complete the application, did you receive the help you needed?
10. Was the Contribution Agreement clear and easy to follow?
11. Did you contact the program officials at the Canadian Environmental Assessment Agency for help to understand the Contribution Agreement?
If you contacted them regarding the Contribution Agreement, did you receive the help you needed?
12. Was the expense claims process clear?
Did the expense claims process work as you expected?
Did you receive the funding in a timely manner once the expense claim was completed?
Conclusion
Thank you for taking the time to participate in this interview.
Annex C: Individuals and organizations invited to participate in an interview
From Canadian Environmental Assessment Agency
- CEAA President
- CEAA VP Operations
- CEAA VP Corporate Services
- Director General, Regional Operations
- Manager, Aboriginal Consultation Unit
- Officials from CEAA, Review Panels Division
- Senior Policy Analyst, Review Panels Division
- Consultation Coordinator, Review Panels Division
- Senior officials from Review Panels Division, CEAA
- Regional Directors and Aboriginal Advisors (teleconferences)
PFP Team
- Senior Manager and Senior Funding Officers, PFP
- Director, National Programs
- Director, Finance and Administration
Other Federal departments and agencies
- Officials from PFP program at National Energy Board
- Advisor, Aboriginal Consultation at Canadian Nuclear Safety Commission
- Partnerships Manager, Consultation & Accommodation unit, Treaties and Aboriginal Government Aboriginal Affairs and Northern Development Canada
Recipients of Contribution Agreements
- Recipients of Contribution Agreements were selected in collaboration with CEAA officials, and drawn from a range of projects in several provinces. A listing of projects is presented below. This list was been prepared in collaboration with Program Management.
Project | Location | Number of invitations to be sent | |
---|---|---|---|
Regular funding recipients | Aboriginal funding recipients | ||
Shelburne Basin Venture Exploration Drilling Project | Atlantic | 2 | 1 |
Sisson Project (Tungsten and Molybdenum Mine) | NB | 1 | 1 |
Black Point Quarry Project | NS | 1 | 0 |
Arnaud Mining Project | Quebec | 1 | 1 |
BlackRock Mining Project | Quebec | 0 | 1 |
Dumont Nickel Mine Project | Quebec | 1 | 0 |
Rainy River Project | Ontario | 2 | 2 |
Côté Gold Mine Project | Ontario | 1 | 2 |
Keeyask Generation Project | Manitoba | 0 | 1 |
Tazi Twé Hydroelectric Project | Saskatchewan | 0 | 1 |
Robb Trend Coal Mine Expansion Project | Alberta | 1 | 1 |
Murray River Coal Project | BC | 1 | 1 |
Pacific Northwest LNG Project | BC | 3 | 1 |
Brucejack Gold Mine Project | BC | 0 | 1 |
Total invited | 14 | 14 |
Appendix D: Documents Reviewed
AANDC. Aboriginal Consultation and Accommodation - Updated Guidelines for Federal Officials to Fulfill the Duty to Consult - March 2011 Accessed at: http://www.aadnc-aandc.gc.ca/eng/1100100014664/1100100014675
Canada. Justice Laws Website. Canadian Environmental Assessment Act, 2012. Sections 57 and 58. Accessed at: http://laws-lois.justice.gc.ca/eng/acts/c-15.21/page-1.html.
Canada. Justice Laws Website. Financial Administration Act. In particular, section 42.1. Accessed at: http://laws-lois.justice.gc.ca/eng/acts/F-11/page-25.html#docCont.
Canada. Office of the Auditor General of Canada. Chapter 4—Implementation of the Canadian Environmental Assessment Act, 2012. 2014 Fall Report of the Commissioner of the Environment and Sustainable Development. Accessed at: http://www.oag-bvg.gc.ca/internet/English/parl_cesd_201410_04_e_39851.html#hd4c.
CEAA. CEAA website. In particular, Frequently Asked Questions, applicant materials, policy documents on the CEAA Participant Funding Program (PFP). Accessed at: https://www.ceaa-acee.gc.ca/default.asp?lang=en&n=D75FB358-1. CEAA Registry. Accessed at: http://www.ceaa-acee.gc.ca/050/introduction-eng.cfm.
CEAA. CEAA website. In particular, website pages aimed at providing information regarding the projects listed here: Shelburne Basin Venture Exploration Drilling Project (Atlantic); Arnaud Mining Project (Quebec); Rainy River Project (Ontario); Keeyask Generation Project (Manitoba); Tazi Twé Hydroelectric Project (Saskatchewan); Joslyn North Mine (Alberta); Pacific Northwest LNG Project (BC).
CEAA. PFP. National Program Guidelines. Website version accessed at: https://www.ceaa-acee.gc.ca/default.asp?lang=En&n=9772442E-1&offset=4&toc= hide.
CEAA. Evaluation of the Participant Funding Program. Prepared for CEAA by the Government Consulting Service. Project Number 570-2757 (Program Evaluation and Performance Measurement Services). March 2009.
CEAA. Unpublished document. ““As Is” Model for PFP Business Process”. Provided to evaluator by CEAA, April 28, 2015.
CEAA. Unpublished documents related to the Questionnaire sent to participants of contribution agreements. These include: Questionnaire – EN – 20110421. doc (Satisfaction Questionnaire sent to recipients); Letter – EN – CEAA. Unpublished documents related to the Questionnaire sent to participants of contribution agreements. These include: Questionnaire – EN – 20110421.doc (Satisfaction Questionnaire sent to recipients); Letter – EN – 20120403. doc (cover letter to the Satisfaction Questionnaire sent to recipients); Instructions – CEAA. Unpublished documents related to the Questionnaire sent to participants of contribution agreements. These include: Questionnaire – EN – 20110421.doc (Satisfaction Questionnaire sent to recipients); Letter – EN – 20120403.doc (cover letter to the Satisfaction Questionnaire sent to recipients); Instructions – EN.doc (instruction sheet as part of the Satisfaction Questionnaire sent to recipients); and, Compilation of Responses from Questionnaire FY CEAA. Unpublished documents related to the Questionnaire sent to participants of contribution agreements. These include: Questionnaire – EN – 20110421.doc (Satisfaction Questionnaire sent to recipients); Letter – EN – 20120403.doc (cover letter to the Satisfaction Questionnaire sent to recipients); Instructions – EN.doc (instruction sheet as part of the Satisfaction Questionnaire sent to recipients); and, Compilation of Responses from Questionnaire FY 2014 -2015.xls (compilation recording the responses PFP obtained from returned Satisfaction Questionnaires). There are two worksheets. One records the level of satisfaction of the recipient. The second worksheet records the written comments from recipients.
CEAA. Unpublished document. Funding Disbursed to Recipients FY 2014 -2015.xlsx (contains data on the funding disbursed to recipients in the 2014-2015 FY). A random selection from this listing of files was made. The individual contribution agreement files in this small sample of files was reviewed.
CEAA. Unpublished document. PFP. Procedures. User Guide For Participant Funding Program Officers. Last Updated May 2015.
CEAA. Unpublished document. PFP. Risk-Based Recipient Assessment Framework and Risk Assessment Tool. Draft. Dated May 12, 2014.
CEAA. Unpublished PowerPoint presentation. “Participant Funding Program (PFP) under the Canadian Environmental Assessment Act, 2012: Supporting Public and Aboriginal Participation in Environmental Assessments”. 2014.09.01
CEEA and BC Ministry of the Environment Environmental Assessment Office. Contribution Agreement for the Administration of Fund Pursuant to the Participant Funding Program for Substituted Environmental Assessments: Master Agreement. October 18, 2013; and Schedule 1 to the Master Agreement: Participant Funding Program – Project: LNG Canada Export Terminal Project. May 26 & 27, 2014.
CEAA and Office of the Comptroller General. “Audit of Core Controls” and CEAA “Management Action Plan and Response”. June 2014. In particular, recommendations 12 and 13 and the CEAA Management Action Plan and Response. Both the OCG Audit report and the CEAA Management Action Plan and Response were accessed at: https://www.ceaa-acee.gc.ca/default.asp?lang=En&n=CAFA626C-1.
Canadian Nuclear Safety Commission. Participant Funding Program. Website. Accessed at: http://nuclearsafety.gc.ca/eng/the-commission/participant-funding-program/index.cfm.
Environment Canada. Website. The Green Source Funding Database – Participant Funding Program (sponsored by CEAA). Accessed at: https://www.ec.gc.ca/financement-funding/sv-gs/search_results_e.cfm?action=details&id=292&start_row=1&all_records_details=fund&type_of_funder=Federal.
Iler Campbell LLP. “Intervenor Funding and Access to Environmental Justice: Time for the Ontario Political Parties to revisit this issue?” Paper published to Iler Campbell LLP Barristers and Solicitors Website. Toronto. August 2011. Accessed at: http://www.ilercampbell.com/blog/wp-content/uploads/Intervenor-Funding-and-Participation.pdf.
Kirchhoff, Dennis, Gardner, H. L., and Tsuji, L. J. “The Canadian Environmental Assessment Act, 2012 and Associated Policy: Implications for Aboriginal Peoples”. The International Indigenous Policy Journal, 4(3). 2013.06.28. Accessed at http://ir.lib.uwo.ca/cgi/viewcontent.cgi?article=1139&context= iipj.
National Energy Board. Participant Funding Program. Website. Accessed at: http://www.neb-one.gc.ca/prtcptn/hrng/pfp/prtcpntfndngprgrm-eng.html.
National Energy Board. Participant Funding Program Guide. December 9, 2014 Accessed at: http://www.neb-one.gc.ca/prtcptn/hrng/pfp/prgrmgd-eng.html.
Natural Resources Canada. “Forging Partnerships – building Relationships: Aboriginal Canadians and Energy Development. Report to the Prime Minister by Douglas R. Eyford”. 29 November 2013. Accessed at: http://www.nrcan.gc.ca/sites/www.nrcan.gc.ca/files/www/pdf/publications/ForgPart-Online-e.pdf.
Stratos Inc. Evaluation of the Cabinet Directive on the Environmental Assessment of Policy, Plan and Program Proposals. Final Report. June 30, 2009. Accessed at: http://www.ceaa.gc.ca/Content/5/D/1/5D1C8F32-9457-4C5C-9589-541753A446E7/SEA_Evaluation_Summary_Report-eng.pdf
Treasury Board of Canada. Treasury Board Policy on Transfer Payments. Accessed at: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=13525; Directive on Transfer Payments. Accessed at: Treasury Board of Canada. Treasury Board Policy on Transfer Payments. Accessed at: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=13525; Directive on Transfer Payments. Accessed at: http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id= 14208.
Appendix E: File Review Sample
Project | Location | Selected sample of contribution agreement Files reviewed (Selected in collaboration with Program Management and stratified to include projects from across Canada) | Random Sample of contribution agreement files reviewed |
---|---|---|---|
Shelburne Basin Venture Exploration Drilling Project | Atlantic | 4 | 1 |
Sisson Project (Tungsten and Molybdenum Mine) | NB | 2 | 1 |
Black Point Quarry Project | NS | 2 | |
Arnaud Mining Project | Quebec | 3 | |
BlackRock Mining Project | Quebec | 1 | |
Dumont Nickel Mine Project | Quebec | 1 | |
Rainy River Project | Ontario | 4 | 1 |
Côté Gold Mine Project | Ontario | 3 | |
Keeyask Generation Project | Manitoba | 1 | |
Tazi Twé Hydroelectric Project | Saskatchewan | 1 | |
Pierre River Mine | Alberta | 1 | |
Robb Trend Coal Mine Expansion Project | Alberta | 4 | 1 |
Site C Clean Energy (part of random sample) | BC | 3 | |
Murray River Coal Project | BC | 3 | |
Pacific Northwest LNG Project | BC | 4 | 4 |
Brucejack Gold Mine Project | BC | 1 | |
Total number of files reviewed | 34 | 12 |
[1] CEAA and Office of the Comptroller General. “Audit of Core Controls” and CEAA “Management Action Plan and Response”. June 2014. In particular, recommendations 12 and 13 and the CEAA Management Action Plan and Response. Please see: https://www.ceaa-acee.gc.ca/default.asp?lang=En&n=CAFA626C-1.
[2] Under sections 57 and 58 of CEAA 2012, the CEAA is required to establish a PFP to facilitate the participation of the public in environmental assessments (EAs). This applies to EAs conducted by the Agency and EAs by review panels. These sections of the Act may be accessed at: http://laws-lois.justice.gc.ca/eng/acts/c-15.21/page-17.html#h-25.
[3] The CEAA Public Registry may be accessed at: http://www.ceaa-acee.gc.ca/050/index-eng.cfm.
[4] The Consulting with Canadians portal may be accessed at: http://www1.canada.ca/consultingcanadians/?_ga=1.154248895.660974279.1442251782.
[5] $1.469 million is continuing, permanent A-base funding. $3 million is time-specific B-base funding which was renewed in Budget 2015 and is in place for the period from April 1, 2015 until March 31, 2020.
[6] Source of financial data is CEAA, Finance and Administration.
[7] The Funding Decision Criteria, set out at section 4 of the program terms and conditions are: the potential environmental effects of the project; the potential impacts of the project on potential or established Aboriginal rights to title; the size and location of the project; the diversity of issues likely to be involved in the assessment; the level of public interest, including that of potentially affected Aboriginal groups, on the project; the number of funding applications that may be expected; any maximum funding allocation per recipient that may have been pre-established by the President of the Agency for any or all of the eligible activities identified in Section 5 of the terms and conditions; and, available resources.
[8] TB Transfer Payment Policy, section 6.5.9, requires the establishment of reasonable and practical departmental service standards for transfer payment programs.
[9] http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=25750§ion=text.
[10] CEAA’s PFP service standards are published at: http://www.ceaa.gc.ca/default.asp?lang=en&n=9772442E-1&offset=4&toc=hide#ccs.
[11] Loc. Cit.
[12] PFP officials have identified the following as dormant projects which have been delayed by the proponent as of October 15, 2015: Eagle’s Nest; Fire Lake North Iron Ore; Griffith Iron Ore; Harper Creek; Highway 905 – All Weather; Hopes; Josephine Cone Mine; Kingvale-Oliver Natural Gas; Kipawa Rare Earths; Marathon PGM Mine; Niobec Mine Expansion; Raven; Schaft Creek Mine; Spanish Mountain Gold Mine; and Victor Diamond Mine Ext. Additional information regarding these projects is accessible at the CEAA registry website, http://www.ceaa.gc.ca/050/index-eng.cfm
[13] http://www.ceaa-acee.gc.ca/default.asp?lang=En&n=1A09C820-1&offset=2&toc=hide
[14] These are presented at “Note 9”. Please see: http://www.ceaa-acee.gc.ca/default.asp?lang=En&n=9156A93A-1
[15] From Section 7 of the Program Terms and Conditions (June 2013): “The maximum level (stacking limit) of Total Government Assistance (federal, provincial/territorial and municipal assistance for the same eligible expenditures) for this FPF will not exceed 100% of eligible expenditures . In the event that actual Total Government Assistance to a recipient exceeds the stacking limit, it will be necessary for the Agency to adjust its level of contribution (and seek reimbursement, if necessary) so that the stacking limit is not exceeded.”
[16] This additional precision could be included in the Ineligible expenses section of the National Guidelines. For example, a text could be added at page 5 such as:
The Agency's Participant Funding Program has a ‘stacking limit’, that is, a maximum permitted amount of combined funding from federal, provincial, territorial and municipal governments for participation in any one Environmental Assessment project or associated Crown consultation activities. Combined funding from governments to the Recipient for participation in an Environmental Assessment or associated Crown consultation cannot exceed 100% of eligible expenditures. CEAA uses the attestation provided by the Recipient in the application and in the claim forms to verify that a Recipient's funding request and expenditure claims conform to this stacking limit. If the stacking limit is exceeded, CEAA will adjust its level of contribution, and if necessary, seek reimbursement so that the stacking limit is not exceeded. A paragraph regarding the stacking limit forms part of our standard Contribution Agreement with Recipients.
Page details
- Date modified: