Species at risk program frameworks : chapter 6

6. Evaluation Strategy

Evaluation studies provide information, beyond ongoing monitoring, about key aspects of Program operations and outcomes, as well as the continued relevance and possible alternatives.

Since the implementation of SARA and the management of the SAR Program are undertaken by EC, DFO and PC, the core departments will work co-operatively on SAR Program evaluations. The Audit and Evaluation Branch of EC will chair the management of the evaluations in close consultation with its counterparts at DFO and PC.

6.1 Overall Approach

The overall approach to monitoring and evaluating any program is one of staged expectations, learning and adjustment. The overall approach will be guided by the Program results logic (logic models) and performance measurement strategies presented in sections 3 and 5

The stages of the evaluation study are:

The Evaluation Report will summarize findings by

6.2 Evaluation Issues

In developing the evaluation issues, the following factors were considered:

The evaluation questions fall under four broad issue categories:

Table 14 presents the evaluation issues and questions, and data sources and methods. For each question, multiple lines of evidence are suggested to increase the reliability and validity of the evaluation information.

Table 14: Evaluation Issues, Data Sources and Methods
Evaluation Theme and Questions Data Source Data Collection / Analysis Methods
Program Rationale (SAR Program)
Is there a legitimate and necessary role for the government in the protection and recovery of species at risk?

Is the current role of the federal government appropriate or are there areas that are candidates for re-alignment with the provinces, territories or others?

What activities could be transferred to the private or voluntary sectors, or other level of government?
Literature

Program managers

Experts

Program partners

Aboriginal people

Program stakeholders
Literature review

Interviews

Expert panel
Program Design and Delivery (for each department, SAR element)
Are the roles and responsibilities of COSEWIC, the core departments and P/Ts in the implementation of the Accord and SARA clear? Is there an appropriate accountability framework in place?

Do the governance structures and mechanisms support interdepartmental and interjurisdictional cooperation and consistency in the application of SARA?

To what extent do decision-making and planning processes allow for strategic ranking of species and prioritization of activities? To what extent are multi-species and ecosystem-based analyses used?

Are consultations with partners, Aboriginal people and stakeholders effective? Are there ongoing opportunities for partners, Aboriginal people and stakeholders to provide input?

Do SAR Program policies support consistent program delivery and implementation of SARA across the core departments?

Is there an effective performance monitoring system in place to support program management and demonstrate results?

Is the overall capacity (resources) commensurate with the program design, delivery and results expectations?

To what extent have Program risks been effectively managed? Is the risk management strategy adequate?  Have new risk areas emerged?
Program managers

Program partners

Aboriginal people

Inter-departmental Action Plan Status Report

Program measurement strategy / indicators
Program documentation review

Interviews

Surveys

Case studies

Workshop
Program Success / Impacts (for each core department, by SAR element)
To what extent has the SAR Program generated the expected outputs? See Table 13 for a list of outputs and indicators. Performance measurement strategy (Table 13)

Program managers
Performance indicator assessment

Program documentation review

Interviews

Surveys
To what extent have the expected outcomes been achieved? (See Table 13 for a list of expected outcomes and indicators.)  Performance measurement strategy (Table 13)

Program managers

Experts

Program partners

Aboriginal people

Program stakeholders
Performance indicator assessment

Interviews

Surveys

Case studies

Workshop
To what extent are federal and provincial governments collaborating in support of the Accord and the Act? Are there effective mechanisms in place to coordinate delivery (e.g., bilateral agreements)?
Have the mandatory requirements of SARA been met? Is the intent of the Act being met?
What are the barriers to success?
Program managers

Experts

Program partners

Aboriginal people
Interviews

Surveys

Case studies

Workshop
Cost Effectiveness (SAR Program)
Are there better ways of achieving the results, including alternatives for delivery? Could efficiency be improved?

To what extent do SARA and the SAR Program complement or duplicate other federal legislation and/or provincial or territorial legislation and programs?

What is the value of a multi-species or ecosystem-based approach as compared with a single-species approach?

Is the overall SAR program affordable?  If not, what programs or activities would be abandoned?
Evaluation findings, including program leverage

Program financial data

Core department SAR Program managers

P/T SAR managers
Synthesis of evaluation findings

Analyses of delivery costs; program documentation review

Interviews

Surveys

Workshop

6.3 Evaluation Methodologies

Program evaluation methodologies that could be considered include

6.4 Evaluation Timing

SARA legislation must be reviewed by Parliament after it has been in force for five years. The five-year review of SARA is expected to begin in mid-2008. 

Given that the Formative Evaluation of the SAR Program was completed in 2006 and was undertaken at the early stages of implementation of the SAR Program and made several recommendations, the SAR Program RMAF and RBAF will be in place in 2007-2008, and the Parliamentary Review of SARA will be initiated in mid-2008; the outcome evaluation of the SAR Program will be undertaken in 2010-2011 (five years after Formative Evaluation, three years after the establishment of the RMAF and RBAF and possibly one year after Parliamentary Review, depending on when the review is completed). This will provide adequate time for the SAR Program to undertake required adjustments in response to the recommendations from the Formative Evaluation and possibly the Parliamentary Review, and to gather at least three years worth of information associated with the RMAF and RBAF’s performance indicators, which are essential for undertaken a meaningful and valuable evaluation. Furthermore, this will allow the SAR Program to return to government in 2011-2012 on the need for additional funding (the five-year funding received from the government is ending in 2011-2012) with the support of the outcome evaluation report.

An evaluation plan for the conduct of the outcome evaluation will be developed in 2009-2010, prior to the conduct of the outcome evaluation. The evaluation plan will be developed by the core departments. The Audit and Evaluation Branch of EC will chair the management of the evaluation plan in close consultation with its counterparts at DFO and PC.

6.5 Evaluation Costs

An estimated budget of $250,000 will be required to develop the evaluation plan and undertake the outcome evaluation. The funds for such work will be proportionally distributed by the level of funds available to each organization from the federal Species at Risk Program.

Page details

Date modified: