2013 Annual Report on the Health of the Evaluation Function

1.1 Purpose of this report

The 2013 Annual Report on the Health of the Evaluation Function fulfills a key responsibility of the Treasury Board of Canada Secretariat (Secretariat) to monitor and report annually on government-wide evaluation priorities and the health of the evaluation function.

The report focuses on the performance of large departments and agencies subject to the Policy on Evaluation; it does not include information on small departments and agencies.^{See Footnote 1} The report considers changes between 2011–12 and 2012–13 and provides a government-wide view of the health of the evaluation function over the renewed policy's four-year phase-in period from April 2009 to March 2013.

1.2 Areas addressed in this report

The report addresses six key areas:

• resources for evaluation;
• coverage;
• governance and support;
• quality of evaluations;
• use of evaluations; and
• support provided by the Secretariat.

1.3 Information sources used in preparing this report

This report draws information from the Secretariat's ongoing monitoring of the function, particularly via the receipt of departmentally approved departmental evaluation plans and reports; the annual fiscal year-end Capacity Assessment Survey of departmental evaluation functions; other documents submitted to the Secretariat; and regular interactions with departmental evaluation units. All information used in this report is actual reported data following the close of the fiscal year.

2.1 Financial resources

Expenditures on evaluation increased when the current Policy on Evaluation was introduced, going to a high of $61.2 million in 2009–10. At the close of the phase-in period, expenditures were at $56.2 million. This amount was overall slightly below the level of expenditure by evaluation functions that was in place prior to the introduction of the renewed policy in 2009.

As shown in Table 1, the annual expenditures of evaluation functions in 2012–13 declined relative to 2011–12 levels:

• Total financial expenditures by evaluation functions were reported at $56.2 million in 2012–13, which represents a decrease of approximately 3 per cent from 2011–12 figures. A decrease in evaluation expenditures is consistent with departmental decreases in overall operating expenditures over the same period. The median amount that departments devoted to their evaluation functions was $1.1 million in 2012–13. This represents a decrease from the 2011–12 median of $1.3 million.
• In 2012–13, the amount spent on evaluation across the Government of Canada was 0.5 per cent of the combined dollar value of all the programs evaluated in 2012–13.
• The operating and maintenance expenditures of evaluation functions in 2012–13 decreased by 17 per cent (or $0.8 million) from 2011–12 levels.
• In 2012–13, there was an increase of 4.6 per cent over 2011–12 figures for salary expenditures. This area of expenditure continued to represent the largest component of total expenditures by the function at 73 per cent.

Expenditures by evaluation functions on professional services decreased by nearly 44 per cent, from $20.5 million in 2008–09 to $11.6 million in 2012–13. As a proportion of total expenditures, professional services went from representing 36 per cent of overall expenditures in 2008–09 to representing 21 per cent of overall expenditures in 2012–13.

Conversely, salary expenditures increased from 56 per cent of overall expenditures to 73 per cent of overall expenditures between 2008–09 and 2012–13. Since the introduction of the 2009 Policy on Evaluation, a greater proportion of financial expenditures has come to be associated with the expenses of salaried staff versus expenditures on professional services. 

2.2 Human resources

The number of salaried evaluation resources was greater on average by about 12 per cent throughout the policy phase-in period relative to the level of resources in place prior to the introduction of the policy in 2009. As noted, throughout the phase-in period departments shifted their use of financial resources from expenditures on professional services to expenditures on internal human resources.

As shown in Table 2, the number of full-time equivalents (FTEs) dedicated to the evaluation function fluctuated throughout the phase-in period and experienced a small decline between 2011–12 and 2012–13:

• Between 2011–12 and 2012–13, there was a decrease of 18 FTEs, from 477 FTEs to 459 FTEs.
• The median number of FTEs per department or agency was 10.7 FTEs, which is a decrease from 13.3 FTEs in 2011–12.
• While the absolute number of FTEs declined in 2012–13, salary expenditures increased marginally during the same period. 

2.3 Contracted services

Contracted services include any contracted expenditures associated with planning and conducting evaluations. They are a subset of professional services and speak more specifically to the use of external contractors for planning and conducting evaluations.

In 2012–13, only 51 per cent of evaluations (62 of 122 evaluations) involved contractors for at least some part of the work, while the remainder did not involve any contractors. In comparison, 73 per cent of evaluations involved contractors in 2010–11, and 64 per cent in 2011–12.

In 2012–13, total contracted expenditures reported by departments for evaluation work was $8.1 million, which represents 14 per cent of all evaluation resources expended over the fiscal year. In comparison, the total contracted cost of evaluation services in 2011–12 was $12.4 million, or 21 per cent of all evaluation resources expended in that year.

3.1 Coverage of ongoing programs of grants and contributions

This year's report is the second report to cover a five-year period since the advent of the legal requirement to evaluate all ongoing programs of grants and contributions every five years. This legal requirement came into force on December 12, 2006, when the 2006 Federal Accountability Act amended the Financial Administration Act (FAA).

During the 2012–13 fiscal year, the Secretariat examined compliance with FAA coverage requirements via the use of a confirmation mechanism, which involved assessing financial data reported by departments, reviewing departmental evaluation plans and reports completed in-year, and considering the data available in the Main Estimates.

Results of the assessment process showed that 22 of the 31 large departments and agencies that had administered ongoing programs of grants and contributions from 2008–09 to 2012–13 had fully evaluated these programs, while 9 had not. The latter had partially evaluated these programs: for 5 large departments and agencies, unevaluated programs represented less than 5 per cent of their total spending on ongoing programs of grants and contributions; for the remaining 4, unevaluated programs represented more than 5 per cent of their total ongoing grant and contribution spending.

The Centre of Excellence for Evaluation (CEE) also examined coverage of ongoing programs of grants and contributions as a proportion of grants and contributions displayed in the Main Estimates.^{See Footnote 2} These figures can be found in Table 3. Findings include the following:

• The cumulative government-wide evaluation coverage of ongoing programs of grants and contributions over the five-year period from 2008–09 to 2012–13 is estimated at 88 per cent.^{See Footnote 3}
• The proportion of ongoing program of grant and contribution spending evaluated in 2012–13 was estimated at 11.9 per cent. This varied from 2011–12 levels when the proportion evaluated was at 18.5 per cent. Fluctuations in coverage levels are anticipated from year to year, given the variability in the funding renewal schedule of grants and contributions programs.
• Over the phase-in period, the average annual coverage of ongoing program spending of grants and contributions was approximately 17.6 per cent.

3.2 Coverage of direct program spending

From 2009–10 to 2012–13, departments were given the time to adjust their capacity in order to prepare to evaluate all direct program spending every five years, commencing April 1, 2013.^{See Footnote 4} In order to be fully compliant with the Policy on Evaluation, departments will need to achieve full evaluation coverage of all direct program spending within five years, namely by the end of 2017–18.

Estimated cumulative coverage of direct program spending over the four years since the introduction of the Policy on Evaluation was 49.4 per cent, with average annual coverage 12.4 per cent (see Table 4).

The level of annual coverage of direct program spending varied noticeably over the phase-in period. Between 2011–12 and 2012–13, there was a decrease of 30 per cent in the level of direct program spending evaluated. Similar increases or decreases were observed in previous years.

Results from the 2012–13 monitoring process also indicate departments' progress over the phase-in period in building capacity to achieve full coverage between 2013–14 and 2017–18. In 2012–13, a total of 97 per cent of departments achieved coverage ratings of "acceptable" or "strong" (see Figure 1) in terms of their monitored performance, which was an increase from 90 per cent of departments in 2011–12 receiving these same ratings.^{See Footnote 5}

Looking ahead, a review of 2013–14 to 2017–18 departmental evaluation plans shows that almost all departments plan to evaluate all of their direct program spending over the five-year period. This is a significant increase from the plans from the previous year, which showed that only 62 per cent of departments projected full coverage. This likely reflects the fact that the full coverage requirement came into force in 2013–14.

Figure 1 – Text version

This bar chart shows the percentage of departments that received each of the four possible assessment ratings for evaluation coverage in 2012–13. 43.8 per cent of large departments and agencies were rated "strong," 53.1 per cent were rated "acceptable," none (0 per cent) were rated "opportunity for improvement," and 3.1 per cent were rated "attention required."

Source: Secretariat Monitoring: Evaluation Coverage – Assessment Results

As in 2011–12, data from 2012–13 continue to suggest that departments have increased the level of coverage of direct program spending that is not covered by the evaluation of ongoing programs of grants and contributions. Further, departments have increased this level of coverage while continuing to evaluate their ongoing programs of grants and contributions as required under section 42.1 of the FAA (see Figure 2):

• The evaluation of direct program spending averaged 12.4 per cent per year over the four-year phase-in period (2009–10 to 2012–13), compared with 7.4 per cent in the year preceding implementation of the 2009 policy.
• The proportion of ongoing program spending of grants and contributions evaluated compared with other direct program spending evaluated fluctuated over time, although it remained comparatively similar between 2011–12 and 2012–13. 
• In 2012–13, non-grant and contribution program spending made up 60 per cent of total direct program spending evaluated; program spending of grants and contributions represented 40 per cent.

Figure 2 – Text version

This bar chart shows the proportion of direct program spending evaluated that is either ongoing grant and contribution spending or non-grant and contribution spending. It shows the change in proportion of each from 2008-09 to 2012-13. It shows that departments have increased the level of coverage of direct program spending that is not covered by the evaluation of ongoing programs of grants and contributions.

Source: Secretariat Monitoring: Departmental Evaluation Plans and Reports

As shown in Table 4, information on the number of evaluation reports approved indicates the following:

• The average annual number of evaluations approved was higher within the four-year phase-in period (2009–10 to 2012–13) than prior to the policy's introduction.
• Over the four-year phase-in period, evaluation reports covered on average an estimated $78 million in direct program spending. This figure represents an increase over pre-implementation levels, where the average level of direct program spending covered was approximately $44 million.
• In 2012–13, each approved evaluation report covered an estimated $86 million in direct program spending. This represents a decrease from 2011–12 figures where on average each evaluation covered approximately $104 million in direct program spending.^{See Footnote 6}

8.1 Summary

Annual reports on the health of the evaluation function have tracked changes in the government-wide evaluation function since the introduction of the current Policy on Evaluation on April 1, 2009.

Despite decreases in a number of metrics between 2011–12 and 2012–13 (such as financial and human resources, and coverage rates), the information contained in this report suggests that federal evaluation functions now demonstrate an ability to cover a greater amount of direct program spending as compared with the period prior to the introduction of the 2009 Policy on Evaluation.

Most large departments and agencies continue to fully meet the requirement set out in the FAA to evaluate all ongoing programs of grants and contributions over five years.

Departments have maintained solid governance of the evaluation function. While the quality of evaluation reports is generally acceptable, opportunities remain to improve reports through enhanced access to performance measurement information, improved rigour of evaluation design, and better linkage of findings to conclusions and recommendations.

Finally, the use of evaluation in support of decision making was assessed to be largely acceptable, with further exploration of the use and impact of evaluation results anticipated in future rounds of monitoring.

8.2 Next steps

Monitoring results from 2012–13 have been taken into consideration as part of the evaluation of the Policy on Evaluation conducted during 2013–14. The evaluation serves to address recommendations in the Spring 2013 Report of the Auditor General of Canada, Chapter 1, "Status Report on Evaluating the Effectiveness of Programs" which called for the Secretariat to review, in consultation with departments, the requirement to evaluate all direct program spending over a five-year cycle and to address all five core issues.

Moving forward, the CEE will:

• Continue to enhance the criteria used to assess the evaluation function. The 2013–14 monitoring framework was modified to focus more heavily on the areas of performance measurement that support the utility and use of evaluation information to inform decision making and the continuous improvement of the evaluation function;
• Continue to enhance the use of evaluation to inform decision making and expenditure management; and
• Consult departments to support their needs for capacity-building initiatives.

Coverage:

• Coverage of ongoing direct program spending.
• Coverage of ongoing programs of grants and contributions.

Governance and Support:

• Governance—A departmental evaluation committee supports the evaluation function.
• Independence—The head of evaluation serves as a strategic advisor to the deputy head.
• Planning and Improvement—There is sufficient capacity to plan and carry out evaluation, monitoring and oversight activities, as required.
• Resources—Adequate resources are allocated to enable the evaluation function to operate effectively to meet the requirements of the Treasury Board's Policy on Evaluation and the evaluation needs of the organization.

Quality:

• Adequacy of evaluation reports and their ability to address the requirements of the Standard on Evaluation for the Government of Canada, including the following:
  • Coverage of issues—Evaluation reports consistently address the five core issues from Annex A of the Treasury Board's Directive on the Evaluation Function (2009) pertaining to relevance and performance.
  • Methodology—Evaluation reports consistently include the following elements to explain the methodology:
    • A clear and appropriate evaluation approach and design are articulated, including a description of how risk was considered in the design;
    • The context is explained by describing the objective and timing of the evaluation, and how it fits into the overall operations of the organization and its rationale;
    • A description of data collection methods;
    • A description of sampling (e.g., rationale for selection, sample size, sample method, limitations of sample); and
    • The appendices include data collection instruments (e.g., surveys, checklists), or the instruments are clearly referenced in the report and are available upon request.

Limitations:

• Evaluation reports explain the constraints and limitations of the evaluation with respect to design, methods and consultations.
• Actual and potential biases in the data and their reliability are explained in terms of the impact on evaluation findings.
• The report explains how the evaluation attempted to overcome the limitations.

Findings and Conclusions:

• Evaluation reports consistently present findings that are substantiated by the analysis of qualitative and quantitative data.
• Evaluation reports represent clear and valid conclusions on relevance and performance that flow logically from the report's central findings.

Recommendations:

• Evaluation reports contain recommendations that flow logically from the evaluation findings and conclusions, that are clearly stated and actionable, and that can be implemented in a practical and realistic to way.

Use:

• Use of evaluation in support of the organization's policy or program improvement and expenditure management, as indicated by:
  • The quality of its management response and action plans; and
  • The level of implementation of the management response and action plans.
• The use of evaluation in support of expenditure management, Cabinet decision making and public reporting, as indicated by:
  • The extent to which the results of evaluation are used to inform Treasury Board submissions, Memoranda to Cabinet, departmental performance reports, and expenditure reviews.
• Dissemination of reports to the Treasury Board of Canada Secretariat.
• Dissemination of reports to Canadians.