2011 Annual Report on the Health of the Evaluation Function
Catalogue No. BT1-20/2011E-PDF
© Her Majesty the Queen in Right of Canada,
represented by the President of the Treasury Board, 2012
This document is available in alternative formats upon request.
Table of Contents
- Message From the Secretary of the Treasury Board
- 1. Introduction
- 2. Thematic Focus: Skilled Evaluators—Supporting Increased
Access to Qualified Evaluators Across the Federal Evaluation
- 3. Tracking Government-Wide Progress in Implementing the 2009
Policy on Evaluation: Current State and Recent Progress
- 4. Leadership Provided by the Treasury Board of Canada Secretariat
- 5. Summary and Conclusions
- Appendix: Methodology and Data Sources
Message From the Secretary of the Treasury Board
Evaluation serves many purposes in the Government of Canada. It can be used by deputy ministers, ministers and parliamentarians to improve government policies and programs, ensure effective and efficient allocation of resources to priorities, and provide assurance to Canadians that they are getting value for money from their tax dollars. Since the renewal of the Policy on Evaluation in April 2009, the federal evaluation function has been shifting its focus and increasing its coverage of federal programs in order to provide more comprehensive support to ministers for spending proposals and expenditure reviews.
Deputy heads of departments and agencies should continue to build the foundation for supporting sound expenditure management and for strengthening their respective Minister's proposals, for example, by:
- Implementing strategies for ongoing performance measurement at the program level;
- Conducting timely evaluations of program relevance, effectiveness, efficiency and economy; and
- Implementing approved action plans from evaluations.
As leader for the federal evaluation function, I want to underscore that investing in broad evaluation coverage is important for supporting accountability and responsible government spending overall. Having information available about the relevance and performance of programs supports decision makers in making informed spending choices.
In achieving broad evaluation coverage, investments in individual evaluations should be "right-sized" to suit decision-making needs. As evaluation coverage builds from year to year, providing a foundation of information to support decision making, departments should actively seek to implement more cost-effective, calibrated evaluation designs. Deputy heads and heads of evaluation alike should note that the Policy on Evaluation enables cost-effective evaluations by giving departments considerable flexibility for customizing the scope and scale of evaluations and for grouping programs together for evaluation purposes.
As the Government of Canada's base of evaluation information grows, I encourage deputy heads to benefit fully from the support that evaluations can provide for decision making by calling upon their departmental heads of evaluation to share strategic insights and knowledge integrated from the full breadth of evaluations.
Secretary of the Treasury Board
Resources for the federal evaluation function were relatively stable from 2009–10 to 2010–11. In 2010–11, annual financial resources for evaluation across all large departments and agencies (LDAs) were approximately $67.4 million, with a median of $1.6 million. As for previous years, salaries represented the largest component of total resources for the function in 2010–11, at 55 per cent. A modest increase of 4.8 per cent over 2009–10 levels brought the number of full-time equivalents (FTEs) to 497 in 2010–11, with the median being 13.4. FTEs include evaluation specialists, support staff and executives.
Grants and Contributions (Gs&Cs) programs
The Financial Administration Act requires that all ongoing programs of Gs&Cs be evaluated every five years. The first five-year period for meeting this legal requirement ends in December 2011. Across all LDAs, evaluation coverage continued to expand between 2009–10 and 2010–11, but the pace of evaluating ongoing Gs&Cs slowed from 33.2 per cent to 7.4 per cent annual coverage. The Treasury Board of Canada Secretariat's monitoring shows that cumulative government-wide coverage of Gs&Cs in the four-year period from April 1, 2007, to March 31, 2011, reached 63 per cent. One year remains in the first five-year period for departments to meet the legal requirement for evaluating all ongoing Gs&Cs.
Other direct program spending
The Policy on Evaluation requires all other types of ongoing direct program spending (i.e., excluding Gs&Cs) to be evaluated every five years beginning in April 2013. Until that time, coverage of other direct program spending remains risk-based and at the discretion of individual deputy heads. Although the pace of expanding Gs&Cs coverage slowed significantly in 2010–11, coverage of other types of ongoing direct program spending (i.e., non-Gs&Cs) grew at a faster rate. Twice the dollar value of non-Gs&Cs direct program spending was evaluated in 2010–11 as in 2009–10 ($3.70 billion compared with $1.83 billion). Importantly, until its transitional period ends on March 31, 2013, the Policy on Evaluation calls for departments to plan their coverage of other direct program spending using a risk-based approach.
Looking at all types of direct program spending (i.e., Gs&Cs and non-Gs&Cs together), a slower pace of expansion in coverage was observed from 2009–10 to 2010–11, with annual increases in coverage measuring 14.2 per cent and 6.7 per cent, respectively. While several factors may explain why slower growth in overall coverage was observed in 2010–11, the Secretariat is taking steps to support further progress on coverage. Based on its continuous monitoring of the function, the Secretariat has preliminary indications that the annual coverage rate will rebound significantly in 2011–12.
Governance and support for the evaluation function
In general, departments have put governance structures in place to ensure neutral evaluation functions. All LDAs have a departmental evaluation committee, of which three quarters are chaired by a deputy head. Most departments have ensured that their head of evaluation has direct and unencumbered access to the deputy head by having the heads of evaluation report to the deputy head either administratively (51 per cent) or functionally (31 per cent).
The level of support to evaluations provided by program-led ongoing performance measurement was similar to that observed in 2009–10. Sixty per cent of LDAs assessed under the Management Accountability Framework (MAF) assessment process had evaluations that usually or almost always cited difficulties in assessing program effectiveness due to insufficient performance data. To compensate for insufficient data, evaluators have to spend additional time and resources to collect the needed data themselves or to implement other approaches.
Quality of evaluations
The quality of evaluation reports remained high overall across LDAs, with 88 per cent of departments receiving quality ratings of "acceptable" or "strong" through the MAF assessment process. These ratings were based on an assessment of the quality of the evaluation methodologies used, whether program value for money was addressed in evaluations and the quality of recommendations, among other criteria.
Use of evaluations
As viewed through the MAF assessment process, evaluation use was "acceptable" or "strong" in 81 per cent of LDAs. Departments reported greater consideration of evaluations in Treasury Board submissions, Departmental Performance Reports and in the Strategic Review process than in Memoranda to Cabinet.
In consultations held in November 2011, departmental heads of evaluation indicated that when working with limited evaluation resources, choices must often be made between the range of uses that each evaluation will support and the overall evaluation coverage of the department's program base that can be achieved.
Although departments use evaluations for internal decision making, Treasury Board of Canada Secretariat analysts use evaluation evidence when examining and providing advice on funding proposals to the Treasury Board for consideration, and on proposals put forward during expenditure reviews such as Strategic Reviews. During Strategic Reviews conducted in 2010–11, the Secretariat noted that departments that had low evaluation coverage were usually among the least effective at explaining program results meaningfully; those that had comprehensive evaluation coverage of their programming generally provided more meaningful input to Strategic Reviews. Although Secretariat analysts indicated that evaluations that had a targeted focus were very useful for informing decisions related to individual Gs&Cs programs, in order to support a variety of expenditure management decisions, effective evaluation coverage required an appropriate balance between high-level and targeted evaluation coverage.
A large majority of LDAs (80 per cent) reported that they have systematic tracking of management action plans arising from evaluation recommendations. Of all the management action plan items that LDAs planned to complete in 2009–10 (1,247 items), 53 per cent had been fully implemented and 36 per cent had been partially implemented.
Leadership provided by the Treasury Board of Canada Secretariat
The Secretariat developed a 2011–12 action plan to outline priority activities for supporting policy implementation and for promoting overall progress in the federal evaluation function. The action plan encompassed the following four broad areas:
- Increasing access to skilled evaluators;
- Developing and sharing effective evaluation approaches;
- Improving the quality and use of evaluations; and
- Enhancing the Secretariat's leadership for the evaluation function within the Government of Canada.
To increase access to skilled evaluators, the Secretariat undertook a series of initiatives. It introduced Leadership Competencies for Federal Heads of Evaluation, putting the Government of Canada at the international forefront of establishing a set of expected skills and abilities for leaders within the evaluation function. Other initiatives included finalizing a Framework for Professional Development for Evaluators and conducting a survey to identify the learning needs of those who work in the federal evaluation function. The Secretariat is currently engaging partners to develop training opportunities that will support government-wide evaluation learning priorities.
The Secretariat continued developing guidance on effective evaluation approaches, such as guidance on the governance of evaluations of horizontal initiatives, on the evaluation of policy functions and programs, on theory-based evaluation approaches, on resource-inclusive evaluations, and on calibrated evaluations. A guide to developing a departmental evaluation plan was completed in late 2010–11, and wide distribution of the guide among departments took place in 2011–12.
The Secretariat used several other means for sharing effective approaches among departments, such as quarterly meetings with heads of evaluation; workshops on leading practices, presentations and panel appearances at conferences; and exchanges through an online forum for members of the federal evaluation community.
The Secretariat continued to promote the quality of evaluations and their use in the Government of Canada by continuously monitoring evaluation quality and use, as well as by advising and supporting other analysts of the Treasury Board of Canada Secretariat who consider evaluation evidence when reviewing departmental proposals to the Treasury Board and to Cabinet. Through the annual MAF assessments of the quality and use of evaluation in departments, the Secretariat continued to identify and communicate areas for improvement in departmental evaluation functions.
To enhance its leadership for the federal evaluation function during 2010–11, the Secretariat built strong engagement from the ADM Champion Committee on Managing for Results, an interdepartmental advisory committee of assistant deputy ministers chaired by the Deputy Assistant Secretary of the Secretariat's Expenditure Management Sector. The committee's mandate is to champion advances in results-based management and evaluation within departments and agencies, and to lead to a better integration of performance information in expenditure management decision making. Since its launch in October 2010, the committee has focused on sharing approaches and good practices for integrated planning and for capitalizing on performance information generated by the evaluation function in relation to the Policy on Management, Resources and Results Structures.
Summary and conclusions
This 2011 annual report documents encouraging signs within the government-wide evaluation function during the first two years of implementing the 2009 Policy on Evaluation, including stable financial and human resources, and progress on the Secretariat's collaborative efforts to increase access to skilled evaluators, established departmental governance structures, and the continuing high quality of evaluation reports. Evidence from consultations with the Secretariat's program sectors, whose analysts use evaluation evidence when providing advice to the Treasury Board, also shows that some departments are successfully designing their evaluations and evaluation coverage to inform key decision-making processes.
This annual report also reveals that although resources for the government-wide function remained relatively stable in 2010–11, coverage of ongoing Gs&Cs and overall coverage of direct program spending accumulated at a slower pace. This report describes several factors that affect coverage that the Secretariat will explore further. The Secretariat's ongoing monitoring in 2011–12 gives preliminary indications that the growth rate of evaluation coverage will increase significantly over 2010–11 levels.
In addition to the positive signs noted above, this report highlights some challenges and areas for further progress in the function, including the following:
- Better implementation of ongoing performance measurement in program areas, including capacity building among program managers, to more effectively support the evaluation function in conducting evaluations;
- Helping departmental heads of evaluation to cost-effectively design evaluations and evaluation coverage, including leveraging previously conducted evaluations, in order to achieve best value with their available evaluation resources while considering key users and uses; and
- Enhancing and accelerating strategic use of evaluation information while balancing needs for targeted evaluations to inform decision making.
1.1 Purpose of this report
The 2011 Annual Report on the Health of the Evaluation Function, the second annual report on this function, provides the Treasury Board and Canadians with information on recent trends in the evaluation function related to infrastructure, governance and support for the function, financial and human resources, evaluation coverage, quality of evaluations, and the use of evaluation in decision making. Starting this year, each annual report will focus on a theme of current importance for the federal evaluation function. This report has the theme of "Skilled evaluators: Supporting increased access to qualified evaluators across the federal evaluation function."
This report serves the following purposes:
- To inform the Government of Canada, parliamentarians and Canadians about how implementation of the Policy on Evaluation is progressing across departments;
- To support deputy heads of departments by identifying areas of improvement for departmental evaluation functions; and
- To assist the Treasury Board of Canada Secretariat in its functional leadership role by identifying what improvements are needed in the government-wide evaluation function in order to meet the policy's objective.
1.2 Information sources used in preparing this report
This report draws from the following sources1:
- The Treasury Board of Canada Secretariat's ongoing monitoring of the function, through a variety of means, including an annual Capacity Assessment Survey of departmental evaluation functions, annual assessments of departmental functions under the MAF assessment process, and regular interactions and engagement with departmental evaluation units;
- A targeted consultation held in November 2011 with departmental heads of evaluation or directors of evaluation, focusing on challenges and opportunities in implementing the Policy on Evaluation;
- A consultation with executives within the Secretariat's program sectors, whose analysts use evaluation evidence when providing advice to the Treasury Board; and
- A learning needs assessment conducted in November and December 2011 among evaluators, evaluation managers, directors and heads of evaluation across departments and agencies.
Since the publication of the 2010 Annual Report on the Health of the Evaluation Function, the methodology used to analyze and present data in this year's annual report has been modified and improved; readers will note some changes in previously reported figures. To establish a more stable basis for future reports, data that have been collected from departments and agencies after the end of each fiscal year (actual data) are used wherever possible, rather than data collected during each fiscal year. In addition, this report focuses on LDAs for which the Secretariat's monitoring provides continuous, reliable data.
1.3 Context of the evaluation function in 2010–11
Key role for evaluation in expenditure management
The Government of Canada's Policy on Evaluation (2009) emphasizes a role for the evaluation function in informing government expenditure decisions by providing neutral and credible evidence about the value for money of federal programs. Accordingly, the evaluation function has begun to build a base of evidence on the relevance, effectiveness, efficiency and economy of government programs.
Time frames for meeting legal and policy requirements for evaluation
The Federal Accountability Act of 2006 amended the Financial Administration Act to require that all ongoing programs of Gs&Cs be evaluated every five years. The first five-year period for meeting the legal requirement to evaluate all of these programs ends in December 2011.
Key coverage requirements of the 2009 Policy on Evaluation:
- All direct program spending (excluding grants and contributions) is evaluated every five years (implementation will begin in April 2013).
- All ongoing programs of grants and contributions are evaluated every five years, as required by section 42.1 of the Financial Administration Act (implementation began in December 2006).
- The administrative aspect of major statutory spending is evaluated every five years (implementation began in April 2009).
When it was renewed in April 2009, the Policy on Evaluation expanded coverage requirements by calling for all ongoing direct program spending to be evaluated every five years.
With this expansion of coverage requirements, and based on the advice of an advisory committee of deputy heads2 in 2008, a four-year transitional period was adopted to give departments adequate time to build their evaluation capacity. In all, departments have nine years to ramp up their capacity. The first five-year period over which LDAs will be required to achieve full coverage of ongoing direct program spending is 2013–14 to 2017–18.
During the four-year transition period specified by the policy, decisions on planning evaluation coverage of direct program spending (other than program spending on ongoing Gs&Cs) remain risk-based and at the discretion of individual deputy heads. It is important to note that the policy's transition period does not apply to programs that must be evaluated every five years in accordance with the legal requirement.
Figure 1 outlines the different time frames for meeting legal and policy requirements for evaluation.
At the end of 2010–11, halfway into the policy's four-year transition period for adjusting to expanded coverage requirements, departments and agencies are adapting their evaluation practices and making progress toward meeting the policy's objective.
During this transition phase, the Secretariat observed that some departments have directed more resources toward their evaluation functions to support the evaluation of more of their direct program spending. Some departments have begun to use more cost-effective evaluation designs, such as grouping direct program spending for evaluation purposes.
2. Thematic Focus: Skilled Evaluators—Supporting Increased Access to Qualified Evaluators Across the Federal Evaluation Function
2.1 Initiatives to address a key challenge identified in 2010
In last year's report on the health of the evaluation function, hiring skilled evaluators was identified as a major government-wide challenge because too few were available. This finding was based on observations from deputy heads and departmental heads of evaluation.
In response, the Treasury Board of Canada Secretariat's Centre of Excellence for Evaluation (CEE) targeted this challenge as a major area of emphasis for its own action plan. To support departmental access to skilled evaluators, the CEE undertook efforts in several areas during 2010–11 and beyond.
Major initiatives are described in the following sections.
Leadership competencies for federal heads of evaluation
In September 2011, after extensive consultation with departments and agencies, the Secretary of the Treasury Board shared the Leadership Competencies for Federal Heads of Evaluation. The introduction of this set of expected skills and abilities puts the Government of Canada at the international forefront of establishing evaluation competencies.
These leadership competencies describe the behaviours that heads of evaluation need to demonstrate in their roles as strategic advisors to deputy heads and primary evaluation experts within their respective organizations. Major competency categories covered include evaluation expertise, values and ethics, strategic thinking, engagement, and management excellence.
The leadership competencies are intended to serve as a tool for deputy heads in selecting heads of evaluation, in reviewing their performance, and in assessing and addressing their training needs.
Federal heads of evaluation in LDAs are responsible for meeting these competency requirements by March 31, 2013.
The Secretariat's Framework for Professional Development for Evaluators, and its component elements that are under development, will provide support to heads of evaluation in their efforts to meet competency requirements.
Framework for Professional Development for Evaluators
As part of its ongoing efforts to provide leadership in advancing evaluation practices across the federal government, the Secretariat's CEE began developing a framework for professional development to support heads of evaluation in identifying learning priorities and professional development opportunities to meet both their own needs and the needs of their evaluators and evaluation managers. The framework was launched in early 2011–12, and development work continues on the individual elements of the framework.
When consulted in fall 2011, heads of evaluation in some departments indicated that a lack of skilled evaluators remained a challenge for their evaluation functions. One approach to addressing this gap is to take steps to increase opportunities for professional development for those who work in evaluation functions across the Government of Canada. Heads of evaluation expressed a need for more learning opportunities for evaluators at all levels.
To identify the variety of relevant courses already available to evaluators through Canadian universities, the Secretariat asked the Consortium of Universities for Evaluation Education (CUEE) to develop an inventory of evaluation-related courses offered by its member universities. The CUEE also mapped the relevant courses in the inventory to the Secretariat's draft Competency Profile for Federal Public Service Evaluation Professionals. In addition, the Secretariat worked with the Canada School of Public Service to map its relevant courses to the Competency Profile for Federal Public Service Evaluation Professionals in the same manner. The Secretariat is currently using these two mappings to identify gaps where more learning opportunities need to be developed for federal evaluators in 2012–13 and beyond.
Online survey of learning needs: Highlights of results
To identify evaluators' learning priorities, in November and December 2011 the Secretariat conducted an online survey of those working in the federal evaluation function. Evaluators were asked to provide information about their own learning needs, and evaluation managers and departmental heads of evaluation were asked to provide information about their needs as well as the needs of their evaluation staff.
Survey results showed that evaluators and evaluation managers are generally highly educated, with almost three-quarters (73 per cent) holding a master's degree or Ph.D. Nevertheless, more than half of survey respondents (55 per cent) indicated that they have unmet training and professional development needs that are related to performing their current evaluation duties, and 68 per cent indicated that they have unmet needs related to their career advancement in the field of evaluation. This finding is perhaps not surprising, given that the majority of survey respondents (almost 55 per cent) indicated that they have five or fewer years of experience in the field of evaluation. Figure 2 illustrates the number of years that evaluators surveyed had been working in the field of evaluation. At the same time, a significant proportion of respondents (38 per cent) have worked in the evaluation field between 6 and 15 years, and 7 per cent have been in the field for 16 to 25 years.
When asked to identify their main sources of evaluation-related training, half of the respondents indicated self-study, for example, reading, online research and hands-on practice. Notwithstanding the prevalence of self-study, the vast majority of evaluators and evaluation managers have also received evaluation-related training while in the federal government (87 per cent), with most of this training received within the last two years (79 per cent of those who received training). The most common sources of evaluation training received by survey respondents were the Canadian Evaluation Society (44 per cent), other sources such as The Evaluators' Institute in Washington, D.C. (25 per cent), universities or colleges (16 per cent), the American Evaluation Association (8 per cent), their own Government of Canada department or agency (4 per cent), and the Canada School of Public Service (2 per cent).
However, the key challenges noted by most respondents were that the level of training available was often too basic for their needs (60 per cent), that training was too expensive (51 per cent), that they were unaware of the availability of training (40 per cent), or that they did not have the time to attend training (38 per cent). These findings suggest that in order to support its Framework for Professional Development for Evaluators, the Secretariat and its learning partners should consider pursuing the development of affordable advanced training in specialized subject areas.
When asked to identify topics that should be covered in training offerings, survey respondents indicated a need for training to orient new federal evaluators to the Government of Canada context for the evaluation function.
Although evaluators and evaluation managers were interested in a range of other topic areas, when asked to select their top learning priorities from a list of topics, they most frequently chose the following areas:
- Designing evaluations and identifying evaluation questions;
- Quantitative analysis and interpretation;
- Economic analysis (cost-effectiveness and cost-benefit); and
- Qualitative data analysis and interpretation.
Respondents expressed their openness to a wide variety of professional development approaches and formats, including hands-on training, small-group interactive case-based studies, workshops, mentoring or peer coaching, and in-class instruction.
Following a full analysis of the online survey data, the Secretariat will further consult with departmental heads of evaluation and partners involved in developing training opportunities and consider actions it could take to support government-wide evaluation learning priorities.
Facilitating access to external evaluators
In recognition of departments' need for better access to external evaluators, the CEE engaged with departments and with Public Works and Government Services Canada (PWGSC) in developing procurement approaches and tools for evaluation and performance measurement services. Throughout this process, the CEE acted as a technical advisor to PWGSC with the objective of ensuring that the needs of the federal evaluation community would be understood and reflected in the procurement tools. Under the umbrella of its Professional Services National Procurement Strategy, PWGSC consulted with departments and suppliers on the development of standing offers and supply arrangements for evaluation and performance measurement services, and ultimately solicited proposals from suppliers in July and August 2011. Near the end of 2011, PWGSC was in the latter stages of establishing these standing offers and supply arrangements.
3. Tracking Government-Wide Progress in Implementing the 2009 Policy on Evaluation: Current State and Recent Progress
3.1 Financial resources
As shown in Table 1, financial resources for the federal evaluation function increased over several years but began to stabilize in 2010–11. In that fiscal year, annual resources for evaluation across all LDAs were approximately $67.4 million, representing a slight increase of 0.9 per cent from the previous year. As for previous years, salaries represented the largest component of total resources for the function in 2010–11, at 55 per cent. Although resources for salaries and operating and maintenance grew relative to 2009–10 levels, those for professional services and other expenditures decreased.
In 2010–11, the median amount that LDAs devoted to their evaluation functions was $1.6 million. Across all LDAs, resources spent on evaluation represented less than one tenth of 1 per cent of direct program spending reported in the Main Estimates.
|2010–11table 1 note *
Table 1 Notes
Totals may not add due to rounding. Includes organizations defined as LDAs under the Policy on Evaluation, as determined each fiscal year. The list of LDAs may vary slightly from one year to the next.
|Salarytable 1 note †||28.4||30.8||33.9||37.3|
Servicestable note 2 †
Maintenance (O&M)table 1 note †
|Othertable 1 note ‡||6.7||8.8||13.7||11.8|
|% Annual Increase||N/A||6.3||9.7||0.9|
3.2 Human resources
Like financial resources for the evaluation function, human resources for this function have been increasing over several years. As shown in Table 2, a modest increase of 4.8 per cent over 2009–10 levels brought the number of FTEs in LDAs to 497 in 2010–11. The median number of FTEs for LDAs in 2010–11 was 13.4.
|2007–08||2008–09||2009–10||2010–11table 2 note *|
Table 2 Notes
FTEs shown represent combined A-based and B-based resources, reported by departments after the end of each fiscal year.
Includes organizations defined as LDAs under the Policy on Evaluation, as determined each fiscal year. The list of LDAs may vary slightly from one year to the next.
|Full-Time Equivalents (FTEs)||409||418||474||497|
|% Annual Increase||N/A||2.2||13.4||4.9|
In last year's report, a key challenge identified by deputy heads and departmental heads of evaluation was finding skilled evaluators. Although this challenge still exists, it remains a greater challenge for some departments than for others. During the Secretariat's consultations with heads of evaluation, many noted that compared with previous years, there has been less movement of staff in and out of their evaluation units, perhaps due to the uncertain economic context. In such a context, evaluation units that already possess sufficient expertise may have little need to find new evaluators, while those units that lack expertise may find it difficult to staff evaluator positions. The need to provide training to existing staff therefore remains important. As reported in the 2010 Annual Report on the Health of the Evaluation Function, the CEE within the Treasury Board of Canada Secretariat targeted this as an area for providing support to the government-wide function. Section 2 of this report provides more information on the learning needs assessment conducted by the Secretariat and on its Framework for Professional Development of Evaluators.
3.3 Contracted resources
In 2010–11, 73 per cent (99 of 136) of evaluations conducted by LDAs involved the use of contractors for at least some part of the work, whereas the remaining 27 per cent did not use contractors at all. This represents a slight decline in the prevalence of using contracted resources for the conduct of evaluations when compared with the 75 per cent of evaluations that involved contractors in 2009–103. The total contracted cost for evaluation work in LDAs in 2010–11 was $13.8 million, which represents 20 per cent of all evaluation resources expended in 2010–11. In comparison, the total contracted cost of evaluation services in LDAs in 2009–10 was $20.3 million, or 30 per cent of all evaluation resources expended in that year.
3.4.1. Coverage of ongoing Gs&Cs
Across LDAs, the rate of evaluating ongoing Gs&Cs slowed from 33.2 per cent incremental coverage in 2009–10 to 7.4 per cent in 2010–11 (see Table 3). On average, departments with ongoing Gs&Cs programs need to annually cover 20 per cent of this type of program in order to achieve 100 per cent coverage over five years. Over the four-year period from April 1, 2007, to March 31, 2011, the average annual coverage of Gs&Cs was about 16 per cent. The Secretariat's monitoring of LDAs shows that notional cumulative government-wide coverage in the same four-year period reached 63 per cent.
Because the first five-year period for meeting the legal requirement to evaluate all Gs&Cs ends three quarters into the 2011–12 fiscal year, i.e., in December 2011, cumulative five-year coverage of Gs&Cs will be reported in the 2012 Annual Report on the Health of the Evaluation Function.
3.4.2. Overall coverage of direct program spending
Although growth in Gs&Cs coverage slowed significantly in LDAs, the annual rate of coverage of other types of direct program spending increased from 2009–10 levels. As shown in Figure 3, more than twice the dollar value of non-Gs&Cs direct program spending was evaluated in 2010–11 compared with 2009–10: $3.70 billion versus $1.83 billion.
However, overall coverage of all types of direct program spending together (Gs&Cs and non-Gs&Cs), grew more slowly from 2009–10 to 2010–11, with annual coverage rates of 14.2 per cent and 6.7 per cent, respectively. Notional cumulative coverage over the two years since the introduction of the Policy on Evaluation in April 2009 was 20 per cent, and average annual coverage was 10.1 per cent. Looking ahead to future years, 62 per cent of LDAs submitted departmental evaluation plans to the Secretariat that project full coverage of all types of ongoing direct program spending over the five-year period from 2011–12 to 2015–16. This is despite the fact that the first five-year period over which LDAs will be required to achieve full coverage of this spending is from 2013–14 to 2017–18.
Spendingtable 3 note *
Spendingtable 3 note †
Table 3 Notes
|2009–10table 3 note ‡||164||11,999||84,665||14.2||10,167||30,605||33.2|
|2010–11table 3 note ‡||136||6,607||99,325||6.7||2,903||39,145||7.4|
3.4.3. Explaining changes in coverage
The slowdown in coverage from 2009–10 to 2010–11 is likely due to several factors. First, total direct program spending requiring evaluation increased from $84.67 billion in 2009–10 to $99.33 billion in 2010–11.
The Secretariat's monitoring also shows that several LDAs had a number of evaluations, planned for completion during 2010–11, that were carried over into 2011–12. One explanation is that some departments are designing, on average, larger and more complex evaluations that take more time to complete but achieve greater evaluation coverage per dollar spent on evaluation.
For some departments, the alignment of departmental evaluation plans with the renewal schedule of major spending authorities may also have led to slower growth in evaluation coverage during 2010–11.
Finally, because departments are not required to achieve comprehensive coverage of these other types of direct program spending during the policy's transition period, some have planned their coverage using a risk-based approach. Further, instead of conducting only those evaluations that would count for coverage under the policy, some departments opted to undertake evaluation activities in 2010–11 that are not required by the policy but that are nonetheless of value to their individual departments4.
In 2010–11, there was an increase in the number and total cost of evaluation products submitted to the Secretariat that did not count in its evaluation coverage calculations (see Figure 4). The number of evaluation products that did not count for coverage increased by 33 per cent between 2009–10 and 2010–11, and the total funds devoted to producing such evaluation products increased by more than two times, from $2.5 million to $5.5 million.
The Secretariat will continue its monitoring of the function to better understand other factors, such as capacity issues, that may explain variations in annual evaluation coverage. Preliminary indications from the Secretariat's monitoring activities are that annual coverage rates will increase significantly in 2011–12.
For an overall perspective of 2010–11 MAF ratings for evaluation coverage, see Figure 5.
3.5 Governance and support
The Policy on Evaluation requires departments to put governance structures in place to ensure a neutral evaluation function. All LDAs have a departmental evaluation committee (DEC). In 2010–11, 26 of 35 were chaired by a deputy head (74 per cent). On average across 34 LDAs, DECs met on evaluation-related issues five times during 2009–10. Twenty of 35 DECs (57 per cent) serve only the evaluation function.
The Policy on Evaluation requires that heads of evaluation have direct and unencumbered access to the deputy head of their organization. In 2010–11, 18 of 35 (51 per cent) of heads of evaluation reported administratively to the deputy head; another 11 of 35 (31 per cent) reported functionally to the deputy head but administratively to another executive, and the remaining 17 per cent reported functionally and administratively to another senior executive.
Most heads of evaluation (26 of 35, or 74 per cent) fulfill more than one role within their respective organizations. For example, 18 of 35 (51 per cent) of heads of evaluation also fulfill the role of chief audit executive. Other common roles for heads of evaluation relate to leading performance measurement and risk management within their departments, among others.
Thirty-one of 35 departmental functions (88.6 per cent) are co-located with another function, for example, internal audit, performance measurement, strategic planning or finance.
Figure 6 illustrates 2010–11 MAF ratings for governance and support.
3.5.1 Performance measurement support to the evaluation function
The collection of performance data is necessary so that evaluators can perform the efficiency and economy analysis required by one of the core evaluation issues established by the Directive on the Evaluation Function, as well as the analysis of progress toward achieving expected program outcomes as required by another core evaluation issue.
Under the 2010–11 MAF assessment process, 18 of 30 of LDAs (60 per cent) had evaluations that usually or almost always cited difficulties in assessing program effectiveness due to insufficient performance data.
In consultations held in fall 2011, heads of evaluation also expressed concerns about the lack of quality performance measurement data within their departments and the need to build capacity in program areas for designing and implementing performance measurement. They indicated that improving performance measurement would take continuous effort from departments for the foreseeable future. Furthermore, some heads of evaluation highlighted the extent of time and effort that it sometimes takes for the evaluation unit to support program areas with advice on performance measurement and to monitor its implementation. Several heads of evaluation expected that challenges related to ongoing performance measurement would continue to leave gaps in the data required by evaluations and that evaluation units would consequently need to continue to compensate by devoting resources to primary data collection.
For MAF assessments conducted in 2010–11, the Secretariat applied very similar criteria to those used the year before for assessing the quality of evaluation reports submitted by departments. The criteria included the following:
- Addressing value-for-money issues pertaining to program relevance, effectiveness, efficiency and economy;
- The quality of evaluation methodology;
- Reporting of limitations affecting the conduct of evaluations and the impact of limitations on evaluation findings;
- The quality and substantiation of evaluation findings and conclusions;
- The quality of recommendations; and
- The quality of the management response and action plan.
Report quality was high overall across departments, with 88 per cent of departments receiving quality ratings of "acceptable" or "strong" (see Figure 7).
Departments were asked by the Secretariat to report how, and to what extent, evaluation results were used in 2010–11. LDAs5 reported the following frequencies of use (see Figure 8):
- 88.6 per cent of LDAs indicated that almost all ("almost all" was defined as 80 per cent or more) evaluations had their findings considered in Treasury Board submissions during 2010–11; another 2.8 per cent indicated that several evaluations had their findings considered ("several" was defined as 50–79 per cent).
- 40 per cent of LDAs said that almost all evaluations had their findings considered in Memoranda to Cabinet during 2010–11; another 17.1 per cent indicated that several evaluations had their findings considered.
- 82.9 per cent of LDAs indicated that almost all evaluations had their findings considered in the Departmental Performance Report during 2010–11; another 8.6 per cent indicated that several evaluations had their findings considered.
- 75 per cent of LDAs indicated that almost all evaluations had their findings considered in their 2010–11 Strategic Review; another 2.8 per cent indicated that several evaluations had their findings considered.
To assess how well departments used evaluation results during 2010–11, the Secretariat used MAF criteria that were similar to those in the previous year, including the following:
- The extent to which results of relevant evaluations are brought for consideration in Treasury Board submissions, Memoranda to Cabinet, and the organization's Report on Priorities and Planning and Departmental Performance Report;
- The extent of relevant evaluation results available to be brought for consideration in the organization's expenditure reviews;
- Systematic tracking of management action plans arising from evaluation and regular reporting on the status of the implementation of the evaluation recommendations;
- The extent to which management responses and action plans are implemented as planned; and
- The extent to which completed evaluation reports are submitted to the Secretariat as well as posted on the departmental website in a timely manner.
With respect to the implementation of recommendations from evaluations: 80 per cent of LDAs reported that they have systematic and regular tracking of follow-up to management action plans that arise from evaluation recommendations. In addition, of all the management action plan items that LDAs had scheduled for completion in 2009–10 (1,247 items), 53 per cent had been fully implemented, 36 per cent had been partially implemented, 4 per cent had not yet started, 2 per cent had been determined to be obsolete, and 5 per cent were of unknown status.
Overall, the use of evaluations was highly rated across departments, with 81 per cent of departments receiving MAF ratings of "acceptable" or "strong" (see Figure 9).
Trying to deliver useful evaluations while achieving coverage and meeting core issue requirements with limited resources is a complex challenge. Where departmental evaluation functions are working with limited resources, trade-offs must often be made between the range of uses each evaluation will support and the overall evaluation coverage that can be achieved across the department's program base. This challenge is further complicated by the variety of uses and users of evaluation products.
Informed by its regular engagement with departmental heads of evaluation, the Secretariat expects these tensions to ease after most programs have been evaluated at least once, such that the foundation of existing evaluations can be used for designing subsequent evaluations that are more cost-effective and better targeted.
Where it is available, analysts of the Treasury Board of Canada Secretariat's program sectors use evaluation evidence when examining and providing advice on funding proposals to the Treasury Board for consideration and on departmental proposals put forward during expenditure reviews such as Strategic Reviews (e.g., for helping to identify low-performing, low-priority program areas).
During Strategic Reviews conducted in 2010–11, it was noted in some cases that departments that had low evaluation coverage were among the least effective at explaining their program results meaningfully. In addition, although comprehensive evaluation coverage generally resulted in more meaningful input to Strategic Reviews, the most effective support for Strategic Reviews was delivered by evaluations that, either individually or in combination, explained the overall achievements of departmental programs that were targeting similar intended results.
To support other types of uses, such as decisions on changes to individual Gs&Cs programs, evaluations that had a more targeted focus were considered very useful.
Consequently, from the Secretariat's perspective, in order to support a variety of expenditure management decisions, effective evaluation coverage means striking an appropriate balance between high-level and targeted evaluation coverage (e.g., through choices related to evaluating programs one at a time, covering groups of programs all at once through an individual evaluation, or planning and conducting targeted evaluations of smaller programs in a manner that enables subsequent aggregation of evaluation findings so that an overall perspective on the performance of two or more related programs can be constructed).
4. Leadership Provided by the Treasury Board of Canada Secretariat
4.1 2011–12 Action Plan
During 2010–11, the Secretariat began to adjust its oversight and support activities for the federal evaluation function based on insights gained through the following:
- Monitoring data collected through an annual Capacity Assessment Survey of all organizational evaluation functions and through annual MAF assessments of the quality and use of evaluation in departments;
- The 2010 Annual Report on the Health of the Evaluation Function;
- Consultations held with deputy heads and heads of evaluation during 2010–11; and
- Recommendations from the Standing Committee on Public Accounts in its September 2010 report.
To guide its support efforts through 2011–12, the Secretariat developed an action plan to outline priority activities for furthering policy implementation in departments and for promoting overall progress in the federal evaluation function.
The action plan encompassed four broad areas:
- Increasing access to skilled evaluators;
- Developing and sharing effective evaluation approaches;
- Improving quality and use of evaluation; and
- Enhancing the Secretariat's leadership for the evaluation function in the Government of Canada.
The following describes the key initiatives that were completed or launched in each of these areas in 2010–11 and makes note of some activities that are underway in 2011–12.
Initiatives related to increasing access to skilled evaluators were previously discussed in Section 2, "Thematic Focus: Skilled Evaluators—Supporting Increased Access to Qualified Evaluators Across the Federal Evaluation Function."
4.1.1 Developing and sharing effective evaluation approaches
The Secretariat completed its guide to developing a departmental evaluation plan in late 2010–11 and distributed it widely to departments in 2011–12.
Throughout 2010–11, the Secretariat also continued its ongoing development of guidance on effective evaluation approaches. Specifically, the Secretariat undertook the following:
- Led collaborative interdepartmental working groups to develop guidance on the governance of evaluations of horizontal initiatives, guidance on the evaluation of policy functions and programs, and when exploring possible approaches for undertaking calibrated evaluations;
- Developed preliminary guidance on theory-based evaluation approaches and engaged a few departments in undertaking pilots for informing further refinements to the guidance; and
- Developed draft guidance on addressing program resource utilization in federal evaluations and sought departmental feedback on the draft materials.
As each piece of guidance on evaluation approaches is completed over 2011–12 and beyond, the Secretariat will share it widely with departments and agencies.
The Secretariat used several means for sharing effective approaches among departments in 2010–11 and into 2011–12:
- Holding quarterly meetings with heads of evaluation and directors of evaluation to discuss a variety of evaluation issues;
- Holding workshops on leading practices in evaluation;
- Presenting at conferences and participating in panel discussions about evaluation approaches;
- Hosting an online community forum for departmental heads of evaluation;
- Launching an online community forum for federal evaluators; and
- Undertaking targeted outreach activities in various departments.
4.1.2 Improving quality and use of evaluation
The Secretariat continuously monitors evaluation quality and use. In 2010–11, through annual MAF assessments of the quality and use of evaluation in departments, the Secretariat continued to identify and communicate areas for improvement in departmental evaluation functions.
With a view to enhancing its monitoring of evaluation use, and ultimately to promote further use of evaluation in decision making, the Secretariat undertook some preliminary research to better understand the extent and way in which evaluation is currently used in departments, including its use in central agency processes such as Treasury Board submissions, Memoranda to Cabinet and Strategic Reviews. The study examined different categories of use and success factors for the use of evaluation and will serve to enhance methods employed for assessing evaluation use through the MAF process. The research will continue during 2011–12 and beyond.
4.1.3. Enhancing the Secretariat's leadership for the evaluation function in the Government of Canada
Engagement with senior managers in departments and agencies
The ADM Champion Committee on Managing for Results, an interdepartmental advisory committee of assistant deputy ministers chaired by the Deputy Assistant Secretary of the Expenditure Management Sector of the Treasury Board of Canada Secretariat, was launched in October 2010 to champion advances in results-based management and evaluation within departments and agencies, and to lead to a better integration of performance information in expenditure management decision making. Since its launch, the committee has focused on sharing approaches and good practices for integrated planning and for capitalizing on performance information generated by the evaluation function and in relation to the Policy on Management, Resources and Results Structures.
The Deputy Assistant Secretary of the Expenditure Management Sector of the Secretariat also met with departmental evaluation committees in individual departments, on request, to discuss evaluation within the context of their specific organizations.
5. Summary and Conclusions
5.1 Progress in implementing the Policy on Evaluation
This report documents some encouraging signs within the federal evaluation function during the first two years of implementing the 2009 Policy on Evaluation. Overall, financial and human resources within the function are relatively stable, showing slight increases between 2009–10 and 2010–11. With respect to human resources, progress has been made on one of the key challenges for the function that was identified in the 2010 report, as the Secretariat has undertaken several initiatives to support increased access to skilled evaluators.
Broadly speaking, departments have established the necessary governance structures for their evaluation functions, and the quality of evaluation reports remained in the "acceptable" to "strong" range in 2010–11. Consultations with Treasury Board of Canada Secretariat's program sectors, whose analysts consider evaluation evidence in their review of departmental proposals to the Treasury Board, have indicated that some departments are successfully designing their evaluations and evaluation coverage to inform key decision-making processes.
In addition to these signs of progress, this report finds that although resources for the government-wide function remained relatively stable in 2010–11, there were uneven results related to evaluation coverage. The pace of growth in overall evaluation coverage (i.e., combined coverage of Gs&Cs and non-Gs&Cs) decreased. Although growth in coverage of Gs&Cs slowed, coverage of other types of direct program spending increased by a factor of two. Several factors that explain the slowdown in growth of overall evaluation coverage have been discussed. The Secretariat will further assess the reasons for this situation and take action as necessary to support further progress on coverage.
This report also documents that departments are using evaluations in various applications to meet the needs of a variety of users.
5.2 Needed improvements and their challenges for the evaluation function
Determining the causes for the changes in evaluation coverage will be important in order to fully understanding all the challenges for the federal evaluation function in the next few years.
However, some challenges and areas for further progress in the evaluation function are already evident:
- Continuing to improve implementation of ongoing performance measurement in program areas, including capacity building among program managers, to more effectively support the function in conducting evaluations;
- Helping departmental heads of evaluation cost-effectively design evaluations and increase evaluation coverage, including through leveraging previously conducted evaluations, in order to achieve best value with their available evaluation resources while considering key users and uses; and
- Enhancing and accelerating strategic use of evaluation information while balancing needs for targeted evaluations to inform decision making.
The Secretariat's support for government-wide improvement: Looking ahead to 2011–12 and beyond
The Secretariat will provide additional guidance and supports (e.g., workshops on conducting resource-inclusive evaluations) in 2011–12 and 2012–13 to advance evaluation practices within the function, to provide departments with alternative evaluation approaches for consideration and to help them calibrate their evaluations. The Secretariat's CEE will continue its efforts to encourage the sharing of effective practices among departments through regular meetings of heads of evaluation, online forums for heads of evaluation and evaluators, and other channels.
To help departments and users achieve greater utility from their evaluations for a wider range of uses, the CEE will work collaboratively with departments and with the Secretariat's program sectors to improve the flow of information about needs and uses for evaluation. After early discussions with the Secretariat's program sectors, the CEE expects to promote a practice of holding bilateral meetings between the Secretariat's representatives and departments to review and discuss departmental evaluation plans as they are being drafted.
The Secretariat will continue to work with training partners to develop professional development opportunities that support government-wide learning priorities identified through a survey of federal evaluators and consultations with heads of evaluation.
The CEE has been actively collaborating with other policy centres of the Secretariat, notably the policy centre for the Policy on Management, Resources and Results Structures, as it leads the development of integrated guidance for departments on performance measurement and other capacity-building measures targeted primarily to program managers.
Appendix: Methodology and Data Sources
Methodology notes and a description of the primary data sources used to prepare this report are included in this Appendix.
Since the 2010 Annual Report on the Health of the Evaluation Function, some modifications have been made to the methodology used to analyze and present the data in this annual report, with a view to establishing a more solid basis for future reporting. To the extent possible, when reporting aggregate data that have been collected from departments and agencies, notably on financial and human resources within the evaluation function, this report uses data that have been collected after the end of each fiscal year (actual data) rather than data collected during each fiscal year (planned data). In addition, information presented in this report focuses primarily on large departments and agencies (LDAs), as the Secretariat's monitoring provides continuous, reliable data about LDAs.
Information presented in this report also reflects improvements made to the MAF assessment methodology for calculating the rate of evaluation coverage in departments.
In departments and agencies that are subject to the Policy on Evaluation, the Treasury Board of Canada Secretariat monitors evaluations that are led by the head of evaluation of the organization and that are intended to count toward evaluation coverage under the policy. Departments may choose to undertake other evaluations and evaluation-related work intended to meet departmental needs but that does not reflect the coverage requirements of the Policy on Evaluation. Key examples of departmental evaluations that would not be counted in the Secretariat's coverage calculations include the following:
- Evaluations of time-limited programs, which do not require evaluation under the policy unless requested by the Secretary of the Treasury Board;
- Evaluations of programs that have already been evaluated during the previous five-year period (also includes meta-evaluations, i.e., syntheses of evaluations);
- Evaluations that do not address all of the core issues required by the policy, for example, formative evaluations that do not examine program effectiveness; and
- Evaluation-related products such as evaluability assessments or "lessons learned" studies, which are usually created by analyzing several evaluations in a given program area or of a specific program type.
Evaluations and related work that fall outside the requirements of the policy are not reflected in this report, except in section 3.4.3.
Capacity Assessment Survey
The Capacity Assessment Survey (CAS) is an annual survey administered to evaluation units in all departments and agencies that are subject to the Policy on Evaluation and collects information on a fiscal-year basis. Data from 2011 and earlier years of the CAS have been used in this report for comparative purposes. Since 2007, this survey has been administered as part of the broader Management Accountability Framework (MAF) assessment process through which departments are assessed annually on the quality and use of evaluations.
All departments and agencies that are subject to the Policy on Evaluation are asked to complete the CAS; however, some non-response occurs among small departments and agencies (SDAs). Because the annual survey is administered as part of the MAF assessment process, some SDAs that are not subject to MAF assessment in a given year might choose not to complete the survey. In light of SDAs' lower amounts of direct program spending, the deferral of most of the requirements of the Policy on Evaluation as they apply to SDAs and the generalized lower evaluation capacity within these organizations, the expected level of evaluation activity within SDAs is low, and the Secretariat's monitoring of the evaluation function within those departments is scaled to match. With the introduction of the annual report on the health of the evaluation function, the Secretariat is taking steps to improve SDA response rates to the CAS, without placing undue burden on these organizations.
MAF assessment of evaluation quality and use
The MAF is a key performance management tool used by the federal government. Its purpose is to support management accountability of deputy heads and improve management practices across departments and agencies. More specifically, the objectives of the MAF are as follows:
- Clarify management expectations for deputy heads and inform ongoing dialogue on management priorities;
- Provide a comprehensive and integrated perspective on the state of management practices and challenges; and
- Inform the design of risk-based approaches that provide greater delegation of authority for departments and agencies that have strong management performance.
The MAF summarizes the vision behind various management reforms into 10 high-level management expectations of each deputy head. Additional information about the MAF, including published assessments, is available on the website of the Treasury Board of Canada Secretariat.
The MAF assessment process is performed annually by the Secretariat and is based on evidence submitted by departments and agencies. All LDAs and about a third of SDAs are assessed each year, which represents 55 to 60 departments and agencies annually. Each department and agency is assessed against specific criteria outlined under various areas of management.
One area of management under the MAF is evaluation quality and use. In 2010–11, four lines of evidence were used to assess management performance in this area of management:
- Quality of evaluation reports;
- Governance and support for the evaluation function;
- Evaluation coverage of the organization's direct program spending; and
- Use of evaluation to support decision making and reporting.
The MAF assessment results used in this report for LDAs are from 2010 (Round VIII). LDAs are effectively defined by exception according to the definition of SDAs under the Policy on Evaluation. Under the policy, SDAs are defined as having less than $300 million in their Annual Reference Level and revenues credited to the Vote.
Consultation with heads of evaluation
On November 9 and 10, 2011, the Secretariat's Centre of Excellence for Evaluation (CEE) moderated two informal half-day discussions with heads of evaluation or directors of evaluation representing 32 departments and agencies. All departments and agencies subject to the Policy on Evaluation were invited to participate. Participants were asked to provide their perspectives on the main challenges and opportunities for the evaluation function as well as their outlook on the future for the function.
Consultation with users of evaluation information at the Treasury Board of Canada Secretariat
On January 18, 2012, the CEE consulted executives from the Secretariat to obtain their perspectives on how evaluations were being used; what changes they had observed in evaluation in the last couple of years, such as changes in coverage, quality and utility; and what challenges and opportunities existed for broadening the use of evaluation.
Online survey of evaluators' learning needs
An online survey of evaluators, evaluation managers, directors of evaluation and heads of evaluation was conducted between November 24 and December 16, 2011. Through an invitation forwarded by directors and heads of evaluation in departments and agencies, federal evaluators and evaluation managers were invited to participate in the online survey, which was housed on the Secretariat's site. The estimated response rate to the survey was 42 per cent.
Report a problem or mistake on this page
- Date modified: