Study of the Evaluation Function In the Federal Government
Notice to the reader
The Policy on Results came into effect on July 1, 2016, and it replaced the Policy on Evaluation and its instruments.
Since 2016, the Centre of Excellence for Evaluation has been replaced by the Results Division.
For more information on evaluations and related topics, please visit the Evaluation section of the Treasury Board of Canada Secretariat website.
Prepared by the Centre of Excellence for Evaluation
April 2004
Table of Contents
Prior to the introduction of the revised Evaluation Policy, the Treasury Board Secretariat conducted a review of the function to determine the state of evaluation practice within the Government of Canada. The study is based on an evaluation community survey, strategic interviews, various international evaluation case studies, as well as a review of relevant evaluation literature.
The Study of the Evaluation Function in the Federal Government (March 2000) made recommendations for evaluation within the federal government and led to the development of the revised Evaluation Policy.
Issue
A Review of the Program Evaluation function is currently being completed by TBS. The President of the Treasury Board recently tabled in the House of Commons a management framework for the Government of Canada - Results for Canadians. Both these undertakings raise an issue as to the contribution of Program Evaluation to results-based management in the federal government. This note presents main study findings and makes recommendations for articulating the role and positioning of Program Evaluation within the federal government.
Background
- Evaluation was officially introduced into the federal government in 1977 to help improve management practices and controls. With the introduction of the Policy Expenditure Management System (PEMS), a significant step was made towards increasing accountability within the government as well as improving policy-making and priority setting by the government. Key to success is the information base on which such decisions are made. Officials and ministers must have reliable, relevant and objective information available if real improvements in the decision-making process are to be achieved. Program evaluation is considered to be one important source of this information.
- Since the introduction of the policy, the management framework has evolved. An increasing deficit and lack of confidence of citizens in their governments had led to a number of public sector reforms in many jurisdictions, a central element being a focus on results. This continuing focus is changing the way in which government designs, delivers, evaluates and reports on programs and services.
- The Treasury Board and its Secretariat have been working with departments and agencies on a number of initiatives aimed at modernizing management practices across the federal government. This has been a continuing evolution of the past several years - allowing departments the freedom to focus on results while supporting them with the tools needed to become more efficient, effective, responsive and innovative.
- It is within this context that program evaluation is currently being reviewed. The study findings provide important insights as to how evaluation can best support managers in managing for results.
Methodology
The Review study used multiple lines of enquiry to generate information:
- A review of the literature: The review has covered the evolution of evaluation in the Canadian context as well as in other jurisdictions.
- Case studies: The practice of evaluation in other OECD countries has also been examined. Particular attention has been given to the work that is going on in Australia and New Zealand.
- Survey of the Community: Interviews have been conducted with a representative sample of the heads of evaluation in the federal government to get their views and experiences.
- Strategic Interviews: Interviews have been conducted with a sample of ADMs both line and function, as well as with a select group of outside experts, to seek their views on how to position evaluation in a context of modern comptrollership and results-based management.
Findings
Current Situation:
- Evaluation is receiving renewed attention in OECD countries. Evaluation is recognized as "important in a result-oriented environment because it provides feedback on the efficiency, effectiveness and performance of public policies and can be critical to policy improvement and innovation." (OECD, 1999)
- This view is echoed by senior decision-makers in the federal government as well as experts outside the government who see an essential link between policy and program development and evaluation.
- The objective of evaluation is to improve decision-making at all levels. Yet its actual use has often proved to be limited, especially in relation to policy decisions and budget allocations.
- Part of the problem lies in the lack of demand for effectiveness information. "Historically, governments have focused their attention on resource inputs (what they spend), activities (what they do), and outputs (what they produce). Accurate information at this level is important but insufficient in a results focused environment". (Results for Canadians: A Management Agenda for the Government of Canada, March 2000)
Contribution and Usefulness
- Past studies of evaluation in the federal government indicate that the function has not lived up to the original policy expectations set out in 1977, that is, measuring the effectiveness of policies and programs. In fact, evaluations have resulted largely in the operational improvements to and monitoring of programs, rather than more fundamental changes. The reason for this may be that there is a disconnect between the information that evaluators were asked to provide and the kind of information managers needed.
- With the new focus on results, evaluation functions are getting more attention. Evaluation managers report an increased demand for the development of evaluation frameworks and performance measures.
- The evaluation framework is a planning tool used by evaluators. It sets out the performance issues for a program and identifies performance indicators and strategies for collecting information. Frameworks until recently were developed to guide the future evaluation of a program. Although promoted as a tool helpful to program planning and design, it has not been until recently that program managers are now finding evaluation frameworks useful in helping to better design, manage and monitor the performance of programs at all levels.
- The more direct and continuing links between evaluators and managers are seen as a positive development. Evaluation is beginning to be seen less as a check on management and more as an aid to good management.
- The evaluation framework is also proving a useful tool for the management of horizontal initiatives. It is a way of building the logic for an initiative, for identifying performance issues and measures and for demonstrating how the different stakeholders contribute to outcomes.
- The use of evaluation beyond the narrow confines of a single initiative, within a single department is seen as necessary to maximize the government's return on its evaluation investment.
Evaluation and Strategic Management
- Program evaluation is considered a component of strategic management. It is designed as an aid to decision-making and management. Consequently, it is a source of information for program improvement and organizational learning.
- It is also a source of information for managers in their accountability for the delivery of programs. This position seems to characterize the early forms of evaluation under PEMS. Under this regime, evaluation was seen more as an outside challenge function and less as an organizational learning tool. This may help to account for its lack of strategic use by management in departments.
- Evaluation works at different levels. The literature on program evaluation makes a distinction between formative and summative evaluation. Generally speaking formative evaluation is conducted during the operation of a program to provide program managers with information useful in improving the program. Summative evaluation, on the other hand, is conducted when a program has been up and running for a sufficient period of time to provide decision-makers with judgements of worth or merit.
- Studies indicate that the work of evaluation has been largely focused on formative evaluation activities. However, the measurement of effectiveness through demonstrated program results is viewed as an essential feature of the new citizen- focused management regime. Such attributes of programming are believed to be precisely those where (summative) evaluation can play a lead role.
Delivering Evaluation
- Managers need the proper tools and knowledge to manage for results. Evaluation is one of these tools. However, consideration needs to be given to purpose of the evaluation and the level of skill and expertise required. A study carried out by the OECD distinguishes between four types of evaluation practitioners depending on the purpose of the study and the level of expertise required:
- Self-evaluation, or internal evaluation, by managers or an organization is appropriate when the main objectives are program or organizational learning and improved implementation. The program employees are certain to know more about the program than an outsider. However the time and skills of staff may be insufficient, the range of issues covered may be limited and the credibility of findings may also be questioned.
- Evaluation by central management agencies is considered appropriate when the objective is improving budget priorities or assessing the cross departmental results within a given programming sector, and when it is important that the evaluator have close links with the central decision-making process.
- Independent evaluation is appropriate when the objectives are to improve accountability and transparency.
- External evaluations (e.g. research bodies and management consultants) is appropriate when the objective is to provide new perspectives on public policies or when there is a need for specialized evaluation skills.
- In the federal government, there is room to develop the capacity of managers to carry out evaluations on their programs (i.e. self-evaluations). This is beginning through the emphasis on performance measurement. Evaluators in departments are working with managers to help develop measures to monitor program performance.
- As indicated, there is an increased interest in evaluation (or performance) frameworks prior to the implementation of a new or renewed program. Although mandated by policy, these frameworks were not always created, or if created not always implemented. These frameworks help lay out the logic of the program and expected results from inputs through outcomes up to key results commitments. Assisting managers in developing these tools can improve program design and overall management for results.
- It has been convincingly argued that it is possible to construct a performance measurement system that can provide a credible story as to a program's contribution to larger government objectives. This is not intended to replace evaluation but perhaps to use evaluation more strategically. This "contribution" analysis works particularly well where there is direct control over the outcomes, but diminishes in capacity as the relationship between government interventions to actual outcomes sought becomes blurred by the multiplicity of intervening factors or players.
- There is still a need, and an interest in, effectiveness evaluations, and a role for independent and/or external evaluations within a department or agency. Given the cost associated with such undertakings, these evaluations need to be carried out on a more selective basis and with the right expertise. Good program design and measurement information provides the foundation for such studies.
- There is an important role for central agencies in the evaluation of initiatives involving more than one department. The Treasury Board Secretariat has been playing a role in helping the key departments involved in such initiatives to establish evaluation frameworks (e.g. TAGS, Family Violence). The process of building a framework is highly collaborative involving managers, evaluation specialists as well as representatives from key stakeholder groups.
- These horizontal program delivery arrangements are increasing and may require more involvement from the centre to ensure that the measures and strategies for measuring effectiveness are in place.
Evaluation Policy
- In 1994 the Review Policy was created and brought together the various "tools" for ensuring effective management of programs in departments and sought to clarify the role of management and outside experts in the measurement process.
- Rather than clarifying roles, there is concern that the Review Policy has had the opposite effect. In fact the roles of audit and evaluation have been blurred and management has shown little discrimination in the review tools and the nature of the expertise that is required.
- The subsequent program review exercise also led to a serious undermining of the capacity of evaluation functions.
Conclusions
"The key value of evaluations is that they allow for in-depth study of performance and independent assessment of effectiveness of government interventions. The main objectives of evaluations are to improve decision-making, resource allocation and accountability...Evaluations must be part of a wider performance management framework. They can supplement and improve it, but not replace it." (OECD,1999)
There is clearly a need to renew the mandate of evaluation in the new management regime. Evaluation was formalised in the federal government as a key source for information for policy-making and priority setting and accountability. The context in which evaluation operates has evolved. The study shows that the function has a key role to play in an emerging results-based management culture to improve decision-making at all levels.
- In a culture that aims to empower managers to focus on results, there is need to develop management's ability to measure and monitor the performance of their programs. Evaluation (performance) frameworks and logic models are tools that are used by evaluators in planning for eventual evaluations. Too often developed after a program has been implemented, their real value is in their development with program managers and stakeholders prior to the implementation, thereby optimizing good program design.
- Good performance measurement systems are essential to the ongoing monitoring of programs. If well articulated, they can, in many instances, speak to a program's contribution to larger government objectives and thereby support policy decisions as well as demonstrate accountability. However, there is still a need for effectiveness evaluations where, for example, the multiple lines of evidence point in different directions; and, where it is critical to have good information on attribution. In these cases, specific evaluation expertise may be required.
- A results focus is pushing out the boundaries on government programs and initiatives. Management is working on a more "borderless" basis: across departmental boundaries or in partnership with other governments, with business or with not-for-profit. These collaborative arrangements, as they go beyond the accountability of a single entity, will require the innovative use of evaluation resources. Central involvement cannot be ruled out and could take several forms, from a leadership role to facilitate the development of frameworks for the measurement, monitoring and reporting of the initiative to the actual conduct of an eventual evaluation.
- Capacity for the conduct of evaluations has been seriously eroded over the past few years. Many seasoned evaluators either took early retirement or took early departure to pursue work outside the government. Program review had a serious impact on the evaluation function in departments. The function suffered cuts and has been unable to recruit new blood. Although this situation is changing slowly, there is a gap in expertise between the senior evaluators and "new recruits".
Recommendations
The following recommendations flow naturally from the study and aim to re-establish evaluation in a manner that will support managing for results and link much closer with the improved design and management of programs while supporting the principles of modern comptrollership.
Becoming Central to Performance Management and Program Improvement
Recommendation 1: Program evaluation be linked much closer with the improved design and performance management of programs while supporting management learning.
There is a clear need for managers, and their employees, in the federal government to become more knowledgeable about performance measurement and evaluation if they are to effectively manage for results. Evaluation specialists have a role to play as knowledge brokers. The tools of evaluation, such as logic models, frameworks and measurement, are also tools of good program design. Management will need to learn how to use these and other methods of evaluation to improve program performance and better manage for results. Evaluators must become more directly involved.
The transfer of knowledge to managers can take many forms. Besides the more traditional classroom training, there is the partnering of evaluators with program managers and their staff on the development of evaluation/performance frameworks for their programs. This is currently underway in some departments and should be encouraged in all. The Treasury Board Secretariat has been working on a set of guidelines for such frameworks, which will be welcomed by managers.
Evaluating Effectiveness
Recommendation 2: That effectiveness evaluation be a part of the overall performance management plan of a department/agency.
Good performance measurement systems are required to be able to manage for and report on results. However, these systems do not preclude the need for periodic evaluation, and in particular, effectiveness evaluation. In fact, they are a necessary to evaluation. Effectiveness (summative) evaluations provide information on the overall impact and effects of an initiative, as opposed to the formative evaluations, which focus on more operational aspects. Effectiveness evaluations should form part of an overall departmental performance management plan.
There is a continued role for evaluation experts in organizations to carry out evaluations both from a methodological perspective as well as from a position of independence. This does not mean that management is not involved. Managers know the programs. They are the "subject matter experts". Studies show that evaluations are more successful if they are done in collaboration with key participants (evaluators, users, and stakeholders).
Evaluating from the Centre
Recommendation 3. The Treasury Board Secretariat play a more active role in the development of strategies for the measurement, monitoring and evaluation of program initiatives that cut across organizational boundaries and accountabilities.
As the government moves towards a more borderless approach to delivering programs and services to citizens, there is a need for more collaboration on the design, measurement and evaluation of such initiatives. There is an opportunity for more leadership from the centre in such exercises to ensure that the proper tools are in place for the ongoing monitoring and evaluation of government-wide initiatives. This information is needed to support the Treasury Board as a Cabinet Committee as it relies on such information to assist it in its analyses of resource and results information on a "whole of government" basis.
Building Capacity
Recommendation 4. The capacity to carry out evaluations on federal government programs be re-built. This will need to include the identification of core competencies and a recruitment and development plan.
There is a need to rebuild the evaluation capacity of the federal government. The Canadian federal government, which at one time was leading the way in the evaluation of its programs, appears to have fallen behind while countries such as Australia are in the forefront.
This capacity needs to be rebuilt in both departments and agencies and at the centre. A study will be required to determine our current capacity and what will be needed over the immediate and future. Consideration should be given to a recruitment and development program along the lines of the FORD (Financial Officer Recruitment Program) and IARD (Internal Audit Recruitment Program).
Policy Revision
Recommendation 5. The existing "Review" policy, which muddies the distinction between audit and evaluation, be revised to resituate evaluation in the government's new management framework.
Existing standards will also require revision.