MAF 2017 to 2018 results management methodology

From: Treasury Board of Canada Secretariat

Methodology Overview

Sound performance measurement and evaluation, both hallmarks of good management, have an impact on achievements. Performance measurement and evaluation enable organizations to clearly articulate business needs and objectives and measure the degree to which those objectives are being met. They also indicate areas where improvements can be made.

The overall objectives of the Results Management Accountability Framework (MAF) 2017-18 methodology are to ensure:

Departments use performance measures and evaluations to make better decisions; and,
Treasury Board has accurate performance measurement and evaluation information to effectively scrutinize new funding requests.

Given these objectives, the indicators have been grouped into two key areas:

Availability of performance information data for decision-making; and,
Use of performance information in decision-making.

The 2017-18 MAF results will provide the following to the three key audiences listed below:

Deputy Heads:

Understand the quality of Results Appendices in Treasury Board submissions in order to effect improvements where needed; and,
Ensure performance information is used in decision-making.

Results Management Community:

Provide an understanding of the state of results development in departments and agencies in order to identify potential areas of improvement.

TBS:

Improve TBS’ challenge function on Results Appendices in Treasury Board submissions; and,
Develop training and capacity targeting areas for improvement across government.

While the methodology will provide insight on how well performance measurement and evaluation are done, it will not represent the entirety of performance for this Area of Management; the questions are limited both in number and in what they measure. As the Policy on Results and the government’s renewed focus on results is still relatively recent, the methodology for 2017-18 is transitional. Future MAF cycles will include new questions and existing questions will be reviewed and further streamlined.

This methodology applies only to large departments and agencies (LDAs). It has been developed in consultation with the Policy on Results Office of Primary Interest representatives.

Questionnaire

In this section
- Availability of performance information data for decision-making
- Use of performance information in decision-making

Availability of performance information data for decision-making

Outcome Statement: Departments and agencies measure and assess their performance well

Rationale: Departments need to understand what is working and what is not, and to make improvements to their programs. Availability of quality performance information is a first step in determining what results are being achieved and where resources are needed.

Indicators and Calculation Method (where applicable)	Expected Result	Policy Reference	Evidence Source and Document Limit	Category
What is the quality of Results Appendices and past performance information in first draft TB Submissions received from the department? Excellent Good Fair Poor None submitted How do you calculate this? We have developed an assessment methodology that we apply to the first engagement draft of the TB submission based on criteria identified in the Detailed Guidance for Writers: Writing a Treasury Board Submission. Only submissions, with Results Appendices, that are presented to Treasury Board in the 2017 calendar year will be included.	TB submissions provide high quality results information to inform decision-making.	Policy on Results, 4.3.10	No Evidence. TBS will conduct the assessment.	Management Practice Practice
What is the quality of Results Appendices and past performance information in final TB submissions received from the department? Excellent Good Fair Poor None submitted How do you calculate this? We have developed an assessment methodology that we apply to the TB submission based on criteria identified in the Detailed Guidance for Writers: Writing a Treasury Board Submission. Only submissions, with Results Appendices, that are presented to Treasury Board in the 2017 calendar year will be included.	TB submissions provide high quality results information to inform decision-making.	Policy on Results, 4.3.10	No Evidence. TBS will conduct the assessment.	Management Practice Practice
What percentage of performance indicators in Results Appendices have been implemented? %: How do you calculate this? Implemented refers to having in place actual data for a performance indicator. This indicator is calculated as: number of performance indicators (included in TB Submissions Results Appendix), which have been implemented, divided by the total number of performance indicators identified in the Results Appendices which should have been implemented. Example: If an indicator was approved in September 2016 with a frequency of annual, then by November 2017 it should have one actual data value to be considered as implemented. If the frequency was every six months, then it should have two data values to be considered as implemented. This would apply for all TB submissions, approved between September 2016 and December 2016, which have Results Appendices. In due course, this information should be in the department’s performance information profiles (PIPs).	Departments and agencies measure and report on results they committed to in TB submissions.	Policy on Results, 4.3.9, 4.3.10	Departments and agencies provide actual results data in a template sent by TBS. TBS will prepopulate the template with the indicators departments and agencies committed to implement. The completed template will be submitted to TBS outside of the MAF Portal.	Management Performance Performance Indicator

Indicators and Calculation Method (where applicable)

Expected Result

Policy Reference

Evidence Source and Document Limit

Use of performance information in decision-making

Outcome Statement: Departments and agencies use performance measurement and evaluation information to make decisions and improve their results.

Rationale: The Policy on Results requires departments to use performance information in resource allocation decisions. This will ensure that resources are best allocated to achieve optimal results.

Indicators and Calculation Method (where applicable)	Expected Result	Policy Reference	Evidence Source and Document Limit	Category
To what extent are performance measurement and analysis thereof factored into senior management resource allocation decisions? Always Often Rarely Never How do you determine this? Provide specific examples of performance data and analysis that were factored into resource allocation decisions. Definitions and examples of senior management, resource allocation decisions, and input are provided in the Glossary.	Departments and agencies use performance information when making resource allocation decisions.	Policy on Results, 3.1, 3.2	One piece of evidence. Departments and agencies must provide: A description of how the department regularly ensures input is provided. Specific examples where such input was provided Always: 5 examples Often: 3 examples Rarely: 1 example Examples should be about half of a page, describing the input and the decision-made. Depending on the examples provided, TBS may ask for clarifications or additional examples.	Management Practice Practice
To what extent are findings and recommendations from evaluations factored into senior management resource allocation decisions? Always Often Rarely Never How do you determine this? Provide specific examples of how the findings and recommendations from evaluations were factored in to resource allocation decisions. Definitions and examples of senior management, resource allocation decisions, and input are provided in the Glossary.	Departments and agencies use performance information when making resource allocation decisions.	Policy on Results, 3.1, 3.2	One piece of evidence. Departments and agencies must provide: A description of how the department regularly ensures input is provided Specific examples where such input was provided Always: 5 examples Often: 3 examples Rarely: 1 example Examples should be about half of a page, describing the input and the decision-made. Depending on the examples provided, TBS may ask for clarifications or additional examples.	Management Practice Practice
What percentage of evaluation management action plan items due to be implemented in the fiscal year were actually implemented? %: No action plans items scheduled to be implemented in 2016-17 How do you calculate this? Number of planned Management Action Plan items scheduled to be implemented in the 2016-17 fiscal year that have been fully implemented, divided by the total number of planned Management Action Plan items scheduled for implementation in the 2016-17 fiscal year. Obsolete recommendations are excluded from the calculation.	Departments make improvements to their programs.	Directive on Results, 4.4.4.1	No Evidence. TBS will conduct the assessment based on the annual Capacity Assessment Survey.	Management Performance Performance Indicator

Indicators and Calculation Method (where applicable)

Expected Result

Policy Reference

Evidence Source and Document Limit

Glossary

(Ordered by question)

Results Appendix (Questions 1-3)

Mandatory appendix where organizations should provide details on the short, medium and long term results associated with the proposal, how they will be measured and what the key targets are. Organizations are also asked to explain how performance information will be overseen to ensure that the intended results are being achieved, supported by key milestones.

Performance Information Profile (Question 3)

The document that contains the performance information for each Program from the Program Inventory.

Senior Management (Questions 4-5)

Senior executive committees within the department where final resource allocation decisions are made or approved.

Resource Allocation Decisions (Questions 4-5)

All allocation decisions to fund programs or services a department delivers, including: :

Internal allocation decisions (e.g., budgeting, responding to new pressures, reallocation during the year or longer term)
New funding requests (e.g., using existing past program results to inform a Memoranda to Cabinet)
Renewal of existing funding (e.g., using results achieved to support program renewal in TB submission)

Input (Question 4)

Analysis, advice and recommendations. For example:

Analysis of trend data for outcome indicators and resources, along with contextual data, shows a program may be achieving some outcomes but not at the level expected. Additional resources may be allocated to the program to improve results.
Analysis of the outcome indicators, resources and contextual information shows that results of a program has plateaued over the past three years, even though resources have increased. Senior Management could decide to limit future investments in the program while looking at alternatives.

Input (Question 5)

Analysis, advice and recommendations. For example:

An evaluation recommendation states the program has a number of barriers that is affecting it from being more effective and that additional resources could overcome certain barriers. An analysis on the minimum level of new resources needed should be part of the input.
An evaluation suggests that a program is achieving its outcomes and is doing so efficiently; however, certain risks to the program or the beneficiaries have increased over the last two years with all indication that it will increase in the future. Senior management may decide to allocate resources targeted at reducing the risks.
While there is no evaluation report on the specific program, the input could be analysis of programs with similar approaches and what lessons were learned, or how effective the approach was found to be.

Management Action Plan (Question 6)

Actions management committed to take to address the recommendations in an evaluation report.

Abbreviated policy and policy instrument names used in the “Policy Reference” column:

For additional definitions of the terms and expressions used in this questionnaire, please refer to the Results Portal.

Page details

Date modified:: 2018-09-21

Language selection

Search