MAF 2016 to 2017 management of integrated risk, planning and performance methodology

Methodology Overview
Questionnaire
Glossary

Methodology Overview

To ensure effective and efficient management of federal organizations and their programs, expenditure management decisions need to be evidence-based and supported by a clear alignment between resources and results.

The objective of the Management of Integrated Risk, Planning and Performance Area of Management (AoM) is to capture a system-wide view of managements’ use of performance information to ensure that programs are achieving their objectives and advancing organizations’ mandates.

The government-wide context of achieving excellence in results has been renewed with the public release of the government mandate letter commitments, the establishment of the Cabinet Committee on Agenda, Results and Communications, the Results and Delivery Unit at the Privy Council Office, and the launch of the new Policy on Results. Collectively, these actions emphasize the growing importance of managing programs with a clear understanding of what they are trying to achieve, how success will be measured, and what has been achieved with the resources used to do so.

With the Policy on Results taking effect July 1, 2016, 2016-17 will be a transition year for departments to begin implementing the new Policy. As a result, this year’s methodology has been streamlined to focus on areas of work that will remain relevant and relatively stable, namely the use of performance information in proposals to Treasury Board (i.e., Treasury Board Submissions) and implementation of the new Policy.

Approach for 2016-17 MAF

As noted in 2014-15 and 2015-16, the methodology for this AoM is following a phased-in approach. Building on the positive results of 2014-15 that revealed mature annual integrated planning practices system-wide, in 2015-16 the methodology was refined to determine whether and how organizations were using their performance measurement data to manage their programs.

Many of the 2015-16 questions yielded positive system-wide results. The majority of departments and agencies had a Performance Measurement Framework (PMF) at an acceptable level of quality; they had developed methodologies for collecting performance data; and they were using performance information to inform and monitor progress on risks, priorities, and resource allocation decisions.

Areas for improving performance measurement were also identified in 2015-16. Departments did not always use performance information as an integral part of Cabinet proposals. This raised questions about how well performance information is being integrated into the funding proposals to Treasury Board.

As a result, the methodology for this AoM has been updated to reflect the 2015-16 findings. The line of questioning on the use of performance information has been updated to identify the type of performance measurement and evaluation information included in proposals to Treasury Board. New questions have been added related to evaluation coverage, governance functions to support management’s performance measurement roles and responsibilities, and strategies for collecting performance data to support the ongoing management and reporting of the department’s programs. These questions are intended to assess government-wide readiness and progress towards implementing the new Policy on Results.

This methodology aims to ensure that departments and agencies are positioned to use quality performance information to support program expenditure management decisions within departments and central agencies. It has been developed in collaboration with departments and agencies. Unless otherwise indicated in individual questions, the reference period for the Management of Results AoM questionnaire is April 1, 2016 to March 31, 2017. A glossary of key terms is provided for reference at the end of this document.

Questionnaire

Creation of quality performance information

Outcome statement: Department or agency has clear governance functions and data collection strategies in place to create quality performance information

Indicators and Calculation Method (where applicable)	Expected Result	Policy Reference^{table 1 note *}	Evidence Source and Document Limit	Category
Table 1 Notes Table 1 Note 1 See the Glossary at the end of this document for the long version of the abbreviated policy and policy instrument names. Return to table 1 note * referrer Table 1 Note 2 Departments have until November 1, 2017 to implement these aspects of the Policy. All other response options came into effect July 1, 2016. Return to table 1 note ** referrer
Does the department have a plan in place to ensure that it can generate and report performance data at the Program Inventory level for the following items in support of the Policy on Results, the Estimates and Public Accounts publications? (Select at least one and all that apply) Performance Results Financial Expenditures Human Resources – Full-Time Equivalents (FTEs) Tagging The department has no plan in place for these purposes Calculation of measure: Department to respond to this question based on departmental information at the time of the draft and final submission deadlines. Note: The term Plan refers to a documented agreement or understanding, which outlines how the department will ensure it has program data available in support of the Policy on Results, the Estimates and Public Accounts publication, and other internal and external uses. The intent of the question is to simply state that the department has a plan in place that will enable them to respond to requests to generate and report performance data at the Program Inventory level.	Departments and agencies have a plan in place to ensure they can collect data to measure and evaluate the performance of their programs, policies and services, and upon request, to provide this information to Treasury Board Secretariat.	Policy on Results, 4.3.6, 4.3.9, 4.3.10	Up to 4 pieces of evidence Evidence can include any existing materials such as a deck, plan, action plan, work plan, report, annotated agenda, record of decision, or similar product/tool Evidence submitted should reflect responses selected Department to select at least one response	Management Practice Practice
Which of the following types of governance does the department have in place in support of the new Policy on Results? (Select all that apply) Central Performance Measurement and Evaluation Committee Head of Performance Measurement^{table 1 note } Head of Evaluation Program officials^{table 1 note } Deputy Head signs off on performance measurement and evaluation information in Treasury Board Submissions Calculation of measure: Department to respond to this question based on departmental information at the time of the MAF draft and final submission deadlines.	Departments and agencies have governance in place to establish performance measurement functions within their organizations to oversee departmental performance, including overall progress on results, and to keep an inventory of programs with performance information for each program.	Policy on Results, 4.3.3, 4.3.5, 4.3.6, 4.3.12, 4.3.14	No Evidence Department to select at least one response	Management Practice Policy Compliance

Indicators and Calculation Method (where applicable)

Expected Result

Policy Reference^{table 1 note *}

Evidence Source and Document Limit

Category

Table 1 Notes

Table 1 Note 1

See the Glossary at the end of this document for the long version of the abbreviated policy and policy instrument names.

Return to table 1 note * referrer

Table 1 Note 2

Departments have until November 1, 2017 to implement these aspects of the Policy. All other response options came into effect July 1, 2016.

Return to table 1 note ** referrer

Does the department have a plan in place to ensure that it can generate and report performance data at the Program Inventory level for the following items in support of the Policy on Results, the Estimates and Public Accounts publications? (Select at least one and all that apply)
- Performance Results
- Financial Expenditures
- Human Resources – Full-Time Equivalents (FTEs)
- Tagging
- The department has no plan in place for these purposes
Calculation of measure:

Department to respond to this question based on departmental information at the time of the draft and final submission deadlines.

Note:

The term Plan refers to a documented agreement or understanding, which outlines how the department will ensure it has program data available in support of the Policy on Results, the Estimates and Public Accounts publication, and other internal and external uses.

The intent of the question is to simply state that the department has a plan in place that will enable them to respond to requests to generate and report performance data at the Program Inventory level.

Departments and agencies have a plan in place to ensure they can collect data to measure and evaluate the performance of their programs, policies and services, and upon request, to provide this information to Treasury Board Secretariat.

Policy on Results, 4.3.6, 4.3.9, 4.3.10

Up to 4 pieces of evidence

Evidence can include any existing materials such as a deck, plan, action plan, work plan,

report, annotated agenda, record of decision, or similar product/tool

Evidence submitted should reflect responses selected

Department to select at least one response

Management Practice
- Practice

Which of the following types of governance does the department have in place in support of the new Policy on Results? (Select all that apply)
- Central Performance Measurement and Evaluation Committee
- Head of Performance Measurement^{table 1 note **}
- Head of Evaluation
- Program officials^{table 1 note **}
- Deputy Head signs off on performance measurement and evaluation information in Treasury Board Submissions
Calculation of measure:

Department to respond to this question based on departmental information at the time of the MAF draft and final submission deadlines.

Departments and agencies have governance in place to establish performance measurement functions within their organizations to oversee departmental performance, including overall progress on results, and to keep an inventory of programs with performance information for each program.

Policy on Results, 4.3.3, 4.3.5, 4.3.6, 4.3.12, 4.3.14

No Evidence

Department to select at least one response

Management Practice
- Policy Compliance

Use of performance information in decision-making

Outcome statement: Department or agency uses performance information to identify and monitor progress against program results

Indicators and Calculation Method (where applicable)	Expected Result	Policy Reference^{table 2 note *}	Evidence Source and Document Limit	Category
Table 2 Notes Table 2 Note 1 See the Glossary at the end of this document for the long version of the abbreviated policy and policy instrument names. Return to table 2 note * referrer
Does senior management use performance information on program efficiency and effectiveness to identify risks, establish priorities and/or support resource allocation decisions? (Select all that apply) Yes, to identify risks Yes, to establish priorities Yes, to support resource allocation decisions Performance information is sometimes used for some or all of the above purposes, but not in a consistent manner No, performance information is not used for any of the above purposes (Go to Question 4) Calculation of measure: Department to select all responses that apply. For example, if all three “Yes” options are selected, the fourth and/or fifth option should not be selected. If the department only uses performance information to establish priorities, it would select option 2. If the department always uses performance information to identify risks, but only sometimes uses it to establish priorities and support resource allocation decisions, the department would select options 1 and 4.	Departments and agencies use performance information to inform their planning and resource allocation decisions.	Policy on Results, 3.1., 3.2	Up to Five Pieces of Evidence Risk identification, priority setting or resource allocation exercise material, i.e., deck, record of decision, annotated agendas or similar product/tool that demonstrates decisions taken were informed by performance information on program efficiency and effectiveness Evidence submitted should reflect responses selected	Management Practice Practice
Does the department or agency track progress on the performance indicators in its proposals to Treasury Board? Yes No Not applicable (no proposals submitted to Treasury Board during time period – proceed to Question 5) Calculation of measure: The time period for this assessment is proposals to Treasury Board received by TBS between November 2015 and November 2016. If the department did not have any proposals to Treasury Board during this time, the question would not be applicable and the department would proceed to Question 5 of the methodology. The assessment time period of November 2015 to November 2016 should be applied for the draft and final MAF submissions. Note: The intent of this question is to understand the extent to which departments are monitoring progress against performance indicators approved in proposals to Treasury Board. How departments track this information is at their discretion. PMF, PM Strategies and/or Performance Information Profiles are examples of documents that could be used.	Departments and agencies track their progress on all performance indicators used to support their proposals to Treasury Board.	Policy on Results, 4.3.9, 4.3.10	No Evidence Department to select one response	Management Practice Practice
What percentage of performance indicators included in proposals to Treasury Board are being tracked to monitor their progress? 85-100% 65-84% 50-64% 0-49% Not applicable (no proposals submitted to Treasury Board during time period, or proposal item(s) for which performance indicators will be tracked have not yet started) Calculation of measure: The time period for this assessment is proposals to Treasury Board received by TBS between November 2015 and November 2016. If the department did not have any proposals to Treasury Board during this time, the question would not be applicable and the department would proceed to Question 6 of the methodology. The assessment time period of November 2015 to November 2016 should be applied for the draft and final MAF submissions. The piece of evidence should be the department's calculation, i.e. Number of Performance Indicators being tracked from TB Proposals divided by Number of Performance Indicators in TB Proposals Note: The intent of this question is to understand the extent to which departments are monitoring progress against performance indicators approved in proposals to Treasury Board. How departments track this information is at their discretion. PMF, PM Strategies and/or Performance Information Profiles are examples of documents that could be used.	Departments and agencies track their progress on all performance indicators used to support their proposals to Treasury Board.	Policy on Results, 4.3.9, 4.3.10	1 piece of Evidence Department to select one response	Management Practice Practice
Which required performance and evaluation information in the department or agency’s proposals to Treasury Board should be strengthened? (Select all that apply) Expected outcomes (e.g., as articulated in the Expected Results section and the Delivery and Expected Results Appendices, as applicable) Performance indicators (e.g., as articulated in the Expected Results section and the Delivery and Expected Results Appendices, as applicable) Evaluation information (e.g., evaluation findings, conclusions, recommendations and management responses/action items from past or ongoing evaluations) Not applicable (no proposals submitted to Treasury Board during time period or proposal(s) did not require strengthened performance information) Calculation of measure: TBS to assess first engagement draft proposals to Treasury Board based on criteria identified in the Detailed Guidance for Writers: Writing a Treasury Board Submission and centrally-held data on proposals to Treasury Board. The time period for this assessment is first draft engagement proposals to Treasury Board received by TBS between September 1 and November 30, 2016. The assessment period is the same for the draft and final MAF submissions. If the department did not have any proposals to Treasury Board during this time, the question would not be applicable and the department would proceed to Question 7. Note: TBS will use the criteria outlined in the Detailed Guidance for Writers: Writing a Treasury Board Submission, under the Expected Results section and the Delivery and Expected Results Appendices sections, as applicable, to assess this question.	Departments and agencies include relevant and appropriate performance measurement and evaluation information about programs to support their proposals to Treasury Board. It is expected that this information be present in the first engagement draft of the proposal to Treasury Board that the department submits to TBS.	Policy on Results, 4.3.10	No Evidence TBS to assess based on criteria identified in the Detailed Guidance for Writers: Writing a Treasury Board Submission and centrally-held data on proposals to Treasury Board	Management Practice Practice
How often does the department include past program results in its proposals to Treasury Board? Always (for proposals with existing programs) Sometimes (for proposals with existing programs) Rarely (for proposals with existing programs) Not Applicable: all proposals to Treasury Board were for new programs that did not have past results Not Applicable: no proposals submitted to Treasury Board during time period Calculation of measure: TBS to assess first engagement draft proposals to Treasury Board based on criteria identified in the Detailed Guidance for Writers: Writing a Treasury Board Submission and centrally-held data on proposals to Treasury Board. The time period for this assessment is first engagement draft proposals to Treasury Board received by TBS between September 1 and November 30, 2016. The assessment period is the same for the draft and final MAF submissions. If the department had proposals to Treasury Board but those proposals were for new Programs that did not have past results, the question is not applicable and the department would proceed to Question 8. If the department did not have any proposals to Treasury Board during this time, the question would not be applicable and the department would proceed to Question 8. Note: TBS will use the criteria outlined in the Detailed Guidance for Writers: Writing a Treasury Board Submission, under the Expected Results section and the Delivery and Expected Results Appendices sections, as applicable, to assess this question. The first three response options assume that a department would not be able to report past results on new programs.	Departments and agencies use performance measurement and evaluation information about programs to support their proposals to Treasury Board. It is expected that this information be present in the first engagement draft of the proposal to Treasury Board that the department submits to TBS.	Policy on Results, 4.3.10	No Evidence TBS to assess based on criteria identified in the Detailed Guidance for Writers: Writing a Treasury Board Submission and centrally-held data on proposals to Treasury Board	Management Practice Practice

Internal and External Performance Reporting

Outcome statement: Department or agency has performance information available for internal and external reporting

Indicators and Calculation Method (where applicable)	Expected Result	Policy Reference^{table 3 note *}	Evidence Source and Document Limit	Category
Table 3 Notes Table 3 Note 1 See the Glossary at the end of this document for the long version of the abbreviated policy and policy instrument names. Return to table 3 note * referrer
What percentage of evaluations planned for fiscal year 2015-16 were actually completed in that year? 0-49% 50-69% 70-89% 90-100% Calculation of measure: TBS to assess based on centrally-held assessments of departmental evaluation plans and submitted evaluations. The assessment will examine evaluations completed in fiscal year 2015-16 compared to those planned to be completed in fiscal year 2015-16, as identified in the 2015-16 Departmental Evaluation Plan. An evaluation is considered complete when it has been approved by the Deputy Head.	All evaluations planned for FY 2015-16 are completed in that year, as per the departmental evaluation plan.	Policy on Results, 4.3.15, 4.3.15.1, 4.3.15.2, 4.3.17	No Evidence TBS to assess based on centrally-held assessments of departmental evaluation plans and submitted evaluations	Management Practice Practice

Indicators and Calculation Method (where applicable)

Expected Result

Policy Reference^{table 3 note *}

Evidence Source and Document Limit

Category

Table 3 Notes

Table 3 Note 1

See the Glossary at the end of this document for the long version of the abbreviated policy and policy instrument names.

Return to table 3 note * referrer

What percentage of evaluations planned for fiscal year 2015-16 were actually completed in that year?
- 0-49%
- 50-69%
- 70-89%
- 90-100%
Calculation of measure:

TBS to assess based on centrally-held assessments of departmental evaluation plans and submitted evaluations.

The assessment will examine evaluations completed in fiscal year 2015-16 compared to those planned to be completed in fiscal year 2015-16, as identified in the 2015-16 Departmental Evaluation Plan.

An evaluation is considered complete when it has been approved by the Deputy Head.

All evaluations planned for FY 2015-16 are completed in that year, as per the departmental evaluation plan.

Policy on Results, 4.3.15, 4.3.15.1, 4.3.15.2, 4.3.17

No Evidence

TBS to assess based on centrally-held assessments of departmental evaluation plans and submitted evaluations

Management Practice
- Practice

Glossary

Core Responsibility

An enduring function or role performed by a department. The intentions of the department with respect to a Core Responsibility are reflected in one or more related Departmental Results that the department seeks to contribute to or influence.

Departmental Result

Departmental Results represent the changes departments seek to influence. Departmental Results are often outside departments’ immediate control, but they should be influenced by Program-level outcomes.

Departmental Results Framework

Consists of the department’s Core Responsibilities, Departmental Results and Departmental Result Indicators.

Effectiveness

The impacts of a program, policy or other entity, or the extent to which it is achieving its expected outcomes.

Efficiency

The extent to which resources are used such that a greater level of output/outcome is produced with the same level of input or, a lower level of input is used to produce the same level of output/outcome. The level of input and output/outcome could be increases or decreases in quantity, quality, or both.

Evaluation

In the Government of Canada, evaluation is the systematic and neutral collection and analysis of evidence to judge merit, worth or value. Evaluation informs decision making, improvements, innovation and accountability. Evaluations typically focus on programs, policies and priorities and examine questions related to relevance, effectiveness and efficiency. Depending on user needs, however, evaluations can also examine other units, themes and issues including alternatives to existing interventions. Evaluations generally employ social science research methods.

Performance Information Profile

The document that identifies the performance information for each Program from the Program Inventory.

Performance Measurement and Evaluation Committee

A committee of senior officials, chaired by the Deputy Head, which oversees departmental performance measurement and evaluation.

Program

Individual or groups of services, activities or combinations thereof that are managed together within the department and focus on a specific set of outputs, outcomes or service levels.

Program Inventory

Identifies all of the department’s programs and describes how resources are organized to contribute to the department’s Core Responsibilities and Results.

Program Official

An individual responsible for establishing, implementing and maintaining the Program’s Performance Information Profile, including ensuring data collection for it. One Program Official is required for each Program in the Program Inventory.

Abbreviated policy and policy instrument names used in the “Policy Reference” column

Detailed Guidance for Writers: Writing a Treasury Board Submission
Policy on Results
Guidance for the Policy on Results
President of the Treasury Board of Canada Mandate Letter

For additional definitions of the terms and expressions used in this questionnaire, please refer to the Appendices of the Policy on Results.

Page details

Date modified:: 2017-05-12

Language selection

Search