Neutral assessment of Employment and Social Development Canada’s evaluation function

Alternate formats

Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.

1. Neutral Assessment Report

1.1 Purpose of the Report

This report sets out the results from the neutral assessment of Employment and Social Development Canada’s (ESDC) evaluation function.

1.2 Objectives

The evaluation function is one (1) of several senior governance functions at ESDC. Its role is to conduct evaluation, which is the systematic and neutral collection and analysis of evidence to judge merit, worth or value. Treasury Board of Canada (TB) Policy sets out the expectations for a federal department’s or agency’s evaluation function in the Policy on Results, July 1, 2016. One (1) of the Policy’s requirements is that deputy heads ensure that a neutral assessment of their departmental evaluation function is conducted at a minimum of once every five (5) years. This is the second neutral assessment of ESDC’s evaluation function.

TB is not prescriptive in terms of the conduct or content of a neutral assessment. Raymond Chabot Grant Thornton Consulting Inc. (RCGT) was engaged by the Internal Audit Services Branch to conduct this neutral assessment of the Evaluation Function at ESDC.

The objectives of the neutral assessment were to address the following questions:

  1. Does the evaluation function conform to TB’s expectations?
  2. Is the evaluation organization well-managed?
  3. Is the function adding value to ESDC?

1.3 Scope and Approach

This neutral assessment exercise was conducted over a four-month period beginning in April 2018 and ending in July 2018. It provides findings and conclusions on the state of the function at the time of the assessment. Findings and conclusions related to the evaluation function relate to the practices at the department level; findings and conclusions that relate to the evaluation organization relate to the practices within the Evaluation Directorate.

To conclude on project-level matters, the exercise reviewed documentation from four (4) evaluation projects completed during the past five (5) years.

To conclude on organization- and function-level matters, the exercise examined practices in place at the time of the assessment.

The framework used for this neutral assessment includes three (3) distinct components, focusing on:

  • Governance of the evaluation function;
  • The evaluation organization’s professional practice at the organization level and at the project level; and
  • Monitoring and performance reporting associated with the evaluation function.

The framework includes criteria that were derived from policy, standards, and directives associated with the practice of evaluation. International standards on evaluation were also considered in developing the framework. Additional criteria were included that reflected sound management practices that were applicable to an evaluation function. The framework criteria are included in Section 2.0 of this document.

The neutral assessment was based on an analysis of documentation and interviews.

1.4 Conclusions from the Neutral Assessment

The conclusions address the three (3) assessment areas of governance; professional practice; and monitoring and performance reporting. Findings that support these conclusions are provided in section 1.5 of this report.

The neutral assessment determined that ESDC’s evaluation function conforms with:

  • TB’s expectations with regard to governance
  • Professional practice expectations at the function level
  • Professional practice expectations at the project level
  • Monitoring and performance reporting expectations

The neutral assessment findings indicate that ESDC’s evaluation function is meeting expectations. Specifically:

  • The function conforms to the policy expectations set out by TB, as well as to other reasonable professional expectations set out in the neutral assessment framework;
  • Evidence indicates that the evaluation function is well-managed; and
  • Evidence indicates that the evaluation function is adding important value to the ESDC.

The findings that support these conclusions are presented below.

1.5 Findings from the Neutral Assessment

The findings presented in this section inform the conclusions on whether the evaluation function either conforms to expectations, or its conformance to expectations is at risk.

Conformance indicates that processes and practices in place are determined to be in accordance with expectations set out in policy, standards, and directives. Where conformance is at risk, the processes and practices in place have been judged to fall short of expectations, and they negatively impact the evaluation function’s capability to perform its responsibilities fully and in a cost-effective manner.

The summary findings are presented below according to the structure of the neutral assessment framework.

1.6 Governance of the Evaluation Function

The neutral assessment’s governance criteria address responsibilities of the Deputy Head with respect to the evaluation function, the departmental Performance Measurement and Evaluation Committee, and the HoE.

1.6.1 Deputy Head Responsibilities: Conforms to Expectations

The Deputy Head is fulfilling required responsibilities related to governance. The Deputy Head has established an evaluation function, appointed a departmental Performance Measurement and Evaluation Committee and a HoE at an executive level, and has approved an Evaluation Plan. The Deputy Head has approved in a timely manner the completed evaluation reports that were reviewed, including associated management action plans.

1.6.2 Performance Measurement and Evaluation Committee Responsibilities: Conforms to Expectations

The departmental Performance Measurement and Evaluation Committee is fulfilling the responsibilities set out in the TB Policy on Results. It is mandated to review and approve evaluation-related matters; it meets policy expectations in terms of its composition; and its stated roles and responsibilities meet TB expectations.

1.6.3 Head of Evaluation Responsibilities: Conforms to Expectations

The HoE is fulfilling required responsibilities related to governance. The Evaluation Directorate conducts its work in a neutral manner; it appears to operate in a cost-effective manner; it supports the departmental Performance Measurement and Evaluation Committee; and evaluators adhere to the TB policy suite that governs evaluation.

1.6.4 Values and Ethics: Conforms to Expectations

The TB Policy on Results does not set out requirements pertaining to values and ethics. However, evaluators are involved in a wide range of activities that require them to act in a professional and ethical manner. For example, they interact with groups across ESDC, they can interact with ESDC’s outside stakeholders, they can initiate and manage contracts, and they have access to privileged information.

Reviewed documents indicate that the Evaluation Directorate acts professionally and in a manner consistent with federal government values and ethics.

1.7 The Evaluation Function’s Professional Practice

The neutral assessment’s professional practice criteria address: planning and conducting work at the function level; planning, conducting and reporting at the project level; and managing the evaluation function. The summary findings are presented below.

1.7.1 Planning and Conducting at the Function Level

Evaluation Planning at the Function Level: Conforms to Expectations

Measures are in place related to the evaluation plan. Documentation indicates that:

  • Adequate measures are in place for the development of the ESDC’s evaluation plan.
  • The evaluation plan is approved and communicated.
  • The most recent evaluation plan presents a strategy and the rationale for the strategy. The coverage decisions are suitably explained.
  • Evaluations have been conducted within the general timeframes for which they were planned.

Roles Related to Evaluation and Performance Measurement: Conforms to Expectations

The neutral assessment analysis determined that the evaluation function is meeting expectations related to conducting evaluations. The completed evaluations that were reviewed addressed core issues.

The neutral assessment analysis determined that the evaluation organization is meeting expectations related to departmental performance measurement.

With respect to performance measurement at the function level, deputy heads are to ensure that that ongoing performance measurement is implemented throughout the department so that sufficient performance information is available to effectively support the evaluation of programs. The HoE has been involved in providing comments on performance measurement information in support of evaluation, reviewing Treasury Board submissions, and providing advice on departmental performance measurement strategies and frameworks.

1.7.2 Planning, Conduct, and Reporting at the Project Level

The following assessment is based on the analysis of documentation available from four (4) completed ESDC evaluation projects:

  • Evaluation of the Extension of Employment Insurance Part I Benefit for Long-Tenured Workers – August 2014
  • Evaluation of Workplace Information and Research Activities of the Labour Program – November 2015
  • Evaluation of the Enabling Fund for Official Language Minority Communities – March 2017
  • Homelessness Partnering Strategy - Summative Evaluation – January 2018

The neutral assessment analysis at the project level includes: project planning, conduct, reporting, and use.

Evaluation Project Planning: Conforms to Expectations

According to the TB Directive on Results, plans should discuss project objectives; primary evaluation issues; resource use; risk and complexity; approach, design and methodology; evaluator competencies; and relevant policies.

The analysis of project information determined that a formal terms of reference was completed for all projects, and that essential project information was included in these terms of reference.

A noteworthy strength of the four (4) reviewed evaluations was the development of an Evaluability Assessment. These assessments provided considerable value added to the discussions about the respective evaluation’s design, focus and expected utility. Interviewed executives indicated that these reports added important value to evaluation projects.

Conduct of Evaluation Projects: Conforms to Expectations

The conduct of evaluation projects includes project leadership, stakeholder consultation, project management and control, use of evidence, and information control.

The evaluation projects were led by ESDC’s HoE, and respective stakeholders were consulted as part of each project, within the limitation of the methodology selected. The evaluation reports indicate that conclusions were based on evidence.

Practices are in place related to project management and control, and information control.

1.7.3 Evaluation Project Reporting: Conforms to Expectations

Project reporting includes the completeness and communication of evaluation reports, and their accessibility to the public.

The reports for the reviewed evaluations are clear, meet the requirements of the Standard on Evaluation, include management responses and action plans where required, and have been submitted to Treasury Board Secretariat.

The completed evaluation reports are posted on ESDC’s website.

The reviewed reports would be more useful and complete if they indicated to the reader the limitations of the methodologies and the resulting impacts on interpreting the findings and conclusions. This could be achieved through an overall statement about the validity and strength of the methodology, as well as cautionary statements in the text itself. These limitations are discussed in projects’ Evaluability Assessments; however, readers of the final report have no indication of these limitations and their impact.

Use of Evaluations: Conforms to Expectations

This requirement includes the use of evaluation findings by the Deputy Head to support senior decision-making. Interviews conducted as part of the neutral assessment indicated that results from evaluations have been used by ESDC’s executive level in program decision-making.

1.7.4 Evaluation Competencies: Conforms to Expectations

The evaluation projects reviewed were conducted by ESDC evaluation personnel, who were experienced evaluators, and by consulting firms that had the qualifications required to execute the work.

1.8 Monitoring and Performance Reporting of the Evaluation Function

The neutral assessment’s Monitoring and Performance Reporting criteria address annual and ongoing reporting obligations.

1.8.1 Monitoring and Performance Reporting by the Head of Evaluation: Conforms to Expectations

The HoE reports evaluation-related matters to the Performance Measurement and Evaluation Committee. Linkages are in place between ESDC’s performance measurement regime and the evaluation function. The HoE reports to the Committee on the implementation of management action plans. Reports have been produced on the state of performance measurement in support of evaluations, and these have been presented to the proper oversight committees.

1.8.2 Monitoring and Performance Reporting by the Deputy Head: Conforms to Expectations

Actions taken by ESDC’s Deputy Head have assured that the department’s evaluation function conforms to policy expectations. This neutral assessment is a component of the performance reporting activity. Also, the HoE provides TBS with information annually on behalf of the Deputy Head.

1.9 Recommended Action

The Evaluation Directorate should ensure that evaluation reports include information on the limitations of the methodology and the resulting impacts of those limitations on the findings presented.

2. Framework for the Neutral Assessment

Section A: Governance

There is conformance with the TB Policy on Results requirements regarding Governance

Section A.1: Deputy Head

There is conformance with the TB Policy on Results requirements regarding the responsibilities of the Deputy Head

Section A.1.a: The deputy head has fulfilled his/her responsibilities in regard to the evaluation function
  1. Establishing and maintaining a robust, neutral evaluation function;
    Reference: PFootnote 1 4.3.11
Section A.1.b: The deputy head has fulfilled his/her responsibilities in regard to the Departmental Performance Measurement and Evaluation Committee
  1. Identifying and chairing a committee of senior officials (the Performance Measurement and Evaluation Committee) to oversee departmental performance measurement and evaluation;
    Reference: P 4.3.3
  2. Ensuring that the Performance Measurement and Evaluation Committee, the head of performance measurement, and the head of evaluation have access to the departmental information needed to undertake the (performance measurement and) evaluation activities resulting from this policy and its related instruments;
    Reference: R 4.3.13
Section A.1.c: The deputy head has fulfilled his/her responsibilities in regard to the head of evaluation
  1. Designating a head of evaluation, at an appropriate level, responsible for leading the evaluation function;
    Reference: P 4.3.12
  2. Demonstrates in his or her work the competencies set by the Secretary of the Treasury Board;
    Reference: P 4.3.12
  3. Who has direct, unencumbered access to the deputy head:
    Reference; P 4.3.12
  4. Ensuring that (the Performance Measurement and Evaluation Committee, the head of performance measurement, and) the head of evaluation have access to the departmental information needed to undertake the (performance measurement and) evaluation activities resulting from this policy and its related instruments;
    Reference: P 4.3.13
Section A.1.d: The deputy head has fulfilled his/her responsibilities in regard to the operations of the evaluation function
  1. The deputy head is responsible for annually approving and providing to the Treasury Board of Canada Secretariat, in the manner and timing prescribed by the Secretariat, a five-year, rolling departmental evaluation plan;
    Reference: P 4.3.15
  2. Ensuring that the planned five-year evaluation coverage described in 4.3.15.2 to 4.3.15.7 of this policy is released annually as prescribed by the Treasury Board of Canada Secretariat;
    Reference:
    P 4.3.16
  3. Approving evaluation reports and summaries in a timely manner and providing them to the Treasury Board of Canada Secretariat as prescribed by the Secretariat;
    Reference: P 4.3.17

Section A.2: Departmental Performance Measurement and Evaluation Committee

There is conformance with the TB Policy on Results requirements regarding the departmental Performance Measurement and Evaluation Committee

Section A.2.a: The departmental evaluation committee has fulfilled its role and responsibilities
  1. Responsibilities of the Departmental Performance Measurement and Evaluation Committee. Reviewing and advising the deputy head on departmental evaluation planning and activities, including the adequacy of the following:
    Reference: DFootnote 2 4.1.2
    • Annual evaluation needs assessment exercises, evaluation coverage and resources for evaluation;
    • The departmental evaluation plan;
    • Plans for specific evaluations, if requested by the head of evaluation;
  2. Reviewing evaluation reports and summaries, including management responses and action plans, and recommending approval to the deputy head;
    Reference: D 4.1.3
  3. Monitoring follow-up on evaluation recommendations and action plans and advising the deputy head of any issues;
    Reference: D 4.1.4
  4. Reviewing and advising the deputy head on the availability, quality, utility, and use of performance information including evaluation;
    Reference: D 4.1.5
  5. Reviewing and advising the deputy head on the neutral assessment of the evaluation function.
    Reference: D 4.1.6

Section A.3: Head of Evaluation

There is conformance with the TB Policy on Results requirements regarding the head of evaluation

Section A.3.a: The head of evaluation has fulfilled his/her responsibilities in regard to governance
  1. Evaluations are conducted in a neutral manner and with integrity in their relationships between evaluators and stakeholders;
    Reference: D Appendix C 2.2.1.11

Section B: Professional Practice

There is conformance with the TB Policy on Results requirements regarding Professional Practice

Section B.1: Departmental Evaluation Plan

There is conformance with the TB Policy on Results requirements regarding the Departmental Evaluation Plan

Section B.1.1: Evaluation Plan

There is conformance with the TB Policy on Results requirements regarding the evaluation plan

Section B.1.1.a: Adequate measures are in place for the development of the evaluation plan
  1. The deputy head is responsible for annually approving and providing to the Treasury Board of Canada Secretariat, in the manner and timing prescribed by the Secretariat, a five-year, rolling departmental evaluation plan that:
    Reference: P 4.3.15
    1. Is informed by an annual evaluation planning exercise that includes consultation with the Treasury Board of Canada Secretariat;
    2. Clearly presents planned evaluation coverage, including coverage of organizational spending and the programsFootnote 3 in the Program Inventory, during the planning period;
    3. Identifies and provides a rationale for the organization’s spending and programs in the Program Inventory that are not scheduled for evaluation during the planning period;
    4. Includes evaluations of all ongoing programs of grants and contributions that have five-year average actual expenditures of $5 million or greater per year;
    5. Includes all evaluations required by applicable legislation and as a result of commitments in submissions approved by the Treasury Board of Canada;
    6. Includes all evaluations requested by the Secretary of the Treasury Board of Canada; and
    7. Includes all evaluation activities required to support centrally-led evaluations or resource alignment reviews;
  2. The deputy head is responsible for ensuring that their department undertakes specific evaluations and participates in centrally- led evaluations requested by the Secretary of the Treasury Board of Canada, paying any associated costs out of the department’s reference level;
    Reference: P 4.6.2
  3. The head of evaluation is responsible for: Assessing evaluation needs through an annual departmental evaluation planning exercise and developing an annual five-year departmental evaluation plan, in accordance with subsection 2.2 of the Mandatory Procedures for Evaluation:
    Reference: D 4.4.5
    1. Evaluations of all ongoing programs of grants and contributions with five-year average actual expenditures of $5 million or greater per year, as per subsection 2.5 of the policy, required to fulfill the Financial Administration Act subsection 42.1;
      Reference: D Appendix B. 2.2.1.1
    2. Evaluations required by applicable legislation and as a result of commitments in submissions approved by the Treasury Board of Canada;
      Reference: D Appendix B. B.2.2.1.2
    3. Evaluations requested by the Secretary of the Treasury Board of Canada;
      Reference: D Appendix B. 2.2.1.3 and,
    4. Evaluation activities required to support centrally-led evaluations or resource alignment reviews.
      Reference: D Appendix B. 2.2.1.4
      Additionally, ensuring that the annual departmental evaluation planning exercise, starting from the principle that all programs and spending not covered by evaluation as a result of subsection B.2.2.1 of this Mandatory Procedure should be evaluated periodically, identifies the department’s five-year evaluation coverage needs by considering:
      Reference: B. 2.2.2
    5. The need to evaluate ongoing programs of grants and contributions with five-year average actual expenditures of less than $5 million per year and assessed contributions to international organizations;
      Reference: D Appendix B. 2.2.2.1
    6. Departmental and program risks, governmental and department priorities and the information needs of major stakeholders;
      Reference: D Appendix B. 2.2.2.2
    7. Information needed to support the department’s spending proposals, including Memoranda to Cabinet and Treasury Board submissions, as well as to support resource alignment reviews;
      Reference: D Appendix B. 2.2.2.3
    8. The recentness and sufficiency of existing performance information for management, decision making, and accountability;
      Reference: D Appendix B. 2.2.2.4
    9. Joint evaluations with other departments of government priorities, horizontal initiatives, and where departmental responsibilities, results or program outcomes are shared or related;
      Reference: D Appendix B. 2.2.2.5
    10. Other planned departmental oversight activities;
      Reference: D Appendix B. 2.2.2.6
    11. Advice from the Treasury Board of Canada Secretariat; and,
      Reference: D Appendix B. 2.2.2.7
    12. Evaluations of high-risk and high-priority spending, programs or other issues;
      Reference: D Appendix B. 2.2.4.3.
  4. The annual five-year departmental evaluation plan identifies:
    1. For each evaluation scheduled for approval in the first year of the plan, the quarter in which it will be approved;
      Reference: D Appendix B. 2.2.4.4
    2. The lead department for each evaluation that will be conducted jointly with other departments;
      Reference: D Appendix B. 2.2.4.5
    3. Extent of planned coverage, including by amount of organizational spending and by program of the Program Inventory;
      Reference: D Appendix B. 2.2.4.6
    4. Extent of organizational spending and programs that will not be evaluated in the planning period, the key reasons for not evaluating, and the year in which the spending or program was last evaluated;
      Reference: D Appendix B. 2.2.4.7
  5. The head of evaluation is responsible for consulting with the Treasury Board of Canada Secretariat when developing annual five-year departmental evaluation plans and on the identification of programs of grants and contributions for use in monitoring.
    Reference: D 4.4.14
Section B.1.1.b: The evaluation plan is approved and communicated
  1. The head of evaluation is responsible for: Following review and recommendation for approval by the Performance Measurement and Evaluation Committee, obtaining the approval of the deputy head for the departmental evaluation plan;
    Reference: D 4.4.6
  2. The head of evaluation is responsible for: Submitting to the Treasury Board of Canada Secretariat the annual departmental evaluation plan and releasing, as required by subsection 4.3.16 of the policy, the planned evaluation coverage described in subsection 4.3.15, excluding subsection 4.3.15.1;
    Reference: D 4.4.7
  3. Mandatory Text for the Deputy Head Departmental Evaluation Plan Confirmation Note.
    Reference: D Appendix B 2.3

Section B.2: Nature of Evaluation

There is conformance with the TB Policy on Results requirements regarding the Nature of Evaluation

Section B.2.1: Evaluation Roles and Responsibilities

There is conformance with the TB Policy on Results requirements regarding the Evaluation Function’s roles and responsibilities

Section B.2.1.a: The evaluation function has fulfilled its responsibilities in regard to conducting evaluations
  1. Evaluations of ongoing programs of grants and contributions with five-year average actual expenditures of $5 million or greater per year required to fulfill the requirements of the Financial Administration Act section 42.1, shall include an assessment of relevance, effectiveness and efficiency;
    Reference: D Appendix B 2.2.5
Section B.2.1.b: The evaluation function has fulfilled its responsibilities in regard to departmental performance measurement
  1. The head of evaluation is responsible for advising program officials on the availability, quality, validity, and reliability of the indicators and information in the Performance Information Profile, including their utility for evaluation;
    Reference: D 4.4.2
  2. The head of evaluation is responsible for supporting program officials in verifying for each relevant Memorandum to Cabinet and Treasury Board submission that plans for performance information and evaluations are sufficient and that information on past evaluations is accurately represented and balanced;
    Reference: D 4.4.1
  3. The head of evaluation is responsible for advising program officials on the availability, quality, validity, and reliability of the indicators and information in the Performance Information Profile, including their utility for evaluation;
    Reference: D 4.4.3

Section B.3: Evaluation Project Process

There is conformance with the TB Policy on Results requirements regarding the Evaluation project process

Section B.3.1: Managing the Evaluation Function

There is conformance with the TB Policy on Results requirements regarding managing the Evaluation Function

Section B.3.1.a: Adherence to the Mandatory Procedures for Evaluation and the Standard on Evaluation
  1. Ensuring that evaluations are undertaken in accordance with the Mandatory Procedures for Evaluation and the Standard on Evaluation;
    Reference: D 4.4.13
Section B.3.2: Evaluation Project Planning Phase

There is conformance with the TB Policy on Results requirements regarding the Planning Phase

Section B.3.2.a: Evaluation project plans are developed and recorded
  1. Evaluations are planned with consideration of using relevance, effectiveness and efficiency as primary evaluation issues, where relevant to the goals of the evaluation;
    Reference: D Appendix C 2.2.1.5
  2. Evaluations are designed and managed so as to deliver value for money for Canadians in terms of the use of evaluation resources;
    Reference: D Appendix C 2.2.1.12
  3. Evaluations are planned with consideration of the risks and complexity associated with the policy, program, priority, unit or theme being evaluated;
    Reference: D Appendix C 2.2.1.4
  4. Evaluations have a clear, robust approach, design and methodology that is documented at the outset of the evaluation, and that allows for the collection and analysis of valid, reliable data;
    Reference: D Appendix C 2.2.1.8
  5. Evaluations include multiple lines of quantitative and qualitative evidence to support findings and conclusions;
    Reference: D Appendix C 2.2.1.10
  6. Evaluations are carried out by an evaluator or evaluation team that possesses the knowledge and competence required and that adopts roles and responsibilities that are articulated in writing at the outset of the evaluation;
    Reference: D Appendix C 2.2.1.2
  7. Evaluations have objectives and purposes that are clearly stated at the outset of the evaluation and that meet the needs of evaluation users;
    Reference: D Appendix C 2.2.1.3
  8. Evaluations are planned to take into account government-wide policy considerations, where relevant, such as gender-based analysis and official languages.
    Reference: D Appendix C 2.2.1.6
Section B.3.2.b: Evaluation projects are coordinated with Communications
  1. Informing the head of communications of survey research included in the departmental evaluation plan, at least for the first year of the plan, before departmental evaluation plans are approved.
    Reference: D 4.4.15
Section B.3.3: Conduct of Evaluation Project Phase

There is conformance with the TB Policy on Results requirements regarding the conduct of evaluations

Section B.3.3.a: Leading evaluation projects
  1. Evaluations are directed by the head of evaluation;
    Reference: D Appendix C 2.2.1.1
  2. Evaluations are guided, where relevant, by an advisory or steering committee chaired by the head of evaluation or a delegate;
    Reference: D Appendix C 2.2.1.7
Section B.3.3.b: Stakeholder involvement in evaluation projects
  1. Evaluations include sufficient and appropriate consultation with major stakeholders;
    Reference: D Appendix C 2.2.1.9
Section B.3.3.c: Evaluation projects are managed and controlled
  1. Evaluations should use effective project management strategies;
    Reference: JCSEEFootnote 4 F1
  2. Evaluations should use resources effectively and efficiently;
    Reference: JCSEE F4
  3. Evaluations should account for all expended resources and comply with sound fiscal procedures and processes;
    Reference: JCSEE P7
  4. Contracted external evaluators comply with the Mandatory Procedures for Evaluation and this Standard, and are free from any actual or perceived conflict of interest;
    Reference: D Appendix C 2.2.3
  5. The requirements set out in subsection C.2.2.3 are reflected in all contracting agreements with external evaluators;
    Reference: D Appendix C 2.2.4
  6. Actual or perceived conflicts of interest that arise with regard to evaluations are addressed promptly so as not to compromise evaluation processes or results;
    Reference: D Appendix C 2.2.5
Section B.3.3.d: There is sufficient and appropriate evidence to achieve the evaluation project objectives and to support conclusions and engagement results
  1. Evaluators must identify, analyze, evaluate, and document sufficient information to achieve the project’s objectives;
    Reference: Based on IIAFootnote 5 2310
  2. Evaluators must identify sufficient, reliable, relevant, and useful information to achieve the engagement’s objectives;
    Reference: Based on IIA 2320
  3. Evaluations are subjected to quality assurance including peer reviews where appropriate;
    Reference: D Appendix C 2.2.1.13
Section B.3.3.e: Relevant information is documented, maintained and controlled
  • Individuals involved in evaluations are informed of the level of confidentiality and privacy that is afforded them under the Privacy Act;
    Reference: D Appendix C 2.2.2
  • Evaluators must document relevant information to support the conclusions and engagement results;
    Reference: Based on IIA 2330
  • The head of evaluation must develop retention requirements for engagement records, regardless of the medium in which each record is stored. These retention requirements must be consistent with the organization’s guidelines and any pertinent regulatory or other requirements.
    Reference: Based on IIA 2330.A2
Section B.3.4: Reporting Phase

There is conformance with the TB Policy on Results requirements regarding the Reporting Phase

Section B.3.4.a: Evaluation reports are complete and appropriately communicated
  1. The head of evaluation is responsible for: Obtaining the deputy head’s approval of evaluation reports and summaries;
    Reference: D 4.4.8
  2. Evaluation reports:
    Reference: D Appendix C.2.2.6
    1. Are written and presented clearly and concisely;
    2. Include the information required to understand and reasonably sustain findings and conclusions;
    3. Provide readers with an appropriate context for the evaluation and the policy, program, priority, unit or theme being evaluated;
    4. Identify the limitations of the evaluation in a way that informs readers about the reliability of findings and conclusions;
    5. For evaluations of program outcomes or outputs, include an accurate assessment of the contribution of the program to its related government priorities and/or departmental results and priorities;
    6. Present a logical flow of findings, conclusions and recommendations;
    7. Identify any exposure to risks noted through the evaluation;
    8. Include clear, actionable recommendations that aim to address the key issues or concerns identified;
    9. Include a management response and action plan prepared by the responsible program official or relevant manager(s).
  3. Draft evaluation reports are submitted by the head of evaluation directly and simultaneously to the deputy head and the Performance Measurement and Evaluation Committee;
    Reference: D Appendix B 2.2.6
  4. Submitting evaluation reports and summaries to the Treasury Board of Canada Secretariat and releasing on web platforms as required by subsections 4.3.17 and 4.3.18 of the policy;
    Reference: D 4.4.9
  5. Evaluation reports and summaries, including complete management responses and actions plans, are released on web platforms, after they are approved by the deputy head as per subsection 4.3.18 of the policy, while safeguarding protected and classified information.
    Reference: D Appendix B 2.2.8
  6. Final approved evaluation reports are submitted to the Treasury Board of Canada Secretariat, as per subsection 4.3.17 of the policy, with a cover letter signed by the head of evaluation indicating the report’s approval date.
    Reference: D Appendix B 2.2.7
  7. Evaluations are completed and approved in a timely manner so that they can be used appropriately, effectively and efficiently.
    Reference: D Appendix C 2.2.1.14
Section: B.3.4.b: Evaluation reports are accessible to the public
  1. The deputy head ensures that that evaluation reports and summaries, including complete management responses and action plans, are released on web platforms as prescribed by the Treasury Board of Canada Secretariat;
    Reference: P 4.3.18
Section B.3.5: Use of Evaluations

There is conformance with the TB Policy on Results requirements regarding the use of evaluation information

Section B.3.5.a: Evaluation information is used
  1. The Deputy head is responsible for ensuring that Treasury Board submissions are supported by Performance Information Profile data and evaluation information;
    Reference: P 4.3.10
  2. The Deputy head is responsible for verifying, in writing, performance measurement and evaluation information contained in or appended to each Treasury Board submission for new or renewed spending or for major revision to authorities;
    Reference: P 4.3.14

Section B.4: Evaluation Competencies

There is conformance with the TB Policy on Results requirements regarding the proficiency and professional care

Section B.4.a: The head of evaluation ensures that evaluation resources have the training, competencies and experience to perform their responsibilities
  1. The head of evaluation is responsible for: Demonstrating in their work the competencies for heads of evaluation as established by the Secretary of the Treasury Board of Canada;
    Reference: D 4.4.10
  2. The head of evaluation is responsible for: Ensuring that departmental evaluators demonstrate in their work the competencies for evaluation specialists established by the Secretary of the Treasury Board of Canada;
    Reference: D 4.4.11
  3. The head of evaluation is responsible for: Ensuring that departmental evaluators have opportunities to develop their competencies and to earn evaluation-related designations or certifications from recognized professional associations and certifying bodies;
    Reference: D 4.4.12

Section C: Monitoring and Performance Reporting

There is conformance with the TB Policy on Results requirements regarding Monitoring and Performance Reporting

Section C.1: Head of Evaluation’s Annual Reporting

Section C.1.a: The head of evaluation has reported at least annually to the departmental Performance Measurement and Evaluation Committee
  1. The head of evaluation is responsible for reporting to the Performance Measurement and Evaluation Committee at least annually on the following:
    Reference: D 4.4.4
    1. The implementation of approved management action plans in response to evaluation recommendations;
    2. The impacts of evaluations including lessons learned, corrective actions taken and influence on resource allocation decisions;
    3. The delivery of evaluations as set out in the departmental evaluation plan and, for approved evaluations, the timeliness of transmission to the Treasury Board of Canada Secretariat and of public release; and
    4. The availability, quality, utility and use of performance information to support evaluations;

Section C.2: Deputy Head

There is conformance with the TB Policy on Results requirements regarding Monitoring and Performance Reporting

Section C.2.a: The deputy head has fulfilled his/her responsibilities in regard to monitoring and performance reporting
  1. The deputy head is responsible for ensuring that their department’s adherence to this policy and its associated instruments is monitored;
    Reference: P 4.6.1.1
  2. The deputy head is responsible for ensuring the conduct of a neutral assessment of the evaluation function at least once every five years;
    Reference: P 4.3.19
  3. The deputy head is responsible for ensuring, where appropriate, that proposals are developed to address such difficulties, gaps and compliance issues;
    Reference: P 4.6.1.2 and
  4. Bringing any significant difficulties, gaps or compliance issues and proposals to address them to the attention of the Secretary of the Treasury Board of Canada;
    Reference: P 4.6.1.3
Section C.2.b: The deputy head has the information to respond to the Secretary of the Treasury Board
  1. The deputy head is responsible for ensuring that the Treasury Board of Canada Secretariat is, upon request, provided with organizational, performance measurement and evaluation information as prescribed by the Secretariat, including to support resource alignment reviews and centrally-led evaluations;
    Reference: P 4.3.9
  2. The deputy head is responsible for ensuring that their department provides the Treasury Board of Canada Secretariat with any information requested on the departmental performance measurement function, evaluation function or evaluation coverage, or on the quality, use, cost and timeliness of performance measurement and evaluation in the department;
    Reference: P 4.6.4

Page details

Date modified: