Evaluation of the Treasury Board of Canada Secretariat Values and Ethics Program

Program background

In 2007, the Public Servants Disclosure Protection Act (PSDPA) came into effect. The PSDPA required each department head to:

• “establish a code of conduct applicable to the portion of the public sector for which he or she is responsible”^{Footnote 4}
• “establish internal procedures to manage disclosures made under this Act by public servants employed in the portion of the public sector for which the chief executive is responsible”^{Footnote 5}
• “designate a senior officer to be responsible for receiving and dealing with…disclosures of wrongdoings made by public servants”^{Footnote 6}

As part of complying with the PSDPA, the Values and Ethics Code for the Public Sector (the public sector code) was launched in 2012. The public sector code outlines the standard of conduct for public servants when they carry out their professional duties. The public sector code states that “values are a compass to guide public servants in everything they do.” The 5 guiding values for the public service are:

1. respect for democracy
2. respect for people
3. integrity
4. stewardship
5. excellence

Ethics are not defined in the public sector code, but the code states that by adhering to the 5 guiding values, “public servants strengthen the ethical culture of the public sector and contribute to public confidence in the integrity of all public institutions.”

Adherence to the public sector code is a condition of employment for all public servants.

In compliance with the PSDPA, TBS created its Departmental Code of Conduct in 2012. The Secretary designated:

• the Associate Secretary as the senior disclosure officer
• protocols for the disclosure of wrongdoing

The Departmental Code of Conduct reflects the 5 values outlined in the public sector code and provides contextual information on their interpretation and application within TBS.

Program governance and resources

The V&E program, the V&E network and the V&E champion role were created at TBS in 2009. The Associate Secretary^{Footnote 7} is the V&E champion and is responsible for chairing the network and promoting the program. The V&E program falls under the Office of the Ombudsman, Ethics and Conflict Resolution in the Human Resources Division (HRD).

The Director General of HRD is responsible for the V&E program budget. The program’s annual budget is $142,000. The program is run by 2 full-time equivalents who deliver all the V&E activities. They report to the Director of Ethics and Conflict Resolution, who reports to the Director General of HRD. The Director of Ethics and Conflict Resolution also serves as the Ombudsman at TBS and reports to the Associate Secretary in this role.

In 2012, TBS established its first V&E plan. TBS is currently implementing its 2016 to 2019 plan. In 2016, managing conflict of interest was delegated to the Ombudsman Office of the Ombudsman, Ethics and Conflict Resolution.^{Footnote 8}

Stakeholders and target audience

The V&E program targets all TBS employees. Mandatory training is targeted to executives, managers and supervisors at all levels. The program also offers advice to TBS retirees and contractors on request.

Values and ethics program objectives, activities and outcomes

The Values and Ethics (V&E) Program sets out the direction and activities that support the implementation of the Departmental Code of Conduct.

The objectives of the V&E program are to:

• promote the application of and adherence to:
  • the Values and Ethics Code for the Public Sector
  • TBS’s Departmental Code of Conduct
  • the Policy on Conflict of Interest and Post-Employment
• increase the visibility of leaders in promoting, modelling, and highlighting values and ethics
• communicate policies, expectations and guidance
• increase awareness of and confidence in the available workplace values and ethics processes and mechanisms

In concert with the Office of the Ombudsman, Ethics and Conflict Resolution, V&E officers act to provide a single service window to employees seeking assistance on issues related to:

• ethics
• harassment
• conflict management
• internal disclosure of wrongdoing

The activities implemented in support of the outcomes of the V&E program fall under 3 themes:

1. training and awareness
2. compliance and monitoring
3. advice and recommendations

Training and awareness includes activities that raise awareness and provide context-specific guidance for the application of values and ethics at TBS. Due to the access that program staff have to employees through the delivery of in-person training, the program staff often include complementary topics that are not part of the V&E program, such as fraud and internal disclosure of wrongdoing.^{Footnote 9}

Compliance and monitoring activities follow up on conflict of interest declarations and departmental reporting, such as the triennial Ethical Climate Survey.

Advice and recommendations activities provide tailored, on-demand advice and interpretation of V&E codes and policies to employees and management.^{Footnote 10}

The V&E program delivers these activities with the expectation that they will lead to the following immediate and intermediate outcomes:

• All employees know how to access advice and assistance on ethical issues
• All employees have enhanced knowledge, awareness, and compliance of public sector values and ethics
• Public sector values guide decision-making and support a respectful work environment
• Responsibility for values and ethics at TBS is shared by management and employees
• There is reduced risk of conflict of interest situations
• TBS has an ethical organizational culture^{Footnote 11}

The desired long-term outcome is a public service that is professional, non-partisan, and highly ethical and that maintains public trust.

Findings: direct program outcomes

Rapid impact evaluation (RIE) defines direct outcomes as “effects that occur directly as a result of the program.”^{Footnote 13} The evaluation findings for the direct program outcomes of the Values and Ethics (V&E) Program are as follows.

Findings: long-term impact

A public service that:

• is professional, non-partisan, highly ethical
• maintains public trust

Conclusion

The evaluation concluded that some elements of the V&E program contributed to TBS having an ethical public service.

The comparison of the V&E program with the counterfactual determined that the tailored advice and recommendations activities and, albeit to a lesser extent, the department-specific training and awareness activities contribute to TBS having an ethical public service. For the V&E program, this underscores the program’s reliance on highly experienced employees to deliver advice, recommendations and training.

Subject matter experts stated that the role of the champion would likely have little impact on the outcomes of the V&E program, while the role of the Ombudsman could have a moderate impact on its outcomes. If the individuals in these roles are visible and are perceived to be role models of ethical behaviour, they would set a positive tone for the organization. If they are not visible or if they are perceived to be unethical, they would negatively impact both the V&E program and the ethical climate of the organization.

Recommendations

1. The V&E program should:
   • define its outcomes (the development of a logic model is suggested for this exercise)
   • revise the TBS Values and Ethics Plan 2016-19, as required
2. In light of the importance that values and ethics discussions and behaviours shown by leadership have on employees in setting the tone for values-based decision-making and ethical behaviour, the V&E program should reinforce the practice of mandatory training for all executives to strengthen their leadership role.
3. Given that the Ombudsman role supports the program’s outcomes and sets a positive tone within the organization, the V&E program should continue to strengthen the Ombudsman role, for example, by establishing a feedback loop to demonstrate that issues brought forward to the Ombudsman are addressed.

Rapid Impact Evaluation (RIE)

A structured way to gather expert assessments of a program’s impact.^{Footnote 20}

RIE engages a wide spectrum of experts in order to:

• increase the acceptance and adoption of findings
• provide a balanced perspective of a program’s impacts to inform executives, managers and program analysts

Key benefits of RIE

• It is relatively quick and low cost: RIE can reduce an evaluation’s duration by up to half
• It does not require pre-existing performance data: this makes it flexible enough to be used during program design, implementation and close-out
• It provides a clear and quantified summary of expert assessments of the program’s impact relative to a counterfactual: this compares 2 situations that are otherwise similar, except for the intervention
• It prioritizes the engagement of external perspectives, which can:
• bring an important viewpoint that is not always included in federal evaluations
• increase the perceived legitimacy and accuracy of the evaluation

3 key steps for the evaluation team

The evaluation team:

1. creates a detailed program summary with all parties to ensure that:
   • the program is understood by everyone
   • the immediate and intermediate outcomes of the program are delineated
   • the list of stakeholders is appropriate and inclusive
   • the proposed counterfactual is feasible, legal and plausible
   • the evaluation questions will meet the needs of the evaluators and of the program
2. uses the program summary and engages experts to determine the degree to which the program attained its outcomes, relative to the counterfactual:
   • What is the likelihood of the outcomes occurring in the program versus the counterfactual?
   • What is the magnitude of the outcomes in the program versus the counterfactual?
   • How important is each immediate outcome to the likelihood of the intermediate outcomes in the program versus the counterfactual?
3. verifies the views set out by the experts by:
   • synthesizing the expert assessments
   • compiling and analyzing additional data sources
   • writing a final report and preparing recommendations

RIE is the ideal approach when:

• evaluation results are needed quickly
• knowledge of impacts is desired but the program’s low materiality and low risk do not warrant a traditional evaluation
• the shared production of knowledge by users and experts increases the salience, legitimacy and credibility of the knowledge
• there is a benefit to comparing the impact of 2 or more possible versions of a program, such as during program design or renewal
• there is a need for perspectives outside the program or government
• the focus is on the assessment of immediate and intermediate outcomes

Figure 1 - Text version

Rapid impact evaluation is divided into 4 phases and each phase is composed of one or more steps.

Phase 1, “plan the evaluation,” is composed of the following 6 steps.

• Step 1: develop the terms of reference
• Step 2: obtain any necessary approvals
• Step 3: plan any other lines of evidence that will be used
• Step 4: identify experts that will be consulted
• Step 5: complete a privacy impact assessment, if needed
• Step 6: begin the contracting process, if needed

Phase 2, “develop the program summary,” is composed of the following step.

• Step 1: In consultation with the technical advisor and relevant program stakeholders, develop the program summary, including the:
  • program profile
  • theory of change
  • direct outcomes and intended impacts
  • program stakeholder map
  • times and locations of the program and each effect
  • scenario-based counterfactuals

Phase 3, “gather assessments of incremental impact,” is composed of the following step.

• Step 1: engage with 3 groups of experts to obtain their assessments of the probability and magnitude of each outcome relative to a scenario-based counterfactual

Phase 4, “expert assessments are analyzed to generate an estimate of the program’s overall impact,” is composed of the following 3 steps.

• Step 1: analyze the data, including calculating the impact
• Step 2: verify the quality of the data
• Step 3: report the results

Description of the values and ethics program counterfactual

The Government of Canada Values and Ethics Code for the Public Sector (the public sector code) remains in place and the Treasury Board of Canada Secretariat (TBS) continues to have its own Departmental Code of Conduct. The deputy head of TBS is responsible and accountable for implementing the public sector code, the Departmental Code of Conduct, and the Policy on Conflict of Interest and Post-Employment. There is a values and ethics (V&E) officer, based in the Human Resources Division, who responds to inquiries and provides advice as requested. There is also a V&E champion.

Affirmation and conflict of interest forms are attached to all letters of offer. Employees are instructed to read hyperlinks on the public sector code, the Departmental Code of Conduct, and the Policy on Conflict of Interest and Post-Employment. They are also instructed to bring private affairs to the attention of their manager if those affairs might be in a conflict of interest with their duties, so that their manager can assist them to resolve and mitigate any conflict.

Standardized Government of Canada training by the Canada School of Public Service is exclusively offered online and is mandatory for managers and executives. There is no departmental network to follow up on training or provide supplementary information.

The Ethical Climate Survey is not implemented and there is no TBS values and ethics plan. Values and ethics results from the Public Service Employment Survey and Public Service Employee Annual Survey are shared with senior management.

Stakeholders

The V&E champion at TBS is the Associate Secretary. On TBS’s internal website, the champion’s role is described as follows:

The Champion of Values and Ethics promotes the ongoing discussion of values and ethics throughout the Secretariat and ensures that employees receive the support and assistance they need to both talk about and resolve the issues that are of concern to them.

As part of ethical leadership, senior management, managers and supervisors are required to “communicate effectively and frequently the expected ethical behaviours, encouraging and promoting the shared responsibility of all TBS employees to foster a values-based culture and a workplace free from harassment, discrimination and bullying.”^{Footnote 21} They are required to take V&E training. All executives are required commit to the values expressed in the Every Day commitment, and their performance is assessed against this criteria.

The TBS Departmental Code of Conduct states:

TBS employees have a shared responsibility, namely, to serve the public interest and ensure public trust. TBS employees are required to become familiar with the provisions of the public sector code and the TBS Departmental Code of Conduct and be able to apply these values in their work.

The V&E network of sector representatives is an informal group that supports the V&E office in advancing public sector values and ethics at TBS, within the context of:

• the TBS mandate and main responsibilities
• the corporate culture
• TBS-specific challenges and needs^{Footnote 22}

Contractors and consultants are third party individuals or companies who provide goods or services to TBS. They are expected to abide by the spirit of the TBS Departmental Code of Conduct. Conflict of interest guidelines apply to contractors and consultants. They should inform TBS of any other contracts that they may have with other department(s) that may be perceived as a conflict of interest.

Program activities

Training and awareness activities:

• provide values and ethics training to all employees
• respond to ad hoc inquiries
• incorporate ancillary ethical issues in training, such as fraud, political or outside of work activities, and wrongdoing
• use evidence from monitoring activities to regularly review and update training material to ensure relevance
• promote a “do right” ethical culture at every opportunity throughout the organization

Compliance and monitoring activities:

• collect, review and follow up on all conflict of interest declarations (including the annual renewal of affirmations via pop ups on employee computer screens)
• prepare recommendations as required, depending on the contents of conflict of interest declarations^{Footnote 23}
• monitor and assist employee compliance with the public sector code and the TBS Departmental Code of Conduct
• fulfill the reporting and compliance requirements of the Management Accountability Framework
• administer the triennial Ethical Climate Survey

Advice and recommendations activities:

• are considered annually as:
• one of the corporate risk categories
• part of the corporate ethical risk analysis
• provide on-demand policy advice and interpretation of the public sector code, the Policy on Conflict of Interest and Post-Employment, and the TBS Departmental Code of Conduct
• support the deputy head of TBS, the V&E network and the V&E champion
• support the resolution of individual and systemic concerns by bringing issues forward to those with the authority to act