Review of the Open Government Initiative

Draft: November 19, 2021

Internal Audit and Evaluation Directorate

Table of Contents

• Executive Summary
• Introduction
• Findings and considerations
• Conclusion
• Appendix A: Open Government Secretariat Response
• Appendix B: Maturity Model Self-Assessment Results

Executive summary

Background

Since 2015, the Public Service Commission of Canada (PSC) has been developing plans to support Canada’s contribution to the open government agenda. The PSC’s Executive Management Committee expects that the organization will work to strengthen its position as a leader in open government. The PSC intends to create efficiencies by proactively releasing data and information, reducing the need for Canadians to make use of the access to information process. It is also exploring tools to engage Canadians to better understand their needs and interests and to target its open government efforts accordingly. It is working to empower employees with the tools, knowledge and support they need to proactively identify, develop and publish PSC assets to the Open Government Portal with purpose, meaning that they are of value and informed by public interest.

Objective of the review

The review was conducted at the request of the PSC’s Open Government Secretariat to support it in developing tools and establishing a baseline to measure the progress of the PSC’s Open Government Initiative. As part of the review, the audit team assessed and confirmed the validity of the secretariat’s maturity model, and  conducted an awareness survey. This review will support the next PSC Open Government Implementation Plan, currently being developed by the secretariat. The plan will include activities, deliverables, commitments and objectives for the next 2 years, as well as an action plan.

Statement of conformance

The engagement is in conformance with the Internal Audit Standards for the Government of Canada as supported by the results of the Quality Assurance and Improvement Program.

Acknowledgements

The audit team would like to thank those who collaborated in this effort to highlight the PSC’s strengths and opportunities for improvement related to open government.

Key takeaways

• The architecture of the maturity model is adequate and follows best practices. The model is simple and scaled to the size of the PSC.
• The content of the maturity model captures the key objectives outlined in the Open Government Implementation Plan, as well as requirements of the Directive on Open Government. The maturity model will help identify the desired level of optimization at the PSC and measure progress against it.
• The self-assessment of PSC maturity conducted by the Open Government Secretariat is accurate and supported by enough evidence. The PSC is between level 2 (fostering transparency) and 3 (open participation). Detailed results can be found in Appendix B.
• The Open Government Secretariat succeeded in establishing a base of employees who are familiar with open government roles, responsibilities and requirements, and can identify assets to be published. However, developing an open-by-default culture will require continued efforts over the coming years to promote aspects of open government initiatives among employees. Most employees are still unfamiliar with publishing processes, roles and responsibilities and technical requirements.
• Awareness about the importance of publishing content on the Open Government Portal is high, but very few PSC employees have done it. The survey showed that most respondents still have issues identifying value-added data and information. The Open Government Secretariat has worked to address this issue, including by identifying themes of interest for open government and launching a “publish with a purpose” pilot at Integration Committee.

Advice

Public Service Commission open government maturity model

• Communicate minimum requirements: When communicating results of the maturity model, the Open Government Secretariat should distinguish between Treasury Board of Canada Secretariat’s minimum requirements for compliance and additional improvements.
• Identify desired level of maturity: It is not always desirable or cost effective to reach a level 5 of maturity for all criteria. The Open Government Secretariat should consult with management to determine the desired level of maturity and discuss the requirements to reach that level.
• Improve terminology: The wording of some indicators could be improved to make it easier for stakeholders to understand what should be done to reach that specific level. Some improvements, such as clarifying the use of the term “target audience,” were already pointed out by the audit team to the Open Government Secretariat during this consultation.
• Include level 0: The Open Government Secretariat should consider adding a level 0 to the model to indicate when no work has been performed on a specific criterion.
• Conduct model testing: The Open Government Secretariat should use the consultation for the 2022–24 Open Government Implementation Plan to further test the model, collect feedback from stakeholders, and improve it based on the needs that will be identified.

Awareness

• Provide and communicate clear instructions on Open Government Initiative: Some respondents have raised concerns about their ability to identify documents and data of value that can be published on the Open Government Portal. Developing and sharing clear guidelines would help employees identify data and documents during their creation and facilitate the publication process. Employees should be able to easily access information about the initiative, how to identify potential candidates, training material, roles and responsibilities and other resources. Currently the information can only be found by searching for past open government posts under PSC Express on Intracom. Other federal organizations have developed specific pages on their intranet sites to support open government objectives.
• Focus awareness efforts on managers: To increase awareness among PSC employees, it is important to set the tone at the top. Developing an awareness strategy that focuses on key managers, instead of all PSC employees, can support the awareness efforts already underway. By promoting training and providing tools, managers will be better equipped to identify potential candidates for open government and to share insights about the process with their teams. This may have a trickle-down effect and lead to greater awareness.
• Conduct regular surveys: As part of benchmarking exercises, it is important to monitor progress. The Open Government Secretariat should conduct a survey at the end of each Open Government Implementation Plan to measure progress and inform future training and outreach activities.

Introduction

Open Government Partnership

The Open Government Partnership is an international initiative with the core objective of securing solid commitments from governments to promote transparency, empower citizens, fight corruption and harness new technologies to strengthen governance. The partnership consists of representatives of 78 countries, 20 local governments and many civil society organizations. These members have committed to the Open Government Declaration, which promotes openness in government, including by sharing more information about governmental activities. Members can achieve this ideal by complying with core eligibility criteria, passing a values check and developing action plans co-created with the public every 2 years, including concrete steps and commitments to promote open government initiatives.

Open government and the Canadian government

In 2012, the Government of Canada joined the Open Government Portal. Since then, the Treasury Board of Canada Secretariat has outlined the commitments and requirements for federal organizations to pursue open government activities. The Directive on Open Government states that the “objective (…) is to maximize the release of government information and data of business value to support transparency, accountability, citizen engagement, and socio-economic benefits through reuse, subject to applicable restrictions associated with privacy, confidentiality, and security.” The 4th National Action Plan on Open Government covers 2018 to 2020. At the time of this report (October 2021), Treasury Board of Canada Secretariat is consulting with Canadians to develop the 5th National Action Plan, after being delayed because of COVID-19. 

Open government at the Public Service Commission

Since 2015, the PSC has developed plans to support Canada’s contribution to the open government agenda. The Executive Management Committee expects that the PSC will work to strengthen its position as a leader in open government. The committee designated an open government champion in November 2016 to provide appropriate leadership and visibility to this initiative and to facilitate its successful implementation. The champion is supported by the Open Government Secretariat, composed of 3 analysts at the time of the consultation. Governance is ensured through the Joint Information Management /Information Technology Committee and Integration Committee, the Data and Open Government Advisory Board, and various working groups.

During the last few years, the secretariat conducted many awareness activities, including open houses, training sessions and communication pieces. Most recently, they have launched the “publish with a purpose” pilot at Integration Committee. The secretariat is currently working on the next Open Government Implementation Plan for 2022 to 2024, including an action plan to pursue open government objectives.

Open Public Service Commission objectives

• Create an open-by-default culture by raising awareness and promoting activities that support openness and transparency.
• Engage to provide value-added data and information by widening the channels of communication to solicit user feedback to identify valuable data and information to release.
• Optimize self-service through innovation by streamlining the release of and access to data.
• Support Government of Canada commitments and expectations by supporting the objectives in the Treasury Board of Canada Secretariat Directive on Open Government and the National Action Plan.

Consultation’s scope and approach

• Objective: The review was conducted at the request of the Open Government Secretariat to support it in developing tools and a baseline to measure the progress of the PSC’s Open Government Initiative. The review consisted of reviewing and confirming the validity of the maturity model developed by the secretariat, and conducting an awareness survey.
• Work performed:
• validated the maturity model developed by the Open Government Secretariat
• assessed the self-assessment against the maturity model
• conducted an awareness survey across the PSC to establish a baseline for measuring the progress of the Open Government Initiative in the following years
• Limitation: The scope of this consultation was established in collaboration with the Open Government Secretariat following the standards for consultative engagement. It excluded an overall assessment of the performance of the Open Government Initiative.
• Authority: Consulting engagement conducted in accordance with the approved PSC Internal Audit and Evaluation Plan 2021–23. This risk-level was identified as medium.

Methodology

• Document review: Key documents were reviewed, including policy, guidance, presentations, reports, performance information and other relevant documents.
• Literature review: The literature review included documents that provided further insight and perspectives on relevant open government issues and maturity assessment.
• Interview: An interview with the Open Government Secretariat was conducted to gain further insight and clarification on the Open Government Initiative.
• Survey: PSC employees were surveyed about the Open Government Initiative to establish a baseline and measure progress in the future.

Findings and considerations

Public Service Commission open government maturity model

Maturity model

Why it matters

A well-developed maturity model will help identify desired improvement for open government initiatives at the PSC and provide a useful tool to measure progress.

What the audit team did

The audit team identified the best practices that make a maturity model reliable and compared them with the PSC’s model, focusing on practices identified and promoted by the Institute of Internal Auditors. These include ensuring the model:

• is strongly aligned with measured topics and business processes
• offers a framework that reflects the desired state and supports the development of improvement plans
• offers a mechanism to provide insight into the improvement path from an immature to a mature process
• offers opportunities for benchmarking with other organizations
• relies on disciplined method that is easy to understand and implement

The audit team also ensured the federal government’s objectives on open government initiatives are reflected in the model. The Directive on Open Government was divided into 16 criteria and matched to the proposed model. The assessment also considered the size of the PSC, resources available within the Open Government Secretariat, and the PSC maturity model’s value-added.

Public Service Commission open government maturity model overview

The Open Government Secretariat has developed a maturity model to measure and report on the PSC’s progress on open government initiatives.
The maturity model contains 24 indicators in 4 categories:

• Category 1: Data/Information, planning and accountability: Measures maturity in the areas of data and information formats, data and information inventories, open government implementation plans, governance and accountability.
• Category 2: Transparency and open-by-default: Measures the PSC’s maturity related to progress towards open-by-default, data and information sharing, and proactive disclosure.
• Category 3: Engaging Canadians and the world: Measures the PSC’s maturity in areas related to awareness, engagement, data and information identification, and open dialogue.
• Category 4: Innovation, prosperity and sustainable development: Measures the PSC’s maturity in areas related to data infrastructure and culture, data access and sharing, data literacy and culture, and data identification.

As with other models, there are 5 levels of maturity achievable for each indicator, which are:

• Level 1: Initial conditions
• Level 2: Fostering transparency
• Level 3: Open participation
• Level 4: Open collaboration
• Level 5: Open co-creation

Maturity model observations

Architecture

The architecture of the maturity model is adequate and follows best practices.

What the audit team found

• The PSC’s maturity model is inspired by popular models like Gartner’s open government maturity model and the United Kingdom government’s model. The content is different and based on the PSC’s unique planned path for program assessment and maturation.
• The model is aligned with key best practices, including:
• multiple dimensions for accessing the open government program
• a defined level of maturity for each criterion
• logical succession of progress that can be measured
• The model will generate appropriate conversation among stakeholders about the desired level of maturity and improvement. It is not too cumbersome. Criteria are relatively simple and easy for most stakeholders to understand.
• The maturity model is simple and scaled to the size of the PSC. It should not require too much work to update the assessment.
• The advantage of the model is its simplicity in comparison to more detailed models (Gartner or Capacity Maturity). It would not be cost-effective for the PSC to develop a more prescriptive model that more fully explains the transitions between levels.
• However, the PSC model does require some interpretation due to limited descriptions of the transitions between levels. It will also rely on a degree of subjectivity and explanation from the Open Government Secretariat about progress.

What can be improved

• Wording: The wording of some indicators could be improved to make it easier for stakeholders to understand what should be done to reach each level. Some improvements, such as clarifying the use of the term “target audience,” were already pointed out by the audit team to the Open Government Secretariat during this consultation. Adjustments might also be necessary to adapt to stakeholders’ preferences and the new government National Action Plan for Open Government.
• Level 0: Maturity models usually have a level 0 to reflect an absence or insufficient work in a specific area. The audit team did not find that level in the PSC model.

For future consideration

• Limited benchmarking opportunities: By creating its own model, the PSC is acting as a leader. At the time of this consultation, no other model was available or ready to be used by other government organizations in Canada. The Treasury Board of Canada Secretariat is also working on creating its own model. Having different categories will complicate future benchmarking with other organizations. The Open Government Secretariat might want to compare with the Treasury Board of Canada Secretariat’s model when available and make some adjustments, if necessary.

Content

The content of the maturity model captures the key objectives outlined in the Open Government Implementation Plan, as well as requirements of the Directive on Open Government.

What the audit team found

• The content of the maturity model captures the key objectives outlined in the Open Government Implementation Plan, as well as requirements of the Directive on Open Government. These include maturity levels for:
• promoting and publishing information of business value
• creating an open-by-default culture
• governance to support the delivery of the Open Government Implementation Plan
• roles and responsibilities and their communication to stakeholders
• collecting and using performance data to support the management and improvement of the Open Government Implementation Plan
• publishing data of business value in an accessible, reusable format and timely manner on the Open Government Portal
• optimizing self-service
• The audit team noted that the categories and indicators provide enough opportunity to capture the progress of open government at the PSC and proper measurement of progress.

What can be improved

Identifying minimum compliance: The maturity model does not distinguish minimum compliance requirements from other improvements. To support decision-making about the desired level of maturity and useful communication of results, the model would benefit from such clarification. 

Self-assessment

The self-assessment conducted by the Open Government Secretariat is accurate and supported by enough evidence. Self-assessment results can be found in Appendix B

What the audit team found

• For the maturity self-assessment review, the Open Government Secretariat substantiated the rationale for each indicator so that the audit team could assess whether there was sufficient proof to support the scores.
• The Open Government Secretariat has developed a pointing system, giving a score of 1 to 5 to each criterion based on the level of maturity achieved. The average is then calculated by category to assess the overall level. 
• This first self-assessment will serve as a baseline for comparing future self-assessments.

What can be improved

• The audit team found that the “information inventory” indicator had not been assigned a score. The Open Government Secretariat explained that the PSC does not have an up-to-date information inventory and that there are no plans to create one, because this is no longer a requirement from the Treasury Board of Canada Secretariat. A score of “0” should be assigned to this indicator, or the indicator should be declared obsolete and removed from the model.

Advice for the maturity model

• Communicate minimum requirements: The Open Government Secretariat should distinguish minimum requirements for compliance from other improvements when communicating results of the maturity model.
• Identify desired level of maturity: It is not always desirable or cost effective to reach a level 5 of maturity for all criteria. The Open Government Secretariat should consult with management to determine the desired level of maturity and discuss the requirements for reaching that level.
• Improve terminology: The wording of some indicators could be improved to make it easier for stakeholders to understand how to reach specific levels. Some improvements, such as clarifying the use of the term “target audience,” were already pointed out by the audit team to the Open Government Secretariat during this consultation.
• Include level 0: The Open Government Secretariat should consider adding a level 0 to the model to indicate when no work has been performed on a specific criterion.
• Conduct model testing: The Open Government Secretariat should use the consultation for the 2022–24 Open Government Implementation Plan to further test the model, collect feedback from stakeholders, and improve it based on the needs that are identified.

Open government awareness at the Public Service Commission

Awareness

Why it matters

Assessing the awareness level of PSC employees helps the Open Government Secretariat better develop the next Open Government Implementation Plan, which includes an action plan to pursue open government objectives. 

What the audit team did

• Objective: Determine PSC employees’ level of familiarity with aspects of the Open Government Initiative and establish a baseline for future assessments. The audit team assessed the PSC employees’ awareness of:
• open government processes for publishing open data and open information
• roles and responsibilities of key partners
• technical requirements for publication
• status of the open-by-default culture by assessing their comfort level in participating in open government initiatives
• Survey methodology:
• a survey was distributed between July 13 and August 4, 2021
• participation rate was 37% (281 respondents)
• results are rounded to remove decimal
• Survey limitation: No distinctions were made between employee group, level or directorate for this survey, as the 2018–20 Open Government Implementation Plan aimed to develop awareness among all employees.

Survey observations

Value of publishing

• Most respondents recognize the value of publishing content on the portal, but only a few have done it.
• Overall, 11% of respondents have published a dataset or information assets on the Open Government Portal. (See Figure 1.)
• 76% of respondents see the value of publishing. (See Figure 2.)
• Respondents who don’t see the value of publishing said:
• there is a lack of information about how useful it is
• they don't see how the information they produce on a daily basis is valuable for Canadians
• The Open Government Secretariat can leverage awareness about the value of publishing to increase participation in open government initiatives.

Figure 1

Figure 2

Processes and requirements

Publication process

• Overall, the level of awareness about the publication process is low.
• A small percentage of employees are moderately to very largely familiar with the publication process. (See Figure 3.)
• The publication of open data and open information relies on the knowledge of very few employees within the PSC.
• Lack of familiarity could be discouraging employees from submitting data or documents for publication.

Figure 3

Publication requirements

• The level of awareness for the technical requirements is not very high among PSC employees.
• Some requirements have benefited from the awareness efforts made at the PSC to respect accessibility and language requirements for publication. (See Figure 4.)
• Respondents who have published datasets or open information are more aware than others of publication requirements.

Figure 4

Roles and responsibilities

• Overall, few employees are familiar to a large or very large extent with the responsibilities of key stakeholders for the Open Government Initiative. (See figures 5, 6, 7, 8.)
• Implementing an open-by-default culture relies on a good understanding of responsibilities by employees. The Open Government Secretariat has made considerable efforts in the last few years to educate PSC employees. The survey shows that those efforts have worked on a small percentage of employees.
• Given the low awareness about processes, it is important for employees to know where to obtain the support they need to publish data and information. Low awareness may be due to a high employee turnover rate and to competing priorities keeping employees busy.

Figure 5

Figure 6

Figure 7

Figure 8

Identification of assets

• Very few respondents are comfortable identifying assets and sharing their work on the Open Government Portal, since most do not know much about open government initiatives and are not sure how to proceed. (See Figure 9.)
• When asked what could make them more comfortable, respondents mentioned:
• training
• tools, guidelines and information on Intracom about the identification process
• simpler approval and publication processes
• more frequent messages in the PSC Express
• examples of published documents or one-pager with examples
• having a designated team member responsible for open government to support them
• The Open Government Secretariat has already developed some of these tools, but the survey showed employees are not well aware of overall support.

Figure 9

Sharing content

• Few respondents are comfortable sharing content on the Open Government Portal. (See Figure 10.)
• When asked what would help them share more work products, answers included:
• more information about what can be shared, what is permitted and what is protected, to help plan the production of information upstream
• more information about open government objectives, what should be published and what should not
• training, guidance and support
• a simplified process

Figure 10

Advice on awareness

• Provide and communicate clear instructions on Open Government Initiative: Some respondents have raised concerns about their ability to identify documents and data of value that can be published on the Open Government Portal. Developing clear guidelines would help them identify data and documents during creation and make it easier to publish. Employees should be able to easily access information about the initiative, how to identify potential candidates, training material, roles and responsibilities, and other resources. Currently the information can only be found by searching for past open government posts under the PSC Express on Intracom. Other governmental organizations have developed pages on their intranet sites to support open government objectives.
• Focus awareness efforts on managers: To increase awareness among PSC employees, it is important to set the tone at the top. Developing an awareness strategy that focuses on key managers, instead of all PSC employees, can support the awareness efforts already underway. By promoting training and providing tools, managers will be more equipped to identify potential candidates for open government and share insights about the process with their teams. This may in turn have a trickle-down effect and lead to raised awareness.
• Conduct surveys biennially: As part of benchmarking exercises, it is important to monitor progress. The Open Government Secretariat should conduct a survey at the end of each Open Government Implementation Plan to measure progress and inform future training and outreach activities.

Conclusion

Based on Open PSC objectives, the audit team concluded the following:

Developing a targeted approach to develop an open-by-default culture

While the Open Government Secretariat has done work to raise awareness, the overall level is still low among PSC employees. Raising awareness by targeting managers in the next Open Government Implementation Plan would set the tone at the top and concentrate efforts on those who have the power and responsibility to actively promote PSC and government priorities. Their engagement would trickle down and lead to more employees being knowledgeable about open government.

Improving the capacity to provide value-added data and information

The survey showed that most respondents still have issues identifying value-added data and information. Efforts have been made by the Open Government Secretariat to address this issue by identifying themes of interest for open government and piloting a “publish with a purpose” initiative at Integration Committee. The secretariat should continue supporting the PSC in identifying value-added data and information, using training, publications, tools or other mechanisms.

Identifying the desired level of optimization and innovation

The development of the maturity model will provide a tool for identifying the desired level of optimization at the PSC. Discussions about the desired level of maturity should take into consideration available resources and the level of engagement from stakeholders to reach the Open PSC objectives.

Supporting Government of Canada commitments and expectations

The maturity model provides the organization with sufficient criteria to assess progress against key objectives outlined in the Open Government Implementation Plan, as well as requirements of the Directive on Open Government. To support decision-making, minimum requirements for compliance with policies should be distinguished from further improvements. While the maturity model tends to be static, it should be adjusted when needed to ensure alignment with the Treasury Board of Canada Secretariat requirements.

Appendix A: Open Government Secretariat Response

Overall approach

Incorporate advice into activities in the 2022–24 Open Government Implementation Plan.

Activity examples:

• each directorate in the organization will work to identify at least one data or information asset that they will publish to the Open Government Portal
• develop an internal communication strategy to target managers and increase employee awareness of open government tools and resources
• incorporate Open Government 101 into employee onboarding to increase awareness

Response: advice for the maturity model

• Communicate minimum requirements: The Open Government Secretariat will update the maturity model to indicate minimum requirements for compliance as per the Directive on Open Government.
• Identify desired level of maturity: The Open Government Secretariat will consult with management to establish what is deemed successful progression and a reasonable target for the maturity model.
• Improve terminology: The Open Government Secretariat will improve the wording for general terms like “target audience” used for indicators in the maturity model to improve stakeholder understanding.
• Include level 0: The Open Government Secretariat will add a level 0 to the maturity model to indicate when no work has been performed on a specific criterion.
• Conduct model testing: The Open Government Secretariat will consult to further test the model, collect feedback from stakeholders, and improve it based on the needs that will be identified.

Response: awareness advice

• Provide and communication clear instructions on the Open Government Initiative:  The Open Government Secretariat is developing an Intracom page for the Open Government Initiative at the PSC. This page will give employees easier access to guidance including the open government process user guide and themes/eligibility assessment tool.
• Focus awareness efforts on managers: The Open Government Secretariat will target managers with engagement activities identified in an internal communications plan.
• Conduct surveys biennially: The Open Government Secretariat will conduct a survey at the end of each Open Government Implementation Plan to measure progress and inform future training and outreach activities.

Appendix B: Maturity Model Self-Assessment Results

Grading: Overall PSC – Level 2.5

1. Data/Information planning and accountability: Level 3

2. Transparency and open by default: Level 2

3. Engaging Canadians and the world: Level 3

4. Innovation, prosperity and sustainable development: Level 4