Evaluation of the PCO Central Innovation Hub

[ PDF version ]

[ * ] An asterisk appears where sensitive information has been removed in accordance with the Access to Information Act and Privacy Act.

Final report
November 19, 2018

Executive summary

The evaluation of the Central Innovation Hub (henceforth “the Hub”) covered the period dating from its establishment in February 2015 until the time it was re-mandated as the Impact and Innovation Unit (IIU) in November 2017. The evaluation was undertaken as part of the Privy Council Office’s (PCO) Risk-Based Audit and Evaluation Plan for fiscal year 2018-19.

In accordance with the Treasury Board Policy on Results, this evaluation focused on the three areas of continued relevancy, efficiency, and results achievement (effectiveness). The evaluation report assesses achievement towards short-term objectives including establishing the Hub’s infrastructure, outreach (including communications and marketing), and outcomes. Where possible, it also examines operational outcomes and progress towards the achievement of longer-term objectives including the expansion of the federal innovation network through a number of communities of practice and committees.

The evaluation used multiple lines of evidence. These included literature and document review, file review, key informant interviews, a review of project reports, and in-depth case studies of selected partnership projects that had been completed by the end of 2017.

Overview of key findings

Relevancy

The evaluation found that the Hub/IIU continues to have a relevant role in supporting government priorities by helping create a Public Service that is agile, as well as capable and supported in undertaking experimentation and igniting innovation. The Hub’s approach was aligned with other international jurisdictions and the evolution of the IIU will continue to bring Canada closer to leaders in public sector innovation.

The Hub was purposely placed at the centre of government to send a clear message across the Public Service of the importance of experimentation and innovation; to drive innovation and change; to help remove systemic barriers; and build capability to deal with complex problems. The Hub leadership recognized the importance of reflecting on its own placement, mandate and usefulness, and have evolved in response to changing needs during the evaluation period.

Recommendations

It is recommended that the IIU:

• Utilize the Deputy Ministers’ Task Force on Public Sector Innovation as a venue to:
  • Tackle complex problems from a systems level.
  • Work with its partners to keep up, if not ahead, of the speed of transformation. This requires looking beyond innovation methods at the program level to system level change.
  • Work collaboratively within established networks to identify and remove impediments to innovation. These include: cultural, procedural, policy and resource capacity (such as IT system and HR).
• Map its relationships to the broader innovation ecosystem (including committees, networks and other labs) that the IIU intends to work with to achieve outcomes. This will be helpful to the IIU as it continues to evolve and in measuring its results.

Efficiency

The evaluation found that the Hub invested a great deal of time in the start-up phase and there is evidence that these efforts have increased buy-in and support its work. There continues to be a central role for the IIU in developing relationships and networks, conducting innovative tests/proof of concepts projects, sharing learnings, and together, with key partners, being a catalyst for change.

For a small but growing group, it is clear that the Hub was, and the IIU continues to be, innovative in how it uses its own resources and leverages external resources to ‘punch above its weight’ in order to have the greatest impact.

Recommendations

It is recommended that the IIU:

• Expand its network of experts and collaborative partnerships to include experts in the private and not-for-profit sectors, who have shared interest in a competitive and thriving society and economy.
• Put in place a process to track demand for IIU services in relation to resources to ensure there is sufficient capacity to meet the needs within the federal Public Service. The Hub will require more resources to accomplish its objectives. Any future assessment of resourcing levels should include an assessment of a cost recovery model.

Effectiveness

The evaluation concluded that the Hub has achieved its short-term and operational outcomes and the IIU is on track to achieve its longer-term outcomes.

There is evidence to demonstrate that the Hub has identified and removed system-wide barriers to innovation which has led to positive policies being introduced to provide authority for departments to use new and innovative approaches to the design of policies, regulations, programs and services. It has created and tested new tools/approaches, and that these are becoming embedded in the policy development and delivery process system-wide. There are a number of networks and mechanisms in place to exchange information and best practices and spread successful experimentation and innovation across the portfolio. The Hub was also an active member of the international community of practice and network of innovation labs.

Recommendations

It is recommended that the IIU:

• Use the annual reports to publish all information related to the projects it supports (that move beyond the expression of interest stage) in order to showcase what has worked, transfer knowledge, and – more importantly – to share lessons learned and identify system-level barriers to innovation. This will send a powerful message and will help create an environment which demonstrates that informed risk-taking and failure is part of the process and an important part of the learning process.
• Focus on building capacity and scaling up innovation by leveraging networks and existing communities of practices such as Policy, HR and Regulatory. Bring together key stakeholders that would directly benefit from a deeper discussion of findings and how they might apply the findings in different areas. Expand the use of learning events such as the Randomized Control Trial workshop.
• [ * ] as part of ongoing operational planning, develop key performance indicators that measure activities and outcomes and ensure that data and information are being collected on a regular basis to inform ongoing monitoring and reporting on demand for services and results. Given that the IIU is part of the Results and Delivery Unit, it has a leadership role in establishing a plan with detailed indicators that measure change at the system levels. Note: In June 2018, the IIU launched its Theory of Change as the framework for planning and measuring results. According to the IIU, the Theory of Change has helped the Unit build consensus and articulate the systems changes, it, and its partners, are seeking to achieve and how they will be realized. The Unit further committed to move the Theory of Change into action by rolling out the monitoring and developmental evaluation strategies.

Introduction

The evaluation of the Central Innovation Hub (henceforth “the Hub”) covered the time period from its establishment in February 2015 until the time it was re-mandated as the Impact and Innovation Unit (IIU) in November 2017. The evaluation was undertaken as part of the Privy Council Office’s (PCO) Risk-Based Audit and Evaluation Plan for fiscal year 2018-19.

In accordance with the Treasury Board Policy on Results, the evaluation focused on the areas of continued relevancy, efficiency, and results achievement (effectiveness).

Given the coverage period, the evaluation primarily assesses achievement towards short-term objectives and outcomes. Where possible, it also covers operational outcomes and progress towards the achievement of longer-term objectives, which are as follows:

1. Evidence that new tools and approaches are embedded in the policy development and delivery process system-wide;
2. Evidence that positive policies are in place to provide authority for departments to use new and innovative approaches in policy, regulations, program, and services;
3. Evidence that Canada, through the Hub, is an active member of the international community of practice and network of innovation labs; and
4. Evidence that mechanisms are established to spread successful innovation across the portfolios (e.g., Hub reports, interdepartmental communities of practice).

Overview of methodology

The evaluation used multiple lines of evidence which included literature and document review; file review; key informant interviews; a review of project reports; and in-depth case studies of selected partnership projects that had been completed by the end of 2017.

The limitations associated with this evaluation were as follows:

• The evaluation focused on the first two years of the Hub’s operations. During this time, the Hub spent the first year establishing its infrastructure. At the same time, during its first two years, the Hub was quickly evolving to respond to the needs of the federal Public Service. This made the assessment of results more challenging given that, in effect, the key activities and focus of the Hub had evolved by the second year. These factors, combined with the fact that a two-year period is generally regarded as a short timeframe to evaluate results achievement, made measurement of long-term outcomes difficult.
• The short timeframe for, and timing of, the evaluation limited the number of interviews that could be undertaken. In addition, the interviews were primarily with former Hub staff or current IIU staff.  However, project partners and an external advisor were interviewed as part of the case study review.
• Given the timeframe, there were a limited number of projects that had been completed (n=16) and therefore a limited number of partners to consult.

Further detail about the methodology and limitations can be found in Appendix A. A copy of the moderator guide used for the semi-structured interview consultations is available in Appendix B. Finally, an overview of the projects reviewed as part of the Case Study review is included in Appendix C.

Context and background on the program

Establishing the Hub to drive innovation in the public sector

In the private sector, innovation is regarded as essential to stay ahead of the competition and survive as a business. Innovation in the public sector, on the other hand, has been slow to take hold and become embedded in the culture. This is beginning to change as governments increasingly recognize that in order to address the complex needs and challenges of the 21st Century and, by extension, the long-term growth and prosperity of Canada – governments must innovate.

In 2013, the Privy Council Office (PCO) released a roadmap to modernize Canada’s Public Service by the end of the decade. Entitled, “Blueprint 2020,” this report was based on the feedback of over 110,000 public servants and prioritized actions for: transformation, improved inter-departmental collaboration, and increased citizen engagement. One of the areas of focus was “innovative practices and networking,” as the interim report found:

Public servants want to develop innovative ideas and approaches to better serve Canada and Canadians. This requires improved information sharing and more ways to connect and collaborate across the Public Service to learn from each other and draw on outside expertise as needed. Many would like to see greater use of communities of practice, enhanced public access to information and more emphasis on engaging external partners and the Canadian public. They also see a need for more risk tolerance to try out new ways of working and serving Canadians.¹

Following the establishment of innovation hubs in countries such as Denmark, United Kingdom and Australia, and in response to Blueprint 2020, the Clerk of the Privy Council and the Secretary to the Cabinet,² set about establish a small innovation hub.

The Hub will provide expertise and advice on emerging areas and will help change the way the Public Service does business. It will also support departments in applying new approaches... to complex policy and program challenges.³

In 2015, the Central Innovation Hub was established in the Plans and Priorities Division of PCO but with a non-traditional open-concept workspace aimed at fostering collaboration in the same fashion as many other federal and international innovation labs.

As noted by the Clerk, locating the Hub at the centre of government, and within PCO specifically, was a deliberate decision intended to send a strong signal that the government values innovation, responsiveness, and the adoption of a philosophy of continuous change: “We are doing it here in the Privy Council Office, at our centre of government, to demonstrate that we think innovation is important.”⁴

[ * ] the Hub expected to provide strong value for money as a system enabler. Along with its partners, the Hub was expected to identify barriers to innovation and help remove them; introduce process efficiencies; as well as test and help introduce better targeting for programs. The Hub had three horizontal objectives:

1. Act as a central resource that provides easy access to information on best practices, new tool, approaches, and techniques.
2. Establish networks and partnerships between project leads and resources to accelerate departmental work and act as a central node in the broader innovation network within the public sector.
3. Act as a direct innovation driver that works with departments to identify and plan projects, playing an active role throughout the lifecycle of the project, as well as documenting results and evaluating results to identify success and key lessons for the system.

In its first Annual Report (March 2016), the Hub outlined its core practice areas as follows:

• Behavioural Insights: Improve the public policy outcomes and optimize programs and services based on a better understanding of the way people behave in real-life situations.
• Design: Apply principles of human-centred design/design thinking to work through stubborn problems, uncover insights, and improve user experience of programs and services.
• Data: Tap into unconventional and massive data sources (i.e., Big Data) to provide information for the government to pursue more innovative approaches in public policy development and implementation.

Additionally, the Hub undertook work to provide the policy rationale and create space for the government to pursue more innovative approaches in public policy development and implementation.

In late 2015, a new federal government was elected with a new agenda and horizontal commitments to strengthen the culture of measurement, evaluation, and innovation in the Public Service. Innovation and results were also outlined in a number of publicly available ministerial mandate letters.

The Hub was originally led by an Assistant Secretary, and supported by 1.5 FTEs. Its initial priorities were to establish the Hub, secure resources (including office space), and staff key positions. In reflecting on these early days, former Hub staff admit they underestimated the time it would take to physically establish the Hub while also engaging with critical partners in other federal departments and agencies in order to: market the Hub and its mandate; provide a rationale and encouragement towards innovation; and linking innovation to improved public services and outcomes for Canadians.

With the election of a new government in 2015, the focus on outcomes to drive, monitor, and report on results and with the creation of the Results and Delivery (R&D) Secretariat in 2016, the Hub was transferred to the R&D Secretariat. The Hub office was also physically relocated to a more centrally located PCO office. The interviewees noted that being closer to senior decision-makers was more important than non-traditional design and functional collaborative space. This decision was also made to better align the new government-wide drive towards using innovation to achieve results. Note: the location of the Hub is discussed further below.

Key activities

In its first two years of operation, the Hub’s key activities focused on the following.

Establishing the Hub’s infrastructure

When established, the majority of staff time was spent on acquiring and establishing the operational space for the Hub and recruiting new staff (either internally, externally or through a variety of flexible arrangements such as micro-assignments/missions and student co-ops).

Outreach (including communications and marketing)

Once established, Hub staff and senior leaders invested in outreach activities including communications and marketing in order to increase understanding and awareness about the purpose and role of the Hub; develop trust with representatives of departments and agencies; work with partners to incorporate innovation into their work; and to establish relations with other jurisdictions. Outreach activity primarily focused on:

• Departments, including the small community of federal and provincial innovation labs;
• Public servants through learning sessions, conferences, and events, such as armchair discussions;
• Senior government executives through Deputy Minister and Associate Deputy Minister committees;⁵
• The Canada School of Public Service, to incorporate innovation into the learning and training curricula; and
• Other jurisdictions provincially and internationally to learn and share best practices.

Information and learning sessions were also held in order to inform and better understand the policy community – including needs and barriers to innovation – and the latest innovative tools and approaches, (i.e. behavioural economics; data analytics; strategic design; innovative finance; crowdsourcing, challenge-prizes, open policy-making; innovative partnerships; disruptive technologies) that were already in use.

Undertake partnership projects

As per its established mandate, the Hub was expected to undertake projects in partnership with federal departments and agencies. Early projects tested Behavioural Insights; use of large data and design-thinking tools and methodologies. With its partners, the Hub:

• Developed common or standard criteria for assessing proposed projects including: areas already tested in other jurisdictions with proven results; alignment with government priorities; scalability across government and capacity.
• Identified potential partners and projects to pursue using new and innovative methods and tools.
• Approved and launched 16 partnership projects (please see Appendix C for an overview of projects undertaken within the timeframe of this evaluation).

Develop innovation networks

The Hub spent a great deal of time reaching out to others in the innovation space both within the federal framework and across national and international jurisdictions. The Hub created and supported the expansion of the federal innovation network through a number of communities of practice and committees (Note: these communities, networks and committees are discussed further below under Effectiveness). These networks continue to support both knowledge exchange and transfer.

In response to a Budget 2017 commitment, the Innovation Hub was asked to lead the Impact Canada Initiative. In recognition of this increasing responsibility, in November 2017, the Hub was re-mandated as the Impact and Innovation Unit (IIU) with a new mandate to act as a centre of expertise to support departments’ use of outcomes-based and expanded approaches to innovation including prize/challenges, new partnership models, innovative financing approaches, impact measurement techniques, and behavioural insights to get better results for Canadians in priority areas.

Evaluation findings

Relevancy

The trend toward public sector innovation hubs

Innovation labs in the public sector began to emerge in the early 2000s, with the creation of Denmark’s MindLab in 2002. Since this time, dozens of equivalents have sprung up around the world. Examples include the OPM Innovation Lab in Washington, DC; the Laboratorio Para La Ciudad (only in spanish) in Mexico City; the Human Experience Lab in Singapore; and the Centre for Excellence in Public Sector Design, Canberra, Australia.

Experience from Nesta and others who are leading in the areas of innovation labs reveals that “all governments need institutions to catalyze change.”⁶ Most governments struggle to find the space, time and resources to envision the future. Investments in innovation labs, such as the Hub, create the structures, space and capacity to envision and test new approaches that can lead to better results for Canadians.

The business rationale for innovation labs or hubs is derived from their potential to accelerate a fundamental shift in public sector institutions and their cultures in order to address increasingly complex problems. For societies to advance, public sector organizations need to continuously find new and innovative approaches that will help grow the economy and meet society’s evolving needs. Faced with shifting demands from an increasingly connected and engaged citizenry and limited resources, traditional public sector approaches to policy, program and service delivery alone are not sufficient to meet those needs. Public sector organizations need to reinvent themselves and be early adopters of new products and technologies. This does not mean abandoning everything old, but it does mean a unique balance of traditional approaches with new innovative methods that involve collaboration and co-development.

A review of the literature and practices found in other jurisdictions demonstrates that innovation labs are an essential mechanism to:

• Break down departmental silos by supporting collaboration and integrating knowledge and expertise;
• Enable cross-disciplinary approaches to policy and services decision-making by enhancing opportunities for new insights, solutions and prototyping;
• Inject creativity and experimentation into the policy design process and bring design solutions to civic problems;
• Ingrain experimentation and risk-taking throughout government;
• Influence policymakers to understand their role in public service innovation;
• Promote best practices;
• Drive cultural change;
• Expose public servants to creativity and experimentation;
• Stimulate and create a Public Service that is more open to working with others to solve key societal challenges and to harness emerging opportunities; 
• Tap into methods and skills usually not available in the public sector; and
• Acknowledge and address shortcoming in public sector innovation models.

The general premise underlying innovation labs is to move beyond a limited number of people working in isolation within an organization on innovation, towards teaching hundreds, if not thousands, of public servants to think like innovators and be open to experimentation in order to unlock creativity, foster experimentation and increase productivity.

Borrowing from the private sector, public sector organizations have been experimenting with innovation labs to create the conditions and work environments to inspire experimentation and innovation. Innovation labs provide opportunities to bring government and external partners together to learn from each other, make connections, develop new skills and be inspired to reach new levels. The literature review found that public sector organizations are increasingly building and experimenting with collaborative partnerships with both the private and not-for-profit sectors, who have shared interest in a competitive and thriving society and economy.

Most public-sector innovation initiatives emphasize real-world experimentation, participatory design, and user-driven co-creation/co-production with the direct involvement of citizens/stakeholders. Behavioural Insights, for example, are being embraced by many contemporary governments including the United Kingdom, United States of America, Australia, Denmark, Germany, France, Singapore and many other counties have found success in applying behavioral science insights into public policy development.

The evaluation did not find any overview or mapping of the Hub in relation to the broader innovation ecosystem and key partners it intended to work with to accomplish its objectives.

Placement of innovation hubs

A review of the literature demonstrates that the challenges and risks public sector innovation hubs encounter commonly stem from the following paradox: with closer proximity to the centre of government comes a greater ability to leverage resources and potentially greater access to leadership and administrative support to effect change, but this proximity may limit the ability to be truly innovative. The opposite is also true – a distance from government enables more autonomy, less bureaucracy and distance from the cultural, structural and procedural impediments to innovation that characterize the public sector, but greater autonomy may limit the resources and support needed to effect change. Interview consultations with Hub staff and project partners indicated that having the Hub located in the PCO helped increase the credibility of the innovation projects and helped build support among senior leaders. In many jurisdictions, a decision often has to be made between the necessity to effectively reconcile the high-level support that is needed with the desire to work at a distance from government to ensure a healthy degree of autonomy and enough flexibility to be innovative.

The review of the literature revealed that public sector innovation spaces are usually separate from the rest of the public sector to permit and support a sufficient degree of autonomy from the structural and procedural barriers to innovation that often characterize the public sector. These include: established operating rules, norms, policies, and protocols; large hierarchical and centralized bureaucratic structures characterized by stability and consistency; departmental silos and restricted ability to connect; limited network structures; a lack of competitive pressure to innovate that stems from the monopolistic nature of public sector organizations; and continued adherence to top-down approaches. A review of the literature illustrates that the most commonly encountered challenges and risks facing public sector innovation hubs stem from insufficient distance and autonomy from government and from the above-mentioned structural and procedural impediments that characterize the public sector. For example, Tonurist, Kattel, and Lamber’s (2015) study on 35 different public sector innovation hubs, identifies the following operational key tenants: (1) large autonomy in setting targets and working methods, (2) structural separation from the rest of the public sector and (3) the ability to attract external funding. McGann, Blomkamp, and Lewis’ 2018 study of 19 different innovation labs found similar findings.

Below are some examples from the innovation labs identified in the literature:

• In Australia, the Policy Innovation and Projects Division is located in the Department of the Prime Minister and Cabinet, similar to Canada’s PCO Central Innovation Hub.
• In the UK, Nesta Innovation began inside government but was eventually spun out as a non-profit foundation.
• The MindLab in Denmark was established between 3 Ministries (Note: in 2018 the MindLab was closed down and parts were evolved into a new Disruption Task Force).

Evolving mandate

At the time of its establishment, there was a limited number of innovation labs across the federal government (estimated to be less than 5). The Hub was established to fill a niche: the need for a centralized player who was outward focused, horizontal, and driven to develop partnerships to test innovative approaches with other departments. Interview consultations revealed the Hub experienced some resistance to its establishment particularly among stakeholders in departments that were already experimenting with new approaches to policy and program design.

Those interviewed noted that while the original mandate of the Hub was to be a convenor, connector, and a central resource for tools to help drive innovation, it became clear that there was less of a need for a ‘hub’ type function. There was a greater need for a more strategic role of removing barriers to innovation and working with partners to solve some of government’s and society’s most complex problems.

As detailed in the Effectiveness section below, the review of documentation and interview consultations showed that the Hub has been responsive to the needs of the federal Public Service, senior leaders and its partners; it has adapted, focusing its efforts and resources to address gaps and needs.

While there is no map of the federal innovation landscape, interview consultations with Hub staff indicate that today there are over 20 innovation labs/spaces across the federal government with some evolving into new models. For example, the Innovation Lab at Innovation, Science and Economic Development Canada, which in 2017, announced it was co-locating with an Ottawa accelerator. Similarly, Nesta has spun out of the centre of government into a not-for-profit, and Denmark’s MindLab’s activities will be carried out through a new Disruption Task Force.

Efficiency

Resources

The following table provides an overview of the resources allocated to the Innovation Hub:

Former Hub staff consulted for this evaluation indicated that in the first year of operations, the Hub invested the majority of its time in establishing the infrastructure and hiring staff. As the Hub became established, leaders recognized the need for different skill sets, particularly in the areas of communications and marketing. Once the Hub began undertaking outreach to departments and agencies, a large portion of the Hub’s work focused on establishing partnerships to develop a shared understanding of what innovation means and its value in a public-sector context, and encouraging departments to take informed risks and undertake innovative and proof-of -concept projects.

Interview consultations indicated that having the Hub in place provided departments with access to internal expertise that would otherwise be contracted to external consultants. Interviews with staff indicate that as the demand for the Hub’s support and expertise grew, it became increasingly clear that the Hub was unable to meet the demand for its services. In response to this demand and traditional delays in staffing, the Hub used alternative approaches to expand its capacity, within their budget constraints. These approaches include the creation of a pool of Trusted Advisors and, later, under the IIU, a Fellows Program.⁹

• The Trusted Advisors was a pool of unpaid experts that the Hub drew upon to provide a peer review function including feedback on the approach and methodology of the Hub’s experimentation or proof of concept projects, as well as review results and inform recommendations.
• The purpose of the IIU’s Fellows program is to leverage resources from outside of the Public Service (e.g., academia, other sectors) for placements in departments as subject matter experts in fields that are currently under-resourced. The cost of the fellows is charged back to participating departments. This approach allowed the IIU to manage risks associated with protracted staffing processes and meet departments’ needs quicker. Currently there are four Behavioural Insights fellows in this program. This number is expected to increase to six in the coming months. Innovative Finance, Impact Measurement, and Data Science Fellows Streams are also planned.

Comparisons to other Jurisdictions

While not an extensive review, the evaluation did attempt to take a snap shot of resources dedicated to similar functions/organizations in other jurisdictions. There are a few limitations to this review including:

1. The review was not extensive but rather limited to a snapshot of available information;
2. The level of demand for the Hub’s services was not assessed; and
3. The comparisons are not made between like organizations – given the differential approach applied in each jurisdiction.

During the evaluation, interview consultations (primarily internal) raised the issue that the Hub did not have sufficient resources to meet demand. When the information was examined based on resourcing of Behavioural Insights-related activities (compared to all of the Hub’s activities), the evaluation found some initial indication that other jurisdictions appear to invest more resources in these activities. Examples are outlined below.

Behavioural Insights Group, British Columbia

The BC government has established the Behavioural Insights Group (BIG), which staffs seven full time positions (1 Executive Director; 1 Director; 2 Senior Behavioural Scientists; 2 Methods Specialists; 1 Knowledge Translator). It has an annual salaries and benefits budget of approximately $630K as well as an annual operating budget of approximately $300K to be used for obtaining contracted services from a pre-qualified supplier list, as well as travel, project work, IT support, conferences, etc. Note: BIG is only focused on behavioural insights and not on other areas such as user-centred design/design solutions and big data and data analytics.

MindLab, Denmark

The MindLab was established as part of three ministries (education, employment and business) and the Odense municipality. It is a cross-governmental innovation lab, which involves citizens and businesses in creating new solutions for societal problems. Initially, the MindLab was staffed with five, full-time employees with a variety of different formal skills ranging from creative facilitation, teambuilding, hosting, and policy development. By 2006, the Lab was evaluated as being under-resourced to meet demand. In 2007, the MindLab increased their staff to 10 fulltime equivalent (FTEs) staff and increased their budget to 1 million EUR. MindLab was also a physical workshop space – intended as a neutral zone for inspiring creativity, innovation and collaboration. MindLab closed down in May of 2018 and, at the time, had over 40 FTEs. A part of the Lab’s activities will be carried on in the new Disruption Task Force which will continue to focus on innovation and digital transformation.

Behavioural Economics Team, Australia

In Australia, the Behavioural Economics Team Australia (BETA) is focussed on behavioural science and insights. BETA has a budget of approximately $4 million per year for the next two years. On top of this, BETA charges its partners for their contributions to projects and has a project revenue target of between $1-$1.5 million per year. BETA’s average staffing level is approximately 27 FTEs. 

BETA is located in the Policy Innovation and Projects Division within the department of the Prime Minister and Cabinet (PMC). Other parts of the Division include:

• Data and Digital Branch
• Project Office
• Office of the National Data Commissioner

The Policy Innovation and Projects Division provides new and creative approaches to policy development, data and digital policy, as well as applying behavioural economics to policy design. These other units have separate staffing and operating budgets.

Behavioural Insights Team, UK¹⁰

The Behavioural Insights Team (BIT) is a social purpose company that is jointly owned by the UK Government (Cabinet Office) and Nesta (the innovation charity); and by its employees.

When BIT was first established it was situated in 10 Downing Street. At the time, it was the world’s first government institution dedicated to the application of behavioural sciences. Below are BIT’s business objectives, which have remained constant overtime:

• make public services more cost-effective and easier for citizens to use;
• improve outcomes by introducing a more realistic model of human behaviour to policy; and, wherever possible,
• enable people to make ‘better choices for themselves’.

BIT is focused on redesigning public services based on ideas from the behavioural science literature. Their trials are highly empirical and BIT tests and trials these ideas so partners understand what works and what doesn’t before ideas are scaled up.

BIT has approximately 150 employees, who have either a strong academic grounding in economics, psychology, research methods and/or a background in government policymaking. Today BIT is located outside government with its headquarters in London and with offices in Manchester, New York, Singapore and Sydney.

Recently BIT established BI Ventures, a team within BIT, which works on building scalable digital products that address social issues. BI Ventures currently has three revenue-generating products in its portfolio, with more in development. BI Ventures builds products using ideas that have been generated within the BIT and uses behavioural insights to design and build scalable products and services that have social impact. BI Ventures has rigorous process of conducting large-scale trials that then go through a process of prototyping and piloting before tests are scaled to a product.

Effectiveness

Achievement of short term and operational outcomes

There is evidence that progress was made towards the achievement of short term and operational outcomes as follows.

The Hub worked with federal partners to identify, plan, and implement projects. By 2015-2016, the Hub had completed four pilot projects in the areas of design-thinking and Behavioural Insights. By the following year, the Hub was exploring a potential 15 projects with 11 departments. Between 2015-2017, the Hub undertook a total of 16 projects. Interview consultations and a review of program documentation found evidence that the Hub worked with its partners to develop a project selection process based on government priorities; the partners’ willingness to explore new ideas and engage directly with the public; as well as involving a design-oriented change that could result in significant improvements in the lives of citizens.

In the early days of the Hub, project selection was based primarily on departments that would engage with PCO and projects that were typically less risky and where practices in other jurisdictions had already demonstrated impact. However, as the Hub undertook initiatives and partnership projects that demonstrated learning and impact, demand has continued to grow for the Hub services to the point where it outstripped capacity. Interviews with staff indicated that they are now able to be more selective of which projects to support, based on the process noted above. With increasing demand, the Hub adhered to its approved project selection criteria which helped ensure the achievement of the shorter-term outcome that efficiency would not be the only driver in project selection.

It was expected that each of the Hub’s partnership engagement/projects would be reviewed by the Hub and its partners and that each project would deliver timely advice to strengthen policy and program outcomes. The Hub had a standard operating procedure of planning and reviewing the outcomes of projects it supports. There was a report produced from each project that provides an overview of the project, key findings, recommendations, and future research opportunities. These reports are published on the IIU’s public-facing website. The IIU and its partners leverage opportunities to present the findings, learning and recommendations from the projects with communities of practices and other learning events.

Progress towards the achievement of longer term outcomes

The evaluation sought to confirm if the Hub was on track towards the achievement of its longer-term outcomes; recognizing that the Hub was not expected to fully achieve these outcomes in the short term. These longer-term outcomes are outlined below:

1. Evidence that new tools and approaches are embedded in the policy development and delivery process system-wide;
2. Evidence mechanisms are established to spread successful innovation across the portfolios (e.g., Hub reports, interdepartmental communities of practice);
3. Evidence that positive policies are in place to provide authority for departments to use new and innovative approaches in policy, regulations, programs and services; and
4. Evidence that Canada, through the Hub, is an active member of the international community of practice and network of innovation lab.

1. There is evidence that the work of the Innovation Hub contributed to the adoption of new tools and approaches embedded in the policy development and delivery process in use in the Government of Canada.

The literature review identified several common leading approaches and tools to support innovations in public programs and services. For example, in the theoretical/conceptual literature on public sector innovation, there is a prevailing emphasis on human-centered design concepts and methods. In addition, in the case studies on public sector innovation initiatives, there is a clear emphasis on real-world experimentation (see Bliss, Bliss and Nidhi, 2014), participatory design and user-driven co-creation/co-production with citizens/stakeholders (see Almirall and Esteve, 2012).

The Hub’s 2017-18 Performance Measurement Strategy explains that the “Hub provided access to new and innovative tools, approaches and techniques in its core practice areas to help public servants apply them to their work”. These included Behavioural Insights; design thinking and leveraging multiple and big data sources as well as the provision of training sessions, workshops or conferences, and/or support material provided through the Hub’s website.

As mentioned previously, the former Hub staff also partnered with departments to co-create and develop innovative policies and programs using new tools and approaches. While the examples below are nested in the Hub’s core Behavioural Insights business line, many projects included collaboration between other Hub practice areas.

Randomized Controlled Trials were the technique used most frequently to assess the possible impact of proposed innovations to programs and service.

The Hub entered into partnerships with a number of departments to undertake projects that tested innovative approaches; some of which included the use of Randomized Controlled Trials.¹¹ Specifically, Behavioural Insights Randomized Controlled Trials were used to test and measure whether innovations improve outcomes such as the uptake of the Canada Learning Bond, Employment and Social Development Canada (ESDC)¹² and increase uptake of the Statistics Canada Farm Financial Survey,¹³ both of which used stratified random sampling to compare results from the use of different messaging and imaging. In the case of the Canada Learning Bond, while the program was already experimenting with nudge theory, the partnership expanded the tools (such as tear away vouchers and client focused checklists) and approaches to reaching targeted audiences/stakeholder groups (e.g., outreach).

ESDC has continued to advance this work forward through their own Innovation Lab, which was established at roughly the same time as the Hub. As part of this project, the Hub also worked with the Government of Ontario as a third partner to explore the inclusion of RESP enrolment in the Service Ontario’s 4-in-1 Birth Bundle program. Interview consultations with ESDC revealed that its internal Innovation Lab took the successes and lessons learned from this initiative and have been looking to apply the learning in other program areas such as Old Age Security and services to Indigenous communities.

Based on the success of the Farm Financial Survey, Statistics Canada has expressed an interest to pursue further work with the Hub’s Behavioural Insights team on more high-impact projects.

The Behavioural Insights team, in partnership with the Department of National Defence (DND), also conducted Behavioural Insights research into the factors that influence women to decide to join the Canadian Armed Forces. This research led to a second project that was based on a social media marketing trial. The project was designed to use targeted social media messaging (testing both image and message congruence to identify ads that engaged individuals within the targeted demographic i.e., females).¹⁴ This work developed into a more long-term engagement between PCO and DND and this collaboration is currently moving into the third phase. Based on the success of the project, DND has created its own Innovation for Defence Excellence and Security (IDEaS) Program, which will seek to build capacity through innovation.¹⁵

The work on Design Thinking looked at specific problems from the perspectives of the end-user. For example, in collaboration with Natural Resources Canada, Hub staff first gathered fact-based insights into how people use energy in their home. These insights were used to frame a project, followed by an exploration of how to tailor incentives to citizens’ energy needs to reduce consumption. The Hub also partnered with Immigration, Refugees and Citizenship Canada (IRCC) to put in place a “user centred” design challenge to improve the application process for the Family Class Sponsorship clients. To disseminate findings and results, IRCC created change leaders who returned to their units and shared their new knowledge about user-centred design.

While projects in this practice area have been primarily exploratory, the Hub’s experience with Design Thinking led to the conclusion that it is more effective when incorporated with other approaches. Based on these learnings, components of Design Thinking have been integrated into future Behavioural Insights projects, rather than through stand-alone projects. For example, the formation of ideas has been incorporated into some of the Behavioural Insights projects.

Within the domain of big data and analytics, the Hub explored a project, in partnership with Health Canada, to undertake web scraping of data from online sales reviews and other social media tools as a way of creating a surveillance system to monitor and detect dangerous toys and health products for Health Canada. While this project was viewed as leading edge, it did encounter a number of barriers related to privacy and access to data. The conceptual exploration, therefore, did not transfer into the project phase. While this initiative was mentioned in the Hub’s 2015-2016 Annual Report, there was no other reference to the outcomes of this initiative. Similar to Design thinking, data and analytics are being integrated into multi-disciplinary projects that include the other core practice areas.

The literature review demonstrated that these techniques have gained increasing penetration in the policy world during the time period of the evaluation and had been previously tested by governments which are further ahead in establishing innovation labs including the UK, US and Denmark. These tools were also already being tested by a limited number of federal departments who were undertaking some Behavioural Insights trials in order to improve program design and outcomes. Interview consultations with Hub staff and project partners indicate that the Hub helped advance these tools and their testing in a federal context and to support more rigorous and outcomes-based evaluation methods.

Evidence from the interviews, file and documentation review indicate that early in its establishment, the Hub provided strategic policy advice to senior officials, such as preparing transition documents, as well as strategic advice in areas such as open policy making and digital engagement. It also developed publications and communications products, such as an article in Policy Options from the former Clerk of the Privy Council on public sector innovation,¹⁶ and contributed material for numerous speeches. (Source: Innovation Hub - Annual Report - 2015-2016).

Partners in the Hub’s project indicated that the Hub supported projects led to innovative program design and testing and ultimately these learnings were transferred to other program and service areas and improved policy and program outcomes. In addition, the sharing of results from these projects created greater interest and demand in working with the IIU moving forward.

2. There is evidence that mechanisms were established that contribute to the successful diffusion of innovation across the portfolios (e.g., Hub reports, interdepartmental communities of practice).

The Hub provided federal government departments with resources and guides to support innovation; links to articles and podcasts; consultation and business advisory services; and expert advice in applying innovation to policy development and service delivery.

In addition, the following activities undertaken or managed by the Hub were mechanisms that were established to spread innovation across the portfolios and are further evidence of key partners and influencers that the Hub works with to effect change.

Deputy Minister Committee on Policy Innovation: The Deputy Minister Committee on Policy Innovation examined trends and technologies with the potential to strengthen or transform policy development and delivery, and tested and assessed innovative approaches that may have enhanced policy outcomes. Although this Committee was founded by the Clerk of the Privy Council, the Hub managed the secretariat function for the Committee and presented on Hub-supported work when it was available; provided guidance on strategic policy issues (e.g., impact of sharing economy, innovative use of Grants and Contributions; and sought feedback on key initiatives (e.g., Impact Canada Initiative)).

Note: The Deputy Minister Committee on Policy Innovation has evolved into the Deputy Ministers Task Force on Public Sector Innovation (Task Force), of which the IIU is the Secretariat. It is mandated to “play an action-oriented role in experimenting with emerging tools and approaches and helping provide public servants with the skills and knowledge to achieve better results for Canadians.” Its work supports departmental examination of emerging trends and technologies, core systems transformation, and experimenting with disruptive technologies within government.¹⁷

Assistant Deputy Minister (ADM) Committee on Experimentation: The Committee was established to advance the experimentation direction to Deputy Ministers (discussed later in the report) as a horizontal mechanism to support mutual learning, collaboration, and partnerships between Committee members. The Committee is also open to guests by invitation to provide alternate perspectives or expertise. Committee materials and insights are available to all public servants. The Committee was established by PCO (through the Hub). Today, the IIU jointly holds secretariat duties with TBS.

Behavioural Insights Community of Practice (BICoP): In 2016, the Hub supported the launch and operations of the Behavioural Economics Community of Practice, an interdepartmental network of behavioural insights researchers and practitioners across the federal government. It originally began as a Community of 20 members, at the Director General level, and focussed narrowly on Behavioural Economics. This community of practice has evolved into the BICoP, a horizontal network of employees, practitioners and researchers across the federal government. The BICoP share information, research methodologies, and experimentation results with a broader Behavioural Insights mandate open to all levels with the federal Public Service. The Hub co-chaired this Community of Practice along with a rotating chair. In addition to sharing information about innovative approaches, tools, and techniques, the BICoP has also held workshops on tools and practices (e.g., randomized control trials). Today it continues to be co-chaired by the IIU, has over 400 members and an online presence.

A similar model is now being explored for challenges/prizes based on the success of this format.

Behavioural Insights Network (BIN): In the third quarter of 2016, the Innovation Hub began outreach to develop an external Behavioural Insights Network, intended to bring together players across governments working in the field of Behavioural Insights. A first meeting of this network took place in early 2017. The Network was intended to foster stronger communications and collaboration in behavioural insights, provides a forum to share experiences, make linkages and explore potential partnerships. Six provinces, one territory and three municipalities participated in the inaugural meeting which took place in early 2017.

Presentations/workshops: In addition to presenting to the committees listed above, Hub staff engaged with other departments and agencies to provide presentations and workshops on the innovative tools and approaches they were using as well as results of projects undertaken to date. Interviewees from Canada Learning Bond, DND and Statistics Canada projects indicated that the results from these projects have been presented often – both in the departments who co-designed the work and beyond (including the BIN and BICoP) to share the results, key lessons learned and to raise interest in the broader applicability of the methods and/or results. While Hub staff often undertook these presentations, the collaborating department also undertook some of these communications and dissemination activities.

In addition, the Hub helped organize workshops with key partners. Two examples include:

• BICoP Design Jam
  • July 5, 2015 - Hosted at ESDC Innovation Lab (also facilitated by ESDC)
  • 50-60 attendees
• BICoP Randomized Controlled Trial workshops
  • Sept 6, 2017
  • co-facilitated by Hub and ESDC Lab
  • 45 attendees

A second workshop was organized in September 2018.

Within the federal innovation ecosystem, while the Hub participated in and supported a number of conferences, meetings and networks, there appears to be a gap in the formal learning and development of skills and competencies (that support innovation) of public servants; either through the Hub, the Canada School of Public Service, or others within the system.

The Hub has also worked in the open, making materials available to all public servants, and putting a great deal of it on the Hub’s public-facing website. The Hub co-designed and delivered innovative projects that were intended to impact policy and service delivery within departments. From these projects, case study reports were developed and posted on the Hub’s public-facing website. For a time, the Hub had a podcast (Tracked Changes) and the IIU now continues to maintain a blog section on the website. Hub staff and the advisory committee were also forward facing, with biographies and descriptions available on the website. This approach is consistent with the 2016 Interim Directive on the Administration of the Access to Information Act and the principle of "open by default”.

While the annual reports and case study reports that were developed during the two years of this evaluation do well at articulating the successes of the Hub, and, while there is some reporting on areas where there is no discernible impact (null report), there was an absence of material outlining and recognizing failures or challenges experienced within projects. As previously noted, the Hub was established at PCO to send a strong signal across government of the importance of experimentation and innovation – recognizing and communicating projects that have come up short on results is part of the process, as is learning from failures. For example, the Hub’s 2015-2016 Annual Report indicated that it was exploring with Health Canada analyzing data from online retailers as an early warning in relation to product safety. This project is never highlighted again in subsequent reports nor is a case study report made available online.

To more fully spread a culture of innovation and experimentation across the federal government, the IIU would do well to be more open about challenges experienced during the design or implementation of supported projects including those that simply did not lead to expected results. By sharing limitations, barriers and failures from projects, the IIU would acknowledge all potential outcomes of innovative policy and service delivery work, demonstrating that experimentation and innovation does not always breed success or lead to better outcomes.

3. There is evidence that positive policies are in place to provide authority for departments to use new and innovative approaches in policy, regulations and program and services.

Interview consultations indicated that during the initial years of the Hub’s operations, there were a number of barriers to experimenting and testing innovative solutions to public policy issues and programs that were identified (e.g., procurement policies, transfer payments terms and conditions, human resource capacity, horizontal rigidity, risk aversion).

The Hub worked with key partners and, in consultation with the Deputy Minister Committee on Policy Innovation, identified a number of barriers to innovation ranging from policies, terms and conditions, HR, IT, procedural and cultural.

Below are some of the areas where the Hub was identified by stakeholders as contributing to or leading to the development of innovative approaches.

The first is the Experimentation Direction for Deputy Heads, co-authored by the Secretary of the Treasury Board and the Deputy Secretary of Results and Delivery (PCO) in December of 2016. This direction provided greater information on expectations of departments to help meet the Treasury Board President’s mandate letter commitment to ensure that Deputy Heads identify a fixed percentage of program funds be put toward experimentation, and to also report on their efforts in Departmental Plans. In addition, departments sought to foster work environments conducive to experimentation, and report on results regardless of the outcome, with a default to public release. The Hub played a co-secretariat role with TBS to the ADM Committee on Experimentation. This Direction was supported by senior leadership at the Innovation Hub (currently the IIU) at PCO.

The second is the Declaration on Public Sector Innovation, which was signed by each of the Federal, Provincial, and Territorial Clerks and Cabinet Secretaries on November 14, 2017, outlining the actions they would take to support public-sector innovation.¹⁸ The Hub/IIU led the drafting of this Declaration. Key commitments among the partners include: seeking new ways to solve complex problems, experimenting and sharing knowledge, collaborating together and more broadly with stakeholders, working on outcomes-based funding models, and establishing partnerships.

Policy changes

The Hub influenced a number of significant policy changes. In 2016, the Deputy Minister Committee on Policy Innovation directed the Hub to research and examine barriers to innovation specifically in relation to Grants and Contributions. The Hub was part of an inter-departmental working group that included TBS and large Grants and Contributions departments and the Hub. The Working Group identified the terms and conditions for Grants and Contributions agreements as restricting new approaches to experimentation, and the formation of partnerships with the private sector. This ultimately led to a commitment from TBS to draft Generic Terms and Conditions that created authorities, provisions, and exceptions in Grants and Contributions programs to allow for greater experimentation. Three specific categories have been approved, effective April 1, 2017, and will be leveraged over the course of the 5-year pilot, led by TBS:

1. Incentive-based funding mechanisms;
2. Prizes/challenges; and
3. Micro-funding.¹⁹

These tools, having already been tested in other jurisdictions, are viewed by all of those consulted as a critical step forward in transitioning away from activity-based funding models towards models based on the achievement of concrete goals and results.

Impact Canada

Based on best practices in other jurisdictions, and building on the innovative Grants and Contributions work referenced above, the Hub led research to explore the possibility of establishing a program to accelerate the adoption of experimental interventions, innovation, challenges and prizes; outcomes-based funding; a platform to support the program; and new impact measurement methodologies in Government.

Between August 2016 and November 2017 [ * ], the Hub had three FTEs (full time equivalents) working on this initiative, which was a significant portion of the Hub’s resources. The Hub developed the business case [ * ] to establish the Impact Canada Initiative within PCO. This was a significant undertaking particularly given that these functions are not typically carried out by PCO and they were not resourced for this type of work. [ * ]. Ultimately, this gave new focus to the Hub’s mandate and helped introduce new tools and broader concepts.

In November 2017, the Hub was re-mandated as the IIU and the name change reflects the Unit’s evolving mandate, which includes housing the Impact Canada Initiative’s challenge platform in partnership with Government of Canada departments.

The IIU will act as a Centre of Expertise to support departments’ use of outcomes-based funding mechanisms, impact measurement techniques and behavioural insights to achieve better results for Canadians in priority areas.

Impact Canada is a whole-of-government effort designed to help departments accelerate the adoption of outcomes-based approaches that are intended to deliver meaningful results to Canadians. Impact Canada promotes the use of a range of innovative approaches that are enabled through the flexible terms and conditions, including:

• Challenges – Issuing prizes for whoever can first or most effectively find a solution to a defined problem, and/or making use of structured, open competitions to solicit proposals to fund the best ideas with the potential to solve thematic problems.
• Pay-for-Results – Using customized instruments to shift the focus towards issuing payments based on funding recipients achieving positive and measureable societal outcomes (e.g., social impact bonds, pay-for-success mechanisms).

The Government of Canada created the Impact Canada Initiative to help fund and support those initiatives that utilize these innovative approaches. Impact Canada and supporting changes have provided a pathway to allocate funds towards experimentation and innovation. The initial areas of focus for Impact Canada are: the Smart Cities Challenge, Clean Tech Impact, Responding to Canada's Opioid Crisis, and Improving Indigenous Outcomes

As noted by two senior level interviewees, the creation of Impact Canada was a significant undertaking for the Hub given its relatively small team. The flexible/whole-of-government terms and conditions and the creation of Impact Canada are seen as defining steps towards creating the conditions for a Public Service that is enabled, open to and supportive of innovation.

4. There is evidence that Canada, through the Hub, was an active member of the international community of practice and network of innovation lab.

Interview consultations and the documentation review revealed that the Hub invested in outreach and the establishment and broadening of networks to help advance the public-sector innovation agenda in Canada. 

Evidence from the evaluation indicate that the Hub has helped expand Canada’s presence and raise Canada’s profile internationally as a leader in the field of public sector innovation. Representatives from the Hub were invited to present at international conferences and meet with representatives of central agencies and cabinet offices from other countries.

During 2015-2016, the Hub established connections with public practitioners at the White House and the World Bank and it conducted outreach with other international innovation-oriented organizations such as MindLab, Nesta, etc. (Innovation Hub - Annual Report - 2015-2016).

In order to bring in external expertise, the Hub launched an Advisory Committee in August 2016 which included leading international academics and practitioners in behavioural insights and design. Its purpose was to “provide an opportunity for the Hub to better assist clients through expanded resources, networks, and additional targeted expertise.” (Q3 Management Report – December 31, 2016).

In 2016, a representative of the Hub, presented an overview of public sector innovation landscape in the Government of Canada; as well as Canada’s approach to innovation and results to the OECD’s Observatory on Public Sector Innovation (OPSI) as well as partnering to undertake a cross-government innovation review. 

In May 2017, a representative from the Hub spoke at the OECD Conference on Behavioural Insights. The Government of Canada, through the PCO Innovation Hub, co-chaired of the OECD Public Observatory for Public Sector Innovation National Contact Points, a network of OECD member countries actively engaged in public sector innovation. At the time, the Hub was working to finalize a partnership agreement with the OECD and Nesta to help advance innovation inside the federal government (March 2017; Meeting notes BICoP April 26 – Final), which is now in place. Hub staff collaborated with Bloomberg Philanthropies in the US and Nesta in the UK on innovative finance and challenges/prizes. The Hub also worked with the MaRS Solutions Labs to better understand the role of design in public policy and to assess the impact of federal innovation hubs and labs. 

In addition, the OECD, through the Observatory of Public Sector Innovation, has recently launched an Innovation Skills Builders Group in which Canada participates.

In supplement to external speaking engagements, the Hub invited representatives of other innovation labs to speak in Canada. Some examples include:

• ADM Committee
  • Simetrica (private company in UK) presented to the Committee on Designing a Strategy to Support the Impact Canada Fund (May 17, 2017)
• Deputy Minister Committee on Policy Innovation
  • [ * ] MaRS Solutions Lab presented on bringing discipline to public innovation (Dec 16, 2016).
  • A presentation from [ * ] Nesta, who spoke about future directions in the areas of social innovation, social finance and experimentation, and offer insights into the Government of Canada’s work in these areas (March 28, 2017).
  • [ * ] Solutions, Development & Innovation, 100 Resilient Cities, The Rockefeller Foundation presented to the CMCPI on June 30, 2017.
  • Simetrica (private company in UK) presented on Designing a Strategy to Support the Impact Canada Fund (May 18, 2017).
  • McKinsey and Company made a presentation on disruptive technologies in government (Nov 10, 2017).

There was also recognition that being a central agency of a federal government does help raise the international profile of the Hub and increases the uptake of Canadian research. The Hub also coordinated federal-provincial-territorial public service efforts through an annual federal-provincial-territorial Clerks and Cabinet Secretaries Conference on Policy Innovation. To date, three conferences have been organized.

While it was recognized that the UK and Australia were ahead of Canada in 2015, a number of interviewees noted that Canada has increased its profile on the international stage and this was attributed to the work of the Hub.

Conclusions and recommendations

Relevancy

The evaluation found that the Hub/IIU continues to have a relevant role in supporting government priorities by helping create a Public Service that is agile, as well as capable and supported in undertaking experimentation and igniting innovation. A key strength of the Hub/IIU is its ability to work with partners to co-design projects that challenge and test traditional policy, program and service design by testing alternatives. The IIU continues to focus on high impact and top line government priorities, as well as selecting projects that are a good investment; ones that allow government to springboard into new programs and services and/or are scalable across government.

The Hub’s approach was aligned with other international jurisdictions and the evolution of the IIU will continue to bring Canada closer to leaders in public sector innovation. There are opportunities to further disrupt the system and take innovation challenges to the next level. Impact Canada is expected to support this continued evolution.

The Hub was purposely placed at the centre of government to send a clear message across the Public Service of the importance of experimentation and innovation; to drive innovation and change; to help remove systemic barriers; and build capability to deal with complex problems. The Hub leadership recognized the importance of reflecting on its own placement, mandate and usefulness, and have evolved in response to changing needs during the evaluation period. This responsiveness has resulted in the re-mandating of the Hub as the IIU with an evolved mandate as described previously in this report.

Recommendations

It is recommended that the IIU:

• Utilize the Deputy Ministers’ Task Force on Public Sector Innovation as a venue to:
  • Tackle complex problems from a systems level.
  • Work with its partners to keep up, if not ahead, of the speed of transformation. This requires looking beyond innovation methods at the program level to system level change.
  • Work collaboratively within established networks to identify and remove impediments to innovation. These include: cultural, procedural, policy and resource capacity (such as IT system and HR).
• Map its relationships to the broader innovation ecosystem (including committees, networks and other labs) that the IIU intends to work with to achieve outcomes. This will be helpful to the IIU as it continues to evolve and in measuring its results.

Efficiency

The Hub invested a great deal of time in the start-up phase and there is evidence that these efforts have increased buy-in and support its work. Today the IIU is working with an increasing number of federal departments and agencies to test Behavioural Insights and prize/challenges approaches.

From an efficiency perspective, there is no reason for every department and agency to operate in isolation when developing capacity and testing innovative approaches to complex problems. There continues to be a central role for the IIU in developing relationships and networks, conducting innovative tests/proof of concepts projects, sharing learnings, and together, with key partners, being a catalyst for change.

For a small but growing group, it is clear that the Hub was, and the IIU continues to be innovative in how it uses its own resources and leverages external resources to ‘punch above its weight’ in order to have the greatest impact. Further investigation is required to assess the appropriate resource levels to meet demand as well as the impact of introducing a cost recovery model.

Recommendations

It is recommended that the IIU:

• Expand its network of experts and collaborative partnerships to include experts in the private and not-for-profit sectors, who have shared interest in a competitive and thriving society and economy.
• Put in place a process to track demand for IIU services in relation to resources to ensure there is sufficient capacity to meet the needs within the federal Public Service. This might require a more in-depth review of the level of resources that are required to fully deliver on its mandate in light of expected demand. Any future assessment of resourcing levels should include an assessment of a cost recovery model.

Effectiveness

The evaluation found evidence that the Hub has achieved its short-term and operational outcomes and the IIU is on track to achieve its longer-term outcomes.

There is evidence to demonstrate that the Hub has created and tested new tools and approaches and that these are becoming embedded in the policy development and delivery process system-wide. When the Hub was introduced, some departments were already testing Behavioural Insights and design-thinking; however, it was recognized that the Hub brought an appropriate level of rigour as well as new ideas/tools that could be tested. There is evidence that the Hub had an impact on its partners given that a number are continuing to advance innovative projects that were co-designed and delivered with the Hub and have transferred the learning to other policy/program areas.

There are a number of networks and mechanisms in place to exchange information and best practices and spread successful experimentation and innovation across the portfolio. These mechanisms include DM and ADM Committees, communities of practices that target employees at all levels, and other resources. The Hub did well in promoting its many successes, however, moving forward, more could be done by the IIU to highlight areas where projects or partnerships faced challenges. This would provide a signal that innovation is not synonymous with success. The Hub was also an active member of the international community of practice and network of innovation labs.

There is solid evidence that positive policies are in place to provide authority for departments to use new and innovative approaches to the design of policies, regulations, programs and services. These include the Declaration of Public Sector Innovation; the Experimentation Direction; terms and conditions for Grants and Contributions; and the Impact Canada Initiative which allows for more innovative approaches to solving significant policy challenges (these include prizes and incentive-based funding models).

The evaluation did not find evidence of a data collection plan to support the Performance Measurement Strategy; however, it may have been developed after the period under evaluation.

Recommendations

It is recommended that the IIU:

• Use the annual reports to publish all information related to the projects it supports (that move beyond the expression of interest stage) in order to showcase what has worked, transfer knowledge, and – more importantly – to share lessons learned and identify system-level barriers to innovation. This will send a powerful message and will help create an environment which demonstrates that informed risk-taking and failure is part of the process and an important part of the learning process.
• Focus on building capacity and scaling up innovation by leveraging networks and existing communities of practices such as Policy, HR and Regulatory. Bring together key stakeholders that would directly benefit from a deeper discussion of findings and how they might apply the findings in different areas. Expand the use of learning events such as the Randomized Controlled Trials workshop.
• [ * ] as part of ongoing operational planning, develop key performance indicators that measure activities and outcomes and ensure that data and information are being collected on a regular basis to inform ongoing monitoring and reporting on demand for services and results. Given that the IIU is part of the Results and Delivery Unit, it has a leadership role in establishing a plan with detailed indicators that measure change at the system levels. Note: In June 2018, the IIU launched its Theory of Change as the framework for planning and measuring results. According to the IIU, the Theory of Change has helped the Unit build consensus and articulate the systems changes, it, and its partners, are seeking to achieve and how they will be realized. The Unit further committed to move the Theory of Change into action by rolling out the monitoring and developmental evaluation strategies.

Management response and action plan

	Recommendations	Management Response and Action Plan
Relevance	Utilize the Deputy Ministers’ Task Force on Public Sector Innovation as a venue to: Tackle complex problems from a systems level. Work with its partners to keep up, if not ahead, of the speed of transformation. This requires looking beyond innovation methods at the program level to system level change. Work collaboratively within established networks to identify and remove impediments to innovation. These include: cultural, procedural, policy and resource capacity (such as IT system and HR).	Agreed: As secretariat to the DM Task Force on Public Sector Innovation, the IIU will continue to provide leadership in helping it make concrete and sustainable change at a systems level. In managing the first cohort of GC Entrepreneurs, the IIU is guiding them to address challenges to systems-level problems in the areas of HR, procurement and Grants and Contributions programming, as well as projects exploring the application of disruptive technologies like Blockchain, AI and real time data collection. Examples of these include: Pay-for-results projects in the areas of opioid detection technology, and improving access to perishable food in the north Innovative approaches to enabling cross-sector mobility Development of a technology-based workplace wellness initiatives intended to better understand and enhance workplace well-being in the federal public service Exploring the use of distributed ledger technology (e.g., Blockchain) to improve service to Canadians Piloting the use of innovative technologies (e.g., AI) for machine-based translations Target date: Ongoing as this is intrinsic to the lead role of the Impact and Innovation Unit. Position responsible: Assistant Secretary, Impact and Innovation Unit
	Map its relationships to the broader innovation ecosystem (including committees, networks and other labs) that the IIU intends to work with to achieve outcomes. This will be helpful to the IIU as it continues to evolve and in measuring its results.	Agreed: The IIU will produce a one page graphic outlining the relationship of the IIU vis-à-vis other actors in the broader federal innovation ecosystem. Target date: March 2019 Position responsible: Assistant Secretary, Impact and Innovation Unit
Efficiency	Expand its network of experts and collaborative partnerships to include experts in the private and not-for-profit sectors, who have shared interest in a competitive and thriving society and economy.	Agreed: On November 1, 2018, the IIU held the inaugural meeting of its new Impact Canada Advisory Committee. This committee is expressly designed to draw in domestic and international private, not-for-profit, as well as academic expertise to guide efforts of the Impact Canada Initiative. Target date: November 2018 - Completed Position responsible: Assistant Secretary, Impact and Innovation Unit
	Put in place a process to track demand for IIU services in relation to resources to ensure there is sufficient capacity to meet the needs within the federal Public Service. This might require a more in-depth review of the level of resources that are required to fully deliver on its mandate in light of expected demand. Any future assessment of resourcing levels should include an assessment of a cost recovery model.	Agreed: The IIU has put in place an Expression of Interest process, which allows IIU to formally track and assess service demands from potential federal partners. Target date: October 2018 - Completed Position responsible: Assistant Secretary, Impact and Innovation Unit
Effectiveness	Use the annual reports to publish all information related to the projects it supports (that move beyond the expression of interest stage) in order to showcase what has worked, transfer knowledge, and – more importantly – to share lessons learned and identify system-level barriers to innovation. This will send a powerful message and will help create an environment which demonstrates that informed risk-taking and failure is part of the process and an important part of the learning process.	Agreed: The IIU continues to be proactive, open and transparent in all aspects of its work, as demonstrated by: Ongoing publication of its annual report Ongoing publication of its case studies Active promotion of Impact Canada projects via impact.canada.ca Ongoing presentations and engagement opportunities at domestic and international events Robust social media engagement Continuing blogs series highlighting work, lessons learned and future plans Target date: Ongoing as this is intrinsic to the lead role of the Impact and Innovation Unit. Position responsible: Assistant Secretary, Impact and Innovation Unit
	Focus on building capacity and scaling up innovation by leveraging networks and existing communities of practices such as Policy, HR and Regulatory. Bring together key stakeholders that would directly benefit from a deeper discussion of findings and how they might apply the findings in different areas. Expand the use of learning events such as the Randomized Controlled Trials workshop.	Agreed: The IIU continues to manage several communities of practice related to behavioural insights at both the federal and federal/provincial/territorial levels. Plans are under way to establish a federal community of practice focused on prizes/challenges. In addition, IIU staff will continue to: Broaden outreach to other functional communities (HR, finance, regulatory) via the DM Task Force on Public Sector Innovation Continue in its role as co-chair of the ADM Committee on Experimentation Regularly present work at national and international fora Utilize the growing cadre of Fellows to work with, train and engage fellow public servants. Explore partnership opportunities with the Canada School of Public Service to scale up the States of Change learning curriculum, currently offered to the GC Entrepreneurs under the DM Task Force on Public Sector Innovation. Target date: Ongoing as this is intrinsic to the lead role of the Impact and Innovation Unit. Position responsible: Assistant Secretary, Impact and Innovation Unit
	[ * ] as part of ongoing operational planning, develop key performance indicators that measure activities and outcomes and ensure that data and information are being collected on a regular basis to inform ongoing monitoring and reporting on demand for services and results. Given that the IIU is part of the Results and Delivery Unit, it has a leadership role in establishing a plan with detailed indicators that measure change at the system levels. Note: In June 2018, the IIU launched its Theory of Change as the framework for planning and measuring results. According to the IIU, the Theory of Change has helped the Unit build consensus and articulate the systems changes, it, and its partners, are seeking to achieve and how they will be realized. The Unit further committed to move the Theory of Change into action by rolling out the monitoring and developmental evaluation strategies.	Agreed: The IIU has developed and implemented a Theory of Change that provides the basis for its performance measurement activities. Target date: October 2018 - Completed Position responsible: Assistant Secretary, Impact and Innovation Unit

Appendix A: Detailed overview of methodology

The evaluation of the PCO Central Innovation Hub was conducted between July and September 2018. The evaluation was based on the following methodologies.

A review of relevant literature

This review was designed to cover a range of themes pertaining to innovation hubs and approaches used in other jurisdictions and sectors to drive innovation in policy and service delivery. The literature review was focused on discovering and analyzing trends, common approaches, and common principles of public sector innovation hubs, including:

• Their intended role within the public administration;
• How they are positioned/structured;
• The impact/effect they are intended to have on government more widely; and
• Emerging results, best practices and lessons learned.

Documentation and file review

The IOG reviewed several dozen files and reports provided by the Innovation Hub related to strategic policy, planning and reporting, as well as documents related to the various communities of practices, committees and international working groups (including presentations and submissions).

Interview consultations

The IOG conducted interview consultations with eight federal employees who were either involved in the creation of the Hub or worked in the Hub during the first two years; as well as the co-chairs of the Deputy Minister Committee on Policy Innovation.

Case study review

The IOG reviewed reports from 8 partnership projects that were undertaken during the two-year timeframe of the evaluation. After reviewing these documents, the IOG selected two projects which were representative of the types of projects the Hub was involved with during its first two years of operation for deeper analysis. This included 6 interview consultations with program staff who worked on the project, project partners and an interview with an external advisor that is part of the Hub’s Trusted Advisors.

These methodologies provided multiple lines of evidence that informed the evaluation findings, conclusions and recommendations.

Evaluation limitations

There were a number of limitations associated with this evaluation, as follows:

• The evaluation was focused on first two years of the Hub’s operations. During this time, the Hub spent the first year establishing the Hub’s infrastructure. At the same time, during its first two years, the Hub was quickly evolving which made the assessment of results more challenging given that, in effect, the key activities and focus of the Hub [ * ] were evolving by year 2. These factors, combined with the fact that two years is generally regarding as a short timeframe to evaluate results achievement.
• The short timeframe for, and timing of, the evaluation limited the number of interviews that could be undertaken. In addition, the interviews were primarily with former or current Hub staff. However, project partners and an external advisor were interviewed as part of the case study review.
• Given the timeframe, there were a limited number of projects that had been undertaken and therefore a limited number of partners to consult.

Appendix B: Interview Guide

Evaluation of the PCO Central Innovation Hub

Introduction

Thank you for agreeing to participate in this evaluation process. The Institute of Governance has been hired to conduct this independent evaluation of the Innovation Hub located at the PCO. What follows is the interview guide to support the evaluation.

All information provided will be kept in strict confidence according to the Treasury Board Policy on Privacy Protection.

Background

This evaluation is largely retrospective in nature and is focused on the establishment and early progress of the Central Innovation Hub (the Hub) during its initial two years. The Hub was first established in 2015 and was rebranded as the Impact and Innovation Unit (IIU) in November 2017, with an expanded mandate to include the delivery of the Impact Canada Initiative (ICI).

The Hub was established to:

• Assist federal departments, agencies, and institutions to manage complex policy and service delivery challenges by providing resources and support to these organizations as they examine potential ways to adopt new and emerging approaches, tools and techniques.
• Help federal government institutions test, document and accelerate innovation, and to build capacity and knowledge to support replication and scaling of successful techniques across the Public Service.
• Amplify and align successful initiatives across departmental lines, and create synergies with the broader work of the Privy Council Office.

The Innovation Hub was intended to take on three functions:

• Central resource: best practices, new tools, approaches, and techniques.
• Connect and convene: establish networks and partnerships. Act as central node in broader innovation network (i.e., with other countries such as UK).
• Innovation driver: work directly with departments to define problems, plan projects, assess solutions, and evaluate initiative. The Hub can also work as an active member of the project team throughout the lifecycle of project.

During the initial years of its establishment, the Hub focused on:

• behavioural insights;
• data and design thinking; and
• the application of tested tools and techniques to different areas.

Scope of the evaluation

The scope of this evaluation is from the time of the Hub’s establishment in 2015 to November 2017. The evaluation does not include the newly expanded mandate.

Interview questions

Background

1. To begin, can you please describe your job title and your responsibilities in relation to the Hub?
2. Can you provide us with the background and context that led up to the establishment of the Hub? As a signature corner stone of the Government of Canada’s Blueprint 2020 – was the Hub positioned for success?
   Probes:
   • Was the Hub properly situated organizationally?
   • Was it properly resourced?

Emerging results

3. Can you explain the first two years of operation? What went well? What delays/barriers were encountered? Has it achieved its expected outcomes? What would you do differently today?
   • How open are other departments that were already working in the innovation lab space such as ISED?
4. Can you point to any new tools and approaches that have become embedded in the policy development and delivery process system-wide as a result of the work of the Hub?
   Probes:
   • Are policies in place to provide authority for departments to use these more innovative approaches to policy, programs and service delivery? Are their barriers to its usage?
5. What mechanisms have been established to spread successful innovation across portfolios (e.g., Hub annual reports/case studies, interdepartmental communities of practice, departmental innovation labs)?
   Probes:
   • Have you seen uptake and experimentation with new innovative approaches across federal departments and agencies as a result of the Hub’s work/activities? Is there evidence of system-wide uptake?
   • Is there evidence that the Hub and the initiatives supported have strengthened policy and program outcomes?
6. Are some departments and agencies more open to experimenting with these new approaches?
7. What has been the role of the Hub in ensuring that Canada is an active member of the international community of practice and network of innovation labs? What is Canada’s position within the international community? What more could be done to position Canada as a leader in public policy and program innovation?
8. Generally, has the Hub been successful in measuring and demonstrating its results? In what areas is it having the greatest impact?
9. Has there been a reduction in overlap or duplication in innovative experiments and research?
10. What have been the barriers to results achievement? How has the Hub tried to overcome these barriers?

Continuous learning

11. Behavioural insights and data and design thinking were the initial signature initiatives. Were these good choices? What role should they play going forward? Should they continue to be a focal point?
12. The Hub has a well-defined process and method for selecting projects (e.g., policy promise, ability to scale, departmental readiness, projects that have been successful in other jurisdictions). Has this process been effective at selecting the right projects for experimentation and proof of concept?
13. In terms of the Hub’s approach and focus, are you aware of different approaches found in other jurisdiction that have been more successful at igniting innovation in public policy and service delivery?
14. How effective has the Hub been at bringing in non-traditional players into government to support innovation and co-design and co-created solutions to complex problems (i.e., private, not-for-profit, international stakeholders)?
15. Does there continue to be a need for the Hub/IIU to spark innovation across government?
    Probe:
    • If there were no Hub/IIU in PCO today, would the Clerk feel the need to create one?
16. Are there alternative approaches that may be more efficient at driving innovation in the federal Public Service?

Conclusion

17. Do you have any other comments to inform the evaluation and/or improve the overall impact of the Innovation Hub/IIU?
18. Is there someone else you would recommend we interview?

~ Thank you for participating in this evaluation process ~

Appendix C: Overview of projects reviewed

As part of the evaluation, the IOG reviewed the following reports on projects undertaken with the support of the Innovation Hub:

• Behavioural Insights Project: Increasing Recruitment of Women into the Canadian Armed Forces, Innovation Hub – Privy Council Office
• Behavioural Insights Project: Canadian Women with a Career in the Armed Forces, Impact and Innovation Unit – Privy Council Office
• Designing Interventions: To Improve Organizational Culture for Persons with Disabilities in Health Canada, Central Innovation Hub (March 2017)
• Behavioural Insights Project: Increasing Take-up of the Canadian Learning Bond, Impact and Innovation Unit – Privy Council Office
• Language of Work: Bilingualism in the Workplace Consultation and Design Workshop Report (December 2016)
• Family Class Design Challenge: Using Service Design Principles to Improve the Experience of Family Class Sponsorship Clients, Immigration, Refugees and Citizenship Canada (IRCC), Impact and Innovation Unit, Privy Council Office, 2017
• Behavioural Insights Project: Increasing Responsive Rates to a Statistics Canada Survey, Innovation Hub – Privy Council Office