Evaluation of Health Canada's Solutions Fund

Final report
February 2023

Prepared by the Office of Audit and Evaluation
Health Canada and the Public Health Agency of Canada

Table of Contents

• List of Acronyms
• Executive Summary
  • Context
  • What we found
  • Recommendations
• Purpose of the Evaluation
• Evaluation Scope and Approach
• Context
• Key Findings: Encouraging the use of innovation and experimentation
• Key Findings: Creating a conducive environment for innovation and experimentation
• Key Findings: Helping participants to succeed
• Key Findings: Solutions Fund project selection
• Key Findings: Supporting the scaling up of Solutions Fund projects
• Key Findings: Opportunities to improve processes associated with the Solutions Fund
• Key Findings: The way forward for the Solutions Fund
• Conclusions
• Recommendations
• Management Response and Action Plan
• Appendices
  • Appendix 1: Data Collection and Analysis Methods
  • Appendix 2: Solutions Fund Timeline (2016 to 2022)
  • Appendix 3: Health Canada Innovation and Experimentation Framework
  • Appendix 4: Financial summary of approved Solutions Fund projects
  • Appendix 5: Summary of funded Solutions Fund projects
  • Appendix 6: Proposals received by SF in Application Rounds
  • Appendix 7: Environmental scan of Innovation and Experimentation projects across Health Canada
  • Appendix 8: Approved Solutions Fund projects by branch, (2018-19 to 2021-22)
• Endnotes

List of Acronyms

ADM

Assistant Deputy Minister

CFO

Chief Financial Officer

CFOB

Chief Financial Officer Branch

CPAB

Communications and Public Affairs Branch

CSB

Corporate Services Branch

DGIP

Director General committee for Investments and Projects

DM

Deputy Minister

EC-FIPT

Executive Subcommittee on Finance, Investments, Projects and Transformation

FTIR

Fourier-transform infrared (spectroscopy)

HC

Health Canada

HECSB

Healthy Environments and Consumer Safety Branch

HPFB

Health Products and Food Branch

IM/IT

Information Management / Information Technology

IMSD

Information Management Services Directorate

MAF

Management Accountability Framework

PSES

Public Service Employee Survey

ROEB

Regulatory Operations and Enforcement Branch

SF

Solutions Fund

SFGC

Solutions Fund Governance Committee

SFRP

Solutions Fund Review Panel

SWOT

Strengths, Weaknesses, Opportunities, and Threats

TBS

Treasury Board of Canada Secretariat

Executive Summary

Context

Health Canada (HC) developed the Solutions Fund (SF) in 2017 in response to the Treasury Board Secretariat (TBS) Experimentation direction for Deputy Heads (2016). This direction requires departments to test new approaches and measure impact in order to instill a culture of innovation, measurement, and evaluation in program and policy design and delivery. The first call for proposals was launched in 2018.

The intention of the SF is to provide opportunities for Health Canada employees to identify, develop, and implement projects that will improve the health and safety of Canadians, enhance the efficiency and effectiveness of the Department, and deliver greater value to taxpayers.

The SF distributes small investments, specifically seed funding, for employee-led innovation and experimentation projects. Projects are funded through the following two streams: 'Stream 1' is Exploration and 'Stream 2' is Experimentation. By testing innovative projects on a small scale to find out what works and what doesn't, HC aims to promote stronger, evidence-based decision making before making larger investment decisions.

This evaluation assessed the achievement of the SF's goals and examined the appropriateness and effectiveness of its current design and implementation. The evaluation covered SF activities from April 2018 to July 2022.

What we found

Overall, the Solutions Fund (SF) is regarded as a well-planned and well-run initiative led by a dedicated team which, in partnership with corporate support services, provides comprehensive supports to SF applicants and participants.

The SF has successfully employed a variety of outreach activities to encourage HC employees to submit applications. The SF is a key contributor in promoting an innovation and experimentation mindset throughout HC. However, there is potential for improvements in sharing project results and lessons learned to further build this culture and apply the lessons learned from SF projects across HC.

The SF has developed a standardized process of proposal review and project selection. The SF team has improved project assessment processes, including the adoption of assessment grids and leveraging expert reviewers across HC and the Government of Canada. The oversight mechanisms for SF projects are well-defined with clear roles for the project team and the SF Governance Committee.

In addition, no promising proposals have been rejected due to a lack of funding.

The SF incorporates a vision of preparing successful innovation ideas to 'scale up' in order to improve program and service delivery, but projects considered to be successful still face the reality of needing branch or department-level support to be implemented, and must still pass required and demanding due diligence processes. For the purposes of this report, the term 'scale-up' refers to any further implementation of a SF-funded project once it has completed its Stream 2 activities. This includes further piloting as well as further investments through Branch or Department-level investment planning processes. Some SF projects have shown promising results through experimentation. Given the relatively recent completion of most Stream 2 projects and the associated processes and requirements to put them into practice, no project has yet been fully implemented. In addition, there is a question of if, and how, promising SF projects could be supported for potential scale-up after they have completed Stream 2.

Recommendations

Various lines of evidence were reviewed as part of the evaluation, including files and documents, performance data, and data from interviews with internal and external interviewees. As a result, two recommendations emerged.

Recommendation 1) Improve the sharing of project results and lessons learned from SF projects across HC.

Sharing project results and lessons learned beyond close-out presentations has proven to be challenging, but there is potential for the Department to maximize the benefits from SF's small-scale experimentation projects. For example, sharing results and lessons learned would allow others to build on recent experiences and avoid duplication of effort or covering off areas that have been previously addressed. The SF team should continue to work with its corporate partners and stakeholders to enhance employee and management awareness and access to SF project experiences and lessons learned.

Recommendation 2) Determine the future role of the SF in supporting the scale-up of successful Stream 2 projects and growing the capacity for innovation and experimentation across HC.

Nine of the 22 projects selected and supported by the SF have completed their Stream 2 activities. However, while some have been piloted, none have yet been implemented at a wider Branch or Department level. The SF should explore how it can build on existing collaborative relationships with relevant corporate service groups to determine what further role the SF itself could play in supporting future 'scale-up' of successful Stream 2 projects.

Although most branches have been represented in employee-led SF projects, the level of engagement in innovation and experimentation varies across HC. Given that innovation and experimentation is a recognized departmental priority, the SF should determine if and how it could play a role in increasing the level of knowledge and action on innovation and experimentation across Health Canada outside of the Solutions Fund itself.

Purpose of the Evaluation

The purpose of this evaluation was to assess the achievement of the goals of the Health Canada (HC) Solutions Fund and examine the appropriateness and effectiveness of its current design and implementation.

The Solutions Fund, administered by the Chief Financial Officer Branch (CFOB), requested this evaluation to ensure that the Solutions Fund is operating as effectively as possible, and to determine if there are any potential improvements that could be made. This is the first HC evaluation of the Solutions Fund (SF).

Evaluation Scope and Approach

The scope of this evaluation covered HC Solutions Fund activities from fiscal year 2018-19 to 2021-22. Multiple lines of evidence, profiled in Appendix 1, were used to address questions that focus on the following:

1. To what extent has the Solutions Fund achieved its goals:
   1. Encouraging the use of innovation and experimentation to improve program and service delivery; and
   2. Creating an environment conducive to experimentation and innovation?
2. Is the current design and implementation of the Solutions Fund appropriate and effective with respect to:
   1. Helping SF participants succeed;
   2. Funding all promising project submissions;
   3. Supporting the scaling up of successful SF projects; and
   4. Encouraging more evidence-based decision making?
3. Are there opportunities to streamline processes without having a negative impact on results?

Context

The Solutions Fund (SF) was designed in 2017 and launched in May 2018 at the request of the HC Deputy Minister, in response to the 2016 Treasury Board Secretariat (TBS) Experimentation Direction for Deputy Heads^{Footnote 1} calling for federal government innovations to address persistent problems. The SF was originally managed by the Communications and Public Affairs Branch (CPAB), as it was originally created and managed by the Blueprint 2020 team, which was housed in that branch. The SF was transferred to the Chief Financial Officer Branch's (CFOB) Costing, Revenues, Investments and Projects Directorate in January 2022. A timeline for the SF is provided in Appendix 2.

The objective of the SF is to provide a supportive environment for employee-led development and testing of innovative ideas via seed funding and technical advice, in order to learn from the successes or failures of promising projects. Manager approval is not required for employees to propose an idea to the SF.

The SF contributes to the goal of fostering a risk-tolerant culture of innovation and experimentation throughout HC, as featured in the Departmental Plan and Departmental Results Reports.^{Footnote 2} It aligns with the 2020 HC Innovation and Experimentation Framework shown in Appendix 3 and the TBS Management Accountability Framework (MAF) under 'Results Management'.

The SF has issued calls every spring and fall for proposals to HC employees, inviting project submissions for Exploration of solutions to problems (Stream 1) and Experimentation for testing a potential solution (Stream 2). The focus of Stream 1 Exploration is to identify a problem or an area of business that needs to be addressed and potential innovative solutions. Stream 2 Experimentation is meant for testing an identified hypothesis or a potential promising solution to a problem or a business need. Costed proposals are vetted by SF staff and relevant experts. Coaching on experimentation design and project management is then provided to improve proposals. Proposals that are deemed ready for consideration are discussed by the SF Review Panel, led by the two Assistant Deputy Minister (ADM) Innovation Champions, who then recommend proposals to the Deputy Minister (DM) for final approval. The progress of funded projects is overseen by SF Governance Committee members who monitor progress, approve project amendments, and participate in the assessment of Stream 1 projects applying for Stream 2 funding.^{Footnote 3}

Between May 2018 and July 2022, the SF selected 22 projects for funding with a total amount of $4.4M committed to the projects. In 2022-23, the allocated annual budget for the SF of $2M was a spending target based on historical investments, not an investment cap. See Appendix 4 for a breakdown of project funding by Stream 1 and Stream 2 and Appendix 5 for detailed project descriptions.

Projects that complete Stream 2 are encouraged to work with their Branch leadership to apply lessons learned, including the implementation or scaling up of promising solutions. At this stage, projects leave the SF experimentation 'sandbox' and may enter the branch or Department-level investment planning process. The SF supports the exploration and testing of projects in an isolated 'sandbox' environment, i.e., a stand-alone configuration with the purpose of experimenting and testing solutions without having the risk of compromising departmental operations or data. This allows the project to fully explore possibilities of a hypothesis by removing restrictions, security measures and risks that would be associated with real-world 'production' environment.

Key Findings: Encouraging the use of innovation and experimentation

The first communications plan, developed in 2018, incorporated the following engagement mechanisms: the Deputy Minister's (DM) message to employees, Broadcast News, the mySource intranet page, and content for Health Television (HTV) broadcasts in HC and PHAC buildings. In 2018, the Solutions Fund (SF) started a GCconnex page in order to network with interested employees. In 2020, the SF launched the HC Innovation and Experimentation page on GCpedia^{Footnote 4} that includes information about the SF concept and has links to resources for starting a proposal.

The SF team has leveraged several other platforms to promote its work, including regular presentations at management meetings, at the Beyond2020 Virtual Innovation Fair in October 2020, and to the HC Young Professionals Network in June 2022. In February 2020, the SF hosted the first PHAC and HC Innovation and Experimentation Symposium which was attended by 500 public servants. The SF has also promoted online training sessions from the Treasury Board Secretariat (TBS) that are hosted by the Canada School of Public Service. Lastly, the SF continues to host a Mindset Matters webinar series. Several applicants indicated that they had heard about the SF through these promotion efforts, especially through Broadcast News.

While outreach efforts have been successful, many internal interviewees, principally participants and staff, as well as a few external interviewees indicated that SF outreach could be improved by engaging with both employees and senior management. Encouragingly, it was noted by staff that, after a session in May 2022 for aspiring managers, there was a substantial increase in the number of SF applications, from two to 15. In response to this, the SF team developed a new communications plan that targets both employees and managers to maximize outreach.

Other ideas voiced by individual interviewees to improve awareness of the SF included greater involvement from the Communications and Public Affairs Branch (CPAB) in its promotion, as the small SF team has had limited time to commit to promotion efforts, adding the SF to the new employee orientation package, and working with other Government of Canada hubs to raise awareness of innovation and experimentation across all departments.

In the context of ongoing promotion of the SF, 47 project proposal applications were received over seven application cycles from 2018 to 2021, as shown in Appendix 6. An average of six proposals were received per call, with the trend decreasing during the time that the Department was responding to the COVID-19 pandemic.

Key Findings: Creating a conducive environment for innovation and experimentation

The Solutions Fund (SF) embodies the innovation and experimentation mindset, as it was specifically developed in response to the 2016 TBS Experimentation Direction for Deputy Heads that requires departments to create an environment that fosters innovation and experimentation. HC has consistently received the highest possible Management Accountability Framework (MAF) scores from TBS for its efforts in experimentation since that management area was introduced in 2018-19. The SF has been highlighted as the primary reason for these high MAF scores and, according to external interviewees, is highly regarded by TBS. TBS has also referred other federal government departments to Health Canada for guidance on innovation and experimentation given the success of the SF. TBS is currently revising the Experimentation section of the MAF to encourage departments to use experimentation early in the program lifecycle and to generate evidence for 'high-yield' investments.

Additionally, the SF and its projects have been consistently highlighted in the Experimentation section of Departmental Plans and Departmental Results Reports since 2018-19. The latest editions of these documents have explained how all of HC's experimentation efforts correspond to the Departmental Innovation and Experimentation Framework, developed by the SF team on request of the DM, and approved in March 2020. See Appendix 3 for the Departmental Innovation and Experimentation Framework.

In a survey conducted by TBS Experimentation Works in 2022, HC had the most respondents involved in an experiment in the last five years. The survey identified the following major barriers to experimentation across various federal departments:

• resources, including lack of time and funding, and limited access to technology;
• process, including compliance with rules, complex procurement processes, and lack of clarity on ethics); and
• culture, including low risk tolerance and a lack of executive support.

The SF addresses all three of these barriers by providing resources, guidance, and by garnering support from upper management. Notably, DM-level support was key to launching and sustaining the SF.

There is evidence that an innovation and experimentation mindset has been growing across HC since the start of the SF. As seen in the chart below, based on the 2020 Public Service Employee Survey^{Footnote 5} (PSES), the proportion of HC respondents who agreed that they are encouraged to be innovative or take initiative in their work rose steadily from 2017 to 2020.^{Footnote 6} The proportion of this positive attitude has been slightly but consistently higher in HC than the average for the federal public service as a whole. A similar trend was recorded in the proportion of HC employees who felt that they would be supported if they proposed a new idea, with 59% responding positively in 2017, and 71% responding positively in 2020. On a related note, HC was selected as one of Canada's top 100 employers in 2022^{Footnote 7} (Globe & Mail), citing the SF as one of the reasons for this selection, due to it inspiring innovative thinking and action from employees.

Question 14: I am encouraged to be innovative or to take initiative in my work.

Note: The percentage of positive responses was calculated by combining 'strongly agree' and 'somewhat agree' responses.

In addition, as shown in Appendix 7, there has been a growing number of active innovation and experimentation projects across HC. In 2020, there were 27 innovation and experimentation projects across the Department, 13 of which were funded by the SF. In 2022, there were 40 such projects, nine of which were funded by the SF.

While all HC branches are represented among the employee-led proposals, employees in some branches are more active and apply and participate more consistently in the SF. As shown in Appendix 8, the Regulatory Operations and Enforcement Branch (ROEB), the Healthy Environments and Consumer Safety Branch (HECSB), and the Health Products and Food Branch (HPFB) have had a higher rate of projects funded by the SF than the other HC branches. Some branches also exhibit integration of innovation and experimentation more at the branch level. For example, ROEB established a Transformation Office to foster branch-level innovations around the same time that SF was launched. Additionally, HECSB and CFOB have included experimentation and innovation in their strategic plans.

A few internal interviewees indicated that ROEB's high degree of engagement was due to individuals promoting and supporting innovation and experimentation to their organizational colleagues. Moreover, the experience gained by ROEB employees from successful SF applications has enhanced ROEB's capacity for further innovation and experimentation.

Though the SF encourages proposals from each of the Department's business lines, the vast majority of proposals and funded projects have involved the information technology (IT) domain, i.e., applying IT solutions to adapt current tasks with the goal of more efficiently and effectively using existing resources. A few staff and governance interviewees have attributed the 'popularity' of the IT focus in proposals to a perception that technology and innovation are highly related. It should be noted that the IT-focused projects funded by SF cover a wide variety of objectives, such as improving the public's experience of accessing HC information and reducing the demand of highly repetitive tasks on HC staff. In addition, innovative projects go beyond enhancing or adapting current practices with IT-related innovations. More recently, SF has funded some policy experiments (e.g., Heart, Individualized Accommodation Passport). Overall, there appear to be opportunities to expand the variety of proposals to the SF by further encouraging employees in all branches to apply, even if they are not focused on IT-related problems or solutions.

As noted by many interviewees, the results and lessons learned from SF projects are not widely disseminated throughout HC. In particular, SF project close-out presentations are not accessible to all HC staff, especially those considering similar innovations. Furthermore, there is no repository that HC staff can access to learn from SF project findings, although this information is available on request.

It was noted that the SF team has taken some measures to share project results through various shared platforms, such as the Statistics Canada RADAR Innovation Database.^{Footnote 8} However, low user uptake or discontinuation of these options has led the SF team to look for alternatives. Some SF governance interviewees proposed that SF project results be included in a new CFOB effort to catalogue lessons learned from large corporate investments. They also proposed that there should be more systematic sharing of SF project results and lessons learned with departmental committees involved in investment planning, including the Director General Committee for Investments and Projects (DGIP) and the Executive Subcommittee on Finance, Investments, Projects and Transformation (EC-FIPT).

A few interviewees, speaking from participant, governance, and support perspectives, highlighted the importance of informing HC employees of all SF project findings to further promote experimentation and to provide evidence for decisions on where and when to apply innovative ideas or when conditions are not yet right.

It is important to highlight that Stream 2 projects are still considered valuable, even without promising evidence of success, as they provide cost-effective lessons on ideas that should not be pursued. For example, the GC Recalls and Safety Alerts Voice Assistant project and the IT National Service Desk Online Chatbot project discovered that the voice assistant and chatbot platforms did not help to improve services as expected at this point in time. It was found that improvements in underlying information infrastructures would be necessary to implement these technologies. These examples can serve to inform other groups that are interested in applying these kinds of technologies for more efficient use of staff resources and improved user experiences.

Some of the projects that have been completed for Stream 2 have shown positive results. For example, Project Cyclops tested an app that would enable inspectors to use a portable device like a smartphone to scan labels on health products. This would allow ROEB to strengthen its inspection processes by cutting down on the time required to review labels and increase its capacity to review a larger number of product labels. See the textbox below for more details.

Another example is the Fourier-transform infrared (FTIR) spectroscopy project to improve HC's lab-based testing for contaminants in natural health products. This project reduced the number of tests required to identify different types of materials and opened the possibility of detecting microorganisms such as moulds, and producing more results with one kind of test, thus increasing lab efficiency and effectiveness.

However, given the relatively recent completion of most Stream 2 SF projects and the associated processes and requirements to put them into practice, no project has yet been fully implemented, although a few are currently with their branches for piloting and next steps. See the section below on 'Supporting the scaling up of Solutions Fund projects' for more details, as well as Appendix 5 for the status of individual SF projects.

Key Findings: Helping participants to succeed

Health Canada's Solutions Fund encourages experimentation in a risk-averse environment through internal and external partnerships. The Solutions Fund team and corporate support services are made available to applicants and participants throughout the funding process. The oversight mechanism for funded projects is well-defined for each project team and the Solutions Fund governance committee.

The Solutions Fund team provides comprehensive and effective coaching and tools to applicants and participants, in collaboration with corporate support services.

According to the majority of interviewees, the Solutions Fund (SF) team plays an outstanding role in facilitating a collaborative coaching relationship between project leads and the SF team, together with corporate support services and subject matter experts. Project teams received individualized coaching and support to help develop their experiment design and refine their proposals. This coaching also extends to supporting the implementation of funded projects. Corporate support services include, but are not limited to, project and financial management, information management, human resources, and procurement services. For example, financial management experts have assisted applicants in completing their project costing template.

Furthermore, one applicant mentioned that SF staff had connected her with three people who had experience with Artificial Intelligence (AI) and were working on AI projects. Some projects have also benefitted from the SF team connecting them with subject matter experts outside of HC, including from TBS and the Canada School of Public Service. Some project participants were also connected with external supports such as universities. For example, Project Apollo worked with HECSB, ROEB, the Information Management Services Directorate (IMSD), CPAB, the Canada School of Public Service's Digital Academy, EcoSchools, and VR Vision for their project.^{Footnote 9}

The involvement of corporate support services in SF projects is started very early by design, where applicants are asked to think about the kind of supports that could benefit their project; this is built into the SF submission proposal form. Additionally, the SF website on GCconnex encourages applicants with an idea to contact them early in the process.^{Footnote 10} The project work plan, which includes clearly defined roles for corporate support services, must be reviewed and signed off by the relevant corporate service representatives.

The SF submission proposal form is part of a suite of tools created to help applicants throughout the project cycle. Other tools include the applicant guide, the project work plan, the project costing template, and the project reporting and update template, all available via the SF's GCpedia page.

Each project team has a project lead and an executive sponsor. Both have clearly specified roles in the management of the project. For example, the project lead is responsible for the planning and completion of activities within the scope of the project plan, while the executive sponsor has spending authority over the project and responsibility for supporting the lead.

All SF-funded projects are subject to the Solutions Fund Governance Committee (SFGC) oversight process, in order to ensure due diligence, project accountability, and responsible management of funds. The SFGC monitors the progress of funded projects to ensure they are meeting work plan objectives and applying sound project management practices. Any amendment to a project's budget must go through the SFGC. To date, four projects have applied for and received approval for a budgetary increase.

Led by the Manager of the SF, the SFGC includes members of the IMSD and Enterprise Architecture teams from the Digital Transformation Branch, the Branch Senior Financial Officer community, CFOB, and the two ADM Innovation Champions.

Some interviewees involved in SF governance and who provide corporate support services identified an emerging challenge to the capacity of the SF team to support an increasing number of SF proposals and projects. They expressed a concern that the SF team may not be able to offer the same high level of one-on-one support to projects that has been identified as a major contributor to the success of SF projects to date. For example, two SF participants mentioned that SF did regular check-ins, visited their Toronto office, and even participated in project workshop activities.

Some project participants, staff, and support service interviewees noted the slow pace of procurement that can affect project timelines; however, this is outside of the control of the SF, which facilitates connections between projects and departmental procurement experts for support. For example, due to procurement delays, two SF projects (Citizen Science and Nitro) requested an amendment to extend the project by four and five months, respectively. Another SF project (RSA Voice) was unable to obtain software licenses due to the length of the procurement process.

Key Findings: Solutions Fund project selection

The Solutions Fund has a well-defined structure and standardized process of proposal review and project selection. While the review and selection process is well-documented and communicated, there are opportunities to better communicate the criteria used to decide which projects should be funded.

The Solutions Fund has an established process of proposal review that leverages expertise within Health Canada and from other federal departments, incorporating a diversity of viewpoints and expertise to inform Solutions Fund Review Panel deliberations.

The proposal review and project selection process has been described in external communications produced by the Solutions Fund (SF), such as the Applicant's Guide, which is available on the SF GCpedia page^{Footnote 11}. The proposal review and project selection process is typically completed within a three-month period.

Submitted proposals undergo an initial review by subject matter experts and technical experts belonging to a variety of disciplines including the topic area of the proposal and relevant corporate services such as finance, information management and information technology (IM/IT), procurement and human resources. At times, subject matter and technical experts from other government departments, identified through the Experimentation Works initiative at TBS, have been invited to review project proposals. More recently, successful SF project applicants were invited to participate as expert reviewers for assessing subsequent proposals.

Each expert applies a standardized Strengths, Weaknesses, Opportunities, and Threats (SWOT) rubric to evaluate proposals and submits their written response with comments and suggestions to the SF Review Panel, thus ending their involvement. Most of the interviewed internal and external expert reviewers expressed a desire to know the results of the Review Panel's decision. It was suggested that feedback on the utility of their reviews would also help improve future engagement of experts with the SF proposal review process.

A few internal expert reviewers indicated that the multi-page SF proposals require substantial time for proper review, which places pressure on the limited capacity of those called upon to contribute to the SF review process. Suggestions for improvement varied from proposal format re-design, such as creating dedicated sections for topics like IM/IT, and human resources requirements, to staggering the timing of proposals assigned for review, thereby reducing the number of proposals being reviewed at any given time.

The SF Review Panel (SFRP) is a small but diverse group co-chaired by the two ADM Innovation Champions. It is composed of managers and employees with backgrounds in seed funding and experimentation design, representing different HC branches and the TBS Experimentation group. SFRP membership has changed between solicitations, involving employees both inside and outside of HC, with the purpose of pulling together diverse expertise and knowledge to analyze proposals. Terms of Reference for the SFRP have been developed outlining these roles and responsibilities.

The SF team assembles an e-binder of proposals and expert reviews for each Review Panel meeting. SF applicants are invited to present their proposals to the SFRP and respond to their questions. The SFRP members review each project proposal and consider expert assessments and applicant presentations before deliberating and voting on final project recommendations to send to the DM. A couple of interviewees involved in the selection process noted that reliance on deliberations runs the risk of a 'rallying effect', whereby panel members may be swayed by opinions of others, particularly ADM Innovation Champions. On the other hand, some participants in SFRP meetings have expressed that the current deliberation process is clear and appropriate. One interviewee suggested that Review Panel members themselves could submit written reviews which could then be deliberated in a live meeting, alongside other review materials.

Project funding is awarded by the DM based on recommendations submitted by the SFRP. Successful applicants are informed by the DM's office. The SF also contacts unsuccessful applicants, informing them of the Review Panel's decision and the reasons for it.

There is support for applicants whose projects are not selected for funding. Some are encouraged to further develop their innovation idea and reapply, while others have been connected to similar projects or to relevant corporate service groups who could assist them. Some proposals that have been judged not to be innovative, but rather target improvements in core business practices, particularly large-scale projects, are referred to the appropriate branch ADMs for their consideration.

Many interviewees, primarily staff and participants, confirmed that no proposals have been rejected due to budget limitations. Instead, proposals were most commonly turned down because of the following:

• issues with the proposed scope of the project, including the need to better articulate the solution hypothesis;
• a need for more data and analysis; and
• a lack of consultation with possible stakeholders and other centres already exploring similar innovative solutions.

Participants are expected to respond to several clear questions within their Stream 1 (Exploration) and Stream 2 (Experimentation) proposal applications, including the problem or business need addressed by the proposal, the proposed solutions, their rationale, the intended beneficiaries, their alignment with departmental priorities, and a detailed budget.

Subject matter and technical experts consider a range of criteria in their review of proposals, such as cost, feasibility, similarity with other projects at HC, etc. These criteria are not explicitly stated within the SWOT framework. Instead, each expert applies their unique perspective to elaborate the strengths, weaknesses, opportunities, and threats of the proposal under review.

At the Review Panel stage, the SF does not explicitly identify the proposal assessment criteria to be used by SFRP members. Instead, the success of the process rests on the individual judgement and group deliberation between members. At the same time, there was a difference of opinion among interviewees representing governance and SF staff members with respect to the effectiveness of this process. Some felt satisfied with the ability of the SF Review Panel to come to a decision based on the quality of written answers to standard questions that applicants provide in their proposals, expert SWOT reviews, and applicants' oral presentations. Others expressed that the SF should create decision criteria to define what counts as 'innovative' for the SF and to better assess the quality of proposals.

Although decision-making criteria are inherently integrated into the application materials and review process, some interviewees representing participant, staff, and governance, as well as support services staff observed that the existing criteria may not be clear enough to all participants and SF review panelists. This points to an opportunity to better communicate the existing criteria to the different SF stakeholders. This could include additional explanations of what is meant by innovation and experimentation, and how projects are assessed.

Key Findings: Supporting the scaling up of Solutions Fund projects

The Solutions Fund incorporates a vision of preparing successful innovation ideas for implementation, or 'scale-up', but projects with potential must still compete with other projects in the larger Departmental Investment Planning processes. In addition, there is a question of if, and how, Solutions Fund projects could be supported after they have completed Stream 2 and the role that the Solutions Fund team might or should play in that process.

Stream 1 projects graduating to the Stream 2 phase do so when there is a well-developed proof-of-concept and experimental design.

Solutions Fund (SF) applicants can gain project management skills through the coached development of their project plans in the Stream 1 phase. The Stream 1 stage of further defining the problem to be addressed and developing the experimental methodology to test a solution also increases the possibility that employee-driven innovation ideas will be approved for Stream 2 funding. Furthermore, Stream 1 preparations raise the chance that the proposed experiment will be conducted as planned to generate useful lessons learned.

Stream 1 project participants wishing to proceed to Stream 2 must submit a Stream 2 proposal, as outlined in the previous section, which provides evidence that is considered in a joint meeting of the SFRP and SFGC. However, some interviewees speaking from the perspective of participants and reviewers pointed out that there may be more room for clarity for project applicants and participants regarding the criteria and process for moving from Stream 1 to Stream 2. In addition, a couple of internal interviewees observed that certain projects have needed considerable support from the SF team and experts to prepare for Stream 2. That said, since the start of the SF, four projects have transitioned from Stream 1 to Stream 2, as they were able to demonstrate the necessary capacity for testing an innovative idea. An additional five projects started directly in Stream 2. See Appendix 5 for more details.

To be approved, SF proposals must show evidence of promise and be supported by research. Innovative ideas are not selected if they do not show promise for improvements to HC programs or service delivery. Once approved, the SF provides project participants with a 'sandbox' environment to test new ideas in a cost-effective way. When projects have completed their Stream 2 experimentation phase, the project participants and relevant managers face a decision on whether to implement the innovation, i.e., investing in its implementation on an ongoing basis, where it would serve to improve current ways of conducting business. These decisions about implementation are supported by the SF project process that requires participants to document each project's concepts and the resulting evidence from testing the chosen innovation.

A few participant and governance interviewees confirmed that the required SF project documentation was designed to align with HC's project management processes in the sense that successful projects will have already fulfilled many of the documentation requirements for the initiation stage of the investment planning process at the branch or departmental levels.

In addition, the connections fostered between SF project participants and the wider SF network of staff, corporate service personnel, managers, and CFOB investment planning officials over the life of successful SF projects have been used to engage those responsible for supporting post-SF implementation decisions and processes. For example, a few governance interviewees noted that the ADM Innovation Champions have liaised with branch ADMs to raise awareness of relevant SF projects and that members of the Digital Transformation Branch Enablement Team working with SF projects have similarly liaised with their colleagues.

While some SF projects have shown potential for service or program improvement, it has been difficult to implement these projects at the branch level. For example, the Fourier-transform infrared spectroscopy (FTIR) project has the potential to produce approximately $800 in savings per test, in addition to saving time, by reducing the need for multiple tests per product and it is being piloted in an operational setting. However, it was suggested that this technology has not yet been more widely implemented due to competing priorities for operational funding, as well as the requirement for potential users to be trained on how to operate this new analytical equipment in a lab setting.

Unlike the SF, ADMs are responsible for investment decisions at the branch level for under $1M, and senior managers are engaged in the departmental approval process for projects of $1M and above, such as through the Director General Committee for Investments and Projects (DGIP) and the Executive Subcommittee on Finance, Investments, Projects and Transformation (EC-FIPT). Evidence of SF project success does not mean that the project aligns well with management investment priorities. Therefore, innovative ideas that have been 'proven' through the SF must still be assessed alongside other pressing investment needs, especially in the context of limited budgets.

Projects supported by the SF have the advantage of having special connections to corporate support services that facilitate rapid response to project needs in areas such as information management and information technologies, human resources, procurement, and financial management. Projects considering scale-up beyond the SF must seek these essential services through regular corporate channels, such as the Departmental Project Management Framework process, where some interviewees said that timelines can be considerably longer. Despite these challenges, positive examples of SF projects taking steps towards implementation were noted, such as ROEB's plans to pilot three projects in a 'real world' operational setting.

Currently, there is no clear mandate for the SF team to support the scale-up of projects that have shown promise after successfully completing the Stream 2 phase. However, the SF team and experts do make connections with those involved in scale-up decisions.

The transfer of the SF to CFOB has strengthened the connections that the SF had previously developed with staff and senior management involved in investment planning decisions for the possible scale-up of proven SF projects. Despite increased connections, a senior management interviewee suggested that more work needs to be done in this area.

A couple of interviewees representing the SF governance structure said that there are now more opportunities to communicate the results of SF projects to senior managers, such as presentations to the DGIP and EC-FIPT committees. In addition, some interviewees have suggested that SF staff could continue to assist projects that have completed the Stream 2 phase, especially during Stream 2 closeout, when project support is conferred onto others. However, the SF team has limited capacity and is focused on coaching current applicants and participants, promoting the SF, facilitating the SF selection process, and supporting currently funded projects.

The SF was transferred from CPAB to CFOB in January 2022 to promote a strategic relationship with the group that considers larger investments for HC, governed by the Departmental Project Management Framework and the Investment Planning and Results Framework. In line with this change, a few governance interviewees noted that a closer alignment could be fostered between the SF and the departmental investment planning process to encourage large projects over $1M to undergo a pilot testing phase prior to committing the Department to a large project expenditure of public funds.

Key Findings: Opportunities to improve processes associated with the Solutions Fund

Generally, the Solutions Fund is working very well. There were some suggestions to improve some aspects of the proposal review process, improve information sharing, and clarify ADM roles and responsibilities.

The evaluation identified a few possible improvements to the Solutions Fund.

According to a few internal interviewees, the Solutions Fund (SF) proposal template could be simplified, as the application process takes a long time due to numerous required details. The length of proposals could be limited, the proposal format could be reorganized into thematic sections, and the financial costing tool could be simplified.

• Suggested improvements to the proposal format could allow applicants to focus on articulating the core problem and innovative solutions.
• Such improvements could also reduce the time required for experts and SFRP members to review the proposal, given that the SF is not the primary role of senior managers and experts who are not part of the SF team itself – this is a 'side of the desk' activity. As an example, it takes approximately three hours to review four proposals and there may be up to 12 proposals to evaluate at any one time.

However, any streamlining or simplification of the proposal format would need to strike a balance between saving time for applicants and reviewers, and providing enough information on critical decision factors for proposal evaluation.

As previously mentioned:

• A few interviewees suggested that the criteria for project selection could be clarified, including criteria for determining which Stream 1 projects are ready to transition to Stream 2.
• Some expert reviewers would appreciate knowing the result of the process that they were called upon to assist, and to receive feedback if their review was useful or not.
• Many interviewees felt that the sharing of project results and lessons learned would benefit employees across HC.

There was feedback from a couple of SF team and senior management interviewees that there is confusion over the roles and responsibilities of the various ADMs responsible for the SF, specifically the two ADM Innovation Champions who co-chair the SF Review Panel, and the CFO who is now responsible operationally for the SF team since the transfer of the SF from CPAB to CFOB in January 2022. Clarity of these responsibilities would make it easier for the SF team when seeking approvals or direction as it relates to the SF.

Key Findings: The way forward for the Solutions Fund

Although the Solutions Fund is seen to be inspiring more innovation and experimentation across Health Canada, it is unclear what role it should or could play beyond offering the current Solutions Fund opportunity for employees.

As discussed previously, a few interviewees indicated that the Departmental Investment Planning process could be inspired by the Solutions Fund (SF) where large projects are piloted to build evidence of potential for success before large investments of time and money are made.

Many interviewees across the various groups believe that each branch has or should have a role to play in expanding innovation and experimentation across the Department and building on the work done through the SF. At the same time, they also identified a need for a centralized group like the SF team to help ensure this culture is fostered across the Department. Such a group would be essential for a variety of reasons, including the following:

• Currently, some branches have more innovation activities than others, and a few interviewees indicated that all branches may not be able to replicate the SF model due to limited resources.
• Even if each branch had its own innovation centre, like ROEB's Transformation Office, there may need to be an overarching centre of innovation and experimentation expertise. Such a team could provide guidance to branches with less expertise and experience in innovation and experimentation, and help standardize activities across the Department.
• A centralized group could also help branches make connections to expertise across and outside the Department and help disseminate project lessons and findings across branches.

A recent SF environmental scan highlighted top-down and bottom-up drivers of innovation within an organization. One of the senior manager interviewees reflected that there could be room for the SF to continue to support employee-led innovation projects while also encouraging leaders to increase innovation and experimentation within their branches.

Conclusions

The Solutions Fund (SF) is regarded as a successful initiative that plays a strong role in inspiring a culture of innovation and experimentation across HC. Managed by a dedicated team, the SF has well-established processes and oversight mechanisms that enable it to operate efficiently, in partnership with corporate support services and experts within and outside HC. By leveraging partnerships and expertise, the SF team engages with applicants and participants to provide comprehensive support, from proposal development and selection to project implementation. Since its launch in 2018, the SF has not faced any budgetary restrictions which may affect project selection or funding. By enabling employee-led development of innovative ideas and experimentation, the SF can identify what works and what does not within a supportive environment that controls for risks and aims to generate lessons that can provide evidence for future decision making on larger investments.

Although the evidence shows that the SF is working well as a corporate funding opportunity for employee-led innovations, the evidence gathered during this evaluation points to a few areas for consideration as the SF evolves into the future, chiefly concerning how Health Canada can maximize the use of knowledge gained from the funded projects.

Recommendations

As a result of the analyses presented in this report, the following two recommendations are proposed.

Recommendation 1) Improve the sharing of project results and lessons learned from SF projects across Health Canada.

While the sharing of project results and lessons learned beyond close-out presentations has proven to be challenging, there is potential for the Department to maximize the benefits from SF's small-scale experimentation projects. For example, sharing results and lessons learned would permit others to build on recent experiences and avoid duplication of effort or covering off areas that have been previously examined or addressed. The SF team should continue to work with its corporate partners and stakeholders to enhance employee and management awareness and enable access to SF project experiences and lessons learned.

Recommendation 2) Determine the future role of the SF in supporting the scale-up of successful Stream 2 projects, and growing the capacity for innovation and experimentation across Health Canada.

Nine SF projects have successfully completed Stream 2 activities. However, while some have been piloted, none have yet been 'scaled up' at a branch or departmental level. The SF should consider exploring how it could build on existing collaborative relationships with relevant corporate service groups to determine what further role the SF itself could play in supporting future 'scale-up' of successful Stream 2 projects.

Although most branches have been represented in employee-led SF projects, the level of engagement in innovation and experimentation varies across HC. Given that innovation and experimentation are recognized Department priorities, the SF should determine if and how it could play a role in growing the level of knowledge and action on innovation and experimentation across Health Canada outside of the Solutions Fund itself.