Evaluation of the Generic Terms and Conditions Pilot Program

On this page

Introduction

This document presents the results of an evaluation of the Generic Terms and Conditions Pilot Program (the Generics), which is managed within the Office of the Comptroller General at the Treasury Board of Canada Secretariat (TBS). The evaluation was undertaken by the Internal Audit and Evaluation Bureau at TBS with the assistance of Goss Gilroy Inc. It was conducted in accordance with the Treasury Board Policy on Results.

The evaluation assessed the impact of the Generics measured by the extent to which the intermediate and long-term outcomes have been achieved. The evaluation:

  • was undertaken between April 2021 and January 2022
  • covered the period from the implementation of the Generics in April 2017 to September 2021

Preamble

Transfer payments, including grants and contributions, represent a large proportion of government spending. They help to further government policy objectives by enabling and engaging a wide range of external skills and resources. Within this context, TBS implemented the Generics in 2017 as part of the government’s effort to foster a culture of experimentation and innovation. The Generics provided departmentsFootnote 1 with the opportunity to implement one of six pre-approved, non-standard funding approaches within existing grants and contributions programs.

In 2017, Innovation, Science and Economic Development Canada launched Innovative Solutions Canada, which is a program that offers departments funding approaches similar to those implemented in the Generics. During the same period, the Privy Council Office launched the Impact Canada Initiative, which also offers departments access to similar funding approaches as those offered by the Generics. Like TBS’s Generics, both the Impact Canada Initiative and Innovative Solutions Canada seek to expand the use of transfer payment programs in new ways though with some key differences. A brief description of each program and their differences is provided in Appendix A. The findings indicate that there would have been benefit to departments coordinating these innovative transfer payment approaches.

The evaluation found that the Generics achieved the expected outcomes to some extent and was well-received. The Generics provided departments with additional flexibility and enabled some innovation. Twenty-three of 32 departments were found to have used at least one of the new funding approaches: the Generics, the Impact Canada Initiative or Innovative Solutions Canada (refer to Table 1). The Generics was not necessarily more efficient relative to traditional grants and contributions approaches. The funding approach used and the enabling conditions present in a department were major contributors to the level of success and efficiency achieved.

Some departments found that the complexities of implementing the Generics were burdensome. In some cases, the skills and knowledge required to use the Generics were considerable, as departments and recipients had to learn how to incorporate outcomes-based thinking and practices. In other cases, the additional administrative burden required to integrate the Generics into existing practices, processes and systems did not present a good value proposition. Greater support from the Office of the Comptroller General to departments could have helped to address some of the challenges faced by the community.

The Generics was implemented as a pilot to:

  • facilitate improvements before becoming permanent
  • better manage potential risks

Specifically, changes were to be made at the mid-point of the pilot based on feedback from departments. The Office of the Comptroller General opted, however, to defer any changes pending the completion of the evaluation, given that at that time the Office of the Comptroller General felt that only one change needed to be made. When the Generics was launched, the Office of the Comptroller General was assumed to have the capacity to:

  • help departments access the Generics
  • track and monitor the performance of the Generics

To some extent these assumptions proved to be correct, but not to a degree that fulfilled community needs or as planned.

Program background

Program description

The Generics is a five-year pilot program that was established to make innovative use of grants and contributions for departments covered by the Policy on Transfer Payments. The Generics is guided by the:

The Generics provides the authorities, provisions and exceptions approved by Treasury Board Ministers to enable specific innovative approaches in the design and administration of transfer payment programs. Departments can append new program authorities to an existing set of terms and conditions for a grant or contribution program in order to facilitate the use of one of six new funding approaches with an emphasis on measuring results rather than on eligible expenditures. The pre-approved appendices to the Generics did not require departments to return to the Treasury Board for approval. After consulting with TBS, ministerial approval was sufficient for implementation. Once approved, the Minister was required to inform the President of the Treasury Board through a letter of confirmation. Departments can also incorporate the Generics into new programs through the usual Cabinet approval process. The following three broad funding categories with six funding approaches were made available through the Generics (refer to Appendix B for definitions).

  1. Incentive-based funding
    1. Outcome achievement payments
    2. Base payment and premium payment
    3. Fixed contributions
  2. Prizes/challenges
    1. Prizes/challenges
    2. Targeted prizes/challenges
  3. Micro-funding
    1. Micro-funding

Roles and responsibilities

In this section

The following are the major stakeholders who have an interest in the Generics.

Potential recipients

Potential recipients may be individuals or entities, including Crown corporations that have successfully applied for funding under a transfer payment program. Funding may take one of the following forms:

  • monetary payments
    or
  • transfers of goods, services or assets from the Government of Canada on the basis of appropriations that do not result in the acquisition by the government of any goods, services or assets

Deputy heads

Deputy heads are responsible for the design, delivery and management of transfer payment programs, including ensuring that departmental transfer payment programs are, and remain, relevant and effective in meeting departmental and governmental objectives.

Deputy heads ensure that due diligence is adhered to and documented when deciding to use the Generics. Deputy heads are responsible for:

  • reviewing and potentially updating the Departmental Results Framework
  • monitoring and minimizing any adverse effects of implementing the Generics through existing or amended strategies to manage risk in transfer payment programs

Departmental managers and practitioners

Departmental managers and practitioners are responsible for accountability, transparency and effective control in the management of transfer payments, which includes providing flexibility to adapt the administrative requirements to the risks involved for applicants and recipients. Generally, however, transfer payment programs are implemented by program officers.

Departmental managers are responsible for documenting the decision to implement the Generics, including:

  • the choice of the appropriate funding approach
  • the rationale for the use of the Generics
  • the expected results
  • monitoring
  • the sharing of lessons learned
  • risk management
  • communication strategies

Departmental centre of expertise on transfer payments

Not all departments have centres of expertise. For those that do, centres of expertise:

  • support program managers in the implementation of the Generics by promoting its use and making training available
  • enable skills development for effective program design, experimentation and results measurement by supporting program managers in the application of experimental methods

TBS program sectors

TBS program sectors are the single window for departments. To use the Generics, departments consult their program sector analyst, who validate the proposed approach with the Financial Management Sector Transfer Payment Policy Centre.

TBS Financial Management Sector Transfer Payment Policy Centre

The Transfer Payment Policy Centre reports directly to the Assistant Comptroller General in the Financial Management Sector at TBS. The centre is responsible for policy development and for interpreting:

The Directive on Transfer Payments outlines elements of program design and other operational requirements that ensure “accountability, transparency and effective control in the management of transfer payments.”

Expected outcomes and assumptions

In this section

Long-term outcomes

  • Federal transfer payment programs reach a broader range of recipients and collaborators
  • Innovations in transfer payment programs benefit Canadians

Intermediate outcomes

  • Departments use the Generics more frequently
  • Departments have the flexibility to design and deliver transfer payments
  • The Generics enables more efficient implementation of programs in departments

Immediate outcomes

  • Senior management leads and provides support to practitioners in the implementation of the Generics
  • Departments have the capacity to use funding approaches in the Generics
  • Departments are aware of the Generics

Assumptions

  • Potential recipients are engaged and supported
  • TBS and departments implement the administrative requirements for the Generics
  • Departments are willing to take risks and experiment
  • TBS has the capacity to support the implementation of the Generics

Evaluation methodology and scope

The evaluation assessed the extent to which the intermediate and long-term outcomes have been achieved. The evaluation tested the immediate outcomes and the assumptions outlined in the logic model as conditions for the achievement of the intermediate and long-term outcomes. For this purpose, a logic model was developed in collaboration with the Office of the Comptroller General (refer to Appendix C).

The evaluation examined the period from implementation of the Generics in April 2017 to September 2021 using the following lines of evidence:

  • document review
  • administrative data review
  • interviews (n=23) with 16 departments and 11 TBS representatives
  • case studies (n=6)
  • a survey of departments (n=32)

In scope for the evaluation were departments with a transfer payment budget in the Main Estimates of Fiscal Year 2019–20. Evaluators worked with the Office of the Comptroller General to identify departments that were using the Generics.

The evaluation documented the impact of the Generics on program outcomes using departmental reports, publications on programs websites and, in one case, a departmental evaluation.

The evaluation did not do a comparative analysis of the different programs offering financial instruments similar to the Generics.

Limitation of the evaluation

The Generics tracker system on the Grants and Contributions Portal was inconsistent and had not been updated since 2018. This data gap was mitigated by using multiple lines of evidence for the evaluation.

The Office of the Comptroller General collected survey data in 2017–18 and 2018–19. The 2018-19 survey data did not allow for trend analysis. Nonetheless, both surveys were used as evidence for the evaluation and to inform the design of the evaluation survey.

Expected outcome: departments use the Generics more frequently

Conclusion

The number of departments that use the Generics has remained static with most accessing the Generics early on. Departments faced barriers and challenges that prevented the use of the Generics, which prompted some to use the initiatives from the Privy Council Office and Innovation, Science and Economic Development Canada.

Evidence shows that of the 32 departments that used innovative transfer payment approaches, 12 used the Generics during the pilot. This is an increase of only one department since 2018–19. The number of projects that used financial instruments supported by the Generics grew slightly over time. By 2020–21, there were 26 programs that had used or are using the Generics compared to 21 programs in 2019–20. The data review shows that most of the departments using the Generics accessed it earlier in the pilot.

All lines of evidence show that departments faced barriers that prevented them from considering the Generics. There were also challenges that made departments reconsider or reassess use of the Generics. The barriers and challenges most often cited are:

  • difficult to incorporate the Generics due to the duration of the pilot program
  • lack of capacity (resources) to implement the Generics even with training and guidance
  • the process to obtain authority to use the Generics is too burdensome
  • limited understanding of the funding approaches and how they could be used
  • the Privy Council Office’s Impact Canada Initiative is a better fit for programs in some departments

Although the number of departments that have used the Generics during the pilot period could suggest low interest, the evaluation found that 23 of 32 departments (72%) with grants and contributions are using one of the three innovative transfer payment initiatives from Innovation, Science and Economic Development Canada, the Privy Council Office and TBS (refer to Table 1). This number suggests there is interest in the types of funding approaches proposed by the Generics. The funding approaches most used through the Generics, Impact Canada Initiative and Innovative Solutions Canada are the prizes/challenges, outcome achievement payments and fixed contributions (refer to Figure 1).Footnote 2

Table 1: transfer payment initiatives and the number of departments that use them
Transfer payment initiative (the department responsible for the initiative) Number of departments
Innovative Solutions Canada (Innovation, Science and Economic Development Canada)

3

Impact Canada Initiative (Privy Council Office)

5

The Generics (Treasury Board of Canada Secretariat)

8

The Generics and Impact Canada Initiative

1

The Generics and Innovative Solutions Canada

3

Impact Canada Initiative and Innovative Solutions Canada

3

None of the above

9

Total

32

Figure 1: innovative funding approaches used by departments (percentage)
Figure 1: innovative funding approaches used by departments (percentage). Text version below:
Figure 1 - Text version
Approach used Percentage of departments
Prizes/challenges 32%
Outcome achievement payments 22%
Micro-funding 14%
Fixed contributions 14%
Base payment and premium payment 11%
Targeted prizes/challenges 8%
Total 100%

Source: 2021 survey on the Generics. Innovative Solutions Canada and Impact Canada Initiative websites were accessed in February 2022.

The interest in these approaches is confirmed by most key informants who believe that the Generics provides flexibility and options for departments to use when conditions for successful implementation exist. Similarly, survey respondents commented that the Generics should be formalized and normalized because it offers flexibility compared to traditional terms and conditions (Ts&Cs).

The following factors are important enablers for departments to use the Generics:

  • the readiness of the departmental program (discussed further in Conditions for implementing the Generics)
  • innovation and the opportunity to experiment
  • whether the department was in the process of modernizing their grants and contributions programs
  • the opportunity to reach new recipients or beneficiaries
  • opportunities to focus on outcomes rather than outputs: as a key informant stated, “using [the] Generics helped highlight the fact that we [departments] have a lot of work to do to get an outcomes mindset across to our recipients”
  • timing: the launch of the Generics coincided with the development of Ts&Cs for a new program

Expected outcome: departments have flexibility to design and deliver transfer payments

Conclusion

The Generics provides flexibility that supports innovative project delivery, but there is still room for improvement.

The Generics provides policy authority to departments to try new funding approaches that are pre-approved by Treasury Board. Departments can design or adapt their programs to use funding approaches offered under the Generics by granting exemptions to existing policy requirements. The Generics supports innovation more so than flexibility, and the latter depended on the type of funding approach used.

Most key informants agree that having access to more tools provides new possibilities for departments in managing transfer payments. The Generics offers flexibility to support innovative projects or program delivery compared to traditional Ts&Cs. Key informants noted the following examples of flexibility:

  • access to a range of funding approaches that were pre-approved by TBS and relatively straightforward
  • more options to explore different ways of delivering their program and determine what works and what doesn’t
  • the ability to reach new types of recipients
  • the ability to fund projects outside of the standard transfer payment agreements
  • financial support based on the achievement of specific outcomes rather than activities or outputs

Survey data supports these views:

  • 50% of departments noted that, to a moderate extent, the Generics provided more flexibility to their programs than traditional Ts&Cs
  • Only 30% of departments described the Generics as providing more flexibility to a large extent
  • On the other hand, 60% of departments noted that, to a large extent, the Generics supported innovative project or program delivery compared to traditional Ts&Cs

Full results are provided in Table 2.

Table 2: the extent to which the Generics contributed to innovative delivery and flexibility, by percentage of departments surveyed
Criterion Not at all Small extent Moderate extent Large extent Very large extent
Innovative delivery

10%

10%

10%

60%

10%

Flexibility

10%

0%

50%

30%

10%

Source: 2021 survey on the Generics.

Evidence from case studies shows that flexibility varied greatly, depending on the approach used. Examples of flexibility as they relate to the following funding approaches:

  • Prizes/challenges:
    • allowing for a staged approach in a contest
  • Incentive-based funding:
    • not being subject to fixed timelines
    • allowing for adjustments and changes by focusing on results
    • allowing departments and recipients to adjust to and handle changes due to the COVID-19 pandemic by focusing on results
  • Base payment and premium payment:
    • facilitating negotiation with recipients because the funding approach was pre-approved by TBS
    • eliminating the need to request additional authorities
  • Micro-funding:
    • eliminating the requirement to write reports at the end of the agreement no matter the amount involved

Key informants and the survey respondents provided examples of inflexibility that, in some cases, prevented departments from using the Generics and, in others, led them to use Ts&Cs offered by the Impact Canada Initiative or Innovative Solutions Canada instead:

  • Prizes/challenges:
    • no flexibility to use an intermediary to design and manage challenges
    • departments do not have capacity to build a platform and run a challenge, so they used the Impact Canada Initiative or Innovative Solutions Canada
  • Base payment and premium payment:
    • payment capped at 15% of total eligible expenditures
  • Fixed contributions:
    • lack of awareness of the similarities of fixed contributions in Appendix K of the Directive on Transfer PaymentsFootnote 3
    • unclear if the issue is awareness of the existence of the Generics
  • Micro-funding
    • the maximum amount eligible is $1000, which is low, so a range of options from $5,000 to $25,000 have been suggested
    • the cost-benefit ratio when delivering micro-funding with the current

$1,000 limit is poor because the cost of implementation is very high, and the perceived benefits are too low

Expected outcome: the Generics enables more efficient implementation of programs in departments

In this section

Conclusion

The Generics did not, for the most part:

  • enable more efficient implementation of programs in terms of streamlined processes
  • reduce the need for authorities
  • reduce workload

The Generics was created, in part, to:

  • remove the need for programs to request additional authorities from the Treasury Board
  • use innovative funding approaches

The intent was to:

  • streamline the implementation of programs
  • deliver programs more efficiently than when using traditional Ts&Cs

Reducing the need for additional authorities

Key informants and survey respondents had mixed views of the impact of the Generics in reducing the need for additional authorities. Survey data shows that only 20% of respondents described the Generics as reducing the need by a large or very large extent (refer to Table 3). Eighty per cent of departments experienced some reduction in the need for additional authorities. Nonetheless, some key informants indicated that the process for the Generics was as long as the process for a Treasury Board submission, which led them to abandon the Generics.

Table 3: the extent to which the Generics increased efficiency, by percentage of departments surveyed
Criterion related to efficiency Not at all Small extent Moderate extent Large extent Very large extent
Reduced need for additional authorities

20%

40%

20%

10%

10%

Streamlined departmental processes

30%

20%

20%

20%

10%

Source: 2021 survey on the Generics.

Streamlining processes

Evidence on streamlining processes is mixed. Among departments that incorporated the Generics through Cabinet approval, the process of adopting the Generics was not viewed as burdensome. The departments that appended the Generics language to existing Ts&Cs saw the process as accessible, but it became difficult when departments fit the new funding approaches into existing programs, processes and systems.

Key informants explained that using the Generics required more work upfront and that implementation was difficult. For some, the workload was only reduced once the project was in place. In other cases, the administrative burden of implementing the Generics remained heavy, which is reflected in the survey data. Seventy per cent of respondents indicated that adopting the Generics had no impact at all on streamlining departmental processes. For those who said there was an impact, 40% said the Generics streamlined processes to a small or moderate extent. Only 30% indicated that the Generics streamlined processes to a large or very large extent.

The 2018–19 survey about the Generics also indicated that the following areas reduced efficiency:

  • more work and time spent upfront; in some instances, programs needed to be restructured in departments
  • more oversight and monitoring by the departmental program that is using the Generics
  • processes were more labour intensive and became less efficient
  • a wide range of expertise in different areas, such as results and research, were needed, which added costs for hiring people or contracting out

Departments have existing processes for:

  • engaging stakeholders
  • making calls for proposals
  • paying recipients through case and financial management systems

Within this context, departments using the Generics had to incorporate new designs, approaches and internal processes. Doing so required:

  • using new agreement templates
  • ensuring that the templates fit within existing legal frameworks
  • integrating them into existing financial systems

For instance, changing agreements to pay recipients based on results instead of expenses required negotiations on performance measurement and results reporting. Departments had to ensure that their financial management systems could support payments based on outcomes rather than eligible expenditures. In some cases, changes had to be made to information technology (IT) applications in order to allow non-standard deadlines, such as the end of the school year. An example of the complexity of amending agreements is found in this statement:

“We had to have agreements to get the funding released to let recipients contract a third party. We had to amend the agreements once the third party did their work to set targets, then we had to keep unencumbered funds for this to occur.”
– departmental representative

Some time-consuming administrative challenges, such as changing IT systems for case and financial management, only needed to be addressed once. However, negotiating outcomes and when outcomes would trigger reimbursement remains an ongoing, difficult and time-consuming task. Key informants explained that this is because both departments and recipients have limited capacity and experience in negotiating agreements under these new approaches. Some key informants also indicated that they were unlikely to try again in the short term because of the upfront work and the fact that the Generics is a pilot program.

Expected outcome: federal transfer payment programs reach a broader range of recipients and collaborators

In this section

Conclusion

The Generics enabled federal transfer payment programs to reach eligible recipients and collaborators that traditional Ts&Cs would not have, to some extent.

The Generics aimed, in part, to encourage new collaborations and reach a broader range of eligible recipients. Increasing reach for both recipients and collaboratorsFootnote 4 varied depending on the objective of the program (for example supporting Indigenous peoples or, in the case of mandatory requirements, targeting individuals aged 16 or older for micro-funding).

Recipients

Survey data shows that among departments that use the Generics, access to a broader range of recipients and collaborators improved to a small to moderate extent when compared to using traditional Ts&Cs. Only a small number of respondents (10%) noted that the Generics improved access to a broader range of recipients and collaborators to a very large extent (refer to Table 4).

Table 4: the extent to which the Generics improved access to recipients and collaborators, by percentage of departments surveyed
Improvements to access Not at all Small extent Moderate extent Large extent Very large extent
Broader range of recipients

20%

30%

30%

10%

10%

Broader range of collaborators

20%

40%

30%

0%

10%

Source: 2021 survey on the Generics.

Key informants agree that the Generics helps programs reach a broader range of eligible recipients. The level of reach depends on the objective of the program, eligibility criteria for recipients, and whether the project is a grant or contribution. The evidence shows that it is easier to reach new recipients with grant funding than with contribution agreements.

Key informants indicated that contribution programs generally reached the same eligible recipients since they are not required to advertise broadly. In these cases, departments target recipients based on:

  • the capacity to use the Generics
  • the willingness to take risks and innovate by measuring outcomes instead of outputs
  • the capacity to meet the Generics tight timelines

Key informants of grant programs found that prizes/challenges and micro-funding allowed them to reach recipients that they would not have otherwise reached with traditional Ts&Cs (such as students and teachers) since traditional Ts&Cs require a public call for proposals.

Table 5 provides examples of recipients who were reached using the Generics and who would not have been reached using traditional Ts&Cs.

Table 5: funding approaches and recipients
Funding approach Reached a broader range of recipients? Who did it reach?
Prizes/challenges

Yes

  • Specific clientele: young entrepreneurs and women entrepreneurs
  • Broad innovation community and academics who may not have otherwise considered defence as an area that they could contribute to
Micro-funding

Yes

  • Individual Canadians
  • Teachers and students
Outcome achievement payments

No

  • Regular recipients willing to take risks
  • Regular recipients with an interest in innovative approaches
  • Recipients who submitted an unsolicited proposal
Base payment and premium payment

No

  • Interested new recipients who responded to a call for proposals
  • Regular recipient who was interested in a new approach promoted by TBS

Collaborators

Survey data shows that among departments that use the Generics, access to a broader range of collaborators improved to a small (40%) to moderate (30%) extent compared to traditional Ts&Cs. Only a small number of respondents (10%) noted that the Generics allowed access to a broader range of collaborators to a very large extent (refer to Table 4).

Evidence from case studies shows that private investments and partnerships occurred when departments used outcome achievement payments and prizes/challenges (refer to Table 6). In these cases, recipients worked with private sector organizations that helped them secure partnerships and shared their risk.

Table 6: funding approaches and collaborators
Funding approach Collaborators
Prizes/challenges

Defence scientists and experts in the field, including:

  • National Research Council Canada experts
  • engineering firms
  • military experts and Canadian troops
  • universities
  • business incubators or accelerators
Outcome achievement payments
  • Private investors willing to provide initial funds
  • Private partnerships

Source: Cases studies for the evaluation, 2021.

Expected outcome: innovation in transfer payments programs benefit Canadians

Conclusion

There is a broad view among departments that the Generics benefits Canadians based on the achievement of program outcomes. In some cases, it is too early to determine the impact of the Generics on program outcomes.

According to survey data, departments that use the Generics believe that the Generics benefits Canadians by supporting outcome achievement (refer to Figure 2). However, the extent to which the Generics is more effective than traditional Ts&Cs is not clear due to the split result shown in Figure 3 and described below.

Departments have a somewhat positive view of the contribution of the Generics to achieving program outcomes. Forty per cent of departments surveyed said that the Generics made a positive contribution to a large or very large extent while 30% described the extent of the positive contribution as moderate. The remaining 30% of departments expressed that the Generics did not contribute or contributed only to a small extent to program outcomes.

Figure 2: the extent to which the Generics made a positive contribution to program outcomes, by percentage of departments surveyed
Figure 2: the extent to which the Generics made a positive contribution to program outcomes, by percentage of departments surveyed. Text version below:
Figure 2 - Text version
Extent to which the Generics made a positive contribution to program outcomes Percentage of departments
Not at all 20%
Small extent 10%
Moderate extent 30%
Large extent 30%
Very large extent 10%
Total 100%
Figure 3: the extent to which the Generics helps solve problems that traditional Ts&Cs do not, by percentage of departments surveyed
Figure 3: the extent to which the Generics helps solve problems that traditional Ts&Cs do not, by percentage of departments surveyed. Text version below:
Figure 3 - Text version
Extent to which the Generics helps solve problems that traditional Ts&Cs do not Percentage of departments
Not at all 10%
Small extent 30%
Moderate extent 20%
Large extent 30%
Very large extent 10%
Total 100%

With regard to whether the Generics can solve problems that traditional Ts&Cs cannot, the opinion of departments surveyed was evenly split. Forty per cent of departments felt that the Generics had a positive impact to a large or very large extent while another 40% felt that the positive impact was small or non-existent. The remaining 20% felt the extent was moderate.

Key informants indicated that programs using the Generics are achieving their outcomes and thus delivering benefits to Canadians. However, they could not say whether the outcomes and benefits would have been achieved without the Generics. Most key informants noted that it was still too early to determine the impact of the Generics on their program results since some projects were not yet complete or had not yet reported on results. In some cases, it was difficult for departments to determine the impact of the Generics on the outcomes of the program. This difficulty was because:

  • the value paid to recipients was low relative to the total amount invested in the program
  • a low number of recipients have reported back to the department

Evidence from case studies shows that programs using the Generics have achieved or are achieving program outcomes. Key informants also provided examples of:

  • organizational outcomes that have been achieved
  • in a few cases, the benefits to Canadians (refer to Appendix D)

Unintended outcomes

The use of the Generics resulted in some unintended positive outcomes, for example:

  • innovative changes to program design and delivery elements
  • use of social media to reach recipients
  • integration of solutions through different domains working together
  • simplification of the application process for recipients
  • reduction in wait times for information requests
  • ability to reach Canadians directly
  • opportunity to demonstrate that a program is achieving outcomes

In some cases, the Generics helped recipients focus on outcomes rather than outputs. The Generics also provided learning opportunities for staff across departments.

Conditions for implementing the Generics

In this section

Conclusion

The conditions for implementing the Generics were not optimal. Although, departments were aware of the Generics, the uptake was low. Departments had challenges implementing the Generics and lacked expertise and support from the Office of the Comptroller General.

Awareness of the Generics

There is strong evidence that departments are aware of the Generics. Most survey respondents (62.5%) indicated that their department is aware or very aware of the Generics. A few key informants did, however, indicate that there was insufficient ongoing promotion and information-sharing about projects using the Generics. This was confirmed by survey evidence and the document review.

Departments have the capacity to use the Generics

All lines of evidence suggest departments have varying degrees of capacity to implement the Generics. How departments view their capacity also varies by funding approach. Almost half of departments rated their capacity as either medium or medium-to-low for all new approaches. Most key informants indicated having:

  • capacity challenges
  • a lack of resources
  • a lack of expertise in using the new approaches

Although key informants using the Generics mentioned having some capacity, implementing the Generics required a level of readiness and willingness to experiment, including:

  • dedicated funds and resources
  • support from the transfer payment community and other departments that have used similar approaches
  • willingness to try something new and experiment
  • know-how
  • willingness to learn
  • ability to think outside the box
  • willingness to take risks
  • well-defined outcomes and indicators

In one case study, key informants mentioned that being in a department that was already experiencing a culture shift toward experimentation and innovation eased the implementation of the Generics. Data from case studies shows that programs that used the Generics were supported by their policy branches or centres of expertise with advice, guidance and central agency liaison. Key informants indicated, however, that support was needed from the TBS Transfer Payment Policy Centre so that departments could:

  • make improvements based on lessons learned, and the successes and failures of other departments
  • share information and templates

Departments implement the requirements of the Generics

Departments largely fulfilled the administrative requirements set out in the guidance and Policy on Transfer Payments (refer to Figure 4). There is, however, a greater need for support through community outreach and the sharing of lessons learned. Survey data shows that the vast majority of implementing departments are aware of, and have largely fulfilled, the administrative requirements. Nonetheless, only 50% of survey respondents indicated having reviewed their performance measurement strategies when using the Generics. One of the reasons given by those who have not reviewed their strategies was that the Generics is a pilot.

Interviews and evidence from case studies show that programs using outcome achievement payments faced challenges in defining outcomes and measuring performance. Addressing these challenges required:

  • a good baseline for measuring results: in the absence of expertise and specific guidelines for this, departments had to use third parties
  • careful negotiation to identify and measure outcomes while maintaining collaborative partnerships with recipients
  • a linking of payments to results: while it was easier to identify the level of pay for high-level results, it was a lot more difficult for low-level results
Figure 4: fulfillment of administrative requirements, by percentage of departments
Figure 4: fulfillment of administrative requirements, by percentage of departments. Text version below:
Figure 4 - Text version
Administrative requirements Yes No
Ensured the right levels of operational and financial controls 100% 0%
Monitored the implementation of the Generics 100% 0%
Shared lessons learned 100% 0%
Documented decision-making when using the Generics 90% 10%
Engaged external stakeholders or communicated to recipients 80% 20%
Reviewed the impact of using the Generics on strategies to manage program risks 80% 20%
Reported annually on the use of the Generics 70% 30%
Developed communications about how to use the Generics 60% 40%
Reviewed departmental results frameworks or performance measurement strategies 50% 50%

Survey data shows that most departments engaged and supported recipients in the use of the Generics to a high or very high extent. This result was confirmed by key informants and by data from case studies. There is evidence of buy-in from recipients though key informants noted that this varies by approach, as does the willingness of recipients to take risks. Providing flexibility and negotiating with key stakeholders are necessary for success and are some of the lessons learned by programs.

TBS has the capacity to support implementation of the Generics

The conditions for TBS’s implementation of the Generics were not optimal. Departments needed greater support than what TBS was able to offer. The document review shows that no full-time equivalent resources were requested by the Office of the Comptroller General from the outset of the Generics. The Transfer Payment Policy Centre assigned only 0.25 full-time equivalents to support departmental implementation of the Generics. Although the Office of the Comptroller General provided 15 training sessions over a five-year period, several key informants indicated that their department did not receive enough support. They made it clear that dedicated support from TBS would have made a difference.

Survey data shows that most departments (62.5%) consulted the Office of the Comptroller General at some point. Among those departments, 45% reported having useful or very useful consultations. Key informants also agreed that consultations with the Office of the Comptroller General were helpful. Almost a third of departments reported having reached out to other organizations, such as the Privy Council Office, Natural Resources Canada and Canadian Heritage, to seek help with review and feedback. Though most departments were aware of Generic Terms and Conditions: A Guide for Practitioners (accessible only on the Government of Canada network) only 43% found the guide useful. Many key informants indicated that the guide was not easily available. The guide had not been updated since it was issued in 2017 because there was no new information or changes to the innovative funding approaches.

The document review identified an original commitment to adjust around the mid-point of the Generics based on feedback from departments. Although departments provided feedback on the Generics, adjustments were not made. Some key informants underlined how TBS could have:

  • tracked the use of the Generics by department and their progress
  • responded to community needs by adjusting or modifying the Generics based on feedback from implementing departments

Considering TBS’s limited ability to support departments and the limited ability of departments to implement the Generics, the Office of the Comptroller General focused on three areas:

  • gathering lessons from the community
  • building awareness
  • assessing the impact of increasing the value of micro-funding

However, the Office of the Comptroller General postponed adjusting the Generics until after the evaluation was completed.

The evaluation found that not all program sector analysts at TBS were aware of the existence of the Generics and, therefore, they could not have enabled or proposed the use of the Generics to departments. Key informants noted that reaching out to other functional areas within TBS to garner additional support for the uptake and implementation of the Generics could have helped improve awareness.

The Generics was launched to foster a culture of experimentation and innovation to help departments “make the transition from funding based on tasks and activities to funding based on the achievement of concrete goals.”Footnote 5 Although, Generic Terms and Conditions: A Guide for Practitioners (accessible only on the Government of Canada network) provides departments with information about experimentation and the Generics, the evidence indicates that there continues to be a lack of understanding of what experimentation entails:

“… Experimentation is about committing to rigorous evaluation and evidence – not just freewheeling ‘trying stuff out.’ It’s about putting in place a structure to learn from trying things out in the world. It is not about just doing things differently – and expecting to succeed.”Footnote 6

In principle, the Generics was set to work; however, only 0.25 full-time equivalents were assigned to the Generics and evidence collected during 2017–18 and 2018–19 was not used to improve the Generics. Rather, adjustments were postponed until the completion of this evaluation.

In summary, departments wanted to experiment and be innovative. However, challenges emerged due to a lack of capacity and expertise, as well as a lack of support from the Transfer Payment Policy Centre. Some departments were willing to use the Generics if it remains an option and if it goes beyond a pilot program. A few departments tried the Generics in more than one program, while others tried but encountered challenges that prevented them from continuing to use the Generics. Of the latter, some chose to use the Impact Canada Initiative or Innovative Solutions Canada, which offered more support and resources, such as a platform for launching challenges. In the end, it is clear that departments want to have these options in the Policy on Transfer Payments.

Recommendations

  1. It is recommended that the Office of the Comptroller General establish the Generics as a transfer payment instrument for departments on an ongoing basis:
    • to be relevant and enhanced innovation and flexibility
    • to have a variety of positive unintended outcomes
    • to hinder adoption in some cases due to the short-term duration,
  2. It is recommended that the Office of the Comptroller General increase its capacity (skills and resources) to:
    • play a more effective role in supporting departments in their use of the Generics
    • incorporate the feedback provided by departments for possible improvements
    • regularly update and make available the Generic Terms and Conditions: A Guide for Practitioners
    • improve performance measurement and reporting
  3. It is recommended that the Office of the Comptroller General strengthen its role in the transfer payment community by facilitating greater communication within the community, especially between departments that have used the Generics and those that are considering the Generics.
  4. It is recommended that TBS consult with partners, such as the Privy Council Office’s Impact Canada Initiative, to coordinate the Generics with parallel initiatives and improve government-wide access to innovative transfer payment approaches, as well as the ability to implement such approaches.

Appendix A: comparison of three departments offering access to innovative funding approaches for transfer payments

Department (name of transfer payment program) Funding category and approach Characteristics
Treasury Board of Canada Secretariat (Generic Terms and Conditions Pilot Program)

Incentive-based funding (contribution):

  • fixed contributions
  • base payment and premium payment
  • outcome achievement payments

Prizes/challenges (grant)

Micro-funding (grant)

  • The pilot is run using existing resources (no new funding)
  • Activities to launch the pilot included training and meeting with various departments
  • Ongoing advice and guidance provided by the Office of the Comptroller General
  • Primary supporting documentation provided through Generic Terms and Conditions: A Guide for Practitioners
Privy Council Office (Impact Canada Initiative)

Outcome-based funding:

  • is a mix of funding models, flexible authorities and methodologies
  • enables the use of grants, contributions or a mix based on need or design
  • includes challenge-based models, such as prizes/challenges, accelerators, grand challenges
  • includes other pay-for-success models, such as social impact bonds
  • may take the form of micro-funding
  • The program has been running with new dedicated resources provided in 2017–18
  • A dedicated centre of expertise within the Privy Council Office was established to help departments with researching, designing and delivering challenges
  • The Impact Canada Initiative challenge platform and engagement team were created to:
    • support the implementation of the challenge program
    • communicate and share policy and behavioural science research and results
  • A fellowship program was established to recruit external talent to fill gaps in capacity and skills in partner departments
  • The program was designed with authorities that are ongoing, pending assessment by 2022
  • A Minister at the Privy Council Office is responsible for granting access to departments to Impact Canada Initiative terms and conditions, which can be applied to new or pre-existing programs
Innovation Science and Economic Development Canada (Innovative Solutions Canada)

Prizes/challenges

  • $100 million in dedicated funding was provided to 20 departments in support of this initiative for three years ending in March 2020Footnote 7
  • The Innovative Solutions Canada platform and website were created to support implementation of the initiative
  • At this time, the program’s challenge stream continues to be funded through the end of 2022–23
  • The program includes both contracts and grants as options for the prizes/challenges

Appendix B: funding categories and approaches

Funding category Funding approaches and how they are used

Incentive-based funding
For approaches that are based on the achievement of predetermined outcomes, rather than solely on a reimbursement of eligible expenditures incurred by the recipient.

  • Outcome achievement payments, which are based on the achievement of predetermined, measurable outcomes rather than the expenditures incurred
  • Base payment and premium payment structure, which allows the payment of a premium if and when certain milestones are achieved
  • Fixed contributions, which compensates outcomes achieved for predetermined activities that are consistent with program objectives regardless of the expenditures incurred

Prizes/challenges
For approaches that are grants; the value of such grants is determined when the prizes/challenges are issued. There is no direct link between the cost incurred by the recipient and the value of the prizes/challenges.

  • Prizes/challenges, which are used to address a significant challenge where a satisfactory solution does not yet exist
  • Targeted prizes/challenges, which are used when a short list of eligible recipients can be identified and targeted for participating in a prizes/challenges process

Micro-funding
For approaches that are low risk and involve low-value grants

  • Micro-funding, which is used to administer low-risk projects consisting of low-value payments to individuals and not-for-profit organizations

Appendix C: logic model for the Generic Terms and Conditions Pilot Program (the Generics)

Long-term outcomes (benefits)

Federal transfer payment programs reach a broader range of recipients and collaborators

Innovations in transfer payment programs benefit Canadians

Assumption

Potential recipients are engaged and supported

Intermediate outcomes
(changes in behaviour)

The Generics enable more efficient implementation of programs in departments

Departments have the flexibility to design and deliver transfer payments

Departments use the Generics more frequently

Assumption

The Treasury Board of Canada Secretariat (TBS) and departments implement the administrative requirement for the Generics

Immediate outcomes
(changes in capacities, opportunities, motivations)

Senior management leads and provides support to practitioners in the implementation of the Generics

Departments have the capacity to use funding approaches in the Generics

Departments are aware of the Generics

Assumption

Departments are willing to take risks and experiment

TBS has the capacity to support the implementation of the Generics

Reach
(the beneficiaries)

Deputy heads, chief financial officers, centres of expertise on transfer payments

Program managers and practitioners

Outputs
(goods we produce, services we provide)

Communications and tools: frequently asked questions, GCpedia, training documents

Governance committees

Policy instrument and guide for practitioners

Reports and interpretation

Activities
(what we do)

Rollout by TBS

Monitoring and oversight

Policy instrument development

Advice

Appendix D: examples of outcomes and benefits achieved

In this section

The following are examples of departmental programs, including:

  • the transfer payment initiative that was used for the program
  • the funding approach used
  • the outcomes and benefits achieved by the program

Canada Economic Development for Quebec Regions: The Fast Forward Challenge Pilot Project (2019)

Funding approach

  • Prizes/challenges

Program outcomes

  • Grants awarded to start-up businesses (young and women entrepreneurs) help these businesses succeed

Organizational outcomes

  • Innovations in external communications to enhance the visibility of departments among a new clientele and foster an innovation-based work culture
  • High level of satisfaction among the businesses that requested an application package: the number of requests (273) and the number of applications (73) were considered excellent
  • The non-repayable prize ($50,000) and the number of prizes were the most appealing features

About the challenge

  • The challenge allowed departments to expand its pool of entrepreneurs
  • The challenge:
    • helped five winning businesses succeed by using other sources of funding and hiring qualified employees
    • ensured the survival of two other winning businesses
  • The challenge helped refresh the image of the government’s services through:
    • numerous consultations
    • advertisements aimed at both the targeted clientele and external stakeholders

National Defence: Innovation for Defence Excellence and Security (IDEaS) (2017)

Funding approach

  • Prizes/challenges

Program outcomes

  • An effective mechanism to engage innovators and solicit ideas to address defence challenges
  • Any solution answering the challenge statement could be proposed
  • Round 1:
    • was made easily accessible to a wide variety of entrants
    • succeeded in generating a broader interest among innovation community
  • In Round 2, winners with different technical domains worked together to develop an integrated solution

Benefits for innovators, companies, Canadians

  • Academics were able to access new resources by partnering with companies
  • Support for research and innovation
  • Technologies and solutions proposed through Pop Up City may have potential beyond improving the efficiency of deployed camps

Canadian Heritage: Promotion of Linguistic Duality

Funding approach

  • Micro-funding

Program outcomes

  • Some business process efficiencies were achieved, for example waiting periods for information requests improved from three or four weeks to two days
  • Projects were able to offer a better user experience
  • Innovations in business processes were tested
  • The experience could pave the way for other departments to use the Generic Terms and Conditions Pilot Program (the Generics)

Survey

  • Results from a survey of funded and not funded teachers and of participating students (n=800) revealed that:
    • the project had an impact on the community and there were possible benefits for Canadians
    • a majority of youth (87%) felt that they were able to express themselves in French by doing the activity that their teacher had prepared

Public Health Agency of Canada: Healthy Canadians and Communities Fund (2021)

Funding approach

  • Outcome achievement payments

Program outcomes

  • The fund:
    • was launched in 2018–19
    • is currently delivering programs
    • expects that using the Generics will lead to positive results that support program outcomes
  • Payments are based on a two-point change between the baseline and the subsequent assessment according to the average rate of change in systolic blood pressure

Organizational outcomes

  • Administration of projects was facilitated by:
    • understanding the processes for reporting
    • setting payment schedules
    • determining eligible expenditures

Global Affairs Canada: International Development Assistance Program – Colombia

Funding approach

  • Outcome achievement payments

Program outcomes

  • The program ends in 2025 and it is too early to determine the impact of projects on the program’s results
  • Funding from the private sector will help recipients achieve greater impact
  • Capacity was increased to adjust to major contingencies, such as the COVID-19 pandemic
  • The funding approach is more flexible, which helps recipients deal with changing situations more efficiently than traditional terms and conditions (Ts&Cs)

Immigration, Refugees and Citizenship Canada: The Settlement Program

Funding approach

  • Base payment and premium payment

Program outcomes

  • Too early to determine the impact of projects on program results

Organizational outcomes

  • Increase learning on results and deliver outcomes for transfer payments
  • Contribute to the switch from output-based results to outcome-based results for transfer payments in departments

Appendix E: Management Response and Action Plan

In this section

The Treasury Board of Canada Secretariat (TBS) - Financial Management Sector (FMS) of the Office of the Comptroller General (OCG), has reviewed the evaluation completed by the TBS - Internal Audit and Evaluation Bureau and agrees with its recommendations. Proposed actions to address the recommendations are outlined in the table below. 

Recommendation 1

Given that the Generics pilot was shown:

  • to be relevant and enhanced innovation and flexibility
  • to have a variety of positive unintended outcomes
  • to hinder adoption in some cases due to the short-term duration,

it is recommended that the Office of the Comptroller General establish the Generics as a transfer payment instrument for departments on an ongoing basis.

Management response

The OCG agrees that the Generics should be established for departments’ use on an ongoing basis.  

Proposed actions for recommendation 1 Start date Targeted completion date Office of primary interest

OCG sought and received approval from the Treasury Board to continue the Generics beyond March 31, 2022 (i.e. extended on an ongoing basis with no end date).  This approval supports the continued use of innovative financial approaches e.g., outcome achievement payments, fixed contribution, base + premium payment, prize/challenge, and micro-funding.

Fall 2022

Completed in March 2022

FMS

Recommendation 2

It is recommended that the Office of the Comptroller General increase its capacity (skills and resources) to:

  1. play a more effective role in supporting departments in their use of the Generics
  2. incorporate the feedback provided by departments for possible improvements 
  3. regularly update and make available the Generic Terms and Conditions: A Guide for Practitioners
  4. undertake and improve performance measurement and reporting

Management response

The OCG agrees that increased capacity would enable FMS to more effectively support the transfer payment community in the continued implementation and reporting of Generics.

Proposed actions for recommendation 2  Start date Targeted completion date Office of primary interest

FMS will:
Consult with key partners, such as the Privy Council Office (PCO) and Innovation, Science and Economic Development Canada (ISED), to inform an assessment of the appropriate level of capacity/resources required to more effectively support the transfer payment community in the continued implementation and reporting of Generics. Based on this assessment, FMS will develop a proposal to request additional resources.

In the absence of additional resources, FMS will focus on strengthening its role within the transfer payment community to provide support for the continued implementation of Generics as outlined in managements response to recommendation 3. 

September 2022

March 2023

FMS

Recommendation 3

It is recommended that the Office of the Comptroller General strengthen its role in the transfer payment community by facilitating greater communication within the community, especially between departments that have used the Generics and those considering the Generics. 

Management response

The OCG agrees that it can further support departments in the use of Generics by facilitating greater communication within the transfer payment community on uptake and the sharing of lessons learned and best practices.  

Proposed actions for recommendation 3 Start date Targeted completion date Office of primary interest

FMS will:

  • Increase the discoverability of its GCPedia page, and the Guide to Practitioners located therein, by including a link to the GCPedia G&C Portal in all team member signature blocks;
  • Conduct an annual survey to collect information/indicators from departments on the use of Generics, including: take-up; implementation/capacity; recipient reach and program impact; and lessons learned
  • Review and update the Generic Terms and Conditions: A Guide for Practitioners;
  • Promote use and create awareness of Generics through:
    1.  TBS Program Sector (along with existing guidance material), to encourage departments’ use of innovative financial approaches at the design stage of new programming. 
    2. The transfer payment community through transfer payment committee meetings and as needed through other governance committees, including the Deputy Chief Financial Officer monthly meetings, through email communication, etc.

June 2022

December 2023

FMS

Recommendation 4

It is recommended that TBS consult with partners such as Privy Council Office’s Impact Canada Initiative, to coordinate the Generics with parallel initiatives and improve government-wide access to innovative transfer payment approaches, as well as the ability to implement such approaches.

Management response

OCG agrees with the recommendation and will consult with key partners such as PCO and ISED to coordinate the Generics with parallel initiatives.

Proposed actions for recommendation 4 Start date Targeted completion date Office of primary interest

FMS will:
Consult with key partners such as PCO and ISED to determine how best to improve coordination of Generics with parallel initiatives and raise awareness of and promote the availability and discoverability of innovative transfer payments approaches, including the sharing of best practices to support departments’ ability to implement. Based on the feedback gathered during the consultations, an approach will be developed and implemented.

September 2022

March 2024

FMS

Page details

Date modified: