Evaluation of the Research and Policy Initiatives Assistance Program

From: Treasury Board of Canada Secretariat

On this page

Introduction

This report presents the results of the Research and Policy Initiatives Assistance Program (RPIA) evaluation, an unfunded, departmental contribution program co‑administered by Priorities and Planning (P&P) and the Corporate Services Sector (CSS) at Treasury Board of Canada Secretariat (TBS).

The TBS Internal Audit and Evaluation Bureau (IAEB) undertook the evaluation in accordance with the Treasury Board (TB) Policy on Results

The evaluation assessed RPIA’s efficiency and design alternatives, responsiveness and performance. The targeted assessment of program performance examined the extent to which RPIA achieved its immediate outcomes. IAEB undertook the evaluation from October 2020 to November 2021 and examined program activities between October 2017 to March 31, 2021.

The program’s terms and conditions (Ts&Cs) will expire on June 15, 2022, with renewal subject to the President’s assessment and approval. The RPIA Office (RPIA/O) requested this evaluation to inform the upcoming renewal.

Results at a glance

RPIA has existed since 2003. One finding bears particular reflection: the RPIA Program is a low-value, low-risk program and funding mechanism that performs very well in many ways, but is hampered in the area of efficiency, which is the primary reason for its creation.

Efficiency and design alternatives: With the program changes introduced in 2017 and at the level of use across sectors, the current resources are not adequate to administer the program. RPIA’s overall efficiency and its efficiency relative to two comparator programs have decreased since the last evaluation. Despite this, RPIA remains more efficient compared to the TB submission process.

The 2017 re‑design included changes that calibrated RPIA’s accountability, tracking and performance measurement mechanisms to other grants and contributions (G&C) programs, but at the expense of efficiency. Sectors reported a burdensome administrative process with lower dollar‑value projects, questioning RPIA’s overall value proposition.

Sectors were more satisfied with advice and guidance received from RPIA/O and CSS’s Financial Management Directorate (FMD) than they were with RPIA’s ease of access and clarity of roles and responsibilities.

For many projects, key informants (KIs) reported no suitable alternatives to RPIA.

Responsiveness: The new annual outreach and communications strategy helped grow program visibility across TBS, resulting in more program enquiries and use across sectors.

RPIA was responsive to sectors’ and funding recipients’ needs and priorities in many ways. RPIA projects also supported TBS during the COVID‑19 pandemic and enabled certain sectors to quickly scale up mission‑critical activities in new and emerging priority areas. Many sectors viewed RPIA as a versatile, flexible and necessary corporate tool.

Yet, despite most KIs reporting good fit between their projects and RPIA’s Ts&Cs, others raised concerns about the level of effort to fit projects within the T&Cs or the parameters of a contribution mechanism. Some sectors also reported longer‑than‑expected processing and approval timelines.

Findings on RPIA’s responsiveness and efficiency also suggest growing pressure to fulfill an ongoing, unmet need at TBS for public policy research.

Performance: RPIA is meeting its immediate outcomes to some extent. It is contributing to greater access to an expanded knowledge base related to public sector or public policy issues. However, the program has been unable to increase projects’ overall leverage of resources from external sources. This may be a result of increased stacking limits from 50% to 100% of eligible expenses. Issues with the theory of change and the current performance measurement strategy were also identified.

Role of the RPIA and its context

Background: Established in 2003, RPIA is a departmental class contribution program used to transfer monies to third party organizations for policy and research initiatives that contribute to the advancement of public policy research and public service management.

Prior to RPIA, sectors prepared individual TB submissions to fund low‑dollar, low‑risk initiatives, but later deemed these inefficient and costly. RPIA was created to provide a cost‑effective funding option for sectors to partner with third party organizations. The class contribution authority was provided under the 2000 Policy on Transfer Payments.

RPIA was first evaluated in 2008‑09 and again in 2013‑14. Core evaluation issues of program relevance, performance, efficiency and design alternatives were assessed.

Program expiry and renewal: On March 31, 2014, RPIA’s Ts&Cs expired but were renewed in 2017.

Informed by the 2013‑14 evaluation recommendations, RPIA’s 2017 renewal included changes to increase program awareness within TBS, bring the program’s Ts&Cs into alignment with the 2008 Transfer Payments Policy, and strengthen program oversight, performance measurement and reporting.

Program profileFootnote 1

Expected outcomes

RPIA’s expected outcomes, as shown in its logic model (see Appendix A), are:

Immediate outcomes

  • Access to an expanded knowledge base related to public sector/public policy issues
  • Increased leverage of resources from external partners

Intermediate outcomes

  • Increased contribution to knowledge related to emerging public sector or public service policy issues

Final outcomes

  • Contribute to the improvement, advancements and promotion of public sector policy research and public sector management

Governance, roles and responsibilities: RPIA is co‑administered by the RPIA/O, within the P&P Sector, and the FMD within the CSS. Funding for eligible projects comes from sponsoring sectors’ existing operating budgets.

RPIA/O is responsible for overall program administration and oversight, raising awareness of RPIA across TBS sectors, working with interested sectors to determine project proposal eligibility, and assessing proposals against RPIA’s Ts&Cs. RPIA/O also collects and analyzes results data from projects funded under the program and provides project status reports to senior management and the President’s office on forecasted contributions over $100,000.

FMDFootnote 2 conducts due diligence of each proposal, assessing each against financial policies and RPIA’s Ts&Cs. FMD also reviews proposed contribution agreements, ensures that sponsoring sectors have the necessary funds for projects, and issues payments in accordance with contribution agreements.

Sponsoring sectors undertake initial assessments of proposed projects, and work with potential recipients to complete project proposals and draft contribution agreements. Sectors also prepare approval documents,Footnote 3 ensure sign‑off by FMD (DCFO) and RPIA/O, and coordinate signing of contribution agreements. They are responsible for monitoring implementation, including milestone objectives and ensuring that recipients complete post‑project evaluations.

Target population, eligibility and limits: RPIA supports eligible projects in three categories:

  1. Research projects that are information-driven and support policy and legislative initiatives (research studies, research/discussion papers, publications)
  2. Innovation and experimentation projects that test innovative approaches and the acquisition of data and knowledge, which can be translated into program and policy action through evidence‑based decision-making (pilot projects, best practices, models/tools)
  3. Engagement projects that promote openness and transparency, and consist of direct communication (workshops, conferences, seminars, symposiums) or cross-sectoral and interjurisdictional consultation with experts and stakeholders, as well as informative and idea‑generating innovative engagement sessions with the general public

To be considered eligible, potential project proposals must demonstrate broad, public benefits beyond TBS.Footnote 4 ,Footnote 5

Eligible third‑party recipients include individuals, not‑for‑profit organizations, post‑secondary institutions, other levels of governments, international organizations, non‑government corporations and Indigenous communities.Footnote 6 These recipients would typically have, as part of their mandate, to pursue/encourage research, and the advancement of public sector policy and public service management.

Eligible expenses: All reasonable expenses directly related to the activities of the project, including professional fees, travel costs, salaries, supplies, communications costs, rental equipment and facilities.

Capital expenditures and activities of an ongoing nature are ineligible.

Funding limits: The maximum contribution per project is up to $500,000 over the life of the project. The maximum stacking level of the total government assistance is 100% of the eligible project expenditures.

Internal stakeholders: RPIA is available to all TBS sectors that want to contribute to ad hoc initiatives with eligible recipients to lever expertise and/or networks to advance public sector policy and public service management.

Program resources, budget, and cost of initiatives: RPIA has no full‑time dedicated resources. P&P uses existing full-time equivalents (FTEs), with an allocated level of effort of five hours per month at the EC‑03 level. The 2017 estimated level of effort for sponsoring sectors is five hours per project.Footnote 7 ,Footnote 8

RPIA has no targeted annual minimum funding amounts, nor is there any allocated funding for RPIA contributions.Footnote 9 TBS sectors support eligible projects through contributions provided directly from their budget.

Evaluation methodology and scope

The evaluation assessed RPIA’s responsiveness, efficiency, design alternatives and performance in relation to its immediate outcomes. The lines of evidence included:

  • document review
  • administrative data review
  • key informant interviews (n = 27)
  • survey of sectors with enquiries or incomplete/withdrawn RPIA applications (n = 3)

Only sectors in direct contact with the RPIA program office were included in the evaluation.

Limitations of the evaluation

The challenges and mitigations encountered in the conduct of this evaluation are outlined below.

Missing post‑project evaluation forms: Evaluators located post‑project evaluation forms for only 25% of completed projects. Evaluators used multiple lines of evidence to mitigate this issue.

Limited assessment of incomplete applications: Less than one-fifth (17%) of cases had documented reasons for discontinuation. Evaluators supplemented this with survey responses (n = 3) from sector contacts with enquiries and/or partial applications and, where available, data from KI interviews. Only a limited assessment of barriers to program access and use of RPIA alternatives among this group was possible.

1. Findings

In this section

Efficiency and design alternatives

Key evaluation question: To what extent did the RPIA re‑design improve or hinder program delivery and achievement of outcomes?

Conclusion: RPIA’s redesign both helped and hindered program delivery and achievement of immediate outcomes to some extent.

Early into the program re‑launch, RPIA/O developed the first ever business process and provided online access to descriptive materials and application templates. Interview evidence indicates that many sectors appreciated this, but also noted that it was overly cumbersome, too long to read, and in certain instances did not convey all the information needed before sectors engaged in the full application process.

Some KIs went further, noting negative effects of the process on relationships with partner organizations. In a few instances, this led to delays in project start dates and, in one project, required the rescheduling of an entire conference.

Sectors with projects involving smaller recipient organizations noted that RPIA’s resource-intensive application process exacerbated organizational capacity constraints. Smaller organizations, they reported, simply have too little budget and too few staff to engage in such an onerous business process (application and reporting). These findings are particularly noteworthy given that one of the original purposes of RPIA was to provide an agile, efficient funding mechanism to sectors.

RPIA/O also introduced a risk‑based approach to project approvals by delegating signing authority one level down from the Secretary.Footnote 10 This would streamline the application process for lower dollar‑value projects while ensuring that the Secretary was kept informed of RPIA commitments made above $100,000. However, there was no evidence that this approach reduced the level of effort or application processing time.

Another element of the re‑design included centralizing the program office within P&P and adopting a co‑administration model between P&P and CSS‑FMD.

However, KIs identified an absence of deep experience with G&C programs within P&P, resulting in significant time and resource use when RPIA/O was first established but, according to one interviewee, with few alternative options, “no one else in [P&P] had knowledge [about G&Cs] so I don’t know who else we could have given [the program] to”.

According to RPIA/O interviewees, the main benefit of housing the central administration office within P&P was additional rigour in the policy analysis/challenge function and oversight. Co‑administration between P&P and CSS‑FMD was reported by some to have contributed to improved delivery by having the two lenses assessing project applications and improving program visibility across two distinct functional communities.

Key evaluation question: What might have occurred in the absence of RPIA?

Conclusion: Many projects would have had no alternative in the absence of RPIA and would have experienced significant negative effects.

Most recipients indicated that without funding through RPIA, projects would likely not have occurred, even in modified form. In many instances, it was difficult to tease apart whether these effects would be due specifically to an absence of RPIA funding or an absence of TBS engagement and sponsorship of a project. A few recipients indicated that project outcomes would have taken much longer, in some instances years, to materialize, if at all, in alternative funding scenarios.

For example, several sector KIs reported that their projects might have been able to proceed with a contract, but that significant features or goals of the project would have had to be changed, a loss of quality in final product would have resulted, or partnerships would have been lost. In particular, nearly all sector KIs stated that selection of specific funding recipients was necessaryto achieve project goals.

Key evaluation question: What are the ongoing barriers to program efficiency?

Conclusion: The current management model and level of resourcing is inadequate to administer the program at its current level of use. Overall program efficiency has declined and is lower than comparator programs.

The level of effort estimated for the RPIA/O is five hours per month (0.25 FTE) at the EC‑03 level. However, the design changesFootnote 11 coupled with the increase in program uptake across sectors have resulted in higher levels of effort to administer the program than was initially estimated. Nearly all former and current RPIA/O staff indicated this was a significant underestimate of the actual resources needed to administer the program.

Many sponsoring sectors reported that the program’s administrative burden was heavy. The time investment relative to the funding amount was enough for a few sectors to reconsider using RPIA in the future. Echoing the findings from the prior evaluation, some recipient organizations indicated that the time and effort for low‑cost initiatives did not offer an enticing value proposition, relative to the administrative burden.

Most (70%) sponsoring sectors and some (33%) funding recipients reported that the application timelines, including developing the contribution agreement, were longer than anticipated.Footnote 12 The document review found that only two projects completed the process in less than RPIA’s publishedFootnote 13 8‑10 week estimate. In contrast, one comparator program (the evaluation looked at two comparator programs, one at Department of Finance Canada and the other at Industry, Science and Economic Development Canada [ISED]) estimated the average time to process an application was approximately one month.

“One of the biggest criticisms from us and our clients is that the process isn’t efficient. It definitely got clearer and expansion of eligibility criteria made [the program] more responsive, but it’s still not efficient”.

The previous evaluation found TBS’s RPIA to be the most efficient among the three comparator programs.Footnote 14 At the time, none of these programs, including RPIA, had dedicated resources. However, unlike the two comparator programsFootnote 15 in the current evaluation, TBS’s RPIA reported increases in the level of resources for administration. In the current evaluation, neither of the comparators actively promoted their programs and both had much lower use than TBS.Footnote 16 Both comparators had simpler application and assessment processes than TBS. The overall effect has been a decrease in the efficiency of RPIA relative to its comparators.

Contextual factors more unique to TBS may have resulted in further differentiation between TBS’s RPIA and its comparators. In particular, TBS has no other G&C program that supports the same kind of work as RPIA. Several KIs suggested this affects comparability across RPIA programs in two ways:

  1. Comparator departments have alternatives to RPIA and
  2. In the absence of an alternate program at TBS, coupled with the high staff turnover, sectors have comparatively less experience with the RPIA’s use and administration and with G&C programs in general

In addition, several sponsoring sectors noted that RPIA is different from most other G&C programs in that it can fund research projects. This accounts for most of RPIA projects. Several sponsoring sector and funding recipients spoke about the well‑known challenges of demonstrating outcomes of research.

KIs also reported additional contextual factors at TBS working against program efficiency:

  • frequent staff turnover within sectors requiring RPIA/O to re‑educate sectors on the use of contribution agreements
  • the program’s ad hoc nature and
  • the absence of a dedicated position or role within RPIA/O with requisite G&C experience

Interview evidence suggested that as part of RPIA’s re‑design, the goal of strengthening its accountability and performance measurement included establishing the same accountability, tracking and performance measurement mechanisms as other G&C programs. However, G&C programs are not typically used to fund research projectsFootnote 17 and measuring their outcomes can be difficult and imprecise. Nonetheless, this explains the introduction of a business process more typical of a higher‑risk and/or higher‑value G&C program.

This also explains why the performance measurement strategy introduced at the re‑design is more complex than either comparator program, both of which still have only minimal performance monitoring.

Several KIs, both with and without experience using RPIA prior to 2014, reported that the re‑design had led to a heavier, more burdensome business and administrative process.

Despite the inefficiencies, nearly all sectors reported being “very or extremely satisfied” with the advice and guidance received from RPIA/O and CSS‑FMD. That said, only 25% of sectorsFootnote 18 reported being “very or extremely satisfied”with RPIA’s ease of access and use.

The clarity of RPIA’s roles and responsibilities was not as satisfactory – especially in the earlier period of the re‑launch – with less than half (45%) of sectors being “very or extremely satisfied,”although levels of satisfaction have increased with more recent projects.

Suggestions for program improvements from sponsoring sectors included:

  • developing clearer, more succinct guidance materials and decision‑aids to support sector managers in deciding if RPIA is the right vehicle for a project, including potential project implications with the use of a contribution authority
  • re‑examining the business process to identify areas that could be streamlined
  • introducing a risk‑based assessment to strike a better balance between the cost and risk of a given project and the level of effort required to put the contribution agreement in place

Responsiveness

Key evaluation question: How and why has demand for RPIA changed since 2017?

Conclusion: The annual outreach and communications strategy played a role in the increased uptake of the program. Though RPIA was responsive in many ways, several barriers were identified.  

Effects of re‑design on program visibility and use

Low awareness and visibility of RPIA in past evaluations were noted as significant barriers to overall program responsiveness. As a result, RPIA/O implemented an annual outreach and communication strategy. 

“The redesign did influence [program] demand. Establishing it on Infosite and including it in the In‑brief once a year has increased demand. The marketing makes a difference”. ~ RPIA/O interviewee

Since October 2017, RPIA has been used by sectors to fund 14 initiatives and 10 third party recipients (see Appendix B for the list of funded initiatives, beneficiaries, and TBS contributions to projects). Sponsoring sectors contributed an average of $63,000 per initiative, with TBS contributions totaling $881,998.

This is a large increase since the last evaluation and represents double the anticipated volume. Evaluators also noted more than a three‑fold increase in the number of sectors using the program, most (71%) with no prior use of RPIA, and a two‑fold increase both in number and typeFootnote 19 of organizations funded.

In addition to the effect of the annual outreach and communication activities on program visibility and use, several RPIA/O interviewees reported that having CSS‑FMD as co‑leads also increased awareness of the program within their respective client sector. This led to more uptake as some sectors talk to their financial management advisors when exploring ways to engage external stakeholders.

Another design change made to the program was an increase in the stacking level to 100% of eligible costs,Footnote 20 from the previous 50%. The document review was unable to locate a rationale for this change. However, interview evidence indicated two reasons for the change to the stacking limit:

  1. to reduce potential financial barriers to partnerships and projects and
  2. to allow for more ambitious projects with greater focus on department interests

KIs did not indicate that the change to the stacking limit influenced their decision to use the program. Yet, the document review found that TBS contributed to more than half of eligible costs in over a third of projects, which had not occurred prior to 2017. Consistent with the rationale underlying the change, several KIs indicated that the stacking limit increase may have reduced the need for sectors to secure additional partnerships and funding for projects. The significance of this on the program’s achievement of its immediate outcomes is discussed in the Performance section.

The re‑design also added a third category to the stream of eligible RPIA project types. About one-third (36%) of the 14 projects funded through RPIA related to the new stream, Innovation and Experimentation (I&E) (see Figure 1).

Figure 1. Number Footnote 21 of funded projects, by project stream
Number of funded projects, by project stream
Research Engagement Innovation and experimentation
Number of projects 12 10 5

Sector KIs did not identify challenges in fitting projects within the different streams, with several sectors and RPIA/O interviewees reporting that these were broad and flexible enough to cover a wide range of projects and topics under the broad banner of public administration.

“RPIA has funded a lot of different things, it’s versatile, and almost anything can fit under public administration”. ~ former RPIA/O interviewee

A versatile, flexible and necessary mechanism

“This kind of program should be used by more government departments since it can be extremely responsive and can allow [external] organizations to contribute when [for example] a legislative window opens before it closes. Without this kind of program, there is less timeliness and overall less responsiveness to issues as they come up”. ~ RPIA funding recipient

KI interviews indicated that, overall, RPIA is a versatile, flexible and necessary funding mechanism for TBS.

Most sponsoring sectors noted that RPIA’s Ts&Cs and eligibility criteria were a good fit with their projects, flexible enough to be applied broadly, and that the program and its objectives are coherent. Most interviewees also reported that a contribution program like RPIA was necessary to achieve their project goals and saw RPIA as the sole suitable mechanism available to them.

Yet, as described in the upcoming section on barriers to program responsiveness, sectors reported uncertainty about whether a contribution was the most appropriate funding mechanism and the level of effort needed to align the project to RPIA’s T&Cs in a little less than half of funded projects.

A few sectors with narrow project launch timelines reported that RPIA was the only option available and that, without RPIA, projects would have likely been delayed or cancelled altogether.

COVID‑19 response

Interviews with several sectors demonstrated RPIA’s adaptability to changing contexts, notably in assisting TBS to respond to the COVID‑19 pandemic.

For example, one project brought TBS’s senior ranks together with public service experts to devise options and solutions to challenges posed by the pandemic. A second project focused on developing lessons from the pandemic, with the funding recipient noting that RPIA had accelerated the timelines for this discussion and that it likely could not have occurred without RPIA.

RPIA also supported timely adaptations during the pandemic, allowing the funding recipient to build skills in virtual event delivery and contributing to the highest number of participating countries in the event’s history.Footnote 22

Responding to gaps in areas of emerging priority

In certain instances, RPIA supported sectors in generating quick and much‑needed cross‑sectoral engagement in support of new or expanded mandates.

For example, the use of RPIA to fund the Centre for Greening Government (CGG) project, “Prioritizing Risks to Canada and the Federal Government of a Challenging Climate Assessment,” was described as instrumental and innovative. It was instrumental in attracting engagement and resources from other federal departments. It was also innovative in that it supported contribution to the larger climate change stakeholder community. Interviewees reported that without RPIA, a new contribution program would have been needed prior to undertaking similar projects. To be without a G&C program was not an option for such mission‑critical initiatives.

Similarly, one comparator department noted its RPIA program filled a crucial gap between the emergence of new priorities and the establishment of new G&C programs. In this organization, RPIA serves sector needs for ad hoc projects and is a mechanism for working on new or emerging priorities.

In another project example, sector KIs reported that RPIA helped (or could) help unlock a long‑standing divide in the public service between researchers and public policy practitioners. According to them:

“There’s no real chance to engage properly with researchers. RPIA is a good way to bridge the academic and public service practitioner gap … the $15K [contribution through RPIA] was a bargain to get this level of collaboration, both to spearhead research in general and to make it relevant to the public service. We heard from many event participants: why don’t we do this more often?”

Barriers to program responsiveness

“The program office was responsive to our needs, but we were victims of the process…The [program office] was responsive but the process is unresponsive”. ~ RPIA sponsoring sector

Although most sponsoring sectors and funding recipients described RPIA as responsive and well‑suited to their projects, many described significant challenges to responsiveness.

Interview evidence identified:

  1. uncertainty about whether a contribution was the most appropriate funding mechanismfor certain projects and
  2. the level of effort needed to align certain projects to RPIA’s T&Cs

A number of sectors reported that after their projects started, they may not have been well‑suited to RPIA. While the end result was positive, a more suitable funding mechanism would have been preferable, although none was identified. This may indicate a need to communicate more effectively with sponsoring sectors around projects that would be appropriately funded by a contribution. Previous evaluations also noted this challenge, showing there is still significant need for RPIA/O to support and educate sectors in distinguishing contributions from contracts.   

Aligning project proposals with RPIA Ts&Cs

Many KIs spoke about the level of effort needed to align projects with RPIA’s T&Cs which often resulted in numerous rounds of comments, questions, and feedback with RPIA/O and CSS‑FMD. This was especially evident in sectors with no prior use of RPIA, projects with stronger elements of basic policy research, and projects engaging international organizations. This was particularly the case when addressing eligible project costs and activities, and/or articulating broader benefits beyond TBS.

This finding was especially pronounced for applications received in the first two years after the re‑launch, coinciding with the earliest period of P&P’s newly established role as program administrator. KIs from that period pointed to a steep learning curve required to administer the program since it was an activity largely unrelated to their existing areas of expertise.Footnote 23 In addition, there was an immediate need to provide extensive application support to sectors, many without prior experience preparing contribution agreements, or using RPIA or other G&C programs.

Sectors recognized the legitimacy of the program’s eligibility criteria and constraints. Yet, in several instances, they struggled to ensure informal contribution commitments were within the confines of RPIA’s eligibility criteria prior to RPIA/O’s assessment of the project.

Nonetheless, nearly all sponsoring sectors recognized the need for a contribution mechanism like RPIA at TBS.

Other barriers to program access and use

Reasons for discontinuing the RPIA application and approval process were not found in the document review, the survey of sectors with enquiries to RPIA/O or inactive/incomplete applications, or by KI interviews. Data from the sector survey suggests that reasons for discontinuation lie beyond the program’s direct control (for example, projects are altogether cancelled or delayed as a result of changes in sector priorities). Yet, KI interviews suggest that the new business process and application processing timelines may have led certain sectors to reconsider the use of the program, at least in the earlier period of the program’s re‑launch.

Lastly, the document review and KI data show that enquiries from sectors with no prior use of the program were much less likely to proceed to full application and approval without

  1. some involvement from individuals with prior RPIA experience
  2. prior involvement with RPIA/O and/or the Strategic Policy Group at P&P or
  3. prior use of similar programs in other departments

Performance

Key evaluation question: To what extent have RPIA immediate outcomes been achieved?

Conclusion: RPIA has contributed to greater access to expanded knowledge in public sector or public policy areas. However, it has not increased its overall leverage of resources from external sources.

Immediate outcome: Access to expanded knowledge base related to public sector or public policy issues

RPIA/O performance data showed a total of 31 organizationsFootnote 24 engaged and 32 topics explored through project activities supported by the 14 contribution agreements. Project topics included regulatory modernization, leadership, public policy, performance, learning from the pandemic and evidence‑based decision‑making.

Most funding recipients said RPIA contributed “greatly or moderately” to increased project reach (number of individuals or organizations engaged) and the number or range of topics explored in projects. Similarly, nearly all sponsoring sectors and funding recipientsFootnote 25 also said RPIA contributed “greatly or moderately” to increased access to an expanded knowledge base.

Immediate outcome: Increased leverage of resources from external partners

According to the most recent RPIA annual report, a total of $584,998 across the 11 completed projects generated close to $1.5 million of external funding from partners.

Some sectors and funding recipients reported that RPIA funding helped to attract additional resources to their project. Others, however, said RPIA did not contribute to securing additional resources. In certain cases, this was a result of the sector already having all necessary funding and partnerships in place prior to approaching RPIA. In other instances, sectors had no need to secure more external resources since the total allowable TBS contribution increased from 50% to 100% of eligible project costs.

According to one RPIA/O interviewee,“…the moment you have 100% TBS contribution limit (vs 50%), it takes away absolutely needing partners or trying to lever [more resources]”.

As a result, the program did not meet its performance target of a 25% decrease in the project‑level ratios of TBS funding to external funding from 2014 levels.

Some KIs also found it challenging to separate out or otherwise qualitatively estimate the extent of RPIA’s contribution to its intended outcomes, for several reasons:

  • Projects often had multiple sources of funding
  • Reach‑related activities (such as report dissemination) were not always funded specifically by RPIA
  • Recipients did not always track the use of the results, products or reports generated from the project

In other instances, sectors indicated that it was not RPIA funding that attracted additional resources, but rather the involvement of TBS as a central agency (as opposed to RPIA itself) that helped secure funding or in‑kind resources.

2. Recommendations

  1. It is recommended that RPIA/O increase program efficiencies by:
    1. examining program resources to ensure alignment with current needs and program use
    2. posting updated tools and supports for sponsors’ review prior to them engaging the program
  2. It is recommended that the program’s performance be improved by:
    1. updating the application process, including application materials and processing times posted on the RPIA website
    2. ensuring that post‑project evaluations are received from sponsors prior to project close‑out
  3. It is recommended that the logic model and performance indicators be reviewed to better reflect the program’s primary purpose(s) and changes stemming from the 2017 re‑design, including the stacking limit.

Appendix A: RPIA logic model

Figure 2: RPIA logic model
Text version below:
Figure 2 - Text version

This graphic illustrates the logic model of the Research and Policy Initiatives Assistance Program (RPIA Program). The logic model shows that the final outcomes of the program are to“contribute to the improvement, advancements, and promotion of public sector policy research and public sector management”.

The RPIA Program has one intermediate outcome that aims to contribute to its final outcome. It is presented as follows:“Increased contribution to knowledge related to emerging public sector or public service policy issues”.

The program’s two immediate outcomes aim to contribute to the intermediate outcome. They are presented as follows:“Access to expanded knowledge base related to public sector or public policy issues, and increased leverage of resources from external partners”.

The program has three output clusters that are intended to lead to the achievement of its immediate outcomes. The first cluster of outputs includes contribution agreements, tools and guidance. The second cluster of outputs contains conferences, workshops, seminars, pilot projects, and other knowledge and outreach activities. The third cluster of outputs includes research, analysis, publications and data collection.

Three activities are intended to lead to the clusters of outputs. The first activity is undertaken by Corporate Services Sector, the RPIA Program Office and sponsoring managers, and is presented as“establishing expedient and flexible contribution agreements and providing tools and guidance”. The second and third activities, which are undertaken by sponsoring managers, are scanning academic and professional communities related to public sector or public service policy issues and identifying opportunities to level external partnerships.

Appendix B: RPIA projects and TBS contribution amount by year (2017‑18 to 2020‑21)

Start year Beneficiary Initiative TBS Contribution Amount
$
TBS Contribution Amount
% of total initiative funding
2017‑18 Council of Canadian Academies Climate Change Risks Facing Canada $0Footnote 26 0%
2017‑18 Public Policy Forum Public Policy Forum Fall Dinner $90,000 31%
2018‑19 Public Policy Forum Open Governance and Application Programming Interfaces: Delivering on Canada’s Digital Service Potential $23,000 60%
2018‑19 Union of BC Indian Chiefs Impacts of Bill C‑58 on First Nations’ Access to Information: Review and Engagement $65,037 100%
2018‑19 Mowat Centre Innovation in Evidence: Fostering Integrated Social Practice in Canada $16,500 8%
2018‑19 OECD Multi‑country senior civil service leadership project $30,461 13%
2019‑20 CD Howe Institute Policy Research Program Supporting Federal Regulatory Modernization $50,000 87%
2019‑20 OECD Research on Incorporating Competitiveness Impacts into Regulations $260,000 100%
2020‑21 Institute of Public Administration of Canada Dialogue Study Team on Nimbleness, Accountability and Oversight in Times of Crisis $0Footnote 27 0%
2020‑21 Canada Green Building Council Net Zero Carbon Retrofit Costing Study $30,000 8%
2020‑21 Public Policy Forum Public Policy Forum Fall Dinner $20,000 29%
2020‑21 OECD Conference on Measuring Regulatory Performance $130,000 UnknownFootnote 28
2020‑21 University of British Columbia (Okanagan) Crowding‑in or crowding‑out? Analysis of innovation government support for firms located in Canada $79,500 47%Footnote 29
2020‑21 Saint Mary’s University Measurement of Complementarities and Spillovers in Technological Innovation $87,500 67%Footnote 30
Total paid by TBS $881,998 42%

Appendix C: Management Response and Action Plan

Program management has reviewed the evaluation and agrees with the recommendations of the report.
Recommendations Proposed Action Start Date Targeted Completion Date Office of Primary Interest

Recommendation 1

It is recommended that RPIA/O increase program efficiencies by:

  1. Examining program resources to ensure alignment with current needs and program use.
  2. Posting updated tools and supports for sponsors’ review prior to them engaging the program.

We agree with the recommendation. Priorities and Planning will work with Corporate Services Sector to review existing program resources and ensure that they are aligned with the current needs of the RPIA Program, as well as its use by sectors across the department.

Priorities and Planning will also update existing tools and supports and will consult the Strategic Communications and Ministerial Affairs sector to explore new ways to support sponsors, including simplifying and updating the tools available to them on the Infosite to help streamline the user experience.

Corporate Services Sector will work to develop guidance for the department on the differences between contracts and contributions.

September 2022

Ministerial approval of renewed program will be sought in Summer 2022

Completion by: March 2023

Completion by: March 2023

Priorities and Planning and Corporate Services Sector

Recommendation 2

It is recommended that the program’s performance be improved by:

  1. Updating the application process, including application materials and processing times posted on the RPIA website. 
  2. Ensuring that post project evaluations are received from sponsors prior to project close-out.

We agree with the recommendation. Priorities and Planning will consult with Corporate Services Sector to update the application process, including exploring ways of improving application materials and decreasing processing times for applications, while still maintaining robust oversight and financial controls. Client feedback will be sought on the program changes after a trial period. Priorities and Planning will also work with the Corporate Services Sector to ensure that post project evaluations are received from sponsors prior to closing-out a project.

April 2022

March 2023

Priorities and Planning

Recommendation 3

It is recommended that the logic model and performance indicators be reviewed to better reflect the program’s primary purpose(s) and changes stemming from the 2017 re-design including the stacking limit.

We agree with the recommendation. Priorities and Planning will work with the Corporate Services Sector to review the logic model and performance indicators to ensure they better reflect the program’s primary purposes of supporting projects that contribute to the effectiveness of public service management practices through the improvement, advancement, and promotion of public policy research, experimentation and engagement.

April 2022

March 2023

Priorities and Planning

Page details

Date modified: