Evaluation of the Social Development Partnerships Program

From: Employment and Social Development Canada

On this page

Alternate format

Evaluation of the Social Development Partnerships Program [PDF - 130 MB]

Large print, braille, MP3 (audio), e-text and DAISY formats are available on demand by ordering online or calling 1 800 O-Canada (1-800-622-6232). If you use a teletypewriter (TTY), call 1-800-926-9105.

List of figures

List of tables

Executive summary

This report presents the findings and recommendations from the Evaluation of the Social Development Partnerships Program (referred to hereafter as “the Program”). It is a nationally delivered program that works in partnership with the not-for-profit sector to support people with disabilities, children and families, official language minority communities and other vulnerable populations. The evaluation of the Program covers the period covering the fiscal year 2012 to 2013 to the fiscal year 2016 to 2017, and focuses on:

  • the Program’s Disability component, specifically the 2012 Call for Proposals projects
  • the Program’s Children and Families component, specifically the Social Partnerships Pilot projects

While both components of the Program fall under the same branch, the distribution of funding and program design fall under the responsibility of two different Ministers and directorates within ESDC.

Background

In the fiscal year 2011 to 2012 (a fiscal year runs from April 1 to March 31), the Program introduced partnerships and leveraging requirements, as well as the third-party delivery model to test whether these features would improve the capacity of not-for-profit organizations to address social issues facing vulnerable populations. These approaches were tested extensively by the Children and Families component of the Program through the Social Partnerships Pilots, and to a lesser extent by the Disability component of the Program.Footnote 1 The implementation of these new requirements and delivery model was part of the evolution and renewal of the Program to move away from directed core funding (for the Disability component) and to test whether partnership oriented models could increase the pool of resources brought to bear on a given social issue, and as a result improve the Program’s impacts on vulnerable populations.

Evaluation scope

The evaluation assessed the extent to which the new funding requirements and delivery model increased the capacity of funded organizations to address issues facing vulnerable populations, in comparison to a more traditional project-based funding approach. It also assessed the Program’s relevance and results measurement strategy that was in place during the evaluation period.

Limitations

While the evaluation sought to assess the Program’s contribution toward improving the capacity of funded organizations to address issues facing vulnerable populations, it was limited in its ability to do so given the lack of data collected on beneficiaries. The evaluation did not seek to determine the Program’s attribution toward the social inclusion of vulnerable populations given that the financial support provided by the Program is relatively small, there are numerous government interventions in support of the same goal, and that there are many external factors that could influence social inclusion such as the state of the economy.

Key findings

Relevance

  • There is a continued need for the Program to support vulnerable populations through collaboration with, and funding for, not-for-profit organizations.
  • Not-for-profit organizations are well positioned to support vulnerable populations due to their connections to communities and understanding of their needs.

Effectiveness

  • The Program was successful in helping guide organizations toward meeting their leveraging goals. The majority of funding recipients met the leveraging targets set by the Program. The 17 Disability component funding recipients self-reported having collectively leveraged around $2.8 million worth of cash and in-kind resources from non-federal sources. Similarly, the 18 Social Partnerships Pilot funding recipients leveraged over $8 million.Footnote 2 However, the leveraging requirements did not translate into new funding sources or significant amounts of cash contributions for most organizationsFootnote 3.
  • The Program encouraged organizations to be innovative in establishing new partnerships. However, funded organizations emphasized existing partnerships in the implementation of their projects for several reasons, including a history of collaboration, a mutual understanding of the needs of target populations, and that new partnerships require a lot of time to develop and be effective.
  • The third-party delivery model is well-positioned to improve the reach of federal funding to smaller not-for-profit organizations, and to strengthen the capacity of not-for-profit organizations in a way that improves overall service delivery to communities.
  • The new requirements and delivery model introduced by the Program increased the capacity of funded organizations to implement their activities during the funding period. However, in most cases the funded organizations reported not having the capacity to continue delivering their activities beyond the funding period.
  • The funded organizations that secured large financial commitments from provincial governments were able to continue supporting vulnerable populations through their projects beyond the funding period. There is potential for the Program to explore greater collaboration with potential for effective and successful partnerships between the federal and provincial governments in supporting vulnerable populations.
  • The Program’s performance measurement was narrowly focused on partnering and leveraging, as opposed to results/outcomes on vulnerable populations. As a result, the calls for proposals and performance measurement strategy that were in place during the evaluated period did not enable a proper assessment of how the Program and the projects are achieving their stated objectivesFootnote 4.
  • Both components of the Program have improved their performance measurement strategies since the period covered by the evaluation. Most notably, more data is being systematically collected by the Program on project outcomes through increased collaboration between the Program and funding recipients with the development of evaluation plans.

Other findings

  • There are indications that the Program may be broadly contributing to the long-term well-being and social inclusion of vulnerable populations by funding not-for-profit organizations to deliver social development projects. For example, the funded organizations reported many positive results in the implementation of their projects, including greater awareness of barriers to social inclusion within communities and a greater sense of confidence, independence and well-being among participating individuals.

Recommendations

The Evaluation suggests the following 2 recommendations:

  1. Ensure that the Program and funding recipients have a common understanding of the Program’s objectives and expected results, as well as the key concepts (for example social inclusion, partnerships, sustainability), in order to enable better performance measurement.
  2. Improve monitoring and reporting templates (for example progress and final reports), as well as data collection tools (for example surveys), to capture detailed information on how the Program is impacting its intended beneficiaries.

Management response and action plan

Introduction

The Social Development Partnerships Program is a Grant and Contribution program that supports investment in not-for-profit organizations to help improve life outcomes for people with disabilities, children and families, and other vulnerable populations. The program has a $20 million annual budget divided in two components: Disability ($11 million) and Children and Families ($9.1 million).

The Program is aligned with ESDC’s vision with respect to individuals and families, namely to “build a stronger and more inclusive Canada, support Canadians in making choices that help them live productive and rewarding lives, and improve Canadians quality of life”. The Program supports ESDC’s core responsibility for social development which seeks to increase inclusion and opportunities for the participation of Canadians in their communities.

The Social Development Partnerships Program supports Government of Canada priorities through investment in not-for-profit organizations aiming to improve the life outcomes for people with disabilities, children and families, and other vulnerable populations. The use of Grant and Contribution investments represents a flexible and cost-effective way to support the role that communities, not-for-profit and voluntary sector organizations play in helping people with disabilities, families and children, and other vulnerable populations to be resilient and to provide them with the tools and skills to respond to current and emerging social issues.

The key findings presented in the evaluation indicate that the Program is making progress towards achieving its intended objectives and outcomes. The evaluation report includes two recommendations and Management supports both. The Management Response and Action Plan will be implemented by the Income Security and Social Development Branch in collaboration with the Program Operations Branch and relevant stakeholders where appropriate. In addition, the Income Security and Social Development Branch will continue to lead improvements to the Program Performance Information Profile.

It is important to note that the Program consistently endeavours to consult with external partners, including funded recipients on balancing the desire and need for robust performance measurement information, cost-effectiveness and efficiency.

Recommendation 1: Ensure that the Program and funding recipients have a common understanding of the Program’s objectives and expected results, as well as the key concepts (for example social inclusion, partnerships and sustainability), in order to enable better performance measurement.

Response

The program is already striving to address this recommendation. For example, funded recipients are required to collect relevant performance data and to report on the results of their projects. As such, a comprehensive communication approach to ensure that these recipients are well informed of Program objectives, expected results and key priorities will improve the quality of results reported by them and enhance their projects’ alignment with the Program’s goals and priorities. In addition, a common understanding of key concepts by both Program and funded recipients will enhance the quality of performance measurement.

More specifically, in the recent Social Development Partnerships Program Children and Families component Call for Concepts (focusing on Early Learning and Child Care), the objectives of the Program and those of the Call itself were clearly outlined in the applicant guide and the application template. Key terms and concepts were defined in the applicant guide. In their concept application, applicants were required to demonstrate the link between the goals and the expected outcomes of their proposed concepts and the objectives of the Children and Families component. A component of the assessment was on how well their proposed concepts aligned with the program objectives and priorities.

With respect to the Social Development Partnerships Program Disability component, and as part of the renewal of its operating funding to national organizations, launched in 2017, the Department consulted with the disability community through a Steering Committee to provide insight on principles and priorities that would contribute to the development of a program Performance and Accountability Framework. Under the renewed Disability component, recipients will be required to report on results achieved using a minimum set of baseline performance indicators. The Performance and Accountability Framework identified four principles to which performance indicators will be developed to measure the results of the Disability component operating funding to contribute to the health and maturity of the disability sector and ultimately to advance the social inclusion of Canadians living with disabilities. The performance indicators will measure organizational capacity in the following broad areas:

  • governance and accountability
  • effective leadership and operations
  • developing and maintaining partnerships
  • measurable impact

Management action plan

  • The Program will work with relevant stakeholders, including internal partners, to develop common definitions for the Program’s key terms and priorities such as social inclusion, sustainability, leveraging and partnership. The definitions will then be communicated clearly with all funding applicants to enhance their understanding of the Program’s expectations
    • Completion date: Spring 2019
  • The Program will continue to develop and improve tools and products, making them widely available, to ensure that funded recipients are well informed of the Program’s objectives, performance indicators, expected results and key priorities.
    • Completion date: Fall 2019

Recommendation 2: Improve monitoring and reporting templates (for example progress and final reports), as well as data collection tools (for example surveys), to capture detailed information on how the Program is impacting its intended beneficiaries.

Response

The ability to capture detailed information on how the Program is impacting its intended beneficiaries is essential in demonstrating efficiency in the management of the Social Development Partnerships Program. The detailed information on the Program’s impact will also inform relevant direction to program policy, program design and service delivery.

Since the implementation of the Children and Families component innovation pilots (from 2012 to 2015), the component has engaged in ongoing revisions to the final reporting template that funded recipients use to submit their final reports upon completion of their projects. These revisions include:

  • customized final reporting templates for each Call for Concepts/Proposals
  • request for information on specific project outcomes, reach of project and its impact on project beneficiaries
  • additional requirements for information, such as gender and target populations to enable Gender-based Analysis Plus analysis on project beneficiaries

The Children and Families component communicates performance measurement expectations to funding recipients throughout the application and funding processes. This includes discussions on performance measurement expectations at the start of the planning and designing of each Call for Concept/Proposal, through the development of the application guide, the assessment of concepts/proposals and the development of the terms of agreements for funded projects. Solicited Children and Families component project proposals are also reviewed from a performance measurement perspective prior to the assessment and approval of potential applications for funding. Projects are monitored during their implementation in order to ensure that evaluation plans including relevant result measurement indicators are developed and reported on.

To capture detailed information on the impact Disability component funding will have on its intended beneficiaries for projects from April 1, 2017 to March 31, 2018, the Performance Report Indicator Template for project funding was revised to include additional questions pertaining to partnerships, gender-based analysis, evaluation, success stories and lessons learned. For future performance reporting periods, the Department will consider whether further changes to the template are required.

Furthermore, the Program will continue to consider how the current data collection strategy can be improved, by revisiting the template with an aim to make it more concise, and ensure the continued applicability of the data collected to be in line with the performance measurement strategy.

Management action plan

  • In collaboration with funded recipients, the Program will continue to improve the monitoring and reporting templates to enhance the quality of data collected on how the Program is impacting its intended beneficiaries.
    • Completion Date: Fall 2019
  • To further improve the capacity of funded recipients to conduct relevant performance measurement of their projects, the Program will ensure funding is dedicated and support is available for the development of evaluation plans, including data collection, data analysis and reporting.
    • Completion Date: Fall 2019

1. Introduction

This report presents the findings from the Evaluation of the Social Development Partnerships Program covering the period starting from the fiscal year 2012 to 2013 to the fiscal year 2016 to 2017. It builds on four technical reports, namely, a file review, document review, key informant interviews with internal and external stakeholders, and case studies with selected funded projects by triangulating their findingsFootnote 5.

1.1 Program description

The Social Development Partnerships Program is a nationally delivered program that makes strategic investments to support government priorities related to people with disabilities, children and families, the voluntary sector, official language minority communities and other vulnerable populations. It works in partnership with not-for-profit organizations to help improve life outcomes of these target groups.

The Program supports Employment and Social Development Canada’s (ESDC) strategic outcome related to Income security, access to opportunities and well-being for individuals, families and communities.Footnote 6

The annual Grants and Contributions budget of the Program is over $20 million. The annual budget is divided among two components, where $11 million is allocated to the Disability component and slightly over $9 million is allocated to the Children and Families component. The current logic models for the Disability, and Children and Families components are provided in Appendix E and Appendix F respectivelyFootnote 7.

The two program components are described in greater detail below.

Disability component

The Disability component is overseen by the Office for Disability Issues within ESDC’s Income Security and Social Development Branch, and delivered by the Program Operations Branch within Service Canada. It supports projects intended to improve the participation and inclusion of people with disabilities in all aspects of Canadian society. The Minister of Public Services and Procurement and Accessibility is accountable for the Disability component.

The Disability component provides both operating and project funding. Operating funding is provided to increase the capacity of not-for-profit organizations with a primary mandate to promote the social inclusion of people with disabilities and to affect change among these populations. Project funding is provided to support a wide range of initiatives that address social issues and barriers that confront people with disabilities; develop tools and resources that address the needs of individuals, their families and communities; support services for people with disabilities; and foster partnerships and collaborative networks.

Children and Families component

The Children and Families component is overseen by the Community Development and Homelessness Partnerships Directorate within ESDC’s Income Security and Social Development Branch, and delivered by the Program Operations Branch within Service Canada. The component provides funding to eligible not-for-profit organizations for projects that help to improve the lives of children and families and other vulnerable populations. The Minister of Families, Children and Social Development is accountable for the Children and Families component.

There are three funding allocations within the Children and Families component that are integrated in the Terms and Conditions:

  • The Social Partnerships Initiative in Official Language Minority Communities (OLMC), which is part of the Roadmap for Canada’s Linguistic Duality, 2013–2018: Education, Immigration, Communities. The funded projects focus on youth in transition, changing demographics (seniors) and vulnerable populations in official language minority communities.
  • Canada's Volunteer Awards, which recognizes the enormous contributions that volunteers, not-for-profit organizations and businesses across the country make to help people and their communities.
  • The analysis and dissemination of the General Social Survey – Giving, Volunteering and Participation. This is an annual commitment for the Children and Families component to fund a single organization to analyze and disseminate results from the survey to not-for-profit organizations across Canada.

2. Evaluation scope and objective

The evaluation looks at the projects funded under the 2012 Call for Proposals for the Disability component, as well as the Social Partnerships Pilots under the Children and Families component. These projects were initiated and completed between the fiscal year 2012 to 2013 and the fiscal year 2016 to 2017.

The scope of the evaluation was developed in consultation of the Program and was approved at the June 2017 Performance Measurement and Evaluation Committee meeting. The following is an overview of the initiatives that were included or excluded from the evaluation. The initiatives that were excluded may be part of the scope of a future evaluation of the Program.

Children and Families component

  • Covered: Social Partnerships PilotsFootnote 8.
  • Not covered: Children and Families (ongoing projects), Canada’s Volunteer Awards, General Social Survey on Giving, Volunteering and Participating, and the Social Partnerships Initiative in Official Language Minority Communities.

Disability component

  • Covered: 2012 Disability Call for ProposalFootnote 9.
  • Not covered: Directed/core-funding of the national disability organizations and provincial/territorial organizations, 2016 Call for Proposal on Innovation, and the 2016 Call for Proposal on the Federal Accessibility Legislation.

The evaluation focuses on partnerships and leveraging requirements, as well as the third-party delivery modelFootnote 10, which were introduced by the Program in 2012. The aim of the evaluation is to assess whether these approaches and delivery model improved the capacity of the funded organizations to address social issues facing vulnerable populations. The evaluation also examines issues related to the program design and delivery, and explores the program’s result measurement strategies.

It should be noted that these requirements and delivery model were more extensively tested by the Children and Families component than the Disability component. This is due to the fact that the Social Partnerships Pilots were part of a broader initiative within the Department to test innovative approachesFootnote 11 to address complex social issues. While the 2012 Disability component Call for Proposals was not established for the specific purpose of testing these approaches, it did incorporate mandatory leveraging as one of the Call’s requirements.

3. Key findings

3.1 Relevance

Need for the program

There is a continued need for the Program to address various social issues facing vulnerable populations in Canada .

The file review and case studies indicate that the Program has, through the projects it funds, identified and addressed various needs, gaps, and social issues facing vulnerable populations, including barriers to social inclusion, during the funding period.Footnote 12

While funding recipients that were interviewed described needs as having been addressed during the course of the projects, many also expressed that these needs were ongoing and remained in the communities that were targeted by the projects. Therefore, not-for-profit organizations require continued support from the Program to address various needs, gaps, and social issues facing vulnerable populations.

Need to work with not-for-profit organizations

Working with not-for-profit organizations is perceived as a valuable approach to identify and address the needs of vulnerable populations and to deliver the Program.

Findings from the evaluation suggest that working with not-for-profit organizations is a good approach to tackling social issues faced by vulnerable populations. For example, key informantsFootnote 13 indicated that not-for-profit organizations have the proper connections and relationships necessary to serve vulnerable populations due to their operational knowledge and long history of working directly with, and delivering services to, vulnerable populations.Footnote 14

In general, key informantsFootnote 15 perceive the federal government’s role as an enabler in helping not-for-profit organizations deliver services to vulnerable populations. The document review indicates the complementary nature between not-for-profit organizations and governments in service delivery to vulnerable populations.Footnote 16 The former offers advantages, such as small scale and diversity of services, while the latter compensates the not-for-profit organizations’ inability to generate an adequate scale of resources by generating revenue.

However, the document review and case studies also highlight an ongoing challenge for not-for-profit organizations, which is the lack of financial capacity brought by decreasing revenues from government sources. This affects their capacity to deliver services to vulnerable populations, especially in an environment where demand for such services is increasing.Footnote 17

Alignment with federal policies and priorities, and departmental strategic outcomes

The Program’s objectives are aligned with, and contribute to, the Government of Canada’s policies and priorities and the department’s strategic direction.

The Program’s objectives and the strategies it uses to achieve its goals are aligned with the Government of Canada and the Department’s push for innovation to address social issues.Footnote 18 The document review highlights both the federal government’s and the Department’s commitment to solving public policy challenges through innovative approaches, which includes working with multi-sectoral partners to serve populations in need.

Appropriateness of federal roles and responsibilities in delivering the program

The Government of Canada has legal roles and responsibilities toward vulnerable populations, which the Program promotes through its activities.

The document review indicates that the Government of Canada has legal roles and responsibilities to ensure the social inclusion of vulnerable populations.Footnote 19 It also indicates that using an ‘all of Canada’ approach to responding to social inclusion may raise intergovernmental issues due to the fact that social inclusion interacts with many areas for which provinces are constitutionally responsible, such as poverty, housing, education and health care.Footnote 20

However, the federal government is not aiming to substitute from the responsibilities of the provinces in these areas. Rather, it aims to provide complementary support, for instance by fostering an environment that is supportive of social development projects. This is the case of the Social Development Partnerships Program, which encourages not-for-profit organizations to form multi-sectoral partnerships in an effort to promote innovative ways to address barriers to social inclusion facing vulnerable populations.

Similarities with other programs

The Program distinguishes itself from other federal programs by focusing on the creation of cross-sector partnerships.

There are several programs across the Government of Canada that support vulnerable populations.Footnote 21 However, the document review found no other currently existing government program that simultaneously supports social not-for-profit organizations and embeds the creation of cross-sector partnerships, which is integral to the Program’s delivery model.

3.2 Effectiveness

Expected results of the approaches and delivery model

During the evaluation period, the Program tested partnerships, mandatory leveraging and the third-party delivery model as a way to better support the capacity of not-for-profit organizations to address social issues faced by vulnerable populations.

The expected results of the implementation of these approaches included the establishment of new partnerships to improve community outcomes, the diversification of funding sources to increase the pool of resources for greater impact, and greater sustainability for the funded organizations.

Overall, it was found that the indicators that were developed to track these outcomes did not sufficiently consider beneficiaries, which would have been useful in determining outcomes on vulnerable populations.

In the fiscal year 2011 to 2012, the Program tested partnerships and leveraging requirements, as well as the third-party delivery model, to better support not-for-profit organizations to address social issues facing vulnerable populations. Figure 1 illustrates these different funding approaches.

Figure 1. Program delivery methods and funding approaches
Description immediately follows the chart.

Source: ESDC Evaluation Directorate Staff (2018)

Text description of figure 1

Figure 1 illustrates the different delivery methods and funding approaches used by the Social Development Partnerships Program during the funding period. The figure is divided into three main sections: conventional project funding, project funding with partnerships and leveraging requirements, and the third-party delivery model. All three sections describe a chain of events, using the Social Development Partnerships Program as a starting point. The different players are portrayed as circles. Arrow symbols link circles to each other, and represent a sequence and/or action.

The first section describes conventional project funding. Using the Program as a starting point, it illustrates the following sequence. The Program allocates project-based or core-funding to select not-for-profit organizations, and these organizations provide services to vulnerable populations.

The second section describes project funding with partnerships and leveraging requirements. Using the Program as a starting point, it illustrates the following sequence. The program allocates project-based funding to select not-for-profit organizations. These organizations enlist the help of partners and donors, which provide them with resources. The funded organizations then provide services to vulnerable populations. The circle representing the vulnerable populations is portrayed as larger than in the first section. This is to illustrate the expectation that funded not-for-profit organizations are able to reach more vulnerable populations using the resources provided by partners and donors.

The final section describes the third-party delivery model. Using the Program as a starting point, it illustrates the following sequence. The program allocates large project-based funding to a select number of large not-for-profit organizations. These large organizations then distribute grants to smaller not-for-profit organizations. Finally, each of these smaller not-for-profit organizations provide services to vulnerable populations. As in the previous section, the circle representing vulnerable populations is portrayed as larger than in the first section to illustrate greater reach through this approach.

These funding approaches were part of a broader initiative within the Department to test innovative approaches to respond to complex social issues. In the case of the Social Development Partnerships Program, the purpose of the testing was to inform the feasibility of using federal grants and contributions to leverage additional non-federal investments in mutually beneficial, locally-based solutions to complex social issues of importance to communities and the Government.

A review of the relevant Program documents suggests that there were three main results/outcomes that the Program was expecting to achieve with the partnerships, leveraging requirements, and the third-party deliver model. Notably, it was expected that the funding recipients would:

  1. engage new partners, or deepen relationships with existing ones, in a way that improves community outcomes (in other words for vulnerable populations);
  2. diversify their sources of funding, thereby increasing the pool of resources brought to bear on any given issue for greater impact; and
  3. achieve greater sustainability through partnerships and leveraging of new resources.

Overall, the evaluation found that the indicators that were developed by the Program to inform progress toward these results/outcomes were more focused on the funded organizations’ activities (for example partnerships formed, amount of resources leveraged) than outcomes on beneficiaries, which are more closely tied with Program objectives. In order to verify whether partnerships, leveraging requirements, and the third-party delivery model truly improved the organizations’ ability to address social issues facing vulnerable populations, data would have needed to be collected on the beneficiaries.

Outcomes of the approaches and delivery model

Partnerships

The funding recipients more commonly engaged existing partners due to their long history of working with them and their mutual understanding of the needs of the target populations. It was difficult to assess partnerships related issues overall, especially the engagement of new partners, due to the lack of clear definitions surrounding the concept and limitations in the reporting.

The program funding requirements put an emphasis on forming multi-sectoral partnerships with new partners, or strengthening relationships with existing ones, during the funding periodFootnote 22.

The file review and case studies indicate that the funding recipientsFootnote 23 mostly formed partnerships with existing partners, who often had a more formal and central role in the execution of the funded projects (for example active members in project advisory committees). Many funding recipients brought existing partners to the projects because of their long history of working together and their mutual understanding of the needs of their target populations. These well-established relationships were considered an important source of success in conducting the projects.

According to the file review and key informant interviews, factors such as mutual trust and interest were important in ensuring strong and sustainable partnerships. In contrast, many funding recipients indicated that newer partnerships required more time to be effective. While new partnerships were also formed, those new partnerships were often informal and sometimes only limited to the project period.

While the above findings suggest that the existing partners played a greater role than the new ones, it is difficult to assess the overall landscape of partnerships due to the fact that partnerships were defined in a way that covered all sorts of contributorsFootnote 24 (for example a one-time donor, regular/ongoing donors). It was also difficult to assess because the reporting requirements did not ask for additional distinctions, such as whether a partner is existing, new , formal, or informal. By further defining the concept of partnerships and communicating it to the funding applicants, the Program could garner a better understanding of the effectiveness of the partnerships approach.

While the above findings suggest that the existing partners played a greater role than the new ones, it is difficult to assess the overall landscape of partnerships due to the fact that partnerships were defined in a way that covered all sorts of contributors (for example a one-time donor, regular/ongoing donors). It was also difficult to assess because the reporting requirements did not ask for additional distinctions, such as whether a partner is existing, newFootnote 25, formal, or informal. By further defining the concept of partnerships and communicating it to the funding applicants, the Program could garner a better understanding of the effectiveness of the partnerships approach.

Achievement of leveraging requirements

Almost all of the funding recipients from the Disability and Children and Families components of the Program were able to meet the required leveraging targets. The in-kind contributions accounted for a bigger portion of the leveraging compared to cash contributions for most projects.

The file review showed that all of the Disability and Children and Families funding recipients leveraged cash and/or in-kind contributions throughout the course of the projects. Leveraged amounts were self-reported by the organizations, and in most cases, no further verification was conducted to confirm whether these amounts were received.

As shown in Table 1, all of the Disability funding recipients exceeded the basic requirement to secure at least 10% of total eligible project costs from non-federal government sources. In fact, on average, 31% of total eligible project costs were leveraged from non-federal sources.

Table 1. Target and achieved leveraging ratios by group/pilot type
Group / pilot type Target leveraging ratio ESDC grants and contributions Amount of cash and in-kind contributions leveraged Achieved leveraging ratio
Disability component 1:0.10 $ 9,046,055 $ 2,792,153 1:0.31
Community Innovation 3:1 $ 3,194,582 $ 1,754,055 3:1.7
Community Partnerships 1:1 $ 624,319 $ 600,138 1:≈1
Third Party Leadership 1:3 $ 2,233,023 $ 6,062,920 1:2.7

Source: ESDC Evaluation Directorate Staff (2018)

For the Children and Families component, the file review indicates that the majority (71%) of the Community Innovation Pilot projects met their leveraging goal. The Community Partnership Pilot was close to meeting its target, and one out of three organizations under the Third-Party Leadership Pilot met its leveraging goal. Most pilots/groups leveraged mostly in-kind contributions with the exception of the Third-Party Leadership Pilot. There is no standard way of determining the value of these contributions, and therefore the monetary value cannot be verified.

Sources of non-federal funding

The funded organizations mostly leveraged from other not-for-profit organizations, and to a lesser extent other levels of government. Given that not-for-profit organizations are limited in their capacity to provide resources, it is unlikely that the pool of resources brought to bear on the projects was significantly increased.

One of the goals of the leveraging requirements was to diversify the funding recipients’ sources of funding in an effort to increase the pool of resources brought to bear on any given social issue.

The file review indicates that not-for-profit organizations were the main providers of cash and in-kind contributions for the funding recipients, followed by other levels of government. For smaller organizations such as the ones under the Community Innovation Pilot, contributions from the not-for-profit sector comprised half of total leveraged resources.

In the case of the Disability component, a significant portion of the leveraged resources were provided by the funded not-for-profit organizations themselves. Table 2 shows that not-for-profit organizations, including recipient organizations, were the main contributors to funded projects.

Table 2. Sources of non-federal contributions leveraged by group / pilot type
Group / pilot Recipient organizations Non-profit organizations Regional or municipal government Provincial governments Private sector Other Unknown Total
Disability component 43% 24% <1% 9% 13% 9% 2% 100%
Community Innovation N/A 50% 11% 9% 7% 23% 0% 100%
Community Partnerships N/A 81% 0% 0% 19% 0% 0% 100%
Third Party Leadership N/A 36% 7% 26% 12% 18% 0% 100%

Source: ESDC Evaluation Directorate Staff (2018)

*The Disability component of the Program included “recipient organizations” as a distinct category for the sources of non-federal contributions, while the Children and Families component did not. It is likely that the recipient organizations from the Children and Families component also contributed their own resources, and that this was captured in the “not-for-profit organizations” field.

**This category may include individuals, or other donors that were not classified by the funding recipients.

The findings from the key informant interviews indicate that not-for-profit organizations have limited capacity to provide financial resources. For example, the funding recipients demonstrated difficulty sustaining their own projects due to lack of cash resources, and in some cases, similar limitations for partner not-for-profit organizations were also highlighted. While not-for-profit organizations provided a lot of in-kind contributions, these did not always translate into new resources for the projects. For example, one Children and Families component project tracked the time spent by its main group of pre-existing partners on advisory committees and other project-related activities as an in-kind contribution. In this case, these contributions accounted for 67% of all resources leveraged for the projects.

Given these limitations, it is unlikely that the pool of resources dedicated to the projects was significantly increased through leveraging other not-for-profit organizations. There is also the added issue of whether cash contributions that are made by other not-for-profit organizations are actually new resources or recycled money from within the public sectorFootnote 26, which further bring into question whether the pool of resources was actually increasing.

Challenges in leveraging the private sector

Leveraging from the private sector was challenging and less successful than expected, notably due to the sector’s preference for funding projects with clear proof of success.

The Program documentation suggests that there was an expectation for the Social Partnerships Pilots to leverage from the private sector. The case studies, file review and key informant interviews note that leveraging from the private sector was a challenge for all of the funding recipients. The file review indicates that funding recipients perceive that private sector organizations prefer concrete and reliable projects, and require proof of success upfront prior to providing funding. This suggests that the projects may not have been able to demonstrate expected results to the private sector. One reason given by funding recipients from the third-party delivery model on the lack of private sector contributions was that engaging the private sector was not part of the culture of community foundations.

The third-party delivery model

The third-party delivery model is well-positioned to strengthen the capacity of not-for-profit organizations to better serve vulnerable populations, and to expand the reach of federal funding to smaller not-for-profit organizations.

Findings from the third-party delivery model case studyFootnote 27 indicate that funding through third-party organizations may build the capacity of not-for-profit organizations to improve service delivery in a given community. The organization that was part of this case study reported using its local credibility to bring together various service delivery agencies in the area. By taking on the role of an honest broker, it was able to provide a structure for conflict resolution for the different agencies. In this case, this approach helped service providers in the community bridge long-standing differences, which resulted in an overall improvement to the community response system. The organization attributed this success to the fact third-party organizations:

While these findings cannot be generalized to all third-party organizations, Program officials have highlighted similar potential benefits to the third-party model. Many Program officials view third-party organizations as having the ability to use their on the ground knowledge to direct federal funding to areas most in need. Similarly, they also feel that the third-party delivery model increases access to federal funding for smaller not-for-profit organizations. This is supported by one of the partner representatives from the case study, who expressed appreciation for the fact that third party organizations could seek out leveraging opportunities on a grander scale with relative ease, thus relieving that pressure from smaller not-for-profit organizations.

Other benefits to the third-party model that were highlighted by Program officials include a greater understanding of the social issues and challenges faced by vulnerable populations, and the ability to rely on a diverse network of partners with different expertise to analyze and tackle social issues through various angles.

Efforts to improve funding recipients’ capacity and sustainability

Partnerships and leveraging requirements were part of the Program’s efforts to improve the capacity and sustainability of funded organizations. However, the Program and funding recipients do not appear to have the same understanding of organizational capacity and sustainability.

For both components of the Program, there was an expectation that leveraging resources and collaborating with partners would improve the capacity of the funded organizations to address social issues facing vulnerable populations, including barriers to social inclusionFootnote 28. There was also an expectation that these funding approaches would increase the sustainability of these organizations.Footnote 29

Overall, it would appear that the funding recipients did not have the same understanding of capacity-building as the Program.Footnote 30 In the case studies, most organizations were referring to the capacity of their projects as opposed to their organization. Sustainability was also understood more in the context of the project rather than the organization. This suggests that the funded organizations were not forming partnerships and leveraging contributions with the goal of making their organizations more sustainable.

Impact of the funding approaches on capacity

Collaboration with partners and leveraging of cash/in-kind contributions improved the capacity of funding recipients to implement their activities within the funding period.

EvidenceFootnote 31 suggests that partners and/or cash and in-kind contributions supported the capacity of funding recipients in the implementation of project activities. It was noted that partners and or leveraged resources improved capacity by expanding the reach of the projects in terms of social issues addressed, stakeholders reached, and beneficiaries served. For example, many partners were brought on board the Disability projects for their expertise and connections to different disability communities in order to make the project outputs more inclusive of different disabilities. For the Children and Families component, partnerships were made with local service providers in order to identify local needs and reach vulnerable populations. On occasion, cash was used to expand the scope of the project to serve more regions, and thus more beneficiaries.

Impact of the funding approaches on sustainability

Despite the Program’s efforts to enhance the sustainability of funded organizations through leveraging requirements and partnerships, many funding recipients reported not having the capacity to continue addressing needs, gaps, barriers and social issues facing vulnerable populations through their projects beyond the federal funding period.

The file review and case studies indicate that many projects from both components of the Program were not able to continue past the period of federal funding. More specifically, the activities were scaled back or sometimes cancelled beyond the end of the federal funding period due to insufficient financial resources. This suggests that the partnerships and leveraging requirements did not lead to increased sustainability of the funded not-for-profit organizations.

Sustainability and funding

The sustainability of organizations is tied to financial capacity, which may be enhanced through larger financial contributions, longer-term funding, or increased core-funding. Provincial contributions were a key factor in the sustainability of certain projects.

There is evidence that organizations that successfully leveraged large financial contributions had better capacity to sustain their projects. Specifically, there is evidence that many projects that were sustained were able to do so through financial support from provincial governments. The file review provides an example of a third-party organization that received a $1 million endowment fund from a provincial government, which will be used to provide six organizations with $8,000 for 3 years to sustain their programming. These findings suggest a potential for effective and successful partnerships between the federal and provincial governments in supporting vulnerable populations.

Many key informantsFootnote 32 expressed a need for either longer-term funding (for example 5 years), or core funding, to be sustainable. Some program officials have also noted that the lack of long-term or core-funding is a barrier to organizational capacity. Therefore, the Program may consider new ways outside of partnerships and leveraging to improve the sustainability of the not-for-profit sector to deliver services to vulnerable populations.

Measuring the program’s performance

Program funding processes and expected results

The Children and Families component used solicited and unsolicited calls to fund the projects, while the Disability component used an open call for proposal. Overall, the funding processes and the required supporting documents did not enable applicants to properly set up a strategy to measure results/outcomes in a way that informs the Program’s progress toward its stated objectives.

The Children and Families component used solicited and unsolicited calls to fund the projects under the different Social Partnerships Pilots. Despite the fact that the Program intended to assess how new approaches would improve results and lead to better outcomes and impactsFootnote 33, Program officials that were interviewed indicated that the expected results and assessment criteria for the calls were mostly focused around the achievement of partnership and leveraging requirements, as opposed to results/outcomes on vulnerable populations. Although the performance strategy was narrowly focused on leveraging and partnering, data on the target populations would have been needed to demonstrate how the Program is continually working towards its objectives.

The Disability component used an open call for proposals to fund the projects. According to the document review, the assessment criteria and eligibility requirements were based on the alignment of project objectives with those of the Program, as well as alignment with one or more of four priorities based on the United Nations Convention on the Rights of Persons with DisabilitiesFootnote 34. The 2012 Disability component call for proposals applicant guide did not explicitly identify expected results for the call, but listed examples of expected results that could be reported on.Footnote 35 The document review explains that the applicants were responsible for identifying goals, objectives, activities, outputs, and expected results/outcomes of the projects, as well as performance indicators and a data collection strategy. The document review also explains that the only performance indicators that were systematically collected by the Program over the course of the Disability projects were the number of partnerships formed, and the amount of cash and in-kind contributions leveraged.Footnote 36 As a result, the funding recipients mainly tracked data on leveraging and partnerships.

Performance of the Children and Families component

The performance strategy for the pilots was narrowly focused on leveraging and partnering. As a result, the performance measurement framework did not allow for a robust assessment of the impacts of the Program on vulnerable populations.

According to different lines of evidence, the Program’s Children and Families component used a Developmental EvaluationFootnote 37 approach to measure the results of the Social Partnerships Pilots. The primary data collection method consisted of semi-annual telephone calls with the funded organizations, and the evaluation questions revolved around testing innovation, assessing the capacity of not-for-profit organizations to leverage, and assessing the importance and sustainability of the partnerships on the impact of community outcomesFootnote 38. According to the responses of some Program officials, the Program could not truly assess the effectiveness of the Social Partnerships Pilots, for reasons which include:

Challenges such as demonstrating that the new models resulted in better outcomes or impacts as opposed to the previous funding model were also highlighted in the 2017 lessons learned reportFootnote 39. It was also noted that assessment of outcomes and impacts could be improved by the development of consistent and valid data collection processes and methodsFootnote 40.

Performance of the Disability component

The responsibility to set up and measure the expected results was left to the funding applicants. While examples of expected results were communicated in the call for proposal, the Program mainly asked funding recipients to provide data on partnerships and leveraging.

For the Disability component, the funding recipients were responsible for outlining their own objectives, expected results and outcomes, and developing their own results measurement strategy. Although the Program provided examples of expected results that the funding recipients could report onFootnote 41, this strategy did not yield enough data to effectively measure the Program’s performance. In addition, the only data that was systematically captured by the Program during the evaluation period was on partnerships and leveraging.

Improvements to the program’s performance measurement strategy

The Program is integrating evaluation plans into the projects to enhance their capacity to measure expected results/outcomes.

There are indications that the Program is improving its results measurement strategy to better capture the impact of the Program and the funded projects on vulnerable populations. One key initiative that was identified by the Program officials during the interviews was the development of evaluation plans in collaboration with the funding recipients at the on-set of the project. This is to ensure that the not-for-profit organizations are identifying indicators that will help measure the expected results/outcomes of the Program. This strategy is aligned with recommendations made by Program officials during the key informant interviews to improve the Program’s ability to measure its impacts, which is to build the capacity of the Program and the not-for-profit organizations to measure outcomes, impacts, and collect data at the population/individual level. In line with this recommendation, there is also an indication that the Program is integrating Gender-based Analysis Plus into their reporting.

3.3 Other findings

Gender-based Analysis Plus

There are some gaps in the data collection methods that limit the ability to conduct a retrospective Gender-based Analysis Plus of the Program’s initiatives.

The Gender-based Analysis Plus tool assesses the potential impacts of initiatives, programs or services on diverse groups of women, men and gender-diverse people. It also considers many other identifying factors, like race, ethnicity, religion, age, and mental or physical disability. One component of Gender-based Analysis Plus is the extent to which gender considerations are integrated into the data collection guidelines, forms and processes of a program.

While Gender-based Analysis Plus was not a requirement of the government during the evaluation period, the Program did not collect sufficient data at the sub-population level during the funding period in a systematic manner across all funded projects to allow for a thorough retrospective analysis.

Moving forward, the Program may consider ensuring that the collected data can be disaggregated by sex as well as other diversity factors and personal characteristics to be able to assess the reach and impacts of the program on different groups, identify potential disparities, and better monitor unanticipated gender-specific differences.Footnote 42

Program contribution to the well-being and social inclusion of vulnerable populations

Through its activitiesFootnote 43, the Program may be contributing to the well-being of vulnerable populations and their inclusion in communities and the workplace.

Although there is a lack of data on the outcomes of the Program, there are several positive aspects to the funded projects which can be interpreted as contributing to the well-being of vulnerable populations and their social inclusion. Notably, the funded organizations have reportedFootnote 44 that their projects resulted in:

4. Areas for consideration

The findings from the Evaluation indicate that the Program did not employ an adequate framework to measure progress toward its stated goals. In particular, the Evaluation noted that the calls for proposals did not explicitly define overarching expected results/outcomes, aligned with the program’s broader outcomes, for the funded organizations to report on. This reduced the possibility that the funding recipients could report on the appropriate results.

The potential funded organizations share responsibility in measuring the performance of the Program, given that they are the main vehicle through which it is delivered. The funded organizations are expected to properly identify and define the issues they are addressing, and determine the expected results of their own initiatives. In turn, this will ensure that both the Program and the funded organizations are facilitating the measurement of outcomes and impacts related to vulnerable populations.

The evaluation recognizes that any additional responsibilities required on behalf of the funded organizations with regard to data collection and reporting may result in more time and resources having to be invested in such activities, which may take away from project delivery. The challenge of balancing reporting requirements with project delivery has been noted in the different lines of evidence and Program documents. Therefore, additional support and resources could be provided by the Program to improve the evaluation capacity of not-for-profit organizations, if necessary.

5. Recommendations

The Evaluation suggests the following 2 recommendations:

  1. Ensure that the Program and funding recipients have a common understanding of the Program’s objectives and expected results, as well as the key concepts (for example social inclusion, partnerships, sustainability), in order to enable better performance measurement.
  2. Improve monitoring and reporting templates (for example progress and final reports), as well as data collection tools (for example surveys), to capture detailed information on how the Program is impacting its intended beneficiaries.

Appendix A – Evaluation questions

Issues, evaluation questions, indicators and line of evidence

Relevance:

1. Is there a continued need for the program to seek new ways to address social issues faced by children and families, and people with disabilities?

1.1 Evidence that current gaps and/or needs are addressed by not-for-profit organizations

  • Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

1.2 Evidence that working with not-for-profit organizations help to adequately support vulnerable populations

  • Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

1.3 Evidence on current gaps identified that are not addressed or covered by the program

  • Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

2. Does the program align with and contribute to the Government of Canada policies and priorities, and ESDC

2.1 Evidence of consistency between the program priorities and the department strategic outcomes as well as the federal government policies and priorities strategic outcomes?

  • Lines of evidence: Document review

3. Are the current roles and responsibilities of the federal government appropriate in delivering the program?

3.1 Evidence that the program activities are aligned with federal roles and responsibilities in regards to social inclusion

  • Lines of evidence: Document review, key informant interviews (internal)

3.2 Evidence that similar programs or services are not offered within or outside the federal government

  • Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

Effectiveness:

4. Was an appropriate framework in place to facilitate results measurement of the Social Partnerships Pilots of the Children and Families component as well as the funded projects of the 2012 Disability component call for proposal? What could be improved for the future?

4.1 Evidence of result measurement strategy as part of the call for proposals that clearly identifies the expected results and assessments criteria to support the funding decisions

  • Lines of evidence: Document review, key informant interviews (internal), case studies

4.2 Percentage of projects who indicated having developed and used outcome performance indicators and evaluation processes to measure project results

  • Lines of evidence: File review

4.3 Evidence on the evaluation methods and/or measurement tools to evaluate the project results and progress towards contributing to the program’s expected outcomes

  • Lines of evidence: Document review, file review, key informant interviews (internal), case studies

4.4 Evidence on how funded organizations and the program may enhance their capacity in regards to social impacts measurement in the context of social innovation and financing

  • Lines of evidence: Document review, key informant interviews (internal), key informant interviews (external), case studies

5. Did these models improve the capacity of funded organizations to address barriers to social inclusion?

5.1 Evidence as to whether these new socially innovative models (in other words, leveraging, partnerships and third party model) were able to better serve the funded organizations in enhancing their ability/capacity to impact vulnerable populations in addressing barriers and challenges to social inclusion

  • Lines of evidence: Document review, key informant interviews (internal), key informant interviews (external), case studies

Disability component only questions

6. To what extent have partnership requirements among funded organizations under the 2012 Disability call for proposal been successful in:

6.1 Attracting partners to funded projects?

6.1.1 Number of partnerships created or strengthened, by sector (for example, private, public, not-for-profit, etc.)

  • Lines of evidence: File review

6.1.2 Evidence on successes, challenges and level of efforts in attracting and retaining partnerships (for example, collaborations, alliances, networks, etc.)

  • Lines of evidence: File review, key informant interviews (external), case studies

6.1.3 Approaches adopted to attract partnerships

  • Lines of evidence: File review, key informant interviews (external), case studies

6.2 Maximizing partners’ contributions to funded projects?

6.2.1 Evidence that partners contributed to improving organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional partnerships; replicating and scaling project

  • Lines of evidence: File review, key informant interviews (external), case studies

6.3 Developing successful and sustainable partnerships (what made the partnerships more successful and sustainable)?

6.3.1 Evidence that partnerships created or strengthened were effective and/or continued even after the funded projects ended (for example, types of collaborations, roles or contributions of partners)

  • Lines of evidence: File review, key informant interviews (external), case studies

6.4 Ensuring more inclusivity of organizations and people with disabilities?

6.4.1 Evidence on the types and complexities of social issues and people with disabilities that were addressed and/or reached by the organizations and through their partnerships

  • Lines of evidence: File review, key informant interviews (external), case studies

6.4.2 Evidence that partnerships enhanced the capacity of organizations in ensuring greater inclusivity of disability organizations and in terms of better outcomes for more people with various types of disabilities

  • Lines of evidence: Key informant interviews (external), case studies

6.5 Expanding the reach of funded organizations?

6.5.1 Evidence that partnerships led to greater reach for the funded organizations in terms of dealing with more social issues, serving more people with disabilities, regions across Canada, official languages, etc.

  • Lines of evidence: File review, key informant interviews (external), case studies

7. To what extent have leveraging requirements among the organizations funded under the 2012 Disability call for proposal been successful in:

7.1 Leveraging investments from non-federal organizations?

7.1.1 Percentage of funded recipients that leveraged cash and/or in-kind contributions from non-federal sources

  • Lines of evidence: File review

7.1.2 Percentage of total cash and/or in-kind contributions leveraged from non-federal sources for each type of contributor (for example, public, private, not-for-profit, etc.)

  • Lines of evidence: File review

7.1.3 Amount of leveraged cash and/or in-kind contributions from non-federal sources for every dollar invested by the program (ratio)

  • Lines of evidence: File review

7.1.4 Approaches adopted to leverage cash and/or in-kind investments by non-federal sources

  • Lines of evidence: File review, key informant interviews (external), case studies

7.1.5 Evidence on successes, challenges and level of efforts in leveraging of cash and/or in-kind contributions from non-federal partners

  • Lines of evidence: File review, key informant interviews (external), case studies

7.2 Maximizing expertise in the funded projects?

7.2.1 Evidence on the use of cash and/or in-kind contributions leveraged in the funded projects by category (for example, administrative/professional services, volunteer time, office space, evaluation, etc.)

  • Lines of evidence: File review, key informant interviews (external), case studies

7.2.2 Evidence that cash and/or in-kind investments from non-federal sources contributed to improving organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional investments; replicating and scaling project

  • Lines of evidence: File review, key informant interviews (external), case studies

7.3 Ensuring more inclusivity of organizations and people with disabilities?

7.3.1 Evidence that cash and/or in-kind contributions from non-federal sources enhanced the capacity of organizations in ensuring greater inclusivity of disability organizations and in terms of better outcomes for more people with various types of disabilities

  • Lines of evidence: Key informant interviews (external), case studies

7.4 Expanding the reach of the funded organizations?

7.4.1 Evidence that cash and/or in-kind contributions from non-federal sources led to greater reach for the funded organizations in terms of dealing with more social issues, serving people with disabilities, regions across Canada, official languages, etc.

  • Lines of evidence: Key informant interviews (external), case studies

7.5 Sustaining both monetary and in-kind investments from non-federal organizations?

7.5.1 Evidence that cash and/or in-kind contributions from non-federal sources continued even after the projects ended (for example, nature, scope and/or use of contributions, etc.)

  • Lines of evidence: Key informant interviews (external), case studies

8. Has the 2012 Disability call for proposal been successful in creating and transferring knowledge and approaches that address barriers to social inclusion faced by people with disabilities?

8.1 Number of knowledge and/or approaches developed by type (for example, programs, services, resources, tools, pilot projects, best practices, training guides, innovative solutions, etc.)

  • Lines of evidence: File review, key informant interviews (external), case studies

8.2 Evidence that the knowledge and approaches developed were shared and disseminated to the target population and other stakeholders, including partners

  • Lines of evidence: File review, key informant interviews (external), case studies

8.3 Percentage of the knowledge and approaches developed that reached the target audience (for example, people with disabilities, organizations, municipalities, employers, clinicians, etc.)

  • Lines of evidence: File review, key informant interviews (external), case studies

8.4 Percentage of funded projects that have developed tools and approaches that are scalable, transferable and sustainable

  • Lines of evidence: File review, key informant interviews (external), case studies

Children and Families component only questions

9. To what extent were requirements to leverage support from other organizations successful in:

9.1 Leveraging monetary and in-kind investments from non-federal organizations?

9.1.1 Percentage of funded recipients who were able to reach their leveraging targets through cash and/or in-kind contributions from non-federal sources

  • Lines of evidence: File review

9.1.2 Percentage of total cash and/or in-kind contributions leveraged from non-federal sources for each type of contributor (for example, public, private, not-for-profit, etc.)

  • Lines of evidence: File review

9.1.3 Amount of leveraged cash and/or in-kind contributions from non-federal sources for every dollar invested by the program (ratio)

  • Lines of evidence: File review

9.1.4 Approaches adopted to leverage cash and/or in-kind investments by non-federal sources

  • Lines of evidence: File review, key informant interviews (external), case studies

9.1.5 Evidence on successes, challenges and level of effort in leveraging of cash and/or in-kind contributions from non-federal sources

  • Lines of evidence: File review, key informant interviews (external), case studies

9.2 Enhancing the contribution of both monetary and in-kind investments towards achieving projects’ intended outcomes?

9.2.1 Evidence on the use of cash and/or in-kind contributions leveraged in the funded projects by category (for example, administrative/professional services, volunteer time, office space, evaluation, etc.)

  • Lines of evidence: File review, key informant interviews (external), case studies

9.2.2 Evidence that cash and/or in-kind contributions from non-federal sources improved organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional investments; replicating and scaling project

  • Lines of evidence: File review, key informant interviews (external), case studies

9.3 Sustaining both monetary and in-kind investments from non-federal organizations?

9.3.1 Evidence that cash and/or in-kind contributions from non-federal sources continued even after the projects ended (for example, nature, scope and/or use of contributions, etc.)

  • Lines of evidence: File review, key informant interviews (external), case studies

9.4 Expanding the reach of funded organizations?

9.4.1 Evidence that cash and/or in-kind contributions from non-federal sources led to greater reach for the funded organizations (for example, social issues, vulnerable populations, regions across Canada, official languages, etc.)

  • Lines of evidence: Key informant interviews (external), case studies

10. To what extent were partnership requirements, as per funding agreements, successful in:

10.1 Attracting new partners to contribute to funded projects?

10.1.1 Number of partnerships created or strengthened, by sector (for example, private, public, not-for-profit, etc.)

  • Lines of evidence: File review

10.1.2 Evidence on successes, challenges and level of efforts in attracting and retaining partnerships

  • Lines of evidence: File review, key informant interviews (external), case studies

10.1.3 Approaches adopted to attract partnerships (for example, collaborations, alliances, networks, etc.)

  • Lines of evidence: File review, key informant interviews (external), case studies

10.2 Maximizing partners’ contributions toward the achievement of project intended outcomes?

10.2.1 Evidence that partners contributed to improving organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional partnerships; replicating and scaling project

  • Lines of evidence: File review, key informant interviews (external), case studies

10.3 Developing successful and sustainable partnerships?

10.3.1 Evidence that partnerships created or strengthened were effective and/or continued even after the funded projects ended (for example, types of collaborations, roles or contributions of partners)

  • Lines of evidence: File review, key informant interviews (external), case studies

11. To what extent did the third party delivery (intermediary) approach:

11.1 Result in leveraging from non-federal organizations?

11.1.1 Percentage of funded recipients who were able to reach their leveraging targets through cash and/or in-kind contributions from non-federal sources

  • Lines of evidence: File review

11.1.2 Percentage of total cash and/or in-kind contributions by non-federal sources leveraged from each type of contributor (for example, public, private, not-for-profit, etc.)

  • Lines of evidence: File review

11.1.3 Amount of leveraged cash and/or in-kind contributions from non-federal sources for every dollar invested by the program (ratio)

  • Lines of evidence: File review

11.1.4 Approaches adopted to leverage cash and/or in-kind contributions from non-federal sources

  • Lines of evidence: File review, key informant interviews (external), case studies

11.1.5 Evidence on successes, challenges and level of efforts in leveraging of cash and/or in-kind contributions from non-federal partners

  • Lines of evidence: File review, key informant interviews (external), case studies

11.1.6 Evidence that the third party (intermediary) approach can leverage more easily and effectively pre-existent resources – in terms of partners, contributions, expertise, etc. – towards addressing different types of social issues faced by vulnerable populations and communities

  • Lines of evidence: Key informant interviews (external), case studies

11.2 Develop successful and sustainable partnerships?

11.2.1 Evidence that partnerships created or strengthened were effective and/or continued even after the funded projects ended (for example, types of collaborations, roles or contributions of partners)

  • Lines of evidence: File review, key informant interviews (external), case studies

11.2.2 Establishment of new networks and expansion of these networks beyond the length of the project

  • Lines of evidence: File review, key informant interviews (external), case studies

11.3 Expand the reach of funded organizations?

11.3.1 Evidence that the third party (intermediary) approach led to greater reach for the funded organizations (for example, social issues, vulnerable populations, regions across Canada, official languages, etc.)

  • Lines of evidence: Key informant interviews (external), case studies

11.4 Influence system change?

11.4.1 Evidence that the third party (intermediary) approach may have a better understanding of social issues and challenges faced by vulnerable populations and communities

  • Lines of evidence: Key informant interviews (internal), key informant interviews (external), case studies

11.4.2 Evidence that the third party delivery (intermediary) approach is better positioned to increase community and/or organizational capacity towards addressing different types of social issues and challenges faced by vulnerable populations and communities

  • Lines of evidence: Key informant interviews (internal), key informant interviews (external), case studies

11.4.3 Evidence that the third party delivery (intermediary) approach improved or expanded services to vulnerable populations, thus increasing the impacts of federal spending

  • Lines of evidence: Key informant interviews (internal), key informant interviews (external), case studies

Appendix B - Data collection methods and limitations

In total, 4 lines of evidence were conducted as part of the evaluation:

  • a file review
  • a document review
  • key informant interviews with internal and external stakeholders
  • case studies

A technical report was produced for each line of evidence.

The Evaluation Directorate conducted the file review, document review and part of the key informant interviews in house, while the case studies and key informant interviews (external) were conducted by a consulting firm. All 4 reports were finalized in consultation with the Program areas. The final report was also peer reviewed by two academics with expertise relating to the themes addressed in the evaluation.

The evaluation sought to assess the Program’s contribution toward improving the capacity of funded organizations to address barriers to social inclusion, but was limited in its ability to do so given the lack of data collected on beneficiaries. The evaluation did not seek to determine the Program’s attribution toward the social inclusion of vulnerable populations given that the financial support provided by the Program is relatively small, there are numerous government interventions in support of the same goal, and that there are many external factors that could influence social inclusion such as the state of the economy.

The following limitations are noted with respect to evaluation data and their collection methods:

  • There was an overall lack of adequate data to respond to the various evaluation questions and lines of evidence. Notable challenges include:
    • The information contained in the project filesFootnote 46 offered limited insight on the link between leveraging and partnering indicators and the Program’s intended outcomes and impacts.
    • Insufficient number of Program officials interviewedFootnote 47 had sufficient knowledge of the funded projects and the surrounding processes covered during the evaluation period (for example intake procedures, performance measurement strategies). This is likely due to staff turnover within the Program.
    • Insufficient number of interviewsFootnote 48 with project representatives and partners that could provide detailed information on the funded projects. This is likely due to the time elapsed since the period covered by the evaluation, staff turnover within organizations, and limited knowledge of funded projects.
  • For the case studies, a small sample of organizationsFootnote 49 was selected purposefully by the Evaluation Directorate in collaboration with the Program. The sample size should not be considered representative of the broader population. This is also true for the external key informant interviews, which were selected from the case study organizations. To address this limitation, findings were triangulated with other lines of evidence. Notably, the file review covered historic files from all 37 projects.
  • The value of the in-kind contributions leveraged is not standardized (in other words two organizations may value the same in-kind contribution differently), and therefore leveraging data should be interpreted with caution. Additionally, leveraging information for the Disability component was not compiled or tracked in a way that ensures reliable dataFootnote 50.
  • The lack of data at the population level during the period covered severely limits the ability of the Evaluation to conduct a retrospective Gender-based Analysis Plus. However, according to Program officials, Gender-based Analysis Plus is being integrated into the Program reporting.

Appendix C – List of technical reports

ESDC (2019). The Evaluation of the Social Development Partnerships Program: Document Review Technical Report. (Not published, but available on demand.)

ESDC (2019). The Evaluation of the Social Development Partnerships Program: File Review Technical Report. (Not published, but available on demand.)

ESDC (2019). The Evaluation of the Social Development Partnerships Program: Key Informant Interview Technical Report. (Not published, but available on demand.)

ESDC (2019). The Evaluation of the Social Development Partnerships Program: Case Study Technical Report. (Not published, but available on demand.)

Appendix D - Program evolution, renewal and activities

Disability component

The funding model of the Disability component consists of an operating funding allocation of $6 million per year to national disability organizations; $2 million per year to regional organizations under the Community Inclusion Initiative; and an allocation of $3 million per year for unsolicited projects and funding through call for proposals.

In 2012, an open Call for Proposals was launched which introduced the requirements to leverage funding from non-federal government sources (mandatory 10% leveraging requirement) along with the strengthening of a results orientation and outcome reporting. A total of seventeen projects received funding. These projects ended on March 31, 2016.

In spring 2016, the Minister of Sport and Persons with Disabilities approved the renewal of the Disability component of the program with the goal of designing a Performance and Accountability Framework in collaboration with the disability community, to ensure fairness and transparency in the distribution of funding. The framework sets expectations in the areas of governance, partnerships, sustainability and outcomes.

Children and Families component

Until 2009 (including the 2009 call for proposal), the Children and Families funding model used an open call for proposals to allocate $8.3 million a year to not-for-profit organizations using grants and contributions.

Since then, the Program has evolved, beginning with the introduction of Social Partnership Pilots to test the concept of social innovation and if a more partnership oriented model could demonstrate more of an impact on vulnerable populations. Leveraging, partnerships and third-party delivery were concepts introduced to the funding models as a means to improve the capacity of not-for-profit organizations to better address social issues facing vulnerable populations. Therefore, three types of Pilots were launched in the fiscal year 2011 to 2012 and were expected to inform the evolution of the program by testing the capacity of not-for-profit organizations of different sizes to leverage non-federal contributions (monetary and/or in-kind) and partners from multiple sectors.

The delivery model of the program has been evolving to focus on increasing impact on vulnerable populations through mandatory leveraging and partnerships. Several of the projects under the Children and Families component are now delivered using a third-party delivery model (intermediaries), where large not-for-profits and charities (for example foundations) leverage non-federal investments and multi-sector partnerships to distribute grants to smaller not-for-profits and charities.

Appendix E - Disability component logic model

ESDC Departmental Strategic Outcome: Income Security, Access to Opportunities and Well-Being for Individuals, Families And Communities
Description immediately follows the chart.
Text description for Appendix E

Appendix E illustrates the logic model of the Social Development Partnerships Program’s Disability component, which outlines from the bottom to the top, its inputs, activities, outputs, immediate outcomes, intermediate outcomes, ultimate outcome and the strategic outcome the program intends to achieve or to which it intends to contribute.

1. Inputs

Input 1.1 Program Budget

Input 1.2 Human Resources

Input 1.3 Information and Technology systems

2. Activities:

Activity 2.1 Administer grant and contribution funds

Activity 2.2 Engage stakeholders

Activity 2.3 Measure program performance and provide analysis

3. Outputs

Activities 2.1, 2.2 and 2.3 are expected to produce the following output:

  • Output 3.1: Projects are funded based on evidence-based identified needs and priorities

4. Immediate Outcomes

Output 3.1 is expected to produce the following immediate outcomes:

  • Immediate Outcome 4.1: Not for profit sector establishes formal partnerships to address barriers to social inclusion faced by people with disabilities
  • Immediate Outcome 4.2: Not for profit sector develops and transfers knowledge and approaches that address barriers to social inclusion faced by people with disabilities

5. Intermediate Outcomes

Immediate Outcome 4.1 is expected to contribute to the following intermediate outcome:

  • Intermediate Outcome 5.1 Not-for-profit sector and partners work collaboratively to address existing and evolving social issues facing people with disabilities

Immediate Outcome 4.2 is expected to contribute to the following intermediate outcome:

  • Intermediate Outcome 5.2 Not-for-profit sector and partners provide access to information, programs and services that promotes inclusion of people with disabilities in society

6. Ultimate Outcome

Intermediate outcome 5.1 and 5.2 are expected to contribute to the following ultimate outcome:

  • Ultimate Outcome 6.1: Social Inclusion of People with Disabilities

7. ESDC Strategic Outcome

This ultimate outcome contributes to the department’s strategic outcome: Income Security, Access to Opportunities and Well-Being for Individuals, Families And Communities

Appendix F - Children and Families Component logic model

ESDC Departmental Strategic Outcome: Income Security, Access to Opportunities and Well-Being for Individuals and Communities
Description immediately follows the chart.
Text description for Appendix F

Appendix F illustrates the logic model of the Social Development Partnerships Program’s Children and Families component, which outlines from the bottom to the top, its inputs, activities, outputs, immediate outcomes, intermediate outcomes, ultimate outcome and the strategic outcome the program intends to achieve or to which it intends to contribute.

1. Inputs

Input 1.1 Program Budget

Input 1.2 Human Resources

2. Activities:

Activity 2.1 Administer grant and contribution funds

Activity 2.2 Engage stakeholders

Activity 2.3 Measure program performance and provide analysis

Activity 2.4 Disseminate knowledge and promote innovation

3. Outputs:

Activities 2.1, 2.2, 2.3 and 2.4 are expected to produce the following outputs:

  • Output 3.1: Projects are funded based on evidence-based identified needs and priorities
  • Output 3.2: Canada’s Volunteer Awards
  • Output 3.3: Analysis of General Social Survey – Giving, Volunteering and Participating

4. Immediate Outcomes

Output 3.1, 3.2 and 3.3 are expected to produce the following immediate outcomes:

  • Immediate Outcome 4.1: Increased resources to communities to address barriers to social inclusion
  • Immediate Outcome 4.2: Increased multi-sector partnerships to address barriers to social inclusion

5. Intermediate Outcomes

Immediate Outcomes 4.1 and 4.2 are expected to contribute to the following intermediate outcome:

  • Intermediate Outcome 5.1: Improved capacity of NFPs to address barriers to social inclusion

6. Ultimate Outcome

Intermediate outcome 5.1 is expected to contribute to the following ultimate outcome:

  • Ultimate Outcome 6.1: Increased Social Inclusion of Children and Families, and Vulnerable Individuals

7. ESDC Strategic Outcome

This ultimate outcome contributes to the department’s strategic outcome: Income Security, Access to Opportunities and Well-Being for Individuals, Families And Communities.

Page details

Date modified: