Evaluation of the Social Development Partnerships Program

Alternate format

Executive summary

This report presents the findings and recommendations from the Evaluation of the Social Development Partnerships Program (referred to hereafter as “the Program”). It is a nationally delivered program that works in partnership with the not-for-profit sector to support people with disabilities, children and families, official language minority communities and other vulnerable populations. The evaluation of the Program covers the period covering the fiscal year 2012 to 2013 to the fiscal year 2016 to 2017, and focuses on:

• the Program’s Disability component, specifically the 2012 Call for Proposals projects
• the Program’s Children and Families component, specifically the Social Partnerships Pilot projects

While both components of the Program fall under the same branch, the distribution of funding and program design fall under the responsibility of two different Ministers and directorates within ESDC.

Background

In the fiscal year 2011 to 2012 (a fiscal year runs from April 1 to March 31), the Program introduced partnerships and leveraging requirements, as well as the third-party delivery model to test whether these features would improve the capacity of not-for-profit organizations to address social issues facing vulnerable populations. These approaches were tested extensively by the Children and Families component of the Program through the Social Partnerships Pilots, and to a lesser extent by the Disability component of the Program.^{Footnote 1} The implementation of these new requirements and delivery model was part of the evolution and renewal of the Program to move away from directed core funding (for the Disability component) and to test whether partnership oriented models could increase the pool of resources brought to bear on a given social issue, and as a result improve the Program’s impacts on vulnerable populations.

Evaluation scope

The evaluation assessed the extent to which the new funding requirements and delivery model increased the capacity of funded organizations to address issues facing vulnerable populations, in comparison to a more traditional project-based funding approach. It also assessed the Program’s relevance and results measurement strategy that was in place during the evaluation period.

Limitations

While the evaluation sought to assess the Program’s contribution toward improving the capacity of funded organizations to address issues facing vulnerable populations, it was limited in its ability to do so given the lack of data collected on beneficiaries. The evaluation did not seek to determine the Program’s attribution toward the social inclusion of vulnerable populations given that the financial support provided by the Program is relatively small, there are numerous government interventions in support of the same goal, and that there are many external factors that could influence social inclusion such as the state of the economy.

Key findings

Relevance

• There is a continued need for the Program to support vulnerable populations through collaboration with, and funding for, not-for-profit organizations.
• Not-for-profit organizations are well positioned to support vulnerable populations due to their connections to communities and understanding of their needs.

Effectiveness

• The Program was successful in helping guide organizations toward meeting their leveraging goals. The majority of funding recipients met the leveraging targets set by the Program. The 17 Disability component funding recipients self-reported having collectively leveraged around $2.8 million worth of cash and in-kind resources from non-federal sources. Similarly, the 18 Social Partnerships Pilot funding recipients leveraged over $8 million.^{Footnote 2} However, the leveraging requirements did not translate into new funding sources or significant amounts of cash contributions for most organizations^{Footnote 3}.
• The Program encouraged organizations to be innovative in establishing new partnerships. However, funded organizations emphasized existing partnerships in the implementation of their projects for several reasons, including a history of collaboration, a mutual understanding of the needs of target populations, and that new partnerships require a lot of time to develop and be effective.
• The third-party delivery model is well-positioned to improve the reach of federal funding to smaller not-for-profit organizations, and to strengthen the capacity of not-for-profit organizations in a way that improves overall service delivery to communities.
• The new requirements and delivery model introduced by the Program increased the capacity of funded organizations to implement their activities during the funding period. However, in most cases the funded organizations reported not having the capacity to continue delivering their activities beyond the funding period.
• The funded organizations that secured large financial commitments from provincial governments were able to continue supporting vulnerable populations through their projects beyond the funding period. There is potential for the Program to explore greater collaboration with potential for effective and successful partnerships between the federal and provincial governments in supporting vulnerable populations.
• The Program’s performance measurement was narrowly focused on partnering and leveraging, as opposed to results/outcomes on vulnerable populations. As a result, the calls for proposals and performance measurement strategy that were in place during the evaluated period did not enable a proper assessment of how the Program and the projects are achieving their stated objectives^{Footnote 4}.
• Both components of the Program have improved their performance measurement strategies since the period covered by the evaluation. Most notably, more data is being systematically collected by the Program on project outcomes through increased collaboration between the Program and funding recipients with the development of evaluation plans.

Other findings

• There are indications that the Program may be broadly contributing to the long-term well-being and social inclusion of vulnerable populations by funding not-for-profit organizations to deliver social development projects. For example, the funded organizations reported many positive results in the implementation of their projects, including greater awareness of barriers to social inclusion within communities and a greater sense of confidence, independence and well-being among participating individuals.

Recommendations

The Evaluation suggests the following 2 recommendations:

1. Ensure that the Program and funding recipients have a common understanding of the Program’s objectives and expected results, as well as the key concepts (for example social inclusion, partnerships, sustainability), in order to enable better performance measurement.
2. Improve monitoring and reporting templates (for example progress and final reports), as well as data collection tools (for example surveys), to capture detailed information on how the Program is impacting its intended beneficiaries.

Management response and action plan

Introduction

The Social Development Partnerships Program is a Grant and Contribution program that supports investment in not-for-profit organizations to help improve life outcomes for people with disabilities, children and families, and other vulnerable populations. The program has a $20 million annual budget divided in two components: Disability ($11 million) and Children and Families ($9.1 million).

The Program is aligned with ESDC’s vision with respect to individuals and families, namely to “build a stronger and more inclusive Canada, support Canadians in making choices that help them live productive and rewarding lives, and improve Canadians quality of life”. The Program supports ESDC’s core responsibility for social development which seeks to increase inclusion and opportunities for the participation of Canadians in their communities.

The Social Development Partnerships Program supports Government of Canada priorities through investment in not-for-profit organizations aiming to improve the life outcomes for people with disabilities, children and families, and other vulnerable populations. The use of Grant and Contribution investments represents a flexible and cost-effective way to support the role that communities, not-for-profit and voluntary sector organizations play in helping people with disabilities, families and children, and other vulnerable populations to be resilient and to provide them with the tools and skills to respond to current and emerging social issues.

The key findings presented in the evaluation indicate that the Program is making progress towards achieving its intended objectives and outcomes. The evaluation report includes two recommendations and Management supports both. The Management Response and Action Plan will be implemented by the Income Security and Social Development Branch in collaboration with the Program Operations Branch and relevant stakeholders where appropriate. In addition, the Income Security and Social Development Branch will continue to lead improvements to the Program Performance Information Profile.

It is important to note that the Program consistently endeavours to consult with external partners, including funded recipients on balancing the desire and need for robust performance measurement information, cost-effectiveness and efficiency.

Recommendation 1: Ensure that the Program and funding recipients have a common understanding of the Program’s objectives and expected results, as well as the key concepts (for example social inclusion, partnerships and sustainability), in order to enable better performance measurement.

Response

The program is already striving to address this recommendation. For example, funded recipients are required to collect relevant performance data and to report on the results of their projects. As such, a comprehensive communication approach to ensure that these recipients are well informed of Program objectives, expected results and key priorities will improve the quality of results reported by them and enhance their projects’ alignment with the Program’s goals and priorities. In addition, a common understanding of key concepts by both Program and funded recipients will enhance the quality of performance measurement.

More specifically, in the recent Social Development Partnerships Program Children and Families component Call for Concepts (focusing on Early Learning and Child Care), the objectives of the Program and those of the Call itself were clearly outlined in the applicant guide and the application template. Key terms and concepts were defined in the applicant guide. In their concept application, applicants were required to demonstrate the link between the goals and the expected outcomes of their proposed concepts and the objectives of the Children and Families component. A component of the assessment was on how well their proposed concepts aligned with the program objectives and priorities.

With respect to the Social Development Partnerships Program Disability component, and as part of the renewal of its operating funding to national organizations, launched in 2017, the Department consulted with the disability community through a Steering Committee to provide insight on principles and priorities that would contribute to the development of a program Performance and Accountability Framework. Under the renewed Disability component, recipients will be required to report on results achieved using a minimum set of baseline performance indicators. The Performance and Accountability Framework identified four principles to which performance indicators will be developed to measure the results of the Disability component operating funding to contribute to the health and maturity of the disability sector and ultimately to advance the social inclusion of Canadians living with disabilities. The performance indicators will measure organizational capacity in the following broad areas:

• governance and accountability
• effective leadership and operations
• developing and maintaining partnerships
• measurable impact

Management action plan

• The Program will work with relevant stakeholders, including internal partners, to develop common definitions for the Program’s key terms and priorities such as social inclusion, sustainability, leveraging and partnership. The definitions will then be communicated clearly with all funding applicants to enhance their understanding of the Program’s expectations
  • Completion date: Spring 2019
• The Program will continue to develop and improve tools and products, making them widely available, to ensure that funded recipients are well informed of the Program’s objectives, performance indicators, expected results and key priorities.
  • Completion date: Fall 2019

Recommendation 2: Improve monitoring and reporting templates (for example progress and final reports), as well as data collection tools (for example surveys), to capture detailed information on how the Program is impacting its intended beneficiaries.

Response

The ability to capture detailed information on how the Program is impacting its intended beneficiaries is essential in demonstrating efficiency in the management of the Social Development Partnerships Program. The detailed information on the Program’s impact will also inform relevant direction to program policy, program design and service delivery.

Since the implementation of the Children and Families component innovation pilots (from 2012 to 2015), the component has engaged in ongoing revisions to the final reporting template that funded recipients use to submit their final reports upon completion of their projects. These revisions include:

• customized final reporting templates for each Call for Concepts/Proposals
• request for information on specific project outcomes, reach of project and its impact on project beneficiaries
• additional requirements for information, such as gender and target populations to enable Gender-based Analysis Plus analysis on project beneficiaries

The Children and Families component communicates performance measurement expectations to funding recipients throughout the application and funding processes. This includes discussions on performance measurement expectations at the start of the planning and designing of each Call for Concept/Proposal, through the development of the application guide, the assessment of concepts/proposals and the development of the terms of agreements for funded projects. Solicited Children and Families component project proposals are also reviewed from a performance measurement perspective prior to the assessment and approval of potential applications for funding. Projects are monitored during their implementation in order to ensure that evaluation plans including relevant result measurement indicators are developed and reported on.

To capture detailed information on the impact Disability component funding will have on its intended beneficiaries for projects from April 1, 2017 to March 31, 2018, the Performance Report Indicator Template for project funding was revised to include additional questions pertaining to partnerships, gender-based analysis, evaluation, success stories and lessons learned. For future performance reporting periods, the Department will consider whether further changes to the template are required.

Furthermore, the Program will continue to consider how the current data collection strategy can be improved, by revisiting the template with an aim to make it more concise, and ensure the continued applicability of the data collected to be in line with the performance measurement strategy.

Management action plan

• In collaboration with funded recipients, the Program will continue to improve the monitoring and reporting templates to enhance the quality of data collected on how the Program is impacting its intended beneficiaries.
  • Completion Date: Fall 2019
• To further improve the capacity of funded recipients to conduct relevant performance measurement of their projects, the Program will ensure funding is dedicated and support is available for the development of evaluation plans, including data collection, data analysis and reporting.
  • Completion Date: Fall 2019

1. Introduction

This report presents the findings from the Evaluation of the Social Development Partnerships Program covering the period starting from the fiscal year 2012 to 2013 to the fiscal year 2016 to 2017. It builds on four technical reports, namely, a file review, document review, key informant interviews with internal and external stakeholders, and case studies with selected funded projects by triangulating their findings^{Footnote 5}.

1.1 Program description

The Social Development Partnerships Program is a nationally delivered program that makes strategic investments to support government priorities related to people with disabilities, children and families, the voluntary sector, official language minority communities and other vulnerable populations. It works in partnership with not-for-profit organizations to help improve life outcomes of these target groups.

The Program supports Employment and Social Development Canada’s (ESDC) strategic outcome related to Income security, access to opportunities and well-being for individuals, families and communities.^{Footnote 6}

The annual Grants and Contributions budget of the Program is over $20 million. The annual budget is divided among two components, where $11 million is allocated to the Disability component and slightly over $9 million is allocated to the Children and Families component. The current logic models for the Disability, and Children and Families components are provided in Appendix E and Appendix F respectively^{Footnote 7}.

The two program components are described in greater detail below.

Disability component

The Disability component is overseen by the Office for Disability Issues within ESDC’s Income Security and Social Development Branch, and delivered by the Program Operations Branch within Service Canada. It supports projects intended to improve the participation and inclusion of people with disabilities in all aspects of Canadian society. The Minister of Public Services and Procurement and Accessibility is accountable for the Disability component.

The Disability component provides both operating and project funding. Operating funding is provided to increase the capacity of not-for-profit organizations with a primary mandate to promote the social inclusion of people with disabilities and to affect change among these populations. Project funding is provided to support a wide range of initiatives that address social issues and barriers that confront people with disabilities; develop tools and resources that address the needs of individuals, their families and communities; support services for people with disabilities; and foster partnerships and collaborative networks.

Children and Families component

The Children and Families component is overseen by the Community Development and Homelessness Partnerships Directorate within ESDC’s Income Security and Social Development Branch, and delivered by the Program Operations Branch within Service Canada. The component provides funding to eligible not-for-profit organizations for projects that help to improve the lives of children and families and other vulnerable populations. The Minister of Families, Children and Social Development is accountable for the Children and Families component.

There are three funding allocations within the Children and Families component that are integrated in the Terms and Conditions:

• The Social Partnerships Initiative in Official Language Minority Communities (OLMC), which is part of the Roadmap for Canada’s Linguistic Duality, 2013–2018: Education, Immigration, Communities. The funded projects focus on youth in transition, changing demographics (seniors) and vulnerable populations in official language minority communities.
• Canada's Volunteer Awards, which recognizes the enormous contributions that volunteers, not-for-profit organizations and businesses across the country make to help people and their communities.
• The analysis and dissemination of the General Social Survey – Giving, Volunteering and Participation. This is an annual commitment for the Children and Families component to fund a single organization to analyze and disseminate results from the survey to not-for-profit organizations across Canada.

2. Evaluation scope and objective

The evaluation looks at the projects funded under the 2012 Call for Proposals for the Disability component, as well as the Social Partnerships Pilots under the Children and Families component. These projects were initiated and completed between the fiscal year 2012 to 2013 and the fiscal year 2016 to 2017.

The scope of the evaluation was developed in consultation of the Program and was approved at the June 2017 Performance Measurement and Evaluation Committee meeting. The following is an overview of the initiatives that were included or excluded from the evaluation. The initiatives that were excluded may be part of the scope of a future evaluation of the Program.

Children and Families component

• Covered: Social Partnerships Pilots^{Footnote 8}.
• Not covered: Children and Families (ongoing projects), Canada’s Volunteer Awards, General Social Survey on Giving, Volunteering and Participating, and the Social Partnerships Initiative in Official Language Minority Communities.

Disability component

• Covered: 2012 Disability Call for Proposal^{Footnote 9}.
• Not covered: Directed/core-funding of the national disability organizations and provincial/territorial organizations, 2016 Call for Proposal on Innovation, and the 2016 Call for Proposal on the Federal Accessibility Legislation.

The evaluation focuses on partnerships and leveraging requirements, as well as the third-party delivery model^{Footnote 10}, which were introduced by the Program in 2012. The aim of the evaluation is to assess whether these approaches and delivery model improved the capacity of the funded organizations to address social issues facing vulnerable populations. The evaluation also examines issues related to the program design and delivery, and explores the program’s result measurement strategies.

It should be noted that these requirements and delivery model were more extensively tested by the Children and Families component than the Disability component. This is due to the fact that the Social Partnerships Pilots were part of a broader initiative within the Department to test innovative approaches^{Footnote 11} to address complex social issues. While the 2012 Disability component Call for Proposals was not established for the specific purpose of testing these approaches, it did incorporate mandatory leveraging as one of the Call’s requirements.

3. Key findings

3.1 Relevance

Need for the program

The file review and case studies indicate that the Program has, through the projects it funds, identified and addressed various needs, gaps, and social issues facing vulnerable populations, including barriers to social inclusion, during the funding period.^{Footnote 12}

While funding recipients that were interviewed described needs as having been addressed during the course of the projects, many also expressed that these needs were ongoing and remained in the communities that were targeted by the projects. Therefore, not-for-profit organizations require continued support from the Program to address various needs, gaps, and social issues facing vulnerable populations.

Need to work with not-for-profit organizations

Findings from the evaluation suggest that working with not-for-profit organizations is a good approach to tackling social issues faced by vulnerable populations. For example, key informants^{Footnote 13} indicated that not-for-profit organizations have the proper connections and relationships necessary to serve vulnerable populations due to their operational knowledge and long history of working directly with, and delivering services to, vulnerable populations.^{Footnote 14}

In general, key informants^{Footnote 15} perceive the federal government’s role as an enabler in helping not-for-profit organizations deliver services to vulnerable populations. The document review indicates the complementary nature between not-for-profit organizations and governments in service delivery to vulnerable populations.^{Footnote 16} The former offers advantages, such as small scale and diversity of services, while the latter compensates the not-for-profit organizations’ inability to generate an adequate scale of resources by generating revenue.

However, the document review and case studies also highlight an ongoing challenge for not-for-profit organizations, which is the lack of financial capacity brought by decreasing revenues from government sources. This affects their capacity to deliver services to vulnerable populations, especially in an environment where demand for such services is increasing.^{Footnote 17}

Alignment with federal policies and priorities, and departmental strategic outcomes

The Program’s objectives and the strategies it uses to achieve its goals are aligned with the Government of Canada and the Department’s push for innovation to address social issues.^{Footnote 18} The document review highlights both the federal government’s and the Department’s commitment to solving public policy challenges through innovative approaches, which includes working with multi-sectoral partners to serve populations in need.

Appropriateness of federal roles and responsibilities in delivering the program

The document review indicates that the Government of Canada has legal roles and responsibilities to ensure the social inclusion of vulnerable populations.^{Footnote 19} It also indicates that using an ‘all of Canada’ approach to responding to social inclusion may raise intergovernmental issues due to the fact that social inclusion interacts with many areas for which provinces are constitutionally responsible, such as poverty, housing, education and health care.^{Footnote 20}

However, the federal government is not aiming to substitute from the responsibilities of the provinces in these areas. Rather, it aims to provide complementary support, for instance by fostering an environment that is supportive of social development projects. This is the case of the Social Development Partnerships Program, which encourages not-for-profit organizations to form multi-sectoral partnerships in an effort to promote innovative ways to address barriers to social inclusion facing vulnerable populations.

Similarities with other programs

There are several programs across the Government of Canada that support vulnerable populations.^{Footnote 21} However, the document review found no other currently existing government program that simultaneously supports social not-for-profit organizations and embeds the creation of cross-sector partnerships, which is integral to the Program’s delivery model.

3.2 Effectiveness

Expected results of the approaches and delivery model

In the fiscal year 2011 to 2012, the Program tested partnerships and leveraging requirements, as well as the third-party delivery model, to better support not-for-profit organizations to address social issues facing vulnerable populations. Figure 1 illustrates these different funding approaches.

Source: ESDC Evaluation Directorate Staff (2018)

Text description of figure 1

Figure 1 illustrates the different delivery methods and funding approaches used by the Social Development Partnerships Program during the funding period. The figure is divided into three main sections: conventional project funding, project funding with partnerships and leveraging requirements, and the third-party delivery model. All three sections describe a chain of events, using the Social Development Partnerships Program as a starting point. The different players are portrayed as circles. Arrow symbols link circles to each other, and represent a sequence and/or action.

The first section describes conventional project funding. Using the Program as a starting point, it illustrates the following sequence. The Program allocates project-based or core-funding to select not-for-profit organizations, and these organizations provide services to vulnerable populations.

The second section describes project funding with partnerships and leveraging requirements. Using the Program as a starting point, it illustrates the following sequence. The program allocates project-based funding to select not-for-profit organizations. These organizations enlist the help of partners and donors, which provide them with resources. The funded organizations then provide services to vulnerable populations. The circle representing the vulnerable populations is portrayed as larger than in the first section. This is to illustrate the expectation that funded not-for-profit organizations are able to reach more vulnerable populations using the resources provided by partners and donors.

The final section describes the third-party delivery model. Using the Program as a starting point, it illustrates the following sequence. The program allocates large project-based funding to a select number of large not-for-profit organizations. These large organizations then distribute grants to smaller not-for-profit organizations. Finally, each of these smaller not-for-profit organizations provide services to vulnerable populations. As in the previous section, the circle representing vulnerable populations is portrayed as larger than in the first section to illustrate greater reach through this approach.

These funding approaches were part of a broader initiative within the Department to test innovative approaches to respond to complex social issues. In the case of the Social Development Partnerships Program, the purpose of the testing was to inform the feasibility of using federal grants and contributions to leverage additional non-federal investments in mutually beneficial, locally-based solutions to complex social issues of importance to communities and the Government.

A review of the relevant Program documents suggests that there were three main results/outcomes that the Program was expecting to achieve with the partnerships, leveraging requirements, and the third-party deliver model. Notably, it was expected that the funding recipients would:

1. engage new partners, or deepen relationships with existing ones, in a way that improves community outcomes (in other words for vulnerable populations);
2. diversify their sources of funding, thereby increasing the pool of resources brought to bear on any given issue for greater impact; and
3. achieve greater sustainability through partnerships and leveraging of new resources.

Overall, the evaluation found that the indicators that were developed by the Program to inform progress toward these results/outcomes were more focused on the funded organizations’ activities (for example partnerships formed, amount of resources leveraged) than outcomes on beneficiaries, which are more closely tied with Program objectives. In order to verify whether partnerships, leveraging requirements, and the third-party delivery model truly improved the organizations’ ability to address social issues facing vulnerable populations, data would have needed to be collected on the beneficiaries.

Outcomes of the approaches and delivery model

Partnerships

The program funding requirements put an emphasis on forming multi-sectoral partnerships with new partners, or strengthening relationships with existing ones, during the funding period^{Footnote 22}.

The file review and case studies indicate that the funding recipients^{Footnote 23} mostly formed partnerships with existing partners, who often had a more formal and central role in the execution of the funded projects (for example active members in project advisory committees). Many funding recipients brought existing partners to the projects because of their long history of working together and their mutual understanding of the needs of their target populations. These well-established relationships were considered an important source of success in conducting the projects.

According to the file review and key informant interviews, factors such as mutual trust and interest were important in ensuring strong and sustainable partnerships. In contrast, many funding recipients indicated that newer partnerships required more time to be effective. While new partnerships were also formed, those new partnerships were often informal and sometimes only limited to the project period.

While the above findings suggest that the existing partners played a greater role than the new ones, it is difficult to assess the overall landscape of partnerships due to the fact that partnerships were defined in a way that covered all sorts of contributors^{Footnote 24} (for example a one-time donor, regular/ongoing donors). It was also difficult to assess because the reporting requirements did not ask for additional distinctions, such as whether a partner is existing, new , formal, or informal. By further defining the concept of partnerships and communicating it to the funding applicants, the Program could garner a better understanding of the effectiveness of the partnerships approach.

While the above findings suggest that the existing partners played a greater role than the new ones, it is difficult to assess the overall landscape of partnerships due to the fact that partnerships were defined in a way that covered all sorts of contributors (for example a one-time donor, regular/ongoing donors). It was also difficult to assess because the reporting requirements did not ask for additional distinctions, such as whether a partner is existing, new^{Footnote 25}, formal, or informal. By further defining the concept of partnerships and communicating it to the funding applicants, the Program could garner a better understanding of the effectiveness of the partnerships approach.

Achievement of leveraging requirements

The file review showed that all of the Disability and Children and Families funding recipients leveraged cash and/or in-kind contributions throughout the course of the projects. Leveraged amounts were self-reported by the organizations, and in most cases, no further verification was conducted to confirm whether these amounts were received.

As shown in Table 1, all of the Disability funding recipients exceeded the basic requirement to secure at least 10% of total eligible project costs from non-federal government sources. In fact, on average, 31% of total eligible project costs were leveraged from non-federal sources.

Source: ESDC Evaluation Directorate Staff (2018)

4. Areas for consideration

The findings from the Evaluation indicate that the Program did not employ an adequate framework to measure progress toward its stated goals. In particular, the Evaluation noted that the calls for proposals did not explicitly define overarching expected results/outcomes, aligned with the program’s broader outcomes, for the funded organizations to report on. This reduced the possibility that the funding recipients could report on the appropriate results.

The potential funded organizations share responsibility in measuring the performance of the Program, given that they are the main vehicle through which it is delivered. The funded organizations are expected to properly identify and define the issues they are addressing, and determine the expected results of their own initiatives. In turn, this will ensure that both the Program and the funded organizations are facilitating the measurement of outcomes and impacts related to vulnerable populations.

The evaluation recognizes that any additional responsibilities required on behalf of the funded organizations with regard to data collection and reporting may result in more time and resources having to be invested in such activities, which may take away from project delivery. The challenge of balancing reporting requirements with project delivery has been noted in the different lines of evidence and Program documents. Therefore, additional support and resources could be provided by the Program to improve the evaluation capacity of not-for-profit organizations, if necessary.

5. Recommendations

The Evaluation suggests the following 2 recommendations:

1. Ensure that the Program and funding recipients have a common understanding of the Program’s objectives and expected results, as well as the key concepts (for example social inclusion, partnerships, sustainability), in order to enable better performance measurement.
2. Improve monitoring and reporting templates (for example progress and final reports), as well as data collection tools (for example surveys), to capture detailed information on how the Program is impacting its intended beneficiaries.

Appendix A – Evaluation questions

Issues, evaluation questions, indicators and line of evidence

Relevance:

1. Is there a continued need for the program to seek new ways to address social issues faced by children and families, and people with disabilities?

1.1 Evidence that current gaps and/or needs are addressed by not-for-profit organizations

• Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

1.2 Evidence that working with not-for-profit organizations help to adequately support vulnerable populations

• Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

1.3 Evidence on current gaps identified that are not addressed or covered by the program

• Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

2. Does the program align with and contribute to the Government of Canada policies and priorities, and ESDC

2.1 Evidence of consistency between the program priorities and the department strategic outcomes as well as the federal government policies and priorities strategic outcomes?

• Lines of evidence: Document review

3. Are the current roles and responsibilities of the federal government appropriate in delivering the program?

3.1 Evidence that the program activities are aligned with federal roles and responsibilities in regards to social inclusion

• Lines of evidence: Document review, key informant interviews (internal)

3.2 Evidence that similar programs or services are not offered within or outside the federal government

• Lines of evidence: Document review, file review, key informant interviews (internal), key informant interviews (external), case studies

Effectiveness:

4. Was an appropriate framework in place to facilitate results measurement of the Social Partnerships Pilots of the Children and Families component as well as the funded projects of the 2012 Disability component call for proposal? What could be improved for the future?

4.1 Evidence of result measurement strategy as part of the call for proposals that clearly identifies the expected results and assessments criteria to support the funding decisions

• Lines of evidence: Document review, key informant interviews (internal), case studies

4.2 Percentage of projects who indicated having developed and used outcome performance indicators and evaluation processes to measure project results

• Lines of evidence: File review

4.3 Evidence on the evaluation methods and/or measurement tools to evaluate the project results and progress towards contributing to the program’s expected outcomes

• Lines of evidence: Document review, file review, key informant interviews (internal), case studies

4.4 Evidence on how funded organizations and the program may enhance their capacity in regards to social impacts measurement in the context of social innovation and financing

• Lines of evidence: Document review, key informant interviews (internal), key informant interviews (external), case studies

5. Did these models improve the capacity of funded organizations to address barriers to social inclusion?

5.1 Evidence as to whether these new socially innovative models (in other words, leveraging, partnerships and third party model) were able to better serve the funded organizations in enhancing their ability/capacity to impact vulnerable populations in addressing barriers and challenges to social inclusion

• Lines of evidence: Document review, key informant interviews (internal), key informant interviews (external), case studies

Disability component only questions

6. To what extent have partnership requirements among funded organizations under the 2012 Disability call for proposal been successful in:

6.1 Attracting partners to funded projects?

6.1.1 Number of partnerships created or strengthened, by sector (for example, private, public, not-for-profit, etc.)

• Lines of evidence: File review

6.1.2 Evidence on successes, challenges and level of efforts in attracting and retaining partnerships (for example, collaborations, alliances, networks, etc.)

• Lines of evidence: File review, key informant interviews (external), case studies

6.1.3 Approaches adopted to attract partnerships

• Lines of evidence: File review, key informant interviews (external), case studies

6.2 Maximizing partners’ contributions to funded projects?

6.2.1 Evidence that partners contributed to improving organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional partnerships; replicating and scaling project

• Lines of evidence: File review, key informant interviews (external), case studies

6.3 Developing successful and sustainable partnerships (what made the partnerships more successful and sustainable)?

6.3.1 Evidence that partnerships created or strengthened were effective and/or continued even after the funded projects ended (for example, types of collaborations, roles or contributions of partners)

• Lines of evidence: File review, key informant interviews (external), case studies

6.4 Ensuring more inclusivity of organizations and people with disabilities?

6.4.1 Evidence on the types and complexities of social issues and people with disabilities that were addressed and/or reached by the organizations and through their partnerships

• Lines of evidence: File review, key informant interviews (external), case studies

6.4.2 Evidence that partnerships enhanced the capacity of organizations in ensuring greater inclusivity of disability organizations and in terms of better outcomes for more people with various types of disabilities

• Lines of evidence: Key informant interviews (external), case studies

6.5 Expanding the reach of funded organizations?

6.5.1 Evidence that partnerships led to greater reach for the funded organizations in terms of dealing with more social issues, serving more people with disabilities, regions across Canada, official languages, etc.

• Lines of evidence: File review, key informant interviews (external), case studies

7. To what extent have leveraging requirements among the organizations funded under the 2012 Disability call for proposal been successful in:

7.1 Leveraging investments from non-federal organizations?

7.1.1 Percentage of funded recipients that leveraged cash and/or in-kind contributions from non-federal sources

• Lines of evidence: File review

7.1.2 Percentage of total cash and/or in-kind contributions leveraged from non-federal sources for each type of contributor (for example, public, private, not-for-profit, etc.)

• Lines of evidence: File review

7.1.3 Amount of leveraged cash and/or in-kind contributions from non-federal sources for every dollar invested by the program (ratio)

• Lines of evidence: File review

7.1.4 Approaches adopted to leverage cash and/or in-kind investments by non-federal sources

• Lines of evidence: File review, key informant interviews (external), case studies

7.1.5 Evidence on successes, challenges and level of efforts in leveraging of cash and/or in-kind contributions from non-federal partners

• Lines of evidence: File review, key informant interviews (external), case studies

7.2 Maximizing expertise in the funded projects?

7.2.1 Evidence on the use of cash and/or in-kind contributions leveraged in the funded projects by category (for example, administrative/professional services, volunteer time, office space, evaluation, etc.)

• Lines of evidence: File review, key informant interviews (external), case studies

7.2.2 Evidence that cash and/or in-kind investments from non-federal sources contributed to improving organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional investments; replicating and scaling project

• Lines of evidence: File review, key informant interviews (external), case studies

7.3 Ensuring more inclusivity of organizations and people with disabilities?

7.3.1 Evidence that cash and/or in-kind contributions from non-federal sources enhanced the capacity of organizations in ensuring greater inclusivity of disability organizations and in terms of better outcomes for more people with various types of disabilities

• Lines of evidence: Key informant interviews (external), case studies

7.4 Expanding the reach of the funded organizations?

7.4.1 Evidence that cash and/or in-kind contributions from non-federal sources led to greater reach for the funded organizations in terms of dealing with more social issues, serving people with disabilities, regions across Canada, official languages, etc.

• Lines of evidence: Key informant interviews (external), case studies

7.5 Sustaining both monetary and in-kind investments from non-federal organizations?

7.5.1 Evidence that cash and/or in-kind contributions from non-federal sources continued even after the projects ended (for example, nature, scope and/or use of contributions, etc.)

• Lines of evidence: Key informant interviews (external), case studies

8. Has the 2012 Disability call for proposal been successful in creating and transferring knowledge and approaches that address barriers to social inclusion faced by people with disabilities?

8.1 Number of knowledge and/or approaches developed by type (for example, programs, services, resources, tools, pilot projects, best practices, training guides, innovative solutions, etc.)

• Lines of evidence: File review, key informant interviews (external), case studies

8.2 Evidence that the knowledge and approaches developed were shared and disseminated to the target population and other stakeholders, including partners

• Lines of evidence: File review, key informant interviews (external), case studies

8.3 Percentage of the knowledge and approaches developed that reached the target audience (for example, people with disabilities, organizations, municipalities, employers, clinicians, etc.)

• Lines of evidence: File review, key informant interviews (external), case studies

8.4 Percentage of funded projects that have developed tools and approaches that are scalable, transferable and sustainable

• Lines of evidence: File review, key informant interviews (external), case studies

Children and Families component only questions

9. To what extent were requirements to leverage support from other organizations successful in:

9.1 Leveraging monetary and in-kind investments from non-federal organizations?

9.1.1 Percentage of funded recipients who were able to reach their leveraging targets through cash and/or in-kind contributions from non-federal sources

• Lines of evidence: File review

9.1.2 Percentage of total cash and/or in-kind contributions leveraged from non-federal sources for each type of contributor (for example, public, private, not-for-profit, etc.)

• Lines of evidence: File review

9.1.3 Amount of leveraged cash and/or in-kind contributions from non-federal sources for every dollar invested by the program (ratio)

• Lines of evidence: File review

9.1.4 Approaches adopted to leverage cash and/or in-kind investments by non-federal sources

• Lines of evidence: File review, key informant interviews (external), case studies

9.1.5 Evidence on successes, challenges and level of effort in leveraging of cash and/or in-kind contributions from non-federal sources

• Lines of evidence: File review, key informant interviews (external), case studies

9.2 Enhancing the contribution of both monetary and in-kind investments towards achieving projects’ intended outcomes?

9.2.1 Evidence on the use of cash and/or in-kind contributions leveraged in the funded projects by category (for example, administrative/professional services, volunteer time, office space, evaluation, etc.)

• Lines of evidence: File review, key informant interviews (external), case studies

9.2.2 Evidence that cash and/or in-kind contributions from non-federal sources improved organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional investments; replicating and scaling project

• Lines of evidence: File review, key informant interviews (external), case studies

9.3 Sustaining both monetary and in-kind investments from non-federal organizations?

9.3.1 Evidence that cash and/or in-kind contributions from non-federal sources continued even after the projects ended (for example, nature, scope and/or use of contributions, etc.)

• Lines of evidence: File review, key informant interviews (external), case studies

9.4 Expanding the reach of funded organizations?

9.4.1 Evidence that cash and/or in-kind contributions from non-federal sources led to greater reach for the funded organizations (for example, social issues, vulnerable populations, regions across Canada, official languages, etc.)

• Lines of evidence: Key informant interviews (external), case studies

10. To what extent were partnership requirements, as per funding agreements, successful in:

10.1 Attracting new partners to contribute to funded projects?

10.1.1 Number of partnerships created or strengthened, by sector (for example, private, public, not-for-profit, etc.)

• Lines of evidence: File review

10.1.2 Evidence on successes, challenges and level of efforts in attracting and retaining partnerships

• Lines of evidence: File review, key informant interviews (external), case studies

10.1.3 Approaches adopted to attract partnerships (for example, collaborations, alliances, networks, etc.)

• Lines of evidence: File review, key informant interviews (external), case studies

10.2 Maximizing partners’ contributions toward the achievement of project intended outcomes?

10.2.1 Evidence that partners contributed to improving organizational capacity in terms of: increased project awareness in communities/facilitated connections to communities; increased resources; filled gap in services/needs; new ideas/approaches; leveraging additional partnerships; replicating and scaling project

• Lines of evidence: File review, key informant interviews (external), case studies

10.3 Developing successful and sustainable partnerships?

10.3.1 Evidence that partnerships created or strengthened were effective and/or continued even after the funded projects ended (for example, types of collaborations, roles or contributions of partners)

• Lines of evidence: File review, key informant interviews (external), case studies

11. To what extent did the third party delivery (intermediary) approach:

11.1 Result in leveraging from non-federal organizations?

11.1.1 Percentage of funded recipients who were able to reach their leveraging targets through cash and/or in-kind contributions from non-federal sources

• Lines of evidence: File review

11.1.2 Percentage of total cash and/or in-kind contributions by non-federal sources leveraged from each type of contributor (for example, public, private, not-for-profit, etc.)

• Lines of evidence: File review

11.1.3 Amount of leveraged cash and/or in-kind contributions from non-federal sources for every dollar invested by the program (ratio)

• Lines of evidence: File review

11.1.4 Approaches adopted to leverage cash and/or in-kind contributions from non-federal sources

• Lines of evidence: File review, key informant interviews (external), case studies

11.1.5 Evidence on successes, challenges and level of efforts in leveraging of cash and/or in-kind contributions from non-federal partners

• Lines of evidence: File review, key informant interviews (external), case studies

11.1.6 Evidence that the third party (intermediary) approach can leverage more easily and effectively pre-existent resources – in terms of partners, contributions, expertise, etc. – towards addressing different types of social issues faced by vulnerable populations and communities

• Lines of evidence: Key informant interviews (external), case studies

11.2 Develop successful and sustainable partnerships?

11.2.1 Evidence that partnerships created or strengthened were effective and/or continued even after the funded projects ended (for example, types of collaborations, roles or contributions of partners)

• Lines of evidence: File review, key informant interviews (external), case studies

11.2.2 Establishment of new networks and expansion of these networks beyond the length of the project

• Lines of evidence: File review, key informant interviews (external), case studies

11.3 Expand the reach of funded organizations?

11.3.1 Evidence that the third party (intermediary) approach led to greater reach for the funded organizations (for example, social issues, vulnerable populations, regions across Canada, official languages, etc.)

• Lines of evidence: Key informant interviews (external), case studies

11.4 Influence system change?

11.4.1 Evidence that the third party (intermediary) approach may have a better understanding of social issues and challenges faced by vulnerable populations and communities

• Lines of evidence: Key informant interviews (internal), key informant interviews (external), case studies

11.4.2 Evidence that the third party delivery (intermediary) approach is better positioned to increase community and/or organizational capacity towards addressing different types of social issues and challenges faced by vulnerable populations and communities

• Lines of evidence: Key informant interviews (internal), key informant interviews (external), case studies

11.4.3 Evidence that the third party delivery (intermediary) approach improved or expanded services to vulnerable populations, thus increasing the impacts of federal spending

• Lines of evidence: Key informant interviews (internal), key informant interviews (external), case studies

Appendix B - Data collection methods and limitations

In total, 4 lines of evidence were conducted as part of the evaluation:

• a file review
• a document review
• key informant interviews with internal and external stakeholders
• case studies

A technical report was produced for each line of evidence.

The Evaluation Directorate conducted the file review, document review and part of the key informant interviews in house, while the case studies and key informant interviews (external) were conducted by a consulting firm. All 4 reports were finalized in consultation with the Program areas. The final report was also peer reviewed by two academics with expertise relating to the themes addressed in the evaluation.

The evaluation sought to assess the Program’s contribution toward improving the capacity of funded organizations to address barriers to social inclusion, but was limited in its ability to do so given the lack of data collected on beneficiaries. The evaluation did not seek to determine the Program’s attribution toward the social inclusion of vulnerable populations given that the financial support provided by the Program is relatively small, there are numerous government interventions in support of the same goal, and that there are many external factors that could influence social inclusion such as the state of the economy.

The following limitations are noted with respect to evaluation data and their collection methods:

• There was an overall lack of adequate data to respond to the various evaluation questions and lines of evidence. Notable challenges include:
  • The information contained in the project files^{Footnote 46} offered limited insight on the link between leveraging and partnering indicators and the Program’s intended outcomes and impacts.
  • Insufficient number of Program officials interviewed^{Footnote 47} had sufficient knowledge of the funded projects and the surrounding processes covered during the evaluation period (for example intake procedures, performance measurement strategies). This is likely due to staff turnover within the Program.
  • Insufficient number of interviews^{Footnote 48} with project representatives and partners that could provide detailed information on the funded projects. This is likely due to the time elapsed since the period covered by the evaluation, staff turnover within organizations, and limited knowledge of funded projects.
• For the case studies, a small sample of organizations^{Footnote 49} was selected purposefully by the Evaluation Directorate in collaboration with the Program. The sample size should not be considered representative of the broader population. This is also true for the external key informant interviews, which were selected from the case study organizations. To address this limitation, findings were triangulated with other lines of evidence. Notably, the file review covered historic files from all 37 projects.
• The value of the in-kind contributions leveraged is not standardized (in other words two organizations may value the same in-kind contribution differently), and therefore leveraging data should be interpreted with caution. Additionally, leveraging information for the Disability component was not compiled or tracked in a way that ensures reliable data^{Footnote 50}.
• The lack of data at the population level during the period covered severely limits the ability of the Evaluation to conduct a retrospective Gender-based Analysis Plus. However, according to Program officials, Gender-based Analysis Plus is being integrated into the Program reporting.

Appendix C – List of technical reports

ESDC (2019). The Evaluation of the Social Development Partnerships Program: Document Review Technical Report. (Not published, but available on demand.)

ESDC (2019). The Evaluation of the Social Development Partnerships Program: File Review Technical Report. (Not published, but available on demand.)

ESDC (2019). The Evaluation of the Social Development Partnerships Program: Key Informant Interview Technical Report. (Not published, but available on demand.)

ESDC (2019). The Evaluation of the Social Development Partnerships Program: Case Study Technical Report. (Not published, but available on demand.)

Appendix D - Program evolution, renewal and activities

Disability component

The funding model of the Disability component consists of an operating funding allocation of $6 million per year to national disability organizations; $2 million per year to regional organizations under the Community Inclusion Initiative; and an allocation of $3 million per year for unsolicited projects and funding through call for proposals.

In 2012, an open Call for Proposals was launched which introduced the requirements to leverage funding from non-federal government sources (mandatory 10% leveraging requirement) along with the strengthening of a results orientation and outcome reporting. A total of seventeen projects received funding. These projects ended on March 31, 2016.

In spring 2016, the Minister of Sport and Persons with Disabilities approved the renewal of the Disability component of the program with the goal of designing a Performance and Accountability Framework in collaboration with the disability community, to ensure fairness and transparency in the distribution of funding. The framework sets expectations in the areas of governance, partnerships, sustainability and outcomes.

Children and Families component

Until 2009 (including the 2009 call for proposal), the Children and Families funding model used an open call for proposals to allocate $8.3 million a year to not-for-profit organizations using grants and contributions.

Since then, the Program has evolved, beginning with the introduction of Social Partnership Pilots to test the concept of social innovation and if a more partnership oriented model could demonstrate more of an impact on vulnerable populations. Leveraging, partnerships and third-party delivery were concepts introduced to the funding models as a means to improve the capacity of not-for-profit organizations to better address social issues facing vulnerable populations. Therefore, three types of Pilots were launched in the fiscal year 2011 to 2012 and were expected to inform the evolution of the program by testing the capacity of not-for-profit organizations of different sizes to leverage non-federal contributions (monetary and/or in-kind) and partners from multiple sectors.

The delivery model of the program has been evolving to focus on increasing impact on vulnerable populations through mandatory leveraging and partnerships. Several of the projects under the Children and Families component are now delivered using a third-party delivery model (intermediaries), where large not-for-profits and charities (for example foundations) leverage non-federal investments and multi-sector partnerships to distribute grants to smaller not-for-profits and charities.

Appendix E - Disability component logic model

Text description for Appendix E

Appendix E illustrates the logic model of the Social Development Partnerships Program’s Disability component, which outlines from the bottom to the top, its inputs, activities, outputs, immediate outcomes, intermediate outcomes, ultimate outcome and the strategic outcome the program intends to achieve or to which it intends to contribute.

1. Inputs

Input 1.1 Program Budget

Input 1.2 Human Resources

Input 1.3 Information and Technology systems

2. Activities:

Activity 2.1 Administer grant and contribution funds

Activity 2.2 Engage stakeholders

Activity 2.3 Measure program performance and provide analysis

3. Outputs

Activities 2.1, 2.2 and 2.3 are expected to produce the following output:

• Output 3.1: Projects are funded based on evidence-based identified needs and priorities

4. Immediate Outcomes

Output 3.1 is expected to produce the following immediate outcomes:

• Immediate Outcome 4.1: Not for profit sector establishes formal partnerships to address barriers to social inclusion faced by people with disabilities
• Immediate Outcome 4.2: Not for profit sector develops and transfers knowledge and approaches that address barriers to social inclusion faced by people with disabilities

5. Intermediate Outcomes

Immediate Outcome 4.1 is expected to contribute to the following intermediate outcome:

• Intermediate Outcome 5.1 Not-for-profit sector and partners work collaboratively to address existing and evolving social issues facing people with disabilities

Immediate Outcome 4.2 is expected to contribute to the following intermediate outcome:

• Intermediate Outcome 5.2 Not-for-profit sector and partners provide access to information, programs and services that promotes inclusion of people with disabilities in society

6. Ultimate Outcome

Intermediate outcome 5.1 and 5.2 are expected to contribute to the following ultimate outcome:

• Ultimate Outcome 6.1: Social Inclusion of People with Disabilities

7. ESDC Strategic Outcome

This ultimate outcome contributes to the department’s strategic outcome: Income Security, Access to Opportunities and Well-Being for Individuals, Families And Communities

Appendix F - Children and Families Component logic model

Text description for Appendix F

Appendix F illustrates the logic model of the Social Development Partnerships Program’s Children and Families component, which outlines from the bottom to the top, its inputs, activities, outputs, immediate outcomes, intermediate outcomes, ultimate outcome and the strategic outcome the program intends to achieve or to which it intends to contribute.

1. Inputs

Input 1.1 Program Budget

Input 1.2 Human Resources

2. Activities:

Activity 2.1 Administer grant and contribution funds

Activity 2.2 Engage stakeholders

Activity 2.3 Measure program performance and provide analysis

Activity 2.4 Disseminate knowledge and promote innovation

3. Outputs:

Activities 2.1, 2.2, 2.3 and 2.4 are expected to produce the following outputs:

• Output 3.1: Projects are funded based on evidence-based identified needs and priorities
• Output 3.2: Canada’s Volunteer Awards
• Output 3.3: Analysis of General Social Survey – Giving, Volunteering and Participating

4. Immediate Outcomes

Output 3.1, 3.2 and 3.3 are expected to produce the following immediate outcomes:

• Immediate Outcome 4.1: Increased resources to communities to address barriers to social inclusion
• Immediate Outcome 4.2: Increased multi-sector partnerships to address barriers to social inclusion

5. Intermediate Outcomes

Immediate Outcomes 4.1 and 4.2 are expected to contribute to the following intermediate outcome:

• Intermediate Outcome 5.1: Improved capacity of NFPs to address barriers to social inclusion

6. Ultimate Outcome

Intermediate outcome 5.1 is expected to contribute to the following ultimate outcome:

• Ultimate Outcome 6.1: Increased Social Inclusion of Children and Families, and Vulnerable Individuals

7. ESDC Strategic Outcome

This ultimate outcome contributes to the department’s strategic outcome: Income Security, Access to Opportunities and Well-Being for Individuals, Families And Communities.