Evaluation of the New Horizons for Seniors Program

On this page

List of abbreviations

NHSP
New Horizons for Seniors Program
ESDC
Employment and Social Development Canada

List of figures

Executive summary

The New Horizons for Seniors Program provides an average of $48.9 million every year in grant and contribution funding. The funded projects aim to empower seniors in their communities and improve their well-being.

The program has 2 funding streams:

  • the Community-based funding stream provides organizations with up to $25,000 per project per year
  • the Pan-Canadian funding stream for the fiscal year 2015 to 2016 call for proposals focused on Collective Impact Initiatives. It gave funding of up to $750,000 for up to 3 years. The Collective Impact Initiatives are groups of organizations that work together to achieve common goals

The evaluation covers fiscal years from 2015 to 2016 to 2018 to 2019. It meets the Financial Administration Act  requirement to evaluate the program every 5 years. The evaluation drew from a variety of findings from 5 lines of evidence.

Main finding 1

Overall, the New Horizons for Seniors Program is a benefit to seniors and the communities where projects are in operation. This includes a positive influence on well-being, engagement in volunteering and the positive view that the projects impact communities.

Several lines of evidence point to the projects having a positive influence on seniors’ well-being. All funded organizations surveyed for the evaluation said their projects increased seniors’ participation in their communities. However, not all organizations were able to quantify the increase.

The organizations also indicated that projects helped seniors to connect with others and feel a sense of purpose.

In the case of Pan-Canadian projects, the organizations noted that they become more aware of community resources.

The COVID-19 pandemic stopped data collection from seniors about their experiences with the program.

As well, the small amount of data from communities made it impossible to measure the direct impacts of the projects on the surrounding communities. Data from secondary sources and funded organizations suggest that communities benefited from having seniors involved in funded projects. This showed up in a number of ways, such as:

  • seniors’ sharing of knowledge and skills with the community, including with younger generations
  • seniors feel like they belong to the community and feel more valued

Main finding 2

The program contributed to some aspectsFootnote 1 related to the achievement of the expected outcome of increasing funded organizations’ capacity to support seniors’ initiatives in their communities. However, a review of the definition of capacity and associated indicators for the purpose of clarity, would help to improve the measurability of the expected outcome.

The evaluation found that the funding allocated by the program might have increased the organizations’ capacity to support seniors’ initiatives in their communities. It indicates that the projects contributed to some aspects of building the capacity of organizations in a variety of ways, including:

  • recruiting additional volunteers
  • enhancing physical infrastructure
  • developing networks and partnerships to support community activities

Because the definition of capacity is broad, the evaluation team had challenges. Measuring the extent to which the program contributed to increasing funded organizations’ capacity to support seniors’ initiatives in their communities was difficult.

The evidence related to increasing capacity includes:

  • recruiting volunteers
  • creating partnerships
  • attracting senior participants
  • investing in physical infrastructure

Note: The new Performance Information Profile has information about the Program.

Main finding 3

The Collective Impact approach showed promise in addressing social isolation. Fostering close partnerships, which is necessary to implement the model, however, requires more.

The approach encouraged organizations to coordinate efforts and work together to help address a large-scale issue affecting seniors (social isolation). It also helped organizations learn new approaches and reach new audiences of seniors. The Pan-Canadian stream of the program introduced the Collective Impact approach in 2015. The impact and sustainability of this approach cannot be fully measured so early after implementation. Despite this, all surveyed organizations agreed that the approach helped them in a number of ways, such as:

  • raising awareness of their projects in the community
  • increasing the number of participants and volunteers
  • helping to effectively solve complex community challenges (92% of surveyed Pan-Canadian funding recipient organizations)

The organizations reported a few challenges, such as:

  • the limited time available to understand the Collective Impact approach
  • the lack of clear communication from the Department to organizations about their expectations
  • the fragile collaboration between organizations in the Collective Impact Initiatives
  • the varying levels of knowledge and experience with evaluation, data collection and reporting activities

Main finding 4

There is evidence to suggest that the process used to assess Community-based funding request proposals would have benefited from greater guidance, level of detail and clarity.

Evidence from the document review suggests that terms used to assess proposals could have different meanings to the Service Canada Program Officers. As well, rating scales (such as, poor, average, exceptional) do not have clear definitions and lack indicators. This means that Service Canada Officers used their judgment and previous experience to assess the merit of funding request proposals. Although the 2020 call for proposals did not take place during the evaluation period, the issues noted above have been addressed. This was done by reviewing the program operational directives to support assessment by regional offices.

Evidence from a few key informant interviews pointed to some challenges in obtaining historical information needed to assess the applications. This is due to issues such as the labelling of documents in the Common Systems of Grants and Contributions.

Introduction

This evaluation report presents key findings and observations of both funding streams of the New Horizons for Seniors Program: Community-based projects and Pan-Canadian projects. The evaluation covers program activities over a 4 year period, from fiscal years 2015 to 2016 to 2018 to 2019. The evaluation meets the terms of the Financial Administration Act  and the policy on results.

The report answers the 6 evaluation questions through 3 main themes:

  • program benefits to seniors and communities
  • organizational capacity and the barriers that inhibit social inclusion
  • an innovative approach for wider impact

Annex A lists the evaluation questions. Annex B shows how the evaluation questions reflect each of the 3 themes. Annex C presents the New Horizons for Seniors Logic Model.

The evaluation drew from 5 different lines of evidence. Groups consulted include representatives from the Department and representatives from organizations that were funded through both streams of the program. More details on the lines of evidence is in Annex D.

Due to COVID-19 limitations, the evaluation did not collect any direct information from seniors about their experiences. As a result, the evaluation relied on the following sources:

  • the views of funded organizations
  • documentary sources
  • key informants for data on seniors and communities
More details on the limitations of the evaluation are in Annex E.

Program background

Launched in 2004, this federal grants and contributions program supports projects that empower seniors and enhance their well-being, as well as community vitality. The program encourages seniors to share their knowledge, skills, and experiences with others in the community. The program funding supports 2 streams: Community-based projects and Pan-Canadian projects.

$32 million in annual budget

The design of the Community-based projects recognizes communities as the focal point for program delivery. Funded projects are typically volunteer-based, supported by communities and inspired or led by seniors. The projects are eligible to receive up to $25,000 in funding per project for up to 1 year.

$9 million in annual budget

For fiscal year 2015 to 2016 Call for Proposals, Pan-Canadian funding supports multi-year projects using collaborative approaches to address seniors’ issues. This Collective Impact model uses a collaborative, multi-partner approach. It brings together a group of organizations from different sectors to commit to a common agenda. The objective is to produce significant changes in their community. The funded projects received financial support of up to $750,000 for up to 3 years.

Note: Backbone organizations lead the implementation of the Pan-Canadian projects. Backbone and Collaborator organizations together make up the Collective Impact Initiative.

More information on the program outcomes is in Annex B and Annex C.

Key findings

Program benefits to seniors and communities

Main finding 1

Overall, the New Horizons for Seniors Program is a benefit to seniors and the communities where projects take place. This includes a positive influence on well-being, engagement in volunteering and the positive view that the projects impact communities.

According to funding recipients, Community-based and Pan-Canadian projects positively influenced senior’s well-being

Amongst results from 4 sources of evidence, there is a consensus that the projects of both funding streams positively influenced seniors’ well-being.

57% of funded Community-based organizations felt projects increased seniors’ socialization or ended their social isolation.

Figure 1 presents other benefits as reported by funding recipients.

Figure 1: How Community-based projects influenced seniors’ well-being
Figure 1: How Community-based projects influenced seniors’ well-being - Text description follows
Figure 1 – Text version
Benefits to Seniors  from Community-based projects Percentage
Increased socialization, end social isolation 57%
Enhanced mental health, well-being, physical health 52%
Community improvement, involvement, ownership 30%
Chance to share, gain skills (including technology) 28%
Enhanced sense of belonging, being needed, valued 25%
Provided safe environment, offered inclusiveness 21%
Access to resources, funding 9%
Other 1%
No comment 2%
Don’t know 1% 
  • Source: Survey for the Evaluation of the NHSP (2020), 558 Community-based funding recipient respondents.

The survey asked Community-based funding recipient organizations who would benefit from taking part in the projects:

  • 76% of organizations felt seniors would benefit from the projects
  • followed by the community and youth (39%)
  • persons with disabilities (30%)

Note: Senior participants were not consulted directly in this evaluation. 

Nearly half of Pan-Canadian funding recipient organizations surveyed said the projects:

  • created social and community connections among seniors (48%)
  • reduced barriers to inclusion (43%)
  • provided opportunities for support and learning (39%)

Findings from the interviews and the survey of Pan-Canadian funded organizations also showed that the Collective Impact projects positively influenced participating seniors’ well-being. They said seniors benefitted by:

  • increasing awareness of community resources (50%)
  • becoming more engaged in their community (38%)
  • learning something new (29%)

Figure 2 presents other reported benefits of participating in Collectives Impacts projects.

Figure 2: Benefits to seniors from participation in Collective-Impact Initiatives as reported by surveyed Pan-Canadian funded organizations
Figure 2: Benefits to seniors from participation in Collective-Impact Initiatives as reported by surveyed Pan-Canadian funded organizations - Text description follows
Figure 2 – Text version
Benefits to Seniors from participation in Collective Impact Initiatives Backbone Collaborator Backbone and Collaborator
Increase awareness of community resources 0% 50% 50%
More engaged in their community 0% 38% 38%
Learn about something new 8% 21% 29%
Sense of purpose, increased self-esteem 8% 21% 29%
Share knowledge, build connections with other generations / peers 8% 21% 29%
Positively impacts well-being, health, increase physical activity 8% 17% 25%
New, enhanced programming opportunities 0% 17% 17%
Don’t know 0% 4% 4% 
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Engagement is an important step in increasing seniors’ participation in their communities. The following sections explore this aspect, as well as its effects.

Community-based and Pan-Canadian projects engage seniors to participate and get involved in community projects

91% of surveyed Community-based organizations thought the projects helped seniors become more engaged in their communities. They pointed to the following observations as influencing their thinking:

  • more seniors participating in the community (69%)
  • seniors being part of a more vibrant community (41%)
  • seniors gaining knowledge and communicating of information (29%)

Figure 3 outlines other reported responses.

The types of activities that Community-based organizations reported include:

  • social gatherings (61%)
  • learning events (42%)
  • capital renovations and repairs (32%)

Figure 4 outlines other reported responses.

Figure 3: How the projects helped seniors become more engaged in their communities
Figure 3: How the projects helped seniors become more engaged in their communities - Text description follows
Figure 3 – Text version
How the projects helped seniors become more engaged in their communities Percentage
Grater participation by seniors, more activity, socialization 69%
Part of more vibrant community 41%
Knowledge gained, information communicated 29%
Physical volunteering, performing, cooking 24%
Inclusiveness, Intergenerational, cultural 23%
Leadership, influenced, direction, decision making 21%
Other 6%
No comment 2%
Don’t know 1% 
  • Source: Survey for the Evaluation of the NHSP (2020), 531 Community-based funding recipient respondents. The survey question was:“ Did the NHSP projects help seniors become more engaged in their communities?” and “How so?”
Figure 4: Types of project activities reported by surveyed Community-based funding recipient organizations
Figure 4: Types of project activities reported by surveyed Community-based funding recipient organizations - Text description follows
Figure 4 – Text version
Types of project activities reported by surveyed Community-based funding recipient organizations Percentage
Social gathering 61%
Learning event 42%
Capital renovations or repairs 32%
Arts inspired 19%
Exercise 17%
Project senior led 13%
Address local issue 7%
Day trip 5%
Other 3%
Don’t know 3% 
  • Source: Survey for the Evaluation of the NHSP (2020), 583 funding Community-based funding recipient respondents.

Evidence points to an increase in seniors' engagement. This results from an increase in the number of participants and volunteers that were part of the Collective Impact projects. When the Pan-Canadian organizations were surveyed:

  • 83% agreed that their Collective Impact activities collaboratively helped seniors become more engaged in their communities. The number of respondents is 20
  • 100% agreed that they saw an increase in the number of participants related to their Collective Impact projects. The number of respondents is 24

The survey revealed the Pan-Canadian activities that organizations engaged in as part of the Collective Impact Plan. Activities included:

  • research (58%)
  • programming and social activities (50%)
  • connecting seniors to the services they need (46%)
  • social gatherings (38%)
  • learning events (33%)
Figure 5: Activities that organizations and partners engaged in as part of Collective Impact Plan
Figure 5: Activities that organizations and partners engaged in as part of Collective Impact Plan - Text description follows
Figure 5 – Text version
Activities that organizations and partners engaged in as part of Collective Impact Plan Backbone Collaborator Backbone and collaborator
Research 8% 50% 58%
Programming and social activities 13% 38% 50%
Connecting seniors to services they need 4% 42% 46%
Social gathering 13% 25% 38%
Learning event 4% 29% 33%
Recruit and train volunteers 4% 17% 21%
Home visits and telephone calls 4% 13% 17%
Raising awareness 4% 13% 17%
Developing technology-based tools 4% 8% 13%
Transportation 8% 0% 8%
Address local issue 0% 8% 8%
Other 4% 8% 13%
Don’t know 0% 8% 8% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

The Document review notes that although 100% of Pan-Canadian organizations reported increased senior participation, not all Collective Impact Initiatives provided quantitative data.

Similarly, a lack of community-level data created an uncertainty about the extent to which Pan-Canadian projects contributed to engagement outside of the projects.

Thus, it is difficult to know whether the Collective Impact model was realizing population level results.

Community-based and Pan-Canadian projects engage seniors to volunteer

The program allows a chance for volunteers to get involved. To a large extent, the projects are also successful in attracting them. However, there is a high degree of turnover in volunteers, and organizations often struggle to keep them.

Community-based organizations said why they volunteer:

  • because the projects involve seniors and they contribute to the projects (62%)
  • because there is a desire to improve the community or contribute to community spirit (41%)
  • because they felt a sense of belonging (29%)
  • because the projects provided an opportunity to meet, interact with others and socialize (26%)

Figure 6 outlines other reported responses.

The Pan-Canadian organizations reported success in maintaining their volunteer base.

Over half of the Pan-Canadian organizations suggested in the survey that volunteers continued to volunteer because they felt valued and important.

Figure 6: Why Community-based volunteers continued to volunteer
Figure 6: Why Community-based volunteers continued to volunteer - Text description follows
Figure 6 – Text version
Why Community-based volunteers continued to volunteer Percentage
Opportunity to be involved, contribute to projects 62%
Community spirit, desire to improve community, belief in goals 41%
Sense of belonging , positive team environment, feeling welcome 29%
Opportunity to meet, interact with others, socialize 26%
Opportunity to learn, experience 21%
Opportunity to preserve, record local history, culture 6%
Other 2%
Don’t know 2% 
  • Source: Survey for the Evaluation of the NHSP (2020), 504 Community-based funding recipient respondents.

More details about the results about volunteerism and the New Horizons for Seniors Program are in Annex H.

Community-based and Pan-Canadian projects allow seniors to share their knowledge and experiences

Some examples of seniors sharing knowledge include:

  • seniors participating in a knitting circle and teaching others new techniques
  • seniors leading senior exercise classes
  • seniors leading projects and sharing their organizational skills
  • seniors benefitted from participating in Community-based intergenerational projects

Organizations felt that the projects were beneficial for seniors to:

  • share their knowledge and build connections with other generations and their peers (49%)
  • give seniors a sense of purpose and increased their self-esteem (27%)
Figure 7: How seniors benefitted from participating in an inter-generational project
Figure 7: How seniors benefitted from participating in an inter-generational project - Text description follows
Figure 7 – Text version
How seniors benefitted from participating in an inter-generational project Percentage
Share knowledge, build connections with other generations, peers 49%
Sense of purpose, increased self-esteem 27%
Learn from, understand younger generation 16%
More engaged in community, benefit (general) 13%
Enjoy enthusiasm, keeps them yond, do things they like 11%
Younger generation would benefit 10%
Improved access to technology 10%
Positively impacts well-being, health, increased physical activity 7%
Addresses societal issues and promotes understanding 4%
Building, location improved 4%
Increased awareness of community resources 1%
Other 6%
Do not benefit 2%
Don’t know 3% 
  • Source: Survey for the Evaluation of the NHSP (2020), 408 Community-based funding recipient respondents.

70% of the Community-based survey respondents reported that their projects had an intergenerational component.

Examples of intergenerational projects include:

  • programming that involves joint activities with seniors and youth, which allows for the exchange of knowledge
  • projects that focus on seniors’ knowledge and skills, in which seniors teach about hobbies or topics of interest. Examples of such projects might include:
    • storytelling
    • gardening
    • community history
    • cultural practices
    • language

Organizations did not design the Collective Impact Initiatives with explicit instruction or intention to leverage the knowledge and skills of seniors. Despite this, numerous lines of evidence show that organizations sought senior consultation and contribution to the design, implementation and evaluation. In this way, the design of the projects were to a large extent the result of capitalizing on seniors’ knowledge and skills.

Seniors playing active leadership and advisory roles in running the project allowed them to transfer their knowledge and skills more broadly in the organization.

The program encouraged some seniors to better care for their own health. This helped them become more productive and improved their quality of life. One participant described how they realized that they are not alone in facing limitations and how confident and capable they are.

Inter-generational Pan-Canadian projects benefit seniors

A little more than a third (38%) of the Pan-Canadian projects integrated members of other generations into their project.

Integrating other generations into the projects helped seniors. They were able to share their knowledge, skills and also learn from younger generations.

78% of Pan-Canadian survey respondents felt that seniors benefit from participating in inter-generational projects. This is because they can learn from and understand younger generations, as well as share knowledge and connect with other generations (67%).

Figure 8: How seniors benefitted from participating in inter-generational projects
Figure 8: How seniors benefitted from participating in inter-generational projects - Text description follows
Figure 8 – Text version
How seniors benefitted from participating in an inter-generational project Backbone Collaborator Backbone and collaborator
Learn from, understand younger generation 22% 56% 78%
Share knowledge, build connections with other generations, peers 11% 56% 67%
Younger generation would benefit 11% 11% 22%
Sense of purpose, increased self-esteem 11% 11% 22%
Enjoy enthusiasm, keeps them young, do things they like 0% 11% 11%
Don’t know 11% 0% 11% 
  • Source: Survey for the Evaluation of the NHSP (2020), 9 Pan-Canadian funding recipient respondents.

Community-based projects benefit community members of other generations

The community-based projects contributed to the benefit of community members of other generations.

Figure 9 shows the age-breakdown of community members who benefit from the program’s projects.

Figure 9: Percentage of individuals benefiting from the program by age from fiscal year 2015 to 2016 to fiscal year 2018 to 2019
Figure 9: Percentage of individuals benefiting from the program by age from fiscal year 2015 to 2016 to fiscal year 2018 to 2019 - Text description follows
Figure 9 – Text version
Age Percentage of individuals benefiting from the program
65 and more 49%
45 to 64 27%
25 to 44 13%
Under 25 11% 
  • Source: Document Review Technical Report of the Evaluation of the NHSP (2020).

A senior working with new Canadians described the positive impact the Program has had on new senior immigrants. It helped them to adapt to their new environment, make new friends and begin to feel that Canada is their home.

Community-based and Pan-Canadian funded organizations felt their projects benefit communities

Community-based organizations felt the community benefitted from increased senior involvement because of the following reasons:

  • there was a sense of increased community cohesion (68%)
  • seniors could share their skills and knowledge (with younger people) (30%)
  • there was an enhanced sense of value and energy among seniors (28%)
  • they noticed improved health effects (17%)
Figure 10: How the community benefitted from increased senior involvement in projects
Figure 10: How the community benefitted from increased senior involvement in projects - Text description follows
Figure 10 – Text version
How the community benefitted from increased senior involvement in projects Percentage
Increasing community cohesion 68%
Sharing of skills, knowledge (with younger people) 30%
Seniors feel valued and energized 28%
Positive health impacts 17%
Increasing the volunteer base 15%
Addressing local issues 8%
Other 4%
The community doesn’t benefit 2%
Don’t know 8% 
  • Source: Survey for the Evaluation of the NHSP (2020), 583 Community-based funding recipient respondents. The survey question was: “How did the project help seniors become more engaged in their communities?”.

96% of surveyed Pan-Canadian respondents felt that the New Horizons for Seniors Program fulfilled a community need. The number of respondents is 23.

They also felt that the community benefitted from the Collective Impact Initiative’s projects. They felt this way because of the following reasons:

  • the projects addressed social isolation (42%)
  • the projects increased community cohesion (29%)
  • the projects allowed seniors to share their knowledge and skills with youth (29%)

Community cohesion refers to the idea of people in the community coming together to create a better community for everyone.

Figure 11: How communities benefitted in the last 3 years from increased senior involvement in the Collective Impact Initiative’s projects
Figure 11: How communities benefitted in the last 3 years from increased senior involvement in the Collective Impact Initiative’s projects - Text description follows
Figure 11 – Text version
How communities benefitted in the last 3 years from increased senior involvement in the Collective Impact Initiative’s projects Backbone Collaborator Backbone and collaborator
Address social isolation 13% 29% 42%
Increasing community cohesion 0% 29% 29%
Sharing of knowledge and skills (with younger people) 4% 25% 29%
Increasing the volunteer base 8% 4% 13%
Address local issues 0% 8% 8%
Increase in well-being 4% 0% 4%
Don’t know 0% 17% 17%
  • Note: Communities were not consulted directly in this evaluation.
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipients respondents.

Pan-Canadian funded organizations have noticed that communities are responding to seniors’ needs and interests

Key informants and survey respondents indicated that the Collective Impact activities helped communities to achieve greater senior involvement.

75% of surveyed organizations agreed that they had seen changes in the community. These changes were in relation to seniors programming and social inclusion after the Collective Impact projects began. Most prominently, organizations saw:

  • new programs and collaborations formed that speak to senior’s interests (46%)
  • more available opportunities for seniors (33%)
  • more designated spaces for seniors created by communities (13%)
  • more opportunities to get involved (8%); which allowing them to get to know one another (8%)
Figure 12: The types of changes seen in the community related to seniors programming and social inclusion after the Collective Impact projects began
Figure 12: The types of changes seen in the community related to seniors programming and social inclusion after the Collective Impact projects began - Text description follows
Figure 12 – Text version
The types of changes seen in the community related to seniors programming and social inclusion after the Collective Impact projects began Backbone Collaborator Backbone and collaborator
Address social isolation 13% 29% 42%
Increasing community cohesion 0% 29% 29%
Sharing of knowledge and skills (with younger people) 4% 25% 29%
Increasing the volunteer base 8% 4% 13%
Address local issues 0% 8% 8%
Increase in well-being 4% 0% 4%
Don’t know 0% 17% 17% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Organizational capacity and the barriers that inhibit social inclusion

Main finding 2

The program contributed to some aspectsFootnote 2 related to the achievement of the expected outcome of increasing funded organizations’ capacity to support seniors’ initiatives in their communities. However, a review of the definition of capacity and associated indicators for clarity would help to improve the measurability of the expected outcome.

The program collected data on funded organizations’ ideas of organizational capacity.

For example, funded organizations completed final reports with questions that assessed capacity as:

How did your project support your organization to increase its capacity to support seniors initiatives in your community?

Response categories included:

  • provided/enhanced physical infrastructure which supported program or service delivery
  • recruited more volunteers to support community activities
  • developed partnerships and networks to support community activities
  • expanded organization’s ability to provide activities/programs that benefit seniors

Program officials indicate that funding allocated by the program increased the organizations’ capacity to support seniors’ initiatives in their communities. Organizations reported on aspects that may have contributed to building their capacity in a variety of ways. These included:

  • recruiting additional volunteers
  • enhancing physical infrastructure
  • developing networks and partnerships to support community activities

Program officials interviewed also indicated that the program funding allows the organizations to increase their capacity to support community initiatives. Given that the definition of capacity is broad, the evaluation faced challenges measuring the progress towards the following outcome the extent to which the program contributed to increasing funded organizations’ capacity to support seniors’ initiatives in their communities

One senior describes that the program had provided funds to their community since 2007, which allowed them to run projects that are attended by many local seniors and Elders. One of their projects allowed them to create storyboards detailing scenes that are historically significant to the community.

Program data collected from funded organizations may provide insight into some aspects of organizational capacity. However, the program’s definition of capacity used is unclear. As such, evaluating the progress made towards this outcome required the use of additional effort by the evaluation team.

The evaluation team attempted to identify what capacity could mean in order to assess progress related to the relevant outcomes and evaluation questions. Consult Annex I  for more information and suggested indicators.

Throughout the evaluation, a number of items arose regarding the design and implementation of the program. These items have effects on organizational capacity and its measurement. More information on Evaluators’ observations is in Annex J.

Evaluators identified the barriers to social inclusion for seniors and assessed whether the funded recipients are addressing them. Consult Annex K  for more information.

An innovative approach for wider impact

Main finding 3

The Collective Impact approach showed promise in addressing social isolation. However, fostering close partnerships, which is necessary to implement fully the model, requires more time.

Pan-Canadian funded organizations felt that they had raised the visibility and awareness of the issues affecting senior social inclusion

Through the implementation of the Collective Impact projects, key informants described their efforts in raising awareness of the issues. They recognized that partnerships facilitated the growth in awareness of the projects. All indicated an increase in participants and or volunteers throughout the lifecycle of the projects. The Pan-Canadian survey of organizations showed that:

  • 100% felt the partners in the Collective Impact Initiative helped to raise awareness in the community about the project
  • 92% felt that the Collective Impact approach is an effective approach in solving complex community challenges. The number of respondents is 22
  • partners in the Collective Impact also helped organizations to gain participants (79%) and volunteers (54%)
Figure 13: Partners in Collective Impact Initiatives helped gain participants, volunteers and awareness in the community about project
Figure 13: Partners in Collective Impact Initiatives helped gain participants, volunteers and awareness in the community about project - Text description follows
Figure 13 – Text version
Partners in Collective Impact Initiatives helped gain participants, volunteers and awareness in the community about project Backbone Collaborator Backbone and collaborator
Awareness in the community about your projects 17% 83% 100%
Participants 17% 63% 79%
Volunteers 13% 42% 54% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

In the absence of a tangible measure of visibility and awareness of the issues, this indicator is based on perceptions of the key informants who were interviewed and survey respondents.

Seniors engaged in project design and implementation, as the program intended

The projects’ development and implementation involved seniors at various stages.

Often, organizations looked to seniors as part of a working group or committee.

Projects introduced new programming and built networks

75% of Pan-Canadian organizations thought they had seen changes in the community related to senior’s programming and social inclusion after the projects began. They described what they had observed in the community:

  • the forming of new programs or collaboration
  • the creation of more activities or opportunities for seniors
  • the creation of senior spaces
  • greater opportunities for seniors to network and be social

Pan-Canadian organizations described that they developed and strengthened partnerships and collaboration

Through the key informant interviews and the survey, there is a consensus that Pan-Canadian organizations learned a great deal throughout the process. A quarter of the organizations learned about how important it is to have organizational support throughout their projects.

Figure 14: What was learned about running an organization from implementing project
Figure 14: What was learned about running an organization from implementing project - Text description follows
Figure 14 – Text version
What was learned about running an organization from implementing project Backbone Collaborator Backbone and collaborator
Importance of organization support 8% 17% 25%
Collaboration 0% 21% 21%
Communications 4% 17% 21%
Flexibility 4% 13% 17%
Management (financial, time) 4% 13% 17%
Evaluation techniques 0% 8% 8%
Did not learn anything new 0% 8% 8%
Don’t know 0% 17% 17% 
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Organizational leaders also learned from other organizations in the Collective Impact Initiatives.

Figure 15: How respondents learned from other organizations in the Collective
Figure 15: How respondents learned from other organizations in the Collective - Text description follows
Figure 15 – Text version
How respondents learned from other organizations in the Collective Backbone Collaborator Backbone and collaborator
Teamwork, co-ordination 13.04% 60.87% 74%
Sharing information 8.70% 56.52% 65%
Supporting, understanding seniors’ needs 87.0% 26.09% 35%
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Pan-Canadian collaboration is not without its challenges, but over time, the organizations learned how to address them

The most common challenge cited by the Pan-Canadian organizations in the key informant interviews was in collaborating with or gaining partners. The survey confirmed this; 63% said collaborating with or gaining partners was a challenge.

Figure 16: Challenges in implementing the Collective Impact Plan
Figure 16: Challenges in implementing the Collective Impact Plan - Text description follows
Figure 16 – Text version
Challenges in implementing the Collective Impact Plan Backbone Collaborator Backbone and collaborator
Collaborating with,  gaining collaboration,  partners 17% 46% 63%
Evaluation of projects 4% 33% 38%
Not enough time 8% 21% 29%
Writing Collective Impact Plan, final reports 4% 13% 17%
Staff and volunteer turnover 0% 13% 13%
Geographic 4% 4% 8%
Continuing the project after funding done 0% 4% 4%
Raising awareness about project 0% 4% 4%
Other 0% 4% 4%
No challenges 0% 4% 4%
Don’t know 0% 8% 8% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Organizations also described how the organizations learned to:

  • collaborate with their partners (46%)
  • address the barriers to seniors inclusion (42%)
  • improve their communications and outreach (42%)
Figure 17: How organization improved its administration over past 3 years
Figure 17: How organization improved its administration over past 3 years - Text description follows
Figure 17 – Text version
How organization improved its administration over past 3 years Backbone Collaborator Backbone and collaborator
Collaboration with collaborators, partners 13% 33% 46%
Address barriers to seniors inclusion 13% 29% 42%
Improved communications, outreach 8% 33% 42%
Evaluation techniques 4% 8% 13%
Computer skills 0% 4% 4%
Don’t know 0% 8% 8% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipients respondents.

Pan-Canadian funded organizations expressed how they were able to expand programs and services after the funding has ended

Interviewed organizations explained that though they were not successful in receiving funding through the new 2018 Call for Concepts, there are legacies that remain, including:

  • the knowledge that they gained throughout their projects that are applicable to other projects or circumstances
  • the adaptation of some activities that started during their project into different activities or activities directed at different groups of seniors
  • the beginning discussions of the creation of a Funders Table. The intent of this initiative would bring together the not-for-profit community and funders. Their activities would try to better coordinate and align their abilities and funds

Through these initiatives, the work of the Pan-Canadian Collective Impact projects will continue to address the social inclusion of seniors. Through the survey of Pan-Canadian organizations:

  • 88% thought that they would continue some activities after the funding was complete. The number of respondents is 21
  • 54% said that they would continue to collect data after their project was complete. The number of respondents is 13

There are sustainable legacies of the Pan-Canadian projects

Interviewed organizations said that their participation in the Collective Impact projects allowed them to learn. They described how important it is to be cooperative and transparent in building effective partnerships.

Organizations described in the survey how seniors benefitted by participating in funded projects that were part of the Collective Impact Initiative:

  • half the organizations felt there is an increased awareness of community resources (50%)
  • seniors are more engaged in their communities (38%)
  • roughly about a third of organizations felt that seniors benefitted by learning something new
Figure 18: How seniors benefitted by participating in the projects that were part of Collective Impact Initiatives
Figure 18: How seniors benefitted by participating in the projects that were part of Collective Impact Initiatives - Text description follows
Figure 18 – Text version
How seniors benefitted by participating in the projects that were part of Collective Impact Initiatives Backbone Collaborator Backbone and collaborator
Increase awareness of community resources 0% 50% 50%
More engaged in their community 0% 38% 38%
Learn about something new 8% 21% 29%
Sense of purpose, increased self-esteem 8% 21% 29%
Share knowledge, build connections with other generations, peers 8% 21% 29%
Positively impacts well-being, health, increased physical activity 8% 17% 25%
New, enhanced programming opportunities 0% 17% 17%
Don’t know 0% 4% 4% 
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

There is a strong indication that organizations will continue some activities after ESDC funding ended. However, it is unlikely that the activities will continue in the original form as implemented through the Collective Impact Initiative. Key informants described that without additional funding, they would need to adapt their project. This meant adapting the original activities or the resources developed (example, training modules) for other purposes. Organizations indicated that funding and time were the most dominant challenges that they faced in order to continue after the funding has ended.

There is evidence of the collection and use of performance and evaluation data

There is evidence to suggest that the Pan-Canadian organizations learned about outcome measurement. To some extent, there is also evidence that they may continue collecting data to track progress after the project has finished.

Organizations know that evaluation and data collection is important, but not all are able to evaluate their own projects on their own. Outside of this project, organizations stressed that they are at a disadvantage because of time, funding and ability. As a result, they did not think they could monitor their activities to the same extent they did during this project.

Organizations measured their Collective Impact project results and monitored their progress.

Figure 19: How Pan-Canadian funded organizations measure Collective Impact project results
Figure 19: How Pan-Canadian funded organizations measure Collective Impact project results - Text description follows
Figure 19 – Text version
How Pan-Canadian funded organizations measure Collective Impact project results Backbone Collaborator Backbone and collaborator
Through ongoing evaluations, evaluation reports 13% 21% 33%
Through the collection of data 0% 29% 29%
Through support from the backbone organization 8% 21% 29%
Sent out a survey 8% 13% 21%
Through keeping in contact with participants, staff 4% 13% 17%
Saw an increase in participation 4% 8% 13%
Hired an evaluator 0% 13% 13% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Learning more about the factors that may contribute to the success of the Collective Impact Model, however population impacts are difficult to measure

The introduction of the Collective Impact approach to the non-profit community could have improved with more time because:

  • they needed time to learn the approach
  • they needed time to understand the roles and responsibilities

In hindsight, survey respondents provided advice for others on how to implement a successful Collective Impact project. Implementation is facilitated by:

  • having clear goals / process (50%)
  • anticipating time and resource restraints (38%)
  • collaborating closely with other organizations (33%)

These results are consistent with the key informant interviews

Figure 20: Advice for others on how to implement a successful Collective Impact project
Figure 20: Advice for others on how to implement a successful Collective Impact project - Text description follows
Figure 20 – Text version
Advice for others on how to implement a successful Collective Impact project Backbone Collaborator Backbone and collaborator
Clear with goals, process 4% 46% 50%
Anticipate time, resource restraints 8% 29% 38%
Collaborate with other organizations 8% 25% 33%
Evaluate your project 8% 25% 33%
Involve seniors in planning, implementing 8% 21% 29%
Don’t know 0% 17% 17% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Communication between the Department and Pan-Canadian funded organizations could improve

Evidence pointed to a desire for improved communication between the Department and Pan-Canadian organizations.

Specifically, improvements to the Collective Impact’s implementation are:

  • communicating clearly about expectations of the approach and process
  • providing more detail about the reporting and evaluation process
  • reducing the turnover of the Department to support the organizations
Figure 21: How the Department could have improved its communication of its expectations of the Collective Impact Approach and its implementation
Figure 21: How the Department could have improved its communication of its expectations of the Collective Impact Approach and its implementation - Text description follows
Figure 21 – Text version
How the Department could have improved its communication of its expectations of the Collective Impact Approach and its implementation Backbone Collaborator Backbone and collaborator
Not clear expectations, process 4% 25% 29%
Reporting, evaluation process 4% 17% 21%
Consistent ESDC staff, more ESDC assistance 8% 13% 21%
Longer time frame 4% 13% 17%
Clarity on backbone organization’s role 0% 8% 8%
No improvements (positive comments) 4% 0% 4%
Don’t know 0% 25% 25% 
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

The Department worked with an expert organization, Innoweave, to support the design and implementation of the Collective Impact approach. The key informant interviews and survey results present inconsistent findings regarding Innoweave’s participation with the Collective Impact Initiatives. It suggests Innoweave was a benefit in some circumstances, but not as much in others.

  • 83% of respondents were happy with the support they received from the Department. The number of respondents is 20
  • 54% of respondents were happy with the support they received from Innoweave. The number of respondents is 13

Finding request proposals assessment guidance

Main finding 4

There is evidence to suggest that the process used to assess Community-based funding request proposals would have benefited from greater guidance and level of detail.

Thousands of applicants send proposals to ESDC every year for the program’s Community-based grants. Service Canada program delivery officers in the regions as well as the Regional Committees review the proposals.

The assessment and screening guide, as well as the assessment grids include areas for the reviewer to score applications. Without clear guidelines on some aspects of the assessment, reviewers sometimes rely on their experience and their familiarity with the organizations.

The assessment grids provide a list of reasons to choose from for not recommending a funding request proposal. The assessment grids lack clear definitions of terms. Reviewers would benefit from clear definitions in order to limit potential inconsistencies in their review process. Reviewers could interpret rating scales (such as, poor, average, exceptional) differently.

Service Canada has introduced a quality assurance process to review the consistency of its assessment and recommendations for funding to address the concern of inconsistencies between regional offices. Although the 2020 call for proposals did not take place during the evaluation period, the issues noted above have been addressed by reviewing the program operational directives to support assessments by regional offices.

Clearer definitions and better indicators would assist program delivery officers. This would help them not to rely mainly on their judgment and previous experience to assess the merit of a funding request proposal.

Note: The New Horizons for Seniors Program is a community-driven program based on local and regional involvement. The Regional Committees’ role is to review community-based project proposals and provide a regional perspective to the Department. They play more of an advisory role to Service Canada by providing insight into the assessment of projects rather than by reassessing them.

The assessment grids include the following rationale for projects being unsuccessful in their applications for fundingFootnote 3:

  • weak benefit to seniors or the community
  • weak in community involvement, volunteerism
  • weak in community support
  • weak in meeting program objectives
  • weak in partnerships
  • weak in seniors' involvement
  • weak in value for money
  • weak organizational capacity
  • weak project feasibility
  • weak track record with NHSP or ESDC funding
  • project activities or cost do not meet program's Terms and Conditions

Conclusion and observations

The evaluation used a thematic approach in response to the 6 evaluation questions. This format was optimal because many questions overlapped with each other and focused on 3 key themes:

  • benefits to seniors and communities
  • the capacity of organizations to address barriers to social inclusion
  • the new Collective Impact approach to Pan-Canadian projects

From these themes, one main conclusion emerged and 3 observations to further improve program development in the future.

Main conclusion: The New Horizons for Seniors Program is a benefit to seniors and the communities where the projects operate

The program benefits seniors and their communities through socialization, learning and skills sharing, and mentorship between seniors and with younger generations. The program supports seniors’ well-being by participating in and volunteering for funded projects. This allows seniors to continue contributing to their communities. Funding recipient organizations reported that volunteering gave seniors a sense of belonging, among other benefits. The projects provided participant and volunteer seniors with knowledge and awareness of resources available to them in their communities, as well as engagement and learning opportunities. Funding recipients reported that communities benefit from projects addressing social isolation through:

  • increased community cohesion
  • the knowledge and skills shared by seniors, particularly when mentoring

Observations 1

Over the reference period, there is some indication that funded organizations contributed to building their capacity in a variety of ways. These included:

  • recruiting additional volunteers
  • enhancing physical infrastructure
  • developing networks and partnerships to support community activities

Given the broad definition of capacity, the evaluation faced challenges. Measuring the extent to which the Program contributed to increasing organizations’ capacity to support seniors’ initiatives in their communities was difficult.

A review of the program information profile would enable a greater understanding of the results achieved. A review would include refinements to the following:

  • the definition of some expected outcomes
  • the associated performance indicators
  • the explanation of how the project activities of each funding stream lead to the program’s outcomes

Observations 2

There is some evidence that points to some challenges in obtaining historical information needed to assess the applications. Challenges stem from the organization and labelling of documents in the Common Systems of Grants and Contributions.

Evidence suggests that Service Canada program officers could interpret the terms used to assess funding request proposals differently. Program delivery officers need to rely on their judgment and previous experience to assess the merit of a funding request proposal. This is because they found:

  • rating scales that were not defined clearly
  • an absence of clear indicators

Program officials indicate that Regional Committees also assess each project proposal using the same approach. This allows for reinforcing consistency in the assessment process.

Refinements to the guidance tied to the assessment process of the funding request proposals (assessment guide and scoring tools) would lead to:

  • reinforcing the delivery of the Community-based stream of the program
  • further strengthening the consistency of the assessment process

Observations 3

The Pan-Canadian Collective Impact approach hinges on the success of organizations’ partnerships and relationships, which take time to establish.

The Collective Impact approach showed promise in addressing social isolation. It received positive feedback about the prospect for its success in the future. Participating organizations said building trust and learning to work together was significant to implementing their projects and supporting seniors and their communities.

Despite the importance and usefulness of partnerships, some Collective Impact Initiatives faced challenges building the necessary trust and cooperation with one another. Fostering partnerships and a spirit of collaboration was a necessary piece of the model. It required more time and effort to see the projects reach its full potential.

Annex A: Evaluation questions

  1. To what extent has the program improved the capacity of organizations to address barriers to social inclusion for seniors?
  2. To what extent has the program contributed to an increase in the participation and engagement of seniors in their communities?
  3. To what extent has the Pan-Canadian Collective Impact approach been effectively designed and delivered?
  4. To what extent are early outcomes being achieved in the Collective Impact projects that could lead to systemic and population level changes?
  5. To what extent does the Program’s Collective Impact approach support sustainable approaches to address the social inclusion of seniors?
  6. (a) In what ways and to what extent do seniors participating in funded projects benefit from increased social participation and engagement in their communities?
    (b) In what ways and to what extent do communities benefit from the increased participation and engagement of seniors?

Annex B: Questions mapping

This annex illustrates the evaluation questions that are reflected in each of the 3 themes.

Theme 1: Impacts of the NHSP projects on seniors and communities

  • Evaluation question 2: To what extent has the program contributed to an increase in the participation and engagement of seniors in their communities?
  • Evaluation question 6 (a): In what ways and to what extent do seniors participating in funded projects benefit from increased social participation and engagement in their communities?
  • Evaluation question 6 (b): In what ways and to what extent do communities benefit from the increased participation and engagement of seniors?

Theme 2: How program funding has contributed to organizational capacity of organizations to deliver the program

  • Evaluation question 1: To what extent has the program improved the capacity of organizations to address barriers to social inclusion for seniors?

Theme 3: The application of the Collective Impact Model by the Pan-Canadian stream of the program and reporting on the early results

  • Evaluation question 3: To what extent has the Pan-Canadian Collective Impact approach been effectively designed and delivered?
  • Evaluation question 4: To what extent are early outcomes being achieved in the Collective Impact projects that could lead to systemic and population level changes?
  • Evaluation questions 5: To what extent does the program’s Collective Impact approach support sustainable approaches to address the social inclusion of seniors?

This table describes which evaluation question addresses which program outcome and under which stream. For accessibility reasons, the table has been simplified. Consult the PDF version for the full table.

Stream: Community-based and Pan-Canadian

Evaluation question 1: To what extent has the program improved the capacity of organizations to address barriers to social inclusion for seniors?

  • Immediate outcome 3: Recipient Organizations have capacity to support seniors initiatives in their communities
  • Immediate outcome 4: Recipient organizations recognize and address barriers to social inclusion faced by seniors

Stream: Community-based and Pan-Canadian

Evaluation question 2: To what extent has the program contributed to an increase in the participation and engagement of seniors in their communities?

  • Immediate outcome 1: Recipient organizations develop approaches to engage and retain volunteers
  • Immediate outcome 2: Participating seniors share their knowledge and experience with peers and different generations

Stream: Pan-Canadian

Evaluation question 3: To what extent has the Pan-Canadian Collective Impact approach been effectively designed and delivered?
Evaluation question 4: To what extent are early outcomes being achieved in the Collective Impact projects that could lead to systemic and population level changes?

  • Immediate outcome 3: Recipient Organizations have capacity to support seniors initiatives in their communities

Evaluation question 5: To what extent does the Program’s Collective Impact approach support sustainable approaches to address the social inclusion of seniors?

Stream: Community-based and Pan-Canadian

Evaluation question 6:

  1. In what ways and to what extent do seniors participating in funded projects benefit from increased social participation and engagement in their communities?
  2. In what ways and to what extent do communities benefit from the increased participation and engagement of seniors?
    • Immediate outcome 1: Recipient organizations develop approaches to engage and retain volunteers
    • Immediate outcome 2: Participating seniors share their knowledge and experience with peers and different generations

Annex C: New Horizons for Seniors Program outcomes

ESDC Departmental Strategic Outcome: Income Security, access to opportunities and well-being for individuals, families and communities

Ultimate outcome

Individuals, families, and communities mutually benefit from the participation of seniors.

Intermediate outcomes

  • Communities have the capacity to address local issues by engaging seniors
  • Seniors participate in, and contribute to, communities

Immediate outcomes

  • Recipient organizations develop approaches to engage and retain volunteers
  • Participating seniors share their knowledge and experience with peers and different generations
  • Recipient organizations have capacity to support seniors’ initiatives in their communities
  • Recipient organizations recognize and address barriers to social inclusion faced by seniorsFootnote 4

The evaluation team notes that the evaluation reference period (2015 to 2016 to 2018 to 2019) is not in line with the dates of the updates of logic model (2012 and 2017).

There are close similarities in outcomes between the logic models of 2012 and 2017. However, the evaluation team followed approved documentation and use of the most recent, 2017 version.

The evaluation team also notes that the misalignment of program implementation (meaning activities) and design (meaning logic model) is problematic. It may result in limitations to assessing all years of the reference period with the same outcomes.

Annex D: Lines of evidence

Literature review

The literature review draws on relevant international and Canadian literature. It addresses key issues and themes of interest, as defined by program indicators, including major trends, and discusses links to the program.

Document review

The document review was conducted between January and April 2020 and covered program activities for the aforementioned reference period. In addressing the evaluation questions, evaluators reviewed roughly 500 documents, including:

  • specific project documentation
  • relevant Government of Canada
  • academic sources
  • 2016 Statistics Canada census data

The document review also includes:

  • a qualitative analysis of 40 randomly sampled Community-based projects over the reference period
  • a quantitative analysis of administrative datasets, which includes data from Community-based projects funded from fiscal years 2015 to 2016 to 2018 to 2019 that completed final reports

Key informant interviews

Evaluators conducted 26 interviews between February and May 2020. Evaluators selected key informants based on 2 factors:

  • their high level of experience and knowledge of the program
  • their ability to speak to the evaluation questions and issues

Examples of key informants includes departmental regional committee members (capital region and regional staff), and funding recipients from both streams of the program.

Survey

The evaluation used an online survey with telephone follow-up for both streams of the program (Community-based and Pan-Canadian). The survey used a sample of organizations that received funding between 2015 to 2018. The Community-based stream had a population of 5,513 organizations. The survey invitation was sent by email to 2,000 randomly selected organizations from the sample. The response rate was 32.6%.

The survey sought to reach all Pan-Canadian organizations that received funding for the 3-year cycle, starting in 2015. This included 48 organizations that make up 9 collectives. The response rate was 50%.

Review of statistics Canada data

The analysis used data from Statistics Canada's Canadian Community Health Survey and General Social Survey. It sought to understand seniors' perspectives on issues relevant to program outcomes such as volunteering, social participation, and sense of community.

Annex E: Limitations and challenges

Data quality

The evaluation found limitations related to project administrative data from project files. Noted variations in project reporting variables presented a challenge for reporting on trends. In some cases, data was not available for consecutive years. As well, some key project administrative and performance data were new in the 2018 to 2019 fiscal year. This meant that data was unavailable for previous years within the reference period.

Subjectivity

The subjectivity of the participants and their stake in the program could have influenced the findings of this evaluation. Evaluators tried to avoid this challenge by seeking out examples from organizations or stakeholders and or cross-referencing data to confirm the findings.

Definitions

There are a number of terms used in the logic model and program documents that were not clearly defined. As such, evaluators encountered measurement challenges.

Attribution versus contribution

Immediate outcomes sometimes can be viewed as resulting directly from program outputs. However, intermediate and ultimate outcomes are usually not controlled solely by a single organization or program. Rather, an organization or a program contributes to, and influences the achievement of, these outcomes.

One can speak about progress “on” or “towards” the achievement of the outcomes. But, it is not possible to make claims about the causal effects of the program on those outcomes. This evaluation focuses on contribution.

The specific context of the COVID-19 pandemic impacted the evaluation in the following ways:

Case studies

The evaluation had included case studies at several locations across Canada. COVID-19 caused limits to gatherings. It also posed a risk to seniors participating in the case studies. As a result, the case studies were cancelled.

Interviews

The majority of interviews were completed between February and April 2020. For the few that took place during the pandemic, scheduling interviews were set up with flexibility and accommodation in mind. Interviewers paid attention to the sensitivities of the situation and needs of each respondent. In such times, respondents may not have been as clear or as focused in their responses had the interview taken place under different circumstances.

Survey

The timing of the survey in late May could have influenced:

  • how community based respondents answered questions related to challenges / obstacles:
    • 8% of respondents stated COVID-19 was a challenge in implementing their funded projects
  • the response rate of the Pan-Canadian respondents (50%):
    • respondents received the survey after the majority of potential respondents had learned they would not be awarded additional funding

Annex F: Program background

The New Horizons for Seniors Program is a grant and contribution program that aims to:

  • empower seniors by encouraging them to share their knowledge, skills, and experiences with others in the community
  • enhance seniors’ well-being and community vitality

The New Horizons for Seniors Program delivers 2 separate funding streams: Community-based and Pan-Canadian.

Community-based projects are inspired or led by seniors. They are volunteer-based, supported by communities, and facilitate activities where seniors are engaged, connected, and actively involved in their communities.

Calls for proposals (CFP) seek proposals for Community-based projects for which projects must address at least 1 of the following 5 objectives:

  1. promoting volunteerism among seniors and other generations
  2. engaging seniors in the community through mentoring of others
  3. expanding awareness of elder abuse, including financial abuse
  4. supporting the social participation and inclusion of seniors
  5. providing capital assistance for new and existing community projects and or programs for seniors

Note: Projects funded under Capital Assistance must also meet one other objective.

The Pan-Canadian projects, in 2015, aimed to address seniors’ social isolation using the Collective Impact Model. This model uses a cooperative, partnered approach. It brings together a group of collaborating organizations from different sectors to commit to a common agenda. The intent of its design is to produce significant changes in their community.

To meet the Pan-Canadian funding requirements, applications needed to demonstrate that they met the 5 main requirements for a Collective Impact Model:

  • a common agenda: having the same goal and agreement on how to resolve the issue
  • shared measurement: having common indicators to measure and report success
  • mutually reinforcing activities. This means that each set of activities in which collaborating organizations excel, aligned towards achieving the common agenda and shared measures
  • continuous communication: creating a good dynamic within the collaborating organizations by establishing trust, shared objectives, and motivation
  • the backbone organization: guiding and supporting the Collective Impact collaborating organizations as they work collectively to reach their common agenda

Pan-Canadian

Program objective:

  • funding is aimed at addressing seniors’ isolation through the Collective Impact Model
  • collective Impact is “An approach which brings together different [community] sectors for a common agenda to solve large complex problems…”Footnote 5
  • the approach obliges organizations to work together as a group or “Collective Impact Initiative”, which is composed of a backbone organization and collaborator organizations:
    • the backbone organization guides and supports other organizations in the Collective Impact Initiative and serves a key role in establishing shared measurement systems and data collection
    • the collaborator organizations play a critical part in implementing the projects and collect funding from ESDC
    • partner organizations are important in the delivery of the Collective Impact Initiative, but do not receive funds from ESDC

Delivery:

In the 2015 cohort, approved Pan-Canadian projects were eligible to receive up to $750,000 for up to 3 years in grant funding.

Funding:

  • from fiscal year 2015 to 2016 to 2018 to 2019, the average annual funding was $9 million
  • in the budget 2019, the Pan-Canadian stream funding allocation increased to $13.1 million
  • 238 applications were received and 48 (9 collectives) projects were approved for the fiscal year 2015 to 2016 cohort

Community based

Program objective:

  • Community-based projects must intend to meet at least one of 5 objectives:
    • promoting volunteerism
    • engaging seniors through mentoring
    • expanding awareness of elder abuse
    • support social participation and inclusion of seniors, and
    • providing capital assistance to new and existing community projects or programs for seniors
  • capital Assistance is defined by the program as: “material and equipment that organizations are able to buy, build or repair that offers new activities or ensures the continuity of current activities”

Delivery:

Approved Community-based projects are eligible to receive up to $25,000 per year (and per organization) in grant funding.

Funding:

  • funding allocated annually to Community-based projects averaged at $32.6 million, which has increased to $50 million since the end of the evaluation reference period
  • 13,698 applications were received and an average of 1,849 projects per fiscal year were approved

Annex G: Previous evaluation

Results from the 2015 evaluation of the New Horizons for Seniors Program

The 2015 Evaluation focused primarily on the Community-based project component of the program. Specifically, the evaluation concentrated on synthesizing evidence in lessons learned, promising practices, and success factors.

Lessons learned

The importance of:

  • tailoring projects to the needs and interests of the target population
  • developing partnerships prior to project start-up
  • nurturing of existing partnerships

As well, as the significance of Regional Service Canada staff in assisting some organizations to develop better quality applications.

The need to communicate with applicants and funded recipients in a timely manner to ensure an adequate understanding of Program requirements:

Promising practices

  • Teaching seniors how to use technology and social media to enable isolated seniors to stay in touch with their family, friends and their communities
  • Making use of video and audio media to enhance the reach of project activities
  • Improving the accessibility of facilities for volunteers with disabilities
  • Community engagement and promotion strategies that focus on geographical areas where there is a disproportionate number of funded projects works well

Success factors

Strong administrative capacity and project management skills, a presence of supportive partnerships and a sufficient level of senior involvement in project activities.

Elements that detract from projects’ success in achieving results and objectives include:

  • broad or unfocused project objectives
  • change or loss of project leadership

The overall evidence from the evaluation indicated that the program is making progress towards achieving its direct outcomes.

The evaluation had 3 recommendations:

  • streamline and focus the collection of project performance information in an accessible format to facilitate program monitoring and evaluation. Support the identification and dissemination of promising practices that can be replicated in other communities
  • recognizing the needs and preferences of seniors, leverage traditional methods of communication and external stakeholder organizations to complement Modernization efforts in support of program delivery
  • explore ways to streamline the administration of the program and reduce processing time

Annex H: Organizational capacity as a condition to receive program funding

The program collects and relies on data to inform decision making, however data management processes may be hindering the equitable access to the program funds.

Data for the same organization is not recorded in the same way every year. In order to understand the number of times organizations received funds, Evaluation isolated the organizations by using the following variables:

  • organization legal name
  • organization email address
  • contact name
  • telephone number

This means that some organizations that are applying for and receiving program funding are using slightly different applicant information every year. If the applicants use different information every year, reviewers would not be able to find the saved associated files from previous years.

Evaluation noted that the data in the database differs from the data that National Headquarters supplied to Evaluation regarding the number of times organizations were funded. This may suggest that the reviewers of applications may not have a fulsome picture of the organization’s history. This may in turn, influence their decision-making.

After isolating each organization, Evaluation determined that on average, roughly half of the organizations received funding more than once in the reference period.

Figure 22: Percentage of Community-based organizations and/or individuals funded once and more than once between fiscal year 2015 to 2016 and 2018 to 2019
Figure 22: Percentage of Community-based organizations and/or individuals funded once and more than once between fiscal year 2015 to 2016 and 2018 to 2019 - Text description follows
Figure 22 – Text version
Year Percentage of Community-based organizations and, or individuals funded once Percentage of Community-based organizations and, or individuals funded more than once
2015 43.4% 56.6%
2016 42.5% 57.5%
2017 40.7% 59.3%
2018 50.9% 49.1% 
  • Note: Organizations that are funded more than once between fiscal year 2015 to 2016 and 2018 to 2019 may not necessarily have been funded in consecutive years and are represented in each year that they received funding.
  • Source: Program Documents of the NHSP, fiscal year 2015 to 2016 to 2018 to 2019.

Despite efforts to encourage first time applicants, the inability to review information about how often organizations receive funds was a problem. It may result in awarding funds inadvertently to those organizations that have already gained organizational capacity. Receiving funds often may allow for organizations to gain capacity through implementation of their projects. As such, they are able to build and demonstrate their capabilities further.

Annex I: Defining organizational capacity

In the absence of a clear definition, Evaluation sought to understand more fully what capacity means.

The results of several lines of evidence point to common indicators of capacity. These indicators could include the ability to attract and retain participants and volunteers.

Ability to attract and retain participants and volunteers

There is a strong sentiment amongst key informants that senior participants and volunteers are often inter-changeable throughout the project’s implementation. This means that seniors may start out participating in projects and over time may become a volunteer in running them. Likewise, volunteers in the projects sometimes are not able to continue their commitments to the project, but they may be a consistent participant.

This is significant because it means the reported data on participants or volunteers could be inaccurate.

The success of the program is closely linked to the organization’s ability to attract participants to take part in the activities. Volunteers are likewise integral because of the role they play in delivering the projects. The key informant interviews, the survey and the document review identify that organizations could attract and train participants.

This indicator could be applicable to both streams of the program. Through numerous lines of evidence, recruiting and retaining participants and volunteers appeared to be less of a challenge for the Pan-Canadian funded organizations.

The ability to develop partnerships

Partnerships were seen as contributing to organizational capacity. Keeping in mind that the broad definition of organizational capacity may have influenced this result.

Interviews revealed that participants interpreted ‘partnerships’ in many different ways. Partners could be a neighbour that helps move boxes on the weekend. Partners could also be a community organization that engages in a formal partnership with a Memorandum of Understanding.

The wide interpretation of ‘partnership’ poses a challenge. This is because the partnership role was not consistent to all organizations in the support they offered. Therefore, the extent to which partnerships impacted organizational capacity to deliver programs that address the barriers to senior’s social inclusion is unknown.

Community-based and Pan-Canadian funded organizations develop capacity by learning how to attract participants and volunteers

The document review revealed some activities to promote seniors to volunteer, including:

  • recruiting their friends or peers (75%)
  • holding information sessions or workshops (39%)
  • calling seniors on the phone to encourage their volunteerism (34%)

The key informant interviews indicated similar activities were successful in gaining volunteers.

Figure 23: Proportion of approaches reported to promote volunteerism averaged over fiscal year 2015 to 2016 to 2018 to 2019 (Community-based stream)
Figure 23: Proportion of approaches reported to promote volunteerism averaged over fiscal year 2015 to 2016 to 2018 to 2019 (Community-based stream) - Text description follows
Figure 23 – Text version
Approaches Reported to Promote Volunteerism Averaged over fiscal year 2015 to 2016 to 2018 to 2019 (Community-based stream) Proportion
None, not applicable 8%
Other 26%
Friends, peers 75%
Social media 29%
Web material 19%
Print material 31%
Mailing 10%
Email 32%
Telephone 34%
Local media 30%
Information sessions 39% 
  • Source: Document Review Technical Report of the Evaluation of the NHSP (2020).

The evaluation unearthed some useful approaches to improve volunteer management, including:

  • identifying strengths and abilities of volunteers and placing volunteers in roles where they will be most effective
  • managing the amount of time volunteers act in any given role so that they have opportunities
  • monitoring and supervising volunteers to maintain engagement and avoid problems

The evidence speaks to the ability to gain new and build lasting partnerships as an indicator of capacity

The key informant interviews and the document review suggest that there is a wide interpretation of what partnership includes. Partners could contribute both financial and in-kind resources, and could include any person, organization or group that is connected to the funded organization.

Some examples of a partner could be:

  • a school that runs an inter-generational project with a community-based organization
  • a community centre that lends a space for seniors to meet
  • an individual that lends their time to help set up for an event
  • another organization in the community that trades time or services in exchange for the same and
  • a community member’s financial contribution

Organizations that receive program funds are able to build partnerships

Figure 24: Average percentage of the types of in-kind contributions provided by partners between fiscal year 2015 to 2016 and 2018 to 2019
Figure 24: Average percentage of the types of in-kind  contributions provided by partners between fiscal year 2015 to 2016 and 2018 to  2019 - Text description follows
Figure 24 – Text version
The types of in-kind contributions provided by partners between 2015 to 2016 and 2018 to 2019 Average percentage
Not applicable 7%
Other 42%
Material supplies 55%
Administrative support 55%
Physical space 61%
Knowledge, expertise 85% 
  • Source: Document Review Technical Report of the Evaluation of the NHSP (2020).

Community-based organizations learned from their partners

Figure 25: What partner organizations learned from each other
Figure 25: What partner organizations learned from each other - Text description follows
Figure 25 – Text version
What partner organizations learned from each other Percentage
Ability to learn, share, borrow strategies 47%
Success from using collaboration, partnerships 41%
Value of networking, mutual benefit 33%
Cultural, intergenerational, development, appreciation 25%
Access to, utilization of resources, funding 23%
Other 4%
We didn’t learn from each other 1%
No comment 8%
Don’t know 4% 
  • Source: Survey for the Evaluation of the NHSP (2020), 362 Community-based funding recipient respondents.

Community-based organizations were able to spread awareness of their project through their partnerships

Figure 26: Organizations expressed whether partners helped in gaining participants, volunteers, and awareness
Figure 26: Organizations expressed whether partners helped in  gaining participants, volunteers, and awareness - Text description follows
Figure 26 – Text version
Organizations expressed whether partners helped in gaining… Don’t know (%) No (%) Yes (%)
Awareness in the community 3% 4% 93%
Participants 3% 8% 88%
Volunteers 2% 19% 79% 
  • Note: Totals may not sum to 100% due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 362 Community-based funding recipient respondents.

Pan-Canadian funded organizations learned to deepen their collaborative partnerships

Like the Community-based stream, several lines of evidence identified similar indicators of capacity for the Pan-Canadian stream:

  • recruiting participants and volunteers
  • developing partnerships
  • 91% of funded organizations had been in operation for over 10 years

Key informant interviews suggested that the organizations have been in operation for many years. Organizations said their strong abilities to engage and retain participants and volunteers, as well as their confidence to deliver programming to seniors.

Also, organizations described that the most challenging aspect in implementing the Collective Impact Model was partnership development and collaboration.

The survey of Pan-Canadian funded organizations confirmed this view:

  • 63% Pan-Canadian respondents felt that collaborating with / gaining partners was the most challenging
  • 38% evaluating the project
  • 29% not enough time
Figure 27: Challenges in implementing the Collective Impact Plan
Figure 27: Challenges in implementing the Collective Impact Plan - Text description follows
Figure 27 – Text version
Challenges in implementing the Collective Impact Plan Backbone Collaborator Backbone and Collaborator
Collaborating with, gaining collaborators, partners 17% 46% 63%
Evaluation of project 4% 33% 38%
Not enough time 8% 21% 29%
Writing Collective Impact Plan, final reports 4% 13% 17%
Staff and volunteer turnover 0% 13% 13%
Geographic 4% 4% 8%
Continuing the project after funding done 0% 4% 4%
Raising awareness about project 0% 4% 4%
Other 0% 4% 4%
No challenges 0% 4% 4%
Don’t know 0% 8% 8% 
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

96% of survey respondents agreed that they learned from other organizations in their Collective Impact Initiative

The Pan-Canadian funded organizations specified that they learned through:

  • their teamwork and coordination (71%)
  • their sharing of information with each other (63%)
  • in their support of seniors’ needs (33%)

Annex J: Is Organizational capacity a process, an achievement or both

What is the program’s vision about organization capacity

The program’s view of organizational capacity as an achievement or a process affects the way the program engages with organizations. The way that they view ‘organizational capacity’ could influence a number of aspects of the program’s design and prospects for success:

  • the selection criteria that outlines which organizations get funding
  • the goals that the program is trying to achieve
  • the activities that the organizations carry out
  • the performance measures that monitor progress

If organizations can achieve capacity and capacity is a state in time then there would be no need to gain expertise or grow abilities. This is the case because organizations have already established it. To see capacity in this way, the funding may be seen as short term (project-based).

Project-based funding provides one-time funding for projects without the expectation that the project would continue or receive additional funding in the future.

If organizations engage in a process that builds their capacity over time, it is necessary to recognize that the ‘organizational strength’ may fluctuate.

Funded Community-based organizations often experience high turnover in staff and volunteers. This means there could be a gap in capacity between what the organizations can do year over year. It also means there may be an interruption in the flow of organizational learning.

To see capacity in this way, the funding may be seen as longer term (seed funding).

Seed funding means that there is an intent for the project to continue and grow after the initial funding is over.

There is also the possibility that capacity could be both an achievement (project-based) and a process (seed funding). The distinction is not clear. Measuring capacity is significant as it would allow us to understand at what point in time organizations have it. It would also allow us to understand what organizations are doing to maintain it.

Annex K: Barriers to social inclusion

Determining the barriers to social inclusion for seniors

Despite a lack of definition of organizational capacity, evaluation determined what some indicators could be. In using these indicators, evidence points to the projects contributing to the capacity of organizations to address the barriers to social inclusion.

The following section will outline common barriers to social inclusion identified by both streams of the program. It will also assess whether activities undertaken seek to address those barriers.

Transportation, distance, mobility issues

This is the most dominant barrier. Seniors, in general, feel disappointed when a project requires travel. This could be for a variety of reasons, including self-confidence, mobility, health and family related issues. If transportation is necessary, it is imperative that the mode of transport meet the needs of seniors (such as, subways with elevators or buses with handlebars).

Awareness and knowledge

Word of mouth is the best way to spread awareness. New participants are more likely to come out when they already know someone who is participating. Awareness and knowledge about the projects including the participants, the activities, the accessibility of the space and the organizations themselves, as well as when and where the project is taking place is a barrier.

Poverty and low income (social determinants)

  • Adequate housing
  • Healthy meals
  • Access to medical services

Social isolation

The conditions to be social are not sufficient for seniors to be comfortable and confident in being social.

Language and cultural challenges

Seniors may not feel confident being social with others from a different language or culture.

Activities that funded projects implemented provides some insight into how the projects addressed the barriers to social inclusion

The survey of Community-based organizations showed the most popular activities that project engaged are:

  • social gatherings (61%)
  • learning events (42%)
  • capital renovations / repairs (32%)
Figure 28: Activities undertaken as part of the project
Figure 28: Activities undertaken as part of the project - Text description follows
Figure 28 – Text version
Activities undertaken as part of the project Percentage
Social Gathering 61%
Learning event 42%
Capital renovations or repairs 32%
Arts inspired 19%
Exercise 17%
Projects senior led 13%
Address local issue 7%
Day trips 5%
Other 3%
Don’t know 3% 
  • Source: Survey for the Evaluation of the NHSP (2020), 583 Community-based funding recipient respondents.

The survey of Pan-Canadian organizations revealed the activities that organizations engaged in as part of the Collective Impact Plan:

  • research (58%)
  • programming and social activities (50%)
  • connecting seniors to the services they need (46%)
  • social gatherings (38%)
  • learning events (33%)
Figure 29: Activities that organizations and partners engaged in as part of Collective Impact Plan
Figure 29: Activities that organizations and partners engaged in  as part of Collective Impact Plan - Text description follows
Figure 29 – Text version
Activities that organizations and partners engaged in as part of Collective Impact Plan Backbone Collaborator Backbone and collaborator
Research 8% 50% 58%
Programming and social activities 13% 38% 50%
Connecting seniors to services they need 4% 42% 46%
Social gathering 13% 25% 38%
Learning event 4% 29% 33%
Recruit and train volunteers 4% 17% 21%
Home visits and telephone calls 4% 13% 17%
Raising awareness 4% 13% 17%
Developing technology-based tools 4% 8% 13%
Transportation 8% 0% 8%
Address local issue 0% 8% 8%
Other 4% 8% 13%
Don’t know 0% 8% 8% 
  • Note: Totals may not add up due to rounding.
  • Source: Survey for the Evaluation of the NHSP (2020), 24 Pan-Canadian funding recipient respondents.

Do the projects address the barriers to social inclusion

It is not always clear how the project activities of both streams of the program address the barriers to social inclusion.

Evidence from the Community-based organizations suggests that projects emphasize the ‘social’ aspect of implementing the projects.

Organizations may focus on trying to attract participants by incorporating cultural components in to the project. They may also try to raise awareness through advertising. In so doing, they are addressing awareness as a barrier, as well as barriers posed by differences in language or culture.

Key informant interviews and surveys showed that transportation is a dominant barrier that is top of mind for many groups involved in the evaluation.

Sometimes seniors’ lack confidence in their mobility when using some types of transportation, such as taking subways or getting on a bus. This is closely related to health issues and social determinants of health. Some seniors may not be mobile. Some seniors may not have money to pay for specialized transportation or assisted devices that would help with their mobility.

Despite transportation being a dominant challenge, the activities that projects engaged in don’t address it. This suggests that organizations are not able to overcome this challenge

In the same way, 3 lines of evidence described poverty or low income as a key barrier. This is a complex barrier to solve and one that some organizations have attempted to address in their own way (such as meal delivery).

Breaking down this barrier to allow low-income seniors to participate in projects is difficult. There is some anecdotal evidence of projects addressing low-income issues. However, there is little evidence to suggest that most organizations have made progress in addressing this barrier.

Seniors with low levels of income were often a population that Community-based organizations tried to reach. However, projects were not often designed to address it as a standalone barrier to social inclusion.

By focusing on target audiences in the design of their Collective Impact Plan, Pan-Canadian funded organizations narrowed the focus of possible beneficiaries. This may have resulted in overlooking other barriers.

Report a problem or mistake on this page
Please select all that apply:

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: