Evaluation of the New Horizons for Seniors Program

Official title: Evaluation of the New Horizons for Seniors Program, Final Report, Prepared December 14, 2015 by the Evaluation Directorate, Strategic and Service Policy Branch, Employment and Social Development Canada

Executive Summary

This report presents a summary of the findings for the Evaluation of the New Horizons for Seniors Program (NHSP). Footnote 1 Launched in 2004, the NHSP is a grant and contribution program that aims to enable seniors to contribute to their communities through greater involvement and volunteerism. The NHSP was created in recognition of the knowledge that the participation of seniors in their communities is essential to their well-being and to the well-being of their communities.

Currently the program receives $50 million per year to fund two distinct streams, community based projects and pan-Canadian projects. Through these two streams, the program has funded almost 16,000 projects in over 1,000 communities since 2004. Each fiscal year, approximately 1,800 community-based projects are funded for an average funding amount of $19,000 per project. An analysis of 2011-2012 funded projects indicates that the majority of projects support the social participation and inclusion of seniors.

The evaluation focuses on the community-based project component of the program. The scope of the evaluation was calibrated to address the information needs of program management. Specifically, the evaluation concentrated on synthesizing evidence in three areas: (1) lessons learned (strengths and challenges that have been identified and are to be avoided in the future); (2) promising practices (approaches which have contributed to the achievement of results in particular contexts, but have not been replicated in others); and (3) success factors (elements of a project which contribute to achieving aims). See Annex C for the Glossary of Terms. Evaluation findings cover a four-year period (fiscal year 2011-2012 to 2014-2015).

Key Findings

In relation to program relevance, the NHSP is well placed to address the continued needs of Canada’s increasingly aging population. The program aligns with both the federal government’s role and responsibilities with respect to the well-being of seniors and other vulnerable groups, ESDC’s strategic outcome of supporting income security, access to opportunities and well-being of individuals, families and communities.

Focusing on community-based organizations, the evaluation identified the following lessons learned: tailoring projects to the needs and interests of the target population; developing partnerships prior to project start-up; and the ongoing nurturing of existing partnerships. In relation to program design and delivery, key lessons learned include the importance of regional Service Canada staff in assisting some organizations to develop better quality applications and the need to communicate with applicants and funded recipients in a timely manner to ensure an adequate understanding of NHSP requirements.

Evaluation evidence of promising practices include the following: teaching seniors how to use technology and social media to enable isolated seniors to stay in touch with their family, friends and their communities; making use of video and audio media to enhance the reach of project activities; and improving the accessibility of facilities for volunteers with disabilities. From the perspective of NHSP regional Service Canada staff, community engagement and promotion strategies that are focused on geographical areas where there is a disproportionate number of funded projects was found to work well.

Factors contributing to project success included strong administrative capacity and project management skills, a presence of supportive partnerships and a sufficient level of senior organizational involvement in project activities. Elements that were found to detract from projects’ success in the achievement of results and objectives included broad or unfocused project objectives and change or loss of project leadership.

ESDC currently identifies community-based project success stories on the Departmental website; however, ESDC lacks a systematic process for identifying and communicating community-based lessons learned, promising practices, and project success factors, which were identified by program stakeholders as being of value to further program improvement.

Overall, evidence on program performance indicates that the NHSP is making progress towards achieving its direct outcomes. However, key program performance information is collected in a format that is challenging to analyze. Specifically, data for analysis is manually extracted from paper files (converted to PDFs), which makes it a highly resource intensive activity to determine project results. Furthermore, other than basic applicant information, the data elements that are entered into the Common System for Grants and Contributions (CSGC) database’s Project Life Cycle (PLC) module have limited usefulness for assessing program performance.

Reducing the administrative burden associated with the application screening and assessment process by regional Service Canada staff, as well as the length of time associated with the application review and approval process, were found to be ways in which the program could be made more efficient.

Recommendations

  1. Streamline and focus the collection of project performance information in an accessible format to facilitate program monitoring and evaluation. Support the identification and dissemination of promising practices that can be replicated in other communities.
  2. Recognizing the needs and preferences of seniors, leverage traditional methods of communication and external stakeholder organizations to complement Modernization efforts in support of program delivery.
  3. Explore ways to streamline the administration of the program and reduce processing time.

Management Response

The key findings outlined in the evaluation final report are generally positive and indicate that the community-based project funding stream is consistent with the federal and departmental policy directions and priorities. Drawing on multiple lines of evidence, the evaluation also confirms that the Program is making progress towards achieving its designed objectives and outcomes. The evaluation also sheds light on specific areas for review and improvement, which will be beneficial to program stakeholders, more specifically with respect to exploring and identifying key NHSP project success factors and promising practices. Three recommendations are provided in the evaluation final report. These recommendations are important and the Department will act on them.

Recommendations and responses

1. Streamline and focus the collection of project performance information in an accessible format to facilitate program monitoring and evaluation. Support the identification and dissemination of promising practices that can be replicated in other communities.

The Department agrees with this recommendation. Performance measurement is a critical part of the effective end-to-end management of publicly funded programs and executing this function well depends on the quality and accessibility of information. In addition to the value of data to inform program policy, design and delivery, evidence gathered from each project and analyzed in a collective way will inform other measures taken to identify and share promising practices with communities.

Actions taken
  • In 2012-2013, the final report template to collect community-based project data was modified from a qualitative instrument to one that is now quantitative. Given the volume of projects (i.e., approximately 1,800 annually) and their short duration, this modification allows for more comprehensive analyses of results. Although the data is self-reported, a better sense of how projects have influenced seniors and communities can now be better articulated and shared.
  • Recent efforts were made in 2015-2016 to gather and write success stories. A total of seventeen (17) success stories were written based on information collected from the regions, one of which is published on the Web and serves to promote the program during community engagement in the North.
  • Over the past few years the program has also reinforced the importance of submitting final reports. Starting in 2013-2014, the following sentence was added in the applicant guide:

‘’Reporting on time shows good project management. Should your organization fail to submit a final report on time it may impact your organization’s ability to obtain funding from ESDC in the future.’’

Actions proposed
  • Continue to explore ways to streamline the final report form in light of data relevance.
  • Explore ways to improve the collection of key program performance data, including information on project success factors and stories.
  • Explore the capacity of the Grants and Contributions Online System (GCOS) to capture key performance data.
  • Explore the use of existing CSGC Project Life Cycle fields for the reporting of project success stories and best practices by Service Canada staff.
  • Using new and existing NHSP material, explore ways to showcase project success factors and stories to better engage potential applicants.

2. Recognizing the needs and preferences of seniors, leverage traditional methods of communication and external stakeholder organizations to complement Modernization efforts in support of program delivery.

The Department agrees with this recommendation. The landscape of Canada’s seniors’ population is changing rapidly, and the Department recognizes that there is a digital and urban/rural divide amongst this cohort. The nature of tailored communications to engage seniors’ communities and further tap into non-governmental networks are valuable areas for further exploration.

Actions Taken
  • The Department has maintained overall multimedia outreach activities to potential applicants Footnote 2 including in-person community engagement sessions and through webEx sessions.
  • In addition, special efforts were made within the context of the 2015-2016 call for proposals to reach out to NHSP proponents located in the northern region using letters from the Minister and articles to targeted media.
Actions Proposed
  • Continue to perform targeted outreach activities in the northern region, using existing multimedia methods.
  • Continue to review existing tools, material and methods used for current NHSP community-based activities in light of program clientele needs and preferences, in a cost-efficient manner.
  • Consider planning and implementing more targeted engagement activities with national and regional senior stakeholder organizations.

3. Explore ways to streamline the administration of the program and reduce processing time.

The Department is aware of challenges in program administration and agrees with this recommendation. It intends to explore ways to deliver NHSP community-based funding in a more timely fashion while being cognizant of the need to maintain sound financial stewardship and program oversight.

Actions Taken
  • Planning is underway to establish measures to gain efficiencies in the application review and approval processes. For instance, the role of the Regional Committees has recently been streamlined to minimize some of the steps performed by members.
Actions Proposed
  • Map the current community-based projects process to identify opportunities to further standardise and streamline the application screening and assessment process.
  • Review the roles and responsibilities of ESDC officials and Regional Committees in the assessment and recommendation process to ensure relevance and to achieve greater efficiency.
  • Ensure that initiatives related to the Modernization agenda, such as the implementation of the GCOS, contribute to greater process efficiencies.

1.0 Introduction

This report presents a summary of the findings from the Evaluation of the New Horizons for Seniors Program (NHSP). Results presented in this report are drawn from evidence collected through multiple lines of qualitative and quantitative inquiry.

1.1 Overview of the New Horizons for Seniors Program

Launched in 2004, the NHSP is a grant and contribution program that aims to enable seniors to contribute to their communities through greater involvement and volunteerism. The NHSP was created in recognition of the knowledge that the participation of seniors in their communities is essential to their well-being and to the well-being of their communities. In 2014, the program’s budget was increased by $5 million per year, for a total funding envelope of $50 million per year for the two streams of the program: 1) pan-Canadian projects, and 2) community-based projects. Footnote 3 For fiscal year 2014-2015, the community-based projects component of NHSP accounted for 70% of total program expenditure; while pan-Canadian projects accounted for approximately 16%. Approximately 14% goes towards regional and national program delivery. Footnote 4 According to the program’s Terms and Conditions, eligible community-based projects can be funded under any of the following five objectives:

  1. Promoting volunteerism among seniors and other generations.
  2. Engaging seniors in the community through mentoring of others.
  3. Expanding awareness of elder abuse, including financial abuse.
  4. Supporting social participation and inclusion of seniors.
  5. Providing capital assistance for new and existing community projects and/or programs for seniors.

Delivered at both the national and regional level, NHSP funded projects contribute to the achievement of the program’s direct outcomes, which are identified in the Performance Measurement Strategy as the following:

  • Direct Outcome 1: Recipient organizations adopt approaches to engage volunteers.
  • Direct Outcome 2: Participating seniors share their knowledge and experience with peers and different generations.
  • Direct Outcome 3: Recipient organizations have capacity to support seniors’ initiatives in their communities.
  • Direct Outcome 4: Recipient organizations equip community members to recognize abuse of seniors.

Pan-Canadian projects are administered by the National Headquarters Grants and Contributions Delivery Centre and in the past have focused on increasing the awareness of elder abuse, including financial abuse. Footnote 5 The more recent 2015-2016 call for proposals for pan-Canadian projects addresses the theme of social isolation among seniors and explores the use of social innovation approaches (e.g. social partnerships).

The grants to community-based projects are administered by Service Canada, and intend to address social challenges at the local level and should be “inspired and led by” seniors, on a volunteer basis, who share their knowledge, skills and experiences with others. Footnote 6 Approved community-based projects are eligible to receive up to $25,000 per year, per organization in grant funding. Between fiscal years 2011-2012 to 2014-2015, approximately 1,800 community-based projects are funded each fiscal year with an average funding amount of $19,000 per project. Footnote 7

Analysis of 2011-2012 community-based projects by program management found that the majority of recipients are primarily funded under the objective of supporting social participation and inclusion of seniors (67%), with the remaining projects addressing mentoring (14%) and volunteering (12%). Footnote 8 The majority of project activities were found to contribute to two or more program objectives. Footnote 9 Moreover, findings on the profile of participants and beneficiaries also showed that the majority of projects targeted vulnerable populations (e.g. isolated seniors, persons with disabilities) and often reflect the local population composition. Footnote 10

Eligible community-based project recipients must identify which of the five program objectives a project is intended to address. The NHSP’s eligible recipients include, but are not limited to, the not-for profit sector, for-profit enterprises, municipal governments, educational institutions and Aboriginal organizations. Footnote 11

1.2 Evaluation Scope

The NHSP evaluation is calibrated to address ESDC program management’s principal areas of interest in three areas: (1) lessons learned, (2) promising practices, and (3) success factors related to community-based projects. Additionally, this evaluation was conducted in accordance with the Treasury Board Secretariat Policy on Evaluation (2009) and reports on the issues of relevance and performance (effectiveness, efficiency and economy), including design and delivery. This evaluation was conducted in compliance with Section 41.2 of the Financial Administration Act requiring the evaluation of ongoing grants and contribution programs every five years.

The evaluation reference period covers program activities between fiscal years 2011-2012 and 2014-2015, the period following the previous NHSP evaluation, with increased emphasis on the more recent years of program delivery. The methodology consisted of multiple lines of inquiry, including the following: a document review, which included a review and analysis of up to 100 NHSP projects between 2011-2014 as well as key program documentation; 45 key informant interviews with program experts, regional Service Canada staff, Regional Committee members and community-based projects; a focused literature review; and a review and analysis of available community-based project performance information. For more information on the methodology, please see Annex B.

2.0 Key Findings

2.1 Relevance

In the context of Canada’s aging population and importance of continued community involvement and volunteerism Footnote 12 of seniors Footnote 13 for their well-being and the well-being of their communities, there continues to be a demonstrated need for social development programming for seniors, which NHSP is well placed to address. The objectives of NHSP were found to align with the Government of Canada’s priorities as well as the social development aspect of ESDC’s strategic outcome of supporting income security, access to opportunities and the well-being of individuals, families and communities.

Continued need for the program

Seniors make up the fastest growing segment of the Canadian population. Footnote 14 Over the last 30 years, the number of Canadians 65 and over has more than doubled (up from 2,563,300 in 1984 to 5,585, 300 in 2014). Footnote 15 By 2033, it is projected that seniors will account for approximately 24% of the population, up from approximately 16% in 2014.

Studies indicate that volunteering and community engagement are associated with improved health and well-being, including reduced morbidity and mortality for seniors. Footnote 16 Recently, the Government of Canada’s Action for Seniors (2014) report presented by the National Seniors Council also noted that active participation and involvement of seniors in their communities enables seniors to contribute to their communities and improve their overall health and well-being. Footnote 17

Through the data collected in the key informant interviews and document review, the evaluation found that funded projects are aligned with and contribute to the program’s overall objective of enabling seniors to contribute to and benefit from their communities. Most of the NHSP funded recipients indicated that their project was developed in direct response to an identified need in their community through the conduct of a needs assessment and consultations with seniors. Regional Committee members highlighted that NHSP constitutes a key source of funding for small senior-led not-for-profit groups.

As found in the previous Formative Evaluation of the New Horizons for Seniors Program Capital Assistance Component (2010), the capital assistance component of the program seeks to address concerns that new and existing community-based projects geared towards seniors are at risk from deteriorating facilities and equipment. Footnote 18 Findings from a review of sampled projects conducted for this evaluation indicated a high percentage of projects being funded under the objective of capital assistance, suggesting a continued need for this objective. Footnote 19 NHSP funding is also made available for community-based projects that expand awareness of elder abuse, including financial abuse, which reflects the findings of a report by the National Seniors Council that identifies elder abuse as an under reported phenomenon and a priority that “has remained a largely hidden issue.” Footnote 20

Alignment with government priorities and federal roles and responsibilities

The Government of Canada has consistently reaffirmed its commitment to advancing NHSP’s objectives. Budgets 2005, 2007 and Economic Action Plans 2008, 2010, 2011 and 2014 included continued and increased funding for the program from an initial budget of $8 million Footnote 21 to $50 million annually. Footnote 22

As a signatory to the United Nation’s Madrid International Plan of Action on Ageing, the Government of Canada is committed to ensuring that older persons “age with security and dignity, and continue to participate in their societies as citizens with full rights.” Given that the needs of seniors are complex and multifaceted, the federal government collaborates with provincial or territorial and municipal governments as well as non-government organizations and the private sector to address the needs of seniors. Delivered by ESDC, the NHSP supports the social development aspects of the Department’s strategic outcome of supporting income security, access to opportunities and well-being of individuals, families and communities.

2.2 Design and Delivery

The document review and key informant interviews informed the identification of lessons learned, promising practices, and success factors for community-based projects, which constituted the primary focus of this evaluation (see Glossary of Terms in Annex C). These findings covered various elements of the project life cycle (i.e. design, implementation, delivery) and of program administration (e.g. community engagement, disseminating factors of project success).

Lessons learned in relation to projects

In synthesizing evaluation evidence that brings forth challenges which could be avoided in the future, three lessons learned are noteworthy. First, early engagement of communities by regional Service Canada staff would foster program awareness. Second, assisting organizations to develop sound applications would be of value for organizations that lack expertise. And third, timely communication with applicants to ensure an adequate understanding of NHSP requirements could contribute to stronger applications and ultimately, more effective projects.

The majority of regional Service Canada staff indicated that adequate monitoring of projects and continuous engagement with applicants and funded recipients throughout the lifecycle of the project is necessary to effectively oversee and support the successful outcome of projects. Related to the importance of Service Canada being able to provide timely advice to projects, engaging organizations in the lead up to the call for proposals to foster awareness was found to be a lesson learned from the perspective of multiple key informant groups.

Some key informants (Regional Committee members, regional Service Canada staff) expressed that NHSP should engage organizations (in particular those that lack sufficient capacity) to a greater extent in order to better support the development of sound applications and promote new and innovative project activities. While regional Service Canada staff as a part of their community engagement strategy offer workshops to assist potential applicants with the development of project applications, a key informant with significant knowledge of the program identified community engagement as an “area for improvement.” See Annex F for additional details regarding lessons learned in relation to community-based projects.

Lessons learned in relation to partnerships

For community-based organizations, the development of partnerships prior to project start-up and the ongoing nurturing of partnerships were cited as key lessons learned. Findings from the document review and key informant interviews indicate that most partnerships are with the non-profit sector, with the most prevalent type of support garnered from partners being in-kind support, in the form of time, meeting space, volunteers and resources (e.g. equipment).

Approximately half of regional Service Canada respondents who were able to identify lessons learned in relation to partnerships identified the importance of developing and nurturing partnerships prior to the start of the project, as well as the important role of partners in facilitating linkages between organizations and the community. Linkages between recipient organizations, their partners and the community were found to generate greater awareness of project activities within the community, as well as the exchange of ideas, both of which ultimately benefit the project. A few additional regional Service Canada respondents pointed out that partnerships are often “informal” and that the organizations “tend to go to people they know.”

In a similar vein, organizations reported that having an open communication platform to discuss expectations and commitments was one of the more important lessons learned since “informal partnerships are based on trust.” Similarly, from the perspective of Regional Committee members, open communication in order to foster shared commitment and expectations in advance of the project start-up was the most frequently cited lesson learned.

It was noted that a key factor that may hinder the success of partnerships is the lack of organizational expertise and resources on how to establish and nurture partnerships on the part of community-based organizations. See Annex G for additional lessons learned in relation to partnerships.

Promising practices

In addition to lessons learned, findings from the document review and key informant interview process identified project approaches and activities that were found to be related to achieving project aims in particular contexts. Replicating these promising practices across a broader context, while yet to be determined, holds potential for further improving the program.

For community-based projects, improved accessibility for volunteers, projects that are targeted to vulnerable segments and needs of the project target population, training seniors in the use of technology and social media and use of video and audio media to enhance the reach of project activities were found to be promising practices. For regional Service Canada staff, community engagement and promotion strategies that are focused on geographical areas with low program uptake were noted as a promising practice.

Success factors

Overall among projects, strong administrative capacity and project management skills on the part of community-based organizations and sufficient level Footnote 23 of senior involvement in project activities were found to constitute key factors of project success. Footnote 24 In contrast, the timing of project activities, adequacy of the maximum project funding amount and restrictions regarding transportation costs were found to detract from successful project outcomes across the board.

Citing the quality of applications as a proxy for gauging the administrative capacity and project management skills of applicants, regional Service Canada respondents noted sound administrative capacity (e.g. realistic project outcomes), partnerships (e.g. in-kind support) and sufficient involvement of seniors, and in particular those most in need, as conducive to the successful outcome of projects. Footnote 25 Similarly, a review of previously funded projects found that those with focused and concrete objectives were more likely to achieve their intended outcomes. The review also noted that for projects that target a relatively small segment of the senior population (e.g. ethno-cultural groups), funded recipients enlisted the aid of community organizations with existing networks to better reach the project target population.

Organizations also cited project specific factors that make it easier for potential project participants to access project activities. These included delivering activities in the preferred language of participants, especially in areas where the majority or a large portion of participants are from ethno-cultural minority communities. Delivering the project in a location that is convenient for participants, such as a public library or a religious institution (e.g. Church, Temple) where the target population is used to attending, was also found to contribute to the successful outcome of the projects.

Beyond controlling for the aforementioned factors of project success, the evaluation also identified factors that generally detract from the successful outcome of projects, including the timing of project activities (e.g. winter). Related to this issue, the National Seniors Council also identified project accessibility and transportation for those with disabilities or living in rural and remote areas as key factors that contribute to seniors’ volunteer involvement. Footnote 26 Program funding restrictions with respect to transportation costs were noted by some respondents as potentially detracting from project success. Footnote 27 Similarly, the literature review found that successful volunteer programs address impediments to civic engagement, including life circumstances, of which access to transportation is one of the most important. Footnote 28

Related to the adequacy of the $25,000 maximum funding amount for community-based projects, some key informants noted that for projects that exhibit unique or higher costs such as those in rural or remote areas (e.g. high cost of renting space in northern communities), the funding limit of $25,000 is often insufficient and may detract from its capacity to fully address the needs of seniors in remote and rural communities. It was suggested that increasing the funding amount a project receives may enhance the quality and impact of projects. Moreover, the absence of NHSP funding was cited by the majority of respondents from recipient organizations as a key factor that could have detracted from their capacity to support seniors’ initiatives in their communities. See Annex E for a list of key factors of project success.

Sharing what works

The evaluation found that although sharing of success stories takes place on an on-going basis, the community-based project component of NHSP could benefit from a more systematic process for identifying and sharing lessons learned, promising practices, and success factors as these are seen to hold value for program stakeholders.

From the perspective of regional Service Canada staff, key informants noted that the process for identifying success stories is ad hoc and starts with a request from national headquarters that is relayed through the Senior Development Officers and Business Expertise Consultants, who in turn solicit examples from the Project Officers. It is worth noting that the identification of a success story does not necessarily correlate with identifying lessons learned, promising practices, or success factors. Footnote 29 Requests to identify success stories do not include formal criteria or standardized processes (e.g., definitions, criteria, measures) for selecting a project. Moreover, given that the request pertains to identifying success stories at the end of the project life cycle, Project Officers reported relying on their memory with respect to projects that stood out to them.

Once stories are collected, the NHSP makes use of the ESDC success stories website as the primary means of sharing them. Additionally, key program documents (e.g. applicant guide, factsheets) are used to inform potential applicants and internal personnel of key aspects of strong applications and projects (for a full list of the types of mechanisms employed, see Annex D).

While currently not in use, regional Service Canada staff noted that the departmental Common System for Grants and Contributions (CSGC) Project Life Cycle module has screens for recording “success stories” and “best practices.” Footnote 30 Using these two fields could be one way of increasing the extent to which success stories and best practices are documented and used to inform policy and program development. Footnote 31

Systematically identifying and disseminating key factors of project success (including lessons learned and promising practices) in a format that is accessible to program stakeholders holds potential for scaling up elements of previously funded projects and contributing to future projects achieving their goals.

2.3 Effectiveness

The evaluation found that the NHSP is making progress towards achieving its direct outcomes. However, the current means of collecting and reporting community-based project performance information was found to be highly laborious and time-consuming, producing questionable performance information that is not comparable over time. Furthermore, the CSGC Project Life Cycle module does not provide adequate performance information for evaluation purposes.

Availability and quality of performance data

The NHSP Performance Measurement Strategy identifies, by program outcome, key performance indicators and data sources that are used to systematically collect information in support of departmental program monitoring, assessment and evaluation activities. The final reports for community-based projects Footnote 32 constitute a key source of information Footnote 33 for measuring program performance and effectiveness. Footnote 34 Final reports from NHSP community-based project funded recipients are self-reported and submitted in paper format; these are subsequently scanned and uploaded to the CSGC Project Life Cycle module as PDFs and word documents. In addition, the CSGC Project Life Cycle module is intended to document “all phases of the project life cycle from application to close-out.” Footnote 35

Findings from the data review identified that while various final report templates employed over the evaluation reference period reflect continued changes to simplify and clarify questions in order to secure more relevant project result data, variations between final report templates make it difficult to compile and compare project performance data over time. Moreover, the manual process of extracting final reports and the corresponding performance information is resource intensive (labour and time) with respect to the accessing and analyzing of performance information. Self-reported results and errors in the entry of this information hindered the quality of available performance data. Furthermore, the percentage of missing final reports from the CSGC Project Life Cycle “Supporting Documents” screen (25% over the evaluation reference period) is noteworthy, in that it potentially represents a large amount of performance information that could better inform an understanding of program effectiveness. Footnote 36

Lastly, the review confirmed that much of the current NHSP CSGC Project Life Cycle data are not relevant for reporting on program effectiveness and more broadly, management decision-making. However, through the analysis of Project Life Cycle data, it was found that CSGC Project Life Cycle data captures accurate applicant project information such as organization name, address, proposed budget, funding recommendation, dates of key milestones in the application process and decision.

Progress towards direct outcomes

The program’s logic model identifies the intermediate and ultimate outcomes of the program as “shared outcomes,” on the basis that “attribution [is] distributed across a range of stakeholders,” including recipient organizations, other ESDC programs and different levels of government. Footnote 37 Accordingly, the program’s four direct outcomes and associated indicators were used to gauge the extent to which the program is contributing to the ultimate outcome of having “individuals, facilities and communities mutually benefit from the participation of seniors.” Footnote 38

Overall, from the perspective of key informants and based on the findings from the document review as well the analysis of a sample of previously funded projects, NHSP is making progress towards achieving its direct outcomes. Footnote 39 That is, organizations are reporting adopting approaches to engage with volunteers, sharing of knowledge and experience between different generations are taking place, and NHSP funding is enhancing the capacity of recipient organizations to support seniors’ initiatives. Summary findings by direct outcome are identified below: Footnote 40

Direct Outcome 1: Recipient organizations adopt approaches to engage volunteers.
  • The majority of projects reviewed (70%) reported using different types of approaches to engage volunteers (both new and existing) in the planning, organizing or delivery of project activities.
  • The most widely used approaches are having existing volunteers recruit their friends or peers, holding information sessions/workshops and developing print media.
Direct Outcome 2: Recipient organizations have capacity to support seniors’ initiatives in their communities.
  • Almost 70% of the projects reviewed reported leveraging resources from multiple contributors, with community groups and/or not-for-profit organizations being the most commonly cited contributor.
  • On the whole, roughly equal amounts of cash ($119,644.00) and estimated cash value of in-kind support ($181,496.75) were leveraged. A significant number of partnerships involved the provision of in-kind support.
Direct Outcome 3: Participating seniors share their knowledge and experience with peers and different generations.
  • The principal methods used by seniors included sharing of life experiences, knowledge and skills with their peers and different generations Footnote 41 .
  • Overall, over 60% of recipient organizations reported targeting a specific group, with persons with disabilities constituting the most widely targeted group.
Direct Outcome 4: Recipient organizations equip community members to recognize abuse of seniors.
  • Given that a limited number of the projects reviewed reported “expanding awareness of elder abuse, including financial abuse” as an objective, the document review made use of existing analyses conducted by program management.
  • The majority of the 2011-2012 community-based projects addressed the issue from multiple fronts (i.e. financial abuse, psychological abuse, physical abuse, neglect, sexual abuse and other forms of elder abuse) and employed multiple approaches (e.g. information sessions, partnering and or networking). Footnote 42

Program uptake

The program’s level of reach with respect to its target population was assessed. Footnote 43 Premised on a need to respond to a number of far reaching societal trends, the NHSP targets a broad base of potentially eligible beneficiaries that reflects the broad scope of the program’s five objectives. Information with respect to program demand (i.e. applications received) and program expenditures shows that the demand for projects exceeds available funding. Footnote 44 Between 2011 and 2014, 58% Footnote 45 of applications submitted were approved for funding. Footnote 46 Figure 1 below shows the regional distribution of projects (both received and funded) on a percentage basis. Some evaluation evidence (e.g., document review, key informant interviews) indicates regional differences between percentages of seniors, applications received and funded may be due to factors such as the quality of applications received and program awareness; however, a review of this trend may be beneficial for program monitoring purposes.

Figure 1: Percentage of Applications Received vs. Funded by Region (2011 to 2014)
Applications Received versus Funded by Region (2011-2014)

Source: CSGC Project Life Cycle Module and Statistics Canada 2014 Annual Demographic Estimates

Text Description of Figure 1

Figure 1 is a bar graph that shows the regional distribution of projects received and funded, relative to the regional distribution of seniors. Data used to construct the bar graph was extracted from the Common System for Grants and Contributions as well as Statistics Canada’s 2014 Annual Demographic Estimates.

  • The graph shows that Ontario has the highest concentration of seniors with 38% of the overall senior population, followed by the Western and Quebec provinces with 25% and the Atlantic Provinces with 8% of the senior population.
  • The regional distribution of projects received and funded is shown as a percentage of the total number of applications to the program and total number of applications funded.
  • Between 2011 and 2014, the Western provinces received the highest number of application with 32% and funded the highest number of projects with 33%. Over the same period, Ontario received the second highest number of projects, at 28% and funded the second highest number of projects, at 31%.
  • Quebec received 27% of the total number of applications over this period and funded 25% of the total number of projects funded. Lastly, the Atlantic Provinces 14% of the total number of applications and funded 12% of the total number of funded projects.

Slight differences in the regional distribution of applications received and funded relative to the percentage of seniors may be due to regional variations in the level of program awareness and quality of applications received (which result in applications being screened out, thereby reducing the overall pool of applications).

Raw Data

Applications Received versus Funded by Region (2011-2014)
Region Population 65+ Percentage Received Percentage Funded
Ontario 38% 28% 31%
Western 25% 32% 33%
Quebec 25% 27% 25%
Atlantic 8% 14% 12%

The broad base of potentially eligible NHSP recipients makes it difficult to ascertain the program’s potential reach; however, as shown above the allotment of NHSP funding is aligned with the regional distribution of the older population within Canada. Of note, the absence of information on project participants or beneficiaries in the performance data collected on NHSP funded community-based projects poses limitations, most notably the inability to capture the number of project participants and/or beneficiaries. Overall, recognizing that the demand for the NHSP funding far exceeds available funds, an assessment of the extent to which the program is targeting seniors most in need of NHSP funding (e.g. communities with higher proportions of isolated seniors) would inform a fuller understanding of program reach.

2.4 Efficiency and Economy

Based on the evaluation evidence collected, reducing the administrative burden associated with the screening and assessment process, as well as the length of time associated with the application review and approval process were found to be ways in which the program could be made more efficient. Furthermore, accounting for the ways in which seniors access information will allow for more effective communication with seniors.

The program’s activities and outputs were examined for the purposes of assessing the efficiency and economy of the program. Footnote 47 To that end, the evaluation identified a number of issues with respect to the efficient administration and delivery of the program. On the part of regional Service Canada Project Officers, the repetition of steps and the lack of timely and clear operational directives were found to detract from the quality of the application screening and assessment process as well as entail a degree of administrative burden. From the perspective of community-based projects, the lengthy approval process was cited as detracting from the efficient delivery of the program. Footnote 48 It was recognized that NHSP’s objectives aim to address complex social problems and may require a long period of time to realize.

Project Screening and Assessment Process

During the key informant interview process, the majority of regional Service Canada respondents identified the presence of duplicate steps in the screening and assessment process. Among these respondents, the majority cited the two-pronged assessment stage involving both the Project Officers and Regional Committee members as leading to a repetition of steps. Respondents also felt that the use of paper applications and uploading the findings from the screening and assessment of applications into the CSGC Project Life Cycle module added to the degree of administrative burden.

Capturing the overall sentiment of regional Service Canada key informants, one respondent stated that “the way we presently do it, there is a great deal of duplication. Internally, staff fully assess these applications. And then we take a large number of them to Regional Committee members who then in turn also fully assess and rank these applications. One would think that there is a way to be more efficient.” The screening and assessment of NHSP projects entails the following three stage process to determine if a project should be recommended for funding:

  • Screening
  • Assessment by Project Officers
  • Assessment by Regional Committees

Referring to a verification checklist and a screening grid used in application screening, regional Service Canada staff expressed concerns regarding the repetition of steps. Upon completion of the checklist, Project Officers are instructed to place a copy in the paper file and an electronic version in the CSGC. While the screening stage is meant to ensure that the applicant meets the basic program eligibility criteria, the nature of the questions that are to be assessed requires a sufficient level of knowledge of the project applications, with emphasis placed “on not screening out applications without fully understanding the proposed project activities.” Footnote 49

In comparison to the screening stage, where applications are reviewed to determine if applications meet individual program eligibility requirements, the “assessment” stage entails a greater level of effort on the part of Project Officers. Subsequently, preparations are made for the next stage of the assessment process, which involves Regional Committee members. Footnote 50 Project Officers prepare “three separate lists [A, B, C] of projects identifying eligible projects by a suggested score range” as derived through the application assessment criteria. Footnote 51 Next, Regional Committee members are “asked to review and concur with the results of the assessment of the Projects on the A and C lists” and are to “focus their attention on the group of applications, which are middle range (B List).”

Throughout the screening and assessment of applications, Project Officers are to document the process in the CSGC Project Life Cycle Module for all applications, including those screened out and not recommended for funding. Footnote 52 Findings from the key informant interviews as well as the document review indicate that Regional Committee members play a key role in the delivery of the program by providing a regional perspective, given their expertise and knowledge as well their ties to their communities. Footnote 53 However, to a large extent, the assessment that is conducted by Regional Committees represents a repetition of the work that was already performed by the Project Officers.

Operational Directives

ESDC provides guidance to regional Service Canada delivery personnel through operational directives and ad-hoc support, as required. The majority of Service Canada respondents cited a lack of clear and updated operational directives as an issue that can lead to inconsistencies in the assessment of applications. These include inconsistencies across the provinces with respect to the treatment of applications with missing information and the timeliness of directives on changes to the program such as eligibility criteria. For instance, a Service Canada respondent indicated that directives are not received “until at a very late stage of the assessment period. It will be very helpful if, [for] the [up] coming call for proposals, we are provided with the operational directive ahead of time, like at the beginning of the assessment period. This will help the program officers to become more efficient when we are assessing the files.”

Operational Timelines

From the perspective of some regional Service Canada respondents, tight timelines during the application screening and assessment period were thought to detract from the quality of the application assessment process. For example, a respondent stated that, during the last call-for-proposals, “a challenge we faced was the tight timeframe and deadlines during the assessment period…we are trying to get done a lot in a short amount of time.” In the same vein, another respondent suggested that short timelines are potentially a reflection of inadequate resources and of the timing of the screening and assessment period, which usually takes place during the summer when staff are more likely to be on vacation. Footnote 54

Among the community-based organizations, the timeliness of the application approval process was noted as an area that could benefit from improvement. The vast majority of respondents who identified an issue with the timeliness of the application process cited the length of time between the submission of applications and receipt of funding as “somewhat long.” Analysis of CSGC Project Life Cycle applicant information found that, on average, the application assessment and approval process can extend up to seven months.

Program Awareness and Community Engagement

Related to program awareness and community engagement, Footnote 55 regional Service Canada key informants cited the use of regional community engagement strategies to increase program awareness. Noting that each region is responsible for developing a promotion and engagement plan that “responds to their specific needs,” key informants from regional Service Canada and national headquarters highlighted that program promotion is limited and noted that the program could benefit from increased awareness and diversity of recipients in order to encourage the development of new and innovative projects. Footnote 56 Indicative of this, a respondent tasked with assessing applications from remote and rural communities pointed out that whereas in the previous years on average nine proposals would be submitted from the region, this had dwindled to three proposals during the most recent fiscal year.

To better reach underserved segments on the seniors population, it was noted by key informants that a potential solution may be to focus on areas where “weaknesses lie” such as remote and rural areas where take up or awareness of the program may be low.

Communicating with Seniors: A Mixed Method Approach

As part of ESDC’s modernization efforts, the Department is extensively making use of information technologies and the Internet in order to modernize and improve service delivery. Footnote 57 The ability for a program to communicate to and fully reach its intended participants is an important consideration, which, for example, can affect a range of program activities, including program awareness, program application, reporting on program performance, and sharing of relevant program information. Multiple lines of evidence pointed to the merits of incorporating seniors’ preferred modes of communication, recognizing that projects are often seniors led and include community groups that contain various concentrations of seniors. Research has shown that when seniors seek information, they are more comfortable using radio, television or print media and are likely to approach friends and family. Footnote 58 However, personal, face-to-face contact was regarded as integral to how they accessed information. Footnote 59

The seniors population depends far less on the Internet and modern technologies to communicate, which has resulted in a digital divide in Canada. Footnote 60 While ESDC’s move to modernization is responsive to future senior demographics, particularly the more technologically savvy baby boomers, Footnote 61 it is important to recognize that the current cohort of seniors is not a homogenous group and, for some seniors, their preferred mode of communication, as indicated by the above research, continues to be traditional methods. In addition to a digital divide between the seniors and other segments of the population, there is also an urban/rural digital divide. Footnote 62 The urban/rural divide is an important issue for seniors and community groups because roughly one quarter of seniors in Canada live in rural or remote areas. Footnote 63

The exclusive use of the Internet or automated mechanisms could ignore a large contingent of seniors and community groups with large concentrations of seniors who may be more responsive to traditional means of communicating such as print, telephone or face-to face contact.

Demonstration of Economy

The nature of NHSP’s objectives and community-based approach to delivering the program presents significant challenges for estimating program cost-effectiveness. However, an examination of program budgetary and expenditure data can provide insights with respect to the utilization of resources. With the exception of fiscal year 2011-2012, when project payments were issued in 2012-2013 due to delays resulting from administrative challenges, the program’s operating costs as a percentage of total program expenditure averaged about 10% annually. Footnote 64

Figure 2: Operating Costs as a Percentage of Total Program Expenditure Footnote 65
Program Operating Expenditure
Text Description of Figure 2

Figure 2 is a line graph that shows program operating costs for fiscal years 2010-2011 to 2014-2015 as a percentage of overall program expenditure. Data used to construct the line graph was sourced from Employment and Social Development Canada’s Chief Financial Officer Branch.

  • Total program expenditure and operating costs encompass both the community-based and pan-Canadian components of the program.
  • For fiscal year 2010-2011, operating costs were 11% of the total program expenditure, in 2011-2012 it was 62%, in 2012-2013 it was 6%, in 2013-2014 it was 10% and in 2014-2015 it was 12%.
  • With the exception of fiscal year 2011-2012, when project payments were issued in 2012-2013 due to delays resulting from administrative challenges, the program’s operating costs as a percentage of total program expenditure averaged about 10% annually.

Raw Data

Program Operating Expenditure
Fiscal Year Percentage of Total Program Expenditure
2010-2011 11%
2011-2012 62%
2012-2013 6%
2013-2014 10%
2014-2015 12%

As shown in Figure 3, the regionally allocated funds for fiscal year 2014-2015 are roughly proportional to the regional distribution of seniors in 2014.

Figure 3: Regional Distribution of Senior vs. Allocated Regional Amounts (2014)
Regional Distribution of Seniors versus Allocated Regional Amount
Text Description of Figure 3

Figure 3 is a line graph that shows the regional distribution of seniors vs the amount of money allocated to each region for fiscal year 2014-2015. Data used to construct the line graph was sourced from Employment and Social Development Canada’s Chief Financial Officer Branch as well as Statistics Canada’s 2014 Annual Demographic Estimates.

  • The regionally allocated funds for fiscal year 2014-2015 are roughly proportional to the regional distribution of seniors in 2014.

Raw Data

Regional Distribution of Seniors versus Allocated Regional Amounts
Region Percentage of Funds Allocated Percentage of Population 65+
Atlantic 15% 8%
Quebec 23% 25%
Ontario 31% 38%
Western 31% 25%

Based on analysis of program expenditure data, it was found that on average the program spent 99% of the regionally allocated funds for community-based projects for fiscal years 2012-2013, 2013-2014 and 2014-2015. Footnote 66 However, according to ESDC’s 2013-2014 Departmental Performance Report, as a whole (for both the community-based and pan-Canadian components of the program) the program underspent planned expenditure for that fiscal year by approximately 6%. Footnote 67

Given the high demand for the community-based project component of NHSP, the assessment of proposed projects, with respect to value-for-money, is a likely area for greater focus in ensuring that program resources are optimized. In line with ESDC’s Grants and Contributions Operations Guide, the principal method whereby the program ensures a measure of value-for-money is through the application assessment process, where Project Officers are advised to consider the extent to which the projects “use resources efficiently and effectively” along with the feasibility and sustainability of projects. Footnote 68 Related to the cost-effectiveness of community-based projects, key informants suggested that clustering similar projects in small geographic areas may potentially reduce the number of similar projects that are funded.

3.0 Conclusions and Recommendations

The evaluation found that NHSP is relevant to federal and departmental policy directions and priority areas. An increasingly aging population, the importance of active community involvement and volunteerism to the well-being of seniors and their communities suggest a continued need for the program. Multiple lines of evidence employed by the evaluation confirm that the program is making progress towards achieving its direct outcomes. Most importantly, the evaluation identified lessons learned, promising practices, and success factors that relate to the performance of community-based projects. Areas for improvement include the capturing of relevant, reliable and easy-to-access performance data; reducing the administrative burden on regional Service Canada staff; and reducing the time needed to review and approve applications.

Recommendations

1. Streamline and focus the collection of project performance information in an accessible format to facilitate program monitoring and evaluation. Support the identification and dissemination of promising practices that can be replicated in other communities.

Currently, the means by which performance data are collected and reported on (i.e. scanned PDF’s, hand written final reports), is a highly resource intensive process (e.g., time and labour) that often yields performance data that is difficult to use and of uncertain reliability for monitoring, assessment and evaluation purposes.

While success stories are showcased on the ESDC website, the process by which these are identified could be more standardized (e.g., criteria, measures, etc.). Moreover, these success stories could be better aligned with lessons learned, promising practices and success factors.

2. Recognizing the needs and preferences of seniors, leverage traditional methods of communication and external stakeholder organizations to complement Modernization efforts in support of program delivery.

While ESDC’s move to modernization is responsive to future demographic trends, it is important to recognize that seniors and community groups who represent them are not a homogenous group. In addition, there is also an urban/rural digital divide. This is an important issue for seniors and community groups with large concentrations of seniors as about one quarter of seniors in Canada live in rural areas and small towns.

3. Explore ways to streamline the administration of the program and reduce processing time.

Findings from all lines of evidence confirm the need for ESDC to reduce the degree of administrative burden associated with application screening and the assessment process on the part of regional Service Canada staff, as well as to reduce the length of time between the submission of project applications and receipt of funding on the part of recipient organizations. On average, the application assessment and approval process can extend up to seven months.

Annex A: Scope and Methodology

The focus of the evaluation is to provide relevant information to the Department for program and policy development and to address the areas of interest/knowledge gaps identified by senior ESDC management, which included the need to gain a better understanding of whether promising or best practices with respect to project success are being established amongst community-based projects and at the regional program delivery level, and if so, what the criteria for identifying these practices are. Also important to senior management was exploring how promising or best practices are shared and the process by which results at the community level inform program policy and operation.

The evaluation employed the following methods:

Key Informant Interviews: A total of 45 key informant interviews were conducted amongst Regional Committee members (15), regional Service Canada staff (15), community-based projects (10) and national headquarters program staff (5). Key informant interviews gathered in-depth information, including opinions, explanations, and examples with respect to the evaluation questions, with particular attention paid to the most recent year of program delivery. Interviewees were selected for their experience and knowledge of NHSP. A detailed content analysis of information collected from the interviews was used to identify key themes.

Regional Committee members review project applications and make recommendations to regional Service Canada staff with respect to project funding and are connected to various communities in their region. As noted in the previous evaluation, “Committee members are well-positioned to share their expertise and knowledge of their communities, and thus can help to identify priorities and create linkages to the Program.” Footnote 69

Service Canada staff (project officers, senior development officer and business expertise consultants) have familiarity of projects at the community level, and identify lessons learned for publishing on the ESDC website, so are well positioned to be able to identify lessons learned at the community level. Project officers reported doing the majority of the administrative work related to the assessment and processing of applications, including determining eligibility, coordinating Regional Committee meetings and issuance of payments for approved projects. Senior development officers and business expertise consultants reported being the lead persons for their regions and responsible for the community engagement process, leading the Regional Committee meetings, communicating with national headquarters and overseeing the project officers.

Community-based project interviews were held with senior staff within the community-based organizations whose projects were funded by NHSP. These organizations have direct experience in designing and delivering projects that address the needs of seniors and their communities. The majority of interviewees were Executive Directors who oversaw or were involved in the design and delivery of the NHSP project. These respondents were able to provide first-hand experience in implementing NHSP projects and speak to the success factors and challenges associated with community-based projects.

Lastly ESDC program management where be interviewed in order to gain additional insight from senior management on pertinent evaluation issues and questions. They were considered experts, in the areas of program operations, performance reporting, and data quality.

Document Review: An extensive document review was undertaken to bolster and inform other lines of evidence. Key documentation used in the evaluation were sourced and/or recommended by evaluation working group members. Documents reviewed included a stratified sample of documentation relating to 50 community-based projects from fiscal years 2012-2013 and 2013-2014; 44 community-based project success stories (from fiscal years 2010-2011 to 2013-2014); Key project documents including basic project information; Application form/agreement; Screening form; Assessment grids; Project assessment; Progress report and Final project report. The Common System for Grants and Contributions (CSGC) online project summary pages were also used Footnote 70 ; Program Terms and Conditions of the NHSP; Program Performance Measurement (PM) strategy; previous NHSP evaluations and Speeches from the throne, budget speeches, ESDC’s annual Departmental Performance Report and Report on Plans and Priorities.

Literature Review: A selective and narrowly defined literature review was conducted to gain insights on mechanisms currently used to identify and share knowledge amongst seniors. The review also found some information on lessons learned in relation to building social partnerships, leadership, senior participation and volunteerism, and senior centers. This review is intended to provide contextual information to evaluation findings.

Data Review: The review entailed an assessment of the availability and quality of NHSP community-based project performance data and the relevance and usefulness of CSGC Project Life Cycle module data for evaluation purposes.

Limited Cost-Analysis: Based on the available evidence, the evaluation conducted a limited costing analysis. Financial information pertaining to program expenditures were sourced from departmental reporting documents, the Chief Financial Officer Branch and from the sample of projects reviewed. Given the high demand for NHSP, a closer look at program costs in relation to benefits of the program accounting for the utilization of program resources would allow for a better understanding of the extent to which funds are being optimized, seniors most in need are being served and issues affecting seniors are being addressed in an effective and efficient manner. Footnote 71 However, the program’s objectives of addressing complex social problems which primarily entail qualitative outcomes and the absence of performance information (e.g. number of direct and indirect project beneficiaries) made it difficult to more accurately catalogue the benefits of NHSP funded community-based projects relative to program expenditures.

Challenges and Mitigation Strategies

Response Bias: Key informants who participated in the evaluation may have a vested interest in the continuation of the program. The inclusion of four key informant groups with varying degrees of separation from the program, as well as probing respondents for concrete examples, served to control for this potential bias. To further mitigate this challenge, data was triangulated across multiple lines of evidence where possible.

Small Sample Size: A limitation in the review of project documents was the inability, due to limited time and resources, to review all of the approximately 1,700 projects per year during the reference period established. To overcome this challenge, the evaluation calibrated its efforts, focusing on approximately 100 community-based projects, with representation from all regions. This approach allowed for a more focused assessment of project activities by examining full project documentation in more detail.

Absence of Electronic Performance Data: The absence of electronic performance data that is readily accessible for analysis or review required evaluators to manually extract written or typed information from final reports that were in PDF and Word formats.

Variation in Performance Data Collected: Since there were various final report templates (4) used to collect information on the projects during the evaluation reference period, analysis of performance information from the differing final report templates presented challenges for compiling and comparing performance information over time. To account for this limitation, where possible, efforts were made to standardize/harmonize the performance information contained in each of the templates.

Definition for Seniors

The NHSP does not make use of a formal definition for seniors. According to the Program’s Operational Directives, the term “seniors” is understood to be contextual and for the “purposes of funding eligibility…lets communities define seniors.” Footnote 72 For the purposes of consistency in terminology, the final report employs a working definition of seniors as persons aged 65 and older, Footnote 73 while accounting for other segments of the population (e.g. persons aged 55 to 64 or persons aged 55 and older) when available information was limited to these other segments.

Annex B: NHSP Logic Model

ESDC’s PAA Strategic Outcome: Income security, access to opportunities and well-being for individuals, families and communities.

Ultimate Outcome: Individuals, families and communities mutually benefit from the participation of seniors.
Ultimate Outcome: Individuals, families and communities mutually benefit from the participation of seniors.
Description of NHSP Logic Model

Annex B presents the NHSP logic model, which shows the program’s road map, by outlining the activities the program undertakes and the outputs that are intended to lead to the achievement of expected outcomes.

Program Objective

  • The NHSP is a grant and contribution program that aims to enable seniors to contribute to their communities through greater involvement and volunteerism.

Activities

The program logic model identifies 4 activities:

  1. Program promotion,
  2. Community engagement,
  3. Administering Gs&Cs funds; and
  4. Program measurement, analysis and redesign

Outputs

The 4 activities are intended to produce the following 4 program outputs identified in the logic model.

  1. Funded projects based on established local/regional/national priorities;
  2. promotional plans;
  3. established community-based networks;
  4. Community engagement plans

Direct Outcomes

The 4 outputs are intended to lead to the following 4 direct outcomes of the program.

  1. Recipient organizations adopt approaches to engage volunteers
  2. Participating seniors share their knowledge and experience with peers and different generations
  3. Recipient organizations have capacity to support seniors’ initiatives in their communities
  4. Recipient organizations equip community members to recognize abuse of seniors

Shared Outcomes

The program’s logic model identifies the intermediate and ultimate outcomes of the program as “shared outcomes,” on the basis that “attribution [is] distributed across a range of stakeholders,” including recipient organizations, other ESDC programs and different levels of government.

The identified 4 direct outcomes are expected to produce the following 2 shared outcomes.

  1. Communities have the capacity to address local issues by engaging seniors
  2. Seniors participate and contribute to communities

Ultimate Outcome

The 2direct outcomes are intended to lead to the following ultimate outcome:

  • Individuals, families and communities mutually benefit from the participation of seniors.

Strategic Outcome

The ultimate outcome of the program feeds into the following strategic outcome of Employment and Social Development Canada:

  • Income security, access to opportunities and well-being for individuals, families and communities.

Annex C: Glossary of Terms

Term Definition
Best practice

An approach or action proven—through rigorous external evaluation and internal assessment—to work effectively to produce intended results/successful outcomes. It should also address a common problem faced by a broad spectrum of organizations; function to tackle problems in more than one organizational setting and context; and show replicability on at least a limited scale.

*In some cases a best practice may be based on more intuitive or subjective viewpoints and should be captured.

Source: “Identifying and Promoting Effective Practices

“Best and Promising Practices of Multi-Sectoral Collaboratives: Indicators for Reflection and Assessment”

“What Works and for Whom? - Part 1 - A Hierarchy of Evidence for Promising Practices Research”

Economy

Minimizing the use of resources. Economy is achieved when the cost of resources used approximates the minimum amount of resources needed to achieve expected outcomes.

Source: Treasury Board of Canada Secretariat. Directive on the Evaluation Function

Effectiveness

Assessment of progress toward expected outcomes (incl. immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes.

Source: Treasury Board of Canada Secretariat. Directive on the Evaluation Function

Efficiency

The extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. The level of input and output could be increases or decreases in quantity, quality, or both

Source: Treasury Board of Canada Secretariat. Directive on the Evaluation Function

Innovative Practice

New practices, supports, or systems that an organization—such as a community-based project—has not previously had the opportunity to test in its unique organizational environment. Examples of innovative practices:

  • Leveraging funds
  • New multi sector (strategic) partnerships (private sector); partnerships can either involve resources (Cash or in-kind), or involve collaboration on one or many areas of an intervention.
  • Broadening the use (scalability) of a proven and effective approach to mitigating social challenges.
Lesson learned

Identifies and documents opportunities, risks, and issues of programs, projects, and activities in order to encourage or avoid the practice in the future. Lessons learned can highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcomes, and impact.

*Oftentimes, these lessons learned could be shared with other groups and would help them to better manage the next projects they do.

Source: Glossary of Key Terms in Evaluation and Results-Based Management” (OECD, 2010)

“Lessons Learned”

Project Management Knowledge

Partnership

A partnership is defined as a relationship where two or more parties, having compatible goals, form an agreement to do something together. Partnerships are about people working together in a mutually beneficial relationship, oftentimes doing things together that might not be able to be achieved alone. Such partnerships are driven by:

  • mutual benefits (partnership arrangements are expected to provide value to each party);
  • shared risk, responsibility, and accountability (each partner has a stake in the oversight of the initiative); and
  • pooled resources (each partner invests something of value, be it a monetary or in-kind contribution).

Source: “The Partnership Handbook” (HRSDC, 2000)

Partnership, formal

A collaborative effort, between an agreement holder (recipient) and another party working towards NHSP related program outcomes that is guided by a written agreement. This official or legal agreement should set out the partnership’s purpose, each partner’s respective roles and responsibilities, and the expected results of the partnership.

Source: “The Partnership Handbook” (HRSDC, 2000)

Partnership, informal

A collaboration between an agreement holder (recipient) and another party operating without a written agreement defining the objectives, roles, and responsibilities of each party. While still rooted in a shared vision, it is a more flexible arrangement between parties. An informal partnership can be converted into a formal partnership over time or remain informal, depending on the needs of the partners.

Source: “The Partnership Handbook” (HRSDC, 2000)

Promising practice

An approach or action that has produced successful outcomes for one organization, but lacks sufficient external evaluation to ensure replicability for general use. Internal assessment has, however, indicated that the practice holds potential value for other organizations.

Source: “Identifying and Promoting Effective Practices

“Best and Promising Practices of Multi-Sectoral Collaboratives: Indicators for Reflection and Assessment”

“What Works and for Whom? - Part 1 - A Hierarchy of Evidence for Promising Practices Research”

Relevance

Issue 1: Continued need for the program: Assessment of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians.

Issue 2: Alignment with government priorities: Assessment of the linkages between program objectives, federal government priorities and departmental strategic outcomes.

Issue 3: Alignment with federal roles and responsibilities: Assessment of the role and responsibilities for the federal government in delivering the program.

Source: Treasury Board of Canada Secretariat. Directive on the Evaluation Function .

Social Partnerships

Social Partnerships are an active collaboration between business and not-for-profit and government actors who collaboratively work to accomplish positive social outcomes and community benefits, including benefits to the partners. Partners have a shared recognition of the need for the partnership to achieve shared goals. The commitment of resources such as time, expertise, financial, or volunteers, often a combination of them all, is deeper than a simple monetary contribution. Social Partnerships are sometimes referred to as community-based partnerships or cross-sector partnerships.

Source: Maira May Seitnidi and Andrew Crane. Implementing CSR Through Partnerships: Understanding the Selection, Design and Institutionalisation of Nonprofit-Business Partnerships. Journal of Business Ethics. Spring 2009, 85:413-429.

(Project) Success

A project that achieves their project results, or has other beneficial unintended outcomes.

Success Factor

A key factor that contributes to the success of the project, whereby a favorable or desired result and/or outcome is achieved.

Success story

Showcases positive results from a program, project, or activity. The story will tell the reader the results or impact of the program, how the program success was achieved, and define the public value of a program. A success story can be individual-, organization-, or program-based and will always have positive end results. Examples of what a success story may include:

  • Using Innovative practices;
  • Achieving sustainability without federal funds through the use of other revenue sources. This may include other orders of government, donations (in kind/cash), recruitment (volunteers), private sector involvement, or other models such a cooperative structures or social finance/enterprise mechanisms.
  • Projects reaching their intended outcomes; may also have unintended (positive) outcomes

Source: “How to Write Effective Success Stories”

How to Define Success

Annex D: Mechanisms for Sharing Factors of Project Success

The mechanisms listed below are employed by NHSP to share success stories, as well as key aspects of strong applications and projects with both potential applicants as well as regional Service Canada delivery staff.

Success Stories Webpage

These success stories entail a detailed description of project objectives, design and delivery, along with the results of the funded activities. The ESDC NHSP webpage also provides examples of successful project applicants by making available a list of previously funded projects, listed by fiscal year and dates back to 2011-2012.

Applicant guide

The applicant guide provides key definitions, such an understanding of the term ‘senior’, program objectives and examples such as weak and strong project activities.

Operational Directive

For ESDC and regional Service Canada staff, the NHSP operational directives provide detailed instructions for administering NHSP related activities, namely the screening and assessment of applications. Intended to ensure a high measure of consistency and quality in the assessment of applications and management of projects, the operational directive is another avenue whereby program management shares success factors and lessons learned with respect to how to better assess applications and manage projects along the continuum of the project life cycle.

Regional Promotion and Community Engagement

Documents to assist regional Service Canada staff in their community engagement presentations (e.g. PowerPoint template) are provided by national headquarters. The most recent PowerPoint presentation made available to regional Service Canada staff provides examples of weak and strong project applications as well as tips on how to strengthen the impact of projects.

Regional Committees

Regional Committee members function to add value to the application assessment process through their knowledge and experience with respect to the needs of their communities. The use of Regional Committee members represents an opportunity for the program to leverage the expertise of individuals who are most aware of the needs of their communities and the organizations that provide services for seniors. In this respect, Regional Committee members may bring with them a degree of awareness of the types of projects and organizations that are best suited to address the needs of seniors in their communities.

Factsheets

To further assist NHSP applicants, the NHSP produces a number factsheets containing useful tips and ideas to incorporate into project applications and project design and delivery.

Annex E: Key Factors of Project Success

The following factors are geared towards community-based organizations and are intended to inform aspects of project design, implementation and delivery.

Overarching Theme Project Success Factors Factors Detracting from Success
Project Design
  • Applications showcase sound administrative capacity (e.g. demonstrate project management skills through detailed work plans, realistic project outcomes, costing estimates).
  • Presence of partnerships (e.g. partnerships were cited as increasing the chances of projects achieving their objectives).
  • The inclusion of members of the target population in the planning stages of project design, so as to better align project objectives with the needs and interests of the target group.
  • Project activities incorporate seniors most in need and benefit the wider community (e.g. target population includes isolated seniors and project with an intergenerational component).
  • Applications proposals that contain broad objective(s) with too many activities are less likely to realize their intended outcome(s).
  • Perceived lack of organizational capacity (e.g. viable projects may not receive funding due to the quality of applications that are submitted).
  • Absence of partnerships (e.g. absence or the wrong type of partnership and relying on other contributions or partnerships that have not been confirmed).
Project Implementation
  • Sufficient number of seniors leading and benefiting from the project (e.g. the greater the level of involvement, the more likely the project is addressing a need and continue to operate past the end of the funding period).
  • Extensive and comprehensive communication campaigns to raise awareness of project activities throughout the community. Footnote 74
  • Lack of senior and volunteer involvement (e.g. insufficient involvement of seniors and or volunteers in project activities).
Project Delivery
  • Organizational stability and strong leadership from project start to finish.
  • Supportive partners (e.g. willingness of organizations to participate and contribute resources such as space, volunteers to the project).
  • Ease of project accessibility (e.g. project activities spread out in the community, convenient locations and/or delivery of project activities in the language of target population).
  • Change in leadership and other organizational turnover (e.g. departure of volunteers) due to unforeseen events such as sickness or death.
  • Low level of partner support (e.g. insufficient support by partners detracts from the successful outcome of projects).

Annex F: Lessons Learned in Relation to Community-Based Projects

The following constitute key lessons learned in relation to internal Service Canada activities (e.g. monitoring of projects), program administration and management (e.g. community engagement) and elements of project design, implementation and delivery (e.g. tailored project activities).

Program Administration
  • High utilization of the program for the acquisition or maintenance of assets may be detracting focus from other program objectives.
  • The diversity of Regional Committee members allows for a fuller assessment of applications (e.g. members bring different perspectives and expertise in assessing applications).
Organizational Capacity
  • Community-based project capacity to effectively manage projects and being flexible to adapt in a timely manner to challenges that arise during the project.
  • Being respectful of the mandate of partners in finding a common ground and involving willing partners in projects is a good way to nurture relationships and foster future partnerships.
  • Applicants from rural and remote areas exhibit differences in organizational capacity (e.g. resources such as partnerships may be absent in some of those regions).
Project Design
  • Conducting a community needs assessment prior to project design allows project activities to be focused and more likely to meet the needs of seniors and their communities.
  • Senior involvement in project design and delivery is conducive to better addressing the needs of seniors and their communities.
  • Dedication and expertise of volunteers enhances the quality of project activities.
  • Tailor project activities and objectives to the unique needs of the local senior population (e.g. language or cultural barriers).
  • Projects that benefit the wider community, for example those with intergenerational components were perceived to have a greater impact (e.g. projects that are far reaching were found to work well).
Communication and Community Engagement
  • Actively engaging with potential applicants (e.g. in-person) may be better suited for raising awareness of the program in communities.
  • Sufficient community engagement prior to call for proposals is required to increase awareness and promote new activities to expand the reach of the program (e.g. focus on certain communities such as those in rural areas that may be lacking internet service).
  • Some community-based organizations require assistance in developing strong applications.
  • There is insufficient information sharing between organizations on promising and or best practices regionally and nationally.
  • Making use of existing networks in the community to promote the program helps supplement the absence of dedicated funds towards advertising the program.
  • Early and ongoing communication with organizations helps to ensure adequate understanding of NHSP requirements and contributes to project success (e.g. application development, timely progress reports).

Annex G: Lessons Learned in Relation to Partnerships

On the part of community-based organizations, the following are key lessons learned around the development of partnerships.

Overarching Theme Lesson Learned
Developing Partnerships
  • Organizations lack expertise and internal resources to on establishing and nurturing partnerships.
  • Partner relationships take time to develop and need to be in place prior to the start of the project.
  • A lot of the partnerships are informal and built on trust.
  • Organizations tend to partner with those they are most familiar with.
  • Organizations who have partnered with organizations with similar programming stand a greater chance of success.
  • Open communication platform (e.g. to discuss successes and failures) facilitates shared commitments and expectations, especially with respect to informal partnerships.
Maintaining Partnerships
  • Projects activities that benefit partners involved build stronger relationships and elicit a greater level of commitment from partners. Shared commitments = stronger buy in from partners.
Leveraging Resources
  • Partnerships present an opportunity to leverage resources which results in better programs to serve seniors and the community (e.g. through the sharing of ideas and greater awareness in the community).
  • Insufficient funding amount may require multiple partners – collective impact approach needs to be considered to promote partnerships and successful project outcome.
  • Funded community-based projects can benefit from greater leveraging.

Annex H: Promising Practices

Based on findings from the document review and key informant interview process, the following approaches and project activities described below were found to constitute promising practices.

Overarching Theme Promising Practice
Program Administration
  • Enhancing the capacity of the organizations by removing structural barriers in order to engage seniors with disabilities through improved accessibility. Noting that the prevalence of disability increases with age, this practice holds great potential in reducing the extent of social isolation among seniors with disabilities. Footnote 75
  • Promoting the provision of skills and knowledge of technology (e.g. computer and internet usage) and social media (e.g. Facebook, Skype) can help seniors to keep in touch with family, friends and their community and to help seniors learn how to navigate information technology.
  • Encouraging the targeting of seniors from vulnerable groups (e.g. isolated seniors in rural or remote areas, low-income seniors, seniors from visible minority groups, seniors with mental illness, official language minority seniors in their communities) and/or the unique needs of certain seniors and their communities (e.g. intergenerational interaction and mentoring such as cultural transmission among Aboriginal elders and youth, teaching newly widowed men how to cook) to address the needs of segments of the seniors population that may be underserved.
  • Promoting considerations around revenue generation in projects during the project conception stage (e.g. through advertisements and/or sponsorships and sale of project end products such as calendars) may enhance the sustainability of projects beyond the funding period.
  • Related to the administration of the program, a few regional Service Canada respondents identified the use of a mapping exercise for their region to better identify the geographic distribution of funded projects, to better identify areas that are in need of greater program awareness activities in the next round of regional promotion and community engagement.
Project Design
  • For projects that entail the sharing of knowledge and experiences, the use of video and audio media was found to be a promising practice that could enhance the reach of project activities through exposure to a greater audience.
  • For intergenerational projects, variety of activities designed to attract participants with varied interests, including different age-cohorts, was found to be conducive to increasing number of project participants.

List of Abbreviations

CSGC: Common System for Grants and Contributions

ESDC: Employment and Social Development Canada

NHSP: New Horizons for Seniors Program

PLC: Project Life Cycle module

List of Figures

Figure 1: Percentage of Applications Received vs. Funded by Region (2011 to 2014)

Figure 2: Operating Costs as a Percentage of Total Program Expenditure

Figure 3: Regional Distribution of Senior vs. Allocated Regional Amounts (2014)

Page details

Date modified: