Formative Evaluation : New Horizons for Seniors Program, June 2009

Background

The NHSP provides grants of up to $25,000 to community organizations (non-profit sector organizations, community-based coalitions, networks, ad hoc committees, municipal governments, band/tribal councils and other Aboriginal organizations) for a wide range of projects across Canada that encourage seniors to contribute their skills, experiences and wisdom in support of the social well-being in their communities, to build community capacity and to increase social participation and inclusion of seniors. The NHSP helps to ensure that seniors are able to benefit from and contribute to the quality of life in their communities through social participation and active living.

In its first year of operation (2004/05), the program awarded approximately $5 million in grants. In 2005/06, the grants budget increased to $11.7 million, and in 2006/07, it increased further to $15.6 million.

As a result of changes announced in Budget 2007, the Program now offers three types of funding to organizations: Community Participation and Leadership; Capital Assistance and Elder Abuse Awareness.

Evaluation Scope and Methodology

The formative evaluation covered the first three years of program funding for the Community Participation and Leadership (CPL) component: 2004-05; 2005-06; and 2006-07. It was focused on six specific evaluation issues:

• whether the program is designed so that it can achieve its stated objectives and anticipated outcomes;
• whether the program has been implemented and delivered as intended;
• the efforts made to raise stakeholder awareness of the program;
• the adequacy of the management and performance information systems in place, including whether the appropriate data was being collected to support the summative evaluation;
• the production of intended outputs; and
• the achievement of short-term outcomes

Data collection and analysis for the evaluation took place between February and April 2008. Four main sources of information were used to conduct the formative evaluation: a review of documents; a review of administrative data (from the program database and a random sample of NHSP paper application files); key informant interviews (with NHSP staff and managers across the country, Regional Review Committee (RRC) members, and individuals from groups that represent seniors and/or are active in seniors issues at the national level and in each region); and a review of the results of a survey of funded and non-funded NHSP applicants. This report summarizes the findings from all four lines of inquiry.

Key Findings

Implementation and Delivery

For the most part, the NHSP is being implemented as intended. Each region has an RRC or equivalent, and there is flexibility in how the program is delivered in each region. This flexibility allows the NHSP to be responsive to unique community needs and circumstances.

Funded applicants appear to be satisfied with most elements of the program. Non-funded applicants were less satisfied, in particular with the lack of a clear explanation about why their applications were turned down. Nevertheless, most non-funded applicants indicated that they would reapply. Both funded and non-funded applicants have become increasingly dissatisfied with the amount of time it takes to review the applications.

There appears to be relatively little duplication or overlap with other Canadian programs. There are similar programs at the provincial level in Nova Scotia and Québec, as well as for First Nations groups in British Columbia; but for the most part, the NHSP fills a unique niche in promoting the involvement of seniors in their communities.

Outreach and Promotion

The program's outreach and promotional efforts appear to be adequate. The regional staff have used a wide variety of methods to engage the communities in their regions, including mail-outs of applications kits, presentations to communities or groups, advertisements, and one-on-one communication. Over time, a greater percentage of applications have met the eligibility requirements, indicating that the regional communications and promotional efforts have been effective.

Key informants thought that outreach efforts were most effective when there was in-person contact and assistance with the application. This is taking place in every region.

Survey respondents indicated that the most useful methods of informing applicants about the program are sending information packages in the mail and posting information on the NHSP website. Both funded and non-funded applicants found the website to be an effective communications tool.

Performance Measurement and Monitoring

For the most part, NHSP has the necessary management and performance information and the administrative data/systems in place to effectively manage the program. The program has a Performance Measurement Framework (PMF) in place that identifies performance indicators and describes how program monitoring will be periodically gathered. However, lack of flexibility in the Common System for Grants and Contributions (CSGC) means that some performance measures have to be collected manually from paper files.

The NHSP is currently gathering the information that will be needed to report on the output indicators, with the exception of the number of targeted mailings promoting the program.

The performance indicators for program outputs are appropriate and informative. However, those identified as program outcomes do not give a full flavour of the intended outcomes of the NHSP. Many of these relate to outputs rather than outcomes. There is also some overlap amongst the indicators for different levels of outcomes, and between different outcomes at the same level, which contributes to confusion about what the defining features of each outcome really are. These issues could have an impact on obtaining the data necessary for a summative evaluation.

The CSGC database contains data that are sufficiently accurate and complete to meet most management needs and to inform the program's performance indicators for outputs, with some exceptions.

Production of Outputs

Evidence gathered from documents, key informants and administrative data indicate that the NHSP has produced most of its intended outputs. Regional Review Committees or equivalents have been formed in each region and they establish regional funding priorities each year. Although not all regions have formal promotional and awareness plans, all of the regions undertake a variety of promotional activities to engage target communities.

A total of 1,730 projects were approved in the first three rounds of funding and the approved projects are consistent with the program objectives and regional priorities. One concern expressed is that, as the number of eligible applications increases, so too does the time required to review and approve the applications.

A summary roll-up report is produced each year. It is estimated that the majority of funded projects submit final reports within a year of finishing up their projects ^Footnote1 . Exemplary projects have been identified based on information in the applications, and shared with prospective applicants. Some regions have begun to identify Success Stories based on the outcomes described in the project final reports.

Achievement of Short-Term Outcomes

There is some preliminary evidence that the NHSP is achieving its short-term outcomes. All key informants noted that, for the most part, funding decisions matched the funding priorities and community needs. It supports the engagement of seniors in identifying the community priorities and in selecting project that address those priorities.

Both funded applicants and key informants thought that the program was encouraging the use of seniors' experience, skills and wisdom. Most of the projects involved between 30 and 150 seniors, and preliminary evidence shows that seniors are involved in organizing at least some of the projects.

Most of the funded projects continue beyond the end of the funding. The program seems to increase many organizations' capacity to serve community needs by enabling them to enhance their programs and services.

Networks and partnerships are being formed as a result of NHSP. About three-quarters of the funded organizations developed new partnerships through the NHSP-funded projects. Over half of the projects had funding partners other than the sponsoring organization (usually between 1 and 4), which may contribute to the sustainability of the projects. Funding applications were often made in partnership, in some cases with new partners, in other cases existing partners.

Funded organizations estimated that a total of $52.46 million in cash and in-kind contributions would be made by the sponsoring organizations and funding partners. This includes $37.85 million in cash contributions and $14.61 million in in-kind contributions. These contributions were perceived to contribute to increasing organizational capacity; and may also have contributed to the continuation of projects beyond the funding period

Summary of Recommendations

1. Improve the transparency of the decision-making process by providing clear and detailed written explanations for projects that are not awarded funding.
2. Put measures in place to decrease the amount of time required to make decisions regarding funding and streamline the departmental approval process.
3. Review and update the program's performance indicators for outputs, immediate outcomes, intermediate outcomes, and long-term outcomes.
4. Improve the capture and collation of information about the short-term outcomes of the program.
5. Ensure that CSGC data are complete for any fields that will be used for tracking performance or as inputs for the summative evaluation.

Introduction

In the fall of 2007, a Formative Evaluation of the New Horizons for Seniors Program (NHSP) was undertaken to fulfill a commitment made in the 2004 NHSP Treasury Board Submission. The evaluation of NSHP was conducted by Human Resources and Skills Development Canada (HRSDC). We wish to thank those who participated in the NHSP Formative Evaluation: HRSDC staff; regional Review Committee members, NHSP applicants-both funded applicants and those who applied for but did not receive funding, and stakeholders. We are pleased to provide this management response.

The NHSP Formative Evaluation was designed to focus on three specific evaluation issues: the design, delivery and management of the NHSP, which included progress made towards achieving results (i.e. short-term outcomes). Four main sources of information were used to conduct the Formative Evaluation: a review of documents and files; a review of administrative data; key informant interviews (with senior government officials, NHSP and Service Canada staff and managers, Review Committee members, and representatives from groups that represent seniors and/or are active in seniors' issues); and findings from a survey of funded and non-funded applicants.

NHSP grant funding enables non-profit organizations across Canada to carry out projects that encourage seniors to be involved with their communities by sharing their skills, wisdom and experience. The overall objective is to help ensure that seniors are able to benefit from and contribute to the quality of life in their community through their social participation and active living. The Evaluation concludes that the NHSP is being implemented as intended and that the Program is designed in a way that it can realistically attain its stated objectives and outcomes. Overall, the Formative Evaluation of the NHSP provided management with the assurance that program design and delivery promotes the intended results to be reached. We acknowledge the observations in this Formative Evaluation and intend to take into consideration the key findings and suggestions outlined therein. While some areas for improvement have been identified, generally, the key findings outlined in the formative evaluation are very positive.

The Formative Evaluation made five recommendations to NHSP; below is each recommendation and the NHSP's Management Response.

Program Implementation and Delivery

Recommendation 1: Improve the transparency of the decision making process by providing clear and detailed written explanations for projects that are not awarded funding. Currently, non-funded project proponents can request a debrief indicating precisely why they were not eligible, or how they ranked in comparison to other projects that received funding. According to program representatives, many non-funded applicants do not pursue this option.

• The Department agrees that it is important to have a transparent decision making process. To this end, steps have been taken to ensure transparency, including: detailed decision template letters tailored to the reason(s) for project rejection; encouraging applicants to contact the department for a detailed explanation by a program officer; and, providing a Question and Answer for unsuccessful applicants-on the program website- that includes information on further steps that unsuccessful applicants can follow to obtain more information on the rejected application and/or ideas for future projects.
• While the onus is on the applicant to contact the Department if they would like to discuss their application, program staff also make efforts to work with non-funded applicants to make suggestions on ways to improve their proposals for future Calls for Proposals, and include this as part of their annual outreach and promotion plans.
• Ensuring that applicants fully understand the Program's eligibility criteria when they apply for funding is key to a successful application process. When promoting the Program, Program officers suggest to organizations that they contact them as they develop a project idea to receive feedback on its potential eligibility. We have revised our program materials (application and guidelines) to reinforce this message.

Recommendation 2: Put measures in place to decrease the amount of time required to make decisions regarding funding and streamline the Departmental approval process. For example, allowing ongoing submission of applications, with the Regional Review Committees (RRCs) reviewing applications four times per year (rather than once per year) might be considered as this would likely reduce the amount of time required for funding decisions at any given sitting. It would also put less pressure on organizations to meet deadlines and so could conceivably result in better applications.

• The Department agrees with the principle of this recommendation which is to explore ways in which funding decisions can be made more quickly. Since the launch of the program, applicants have been advised when they could expect a decision – this is included in application package, on the website and in the acknowledgement letter that is sent to organizations when they submit an application. We inform them that the approval/decision-making process normally takes approximately five months from the closing date of the Call for Proposals.
• We continually look for ways in which the application process can be improved. For example, Program staff streamlined the internal approval process through the use of templates and by fast-tracking documents where possible, so that internal efficiencies are achieved. These processes have been adopted by other HRSDC programs as a best practice. However, the high volume of the applications to be processed and the need to schedule assessment around the availability of the external review committee members limits to some extent our capacity to shorten decision timelines.
• In 2008-09, NHQ and regional officials considered the feasibility of introducing an on-going application intake process on a pilot basis--through a working group created to examine this issue. The consensus of this regional working group was that the measure would not be a feasible option at this time. The Program works with a network of external review committees, whose membership includes government officials, seniors, and representatives from seniors and other community organizations. The committee members, who are mainly volunteers, assess and recommend proposals. For the on-going intake process to be effective, the committees would have to meet on a more frequent basis to assess and recommend applications.
• A key concern is the capacity and availability of the review committee members to participate in additional assessment processes. The review committees were established in 2004 when the Program was smaller, both in terms of budget and the number of funding streams. The committees now review a large volume of applications for both the CPL and CA funding components -- over 2200 applications were reviewed in 2008-09. Review committee members make a significant time commitment to come together twice a year for the CPL and CA assessment processes. Many committee members have expressed concerns about the extent of the commitment that is now required of them and would be reluctant to participate in additional meetings.
• The application intake and assessment process will be reviewed in 2009-10 in light of the findings of the implementation review of the CA funding stream, which includes key informant interviews with review committee members.

Performance Measurement and Monitoring

Recommendation 3: Review and update the program's performance indicators foroutputs, immediate outcomes, intermediate outcomes, and long-term outcomes. Indicators should be directly relevant to the output or outcome, feasible to measure over time, and, cumulatively, should provide sufficient benefit to justify the burden of ongoing measurement.

• The Department agrees with this recommendation. The current performance measurement framework (PMF) contains key performance indicators (KPI's) which are relevant to the outputs and immediate/ intermediate outcomes contained in the Program's Logic Model. These KPI's are populated annually for each round of funding-after the project approval process and reports are produced annually. These KPI's are strong output indicators, however, developing strong outcome indicators for the PMF has been challenging. Upon review of the recommendations in the formative evaluation report (i.e. the assessment of NHSP performance measures within the Document Review Technical Report), Program staff did further work on the PMF. This work included adding some indicators as recommended by the evaluators and further refinement of the wording of other indicators. This activity was completed in fall 2008.
• Measuring the achievement of long-term outcomes is very challenging for most programs of a social development nature. In the case of the NHSP, the stated long-term outcome is "vibrant and inclusive communities that benefit from the increased participation of seniors in community life". Measuring the degree of achievement of this outcome is complex due to the attribution effect, i.e. there are other social development initiatives (by various levels of government) that also positively contribute towards the achievement of this outcome. The evaluators have provided some suggested outcome indicators. These require further study, along with other potential indicators, to determine whether appropriate indicators can be identified that can be measured in a practical, cost-effective and ongoing way by project sponsors or through the evaluation process, (e.g. "actual" networking by seniors within funded projects may be difficult to quantify).
• The Impact Evaluation of CPL is now underway and early results, which will provide additional feedback on the program outcomes and measures, will be available in fall 2009. While the Program awaits the results of this evaluation activity, a review will be initiated in Fall 2009 of the NHSP logic model, outcomes and related measures to ensure that both the cause and effect relationships are consistent with the Program's intent and that the appropriate outcomes, indicators and measures can be developed. Guidance will be sought from experts in the field of evaluating social development programs to assist with this exercise that will target on arriving at fewer and better Key Performance Indicators for NHSP.

Recommendation 4: Improve the capture and collation of information about the short-term outcomes of the program. This information is available in the project final reports, but is not captured in a database that can be readily analysed. One possibility would be to adapt the CSGC to capture this type of information.

• The Department agrees with this recommendation. NHQ recognizes that the final reports from projects provide valuable quantitative and qualitative information that is essential to report on a number of the KPI's. The current project management database, the Department's Common System for Grants and Contributions,(CSGC) captures some of the final report information in the Close-out Summary screen. However, this database does not meet the Program's ongoing performance and program reporting requirements as it was not designed in a way that can capture all of the pertinent information from the final reports. An analysis of the feasibility of modifying the CSGC Close-out summary is underway.
• To counter for this weakness in the CSGC, NHSP Program officials have developed a data collection process to capture a set group of success indicators from the final reports information of the Community Participation and Leadership (CPL) component. Program officials in the regions have been tasked with the responsibility of extracting and recording the data from the final reports in a nationally developed tool. This information will be rolled up into a Microsoft Access database which will then allow the data to be synthesized and used for analysis. This activity has commenced and is scheduled for completion in spring 2009. In 2009-10, it will be expanded to capture information for projects funded under the Capital Assistance and Elder Abuse Awareness components of the program.

Recommendation 5: Ensure that CSGC data are complete for any fields that will be used for tracking performance or for the summative evaluation (e.g., number of participants, number of seniors, approval date, OLMC, amount of funding requested, project status).

• The Department agrees with this recommendation. These activities are ongoing and Program officials will continue to stress the importance of comprehensive and accurate data entry. NHQ provides guidance to the Regional program staff on the Program's forms and templates and the CSGC data entry procedures and timelines. In summer 2008, NHQ updated the program's operational manual for CSGC data entry. Regional staff are reminded about new data entry requirements and or emerging issues through e-mails, teleconferences and bulletins. CSGC data entry and the importance of field completion- including non-mandatory fields used in classifying projects-are also addressed during learning forums and orientation/training sessions.
• The new Service Canada structural model for program delivery brings together labour market and social development programming under a consistent organizational structure across regions. This should also provide enhanced opportunities for more consistent processes and training.

Future Application of NHSP Formative Evaluation Recommendations

In summary, the initial conclusions of the evaluation are quite positive and provide good recommendations for improvements to the Department's management of the Program. This evaluation further contributes to the Department's accountability and has resulted in the further refinement of the key performance indicators and the development of a data collection strategy for final reporting information. The evaluation conclusions and recommendations will continue to inform NHSP's planning, priorities and future investments in results reporting.

1.1 Program Description

The NHSP provides grants of up to $25,000 to community organizations for a wide range of projects across Canada that encourage seniors to contribute their skills, experiences and wisdom in support of the social well-being in their communities, to build community capacity and to reduce the risk of social isolation of seniors. The specific objectives, identified in the program's 2004 Terms and Conditions, are to:

• Harness the skills, experience and wisdom of seniors to help themselves and their community;
• Reduce the risk of social isolation of seniors;
• Strengthen social foundations at the community level and invest in social wellbeing;
• Ensure all seniors are able to benefit from, and contribute to, the quality of life in their community through social participation and lifelong active living; and
• Develop governance approaches that encourage provincial, territorial and municipal governments as well as voluntary, social economy and non-profit sector, to work collaboratively within communities to effectively address existing or emerging issues.

1.1.1 Inputs

The NHSP was implemented following the 2004 Budget, which announced $8 million in funding for 2004-05. In response to overwhelming demand, the Government of Canada announced an increase in funding to the NHSP by an additional $5 million (for a total budget of $13 million) in 2005-06, and $10 million (for a total budget of $18 million) in 2006-07. Table 1 provides a breakdown of the budget for each fiscal year.

• La source du tableau ci-dessus Sources: Grants budget from annual roll-up reports; total budget from NHSP Allocation Model & Operations Guide; O&M budget computed by subtracting the grants budget from the total budget

1.1.2 Activities and Outputs

Figure 1 shows the logic model for NHSP as of April 2007, depicting the activities and outcomes of the program during the time period of the evaluation. ^Footnote2 The logic model reflects the activities, outputs, and outcomes of the program, with one exception: the output of "networks and partnerships" was subsequently eliminated as an output ^Footnote3 , and so is not included in this evaluation.

The specific activities of the NHSP are:

• Community Engagement: Informing stakeholders and targeted communities about what the NHSP is trying to accomplish and how organizations can access funding (e.g., through meetings, speaking engagements, stakeholder consultations, distribution of written information, and the NHSP web site). It also includes the establishment of Regional Review Committees and the setting of priorities to guide calls for applications.
• Grants Administration: Administering grants agreements encompasses a range of activities, such as developing the call for applications, screening proposals, assessing and recommending applications, preparing grant letters, monitoring and reporting on the impact of funded projects, and closing out the projects once they are complete.
• Performance Management: Ongoing program management and improvement that includes reviewing project final reports to identify Success Stories that could be shared with the NHSP's stakeholder groups; producing a Best Practices report; continuous review and improvement of program processes; and identification of emerging issues that could affect the program's success.

The outputs of these activities are:

• Identification of funding priorities that are appropriate to the regions.
• Development and implementation of regional promotions/awareness plans, so that groups who might apply and be eligible for funding are aware of the program and understand both the purpose of the funding and the selection criteria.
• Granting of funds to projects that meet program criteria and regional priorities.
• Annual summary reports for the program, continuous program improvements, and examples of Success Stories that can be used as a promotional tool to develop similar projects in other communities.

1.1.3 Grant Eligibility

Activities eligible for NHSP funding include: the active engagement of seniors in the social well-being of their communities; promotion of social citizenship; strengthening of social capital through partnership and network building; investments in community social infrastructure; and knowledge generation and learning that results in social innovation.

Activities must be incremental, augmenting and not duplicating those currently existing in the community. They must be non-profit in nature; address local community priorities and interests; be consistent with NHSP objectives; and respond to departmental priorities. In addition, activities must be inspired by seniors, implemented by seniors and benefit seniors.

1.1.4 Program Outcomes

The funded projects should collectively contribute to the NHSP short-term outcomes, which are expected to be achieved within one to three years. These are:

• Community priorities are addressed;
• Seniors' experience, skills and wisdom are utilized;
• Funded organizations have increased capacity to serve community needs;
• Seniors are connected through networks and partnerships.

The short-term outcomes should to lead to intermediate outcomes that will be achieved within three to five years. The intermediate outcomes for NHSP are:

• Community capacity to respond to existing or emerging social challenges.
• Social participation, empowerment and inclusion of seniors.
• Engagement of seniors in the community.

Over the long-term, the NHSP is intended to result in vibrant and inclusive communities that benefit from the increased participation of seniors in community life.

1.1.5 Program Management and Delivery

The New Horizons for Seniors Division of National Headquarters (NHQ) is responsible for managing the program. This includes responsibility for monitoring all program expenditures and activities, both at the national headquarters and the regional levels, as well as for monitoring projects and their outcomes. Program outcomes are assessed in the context of evaluations.

Service Canada regional staff are responsible for the ongoing delivery of NHSP. Regional staff are also responsible for community engagement and liaison, establishing and supporting the operations of their Regional Review Committees (RRCs) and establishing and maintaining partnerships with provincial and territorial officials on NHSP matters. Financial resources are provided to regions to support these functions. Program delivery does vary by region (e.g., different stakeholder engagement methods are used in different regions; each region has its own set of priorities for funding, etc.).

In all regions except Québec, voluntary Regional Review Committees (RRCs) have been established with representation from all three levels of government, seniors' organizations, seniors themselves, and other people active in seniors' issues. This approach is intended to take account of regional differences in program delivery. Supported by regional staff, RRCs in each province and territory are mandated to set priorities for project solicitation, develop criteria for individual project assessment, and recommend and rank projects for approval by HRSDC.

In Québec, a Joint Management Committee (JMC) fills the role of the RRC. The JMC includes regional NHSP representatives, as well as representatives from the Secrétariat aux aînés du Québec. This arrangement enables the province to coordinate the NHSP funding process with that province's Du coeur à l'action pour les aînés du Québec / Heartfelt Action for Québec's Seniors (formerly the Engagés dans l'action pour les aînés program), which has similar objectives relating to the social participation of seniors.

1.2 Evaluation Context

The formative evaluation was intended to help improve the program. It focused on design, delivery and management issues, and examined activities, outputs and short-term outcomes of NHSP. The evaluation was guided by the following questions.

Program Design

1. Is the program designed in such a way that it can realistically attain its stated objectives and anticipated outcomes?

• Does the logic model reflect how the program currently operates?
• Is the logic model adequately reflecting the program links between the activities and the expected outcomes?

Program Implementation and Delivery

2. Is the NHSP implemented and delivered as intended?

• Should changes be made to program delivery to improve its efficiency and effectiveness?
• Are resources sufficient?

3. What efforts have been made to raise the stakeholder community's awareness of the program?

• What kind of outreach was done?
• How effective was it?
• What are the barriers, if any?

4. Does the program have adequate management and performance information and administrative data/systems to effectively and efficiently manage the program?

• Are the performance measurement indicators adequate and appropriate?
• What is the quality of the available performance data?
• Is the data needed for a summative evaluation being collected?
• Are there gaps to be filled?

Early Results

5. To what extent has the program produced its intended outputs?

• What is the profile of projects in each of the regions?
• What barriers work against achieving the intended outputs?
• What supports the achievement of the intended outputs?

6. To what extent has the program achieved its short-term outcomes?

• What barriers work against achieving the intended outcomes?
• What supports the achievement of the intended outcomes?
• What unintended outcomes have occurred?

2.1 Document Review

Program documents were reviewed at both the national and regional levels. At the national level, documents included the program's Results-Based Management and Accountability Framework (RMAF), Performance Measurement Framework, Program Terms and Conditions, the Early Implementation Review report, the Operations Guide, budgets, annual reports, forms used in delivering the program, and communications with the regions. These documents helped to develop a description of the program. Regional documents included promotions/awareness plans, records of outreach and communications, regional review committee records, examples of innovative projects, and any region-specific tools used in screening and assessing proposals. These documents, by providing information about how the program operated at the regional level, were useful in highlighting the similarities and differences between regions.

The information from the documents was analyzed using content analysis based on the evaluation questions. Relevant information from each document was organized by evaluation question, and the overall findings and conclusions were summarized for each evaluation question, with notations of agreement or discrepancies between different documents.

2.2 Key Informant Interviews

Interviews were conducted in February and March, 2008 with 58 key informants: two senior government officials; 22 NHSP staff and managers (both national and in each region); 14 Regional Review Committee (RRC) members (including all regions except for the Northwest Territories); and 20 representatives from stakeholder groups (i.e., groups that represent seniors and/or are active in seniors issues at the national level and in each region. Some, but not all of these groups, had received funding from NHSP). The breakdown of interviews by group and region is shown in Table 2.

Interviews were conducted by telephone because of the large geographic distances involved in this project. Using telephone interviews made it possible to conduct a large number of interviews within the evaluation budget, and also enabled the evaluators to schedule interviews at times convenient to the respondents.

Key informants were selected purposefully, so that those interviewed were the most knowledgeable within their group about the program as implemented between 2004-05 and 2006-07. The national managers identified appropriate regional staff and senior officials to interview, and regional program staff identified appropriate RRC members and stakeholder groups.

Depending on their level of involvement with the program, key informants were asked about program design, delivery, communication and outreach, administrative/performance information, the process of making funding decisions, and program improvements. Interviews with program staff, managers, and RRC members were approximately one hour. Interviews with senior officials and other stakeholder groups lasted between 20-30 minutes. All but one of the interviews were conducted by telephone.

A coding key was developed after a preliminary review of the interview notes. The coding key included common themes that emerged from the notes. The interview data were then analysed thematically using NVivo qualitative analysis software. Interview responses were coded according to the type of key informant and region, so that themes for each group could be identified if required. Each theme was then summarized, using the words of the respondents if possible.

An indicator of the number of times a theme was mentioned was incorporated into the narrative, using the following descriptors:

• No/none: no individual identified the particular issue or topic.
• Few/very few: one or two individuals expressed a particular opinion.
• A minority: between one-quarter and one-half of the individuals interviewed expressed a particular opinion.
• A majority: the majority of, but not all, individuals were of the same opinion and/or held similar perceptions regarding a selected issue or topic.
• All: All interviewees questioned on the topic expressed the same view or held the same/similar opinion.

2.3 Administrative Data Review

The administrative data review included information stored electronically in the Common System for Grants and Contributions (CSGC), as well as information from the paper files of a stratified random sample of applications and funded projects.

The CSGC contains data on NHSP activities and expenditures, project proposal information, and project reporting information. The review of CSGC data included all applications submitted between 2004-05 and 2006-07, inclusive. The relevant fields were extracted from the database, and imported and analysed using SPSS.

The paper files included application forms, screening and assessment materials, and for funded projects final project reports. A random sample of these files was reviewed, stratified by funding year and application status (ineligible, rejected, and awarded). The purpose was to gather data for the evaluation, as well as to assess the quality of the data in CSGC. The sample included, for each funding cycle and region:

• 5% of applications that were deemed ineligible (sample size = 81);
• 5% of applications that were eligible but rejected (sample size = 37); and
• 10% of applications that were awarded funding (sample size = 173, but only 168 files were provided by the regions).

The sample was drawn by the evaluators, and the regional staff provided copies of the paper files for the sampled projects. Relevant information from the files was entered into an SPSS database and analysed. Qualitative data were coded, using a coding key, prior to being entered into the database.

The accuracy and completeness of the CSGC data were also assessed, with the sample of paper files serving as the reference point for assessing accuracy.

2.4 Survey of Applicants

A telephone survey was conducted in August and September 2007 with two main audiences:

• Organizations that received funding from NHSP; and
• Organizations that applied for funding but did not receive funding (i.e., were deemed ineligible for funding or were rejected).

The survey samples were stratified by region, project component, funding amount and year of funding, as well as by application status (rejected or ineligible) for those who applied for funding but did not receive it. The samples included applicants who had applied for funding between 2004-05 and 2006-07 inclusive.

The survey questionnaires were designed in collaboration with HRSDC Evaluation Directorate. The questionnaires asked about characteristics of the grant recipient organizations, outcomes of the funded projects, and perceptions of various elements of the program. The surveys were conducted during daytime business hours, using Computer Assisted Telephone Interviewer (CATI) software.

A total of 1,745 interviews were completed, including 958 interviews with funding recipients and 787 with applicants who did not receive funding. The response rate for the "funded" sample was 56%, while the response rate for the non-funded sample was 69%. The approximate margin of error for a sample of this size is +/-2.8 for the funded sample and +/-3.1 for the non-funded sample, at the 95% confidence interval.

Responses to open-ended questions were coded thematically. Frequencies were calculated for both open- and closed-ended items. Frequencies were computed separately for those applicants who received funding and those who did not. The data were weighted to ensure that the aggregate sample was representative of the entire universe, based on region, funding cycle, and amount of funding.

2.5 Strengths and Limitations of the Methodology

The evaluation collected data from multiple sources using multiple methods; data triangulation increases confidence in the findings.

The documentation for the NHSP is extensive and the documents are largely consistent with one another, which enhances our confidence in the findings from the document review.

The large number of topics covered in the telephone interviews present both strengths and limitations: it allowed interviewers to capture opinions on a wide range of issues; however, there was little time for interviewers to probe for more in-depth responses. The mode of interview (telephone) also had some disadvantages, as it prevented the interviewer from noticing non-verbal cues.

The NHSP collects information in the Common Systems for Grants and Contributions (CSGC) database about every application it receives. For the most part, the relevant data in this system were complete and accurate. This provides a highly reliable source of data for the evaluation. However, there were some areas where the data were incomplete (e.g., the number of participants in the funded projects was not completed for every project). Furthermore, in some instances, there were slight differences between reports generated by the program and the results obtained through the CSGC, and it was beyond the scope of the evaluation to determine which numbers were most accurate. These instances have been noted in the text of the report, and the source used has been clearly identified.

The review of paper files supplemented the CSGC, as the files included more extensive information about the projects, including project outcomes based on the final reports. Time and budgetary constraints required that a relatively small sample of paper files be drawn. In particular, only 10% of the funded projects were reviewed. This is not a statistically representative sample. Therefore, findings from the paper files (and particularly from the final project reports) need to be treated as a rough estimate rather than a precise reflection of what is happening in all projects.

Finally, information about the short-term program outcomes is based primarily on perceptions, and should therefore be interpreted with caution. Key informants often indicated that they were speculating when providing information about the program outcomes.

3.1 Program Design

Evaluation Question 1: Is the program designed in such a way that it can realistically attain its stated objectives and anticipated outcomes?

• Does the logic model reflect how the program currently operates?
• Is the logic model adequately reflecting the program links between the activities and the expected outcomes?
• Are resources sufficient?

Accuracy of the Logic Model

Figure 1 (see page 3) shows the logic model for NHSP as of April, 2007, depicting the activities and outcomes of the program when it was implemented in 2004. The activities and outcomes are described in section 2.1 of this report.

The logic model is consistent with other program documents reviewed for the evaluation, and accurately reflects the activities, outputs, short-term outcomes of the program, with one exception: the output of "networks and partnerships" identified in the logic model was eliminated as an output and made instead an outcome ^Footnote4 .

There is overlap amongst several of the outcomes in the logic model. For example, the intermediate outcome of engagement of seniors in the community overlaps substantially with that of social participation and inclusion of seniors. This overlap makes the logic model less useful as a tool for a) clarifying the goals of the program, and b) providing direction for the summative evaluation.

Adequacy of Program Resources

The NHSP grant funds are allocated to the regions using a national allocation model. Table 3 presents financial information about the grant funds allocated and awarded within each region, as found in the annual roll-up reports. Looking at this information over time sheds some light on the issue of resource sufficiency.

• La source du tableau ci-dessus Source: annual roll-up reports

The overall grant funds allocated to the regions, as well as the total funds approved, increased steadily over the three-year period, as can be seen in Table 3 as well as Figure 2. In 2004/05, the funds allocated for grants were $5 million, and the total amount approved was the same. In subsequent years, the grants budget increased substantially. In each of these years, the budget exceeded the amount of grants approved. The grants budget allowed NHSP to fund 1,730 projects, including 338 in 2004/05, 614 in 2005/06, and 778 in 2006/07.

The average grant size per funded project was $17,232.

A majority of the staff, managers, senior officials, and RRC members who were interviewed for the evaluation thought that the program budget was sufficient. A few respondents felt that the program budget was insufficient to fund all of the projects they had wanted to support.

The differences in opinion about the sufficiency of the program budget may be due to differences in allocation among the regions. According to the report from the Early Implementation Review, the Resource Allocation Model for NHSP grant funding was developed based on the size of the seniors' population in each region, and was applied consistently across the country. Grant funds were allocated to each of the regions using a base allocation of $50,000 ($60,000 in 2005/06 and $70,000 in 2006/07), which was topped up based on the percentage of seniors in the region. From Table 3, it can be seen that some regions spent more than their allocated amounts, suggesting that the allocated resources were in fact insufficient. However, unused funding from larger regions is commonly re-distributed to smaller regions on an as-need basis. For example, while Saskatchewan has consistently under-spent its allocated budget, Newfoundland and Labrador has paid out over 30% more than its allocated budget for the second and third rounds of funding. Regional overspending occurred mostly in 2005-06 in the territories and Eastern provinces.

Twenty percent of the funded projects were awarded less funding than they had requested ^Footnote5 . These projects were awarded, on average, about $7,000 less than they had requested. Some regions had much larger percentages of projects that were awarded less funding than they had requested, such as the Northwest Territories (69%), Prince Edward Island (64%), Manitoba (48%), and Québec (40%). The evaluation did not identify the reasons for this, but the program may wish to explore this phenomenon further ^Footnote6 .

Grant Size

The majority of the key informants interviewed thought that the grants were of sufficient size. However, a minority thought that the grants were too small, especially in remote areas such as the Northwest Territories or Nunavut (interviewees noted that program materials are more expensive in these regions, because the cost of shipping and transportation is high). It was also noted that while $25,000 was sufficient to support local projects, the ceiling on project funding precluded funding on a province-wide scale. However, funding of province-wide projects is not one of the program's objectives.

A review of the applications in the CSGC database shows that 69% of applicants requested less than $25,000. On average, applicants requested $18,691. Combined with the interview responses, this suggests that for most applicants that are being targeted by the program, the $25,000 grant size is sufficient.

Adequacy of Program Activities/Outputs

Key informants were asked to comment on the likelihood of achieving the program objectives, in light of the activities and resources available. A majority indicated that the program was likely to achieve its objectives. Of these, a majority thought that it was important that the program was grants-based, because the simpler administration, application, and reporting process makes it possible for smaller organizations to apply for the funding. A majority also thought that the use of RRCs to review the applications was particularly effective, as the RRC members bring a lot of experience and knowledge about the community and its needs, and are able to make better decisions than individuals who might not be as familiar with the needs of the region.

A minority of the interviewees thought that the program was somewhat likely to achieve its objectives, and very few thought that it was not at all likely. Some of the concerns raised by these interviewees were that the projects would not continue without ongoing funding, especially when organizations already have ongoing projects that require funding. Another concern was that the projects are by necessity short-term, since the funding covers only a one-year time frame.

Other interviewees noted that the program provides seed money that gives organizations an opportunity to set up projects that address community needs and, if they turn out well, the organization may then be able to sustain them over the long run.

Anticipated Project Outcomes

A review of a sample of the project application forms found the following:

• 84% of applicants expected the project to increase inclusion and decrease isolation for seniors;
• 64% expected it to give seniors an opportunity to express themselves or share their knowledge, skills and values with others;
• 54% expected to build intergenerational understanding and respect; and
• 44% expected to develop seniors' knowledge and skills.

These expected outcomes are consistent with the program objectives.

Summary

• The logic model is consistent with other program documents and for the most part does accurately reflect the activities, output, and outcomes of the program. However, overlap amongst the outcomes reduces the clarity of the model.
• The funds awarded and number of projects funded increased substantially over the three years, as the program budget increased.
• By 2006/07, there seems to have been sufficient funds allocated, at a national level, to fund all projects deemed by the RRCs to be deserving of funding, although some regions required funding beyond their regional allocations.
• Grant size was thought to be sufficient by most (but not all) interviewees. Over two thirds of the applications were for less than the maximum grant per project of $25,000.
• Key strengths of the program design, according to interviewees, were the fact that it is grants-based, and the use of RRCs to review applications.
• Anticipated project outcomes appear to be consistent with the program objectives, which increases the likelihood of achieving the program goals.

3.2 Program Implementation

Evaluation Question 2: Is the NHSP implemented and delivered as intended?

Should changes be made to program delivery to improve its efficiency and effectiveness?

Planned and Actual Implementation

As described in section 2.1.5, the NHSP is intended to be managed at a national level by the National Headquarters (NHQ), which includes overarching planning, communications with the regions, maintaining the NHSP website, staff training, monitoring, and continuous program improvement. The program is delivered by HRSDC/Service Canada regional staff. Each year, the following steps are intended to take place:

1. Regional funding priorities, established by the RRCs, are used as the basis for the Call for Applications.
2. Regional staff engage local organizations to inform them about the Call for Applications and provide assistance in preparing proposals (e.g., by helping them understand what is required and what types of projects are eligible for funding).
3. Applications are submitted on or before the application deadline.
4. Information from the applications is entered into the CSGC database. This information is updated as the application is processed and as funded projects are carried out up to the time projects are closed out.
5. Regional staff screen applications to ensure they meet the program's eligibility requirements. In some limited instances, regional staff may request further information from the applicant.
6. RRCs meet to review the eligible proposals and carry out a more thorough assessment of whether or not the application will contribute to the program goals.
7. RRCs make a recommendation about whether to approve or reject the application.
8. Final approval is made by a delegated authority within Service Canada or HRSDC (a regional director or the Minister).
9. Letters are sent out notifying successful applicants of how much funding they have received. Unsuccessful applicants are informed the reasons why they were not awarded funding.
10. Funding is distributed and the projects are implemented.
11. The local Members of Parliament make announcements in their communities.
12. Regional staff monitor the funded projects (at a minimum, calls are made one month after the funding is issued and one month before the planned end date of the project).
13. Projects submit a final report upon completion of the project.
14. Regional staff review final reports and identify Success Stories.
15. An annual roll-up report and Best Practices report are produced.
16. Program staff identify improvements to the program prior to the next funding round.

Both the interviews and the review of documents indicate that the program is being implemented as intended, with some minor variations. The work processes are well-documented, and the tools and templates for program delivery that were developed by NHQ are being consistently used by the regions. There is some variance and flexibility in how the program is delivered in each region, and this is in line with the NHSP desire to be responsive to unique community needs and circumstances.

The only major variations from the intended process are:

• Exemplary projects are identified based on applications. The program appears to be working towards documenting Success Stories based on the project final reports, but this is not yet done in all regions.
• Although no Best Practices report has yet been produced, program representatives indicated that best practices and improvements are discussed with all Regions at the Annual Learning Forum.

The program is implemented somewhat differently in Québec, where a Joint Management Committee of federal and provincial government representatives has been established to fill the roll of the RRC. Community input is received from Quebec's network of Tables de concertation des aînés. This arrangement is formalized in a Memorandum of Understanding between the Québec government and the government of Canada. The JMC takes into account both the objectives of the NHSP and the objectives of the Du coeur à l'action pour les aînés du Québec / Heartfelt Action for Québec's Seniors (formerly the Engagés dans l'action pour les aînés program, a substantially similar program in Québec) when making recommendations.

Previous Year Program Reporting

Previous year program reporting is intended to ensure continuous program improvement as well as accountability. Outputs should include a summary roll-up report for each round of funding, the identification of "Success Stories" to promote the development of similar projects, and final reports submitted by the funded projects.

Annual summary roll-up reports were produced for each of the first three rounds of funding.

"Exemplary projects" have been identified for each region and posted on the NHSP website. According to key informants, the exemplary projects were identified based on project application forms, and are therefore examples of "good ideas" rather than "projects that had the desired outcomes." It appears that at this point, some regions have gone beyond this to identify "Success Stories" based on the project outcomes as described in the final project reports, and the intent is that all regions will eventually do so. In some regions, staff actively seek out Success Stories, and in one region the Service Canada staff provide a report summarizing the projects and their respective levels of success. This practice varies by region.

Exemplary projects and Success Stories are shared through the following vehicles:

• The regional staff or representatives of funded projects share project examples with prospective applicants (either in person by workshops and community visits, or through newspaper ads, brochures/fact sheets, DVD, word of mouth).
• Exemplary projects are posted on the National website. Applicants are encouraged to look at the website for ideas.
• While Success Stories are beginning to be collected by regional staff in some regions, there does not seem to be a consistent method for distribution of these stories at this point in time.

Barriers and Supports to Achieving the Intended Outputs

Key informants identified very few barriers and supports to the achievement of the intended outputs.

Barriers that were perceived by key informants included stakeholders not working well together, lack of awareness of the program, a lack of understanding of cultural diversity, favouring specific groups or communities over others, and the call for applications being too short. None of these barriers emerged as strong themes among the key informant interviews.

A minority of the key informants identified factors that they thought supported the achievement of the NHSP outputs, including the visibility of NHSP within the community, the available funding, and NHSP staff support. Again, none of these emerged as strong themes among the key informant interviews.

Summary

• Regional Review Committees have been formed that are, for the most part, representative of the diversity of their regions, and RRCs establish regional funding priorities each year.
• Although not all regions have formal promotions and awareness plans, all of the regions except for Québec have sent out targeted mailings to promote the NHSP. All of the regions undertake a variety of promotional activities to engage target communities.
• A total of 1,730 projects were approved in the first three rounds of funding. The approved projects are consistent with the program objectives and regional priorities. The only concern is that, as the number of eligible applications increases, so too does the time required to review and approve the applications.
• A summary roll-up report is produced each year. It is estimated that about 85% of funded projects submit final reports within a year of finishing up their projects. Exemplary projects have been identified and shared with prospective applicants. Some regions have begun to identify Success Stories based on the outcomes described in the project final reports, but not all regions have started this process as yet.

3.6 Short-term Program Outcomes

Evaluation Question 6: To what extent has the program achieved its short-term outcomes?

• What barriers work against achieving the intended outcomes?
• What supports the achievement of the intended outcomes?
• Were there any unintended outcomes?
• The NHSP intended immediate outcomes are:
• Community priorities are addressed;
• Seniors' experience, skills and wisdom are utilized;
• Funded organizations have increased capacity to serve community needs;
• Seniors are connected through networks and partnerships.

Community Priorities are Addressed

Our review of a sample of final reports showed that the most common benefits to the communities were:

• Increased inclusion and decreased isolation for seniors (84%);
• Development of seniors' knowledge/skills (42%);
• Building intergenerational understanding/respect/connections (42%);
• Opportunity for seniors to express selves, share knowledge skills & values (38%);
• Recreation/enjoyment (28%); and
• Promotion of active living, exercise (24%).

Further research is needed to determine the extent to which these benefits are aligned with specific community needs. However, 85% of the funded applicants who were surveyed indicated that their projects contributed somewhat or a lot to enhancing community capacity to respond to existing or emerging challenges, which does suggest that community priorities may be better addressed as a result of the program.

Program Design

The program appears to be designed in a way that it can realistically attain its stated objectives and outcomes. The anticipated outcomes of the funded projects are consistent with the program outcomes and objectives, which increases the likelihood that the program goals will be achieved. The logic model, while consistent with program documents, needs to be revisited with respect to overlapping outcomes.

Most key informants felt that the program budget was sufficient and that the grant size was adequate. A total of 1,730 projects have been over a three-year period. Over two-thirds of the applications were for less than $25,000.

The main strengths of the program design identified by key informants included the fact that it was grants-based and the use of Regional Review Committees (RRC) to review applications.

Implementation and Delivery

For the most part, the NHSP is being implemented as intended. Each region has an RRC or equivalent, and there is flexibility in how the program is delivered in each region, reflecting unique community needs and circumstances.

At the national level, there has been ongoing monitoring, and improvements have been made to the program tools and processes over time. At this time, the only elements of the program that have not been carried out as intended are: 1) the identification of Success Stories based on project final reports, which some regions are not doing yet, and 2) the Best Practices report, which has been replaced with an Annual Learning Forum where best practices and program improvements are discussed with all regions.

Funded applicants were satisfied with most aspects of the program, including the eligibility criteria, final reporting requirements and application guidelines. Although non-funded applicants were less satisfied with some aspects of the program, a majority said that they would re-apply, and would recommend to other organizations that they apply for funding.

Those who did not receive funding felt that the program's eligibility criteria lacked clarity and were dissatisfied with both the transparency of the proposal review process and the explanation as to why their organization was not eligible to receive funding. Funded and non-funded applicants both indicated lower levels of satisfaction with the time it took to receive notification of the funding decision.

There appears to be relatively little duplication or overlap with other Canadian programs. There are similar programs at the provincial level in Nova Scotia and Québec, as well as for First Nations groups in British Columbia; but for the most part, the NHSP fills a unique niche in that there are few other grants programs that promote the involvement of seniors in their communities.

Outreach and Promotion

The regional staff have used a wide variety of methods to engage the communities in their regions, including mail-outs of applications kits, presentations to communities or groups, advertisements, and one-on-one communication.

Many survey respondents became aware of the NHSP through program-related awareness packages (the website, mailed information package, and public notice/newspaper). Most respondents indicated that they would prefer future communications about the program through an information package in the mail or to have the information posted on the NHSP website.

Key informants thought that outreach efforts were most effective when there was in-person contact and assistance with the application. Some form of in-person contact is taking place in every region.

The majority of funded and non-funded applicants had visited the NHSP website and both groups felt that it was an effective communications tool.

5. Recommendations

1. Improve the transparency of the decision-making process by providing clear and detailed written explanations for projects that are not awarded funding. Currently, non-funded projects can request a debrief indicating precisely why they were not eligible, or how they ranked in comparison to other projects that received funding. According to program representatives, many non-funded applicants do not pursue this option.
2. Put measures in place to decrease the amount of time required to make decisions regarding funding and streamline the departmental approval process. For example, allowing ongoing submission of applications, with the RRCs reviewing applications four times per year (rather than once per year) might be considered as this would likely reduce the amount required for decisions at any given sitting. It would also put less pressure on organizations to meet deadlines and so could conceivably result in better applications.
3. Review and update the program's performance indicators for outputs, immediate outcomes, intermediate outcomes, and long-term outcomes. Indicators should be directly relevant to the output or outcome, feasible to measure over time, and, cumulatively, should provide sufficient benefit to justify the burden of ongoing measurement.
4. Improve the capture and collation of information about the short-term outcomes of the program. This information is available in the project final reports, but is not captured in a database that can be readily analysed. One possibility would be to adapt the CSGC to capture this type of information.
5. Ensure that CSGC data are complete for any fields that will be used for tracking performance or for the summative evaluation evaluation (e.g., number of participants, number of seniors, approval date, OLMC, amount of funding requested, project status).

Appendix A – Issues Matrix