Formative Evaluation : New Horizons for Seniors Program Elder Abuse Awareness Component, August 2010

Evaluation Scope and Methodology

The goal of the evaluation is to provide decision makers with evidence related mainly to design and delivery and performance measurement and monitoring systems, but to also address relevance, and early impacts of the CA component. At the time of the data collection for this formative evaluation, only one round of CA projects had been approved (2007-08). Therefore the evaluation report focuses on early success in achieving the intended CA outcomes. Sixteen evaluation questions were addressed.

Four sources of information were used to conduct the formative evaluation: a review of documents, a review of administrative data and project files, key informant interviews, and a survey of funded and unfunded applicants. This report summarizes the findings for these lines of evidence.

Key Findings

Early Success

The CA component is viewed by funded applicants and a majority of key informants in all groups as having a positive impact on the programs and activities (existing and new) provided for seniors and the physical settings in which these are provided. A large majority of funded survey respondents felt their facilities were more comfortable, that the safety needs of seniors were now better addressed, and they have added new programs and services as a result of the improvements made. The latter is a positive unintended outcome as CA was designed primarily to support the continuation of existing programs and services for seniors.

Some organizations (46%) not funded through the CA component went ahead with their projects, using mainly volunteer and in-kind resources, but the majority of these (70%) implemented the projects on a smaller scale than planned. Overall the evidence indicates some limited incrementality in facilities improvements funded by CA.

Based on the survey of applicants, the majority of funded organizations (70%) serve both seniors and non-seniors while some (29%) serve only seniors, indicating the potential for a broader community impact from the improvements made with CA funding.

Design and Delivery

The design and objectives of the CA component are clearly defined in program documents. The CA component was funded through grants to support efficient delivery. According to program documentation, funding the CA component through grants was considered the most efficient method due to the large volume of modest proposals that carry low financial risk.

The organizational structure and the management and administrative systems are considered, for the most part, to support efficient and effective delivery of the CA component. The exceptions are the departmental administrative database system, which is not adaptable to CA performance information requirements, and the delays in project approvals. The evidence also indicates that project monitoring and reporting are not completed for all projects. The majority of Service Canada regional staff and Regional Review Committees (RRC) key informants felt NHSP NHQ is providing appropriate guidance and sufficient support for their work.

Application Process

The evidence indicates that the application process could be improved by simplifying the language in the application form to make it easier to understand, stating the eligibility criteria more clearly, providing assistance with applications, and by providing clearer explanations to applicants of the reasons their applications were rejected. There is also a need to state more clearly the situations when fewer than three estimates for renovation projects will be acceptable. This was seen as problematic for organizations in smaller communities who have few suppliers. In addition, there is a need to ensure that all Regional Review Committees members are fully aware of their role and responsibilities.

Communications

The evidence indicates that the methods used to communicate with organizations about the CA component are largely effective, although unfunded applicants rated communications lower than funded applicants.

The large majority of NHSP NHQ and Service Canada key informants and the document review indicated that the CA component is communicated by regions to potential applicants using a variety of methods. All 10 regions distribute materials by mail or email, all but one region do presentations to groups and visits to organizations to promote the program, and seven regions inform and encourage other federal, provincial or municipal government officials to promote the component. Half the regions do outreach to assist organizations to prepare applications. A consistent communications approach across regions could improve awareness and understanding of CA.

Survey respondents most frequently reported that mail and e-mail are the best ways to reach them. Interestingly most funded survey respondents actually learned about the CA component from another organization.

Operational Costs

CA operational costs were originally estimated to be 14.5% of total CA costs. The resource allocation model suggests that the actual proportion of CA funding allocated to operational costs was 12%. The different is a result of CPL operational funds being used to offset CA operational costs. The CA component is delivered regionally by Service Canada. This delivery model is considered by NHSP NHQ as appropriate to optimize efficiency in program delivery. Determining the exact ratio of operational costs in the regions for each component is challenging as Service Canada does not differentiate between CA and CPL when reporting on operational costs.

Performance Measurement and Monitoring

Overall, the evidence indicates that the performance information on the CA component provides limited support for decision-making and departmental accountability requirements, including the key information needed for a summative evaluation. Departmental key informants indicated a need for more precise information on the CA outputs and outcomes to support program management.

The number of seniors who benefitted from the renovations or improvements made with CA funding is over-estimated as this is based on the number who were served by the organizations and not just those who participated in the service/program impacted by the CA funding.

There are issues with the accessibility, quality and completeness of the performance data in the Common System of Grants and Contributions (CSGC). The number of seniors and other participants were the two data elements most frequently recorded inaccurately (63% and 85% respectively had large differences).

Recommendations

1. Simplify the language used in the application form and ensure the eligibility criteria is communicated clearly to applicants.
   1. Revise the application form to ensure better readability and comprehension tailored to the target audience; and
   2. State the eligibility criteria more clearly, including those related to owning/leasing the building; and provide a clear explanation of situations in which three cost estimates are not required, especially in rural communities where three suppliers may not be available.
2. Ensure all Regional Review Committee members are fully aware of their role and responsibilities.
3. Clarify program outcomes to reflect what the program is specifically trying to achieve and develop a system to collect, monitor and report on program results.

Introduction

The New Horizons for Seniors Program (NHSP) helps to ensure that seniors are able to benefit from and contribute to the quality of life in their community through their social participation and active living. The Capital Assistance (CA) funding, one of the three components within the NHSP, provides grants of up to $25,000 to non-profit organizations for upgrading community facilities and equipment related to existing programs and activities for seniors. CA funding enables non-profit organizations across Canada to maintain or enhance their infrastructure in order to continue seniors' programs/activities.

A formative evaluation of the Capital Assistance (CA) component of the NHSP was undertaken in parallel with the summative evaluation of the Community Participation and Leadership (CPL) component and the formative evaluation of the Elder Abuse Awareness (EAA) component. While some areas for review and improvement were identified, generally, the key findings outlined in the formative evaluation are positive and indicate the component is designed in a way that it can realistically attain its stated objectives and outcomes.

Three recommendations were provided in the CA evaluation report, to which the program area provides a response below. The program area is currently responding to the Budget 2010 commitment of an additional $5M per year of ongoing funding to support projects related to volunteerism and mentoring among seniors and that focus on raising awareness of the financial abuse of seniors. Consequently, the Program management is examining the overall program design and delivery to respond to this commitment and will also use this opportunity to make any necessary adjustments, stemming from NHSP program evaluations.

Recommendations

1. Simplify the language used in the CA application form and ensure the eligibility criteria is communicated clearly to applicants.

1. Revise the application form to ensure better readability and comprehension tailored to the target audience;
2. State the eligibility criteria more clearly, including those related to owning/leasing the building; and provide a clear explanation of situations in which three cost estimates are not required, especially in rural communities where three suppliers may not be available.

The Department agrees with this recommendation.

Actions taken

• The CA application form was designed to handle three types of applications, i.e. funding can be requested to support the replacement of equipment/furniture, or to facilitate renovations/repairs to the facility, (or for both). Consequently, supplemental questions have been asked to determine CA eligibility, such as proof of building ownership or lease agreement, when the applicant is seeking funding for renovation/repair projects.
• Program eligibility is explained in the application form. Program officials have require-ments for three estimates, although preferred, are not a requirement. This has been explained at information sessions.

Actions proposed

• The application form, guidelines and other tools/templates will be reviewed (over the next year) in light of program design changes stemming from the Budget 2010 commit-ments. The proposed program design changes include consolidating the program which will result in Call for Proposals (CFPs) articulating focused program priorities, instead of component-specific CFPs. Furthermore, a single application form will be adopted for the Program.
• Program staff and review committee members will be instrumental in communicating program eligibility criteria and will provide information sessions.

• Ensure Regional Review Committee members are fully aware of their role and responsibilities
• The Department agrees with this recommendation.
• While this recommendation also applies to the CPL component, the focus by some review committees on the proposed budgets in CA proposals rather than on overarching priorities was evident.

Actions taken

• In the past, new members joining a review committee (RC) were informed about their roles and responsibilities through the provision of various tools (i.e. Training Binder for RC members, Terms of Reference and orientation). Messaging and communications on roles, responsibilities and priorities have been reinforced during RC meetings in the Regions.
• The expectation of the role of the RC for CA is to identify priorities for their Region and assess the value of projects to ensure they meet the NHSP overall priorities. The CA process was streamlined to ensure that Service Canada staff would assess the budgets for CA projects and assess the value for money. This reduced the administrative burden on the RC and further allowed for more time to be spent on activities such as priority setting and strategic planning.

Actions proposed

• Program management will provide further direction to regional staff (over the next year) with respect to their role of providing support to the regional committees and ensuring that review committee members understand their role. These employees will directly support the regional committees in their work to establish funding priorities based on evidence gained through community engagement activities, environmental scans and analysis. Committee members are well-positioned to share their expertise and knowledge of their communities, and thus can help to identify priorities and create linkages to the Program. Focusing their efforts on this domain would be the best use of their expertise.
• Clarify program outcomes to reflect what the program is specifically trying to achieve and develop a system to collect and report on program results.
• The Department agrees with this recommendation.
• This recommendation applies to all components. NHSP has evolved since its inception in 2004, adding the two components in 2007. An analysis of the Program's current logic model, which contains the outcomes for the three components, indicates that some "direct" outcomes overlap and outcomes at the "shared" (or intermediate) level, such as community capacity may be difficult to measure. A further challenge is to ensure clarity of the dual purpose of the overall Program; that is, to support the involvement of seniors as contributors and beneficiaries.

Actions taken

• Program officials began revising the program logic model to remove overlap and improve the output and outcome statements to ensure the logic around each component would be clear. This work will continue through the development of a Performance Management (PM) Strategy.
• To counter the limitations in capturing, extracting and managing program-specific performance data, Program officials have developed an interim results reporting data collection tool to capture success indicators from final reports for CPL projects. However, gaps with respect to effective reporting on results remain.

Actions proposed

• By December 31, 2010, a new PM Strategy will be developed to represent the updated outcomes of the NHSP, which stem from the changes related to the Budget 2010 commitments. This Strategy will include program outcomes that are consistent with the program's intent, appropriate performance indicators, and clear roles/responsibilities in regards to data collection and analysis.
• By March 31, 2011, the program's tools and templates will be reviewed and modified to ensure that the appropriate data is being collected to meet the requirements of the new PM Strategy. It is also a program delivery goal to ensure that all forms/tools are client-focused and in plain language.
• Appropriate training (starting Winter/Spring 2011) will be provided to delivery staff to ensure they understand the program's desired outcomes and what requirements will evolve from the PM Strategy.
• Before, during and after the next CFP, the revised Program outcomes and objectives will be articulated to stakeholders through clear communications materials and the Program's Web site.

1.1 Overview of the Capital Assistance Component

1.1.2 Rationale for CA Component

Prior to the introduction of the CA component, the NHSP had provided capital assistance funding to support new – not existing – activities for seniors. The CA component was introduced to fill this gap by providing funding for upgrading community facilities and equipment related to existing programs and activities for seniors in response to a need identified by seniors' organizations.

1.1.3 Eligible Recipients and Activities

Eligible recipients for CA funding are non-profit organizations that have a track record of delivering community programs and activities for seniors. Eligible activities involve the repair, renovation or modification to existing seniors' community facilities, or the purchase of replacement equipment and furnishings which would enable non-profit organizations to continue to maintain their existing seniors' community programs and activities.

1.1.4 Funding Allocation

The total funding allocated annually for CA grants is $6.840 million. Capital Assistance provides up to $25,000 in grant funding per project for renovations, repairs or modifications to community facilities. Of this amount, up to $10,000 may be used for the purchase of replacement furnishings or equipment. Projects are a maximum of one year in duration.

1.1.5 Management Structure

HRSDC NHSP National Headquarters (NHQ) is responsible for management of the CA component including overarching planning, communications with the regions, maintaining the NHSP website, staff training, monitoring, and continuous program improvement. Regional Service Canada staff are responsible for the delivery of this component under the functional direction of NHSP NHQ. Regional responsibilities include community engagement and liaison, establishing and supporting the operations of their Regional Review Committees (RRCs) and establishing and maintaining partnerships with provincial and territorial officials on NHSP matters. The RRCs are comprised of representatives from the federal and provincial/territorial governments, seniors' serving organizations (both private and non-profit), community organizations and seniors.

1.1.6 Application Process

A call for applications is issued annually on a national basis for all regions using a standard application guide and form. Service Canada staff promote the call for applications using various media and direct methods and, on request, provide advice to applicants. Service Canada staff review and screen applications and refer those meeting the CA eligibility criteria to RRCs. RRCs assess applications using standard criteria addressing the benefits Formative Evaluation New Horizons for Seniors Program Capital Assistance Component 3 of the project to seniors, the organization and the community and either reject applications or recommend applications for Ministerial approval.

1.2 Capital Assistance Resources

The annual allocation of CA resources as of 2008-09 is presented in Table 1.1.

1.3 Evaluation Objectives, Issues and Questions

The goal of the formative evaluation of CA is to provide decision makers with evidence related mainly to design and delivery, performance measurement and monitoring systems, operational costs, relevance and early impacts of the CA component. At the time of the data collection for this formative evaluation, only one round of CA projects had been approved (2007-08). Therefore the evaluation report focuses on early success in achieving the intended CA outcomes.

The following evaluation issues ^Footnote1 and questions were addressed:

Relevance

• To what extent is the CA component relevant to the needs of organizations and seniors?

Design and Delivery

• Does the organizational structure support the achievement of the goals and objectives for the CA component?
• Are adequate management and administrative systems in place for efficient and effective delivery of this component?
• Are the objectives and expected outcomes of this new component clear and measurable?
  Does the logic model still accurately reflect the component's expected outcomes?
• Is this component being implemented as intended?
• How effective are communications regarding the CA component?
• Is the CA application process efficient?
• Is the CA application process effective?
• Could delivery of the CA component be changed in any way to improve efficiency and effectiveness?
• Are the types of projects funded under the CA component supporting the achievement of the NHSP intended program outcomes?

Operational Costs

• What is the percentage of operational costs (operation and maintenance) relative to the total annual budget of this component (at the national and regional level) and is this in line with the departmental standards for Grants and Contributions (GS & CS)? ^Footnote2

Performance Monitoring and Measurement

• Are the performance measurement indicators that have been set for the CA component adequate and appropriate?
• What is the quality of the available performance data for the CA component?
• Is the data needed for a summative evaluation being collected? Are there gaps?

Early Success

• To what extent has the CA component reached its expected immediate outcomes related to the participation of seniors and the contribution towards the community?
• Have there been any unintended impacts - positive or negative?

2.1 Overview of Methods

The formative evaluation was structured to collect information on each of the evaluation issues using multiple lines of evidence. Where possible, there was a balance between quantitative and qualitative methods, with qualitative methods providing further description and explanation for quantitative information. Both primary and secondary data sources were used.

The methods used included a document review, administrative data and file review, key informant interviews and a telephone survey of applicants. Appendix B sets out the evaluation matrix showing the evaluation issues and questions addressed by each method.

The evaluation focused on applications and activities involved in Call 1, since, at the time that data was collected, only the first call for CA applications had been implemented and completed.

2.2 Document Review

The document review was conducted to answer evaluation questions related mainly to design and delivery but also to performance measurement and monitoring, and operational costs. A preliminary review of documents also assisted the evaluation team in developing an understanding of the component and in designing the data collection instruments.

The evaluation team reviewed a wide range of documents that provided information for various evaluation questions. These documents related to CA component design and management and included the Treasury Board submission, Results-Based Management and Accountability Framework (RMAF) and Risk-Based Audit Framework (RBAF) for the New Horizons for Seniors Program, Resource Allocation Model, and Performance Measurement Framework. Tools, forms and letters used by regions to promote and deliver the CA component were also reviewed.

2.3 Administrative Data and File Review

The CA administrative data and project files contain various information on the applicants, types of projects and project results as well as on the administration of the CA component. This information was reviewed to address questions related mainly to performance measurement and monitoring but also questions related to design and delivery and early success. Three tasks were involved:

Administrative data review – Data from the Common System for Grants and Contributions (CSGC) database ^Footnote3 that was relevant to specific evaluation questions was extracted and analyzed using a Statistical Package for the Social Sciences (SPSS). Examples of the data extracted include status of the application, amount requested and amount funded, applications by region and organization type.

Project file review – The formative evaluation of CPL (conducted in 2008-09) found that the information in the project files against what was found in the CSGC were for the most part complete and accurate. As a result, a full and comprehensive file review of CA files was not necessary. Therefore, a sample of 80 CA project files was selected from the 1199 applications received in Call 1. This sample was stratified by region and status of application (approved, ineligible and rejected). A total of 67 project files from the sample were provided by regional offices: 42 accepted applications; 17 ineligible applications and 8 rejected applications (the remaining 13 were not submitted within the timeframe for this task). The files contained the funding application, forms related to the review process and the progress and final reports for approved projects. Data relevant to specific evaluation questions was extracted, recorded in a review template and analyzed using SPSS. Examples of the data extracted include dates of acknowledgement and decision letters, purpose of funding requested, immediate outcomes of funded projects. Qualitative information was coded prior to being entered in the template.

Data quality review – This review assessed the accuracy and quality of the data in the CSGC by mapping 12 of 53 fields in the database against information in the sample of project files. The intent of this element of the review was not to allow precise estimation of error rates and completeness (sample sizes were insufficient for this) but rather to identify any systemic problems or major inadequacies with the administrative data and files. Examples of data compared include number of participants, application received date, organization type.

Data from the paper project files was entered into a template pre-populated with administrative data from the CSGC. An analysis of this data was then conducted on each field of the database noting percentage of missing data, irrelevant differences ^Footnote4 and significant differences. ^Footnote5

2.4 Key Informant Interviews

2.4.2 Interview Process

The interview guide was sent to key informants in advance of the interview. It was recognized that this approach might present the risk of bias (respondents saying what they felt the interviewer wanted to hear). However, as the interview guides were lengthy (covering the three NHSP components) it was felt that the interviews would gather more informed opinions if key informants had the opportunity to review the questions in advance and that the benefit of this approach outweighed the risk.

2.4.3 Analysis

Interview notes were captured in an electronic database for analysis. The responses to questions were matched to specific evaluation questions and indicators and synthesized by respondent group. The relative weight of responses within each group was recorded using a rating scale (see section 2.6). The evidence was then analyzed and summarized for each evaluation question and indicator, then rolled up to analyze and summarize for each evaluation question, noting differences or similarities in the opinions across key informant groups.

2.5 Survey of Funding Applicants

2.5.2 Analysis

Much of the analysis compares the survey responses of funded applicants and unfunded projects that went ahead without CA funding. Responses of funded and unfunded applicants were compared for all questions. The unfunded applicants were further analyzed to compare the responses of ineligible and rejected applicants.

Statistical significance of observed differences was tested using the chi-squared goodness of fit test and test statistics are provided with the survey results in this report. This test examines whether the observed differences between the two populations are extreme enough to contradict the null hypothesis of statistical independence. All tests were conducted at the 5% level of significance which means that there is a 5% risk that a statistically significance difference will be found when, in fact, the two populations are statistically independent.

2.5.3 Profile of survey respondents

The funded and unfunded survey respondents were similar on the following demographics:

• Area served: The majority of funded respondents (68%) and unfunded respondents (64%) carried out activities only in one location, followed by some funded (28%) and unfunded (29%) respondents who carried out activities elsewhere in their province/territory. ^Footnote6
• Population served: The majority of funded respondents and unfunded respondents (68% and 69% respectively) served both seniors and non-seniors, followed by some funded and unfunded respondents (27% and 25% respectively) who served only seniors. ^Footnote7
• Staffing: Half of the funded and unfunded respondents had no paid staff, followed by some funded and unfunded respondents (26% and 28% respectively) who had 1 to 4 employees, and a few funded and unfunded respondents (23% and 22% respectively) who had more than five employees. ^Footnote8
• Volunteer base: The funded survey respondents had a slightly larger volunteer base than unfunded respondents. Some funded and unfunded respondents (37% and 29% respectively) had 50 or more regular volunteers; some funded and unfunded respondents had 20 to 49 regular volunteers (32% and 30% respectively); while some funded and unfunded respondents had less than 20 regular volunteers (29% and 38% respectively). ^Footnote9

2.6 Interpretation of Findings

Throughout the text, findings from qualitative and quantitative methods are presented using the following "scale" which corresponds to the proportion of key informant and survey respondents that held similar views.

• "All/almost all" – findings reflect the views and opinions of 90% or more of the respondents in the group.
• "Large majority/most" – findings reflect the views and opinions of at least 75% but less than 90% of respondents in the group.
• "Majority" - findings reflect the views and opinions of 51% but less than 75% of respondents in the group.
• "Half" – findings reflect the views and opinions of 50% of the respondents in the group.
• "Some" - findings reflect the views and opinions of at least 25% but less than 50% of the respondents in the group.
• "A few" - findings reflect the views and opinions of at least two respondents but less than 25% of respondents in the group.

2.7 Challenges and Limitations

The main challenges and limitations that the evaluation encountered related to the key informant interviews:

Conducting simultaneous formative and summative evaluations: Since the formative evaluation of the CA component was conducted concurrently with the formative evaluation of the EAA component and the summative evaluation of the CPL component, most key informants were asked questions about all three components. While this resulted in collecting opinions on a wide range of questions, it also meant that most interviews were lengthy and there was limited time to probe for more in-depth responses.

Balance of informants: Given that there are three NHSP program components and both headquarters and regional staff are involved, it was necessary to include a sufficient number of departmental key informants to provide coverage of the evaluation issues and questions. It was recognized that key informants from an organization that is responsible for a program may be reluctant to provide opinions that are critical of the program. The methodology controlled for this potential bias by including more external key informants (40) than departmental key informants (17).

Program awareness of respondents: As the CA component was in its first year at the time of the interviews, the awareness of the component and the project results varied. The result is that some evaluation questions and indicators were not answered by all respondents. Respondents were asked to self-rate their awareness of the CA component and projects, and this was taken into consideration in the analysis and reporting.

Challenges were also experienced with quality and availability of administrative data, which also had an impact on the survey of applicants:

The main challenge encountered was determining which projects were rejected and ineligible from the project status field in the CSGC database. NHSP personnel provided an algorithm for combining several CSGC fields by which a derived variable could be created which – assuming that all status fields were up-to-date in entering status information – determined whether an application was rejected or ineligible. This derived variable was used for survey stratification with both populations represented proportionally. However, the derived variable produced results which were inconsistent with program statistics based on reports provided directly by the regions. Consultation with program officials determined that the program statistics were correct. Consequently, reporting of responses from rejected and ineligible applicants should be viewed with caution as some of the unfunded applicants have been misclassified. Data comparing funded and unfunded applicants are not affected by this difficulty.

4.1 Design

Program documentation indicates that funding the CA component through grants was considered the most efficient method due to the large volume of modest proposals that carry low financial risk. The program documents reviewed identify the objective of the CA component as being "to help non-profit organizations to maintain their capital infrastructure to support existing community programs and activities that promote active living and social inclusion for seniors". This objective is clear and measureable. On the other hand, there are no intended program outcomes specifically for the CA component epicted in the NHSP logic model (Appendix A). Stated intended outcomes of the logic model are generic to the NHSP program as a whole.

All NHSP NHQ key informants felt that the generic intended outcomes for the NHSP as a whole were sufficiently clear regarding the CA component. ^Footnote11 Reasons given were that, even though this is a generic model for the program overall, the objective of the CA component is quite discrete and measurable. It was noted that there is a broader effort planned to develop outcome models and performance indicators for HRSDC social programs in order to more accurately measure the complex impacts of these programs. The NHSP model will likely be revised as part of this broader effort.

4.2 Delivery Structure and Systems

The document review and NHSP NHQ and Service Canada key informant interviews indicate that the organizational structure supports the achievement of CA objectives, with clearly defined roles for national headquarters (NHQ), regions, and RRCs for all phases of program delivery. The majority of NHSP NHQ and Service Canada key informants felt that the management and administrative systems in place are, for the most part, adequate to support efficient and effective delivery. The majority of Service Canada regional staff felt NHSP NHQ is providing appropriate guidance. The large majority of RRC key informants felt that Service Canada regional staff are providing sufficient support for their work, although a few noted that that the shortage or unavailability of regional staff is a limitation. The large majority of NHSP NHQ, Service Canada and RRC key informants felt that CA is being implemented as planned and there was no evidence in documents to indicate otherwise. ^Footnote12

NHSP NHQ, Service Canada and RRC key informants identified two aspects of the management and administrative system that impact negatively on CA delivery: the CSGC database is considered by the majority of NHSP NHQ and Service Canada key informants as inadequate for effective monitoring and reporting on component outcomes; and theDepartmental approval process for projects which is seen by some key informants as creating significant delays in project approvals.

Internal process

HRSDC implemented a number of steps in the application and project administration process that are supported by and documented using various forms. These steps include an initial screening and internal assessment by Service Canada staff of applications against CA funding criteria, the assessment by individual RRC members of each application against established criteria, a recommendation report reflecting the consensus of the RRC, and, in the case of approved projects, a progress report and final report. The project file review found that the required documentation was included on a majority of files, but progress reports and final reports were absent from 26% and 19% respectively of the files reviewed. This indicates that the monitoring and final reporting may not have been completed in these cases.

4.3 Communications

The various lines of evidence indicate that communications are largely effective but also point to potential areas for improvement. The large majority of NHSP NHQ and Service Canada key informants and the document review (2008-09 Regional Promotional Plans) indicate that the CA component is communicated by regions to potential applicants using a variety of methods that are also used for CPL (Table 4.1), ranging from distributing materials by mail or e-mail (the most common approach) to outreach activities (the least common approach).

• Source from the above table Source: Survey of funded applicants

The funding applicants surveyed most frequently learned about the CA component from material received by mail, or from another organization (Table 4.2).

• Source from the above table Source: Survey of funded applicants; Survey of unfunded applicants

Survey respondents were also asked the best way to communicate information on the CA component to them. They most frequently reported (Table 4.3) that mail and email are the best ways to reach them. Of note is that no respondents indicated they had actually learned about the CA component through email (see Table 4.2) which may indicate that this method could be utilized more by Service Canada in communicating with organizations. Also noteworthy is that one-third of survey respondents had not accessed the NHSP website, which reinforces the importance of NHSP continuing direct contact with organizations.

• Source from the above table Source: Survey of funded applicants; Survey of unfunded applicants

Survey respondents were asked to rate the information provided on the CA. While a majority of both funded and unfunded applicants who responded to the survey rated the availability of information on the CA component and program description in materials and on the website as good or very good, funded applicants gave higher ratings to these aspects of communications. For example, the large majority of funded applicants surveyed (87%) considered the availability of information on the CA component as good or very good, compared to the majority (57%) of unfunded applicants. ^Footnote13

A few NHQ key informants observed that communication success varies depending on the regional emphasis placed on promotion of CA. Some RRC members and stakeholders felt that there is a need for more outreach and in-person contacts with seniors' organizations in order to inform them of the CA component as well as to help them with their applications. Smaller organizations and those in rural areas were seen as having a need for CA funding but having limited capacity to develop proposals for renovations. Very few survey respondents (4% of funded and 10% of unfunded) felt in-person meetings were the best way to inform them about the program, but this does not necessarily reflect their level of interest in meetings to help them prepare applications.

4.4 Application Process

Almost all funded applicants (96%) surveyed were either satisfied or very satisfied with the application process overall, while less than half (49%) of the unfunded applicants gave this rating. ^Footnote14 Among unfunded applicants surveyed, ineligible applicants (15%) were more likely than rejected applicants (4%) to rate overall their satisfaction with the application process as very poor. ^Footnote15 No significant differences were found in the ratings of these two groups of the individual aspects of the application process, including the clarity of CA funding eligibility criteria.

The survey asked applicants about the timeframes for processing applications, and various aspects of the information and assistance provided as part of the application process. Funded applicants rated all these aspects higher than did unfunded applicants. Figure 4.1 below presents the aspects of the application process covered and the survey responses.

Time to receive acknowledgement of application: Most (80%) of funded and half (50%) of unfunded respondents rated the time it took to receive an acknowledgement of their application as good or very good. ^Footnote16

Time to receive a decision on funding: Various lines of evidence provide consistent ratings on the lack of timeliness of decisions on funding. The majority (68%) of funded respondents and some (27%) of the unfunded respondents rated the timeframe to receive a decision on their application as good or very good. ^Footnote17 Key informants were also asked about this. The majority of NHQ NHSP key informants felt the turnaround time is good while the majority of Service Canada informants and some RRC members felt that the turnaround time for decisions is too long. The project file review indicates that decision letters on CA pplications were issued, on average, 90 days after the receipt of the applications, which is well below the departmental standard of five months for communicating decisions on applications. However, this calculation is based on a small sample of project files (53) that contained the information needed to determine turnaround times.

Clarity of criteria: A large majority of funded respondents (83%) rated the clarity of the CA eligibility criteria as good or very good, while some (47%) of the unfunded respondents gave this rating. ^Footnote18 , ^Footnote19 Usefulness of the application guide: A large majority (83%) of funded respondents and a majority (67%) of unfunded respondents rated the usefulness of the application guide as good or very good. ^Footnote20

Assistance provided by Service Canada with application: A majority of funded respondents (68%) rated the assistance provided by Service Canada with their application as good or very good (note that 27% had not requested assistance). Some unfunded respondents (41%) gave this rating (note that 40% had not received assistance). ^Footnote21 There were no substantive regional differences in the ratings of respondents for this question. Responses provided by Service Canada to general enquiries: Most (85%) of the funded respondents and some (48%) of the unfunded respondents rated the responses to their general enquiries as good or very good. ^Footnote22 There were no substantive regional differences in the ratings of respondents for this question.

Response times by Service Canada to general enquiries: Most (81%) of the funded respondents and some (48%) of the unfunded respondents rated the response times as good or very good. ^Footnote23 There were no substantive regional differences in the ratings of respondents for this question.

Fairness of the application process: Most (78%) of the funded respondents and some (38%) of the unfunded respondents rated the fairness of the application process as good or very good. ^Footnote24 NHSP NHQ, Service Canada and RRC key informants were also asked their opinions on the fairness and standardization of the application process. The majority in all three groups felt that the application process for the CA component is standardized across the country and fair. ^Footnote25

A few in each key informant group identified aspects of the process that might negatively impact on fairness, including the extent of regions' community engagement efforts to promote the quality of applications, differences in the methods of dealing with incomplete applications, and a perception that organizations with more resources and in urban centres might be at an advantage to apply for CA funding. With regards to this latter point,applicants are required to get three estimates for renovation projects, and this was seen by some NHSP NHQ and Service Canada key informants as problematic for organizations in smaller communities who have few suppliers. This was also cited as a constraint by a few survey respondents. The CA application form indicates that "if three estimates are not available, please tell us why." It appears from the key informants and the survey of respondents that this potential exemption is either not widely known or not universally applied. The wording of this exemption should clarify the situations where less than three estimates could be provided, such as in rural areas or small communities.

A few survey respondents also felt that the criterion which requires organizations to own the building in order to qualify for building improvement funding was unfair. A review of CA documents indicates this is an inaccurate interpretation of the criteria. The CA documents state that ineligible projects include renovations and/or repairs to buildings owned, operated and maintained by municipal, provincial or territorial governments. Further the CA application form states that applicants must provide proof of building ownership OR a lease agreement with a letter from the property owner confirming they agree with the proposed renovation or repair. This evidence suggests a need to ensure these criteria are clearly communicated to organizations.

Amount of information required in the application: The majority (71%) of funded respondents and some (40%) of the unfunded respondents rated the amount of information required in the application as good or very good. ^Footnote26

Time it took to receive funding: A large majority of funded applicants surveyed (79%) rated the timeframe for receipt of funding as good or very good. Based on information in the funded project files reviewed, almost all cheques were requisitioned at the same time as the decision letter was issued.

Clarity of the reason why the project was not funded: Some (34%) of the unfunded respondents rated the clarity of the reason for the decision on their application as good or very good. The document review indicates that form letters used to communicate with applicants are clear and easy to understand, but precise reasons for rejection of eligible applications are not provided in the decision letter. However, the letter provides a name and telephone number for a Service Canada representative who may be contacted if the applicant wants to discuss the assessment or future proposals. Based on the survey responses, at least some unfunded applicants (32%) seek this additional help. Providing more specific reasons in the letter might address some of the concerns of those who are less satisfied with communications.

Additional help provided by Service Canada to understand why the project was not funded: Some (28%) of the unfunded applicants rated the additional help provided as good or very good (note that 40% of the total had not received additional help). Reporting requirements: Funded applicants were also asked to rate the requirements in the final report they must produce. Almost all (94%) rated these as good or very good.

Reporting requirements: Funded applicants were also asked to rate the requirements in the final report they must produce. Almost all (94%) rated these as good or very good.

Applicant suggestions for improvements

The most frequent suggestions made by funded applicants to improve the application process was to simplify the language in the application form to make it more understandable for seniors and to reduce the repetition in the information requested (17% of respondents) and to have shorter turnaround time for approvals and distribution of funds (10%). Unfunded applicants most frequently suggested simplifying the process (no specifics given) and reducing repetition of questions in the application (19%), clearer and more information on the eligibility criteria (19%) and improved communication and feedback (15%). The majority of Service Canada key informants, some RRC key informants and half the stakeholder key informants ^Footnote27 stated that they were aware of similar concerns as those expressed by the survey respondents with the application process. The application was examined using the Flesch Reading Ease test and was rated as having a less than optimal reading level. ^Footnote28 The form was also reviewed for duplication. While none was found, the form includes numerous 'helpful hints' that lengthen the form and could overwhelm the user.

The majority of NHSP NHQ, Service Canada and RRC key informants felt that the application process is effective and generates the information needed to make funding decisions. ^Footnote29 There were mixed views on whether the RRCs have the necessary expertise (or access to those with the expertise) to make recommendations on construction related projects. The majority of Service Canada and RRC key informants felt that RRCs have at least some members with construction expertise or have access to this expertise to advise on their decisions. However, some NHSP NHQ key informants felt this was not the case and some RRC key informants felt their respective committees lack this expertise and need more time to consult with experts. Following the evaluation data collection period, the evaluation team was informed by NHSP NHQ officials that it is not the role of Regional Review Committees to assess the viability of capital projects. Rather, their role is to assess whether projects meet regional priorities and to make recommendations on funding accordingly. It is the role of Regional Service Canada staff who screen applications for eligibility to determine the feasibility of capital projects.

4.5 Profile of Projects

Capital Assistance criteria state that non-profit groups delivering community-based programs for seniors qualify for funding for improvements to facilities, equipment and furniture. The evidence from the administrative data and survey of applicants indicates that approved CA projects met these criteria. Based on the CSGC, all funded organizations were non-profit and the majority surveyed (68%) operated in only one location. The majority surveyed (70%) served both seniors and non-seniors, indicating the potential for a broader community impact from the improvements made to facilities. The majority of NHSP NHQ and Service Canada key informants and some RRC key informants felt that the CA component is attracting the kinds of projects envisioned, with some commenting positively on the quality and large number of applications in Call 1. ^Footnote30

The large majority of funded applicants surveyed (83%) did not target a specific group of seniors. Immigrant seniors and those in specific cultural groups were identified as the target group of 12% of funded respondents, while 2% identified Aboriginal seniors as their target group. However, the evidence from the project file review (68 files) portrays a profile of organizations serving a wider, more diverse base. Sixty-two percent of funded organizations served seniors in general; 10% served Aboriginal seniors, 9% served seniors with disabilities, 6% served immigrants and 6% served seniors in Official Language Minority Communities.

Appropriateness of performance indicators

The NHSP Performance Measurement Framework sets out 18 performance indicators for the CA component (see Table 6.1). Some of these are used multiple times to measure 48 outputs of the component (e.g. # of CA projects funded is used as an indicator for the outputs of funded projects, community priorities addressed, organizational capacity, community capacity to respond to existing or emerging challenges). None of these 18 indicators measure the intended outcome of maintaining programs and activities for seniors and are thus of limited usefulness in assessing CA results on an ongoing basis and through a summative evaluation.

Table 6.1 CA Performance Indicators

CA Performance Indicators

# of applications received that reflect geographic areas

% of applications deemed "eligible"

# of application deemed "eligible"

Total % of eligible projects approved (National and Regional)

# of capital assistance projects funded

Total # of eligible applications approved (National and Regional)

# of capital assistance applications received

Total $'s of funding (cash and in-kind) levered from partnering organizations involved with funded projects – not including applicant organizations

#/% of funded projects where the recipient organization had any funding partners (cash or in-kind funding) – not including applicant organizations

Total $'s from funding partners – categorized as one of several organization "types"

# of capital assistance projects funded for improvements to facilities

# of beneficiaries (seniors) benefiting from capital assistance projects

# of capital assistance projects for both improvements to facilities and furniture and equipment

# of capital assistance projects for furniture and equipment

Total # of communities receiving funding for projects

# of new organizations receiving funding for projects (i.e. organizations that did not have funding in the previous calls)

# of final reports filed

# of Official Language Minority Community [OLMC] applications

• Source from the above table Source: NHSP Program documents

Accessibility of performance information

Sixteen of the 18 CA performance indicators are captured in the CSGC. Two CA performance indicators (#/% of final reports filed, # of Official Language Minority Community [OLMC] applications) were not found in the CSGC database but can be determined from the paper project files. This presents accessibility and potential accuracy issues for performance measurement and reporting as the files must be reviewed manually and the narrative data coded.

Project final report form

The CA project final report form will be an important source of information for a summative evaluation of the outcomes of this component. The performance information needed for a summative evaluation of the CA outcome is not captured appropriately on this form. The final report form asks which programs and activities were positively impacted by the improvements made but the wording is confusing. There are also several openended questions on impacts on seniors and the community. In the review of project files, it was frequently necessary to make judgments about how the information on results provided by organizations related to the CA intended outcome.

Number of seniors

The number of seniors benefiting from the CA projects is also not captured accurately on the project final report form because of how this question is worded. The form asks for the number of seniors who participate in the organization's programs and activities and not those who participated in programs and activities that were maintained as a result of the CA funding. While in the majority of cases it could be assumed that these would be the same, in the case of projects that provide equipment for specific activities (such as computers) one cannot assume that all seniors served by the organization will benefit from using the new equipment. As a result, the number of seniors benefitting from CA funding is over-estimated.

Common System for Grants and Contributions database

The CSGC does capture narrative information from the project final reports on results, but only 200 characters of this information can be downloaded for analysis. This information is not coded which further limits its accessibility. NHSP initiated a process to code information from the CA and CPL project final reports based on intended program outcomes. The process requires coding of several of open-ended questions none of which very clearly ask about the intended CA outcome. This could be a labour intensive process resulting in poor quality data.

Overall the evidence indicates the need to first revise the project final report to facilitate accurate data capture on the CA component outcome, preferably through several closedended questions complemented by narrative information. This would then at a minimum ensure the efficiency of the manual coding process of this information.

Quality of performance information

The review found that there are some issues with data quality in the CSGC database.

The information in the CSGC database on project results was found to be incomplete. Fifty-two percent of funded projects were shown as In-Process (i.e. not closed out). All CA Call 1 projects were beyond the approved funding period at the time of the administrative data review and should have been closed out. This indicates that the database has not been updated to record project results that relate to output indicators (e.g. number of seniors benefiting from the project) for a majority of projects. An NHSP NHQ key informant confirmed that there are some inconsistencies across regions in the timeliness and thoroughness of this step in NHSP administration.

In the data quality review of 68 CA project files, 64.6% of the data in CSGC accurately matched. The two data elements recorded most accurately in CSGC were the amount of funding approved by HRSDC and the type of organization applying for funding. Significant differences between the CSGC and the project files reviewed were found in 23.4% of the data elements. The number of seniors and other participants were the two data elements most frequently recorded inaccurately (63% and 85% respectively had large differences).

Few NHSP NHQ and Service Canada key informants indicated they have used data on the CA component for decision making. Some felt more precise information on some data elements would improve its usefulness. Examples cited include more defined categories of organizations, types of renovations and equipment improvements, and impacts of the project on revenue generation of the organization. It was noted that work is underway to manually code the project forms to capture some of this information. The project application and final report forms should be revised to capture this kind of information for both program management and the summative evaluation.

The majority of RRC key informants indicated they are getting statistical information on CA and find this useful in doing their work.

Overall, the evidence indicates that some of the key information on project outcomes required for ongoing management and a summative evaluation of the CA component is not captured, some key data is not captured accurately, and some is captured but not readily accessible for analysis. The gaps could be filled by:

• developing a performance indicator for the CA outcome of maintaining programs and activities for seniors;
• revising the CA application and project report forms to more accurately capture data related to this CA outcome and more detailed information on various elements of the component described above; and
• ensuring the data on project application and final report forms is captured completely and accurately in an electronic format. If these gaps in data capture are not filled, it will be necessary for a summative evaluation to rely on a survey of applicants to capture this information.

If these gaps in data capture are not filled, it will be necessary for a summative evaluation to rely on a survey of applicants to capture this information.