Formative Evaluation of the Charities Partnership and Outreach Program

Corporate Audit and Evaluation Branch

April 2008

Table of Contents

• Executive Summary
• Introduction
  • Background
  • Program description
  • Evaluation context
  • Methodology
• Findings
  • Program is relevant within context of Charities Regulatory Reform Initiative
  • Progress on funding projects
  • Original program design has not been fully realized
  • Limited progress in measuring program performance
  • Practices to consider for program improvements
• Conclusion and Recommendations
• Appendix A
• Appendix B

Executive Summary

The Corporate Audit and Evaluation Branch conducted a formative evaluation of the Charities Partnership and Outreach Program (CPOP) to provide information and advice to senior management of the Canada Revenue Agency (CRA) on whether the program is well positioned to meet its objectives and identify areas for improvement. As outlined in the Terms of Reference approved by the Agency Management Committee (AMC) in February 2007, the evaluation focused on program relevance within the context of the CRA and the federal government, program design, implementation and delivery, and program modifications. The scope of the evaluation included fiscal years 2004‑2005 to the second quarter of 2007-2008.

The rationale for the program can be traced back to the Voluntary Sector Initiative, a joint undertaking launched in June 2000 by the Government of Canada and the voluntary sector to enhance their relationship and strengthen the capacity of the sector to better serve Canadians. In May 2003, the federal government released its report, Strengthening Canada’s Charitable Sector: Regulatory Reform, which included a total of 75 recommendations for improvement, of which several focused on the need for the federal regulator to be more proactive in providing information and education to the sector.

In March 2005, Treasury Board (TB) approved the terms and conditions and released funding of $1 million in 2005‑2006, $1 million in 2006‑2007, $2 million in 2007-2008 and $3 million in subsequent years for CPOP. The program was to be phased-in through an incremental approach as was detailed in the Results-based Management and Accountability Framework (RMAF) included in the TB Submission. The Policy, Planning and Legislation Division, Charities Directorate, was originally responsible for program. On April 1, 2006, this responsibility was transferred to the Directorate’s newly created Client Interface and Service Division (CISD).

We understand that the Charities Directorate established this program as part of its ongoing effort to inform and educate registered charities of their obligations under the Income Tax Act. The program is relevant to the CRA and to the federal government because CRA has a responsibility to maintain public confidence in the fairness and integrity of the tax system. This means providing ample support to those who wish to comply with the law in addition to taking the appropriate measures to identify and deal with cases of non-compliance. Registered charities that do not comply with the law are compromising the work of legitimate organizations by reducing public confidence and diverting funds that would otherwise be spent to benefit Canadians.

In pursuing the program design issue, the evaluation found little analysis or documentation that led to the choice of policy instrument, i.e., a contribution program, as the most effective approach to implement the program. We expected that CRA’s decision to select this instrument to carry out the program’s objectives would be based on comprehensive analysis of existing outreach activities and the alternatives available to improve the sector outreach and consultation activities. We did not find evidence that such analysis had been undertaken. A more thorough assessment of program results, through a subsequent summative evaluation, is required to determine the impact of this program on compliance, and if the policy instrument was the most appropriate choice for implementing the program.

Although the RMAF recognized that the CRA is not normally in the business of administering and managing contribution programs, it estimated that the Charities Directorate would be managing 5 to 10 agreements per year. At the time of the evaluation, the program was in its fourth year of operation and the third year that contribution funding has been available. Two calls for applications had been held and, with a new contribution agreement being entered in September 2007, three contribution agreements were in place. The first two contribution recipients had only recently begun delivering training sessions to the voluntary sector in the Fall of 2007. With an additional 3 agreements being entered into in January 2008 a total of six agreements are now in place; however, this would only be for a limited time period.

The evaluation found that the program was not fully implemented as detailed in the original design approved by TB. While the activity of putting in place the contribution agreements has clearly been the focus of CPOP, this is to the exclusion of other key activities including promoting the program, collaborating with the voluntary sector and measuring performance. A performance measuring strategy, as required by the TB Policy on Transfer Payments, has not been implemented. Performance measurement data is required to support the summative evaluation committed by CRA in advance of the renewal of the terms and conditions for this program in March 2010. If performance data is not available, the value of conducting a summative evaluation is questionable.
 
We believe that the program area will continue to make program improvements based the knowledge and experience gained, including the examination of best practices, as the program progresses. In our evaluation report, we identified some practices for the program to consider and put forward 4 recommendations to be addressed by program management. The recommendations, in brief, are as follows:

• revisit the original program design to reassess the feasibility of implementing the program in terms of what was planned as detailed in the TB Submission that was used to obtain funding for this program;
• focus on developing and implementing a performance measurement strategy that will identify the baseline data that is required to measure performance and report on the success of this program;
• develop and implement a communication strategy that clearly identifies the steps that will be taken, both in terms of collaboration and promotion, to help achieve the program objectives; and
• implement the policies, procedures and information system(s) necessary to ensure that quality information is available to support decision-making, the measurement of performance as well as accountability and transparency of the program.

Introduction

Background

1.1   The Canada Revenue Agency (CRA) regulates registered charities under the Income Tax Act (ITA). There are approximately 83,000 registered charities in Canada that are required to file an annual return with the CRA and meet certain requirements of the ITA concerning their expenditures and activities. The ITA exempts registered charities from paying income tax and allows for the issuance of tax receipts for donations that, in turn, can be claimed as tax credits by the donor to obtain partial tax relief. In addition, under the ITA, the CRA has responsibility for determining which organizations are eligible to be registered as charities and ensuring that registered charities comply with established rules and regulations.

1.2   The Charities Directorate is positioned within the Legislative Policy and Regulatory Affairs Branch (LPRAB) of the CRA. Its reported mission is “to promote compliance with the income tax legislation and regulations relating to charities through education, quality service, and responsible enforcement, thereby contributing to the integrity of the charitable sector and the social well being of Canadians.” 

1.3   Over the last few years, the Charities Directorate has been concentrating on developing and implementing a comprehensive reform package designed to strengthen the regulatory environment for charities. Following the Directorate’s “education-first” approach to compliance, the Charities Partnership and Outreach Program (CPOP) was initiated in 2004.

Program description

1.4   The overall objective of CPOP is to ensure that the charitable sector is better educated concerning its regulatory obligations under the ITA, and therefore, better able to comply voluntarily. The main objectives of the program are to:

• raise awareness among the charitable sector of regulatory obligations under the ITA;
• increase the capacity of the charitable sector in meeting regulatory compliance requirements in a sustainable fashion; and
• improve the capacity of the sector to develop and deliver sustainable compliance-based education programs.

1.5   Within the broader context of regulatory reform, the Charities Directorate chose to pursue CPOP’s objectives by way of establishing a contribution program. Under the Treasury Board (TB) Policy on Transfer Payments, transfer payments in the form of grants and contributions can be used to assist the government in meeting its public policy objectives.

1.6   In March 2005, TB approved incremental funding of $12 million per year, beginning in 2004-2005, for the Charities Directorate to implement the Charities Regulatory Reform Initiative, which is aimed at improving the regulatory framework for charities while increasing public and voluntary sector confidence in the regulation of charities. Of this funding, TB specifically approved $1 million in 2005‑2006, $1 million in 2006‑2007, $2 million in 2007-2008 and $3 million in subsequent years for the CPOP. As stated in the terms and conditions included in the TB Submission, the costs of managing and administering the program are funded from existing reference levels.

1.7   As a contribution program, CPOP is administered by the CRA and provides funding, in the form of contribution payments, to eligible registered charities and non-profit organizations serving the charitable sector. Contribution recipients are then responsible for developing and delivering training and education to registered charities on how to comply with the ITA. As indicated in the program objectives above, the program intends to increase the capacity of the sector to meet regulatory compliance requirements in a sustainable fashion so that gains made during the funding stage will be maintained after the investment by government.
 
1.8   Originally, the Policy, Planning and Legislation Division (PPLD) of the Charities Directorate was responsible for the design and early implementation of the program. On April 1, 2006, responsibility for the ongoing management and administration of this program was transferred to the Directorate’s newly created Client Interface and Service Division (CISD).

1.9   Since the start-up of the program, there have been a number of full-time equivalents (FTE) at different levels assigned in various capacities to manage and administer the program. Currently, 2.2 FTEs, consisting of 2 FTEs at the program officer level (PM-05) and .2 of a FTE at the manager level (MG-05) are allocated to the program. Program costs for 2007-2008 are projected to be $170,437 in salaries and $11,998 in other operating and maintenance costs. The evaluation was not able to obtain historical data on program costs. 

Evaluation context

1.10   In the terms and conditions, the Charities Directorate committed to undertake a formative evaluation in April 2006. Formative evaluations are normally conducted a few years into the program and focus on the extent to which the program has been implemented as planned. In April 2006, although the program had been operational for two years, the contribution funding was phased-in and did not begin until the second year. Upon consultation with the TB, the evaluation was delayed until January 2007 to allow for additional progress to be made on the implementation of contribution agreements.

1.11   The purpose of the evaluation was to provide information and advice to the Commissioner and Agency Management Committee (AMC) on whether the program is well positioned and on track to meet its objectives as well as identify areas for improvement. As approved in the Terms of Reference (TOR) that were presented to AMC in February 2007, the evaluation focused on program relevance within the context of the CRA and the federal government, program design, implementation
and delivery, and program modifications. The scope of our work included fiscal years 2004‑2005 to the second quarter of 2007-2008.
 
1.12   The CRA is also committed to conducting a summative evaluation, focused on the achievement of results, and reporting on the findings in advance of the renewal of the terms and conditions in March 2010.

1.13   Further details on this evaluation and the evaluation requirements of this program can be found in Appendix A: About the Evaluation and Appendix B: Glossary of Terms.

Methodology

1.14   Multiple lines of evidence were used to address evaluation issues. Evaluators conducted a review of files and documentation associated with the administration and management of the program with particular emphasis on the application and review process. A literature review of publications and other documentation related to registered charities, non‑profit organizations and the voluntary sector as well as grant and contribution (G&C) programs was also conducted. The first two contribution agreements were used as case studies to gain additional insight into the funding process.

1.15   The evaluation relied extensively on interviews with applicants, agreement holders, committee members and representatives of other federal government departments. The breakdown is provided in Exhibit 1 below.

Exhibit 1:  Breakdown of Interviews

1.16   Based on a review of applications received, the evaluation team was able to identify a cross‑section of interviewees from across major Canadian city centres. In addition to their location, applicants were also selected based on how their application was assessed. Within the CRA, interviews were conducted with representatives of LPRAB and the Finance and Administration (F&A) Branch. In total, we conducted 47 interviews with 58 interviewees.

Findings

Program is relevant within context of Charities Regulatory Reform Initiative

2.1   Government-wide initiative supported regulatory reform of charities .   As identified in the TOR, the evaluation intended to assess the extent to which CPOP is relevant to the CRA and the federal government. Consequently, the rationale for this program was examined and traced back to the Voluntary Sector Initiative (VSI). The Government of Canada and the voluntary sector launched the VSI in June 2000 to enhance their relationship and strengthen the capacity of the sector to better serve Canadians. The VSI resulted in the establishment of the Joint Regulatory Table (JRT) in November 2000, a working group that was co‑chaired by the Director General, Charities Directorate, and a representative of the voluntary sector.

2.2  The JRT was asked to study and make recommendations to improve the legislative and regulatory environment in which the voluntary sector operates. In August 2002, JRT released an interim report and then held public consultations in 21 cities across the country. People from a wide range of voluntary sector organizations, their advisors, provincial government departments and the general public offered their views on the interim report and its recommendations. In total, 524 representatives from 388 organizations participated in the consultations.

2.3   In May 2003, the JRT released its report, Strengthening Canada’s Charitable Sector: Regulatory Reform, which included a total of 75 recommendations for improvement. Due to the number of comments received during the consultation on the issue of education, several of the recommendations focused on the need for the federal regulator to be more proactive in providing information and education to the sector.

2.4   CRA identified strategic measures for enhanced compliance-based outreach and education.  As described above, in 2003 the federal government recognized and recommended that CRA take measures for improving educational activities directed at registered charities. Following this, CRA’s Charities Directorate developed a 2003-2004 Business Case on “Public Awareness and Sector Outreach (Partnership Activities)”, as one of several business cases related to regulatory reform.

2.5   The issue put forward in the business case was the determination and implementation of the most effective means available to improve the voluntary sector outreach and consultation activities of the CRA. The business case reported that efforts to date to educate the sector about its compliance responsibilities had achieved only a degree of success, and innovative ways of reaching out to the sector had not been sought out. Furthermore, the case stated that efforts had not been made to consult extensively with the sector to seek input as to the best means of increasing compliance through education and outreach.

2.6   The document made references to the “widely held views that the current sector outreach and consultation activities, although useful, are insufficient”. The business case did not include, or make reference to, analysis that supported these statements. In addition, the business case stated that improved training and education would assist registered charities in meeting their filing requirements, improve the quality of the information received, and for CRA, reduce the current workload associated with these issues. A new strategy was proposed for education and outreach that would be based on enhanced communications and partnerships with the sector, and establishing a G&C program (the contribution program that became CPOP).

2.7   Sector outreach identified as a key area for regulatory reform.  In April 2004, the Charities Directorate announced a five-year action plan to initiate the Charities Regulatory Reform Initiative in response to the JRT recommendations. The action plan identified public awareness and sector outreach as a key area for regulatory reform. As documented in a media fact sheet issued in April 2004, improved sector outreach was important because registered charities had asked for CRA’s assistance in understanding the rules and obligations associated with being a registered charity. The same fact sheet identified the implementation of a strategic funding program that would provide funds to the voluntary sector to deliver training and education to registered charities as well as supplement other outreach activities as part of an expanded sector outreach program.

2.8   In addition to the findings of the JRT, the Charities Directorate points to selected CRA data, such as annual return, revocation and audit data, as evidence of the need for enhanced outreach activities. For example, the Directorate cites data on the number of registered charities that have their status revoked every year, and suggests that there is a need for outreach. Over the past five fiscal years, 10,336 charities had their registered status revoked, either voluntarily by the charity itself, or by the CRA as a result of the registered charity not filing its annual return, or in other serious instances of non-compliance with the law, for what is termed “for cause”.

2.9    As demonstrated in Exhibit 2 below, the majority of charities have their registered status revoked for failure to file their annual return. An average of 1,181 charities or 1.45% of registered charities lost their registered status over the past five fiscal years. In addition, the 2003-2004 business case also stated that the majority of charities, whose registered status is revoked due to failure to file, reapply to the CRA for registered status.

2.10   The Charities Directorate also points to audit data to support the ongoing need for outreach. As detailed in the Registered Charities Newsletter published in August 2007, of the audits completed in 2006-2007:

• 52% resulted in education letters to the charity to explain the rules and suggest appropriate corrective actions;
• 22% required no changes;
• 20% resulted in compliance agreements, which are negotiated between the CRA and the charity, to address issues of non-compliance; 
• 3% resulted in revocation; and
• 3% were comprised of annulments, pre-registration audits, and voluntary revocations.

Exhibit 2:  Revocation of Registered Charity Status

  2.11   Limited analysis and documentation to support choice of policy instrument.  The decision to implement a contribution program was likely based on the JRT’s recommendation that the CRA find new, innovative ways of delivering education to charities by building partnerships with the sector. However, the evaluation team did not find any evidence that additional research and analysis had been undertaken in relation to the implementation of the contribution program. There was no evidence that existing outreach activities as well as different alternatives and the risks, costs and benefits of each were assessed and compared. The 2003-2004 business case stated that CRA’s existing approach to outreach was ineffective but did not include, or make reference to, research and analysis identifying the reasons why and the recommendations for improvement.

2.12   Furthermore, the TB Policy on Transfer Payments imposes certain requirements as part of the approval process for the terms and conditions of contribution programs. Particular emphasis is placed on the need for the program to measure, report and be accountable for program results. We found that, although a Results-based Management and Accountability Framework (RMAF) was prepared as part of the TB Submission, a performance measurement strategy was not implemented and this has impacted on the program’s ability to report on results. Performance measurement is discussed in greater detail later in this report.

2.13   It should be noted that current program management approached Program Evaluation with concerns regarding the feasibility of implementing the performance measurement strategy and evaluation plan that was included in the RMAF prior to the commencement of this evaluation. Nonetheless, the evaluation team was still required to use the TB Submission, including the RMAF, to assess the extent to which the program had been implemented as planned. 

2.14   This program is consistent with the Charities Directorate on-going effort to inform and educate registered charities of their obligations under the law. The need to educate and inform remains relevant because CRA has a responsibility to maintain public confidence in the fairness and integrity of the tax system. This means providing ample support to those who wish to comply with the law in addition to taking the appropriate measures to identify and deal with cases of non-compliance. Registered charities that do not comply with the law are compromising the work of legitimate organizations by reducing public confidence and diverting funds that would otherwise be spent to benefit Canadians.

2.15   Although the program is considered relevant in terms of the formative evaluation, a more thorough assessment of program results, which is not planned until the summative evaluation, is required to determine the impact of this program on compliance and if the policy instrument was the most appropriate choice for implementing the program.

Progress on funding projects

2.16   As a contribution program, CPOP is administered by the CRA and provides funding, in the form of contribution payments, to eligible registered charities and non‑profit organizations serving the charitable sector. Contribution recipients are then responsible for developing and delivering projects that will increase compliance and build a responsive charitable sector.

2.17   Three contribution agreements are in place.  The program is now in its fourth year of operation and the third year that contribution funding has been available. At the time of the evaluation, two calls for applications had been held and, with a new contribution agreement being entered in September 2007, three contribution agreements were in place. The funding priority for the first call, which was focused on the Registered Charity Information Return (T3010A), resulted in two agreements:  a 32‑month regional agreement in the amount of $237,903 and a 32‑month national agreement in the amount of $1,353,400.

2.18   The second call had two different funding priorities:  (1) fundraising, receipting and maintaining books and records; and (2) foreign activities. At the time of the evaluation, the second call had resulted in a 30‑month national agreement in the amount of $693,830 that was signed on September 1, 2007. Due to the timing of the third agreement, it was not examined in any detail as part of the evaluation. Exhibit 3 on the following page provides an overview of the activities associated with entering into these agreements.

2.19   From the time the first call closed on July 15, 2005, it took approximately 10 and 12 months respectively to enter into the first two contribution agreements. The third agreement was signed approximately 8 months after the second call closed on December 22, 2006. It should be noted that discussions with contribution recipients were put on hold at the request of senior management for a period of time in the Fall of 2007.

Exhibit 3:  Overview For Establishing Agreements

2.20    The RMAF, within the context of discussing data collection issues, recognized that the CRA is not normally in the business of administering and managing contribution programs. It also estimated that the Charities Directorate would be managing 5 to 10 agreements per year and identified that this was a low and manageable volume of agreements. We have recently been advised that a total of four agreements have been entered into as a result of the second call since the completion of the examination phase of the evaluation. If there is overlap between the agreements resulting from the first and second calls, there may be a total of six agreements in place at one time; however, this would only be for a limited time period.

2.21   One of the reasons stated repeatedly by the program area for the number of agreements in place was the newness of contributions programs to the CRA and the need to educate resources both within and outside CISD. In addition, current and former program managers interviewed emphasized that a particularly cautious approach to implementation was taken given the uniqueness of the program within CRA and the high profile nature and risks associated with G&C programs in general.

2.22   Documentation was in place to assist applicants in applying to the program as well as to assist the program in the management and administration of the contribution agreements. In contrast, a lack of program documentation was in place to support the overall management of the program and ensure knowledge is transferred. When documentation was available, it was not always easily accessible or clear as to when the document was created and finalized. This caused some unease as to the reliability and completeness of the documentation.

2.23   Training and education activities by contribution recipients are underway.  Based on information provided by the contribution recipients during the examination phase, a total of 31 sessions were scheduled for completion by November 14, 2007. Based on 11 sessions that have been completed to date, the average number of participants per session was 15. It is not known at this time exactly how many sessions will ultimately be delivered; however, based on the original applications submitted by the contribution recipients and an average of 15 participants per session, it is possible that 3375 participants will attend the 225 targeted sessions by December 2008.
 
2.24   It should be noted that both contribution recipients plan to use technology to make the training and education available to a greater number of individuals, thereby increasing the potential reach of each project. One of the contribution recipients has 140 participants registered for telephone learning sessions that will be held in January and February 2008. Again, the final numbers and the success of this type of training are not known at this time.

2.25   Use of available funding.  As the first year of the program (i.e., 2004-2005) was dedicated to planning, contribution funding was made available in the second year of the program (i.e., 2005-2006). Contribution funding started at $1 million in 2005‑2006 and 2006-2007, increased to $2 million in 2007-2008 and will further increase to $3 million in 2008-2009 and subsequent years. For grants and contribution programs, money not spent in one fiscal year can be carried-forward up to the amount of the allocation for the given fiscal year; if the money cannot be carried over to the next fiscal year, it is considered a lapse. Given this, the program spent $501,200 and had a lapse of $498,800 in 2006-2007. Recent confirmation of 2007-2008 expenditures revealed that the program spent $939,244 and will lapse an estimated $60,756 in 2007-2008. Exhibit 4 below provides an overview of contribution funding for the program.

Exhibit 4:  Contribution Funding

Original program design has not been fully realized

2.26   As described earlier, little documentation was found detailing the original program design. Given this, the RMAF was used to assess the extent to which the program had been implemented as designed. A copy of the logic model provides additional information on the expected program activities, outputs and outcomes and is included (see Exhibit 5 below). As noted previously, prior to the start of this evaluation, the program area had raised questions regarding the appropriateness of the RMAF and wanted to revise it accordingly. Given that the start date had already been delayed, the evaluation team proceeded with the evaluation and committed to providing CISD advice and guidance with respect to updating the RMAF. Program Evaluation, in consultation with CISD, will develop a revised evaluation plan for the summative evaluation upon conclusion of the formative evaluation.  

Exhibit 5:  Charities Partnership and Outreach Program Logic Model

Results-Based Management and Accountability Framework (RMAC),
Treasury Board Submission, Approved March 2005

2.27   Collaboration and promotion activities have not taken place.  Collaborating with the charitable sector and other stakeholders and program promotion were two key activities in the original RMAF. We found little evidence of formal collaboration activities that had been undertaken to engage the charitable sector and key stakeholders to conduct a needs assessment and identify funding priorities. The Charities Advisory Committee (CAC), identified as a key stakeholder, met 7 times from April 2004 to February 2006 before it was cancelled in September 2006. The evaluation team was recently advised that a series of consultations with small and rural charities were held in October 2007 but this initiative was not included in the scope of the evaluation.

2.28   The RMAF identified a number of activities that would be developed as part of a comprehensive communication plan. Examples included the use of targeted distribution lists to notify eligible recipients of calls for applications and the integration of CPOP in existing CRA communication products. Although some work had been done on a draft marketing strategy and communications plan, neither of these tools had been fully established at the time of the evaluation.

2.29   To date, much of the program promotion activity has focused on the calls for applications. Information, such as the funding guide and application (RC4411), assessment criteria, terms and conditions, and frequently asked questions are posted on the CRA Website. The Minister of National Revenue formally launched each call for applications and a news release was issued to that effect. Subsequent news releases were issued to announce the signing of the first two contribution agreements.

2.30   With the exception of interviewees from other federal government departments, all interviewees had some knowledge of the program, and consequently, the interviews could not be used to assess awareness of the program. In spite of this, eight interviewees felt that overall awareness of the program was low within the sector. At least four interviewees were not aware of recent program developments, such as the second call for applications, or the status of the training and education being delivered by contribution recipients.

2.31   Information on the program was also included in three issues of the Registered Charities Newsletter that, as of May 2007, had 14,035 subscribers on its electronic mailing list. However, a total of 34 publications, information circulars and interpretation bulletins related to registered charities are currently published on the CRA website and could potentially be used to further promote the program. One example is the Registered Charities:  What’s New – Important Changes to the Law Affecting Registered Charities (RC4414) that was updated in May 2007.

2.32   CISD is also responsible for the management and administration of the charities roadshows. Roadshows are free information sessions provided to registered charities on a yearly basis on topics of interest, changes to the law and common compliance issues. The evaluation team had discussions with the program area on how roadshows could best be used to promote or complement the CPOP. Exactly how the two programs will work together is not known at this time.

2.33   Program promotion needs to be better targeted.  The eligible recipients for this program include two distinct groups: (1) registered charities as defined by the ITA;and (2) non‑profit organizations serving the charitable sector. The evaluation team found that CISD had access to information that could be used to promote the program to each and every registered charity. The same type of information for non-profit organizations was not available to CISD and subsequently cannot be used in the same manner to promote the program, or to assist CISD in determining the risks associated with non-profit organizations that apply for funding. An analysis to further breakdown the two broad types of eligible recipients into target audiences for promotional purposes has not been undertaken.

2.34   It is also not clear how many eligible recipients would actually be in a position to apply for contribution funding as compared to those that need training and education. We were informed on various occasions of the disparate needs and capacity levels of the charitable sector and that many registered charities do not have the capacity in place to deliver projects of this scope. Criteria is in place to assist with the assessment of applications for funding; however, the program area has not defined expectations that could be used by CRA to determine if an applicant has the capacity to work in partnership with the CRA and deliver the training in a sustainable fashion.

Limited progress in measuring program performance

2.35   As identified in the TOR, this evaluation planned to examine whether the existing performance measurement strategy is appropriate to ensure that program management is able to measure its success in achieving its objectives, and whether this strategy is sufficiently developed and implemented to allow ongoing measurement of program performance. As indicated previously, the TB Policy on Transfer Payments imposes certain requirements on contribution programs and of particular importance is the need for the program to measure and report on results. Results-based performance measurement data is required to support the summative evaluation committed by CRA in the TB Submission. If the required performance measurement data is not in place, the value of conducting a summative evaluation is questionable.

2.36   Contribution recipients are required to report on performance.   Contribution recipients, as part of their agreements, are currently required to provide CISD with an evaluation framework, a mid‑term evaluation report and a final evaluation report. The first two contribution recipients have worked together to develop a feedback questionnaire to be administered to participants upon completion of each training session. Both contribution recipients intend to follow‑up with selected participants after the sessions; however, this will only be done once and will not cover multiple filing periods. Contribution recipients are basically required to self-assess and evaluate the results of their projects.

2.37   Additional program performance measurement is required.   The nature of this program, which relies on third parties to deliver training and education, and its objectives make performance measurement a challenge. CPOP’s objectives focus on increasing awareness of ITA regulatory obligations and the capacity of the charitable sector to meet its obligations and deliver sustainable education. The evaluation team found that the program objectives were not well-defined, measurable or realistic given the nature and size of the program. We also found that the program is not independently assessing performance and measuring results in terms of these objectives. The Independent Blue Ribbon Panel (BRP) on Grants and Contributions, in its report of July 2006, stressed the importance of independent performance measurement and noted that, while G&C programs are designed to achieve public policy objectives, it is rare that a single recipient is in a position to meet these objectives alone.

2.38   The evaluation team found that limited analysis and documentation was available to support the selection of funding priorities and of the data necessary for future performance measurement. Data in relation to the funding priority established for the first call for applications is available from other areas within the Charities Directorate and could be used to establish baselines and assess whether there has been any changes over time. However, it is unclear as to what data is available to the program area that could be used to measure and assess the impact of these funding priorities on compliance behaviour. The results of audits could be used; however, a total of 847 audits were conducted in 2006‑2007 and the number of audits that would be conducted on registered charities that received training is not known at this time.
 
2.39   The program area has raised concerns with respect to the costs of implementing a performance measurement strategy and the confidentiality and privacy of data collected by contribution recipients during the course of their projects. These concerns will need to be taken into consideration in the development of the performance measurement strategy. Regardless, the program needs to be in a position to measure performance and demonstrate its impact to CRA senior management and be in accordance with the TB Policy on Transfer Payments. It should also be noted that, given the limited reach of this program in terms of the overall population of registered charities, it might be difficult to attribute changes in compliance behaviour to this program.   

2.40   As part of updating the RMAF, Program Evaluation is willing to provide CISD with advice and guidance on the development and implementation of a performance measurement strategy upon conclusion of the formative evaluation.

Practices to consider for program improvements

2.41   In addition to the issues of program relevance, design and implementation to date, the evaluation pursued alternatives or modifications to the current program. This section puts forward practices for consideration by program management for program improvement.

2.42   The BRP reported that over 50 departments and agencies spend some $27 billion each year on 800 grants and contributions to individuals, corporations and non-government organizations. G&C programs are used to assist the Government of Canada in meeting its public policy objectives by entering into funding partnerships with organizations that deliver programs and services to Canadians.

2.43   Interviews with selected representatives of Department of Justice Canada, Canadian Heritage, Health Canada, Transport Canada, and Human Resources and Social Development Canada who were responsible for the management, administration and evaluation of G&C programs were conducted as part of this evaluation. As a result of these interviews, it became apparent to the evaluation team that management and administration practices of these programs were quite diverse. In fact, the BRP reported that  “…the manner in which G&C programs are currently managed and administered in the federal government is as diverse as the public policy objectives for which they are put in place to achieve”. Despite the differences, there are many other practices that could be considered by CISD in the ongoing implementation of the program.

2.44   It should be noted that during the evaluation, the program area was reflecting on lessons learned from the first call for applications, and had identified potential modifications to the program for future calls. In particular, the information gathered and experience gained as a result of the first call was used to make improvements to the second call for applications (e.g., revised contribution agreement template).

2.45   As indicated previously, the evaluation team conducted interviews with representatives of the voluntary sector. These interviews, combined with the interviews conducted with other government departments, identified a number of modifications that, upon further investigation, could be made to improve the application process. For example, it was suggested that a 2-tier application process be implemented that would first require the submission of a letter of intent, and then based on the CRA’s assessment of the letter of intent, only then require the submission of a detailed application. It was felt that a 2-tier process would benefit both the CRA and applicants by streamlining the process and prevent applicants from putting significant time and effort into an application that is unlikely to receive funding.

2.46   Another area where applicants felt CRA could make improvements is in the provision of information to applicants at various stages of the application process in order to improve transparency. It was felt that additional feedback could have been provided to applicants on the quality of their applications and areas for improvement. It was also felt that additional information on successful applicants, the status of the projects, and subsequent calls for applications would be beneficial.

2.47   The ARC, which included representatives from the voluntary sector and other government departments, assessed applications and made recommendations for funding. Many interviewees noted that the use of the ARC was a positive feature of the program as it added to the value and credibility of the process and the quality of applicants selected. Building on this practice, it has been suggested that there are areas associated with the ARC that could be formalized (e.g., operating guidelines, feedback to the ARC members on funded projects).

2.48   Other interviewees questioned whether CRA has made full use of organizations (e.g., philanthropic foundations) that are well established in the sector and are already providing training and education on the ITA to their members. These organizations could be used to expand on existing training and education, in particular within specific groups within the voluntary sector, as well as provide a forum for collaboration.

2.49   The program area has also identified areas in which it would like to make modifications. One of these includes having an open call for funding to eliminate the need for applicants to submit applications within a specified period of time. CISD would also like to allow funding to be used for research, and in the 2006‑2008 funding guide, added research and technical studies to the list of projects that are provided as examples that would be eligible for funding. 

2.50   Finally, the BRP provides an additional source of information for potential modifications. In response to the BRP recommendations, the President of the TB announced a series of immediate actions in February 2007 that had led to the development of an action plan to reform the administration of G&C programs, including the creation of a G&C Centre of Expertise. In June 2007, the Charities Directorate advised the evaluation team that a gap analysis of BRP recommendations would be undertaken to identify opportunities for program modifications. We were also advised that the Charities Directorate would continue to monitor the activities of the G&C Centre of Expertise to determine if additional program modifications could be made.

Conclusion and Recommendations

3.1    The creation of CPOP was based on a government-wide initiative that supported strategic initiatives that aimed to improve, through projects, compliance with the regulatory environment in which charities operate. We found that CPOP is relevant to the CRA and the federal government, and the program area points to selected CRA data to demonstrate the need for enhanced outreach and compliance-based education. The evaluation found little analysis or documentation that led to CRA’s choice of policy instrument, i.e., a contribution program, as the most effective approach to implement the program. A more thorough assessment of program results, through a subsequent summative evaluation, is required to determine the impact of this program on compliance and if the policy instrument was the most appropriate choice for implementing the program.

3.2   At the time of the evaluation, three agreements were in place and training within the sector had only recently begun. Little of the approved funding has been spent, and there was a lapse in the first two years that contribution funding was available. Our review of CPOP’s current operations found various deviations from the original program design that was approved by TB. While the activity of putting in place the contribution agreements has clearly been the focus of CPOP, this was to the exclusion of collaboration and promotion activities. More importantly, a strategy to assess and measure overall program performance has not been developed and this raises significant concerns with respect to the CISD’s ability to determine if the program is on track to achieve their objectives. As stated previously, if the required performance measurement data is not in place, the value of conducting a summative evaluation is questionable.

3.3   We believe that CISD will continue to make improvements as knowledge and experience is gained. In the section below, we put forward some recommendations that might assist the program in this respect. Nevertheless, as the CRA is currently committed to conducting a summative evaluation and results‑based data is needed in order to conduct such an evaluation, it is essential that CISD work on developing and implementing a performance measurement strategy to measure the impact of the program. These program results are partic  ularly important due to the renewal of the terms and conditions that is required for this program in March 2010.

Recommendations

Recommendation No.1 

Program management should revisit the original program design to reassess the feasibility of implementing the program in terms of what was planned as detailed in the TB Submission that was used to obtain funding for this program. The current state of the program should be documented as well as the activities that are needed to ensure that the program is fully implemented. If there are activities that are not deemed feasible, the impact of not implementing these activities should be assessed in anticipation of the renewal of the terms and conditions in March 2010.

Management Response

The Charities Directorate agrees with this recommendation. The Directorate plans to modify the Program design as articulated in the RMAF. In addition, the Directorate will continue to build on the significant progress that has been made since implementing the program in 2005-2006. In particular, the Charities Directorate will continue to review and update administrative and program management policies, directives and practices based on Treasury Board policies, recommendations from the Centre of Expertise on Grants and Contributions, once established, as well as legislative amendments and other applicable government policies and practices.

Recommendation No. 2 

Program management should focus on developing and implementing a performance measurement strategy that will identify the baseline data that is required to measure performance and report on the success of this program. Given that CRA has committed to conduct and report on the results of the summative evaluation to TB, this data is needed from both an ongoing program management perspective as well as to ensure that the evaluation team has the data necessary to evaluate the achievement of program objectives during the summative evaluation.

Management Response

The Charities Directorate agrees with this recommendation. The Directorate recognizes the importance of implementing a performance measurement strategy. In the months that follow, the Directorate will study how to strengthen performance measurement to ensure that a strategy is implemented that reflects both the size of the program and the need to demonstrate results. This will include the collection of reliable data to support the on-going administration of the Program as well as any assessments and program evaluations necessary to meet both legal and policy requirements. 

Recommendation No. 3 

Program management should develop and implement a communication strategy that clearly identifies the steps that will be taken, both in terms of collaboration and promotion, to help achieve the program objectives.

Management Response

The Charities Directorate agrees with the importance of promoting the Program. As such, the Charities Directorate will revise its communication strategy to promote program activities consistent with administrative milestones (i.e., calls for proposals, the signing of agreements, projects up-dates and completion). In addition, the Charities Directorate will continue to promote the Program through sector specific presentations, Charities Information Sessions, the Charities Directorate’s external Newsletter and the What’s New publication. 

Recommendation No. 4 

Program management should implement the policies, procedures and information system(s) necessary to ensure that quality information is available to support decision-making, the measurement of performance as well as accountability and transparency of the program.

Management Response

The Charities Directorate agrees with this recommendation and will continue to strengthen its efforts to promote greater accountability and transparency. A new Charities Tracking System has been developed and is currently being implemented across the Directorate. Once the system has been aligned to meet the Program’s needs, staff will be able to more effectively track activities relating to the administration of the Program, which will further assist with decision-making, reporting and overall accountability.  

Final Comments from Program Management

In support of the above Management Response, Program Management will continue to work with Program Evaluation in order to develop an action plan. The Charities Directorate has benefited from the evaluation report. Once the recommendations have been implemented, the Program should realize further efficiencies, increased effectiveness and strengthened accountability. In closing, the Charities Directorate would like to take this opportunity to thank everyone involved in the Formative Evaluation.

Appendix A

About the Evaluation

Purpose
The purpose of this evaluation was to provide information and advice to the Commissioner and senior management on whether the CPOP is well positioned and on track to meet its objectives, and to identify areas for improvements.

Evaluation Issues

This evaluation focused on the three following issues:

1. To what extent is the program relevant to the CRA and the federal government?
   
   This issue examined the rationale for the program in order to determine whether the program is relevant in terms of the current strategic objectives and priorities of the CRA and the federal government.
   
2. To what extent has the program been successfully designed, implemented and delivered to achieve expected benefits?
   
   This issue examined the design and implementation of the program to determine if it was delivered as planned; the extent to which key stakeholders were aware of the program and were involved in program design and delivery; the effectiveness of application, selection and subsequent monitoring processes; and the adequacy of the program resources. This included an examination of the performance measurement strategy to ensure that the program is able to measure progress against its objectives, in particular, in light of the upcoming summative evaluation.
3. Are there alternatives or modifications to the current program that would enhance the realization of Agency objectives?

This issue examined whether the program should undertake a different direction or make modifications to the current design and delivery of the program. It will explore alternative approaches, best practices and potential modifications to the program.

Evaluation Requirements  

In March 2005, Ministers approved the terms and conditions, as part of the TB Submission, for the CPOP. As stipulated in the Policy on Transfer Payments, the TB Submission also included, in the RMAF, the evaluation criteria that would be used to assess the effectiveness of the program.

The CRA committed to undertake a formative evaluation that would report in December 2006. As a formative typically focuses on processes, such as performance measurement, to identify areas for program improvement, the evaluation was planned to begin in 2006‑2007 after the program had been operational for a full year. CRA also committed to undertake a summative evaluation that would report in September 2008 in advance of the renewal of the terms and conditions for this program in March 2010.

It should be noted that the evaluation issues for both the formative and summative evaluation identified in the original RMAF were refocused. As approved in the Terms of Reference, issues related to program relevance and program alternatives or modifications were brought forward and examined as part of the formative evaluation. In addition, an assessment of program results will not take place until the summative evaluation. As a result, a revised evaluation strategy to reflect these changes will be developed as part of this evaluation.

Appendix B

Glossary of Terms

Application Review Committee (ARC) – Applications received through the CPOP that meet the screening criteria are forwarded to an Application Review Committee (ARC). The ARC includes representation from the CRA and the sector. Depending on the nature of the applications, representatives from other federal departments, the provinces, territories, and academia may also be involved in the review. The Committee considers factors such as innovativeness, sustainability, and scope of the project when evaluating the applications.

Blue Ribbon Panel (BRP) – The Blue Ribbon Panel was an independent   panel commissioned in June 2006 by the President of the TB. The panel conducted a six-month review of the administration of federal G&C programs and recommended measures to make the delivery of these programs more efficient while ensuring greater accountability.

Charitable sector – The charitable sector is defined by those charities that are federally registered under the Income Tax Act.

Contribution agreement – Each application that is approved for funding will be subject to a formal agreement specifying the responsibilities of each party; the conditions under which payments will be made in accordance with the TB Policy on Transfer Payments and the Financial Administration Act; and mutually agreed upon measures to assess the success of the activities in achieving the objectives of the project.

Formative evaluation – A formative evaluation is a type of evaluation that has the purpose of ongoing program improvement. It is usually conducted early in the implementation of a program, and used to evaluate if the program is on the right track to achieving its objectives. This type of evaluation usually focuses on issues such as implementation, relevance and possible alternatives or modifications.

Non-profit organization – Under the Income Tax Act, a non-profit organization is an association, club, or society operating exclusively for social welfare, civic improvement, pleasure, recreation, or any other purpose except profit. It is not a charity.

Performance measurement strategy – The selection, development and ongoing use of performance measurements to guide decision-making. The range of information in a performance measurement strategy could include: reach/target group, outputs and outcomes, performance measurements, data
sources, methodology and costs.

Results‑based Management and Accountability Framework (RMAF) - A document which serves as a blueprint for managers to help them focus on measuring and reporting on outcomes throughout the lifecycle of a policy, program or initiative. The RMAF includes: program profile, logic model, ongoing performance measurement strategy, evaluation strategy, evaluation issues and questions, and reporting strategy.

Registered charity – A registered charity refers to a charitable organization, public foundation, or private foundation that was established in Canada and is resident in Canada. It is operated for charitable purposes and must devote its resources to charitable activities. A registered charity has received a registration number from the Canada Revenue Agency and is exempt from paying tax on its revenue. It can issue donation receipts for gifts that it receives.

Summative evaluation – Summative evaluations are typically conducted further into the implementation of a program and focus on the achievement of results. This type of evaluation focuses on program results, especially for making decisions about program continuation, expansion, reduction and funding. It addresses outcomes in order to evaluate the program in terms of effectiveness as compared with similar type programs.

Treasury Board Policy on Transfer Payments – A policy developed by TB to ensure sound management of, control over, and accountability for transfer payments. Transfer payments are made on the basis of an appropriation for which no goods or services are directly received, but which may require the recipient to provide a report or other information subsequent to receiving payment. Three types of transfer payments are grants, contributions, and “other transfer payments”.

Voluntary sector – The voluntary sector is one of the three pillars, along with the public and private sector, that make up Canadian society. The voluntary sector, in its broadest sense, is composed of all non-profit organizations (including registered charities) that exist in Canada.