Executive Summary - Evaluation of the Building Communities through Arts and Heritage for the period 2011-12 to 2014-15

Evaluation Services Directorate

October 17, 2016

Cette publication est également disponible en français.

© Her Majesty the Queen in Right of Canada, 2016

Catalogue No. CH7-10/2016E-PDF
ISBN 978-0-660-23189-1

Overview of the program

The Building Communities through Arts and Heritage (BCAH) program, managed and delivered by Community Engagement Directorate within the Citizen Participation Branch of the Department of Canadian Heritage (PCH), was launched as a new program in 2007 in response to the federal budget which expressed the intent to support local arts and heritage festivals and small capital projects that place an emphasis on local engagement. In August 2009, the Legacy Fund component was added to support larger capital projects that involve the restoration, renovation or transformation of existing buildings and/or exterior spaces.

BCAH provides grants and contributions (Gs&Cs) to non-profit community organizations, aboriginal governments or municipal governments to support them in planning and organizing events and projects that engage citizens in their communities through the performing and visual arts, as well as through the expression, celebration and preservation of local heritage. BCAH is administered through three sub-components:

• The Local Arts and Heritage Festivals component provides funding in support of festivals, events and activities that engage Canadians in their communities through recurring public presentations of local artists and/or of local historical heritage.
• The Community Historical Anniversaries Programming component provides funding in support of non-recurring events and activities that engage Canadians in their communities through the commemoration of major anniversaries of significant local events and/or persons.
• The Community Historical Anniversaries Legacy Fund component provides funding in support of capital projects that engage Canadians in their communities through the commemoration of major anniversaries of significant local events and/or persons.

BCAH is a subprogram under Program – Engagement and Community Participation, as presented in the PCH Program Alignment Architecture (PAA) and contributes to the Strategic Outcome – “Canadians share, express and appreciate their Canadian identity”.

Context and purpose

The evaluation of BCAH covers the period from April 1, 2011 to March 31, 2015. The evaluation was conducted in 2015-2016 by the PCH Evaluation Services Directorate (ESD) with support from the Policy Research Group (PRG) for the literature review and the administration of the surveys.

The purpose of the evaluation was to examine the following:

• the relevance of the program, in particular the extent to which (i) BCAH continues to meet a demonstrable need and responds to the needs of Canadians; (ii) BCAH’s objective aligns with federal government priorities and departmental strategic outcomes; and (iii) BCAH aligns with federal government and PCH roles and responsibilities;
• the performance of the program (effectiveness, efficiency, and economy); and
• other issues pertinent to the program, including performance measurement and program design and delivery.

Methodological approach

The evaluation used the following lines of investigation: literature review; document review; review of project files and administrative databases; interviews with key informants, including PCH officials (National Headquarters and regional), BCAH funding recipients, applicants who had been unsuccessful in obtaining funding and representatives of municipal and provincial governments; case studies; and surveys of funding recipients and applicants who were unsuccessful in obtaining funding.

Limitations and challenges

Limitations of the evaluation were taken into consideration in the presentation of findings, conclusions, and recommendations including: a lack of data on the longer-term outcomes of the program; some gaps in performance data for 2014-2015 for Legacy Fund projects; and the scope of the program involving three different components with different activities, funding models and assessment processes which added to the complexity of the evaluation.

Relevance

The conditions under which BCAH was formed persist today, including few funding sources for small local festivals, commemorative events and for arts and heritage capital projects generally, and more specifically for events and projects with the primary objective of citizen engagement. Canada’s arts and heritage infrastructure is suffering from years of neglect with a deficit of funding support for the restoration, renovation or transformation of existing buildings and/or exterior spaces with local community significance.

Recipients view BCAH’s citizen engagement objective as relevant with the majority of Local Festivals and Community Anniversaries recipients agreeing to “a great extent” that BCAH is an effective mechanism to increase engagement of citizens in their communities. Legacy Fund recipients were somewhat less likely to agree “to a great extent”.

Evaluation evidence indicated consensus among Canadians that arts and cultural experiences contribute to community and individual well-being. BCAH-funded festivals, in particular, played a role in generating a sense of community and belonging, civic engagement and community pride – further solidifying the relevance of federal support in this area.

BCAH was responsive in that it provided funding to almost all eligible Local Festivals and Community Anniversaries applicants (90%) who achieved the minimum required merit score and to 70% of eligible Legacy Fund applicants. However, the amount of funding requested exceeded the available funds. During this period, successful applicants requested $152.5 million and the federal government provided local organizations about 68.3 million to carry out local festivals, community anniversaries and legacy projects in their communities. The average amount of funding per project relative to the eligible amount requested for the four years covered by the evaluation was 41.3% for Local Festivals and 44.1% for Community Anniversaries projects. For Legacy Fund projects, the amount paid to the recipient relative to requested amount averaged 74%.

In the last two years of the evaluation period, the program has received fewer applications, particularly to the Community Anniversaries and Legacy Fund components. No clear reason emerged for the fewer applications. PCH key informants suggested that the fewer applications could be attributed to the normal maturing of the program and factors such as the eligibility requirement linked to a 100 year anniversary or greater in increments of 25 years.

During the period covered by the evaluation, the BCAH program aligned with government priorities aimed at promoting vibrant Canadian communities and supporting arts culture and heritage, and with PCH’s strategic outcome that Canadians share, express and appreciate their Canadian identity. It contributed to two departmental priorities – bringing Canadians together and celebrating our history and heritage. The program currently does not target funding to support Government of Canada (GC) diversity goals or PCH priority groups (for example, Official Language Minority Communities (OLMC), ethno-cultural and Indigenous Canadians and youth).

All key stakeholders agreed that delivery of the program is an appropriate role for the federal government and PCH given the PCH mandate to foster a stronger Canadian identity through active, engaged and inclusive citizenship; the recognition of the importance of a shared civic identity; and the fact that there are few other sources of funding, particularly for small local festivals and, in particular, festivals with an objective of citizen engagement.

Performance: Achievement of expected outcomes

Between 2011-2012 and 2014-2015, PCH funded 3336 projects and provided local organizations about 68.3 million to carry out local festivals, community anniversaries and legacy projects in their communities, contributing to the program’s immediate outcome – local organizations receive financial resources to carry out local festivals, community anniversary and/or legacy projects in their communities. Funding helped recipients offset the costs associated with engaging local artists, artisans and heritage performers and with supporting their volunteers. While BCAH funding had a significant impact on the ability of the majority of recipients to undertake their events, it had a greater impact on Legacy Fund recipients and rural recipients of local festivals. They were more likely to indicate in their survey responses that their event would not have occurred without BCAH funding.

Funded organizations contributed to the achievement of BCAH’s citizen engagement outcomes:

• BCAH funding played a significant role in helping organizations to leverage cash and in-kind support from local community partners. Funded organizations successfully secured local partners and obtained substantial sums of cash and in-kind support from municipal and community partners, as well as non-local partners such as provincial governments. Between 2011-2012 and 2014-2015, BCAH-funded projects secured $298.6 million in cash and in-kind support from municipal and community partners. For every $1 investment by PCH, BCAH projects secured $5 of cash and in-kind support from municipal and community partners.
• Overall, BCAH projects averaged 159 volunteers and 4,366 volunteer hours per project, exceeding its target of 149 volunteers and 3,954 volunteer hours per project.
• BCAH projects provided local artists, artisans and heritage performers with opportunities to be involved in their communities. BCAH projects averaged 158 local artists, artisans and heritage performers per project, exceeding its target of 125 per project.
• BCAH projects provided local citizens with opportunities to be exposed to local arts and heritage. BCAH projects averaged 28,436 visitors/attendees per project, exceeding its target of 22,343.^{Footnote 1}

Legacy Fund projects, by their nature, offer fewer opportunities for citizen engagement. Therefore outcome results were relatively modest in terms of the number of volunteers, artists, artisans and heritage performers when compared with the Local Festivals and Community Anniversaries components.

There was no systematic reporting of outcomes beyond data collected on immediate and intermediate outcomes at the conclusion of funded projects, therefore, it is difficult to discern the achievement of the longer-term, ultimate outcome – citizens across the country are engaged in their communities through local arts and heritage. However, anecdotal evidence from interviews and surveys suggested that BCAH projects generated social benefits for communities, contributed to a greater appreciation of arts and heritage, led to the creation of longer-term partnerships and generated benefits for artists.

Performance: Efficiency and economy

The overall administrative costs incurred by PCH for the management and delivery of the program totaled $15,752,828 for the period 2011-2012 to 2014-2015 which represented 18.7% of total BCAH expenditures. This was higher than the previous evaluation (16.4%) and higher than similar PCH programs, evaluated in 2014 (Canada Cultural Spaces Fund (CCSF) at 7% and Canada Arts Presentation Fund (CAPF) at 11%). Comparing the volume of applications, BCAH’s Local Festivals component had, 1.4 times the applications annually than CAPF. Comparing the Legacy Fund to CCSF, CCSF had four times the number of applications annually.

The proportion of administration costs to total expenditures for the administration of the Legacy Fund and program-related activities undertaken at National Headquarters averaged 20.7% for the four years covered by the evaluation.^{Footnote 2} The administration costs relative to total expenditures was higher in 2014-2015 (25.1%) because the program lapsed $2.7 million in G&C funds. Some Legacy Fund recipients were unable to spend all their funding and a number of projects were not approved. Due to timing, the program was unable to reallocate funds. The proportion of administrative costs to total program expenditures to deliver the Community Anniversaries and Local Festivals components, and related regional activities, was relatively stable, averaging 18.2% over the four years.

Compared with the Local Festivals and Community Anniversaries, Legacy Fund projects had higher administrative costs per application reviewed and projects funded reflecting differences in the characteristics and complexity of Legacy Fund projects; the number of applications approved; the amount of funding; and the structure of the application review process. The Legacy Fund also had the highest cost per outcomes reflecting the fact that legacy projects are fundamentally different from Local Festivals or Community Anniversaries projects, and have lower levels of citizen engagement.

All stakeholders were of the view that BCAH is complementary to any other program delivered by other funding organizations or levels of government. A review of PCH programs concluded that they have different objectives and are designed to fill a specific niche so are complementary. A scan of provincial/territorial (P/T) programs concluded that most have an economic or tourism focus, rather than a citizen engagement focus.

Performance: Other issues

Overall, recipients were satisfied with the design and delivery of the program. Survey results showed lower levels of satisfaction with some aspects of the program including the timeliness of notification of the funding decision, the complexity of the funding application and, in the case of local festivals, the level of funding provided relative to what was requested. The program’s objective to fund all eligible recipients who achieve a minimum required merit score contributes to efficiency and equity and enables the program to support more applicants. However, a number of factors can affect individual contributions and contribute to fluctuations in amounts received by some recurrent festival recipients each year. These fluctuations in funding from year to year and delays and timing of notification of the funding decision created challenges for some recurrent festivals in terms of their ability to plan and to leverage BCAH funding to secure funding from other sources.

Currently, the program does not target more vulnerable organizations in greater need of funding support and stability, such as organizations in smaller communities and rural areas which may have less access to funding. Small festivals and festivals in smaller communities competed for funding with larger, successful and well-established festivals. Organizations responsible for large and well-established festivals who meet the eligibility criteria and who achieve the required merit score are provided funding on the same basis as smaller organizations. Some of these large festivals have a long-standing history of BCAH funding. As shown by the administrative data, funding to these festivals had an impact on the amount of funding available to smaller festivals. The largest 95 projects, representing 3% of total projects, consumed a significant proportion of BCAH funding (16%) because they generally had more eligible expenses and, therefore, received larger amounts of funding whereas the 914 smallest projects, representing 32% of all projects, consumed 9% of BCAH funding.

Between 2011-2012 and 2014-2015, a number of legacy projects involved historic sites designated nationally or under provincial legislation. While the Legacy Fund application requires applicants to indicate that they have consulted and are in compliance with the Standards and Guidelines for the Conservation of Historic Places in Canada, the program does not obtain documentation confirming that projects have complied with the standards and guidelines.

The evaluation found that the program collects data on its immediate and intermediate outcomes and produces an annual performance report. There is strong evidence that the information is being analyzed and used for program improvements. However, the program does not collect data on its ultimate outcome, nor does it track of the program’s contributions to GC diversity goals and PCH priorities (youth, OLMCs, Indigenous and ethno-cultural groups).

Stakeholders identified social and community benefits resulting from BCAH projects (e.g., social cohesion, pride and attachment to their community) but the program’s logic model and performance measurement strategy do not include social, cultural, and community benefits derived from BCAH-funded projects.

Conclusions

BCAH remains relevant. The evaluation evidence confirmed that there is a continued need for federal support to local festivals and commemorative events. The program has been responsive insofar as it funded almost all eligible Community Anniversaries and Local Festivals applicants and to a lesser extent Legacy Fund applicants. However, the demand for funding exceeded the available funds. The program aligned with both federal and departmental objectives and priorities during the period of the evaluation. Promoting citizen engagement in their communities is an appropriate role for the federal government.

While the Program has made on-going efforts to adjust the elements of its funding model to respond to issues with respect to fluctuations in funding year-to-year, it should continue to seek ways to support more vulnerable organizations in greater need of stability, such as organizations in smaller communities and rural areas who may have less access to funding. The program should review it funding to large well-established festivals, and the amounts they receive, to determine their continued need for funding support with the potential reallocation of funds to better support more vulnerable organizations or to projects that support PCH and GC priorities. Also, in the absence of documentation on the methodology used to arrive at the population grids/tools used to determine the merit score, the program should review these tools to ensure that they do not disadvantage smaller communities.

Overall, the program achieved its intended outcomes and exceeded its performance targets for its intermediate outcomes. The funding provided by BCAH is incremental in that most funded projects would have proceeded, but with a reduced scope, without the support provided by PCH. Funded organizations successfully leveraged BCAH funding to secure cash and in-kind support from municipal and community partners. The program had an impact on citizen engagement (partners, volunteers, local artists, artisans and heritage performers and local citizens) in local communities. It also increased the capacity of local community organizations to offer arts and heritage experiences to their communities. Anecdotal evidence and survey data suggested that the projects contributed to social and community benefits, although data on these longer-term impacts are not being systematically collected by the program.

There are opportunities for the program to improve efficiency while improving the service it provides to its funding recipients. The program’s administration ratio was high for the period of the evaluation and the program encountered challenges in meeting its 26 week service standard for notification of funding decisions. However, in the case of the Local Festivals component, the majority of festivals are recurrent recipients, many are low risk and receive relatively small amounts of funding. Among the options the program could consider is further streamlining of application and assessment processes particularly for low-risk files, including the introduction of an accelerated approval process and the approval of funding for more than one year. This would contribute to reducing the application processing time and the time to notify recipients of the funding decision, improve efficiency and increase the satisfaction of recipients with the process. Further efficiencies could also be gained by clarifying and simplifying applications thereby reducing the number of interactions between recipients and program staff. Improvements in these areas would alleviate some of the administrative burden on program staff and recipients.

There are some areas where performance measurement could be strengthened including the identification of outcomes that better reflect the impact of Legacy Fund projects and the collection of data on the programs ultimate outcome and on the program’s contributions to GC diversity goals and PCH priority groups (youth, OLMCs and Indigenous and ethno-cultural communities). Citizen engagement outcomes may be less relevant to Legacy Fund projects and the BCAH outcomes do not appear to resonate as strongly with Legacy Fund recipients suggesting that consideration be given to identifying performance indicators that better reflect Legacy Fund outcomes. Also, the collection of data on longer term impacts, including the social, cultural, and community benefits derived from BCAH-funded projects, would contribute to the ability to measure progress toward the achievement of BCAH’s ultimate outcome and provide evidence of the longer-term impact of BCAH on Canadians. However, the collection of additional data must be balanced, with sensitivity to the burden on recipients and program staff and alignment with the directions being taken with respect to reporting by the Gs&Cs modernization initiative.

Although the Standards and Guidelines for the Conservation of Historic Places in Canada is not legally binding, and the program asks applicants to indicate on their application that they have consulted and complied with the standards and guidelines, to ensure the integrity of the historic sites there should be a greater burden of proof on recipients to demonstrate compliance with the standards and guidelines both before a project proceeds and upon completion of a project.

The way forward

The BCAH program has demonstrated that it is an effective mechanism to support GC priorities to invest in vibrant communities, to celebrate our arts and heritage and to promote PCH citizen engagement objectives by the leveraging of PCH funding to acquire community partners, and by engaging local citizens through volunteerism and attendance. The evaluation evidence indicates that BCAH continues to be relevant in that there is limited funding available to smaller communities and for festivals and commemorative anniversaries with the objective to engage citizens in their communities.

Budget 2016 introduced several broad policy themes of priority to the GC, including: diversity and inclusion; supporting efforts toward national reconciliation of indigenous and non-indigenous Canadians; engaging and supporting youth; and our environment. BCAH, given its flexibility and wide reach into communities across the country, particularly through its Local Festivals component, could be an effective policy instrument to contribute to GC priority themes, by putting a greater emphasis and potentially larger funding allocations to organizations and projects/events that reflect or contribute to the Government’s priority areas.

Although not an explicit criteria for funding, survey evidence found that many of the BCAH-funded festivals had an ethno-cultural or multicultural dimension and some festivals also received funding from the Multiculturalism program at Citizenship and Immigration (CIC). However, as the two programs had different objectives, funding was determined to be complementary. With the transfer of the Multiculturalism program to PCH from CIC in 2015, however, the department will need to undertake a strategic policy exercise to examine interdependencies across PCH policies and programs, including the BCAH Local Festivals component, and consider how each program can best contribute to PCH strategic outcomes and to the Government's policy themes, including its broader diversity and inclusion agenda, and continue to ensure that duplication does not occur between these two programs.

Moving forward will entail a focus on more vulnerable communities and organizations, such as small communities and rural areas which have less access to funding and require funding stability, and on communities and organizations that promote the GC principles of diversity and inclusivity and which support PCH target groups ( youth, OLMCs, and Indigenous and ethno-cultural groups) .

Recommendations

The following five recommendations emerged from the evaluation findings.

Recommendation #1:

To respond to GC diversity goals and PCH priorities such as youth, OLMCs, indigenous and ethno-cultural communities, and the needs of recipient organizations, the Assistant Deputy Minister (ADM) of Citizenship, Heritage and Regions should review BCAH’s funding of Local Festivals, including the assessment criteria and tools, to ensure that they do not present unintended barriers to funding for smaller communities and organizations that support GC and PCH priorities.

Recommendation #2:

It is recommended that the ADM of Citizenship, Heritage and Regions find efficiencies by:

• streamlining the application, assessment and reporting processes, particularly for recurrent, low risk clients;
• reducing the time required to process files to meet established service standard timelines; and
• clarifying and simplifying application guidelines and processes.

Recommendation #3:

It is recommended that the ADM of Citizenship, Heritage and Regions review BCAH’s Performance Measurement Strategy to:

• align with the Departmental Results Framework, under development, and collect data on program impacts; and
• collect data on project contributions to GC diversity goals and PCH priorities, including but not limited to youth, OLMCs, Indigenous communities and projects with a primarily ethno-cultural or multiculturalism focus.

Recommendation #4

It is recommended that the ADM of Citizenship, Heritage and Regions review BCAH’s citizen engagement performance measures in relation to Legacy Fund projects to include indicators that may better measure the outcomes of legacy projects.

Recommendation #5:

For renovation or restoration projects that impact on a site that is recognized as a national or provincial historic site or is municipally recognized, it is recommended that the ADM of Citizenship, Heritage and Regions:

• seek attestation of compliance with the Standards and Guidelines for the Conservation of Historic Places in Canada from recipients as part of the application process and upon completion of the project; and
• obtain the supporting documentation submitted to P/T or municipal governments to demonstrate compliance with the Standards and Guidelines, when available.