Evaluation of the Canada Arts Training Fund 2012-13 to 2017-18

Period from 2012-13 to 2017-18
Evaluation Services Directorate
February 20, 2019

Cette publication est aussi disponible en français.

©Her Majesty the Queen in Right of Canada, 2019.
Catalogue No. CH7-19/1-2019E-PDF
ISBN: 978-0-660-29353-0

On this page:

• List of tables
• List of figures
• List of acronyms and abbreviations
• Executive summary
• 1. Introduction
• 2. Program profile
  • 2.1. Program history
  • 2.2. Program activities, objectives and expected outcomes
  • 2.3. Program management and governance
  • 2.4. Program resources
• 3. Approach and methodology
  • 3.1. Scope, timeline and quality control
  • 3.2. Evaluation questions
  • 3.3. Data collection methods
  • 3.4. Constraints, limits and mitigation strategies
• 4. Findings
  • 4.1. Relevance
  • 4.2. Effectiveness: achievement of expected outcomes
  • 4.3. Government-wide policy considerations
  • 4.4. Efficiency: demonstration of efficiency
• 5. Conclusions
  • 5.1. Relevance
  • 5.2. Effectiveness
  • 5.3. Government-wide policy considerations
  • 5.4. Efficiency
• 6. Recommendations, management response and action plan
• Annex A: evaluation framework
• Annex B: bibliography

List of tables

• Table 1: program logic model
• Table 2: target population and key stakeholders
• Table 3: program resources by fiscal year
• Table 4: evaluation questions by core issue
• Table 5: number and percentage of recurrent and new applicants to CATF, by fiscal year of application
• Table 6: number of applications received and amounts requested, by fiscal year of application
• Table 7: number of recipients and total funding received, by fiscal year and discipline
• Table 8: number and ratio of applications to places by year
• Table 9: number and proportion of students and graduates by fiscal year
• Table 10: amount and percentage of funding sources by type (earned, public, private), by fiscal year
• Table 11: number and proportion of recipients' FVA ratings
• Table 12: number and ratio of graduates to awards, by fiscal year
• Table 13: graduates working professionally in Canada and internationally, by year
• Table 14: number and percentage of applications and recipients from underserved communities, by fiscal year of application
• Table 15: program administrative costs, by fiscal year
• Table 16: recommendation 1—action plan
• Table 17: recommendation 2—action plan

List of figures

• Figure 1: CATF-funded Indigenous organizations' applications, graduates and working professionals compared to CATF-funded non-Indigenous organizations
• Figure 2: number of recipients and amounts approved by region, by year of application
• Figure 3: service standard compliance, by fiscal year

List of acronyms and abbreviations

AHAA

Arts and Heritage Access and Availability Survey

CATF

Canada Arts Training Fund

CCA

Canada Council for the Arts

DTRC

Dancer Transition Resource Centre

PCH

Department of Canadian Heritage

ESD

Evaluation Services Directorate

FVA

Financial Viability Analysis

GBA+

Gender-Based Analysis (plus)

G&C

Grants and Contributions

GCIMS

Grants and Contributions Information Management System

GDP

Gross Domestic Product

NATCP

National Arts Training Contribution Program

NRC

National Review Committee

O&M

Operations and Maintenance

OLMC

Official Language Minority Community

PMERS

Performance Measurement, Evaluation and Risk Strategy

PRG

Policy Research Group

TRC

Truth and Reconciliation Commission

Executive summary

This report presents the findings from the evaluation of the Department of Canadian Heritage's (PCH) Canada Arts Training Fund (CATF) Program.

Program description

CATF is delivered by the Arts Branch of PCH and provides financial assistance to Canadian not-for-profit institutions that specialize in delivering focused, intensive and practice-based arts training to prepare students for professional national or international artistic careers. CATF-funded schools offer professional training at the highest level in disciplines such as dance, theatre, circus arts, visual arts, music (opera, orchestral), and Indigenous and ethnocultural artistic practices. Funding for CATF is provided in the form of contributions, which are allocated according to the program Terms and Conditions. Between 2012-13 and 2017-18, an average of 39 professional arts schools or training institutes were funded annually. The amount paid to funded organizations averaged $22.6 million per year for a total of $131 million over the evaluation period.

Evaluation approach

The evaluation covered the period of 2012-13 to part of 2017-18 and as required by the Financial Administration Act and the Treasury Board Policy on Results (2016), assessed the relevance, effectiveness (including government-wide policy considerations) and efficiency of the CATF program.

Findings

Relevance

Over the evaluation period, there was a high demand for CATF as it is the only program to provide cohesive, multi-year operational funding for professional arts training organizations and is therefore, essential to the arts ecology and future sustainability of arts in Canada. The need for the program was further augmented by government commitment to reconciliation with Indigenous peoples as well as the trend toward tuition-free models among some international arts training organizations.

CATF was responsive to the needs of funded arts training organizations. In particular, CATF supported the development of the highest quality training organizations in Canada which produced and prepared high-quality professional artists for their careers. However, due to high program demand and fixed budgets, CATF did not fund all qualified organizations, did not increase funding to all high-performing schools, and decreased funding to lower performing organizations as well as to two national high-performing organizations. Despite these budget constraints, CATF has engaged in other methods to be responsive to program recipients such as experiment project to explore student recruitment and retention issues at indigenous arts training institutions.

CATF aligns with several PCH priorities as well as federal government priorities. Specifically, CATF aligns with the PCH priorities that support accessibility of Canadian cultural content and arts training in Canada. CATF also aligns with the federal government's public recognition of the importance and contribution of artists to Canadian society including legislation that supports the arts.

There were demonstrated benefits for both Canadians and Arts organizations of the federal government's role in this field. In particular, culture and artistic performances have been significant contributors to GDP, generators of economic growth, jobs and wealth in Canadian communities, and contributed to the cultural identity of Canada. Without public funding, some arts training organizations would not have been viable which in turn might have prevented access by Canadians to strong artistic experiences and access by arts organizations to a robust pool of artists within Canada. Some public funding was available for arts organizations provincially/territorially, however, this funding varied across regions and for some provinces/territories, funding was insufficient or not intended for arts training. CATF complemented other federal level funding offered by the Canada Council for the Arts (CCA), as together they provided a full spectrum of support (i.e. from training support from CATF to performing support from CCA).

Effectiveness

Given CATF's fixed budget, the assessment process was highly competitive and ensured only organizations which best exemplified program objectives were funded, regardless of previous contributions. Further, CATF-funded organizations were recognized as being leaders in their field and have provided relevant and applied training to students through high-quality instruction. However, it was identified that the definition of excellence may not have been appropriate for Indigenous art forms.

CATF-funded organizations had diverse revenue streams to support financial stability. Organizations had a combination of public, earned and private funding. While the amount of funding from each source fluctuated, the proportion of each source remained mostly consistent. Importantly, public funding, such as CATF, assisted training organizations to be financially stable. This is particularly true for Indigenous, ethnocultural and racialized organizations who experienced challenges accessing private funding.

CATF-funded organizations produced highly skilled graduates that are recognized in Canada and internationally for excellence. Overall, one in three graduates received an award in recognition of excellence in their artistic craft. However, there were some issues with using the number of awards graduates received as a measure of success, particularly when attempting to compare regional, national and international awards and when examining students from Indigenous training organizations which may not provide awards or consider it an appropriate measure of success.

Canadians valued and appreciated the work of professional artists. Overall, Canadians held positive views about the value of arts and culture for themselves and for society in general. Seven in ten considered the arts of at least moderate importance to their quality of life. There was also widespread agreement (85%) that arts and culture conveyed a variety of societal benefits. Canadians attended performances, volunteered and donated to the arts. Importantly, CATF supported the development of high-quality training institutions and successful professional artists so that Canadians could have access to high-quality artistic performances.

Government-wide policy considerations

CATF met the official language requirements of section 41 of the Official Languages Act and took measures to ensure that both official languages were represented in the funded organizations. Specifically, CATF required funded organizations to have a bilingual website, information material and (with a few appropriate exceptions) audition process.

CATF personnel has applied GBA+ to identify systemic barriers in their terms and conditions and guidelines and has targeted specific minority communities. Following the PCH departmental plan of 2016-17, the Arts Branch wrote a framework in early 2017 to further this work. However, stakeholders identified a need to further examine the systemic barriers built into the program which prevents access for specific segments of the population, particularly for Indigenous arts training organizations.

Efficiency

CATF utilized consistent and rigorous implementation of eligibility and assessment criteria, site visits and expert assessments to ensure resources dedicated to the program were used efficiently. Based on this data, CATF altered program priorities/program delivery as necessary to achieve a greater impact (e.g. reallocation of funds across different categories of recipients; reallocation of funds from underperforming organizations/organizations that did not demonstrate need to those showing consistent or increased level of excellence). Further, CATF met service standards for acknowledging receipt of applications (80% of applications within 10 days). While the program did not meet service standards for notification of the funding decision from 2012-13 to 2016-17 (80% of applications within 29 weeks), preliminary program data for 2017-18 suggests that timelines for notification have significantly improved.

Alternative approaches or innovations to achieve program outcomes were identified throughout the evaluation. Such as:

• A two-streamed funding model with separate requirements; one stream for large national organizations and one for small organizations (including emergent and Indigenous programs).
• Culturally appropriate approaches to funding Indigenous arts organizations and cultures (e.g. conducting consultations with communities regarding appropriate funding programs, providing consistent feedback opportunities and accommodating alternative applications such as verbal applications.
• Shortened Professional Assessors' Report template to increase consistency and utility to the National Review Committee (NRC) (i.e. replace question on uniqueness with assessor's description of ecosystem of the discipline, ask assessor to define excellence and to rank schools in a discipline as appropriate, to detail methodology, and drop questions on administrative stability or that can be answered through applications or final reports).

Recommendations

Recommendation 1

The evaluation found a need for a mechanism that will recognize the different needs and strengths of national arts training organizations, and emergent and Indigenous organizations within a context of Program high demand and fixed budgets.

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs explore the feasibility and benefits and impacts of a two-stream funding model for CATF within the existing budgets.

Recommendation 2

The evaluation found that recruitment among Indigenous arts organizations remains low, and that CATF policies and tools within the assessment process should be responsive to Indigenous cultural needs.

The evaluation recommends that the Senior Assistant Deputy Minister, Cultural Affairs collaborate with Indigenous arts organizations to identify barriers to a successful application, develop options to address these barriers and implement an approach to ultimately increase the number of high-quality graduates.

1. Introduction

This report presents the findings from the evaluation of the Department of Canadian Heritage's (PCH) Canada Arts Training Fund (CATF). CATF is delivered by the Arts Branch of PCH and is focused on supporting organizations which provide training of the highest standard for Canadians seeking to pursue a professional career in the arts. The evaluation of CATF was conducted in accordance with the requirements of the Financial Administration Act and Treasury Board Policy on Results (2016) and is being undertaken in accordance with the 2017-18 to 2021-22 Departmental Evaluation Plan, approved by the Deputy Minister. The evaluation covers the period of 2012-13 through 2017-18 and addresses the core issues of relevance, effectiveness and efficiency.

2. Program profile

This section outlines the CATF program and provides details on its history, activities, objectives and outcomes, management and governance, and resources. CATF provides financial assistance to Canadian not-for-profit institutions that specialize in delivering focused, intensive and practice-based arts training to prepare students for professional national or international artistic careers. CATF-funded schools offer professional training at the highest level in disciplines such as dance, theatre, circus arts, visual arts, music (opera, orchestral), and Indigenous and ethnocultural artistic practices. The program offers two types of support, the Regular Support Component (i.e. annual or multi-year funding for pre-eminent arts training institutions) and the Development Support Component. As noted on the PCH website, the Development Support Component has been inactive since 2013-14.

2.1. Program history

CATF was created in 1997 as the National Arts Training Contribution Program (NATCP) with a budget of $11 million annually. In 2002, the program's scope was expanded to include Indigenous and culturally diverse arts forms and the budget was increased to $17 million. In 2008-09, the program name was changed to CATF and an increase in funding was provided through Canada's Economic Action Plan. The program's annual budget was increased to $24.1 million ($22.8 million in contributions; $1.3 million in Operations and Maintenance [O&M]) and the program was made an ongoing permanent fund. The funding allowed the program to increase its operating support for organizations already receiving funding, and to offer funding to a number of new organizations. Since 2008-09, funding for CATF has remained static.

An evaluation of CATF in 2014 (covering 2007-08 to 2011-12) recommended that: application guidelines be reviewed to increase clarity with respect to the selection process, and this review should articulate how the needs of long-funded institutions could be met while ensuring that emerging institutions demonstrating excellence could be recognized. In 2015, the Program published new guidelines where content was added to describe the function of CATF's National Review Committee (NRC) as well as considerations that impact the application assessment process. Text was added throughout to emphasize that recipient funding levels can fluctuate, and that CATF provides no guarantee of funding support to new or returning applicants.

2.2. Program activities, objectives and expected outcomes

The objective of CATF was laid out in October 2009, guided the program in the September 2016 Performance Measurement, Evaluation and Risk Strategy (PMERS) and remains unchanged in the Performance Information Profile approved 14 November, 2017. The objective is to "Contribute to the development of Canadian creators and future cultural leaders of the Canadian arts sector by supporting the training of artists with high potential through institutions that offer training of the highest calibre." The concept of "highest calibre", or highest quality, denotes a quality of training that produces graduates who work at a high level professionally, with important artistic roles and often in a leadership capacity in their artistic field.^{Footnote 1}

Table 1 provides the Logic Model illustrating the expected results of the CATF program approved in November 2017, but little changed from the PMERS in force during most of the period under evaluation.

Table 1: program logic model

Activities	Provide financial assistance through transfer payments to support nationally significant Canadian institutions that offer arts training of the highest calibre Review and assess applications Develop and distribute operational tools, program guidelines and application materials Recommend funding Manage and monitor funding agreements Develop partnerships/stakeholder relations/outreach activities Report on results
Outputs	Allocation of contributions to eligible recipients
Immediate Result	Nationally significant Canadian institutions offer arts training of the highest calibre
Intermediate Results	Arts training institutions of the highest calibre are financially and administratively stable Graduates are recognized for excellence in Canada and internationally
Ultimate Result	Canadians value the work of Canadian professional artists

The table 2 outlines the program's target populations and key stakeholders.

Table 2: target population and key stakeholders

Target Population	Key Stakeholders
National arts training organizations that deliver professional training programming for young Canadians in preparation for artistic careers, either as their mandate, or a portion of their mandate. Canadian artists seeking training	Canadian arts employers (performing arts companies, professional arts presenters, art galleries and artist-run centres, etc.) Canada Council for the Arts Provincial/territorial and municipal arts funding bodies

To apply to the CATF program, an organization must complete the application package, which includes the application form and various supporting documents such as a list of the current Board of Directors, articles of incorporation, audited financial statements, business or strategic plan, sample program literature, list of staff awards and training schedule, amongst others. The application along with the support documents is submitted electronically through the Canadian Heritage Online System. The Online System has been available for the past four years.

CATF funds projects that have clear objectives and measurable results. The application is reviewed for eligibility and then assessed by the NRC, which compares and prioritizes it in relation to other applications and funds available. The following is taken into consideration when prioritizing eligible applications: artistic merit, statistical reports, impact and institutional stability. Other complementary sources of information are expert assessments (independent experts to conduct on-site assessments of recipients), reporting results (activity reports, financial reports), national delivery (ensuring that organizations have a national reach), official languages, success of graduates, and health of the discipline and program budget.

CATF funds project-related cash expenses such as:

• Artistic and administrative salaries, fees and benefits
• Curriculum development and training delivery
• Audition fees
• Travel
• Marketing and publicity
• Fundraising
• Administrative expenses
• Facility costs such as rent/mortgage
• Professional development for staff

Media arts training is funded to the extent that it relates to artistic practice and focuses on training people who intend to work as artists using media-arts tools. CATF does not fund training for careers in the commercial film and video industry, nor does it fund capital infrastructure. Total assistance from CATF and other levels of government cannot exceed 90% of the total program expenditures. CATF funds the lesser of 70% or up to a maximum of $6 million.

2.3. Program management and governance

Accountability for CATF program lies with the Assistant Deputy Minister, Cultural Affairs sector. The program is overseen by the Arts Branch at PCH Headquarters. The Arts Branch is responsible for policy development, overall program design, management and budget, monitoring and reporting of program performance and results. It is also responsible for the development and distribution of learning/information/support materials, program guidelines and application forms. It ensures coordination between PCH, its portfolio agencies and other partners that support the arts and heritage sectors. PCH's Grants and Contributions Centre of Expertise and the Financial Management Branch are also involved in the development and management of administrative procedures, due diligence, and data collection related to the delivery of grants and contributions.

2.4. Program resources

CATF funding is provided in the form of contributions, which are allocated according to the program Terms and Conditions. Between 2012-13 and 2017-18, CATF funded an average of 39 professional arts schools or training institutes annually. The amount paid to funded organizations averaged $22.6 million per year for a total of $132 million over the evaluation period. Table 3 provides a summary of CATF contributions for each fiscal year from 2012-13 to 2017-18. Funding for CATF has remained static since 2008-09.

Table 3: program resources by fiscal year

Fiscal year	2012-13	2013-14	2014-15	2015-16	2016-17	2017-18	Total
Contributions	$22,450,000	$22,620,000	$22,750,000	$22,719,000	$22,721,000	$22,653,000	$131,913,000

Source: CATF data

CATF signs multi-year funding agreements with applicants who score high on institutional stability and low on risk. On average, about half of CATF recipients were approved for multi-year funding over the evaluation period. From 2012-13 to 2016-17 between 50% and 58% of all agreement were multi-year. In 2017-18, 69% of agreements were multi-year.

3. Approach and methodology

This section outlines the evaluation approach and methodology including scope, timelines, calibration, evaluation questions, data collection methods, limitations and mitigation strategies. The Evaluation Services Directorate (ESD) of PCH led the evaluation of CATF. Components were completed by ESD, the Policy Research Group (PRG), and an external contractor.

3.1. Scope, timeline and quality control

The evaluation covered the period 2012-13 to part of 2017-18. As required by the Financial Administration Act and the Treasury Board Policy on Results (2016), the evaluation assesses the relevance, effectiveness (including government-wide policy considerations) and efficiency of the CATF program.

The key areas of focus for the evaluation included:

• An emphasis on the impact CATF has had over the period of the evaluation, rather than on output measures and design and delivery.
• Identification of barriers to a successful application by Indigenous communities, ethnocultural and racialized communities, and other issues related to these groups.^{Footnote 2}
• How to continue to support the professional arts community in Canada while making space for new or innovative disciplines or projects.

Most of the data-gathering was conducted jointly with the grouped evaluation of Arts Policy programs, currently underway. The Final Report of the grouped evaluation will include data from CATF in its findings on the combined impact of the Arts Policy programs.

The Evaluation Working Group, which included representatives of CATF, met regularly to discuss evaluation issues and ensure the accuracy of the preliminary findings.

3.2. Evaluation questions

Table 4 below outlines the specific evaluation questions by core issue that were used to guide the evaluation, including the development of data collection instruments and the analyses. More details related to the indicators and data collecting methods can be found in the Evaluation Framework (Annex A).

Table 4: evaluation questions by core issue

Relevance

Core Issue	Evaluation questions
Ongoing need for the program	To what extent is there a need to contribute to the development of Canadian creators and future cultural leaders of the Canadian arts sector by supporting the training of artists with high potential through organizations that offer training of the highest calibre? To what extent was CATF responsive to the demonstrated needs of Canadians?
Harmonization with government priorities and PCH	To what extent did CATF align with PCH priorities and federal government priorities?
Harmonization with government roles and responsibilities	Is supporting the training of artists with high potential through organizations that offer training of the highest calibre an appropriate responsibility for the federal government?

Effectiveness

Core Issue	Evaluation questions
Achievement of expected outcomes	To what extent do nationally significant Canadian organizations offer arts training of the highest calibre? Are arts training organizations of the highest calibre financially and administratively stable? Are graduates recognized for excellence in Canada and internationally? Have there been any long-term impacts because of CATF?

Government Wide Policy Considerations

Core Issue	Evaluation questions
Achievement of Government-wide Policy Considerations	Were all official language requirements met? Did the programs have unintended barriers and issues related to gender-based analysis (GBA+)?

Efficiency

Core Issue	Evaluation questions
Demonstration of efficiency	Were the resources dedicated to the program used efficiently to maximize the achievement of outcomes? Are there more efficient alternatives to achieve the same results?

3.3. Data collection methods

A mixed-method approach was utilized for this evaluation including a document review, literature review, administrative data review, key informant interviews, a focus group and a survey. The following provides a description of each of the data collection tools.

3.3.1. Document review

Over 80 documents relevant to CATF program were reviewed. These documents included but were not limited to: federal government policies and guidelines, departmental and program policies, directives, guidelines, Terms and Conditions, integrated business plans, program audits, meeting minutes, partnering agreements, Memoranda of Understanding and communications and outreach products. Government of Canada documents included: Speeches from the Throne, Federal Budgets, Reports on Plans and Priorities and Departmental Performance Reports and Statistics Canada surveys and reports. Also reviewed were CATF Professional Assessors Reports covering 2013-14 to 2017-18.

3.3.2. Literature review

The review included recently published literature, reports, websites, public opinion research and other sources at the national and international level documenting the current context for professional training in the performing arts. To identify and use high-quality research references, the PCH Knowledge Centre was engaged to obtain scientific articles and reports from respected journals through the EBSCO subscription-based academic research database.

3.3.3. Administrative data review

CATF program administrative data was reviewed for the evaluation. This included:

• Data/statistics provided by the Program
• Data entered into the PCH Grants and Contributions Information Management System (GCIMS)
• Financial data provided by Finance
• Service standard compliance data from Grants and Contributions Centre of Expertise reported on the PCH website

3.3.4. Interviews with stakeholders

Interviews were conducted with both internal and external stakeholders. ESD and the Evaluation Working Group identified a list of stakeholders that could respond to questions about the CATF program. Both English and French guides were developed. A total of 3 interviews with PCH personnel and 12 external stakeholder interviews (with a total of 20 interviewees) were conducted. A description of the external interviewees is provided below.

• 11 of the 12 external key informant interviews were conducted with recipients of CATF
• Disciplines included: theatre, dance, multidisciplinary organizations and music (total n=13)
• Regions included: Ontario (n=4), Quebec (n=3), Manitoba (n=2), British Columbia (n=2) and Alberta (n=1)
• Two interviews (four interviewees) were completed with ethnocultural or racialized organizations.
• One interview guide was collectively completed in written form by five interviewees representing three Indigenous organizations. Specifically, the Aboriginal Trainers Caucus provided a collective written response to the CATF interview guide.

3.3.5. Focus group

A one-hour focus group was conducted with 11 representatives of the Canada Council for the Arts (CCA) including directors from various departments within the organization. The questions for the focus group were adapted from the interview guides and focused on overall relevance and efficiency as well as government-wide policy considerations.

3.3.6. Survey of applicants

Online surveys of funding recipients and unsuccessful applicants of the four Arts Branch programs (including CATF) were posted on-line from August 17 to August 31, 2018.

The response rate among CATF funding recipients contacted was 49% (19 responses). The response rate among unsuccessful applicants to CATF was deemed below the threshold required for statistical validity.

3.4. Constraints, limits and mitigation strategies

The following outlines the key constrains and limits of the evaluation process as well as identifies the mitigation strategies utilized to minimize the impact of these limitations:

• Not all administrative data was available for 2017-18 and some multi-year project final reports had not been received at the time of the evaluation.
• Majority of the external interviewees were recipients (11/12). As a result, there is possible positive response bias. To mitigate this issue, the evaluation utilized other lines of evidence (e.g. literature review, document review, survey, etc.) to support findings and themes from the external stakeholder interviews. As an insufficient number of unsuccessful applicants completed the survey to consider the results valid, this data is not included.
• The evaluation did not include consultations (e.g. interviews or surveys) with students or graduates of CATF-funded training schools. Therefore, the data provided for the indicator pertaining to graduate satisfaction is based on key informant interviews with recipients only. This was also a limitation in the previous evaluation of CATF. This issue is mitigated by cautioning the reader in this section as well as in the body of the report that responses regarding graduate satisfaction are observations from a third party. To mitigate this issue in the future, an additional data source (i.e. such as interviews with graduates of CATF-funded organizations) should be considered.

4. Findings

4.1. Relevance

This section provides the evaluation findings regarding the relevance of the CATF program including the ongoing need for the program, harmonization with government priorities and PCH core responsibilities and harmonization with government roles and responsibilities. The key findings have been organized by evaluation question, with supporting themes and evidence provided below each table.

4.1.1. Relevance: ongoing need for the program

High demand for the program

CATF has experienced increased demand in the last several years, both from returning recipients who request increases in funding and from new applicants requesting new funding. The average total amount requested over the past eight years (2009-17) was $27,805,017 (on a $22,779,440 per year allocation), an average of 18% over budget. Further, most applicants to the CATF program were recurrent (165 of 185). In the last six years, between 83% and 93% of arts training schools that applied for CATF funding were recurrent, suggesting little room for new organizations (Table 5).

Table 5: number and percentage of recurrent and new applicants to CATF, by fiscal year of application

Fiscal year	# of recurrent	% of recurrent	# of new	% of new	# of total applicants	% of total applicants
2012-13	35	89.74%	4	10.26%	39	100.00%
2013-14	25	89.29%	3	10.71%	28	100.00%
2014-15	32	91.43%	3	8.57%	35	100.00%
2015-16	31	86.11%	5	13.89%	36	100.00%
2016-17	27	93.10%	2	6.90%	29	100.00%
2017-18	15	83.33%	3	16.67%	18	100.00%
Grand Total	165	89.19%	20	10.81%	185	100.00%

Source: PCH's Grants and Contributions Information Management System (GCIMS)

Only program to provide cohesive, multi-year operational funding for professional arts training organizations

Stakeholders indicated that the demand for the program was high because CATF is the only program to provide cohesive, multi-year operation funding for professional arts training organizations. Therefore, continued investment in producing strong Canadian professional artists is essential for the arts ecology and the future sustainability of arts in Canada. In a pre-budget submission to the federal government in 2018, the Canadian Arts Coalition identified " In light of the doubling of the budget of the Canada Council, the arts sector is poised to significantly increase productivity. This productivity will mean more jobs for curators, actors, musicians, dancers, directors, designers, and managers and more support for independent artists in writing, film, visual, and media arts. But training of emerging artists and managers, especially from marginalized communities, is essential for the ongoing stability and longevity of the field because we have an aging workforce in comparison to the overall Canadian workforce. There are fewer artists than the overall labour force under 25 years of age (12% vs. 14%) but many more artists 55 and over (25% vs. 19%)."^{Footnote 3}

Government commitment to reconciliation with Indigenous peoples has created a need to change how professional arts training is viewed in Canada

Stakeholders identified that the Federal government's commitment to reconciliation with Indigenous peoples has created a need to change how professional arts training is viewed in Canada and to ensure access to the CATF program. The Aboriginal Trainers Caucus indicated that the need for the program is augmented by the Truth and Reconciliation Commission (TRC) Calls to Action as CATF can provide a culturally specific alternative path to full participation in the arts for Indigenous peoples (i.e. enables Canada's Indigenous people to succeed on their terms and according to their aspirations for artistic expression in their communities, on a National platform and around the world).

World class training is becoming increasingly internationalized and some institutions are moving towards tuition-free models to compete with other leading institutions

World class training is becoming increasingly internationalized and institutions are moving towards tuition-free models to compete with other leading institutions. Some world-class training organizations, such as the Banff Centre, are moving to a tuition-free model to compete for talent with other leading international arts institutions such as Juilliard, the Tanglewood Institute, and the Curtis Institute of Music. Fully subsidizing those who are the very best in their fields is becoming the norm for training at the highest levels.^{Footnote 4} Tuition fees for Canadian students of CATF-funded institutions can range from a few hundred dollars for a short-term workshop, to $30,000 per year for full-year programs. On average, funded institutions receive between less than 10% and up to 67% of their training budget through tuition fees charged to students. Financial support is also often provided to students in funded professional training schools. This helps the training organizations attract their countries' best artists and allows the students to focus full-time on refining their skills, regardless of whether they have other sources of income.^{Footnote 5}

At the meeting with Representatives of National Arts Training Institutions, it was noted by the Royal Conservatory of Music that if the best emerging artists cannot access world-class training in Canada, they go elsewhere. When they leave for their professional training, they tend not to come back, resulting in brain drain.^{Footnote 6} Attracting international students is also important as many training institutions consider foreign students vital to the quality of the training they provide. To become world-class artists, they argue, Canadian students should be exposed to and compete with students from other countries. ^{Footnote 7}

Other new conditions which augmented the need for the program

Stakeholders identified other new conditions which augmented the need for the CATF program. In particular:

• The new CCA funding model which came into effect in 2017-18 ceased funding to artists for postgraduate training in music, theatre and dance, thus increasing a need for CATF funding of these programs
• Artists require training in a wider variety of skills to be successful in their careers (e.g. technology, leadership, marketing and social media, finance, etc.)

For funded organizations, CATF contributed to the development of high-quality arts training organizations in Canada

Over the period covered by the evaluation (2012-13 to 2017-18), CATF program received a total of 185 applications that collectively requested approximately $227 million for their professional arts training organizations. The majority of CATF applications (136 out of 185) were approved for funding (Table 6). The percentage of approved applications ranged from 66.7% in 2015-16 to 82.8% in 2016-17.

Table 6: number of applications received and amounts requested, by fiscal year of application

Application Fiscal year	Total Applications (#)	Total Requested ($)	Successful applications (#)	Successful applications (%)	Requested ($) by successful applicants	Rejected applications (#)	Rejected applications (%)	Requested ($) by rejected applicants
2012-13	39	$33.27M	28	71.80%	$31.73M	11	28.20%	$1.54M
2013-14	28	$33.36M	21	75.00%	$32.03M	7	25.00%	$1.33M
2014-15	35	$44.14M	26	74.30%	$43.33M	9	25.70%	$0.81M
2015-16	36	$39.45M	24	66.70%	$34.39M	12	33.30%	$5.06M
2016-17	29	$49.08M	24	82.80%	$47.23M	5	17.20%	$1.85M
2017-18	18	$27.8M	13	72.20%	$25.76M	5	27.80%	$2.04M
Total	185	$227.1M	136	73.50%	$214.47M	49	26.50%	$12.63M

Source: PCH's Grants and Contributions Information Management System (GCIMS)
*Data includes both single year and multi-year applications; the number of organizations funded has remained relatively consistent as seen in table 7.

Between 2012-13 and 2017-18, there were an average of 39 schools funded yearly through the CATF program across a variety of disciplines such as dance, theatre, music, visual arts and multidisciplinary (Table 7).

Table 7: number of recipients and total funding received, by fiscal year and discipline

	Dance	Theatre	Music	Visual arts	Multidisciplinary	Totals by year
2012-13 (#)	14	11	8	3	3	39
2012-13 ($)	$8,735,000	$7,165,000	$3,925,000	$225,000	$2,400,000	$22,450,000
2013-14 (#)	14	11	8	2	4	39
2013-14 ($)	$8,735,000	$7,210,000	$3,950,000	$195,000	$2,530,000	$22,620,000
2014-15 (#)	14	11	8	2	4	39
2014-15 ($)	$8,840,000	$7,210,000	$3,975,000	$195,000	$2,530,000	$22,750,000
2015-16 (#)	13	11	8	2	4	38
2015-16 ($)	$8,746,000	$7,220,000	$3,988,000	$185,000	$2,580,000	$22,719,000
2016-17 (#)	14	11	8	2	4	39
2016-17 ($)	$8,765,000	$7,148,000	$4,038,000	$170,000	$2,600,000	$22,721,000
2017-18 (#)	15	8	9	1	4	37
2017-18 ($)	$8,545,000	$7,030,000	$4,158,000	$120,000	$2,800,000	$22,653,000
Total # (6 years)	84	63	49	12	23	231
Total $ (6 years)	$52,366,000	$42,983,000	$24,034,000	$1,090,000	$15,440,000	$131,913,000

Source: CATF data

For funded organizations, CATF contributed to the development of high-quality artists who are employed and receiving recognition of their work

Stakeholders agreed that CATF program supported the development of high-quality training organizations in Canada and that these organizations produced and prepared high-quality professional artists for their careers. They also identified that several CATF-funded training programs fill a gap between formal training and beginning a career, during which artists learn important applied skills required to navigate the industry and launch a professional career (e.g. tech, finance, career management, etc.).

Certain qualified intuitions were not being funded and some funded organizations experienced a decrease in funding

There was an increase in demand for CATF funding over the evaluation period (demand outweighs supply), and certain qualified institutions were not funded. Overall, the demand on CATF by current recipients and new applicants represented close to $5 million in funding pressure. To respond to this pressure, CATF had increasingly reallocated funding to ensure that the distribution of funding better aligned with the performance and the financial needs of organizations, and that a limited number of new high-performing applicants would have fair access to CATF support. For example, CATF was not funding all qualified organizations, not increasing funding to all high-performing schools, and decreasing funding to lower performing organizations as well as to two national high-performing organizations.^{Footnote 8Footnote 9Footnote 10}

Between 2015-16 to 2017-18, a total of 24 organizations were impacted by fluctuations in funding:

• Five lower performing organizations have been phased out of the program, and another six were decreased with strict conditions related to future funding.
• Eight organizations were increased, based on strong performance and demonstrated financial needs.
• Three new organizations were funded, each at $100,000 of annual support or less.
• Two highly successful organizations were decreased by 2% (or $100k) and 5% (or $300k) based on the assessment of their financial health and equity in funding. ^{Footnote 11}

Further, based on a review of the Records of Discussions and Decisions of the National Review Committee (NRC), there have been instances where organizations received an excellent rating but could not be funded or receive an increase in funding due to budget constraints. For example, an application for a dance school could not be accommodated due to funding constraints: "good proposal, to be revisited if funds allow, no other equivalent dance school available convenient to residents of that province." Other examples included two currently funded ethnocultural arts schools, about whom the NRC noted that both institutions have high-quality instructors and a high-quality program, but due to a fixed budget could not increase their funding.^{Footnote 12}

Stakeholders stated that there is a need for more funding for CATF to support the expansion of currently funded organizations (e.g. community outreach, meeting PCH priorities including export, digital innovation and creative hubs) as well as new organizations and initiatives.

Challenges with competing for CATF funding against Eurocentric arts organizations and program standards

Stakeholders agreed that smaller organizations (which typically include Indigenous, ethnocultural and racialized organizations) were not able to compete with larger, Eurocentric arts organizations for CATF funding. Further, it was identified that there is an ongoing need to increase accessibility to the funding for organizations that offer training outside of Eurocentric art forms (e.g. Indigenous art forms).

CATF definition of excellence^{Footnote 13} may not be appropriate for Indigenous art forms

Stakeholders indicated that there is a need to look at organizational requirements and program criteria to ensure populations are not systematically excluded. The Aboriginal Trainers Caucus identified that Indigenous peoples have different worldviews which inform the way they understand the concept of training. They require relevant delivery systems and reimagining of the conservatory model (i.e. Indigenous learners require a linguistically and culturally appropriate holistic learning environment that meets the individual and collective needs of the community). They also indicated that CATF was designed to support Eurocentric art forms (e.g. ballet, opera, theatre, etc.). Specifically, they feel that the definition of excellence should be revised for Indigenous art forms. The Caucus believes that demand and admissions would increase at Indigenous arts organizations when the students as well as the organizations are properly supported (i.e. delivery capacity, financial support for students, etc.), as is the case for the most prominent schools.

Overall the Caucus felt that CATF had not been responsive to the needs of Indigenous organizations in terms of addressing the expertise and excellence in the field from an Indigenous perspective and providing a program tailored to meet the reality of the lived experience for most Indigenous Canadians.

Provided funding to a dancer transition resource centre to support Canadian dancers in their professional performing career

The Dancer Transition Resource Centre (DTRC) in Canada receives funding from CATF and is a national charitable organization dedicated to helping dancers make necessary transitions into, within, and from professional performing careers. It also operates as a resource centre for the dance community and the public. Grants are available for retraining and subsistence while in full-time training. Psychiatric treatment for dancers is also available. In addition to federal financial support, the DTRC receives funding from private donors, membership fees, dance company contributions and local governments. ^{Footnote 14}

A study in 2006 found that the key challenges of dancers in transition are related to economic, psychological and educational issues and that many forms of support are helpful to dancers in the career transition process, including: financial assistance, emotional support, counselling programs and services, job search preparation, advice and information, and assistance in education and training. Therefore, programs which support dancers can be particularly helpful for their success.^{Footnote 15} A 2016 literature review on dancer transitions included a review of the four main career transition programs or services (i.e. Career Transition for Dancers in the United States, Dancers' Career Development in the United Kingdom, Omscholing Dansers Nederland in the Netherlands and the DTRC in Canada), Canada's appears unique in receiving the most direct public funding. For example, in the Netherlands, public funding is provided to dancers in the form of financial assistance delivered through Omscholing Dansers Nederland and is designed as a top-up of the Netherlands unemployment benefits.^{Footnote 16}

Recruitment challenges for Indigenous training organizations

Indigenous arts training schools were not attracting the same number of students in comparison to other similar schools. The number of applications for Indigenous arts training schools were much lower in comparison with other schools with similar funding. The numbers of graduates were also relatively low compared to the number of applications, and there was an even lower number of working professionals (Figure 1).^{Footnote 17}

Figure 1: Summary of results for 2012-16 (funding from 0 to $500,000) of CATF-funded Indigenous organizations' applications, graduates and working professionals (left) compared to CATF-funded non-Indigenous organizations (right).

The following challenges were identified for Indigenous training organizations in terms of recruitment of students:

• Accredited post-secondary art training programs offered by colleges and universities have not accommodate Indigenous pathways in those programs and young people face barriers accessing them. However, funding is not available from Band Education Councils for Indigenous arts training schools because they are not accredited by a province and therefore not prioritized for funding support.
• Indigenous training schools have limited financial resources for outreach. Although schools have improved their presence on social media, there is limited direct contact with Band Councils.
• There are no feeder programs in comparison to other schools (e.g. there are no First Nation secondary schools with specialized arts programs).
• Low high school graduation rates.^{Footnote 18}

It is noted that in order to respond to the recruitment challenges, CATF proposed an experiment to test if direct contact with Band Councils and communities would increase successful applications to Indigenous arts training schools. The proposal indicated that funding would be provided to all three indigenous training schools currently funded by CATF. The hypothesis is that including direct contact in outreach efforts should lead to an increase of applications to these schools.^{Footnote 19} It is important to note that this project has not started. A counterproposal from Indigenous training schools was received in early July 2018 and is under consideration.^{Footnote 20} As a result of the consultations with these schools, the program is widening the scope of the project to look at not only successful applications but also the retention of students.

Continuing CATF historic funding support while making space for new recipients or innovative projects

CATF program personnel were interested in exploring ways to continue historic funding support while making space for new recipients or innovative projects. The following provides a summary of some suggestions and conclusions from stakeholders, the document review and literature review:

1. Stakeholders indicated a need to examine how to disseminate funding to large versus small organizations while also including new organizations. Some of their suggestions and/or comments included:
   • Comparing national programs with regional programs that are focused on professional development and mentorship is difficult. It could make sense to have two streams: large organization stream and smaller organization stream.
   • The Canadian Arts Coalition proposed an additional 10 million in funding for CATF broken down so that funded small, medium and large organizations can still do their work and there is funding earmarking for new clients.
2. Stakeholders identified that the funding is allocated unequally among the disciplines and a large percentage is provided to dance, particularly ballet. Stakeholders indicated there is a need to develop a strategy to equalize funding across disciplines and organizations, such as providing more funding for visual arts. For example, in 2016-17, 39% of Canadians respondents attended a visual arts exhibit whereas 26% attended a dance performance.^{Footnote 21} Over the evaluation period, visual arts training organizations received just over $1 million in contributions whereas dance training organizations received over $50 million.
3. The literature review identified a study commissioned by PCH and undertaken by Hill Strategies which completed an international comparison of publicly funded arts training programs. The research found "no other funding program of the highest calibre arts training institutions, administered through a national government, with published mandate, guidelines and criteria as well as open application and evaluation cycles." The most similar international program was in Denmark, where the relationship of the arts training institutions and the government is outlined in a specific national Act. As such, the funding is less open and flexible than a program such as CATF. Further, other program models (e.g., Australia, Denmark, Ireland, Singapore, South Korea) offer funding through direct government appropriations where there might be a lack of openness of many support mechanisms to new applicants; limited information on guidelines and processes, which often results in a smaller number of funded institutions. It was concluded that due to similarities, "Denmark and Australia might be countries to monitor for developments or changes in their funding of high-calibre arts training institutions." ^{Footnote 22}
4. The literature review also identified several options for funded organizations to diversify their revenue streams and increase capacity (with some government support) including such methods as crowdfunding,^{Footnote 23} social finance^{Footnote 24} and shared administrative platforms. ^{Footnote 25}

4.1.2. Relevance: harmonization with government priorities and PCH core responsibilities

CATF aligns with the PCH priorities that support accessibility of Canadian cultural content as well as arts training in Canada

CATF contributed to the PCH strategic outcome: "Canadian artistic expressions and cultural content are created and accessible at home and abroad" by funding high-quality organizations which train professional artists who perform and create for Canadian and international audiences. ^{Footnote 26},^{Footnote 27} As mentioned previously, over the period of the evaluation, CATF funded several professional arts schools. Importantly, data demonstrated that graduates were working professionally in their field to perform and create for Canadian and International audiences (average of 66.5% of graduates).

Further, the Department of Canadian Heritage's Creative Canada Policy Framework states, "We know that the economies of the future will rely on creativity and innovation to create jobs and foster growth. To be competitive in the world, we must invest now to create the conditions of success, to develop and keep our talent in both French and English here at home and to make sure we have a robust domestic market for content on which our international success will depend." The framework lists CATF among the current financial support mechanisms in the federal cultural policy toolkit to support this goal. ^{Footnote 28}

CATF aligns with the Federal Government priorities that have publicly recognized the importance and contribution of artists to Canadian Society and the enacted legislation that supports the arts

CATF aligns with several federal government documents which publicly recognize the importance and contributions of artists to Canadian society (e.g. Status of the Artist Act 1992, Budget Plan 2017, Speech from the Throne 2015, Minister Mandate Letter 2015, etc.). ^{Footnote 29}, ^{Footnote 30}, ^{Footnote 31},^{Footnote 32} For example, the Status of the Artist Act recognizes the importance "to Canadian society of conferring on artists a status that reflects their primary role in developing anframework lists CATF d enhancing Canada's artistic and cultural life, and in sustaining Canada's quality of life". ^{Footnote 33} CATF also aligns with other Acts that establish the Government's commitment to the arts sector and demonstrate that Arts is a priority for the Government. For example, the Canada Council for the Arts Act, ^{Footnote 34} the Multiculturalism Act, ^{Footnote 35}and the Canadian Charter of Rights, ^{Footnote 36} all emphasize the importance of Canada's cultural heritage, including the reflection of the Indigenous cultures of Canada.

At a meeting with representatives of national arts training institutions, the then-Deputy Minister of Canadian Heritage laid out ways in which culture and the objective of CATF aligns with federal government priorities (e.g. helps produce jobs and skilled workers to grow the middle class). Reinforced was the contribution that artists make to the creative economy, including the contribution that higher personal incomes for artists can make to growing the middle class and the contribution that student graduates make to the private sector through their skills.

4.1.3. Relevance: harmonization with government roles and responsibilities

Benefits to the Canadian public

There were demonstrated benefits for both Canadians and Arts and Heritage organizations for the Federal Government's role in this field. For Canadians, artistic performances have been a significant contributor to GDP and generators of economic growth, jobs and wealth in Canadian communities. In 2016, the cultural sector contributed over $59 billion to Gross Domestic Product (GDP) and over 700,000 jobs. Music festivals, theatrical works, and dance and music performances draw tourists and help sustain and grow Canada's cultural tourism market.^{Footnote 37} In 2017 alone, over 20 million tourists visited Canada according to Statistics Canada, which generates significant hotel, restaurant and retail sales, and tax revenues.^{Footnote 38} Finally, a vibrant cultural life in a community is an important factor in attracting the knowledgeable workers companies need to be competitive.^{Footnote 39} Further, stakeholders identified that the funding provided by the federal government for the creation of professional artists supported access to strong artistic experiences for Canadians and contributed to the cultural identity of Canada.

Benefits to Arts Organizations

For Arts organizations, arts training programs of the size and quality necessary to be internationally competitive are not sustainable without public funding. In a meeting with representatives of national arts training institutions, it was noted that"… arts training organizations of this size and impact cannot be financially viable without public funding. Even with diverse revenue streams and private sector support, leading arts training organizations need government support to promote stable and sustainable operations."^{Footnote 40} Further, a comparative report of international arts training programs in 2017 illustrated that many arts training programs have been supported by public funding either through federal ministries of culture (e.g., Denmark, France, Netherlands, Italy, Colombia, etc.) or through education departments and higher education funding bodies (e.g., Australia, United Kingdom, Netherlands, Italy, Czech Republic, Ireland, South Korea, Singapore, etc.).^{Footnote 41} Stakeholders indicated that support for creation of professional artists has helped retain a strong pool of artists within Canada for arts organizations to access and set a national/international example that the sector is a valued industry in Canada.

Provincial and territorial funding for arts training varied across regions

Stakeholders identified that while there is some funding available provincially/territorially (e.g. provincial arts councils), the funding to support high-quality training programs has varied across regions and for some provinces/territories, funding was insufficient or not intended for arts training. In fact, the literature review revealed that although the overall objective behind CATF is shared by both provincial and municipal governments across the country, the intervention targets differed. In general, the provincial funding aimed to create a link between the school environment and the arts. For example, several provincial programs offer opportunities for elementary and high school students to experience cultural and artistic experiences, either through integrating arts education activities into the regular school curriculum or supporting organizations to present arts productions in schools ^{Footnote 42},^{Footnote 43},^{Footnote 44} Further, many local arts councils or municipally-run art grant programs across the country offered programs to support the professional development of artists, with a focus on emerging and involved artists in local communities. ^{Footnote 45},^{Footnote 46},^{Footnote 47} These programs, whose availability and objectives varied from one region of the country to another, complemented and ensured the viability of federal programs.

The Aboriginal Trainers Caucus identified that there have been no other significant stakeholders in the arts training field and few resources to support Indigenous students and training organizations in this sector. It was also identified that Indigenous arts students have faced financial barriers and require a level of support to meet program criteria and to assist with living expenses, which would allow them to stay in Indigenous arts training programs. Other stakeholders that could be involved to complement CATF in meeting these needs were identified as Employment and Social Development Canada or First Nations Universities such as Six Nation Polytechnic. A suggestion was that PCH could support pathways to solving this problem in collaboration with the Canada Council for the Arts and the Aboriginal Trainers Caucus.

CATF and CCA were complementary by providing a full spectrum of support, from training to performing

Stakeholders identified that CATF and CCA complemented one another by providing, between them, a full spectrum of support for artists in Canada (i.e. from training to performing). For example, CCA provided financial support for artists as they moved into their profession (e.g. funds the creative process and the presenting of arts), which will benefit those being trained today at CATF-funded institutions.

4.2. Effectiveness: achievement of expected outcomes

This section provides the evaluation findings regarding the effectiveness of the CATF program and specifically, the achievement of expected outcomes.

CATF assessment processes were competitive ensuring only high-quality organizations were funded

As per CATF guidelines, previous CATF funding did not guarantee continued support. All applications were considered by the NRC and the Department based on the information presented. Both long-term funded and emerging organizations demonstrating excellence were considered through the same rigorous assessment process. Some organizations may have seen a reduction of funding from previous years or lost CATF support if they failed to meet CATF's expectations related to the key assessment areas. ^{Footnote 48},^{Footnote 49}

CATF-funded organizations are recognized as leaders in their fields and student demand for the programs are high

CATF-funded training institutions are recognized among employers as the leaders in their fields. For example, the top three training organizations identified by employers that had hired performing artists in dance during the last five years were all CATF‐funded organizations: Canada's National Ballet School, The School of Toronto Dance Theatre and École de danse contemporaine de Montréal.^{Footnote 50}

Stakeholders identified that the demand from students for CATF-funded programs has been high as these programs are recognized as the best training opportunities in Canada across the disciplines, and for some disciplines/organizations the best training opportunities in the world. Between 2012-13 and 2016-17, a total of 58,029 students applied for 16,122 available places in CATF-funded institutions (Table 8). CATF data indicated that the total overall ratio of the number of applications to available places has remained somewhat consistent, between 2.9 in 2012-13 and 3.9 in the 2013-14 and 2014-15 fiscal years. The number of available places decreased by 30%, from a high of 4,079 available places in 2012-13 to a low of 2,684 in 2014-15, however, removing the results of a single institution that offers short-term training, the Banff Centre, reverses this apparent decline.

Table 8: number and ratio of applications to places by year

Fiscal Year	Number of Applications	Number of Places	Ratio of Applications to Places
2012-13	11,987	4,079	2.9
2013-14	12,420	3,151	3.9
2014-15	10,478	2,684	3.9
2015-16	10,473	2,846	3.7
2016-17	12,671	3,362*	3.8
Total	58,029	16,122	3.6

Source: CATF data
*Removing results from a single institution offering short-term training, Banff Centre for the Arts, reverses apparent decline between 2012-13 and 2016-17 in number of places.

Stakeholders indicated that non-funded training institutions were not able to provide the same universal approach to training and focused on one skill rather than a holistic approach, resulting in the quality of the graduate not being as high as graduates from CATF-funded organizations. Some funded schools provide training in a wide variety of skills, including: directing, designing, grant writing, or networking.

Between 2012-13 and 2016-17, a total of 19,522 students were enrolled and 13,100 graduated from CATF-funded institutions (Table 9). That represented an average 67% graduation rate. The proportion of graduates were somewhat stable across the years, ranging from 63% in 2014-15 to 71% in 2014-15.

Table 9: number and proportion of students and graduates by fiscal year

Fiscal Year	Number of students enrolled	Number of graduates	Percentage of enrolled students that graduated
2012-13	5,189	3,676	71%
2013-14	4,023	2,670	66%
2014-15	3,455	2,176	63%
2015-16	3,157	2,070	66%
2016-17	3,698	2,508*	68%
Total	19,522	13,100	67%

Source: CATF data
*Removing results from a single institution offering short-term training, Banff Centre for the Arts, reverses apparent decline between 2012-13 and 2016-17 number of graduates.

CATF instructors and artistic directors were active in their careers and have received several awards and provided relevant and applied training to students

Stakeholders indicated that instructors and artistic directors at CATF-funded organizations were still active in their careers (i.e. performing and directing nationally and internationally) and have provided students with highly relevant and applied training and resources. Between 2012-13 and 2015-16, a total of 1,300 awards were received by instructors of funded training institutions. The number of awards ranged from a low of 269 in 2015-16 to over 300 in other years. No further analysis was possible due the unavailable data on total number of instructors in funded schools.

Nearly three quarters of surveyed recipients agreed "to a great extent" that CATF funding had: raised the quality of certain aspects of their training

74% of respondents to the survey of recipients agreed "to a great extent" that CATF funding had: contributed to improving/updating their arts training curriculum, and increased the scope and depth of graduates' skills. Nearly as many (68%) agreed to a great extent that it had increased the quality of arts professional training. Only 32% stated that CATF funding contributed "to a great extent" to diversifying training delivery modes or locations, though 42% thought CATF contributed "to a moderate extent" to this objective.

CATF-funded organizations have diverse revenue streams to support financial stability

Stakeholders indicated that having a balance of diverse revenue sources contributed to the financial stability of CATF-funded organizations. Further, it was identified that public funding such as CATF assisted organizations to be financially stable. Between 2012-13 and 2016-17, revenues of CATF-funded training institutions were 50.4% public, 30.2% earned, and 19% private sector (Table 10).

Table 10: amount and percentage of funding sources by type (earned, public, private), by fiscal year

	2012-13 ($)	2012- 13 (%)	2013-14 ($)	2013-14 (%)	2014-15 ($)	2014-15 (%)	2015-16 ($)	2015-16 (%)	2016-17 ($)	2016- 17 (%)
Earned Revenue	$28,429,068	33%	$30,257,334	32%	$32,428,954	33%	$29,348,394	31%	$23,973,911	22%
Public Funding	$28,629,801	53%	$28,166,758	52%	$27,370,263	50%	$26,851,427	48%	$30,804,547	49%
Private Funding	$12,707,187	14%	$14,036,422	16%	$14,281,162	18%	$15,205,300	18%	$23,672,667	29%

Source: CATF data

Stakeholders reported public funding such as: federal, provincial/territorial and municipal funding; earned revenue such as student fees, fundraisers and performing/ticket sales; and private funding such as private donations, corporate sponsorships and foundations. They also identified that the amount of funding from each source has fluctuated (e.g. changes in government or changes in private support levels), but the proportion of each source remained mostly consistent.

Between 2012-13 and 2016-17, 87% of recipients had either "favourable" or "acceptable" Financial Viability Analysis ratings

According to internal stakeholders, the Financial Viability Analysis (FVA) is an internal PCH tool for ensuring the financial stability of an applicant. The Centre of Expertise used the five-point financial analysis tool to assess the audited financial statement of the recipients to provide one of three FVA ratings (unfavourable, acceptable or favourable). The target is to have 80% of recipients with an acceptable or higher FVA rating. FVA was only applied for applications greater than $100K.

Between 2012-13 and 2016-17, recipients that underwent the five-point financial analysis met the target of 80% having an acceptable or higher FVA (Table 11).

Table 11: number and proportion of recipients' FVA ratings

Fiscal year	# of Acceptable or higher	% of Acceptable or higher	# of Unfavourable	% of Unfavourable	Total #	Total %
2012-13	16	84.21%	3	15.79%	19	100.00%
2013-14	17	94.44%	1	5.56%	18	100.00%
2014-15	21	91.30%	2	8.70%	23	100.00%
2015-16	19	90.48%	2	9.52%	21	100.00%
2016-17	16	84.21%	3	15.79%	19	100.00%
2017-18*	9	75.00%	3	25.00%	12	100.00%
Average	16.3	86.61%	2.3	13.39%	18.67	100.00%

Source: PCH's Grants and Contributions Information Management System (GCIMS)
*Data from 2017-18 may be incomplete at the time of the evaluation

Several factors can impact CATF-funded organizations' financial and administrative stability

Stakeholders identified several factors that can impact funded organizations' financial and administrative stability. Specifically:

• Fixed or reduced amount of CATF funding to existing organizations impacts the financial stability of the organizations, as CATF often represents a large portion of recipients public funding or overall funding
• A need for the funded organizations to expand and grow (e.g. e.g. community outreach, meeting PCH priorities including export, digital innovation and creative hubs, etc.)
• Inflation and cost of living impacts financial stability of the organizations on a yearly basis; problematic in terms of increasing operating costs (e.g. leasing and renting) and increasing financial support required by students (e.g. scholarships, bursaries, etc.). Some schools in the United States and Europe are offering free tuition to recruit the best students (i.e. it may become more difficult to compete for the best students in the future).
• Lack of access to a stable and suitable space where administrative personnel can work, and students can also train/showcase their work.

Indigenous, ethnocultural and racialized organizations experienced challenges accessing private funding

According to stakeholders, Indigenous, ethnocultural and racialized organizations have found it more difficult to attain private funding compared to Eurocentric art organizations and did not have the capacity and staff to dedicate to marketing and promotion. The administrative data showed that few (18%) funded training institutions had more than 70% of public funding in the last 5 years. Specifically, between 2012-13 and 2016-17, the number of training institutions with more than 70% of public funding varied between six and eight schools and represented mostly schools providing training in Indigenous or culturally diverse art forms. Although CATF personnel appropriately managed the 70% rule in case of Indigenous and priority groups, stakeholders believe this challenged access to the program.

Indicators were strong to demonstrate financial stability of funded organizations but not administrative stability

An expert assessment is undertaken of each CATF recipient ideally every four years. The assessors are "highly reputable individuals with an active career in a performing arts discipline." The role of the expert assessment is to supplement the evaluation of applicants to CATF undertaken annually by the NRC. Specifically, "the purpose of the artistic assessment is to provide first-hand observation and analysis of the training program… The report focusses on areas regarding artistic standards for training in the discipline and upon the impact and role of training institutions in the discipline."^{Footnote 51}

A review of 19 Assessors' Reports dating from 2013-14 to 2017-18 identified that different measures of administrative stability are needed as assessors often provided narratives regarding administrative stability which did not reflect comparable indicators, leading to the Institutional Stability section being identified as the weakest section. It was not apparent that the Assessors were all qualified or comfortable answering the "how well run, is it?" sub-question. While there is nothing to replace the value of a site visit to observe the harmony and focus on goals administrative efficiency should bring, the annual collection, by recipients, of data to answer standard non-profit performance measures for administrative stability might be a better source of data for some of these sub-questions.^{Footnote 52}

Only 10% of surveyed recipients stated that the absence of CATF funding would have endangered the survival of their program or school

68% of surveyed recipients stated that, in the absence of CATF funding, their educational activities would have gone ahead but on a reduced scale. Those who chose "other" impacts for the absence of CATF funding, a further 16%, wrote descriptions of reduced activity or reduced national reach. 10% stated that the activities would not have taken place or the organizations would have ceased operations. This suggests that a majority of recipients are financially stable.

CATF-funded organizations were recognized by industry as producing highly skilled graduates and as leaders in their field

CATF-funded organizations were recognized by industry as leaders in their field. In 2015, 93% of employers rated graduates of CATF-funded organizations as being better prepared for professional careers. Specifically, positive assessments were given of artists' technical expertise in their discipline (95%), performance qualifications and experience (94%) and professionalism and career readiness (90%).^{Footnote 53} Similar positive assessments were expressed in the 2009 employer survey. ^{Footnote 54} Stakeholders confirmed that many of CATF-funded organizations have had a good industry reputation in Canada as well as internationally, in terms of producing highly skilled and employable artists.

Graduates from CATF-funded training organizations have received regional, national and international awards and majority of graduates are working professionally

Between 2012-13 and 2016-17, there were 13,100 graduates from CATF-funded training programs, 8,705 of these graduates were working professionally in their field (average of 66.5%) and 4,952 had received awards (1 in every 3 grads) (Table 12 and 13).

Table 12: number and ratio of graduates to awards, by fiscal year

Fiscal Year	Number of Graduates	Number of Awards	Ratio of Graduates to Awards
2012-13	3,676	1,109	3.3
2013-14	2,670	1,144	2.3
2014-15	2,176	1,022	2.1
2015-16	2,070	957	2.2
2016-17	2,508	720	3
Total	13,100	4,952	3

Source: CATF data
* In 2016-17, Banff Centre reported a significant drop in employed graduates (70% decrease over 2015-16) impacting the overall average. Without Banff numbers, the employment rate among graduates in 2016-17 is 78%.

Table 13: graduates working professionally in Canada and internationally, by year

Fiscal year	Number of Graduates Working Professionally	Graduates with international career based in Canada (#)	Graduates with international career based in Canada (%)	Graduates with international career residing outside of Canada (#)	Graduates with international career residing outside of Canada (%)
2012-13	2,924	1,053	36%	713	24.4%
2013-14	1,489	172	12%	94	6.3%
2014-15	1,818	197	11%	177	9.7%
2015-16	1,727	170	10%	170	9.8%
2016-17	2,183	182	8%	136	6.2%
Total	10,141	1,774	17%	1,290	12.7%

*The new Annual Statistical Report template changed after 2012-13 which account for large variation in numbers.

During the evaluation period, several alumni and students from CATF-funded national training schools received recognition nationally and internationally for their contributions to the arts. For example, the Governor General's Performing Arts Awards were given to graduates of CATF-funded training organizations for their work in music as well as acting, directing and producing between the years 2012 and 2018. Further, in 2017, an 18-year-old piano student was nominated for a JUNO Award while another 15-year-old piano student was chosen by CBC Music as Canada's next piano superstar. In the same year, a 13-year-old student violinist earned a Jury Prize at the International Jascha Heifetz Competition for Violinists.^{Footnote 55}

Graduates were likely satisfied with their professional training as they gained employment and continued to engage or find employment with their training organizations

Recipients of the CATF program indicated that graduates were likely satisfied with their professional training as they gained employment. As well, many of the graduates continued to engage or find employment with their training organizations, suggesting a personal connection with the program and satisfaction. As mentioned in the limitations, graduates of CATF-funded training organizations were not contacted as part of the evaluation. Some professional training schools indicated that they regularly survey their students and receive positive feedback as well as suggestions on how to improve their training program. Funded organizations could be required to periodically survey their graduates and provide a report to CATF to better respond to this indicator.

Challenges with using the number of awards graduates receive as a measure of success

Graduates from funded organizations have received regional, national and international awards which were submitted to PCH for their review. However, there were some challenges with comparing national and community-level prizes as well as comparing Eurocentric arts awards with other arts that may not recognize awards. Specifically, one stakeholder identified that it was challenging to compare national and community-level prizes when looking at the number of awards achieved by graduates. The Aboriginal Trainers Caucus identified that there may not be "recognized awards" for Indigenous customary practices (e.g. pow wow or storyweaving) as awards are not typically how success is measured in Indigenous communities. They suggest that other measures of quality and more appropriate metrics are needed.

90% of surveyed recipients agreed that CATF funding had a great or moderate impact on the recognition of excellence among their graduates. 95% agreed with the impact on their graduates' subsequent professional careers.

Canadians value the work of professional artists

The Arts and Heritage Access and Availability Survey (AHAA) is a cyclical public opinion survey conducted by PCH and CCA. The survey measures Canadians' recognition of the arts, and attendance at arts and cultural events in Canada. In 2016-17, 2,045 Canadians aged 16 and over were surveyed from December 19, 2016 to January 29, 2017. Some of the key findings included:

Canadian Perception and Appreciation:

• Widespread agreement (85%) among Canadians that arts and culture convey a variety of societal benefits.
• Seven in ten (69%) Canadians considered arts and cultural events to be either very (27%) or moderately important (42%) to their own and their family's quality of life. The importance attributed to the arts has remained similar to 2012 (66% very or somewhat important).

Canadian Attendance and Participation:

• Canadians attended performances (88% in 2016-17), donated to the arts (31% in 2016-17) and volunteered (15% in 2016-17).
• The most popular types of events attended were music performances (61%). Other results found that 41% of Canadians attended a Theatre performance, 39% attended a visual arts exhibit, and 26% attended a dance performance.
• Among Canadians who attended at least one arts event in the past year, the most popular venues were the outdoors (66%) and performing arts facilities (60%).^{Footnote 56}

According to stakeholders, Canadians took pride in the achievement of Canadian artists and wanted to see work by Canadians and not just Americans and other international artists. Further, it was identified that Canadians treasured and celebrated Canadian identity, of which artists are a foundation.

68% of surveyed recipients agreed that Canadians value professional artistic experiences "to a great extent", and 32% felt Canadians value these experiences "to a moderate extent."

CATF supported the development of high-quality training institutions and successful professional artists

CATF funded an average of 14 dance schools per year for an average of $8.7 million. Theatre schools followed with an average of 11 per year and about $7 million. Music schools accounted for 8 per year for an average of $4 million, followed by multidisciplinary and visual arts schools with an average of four per year and $2.5 million and two per year for an average of $180,000 respectively. The distribution of funded projects by region over the period of the evaluation is shown in Figure 2. About half of CATF recipients resided in Ontario (63), followed by Quebec (38), and then Western region (30). Prairies and Northern and Atlantic regions accounted for the smallest number of recipients with four and one respectively. Ontario received the largest amount of funding over the last five years ($78.8 million). Quebec received the second-largest amount ($44.1 million), followed by Western Region with$20 million.

Figure 2: number of recipients and amounts approved by region, by year of application

The survey of performing arts employers confirmed that CATF-funded training institutions were recognized as leaders in their fields, with industry identifying CATF-funded organizations as some of the best training institutes in various disciplines and having graduates that were better prepared for professional careers as opposed to graduates from other programs.^{Footnote 57} Stakeholders indicated that by supporting the development of high-quality training institutions, CATF increased the number of highly trained Canadian artists practicing their craft and successfully employed in Canada and abroad. In fact, during the evaluation period, most graduates from CATF-funded institutions (66.5%) were working professionally in their field and one in three had received an award.

Improvements in the recognition and appreciation of Indigenous artists in Canada, despite differential funding and developmental opportunities

The Aboriginal Trainers Caucus identified that Indigenous artists have now been featured in several major Canadian cities and sought after by mainstream institutions looking to develop a stream of Indigenous inclusion into their programming and work. This has been achieved despite differential funding and developmental opportunities. The Caucus also stated that the arts sector has expanded Indigenous inclusion on culturally and historically appropriate terms. Arts organizations are becoming a repository of Indigenous knowledge that can be transmitted intergenerationally in a sustainable way.

4.3. Government-wide policy considerations

This section provides the evaluation findings regarding the effectiveness of CATF program in meeting government-wide policy considerations.

CATF meets the official language requirements of section 41 of the Official Languages Act and has taken measures to ensure that both official languages are represented in the funded organizations

Part VII (Section 41) of the Official Languages Act mandates states that " the Government of Canada is committed to enhancing the vitality of the English and French linguistic minority communities in Canada and supporting and assisting their development … [and] encourage and cooperate with the business community, labour organizations, voluntary organizations and other organizations or institutions to provide services in both English and French and to foster the recognition and use of those languages."^{Footnote 58}

According to CATF application guidelines, "arts organizations must be accessible to Canadians through a national competitive admission process available in both of Canada's official languages, including a bilingual website and information material, and a bilingual audition process." The application guidelines also identify that there is an Official Languages component to the assessment of each organization applying for CATF funding, and the NRC considers the needs of students from both official language groups when making funding recommendations.^{Footnote 59} External and internal stakeholders confirmed that the CATF program required applications, websites, auditions and other material from funded organizations to be offered in both French and English and that these were monitored by CATF program personnel. As an example of a CATF-funded organization supporting the Official Languages Act and the enhancement of minority official language communities in Canada, the Banff Centre Curriculum outlined the French-language training offered to francophone theatres outside of Quebec. Specifically, the curriculum states: "The Stage intensif de perfectionnement is a successful partnership between Banff, the Association des Théâtre Francophones du Canada and the É cole National De Théâtre du Canada. It is designed to provide sustained training needs of French-language theatre practitioners from outside Quebec. Program participants collaborate with other theatre artists working in French. Their learning is enhanced by group and individual meetings, lectures and workshops. The program is considered one of the fundamental career building blocks for actors, directors and designers in French language theatre."^{Footnote 60}

Between 2012-13 and 2017-18, only one organization from an Official Language Minority Community (OLMC) applied for CATF funding and was found to be ineligible, as fewer than 20% of students came from outside its region.

CATF has targeted specific minority communities as well as instructed personnel to apply GBA+ to identify systemic barriers in their terms and conditions and guidelines

In 2002-03, the program's scope was expanded to target Indigenous and culturally diverse art forms.^{Footnote 61} Between 2012-13 and 2017-18, an average of 35% of all applications received were from schools identified with Indigenous, official language minority or ethnocultural or racialized groups (Table 14). A similar average of 36% of total successful applications came from these underserved communities, suggesting that being a member of an underserved community did not put applicants at a disadvantage in competing with others for funding from CATF. These results were consistent when the data was stratified, and each priority group was examined separately.

Table 14: number and percentage of applications and recipients from underserved communities, by fiscal year of application

Application Fiscal Year	Total Applications (#)	Total Applications from Priority Groups (#)	Priority Groups as % of Total Applications	Total Recipients (#)	Successful Applications from Priority Group (#)	Successful Priority Groups as % of Total Recipients	Requested ($) by Priority Group Recipients	Rejected Applications from Priority Groups (#)
2012-13	39	17	43.59%	28	13	46.43%	$3,379,000.00	4
2013-14	28	8	28.57%	21	6	28.57%	$1,435,360.00	2
2014-15	35	17	48.57%	26	12	46.15%	$4,760,000.00	5
2015-16	36	10	27.78%	24	9	37.50%	$3,345,694.00	1
2016-17	29	8	27.59%	24	7	29.17%	$2,735,000.00	1
2017-18	18	5	27.78%	13	2	15.38%	$1,590,000.00	3
Total	185	65	35.14%	136	49	36.03%	$17,245,054.00	16

Source: GCIMS

Further, there were several documents outlining PCH's commitment to GBA+ including the PCH Departmental Plan 2018-19 for GBA+ and the 2017 Plan for the Development of a GBA+ Framework. The departmental plan outlined corporate initiatives underway to respond to the Government-wide call for action on GBA+:

• The Department has initiated a coordinated policy, planning and research approach to examine GBA+ from a variety of perspectives, including policy considerations, budget requests, data availability, and reporting opportunities.
• The Department's Centre of responsibility for GBA+ will continue to ensure that GBA+ is adequately considered in new policy proposals to Cabinet and funding requests. It will also develop a Diversity and Inclusion Training Toolkit, and continue running its community of practice that includes representatives from all branches of the Department^{Footnote 62Footnote 63}.

Overall, the Department has recognized "a holistic approach is recommended to obtain rigorous, sustained and consistent results." The Department guided its employees to ask: "do programs have the same impact on women and men, Indigenous communities, ethnocultural communities, visible minorities, youth, official language minority communities?" The Department also examined what elements of the program guidelines could indirectly constitute systemic barriers based on the principles of GBA+. Employees were directed to examine the terms and conditions and guidelines of programs, including CATF.^{Footnote 64}

Some CATF-funded organizations created community programs and initiatives to diversify their schools and expand their reach

Stakeholders identified that certain funded organizations created new community programs and initiatives with local organizations and communities to help diversify their organizations and expand their reach. Diversifying and expanding reach with local organizations and communities is consistent with underlying GBA+ principles of tackling inequalities to contribute to an inclusive and democratic society.^{Footnote 65} 2016 and 2017 saw two opinion pieces warning of tokenism: the CEO of CCA identified in his letter "State of the Arts in Canada" that perceived tokenism must be reduced to maximize the incredible potential of Canada's diverse artists.^{Footnote 66} These are opinions to consider when examining how funded organizations are meeting the requirements and outcomes of GBA+.

Opportunity for additional GBA+ analysis

Stakeholders indicated that there are ongoing arts training access issues for most of the populations and organizations listed under GBA+ (e.g. women, Indigenous populations; ethnocultural or racialized communities; persons with disabilities; lesbian, gay, bisexual, transgender, transsexual, queer, questioning and two-spirited community; rural communities, etc.). Some examples provided included:

• Culturally inappropriate criteria and finance challenges for Indigenous populations
• Lack of women in high-level executive positions
• Building and arts programming accessibility issues for persons with disabilities^{Footnote 67}

While CATF conducted some GBA+ analysis, stakeholders said that Indigenous organizations still do not meet the Eurocentric definitions and terms and conditions of CATF program. It was identified that there is a need to further examine the systemic barriers built into the program which prevents access for specific segments of the population.

Indigenous arts organizations faced several key barriers in the arts training sector including fragmented government support, lack of capacity and Eurocentric standards and governance models

The literature review revealed several barriers Indigenous organizations face in the arts training sector: fragmented government support (e.g. multiple grants) leading to lack of stability, lack of capacity for human resource management, lack of facilities, absence of a national touring network for collaboration, lack of acknowledgment and valuing of Indigenous cultural knowledge in mainstream funding systems, lack of recognition by government funding agencies of the unique needs of the Indigenous performing arts sector or the unique skills of cultural managers, lack of financial resources, lack of accreditation among schools offering full-time programs to Indigenous performing artists (i.e. can't compete with mainstream schools), and western standards of the arts and western governance models being ill adapted to Indigenous cultures and practices.^{Footnote 68} Further, as mentioned previously, stakeholders identified that Indigenous organizations found it more difficult to attain private funding. They also indicated that this may challenge access to CATF funding due to program terms and conditions requiring multiple funding sources and no more than 70% in public funding contributions.

The Aboriginal Trainer's Caucus stated that child care, parental care and extended family care impact Indigenous women at all stages of arts training and career development. Currently, Indigenous women and grandmothers continue to fulfill their historic roles of education and intergenerational cultural transmission of knowledge, without any institutional supports. There is a need to accommodate these costs in budgets.

Ethnocultural and racialized organizations experienced many barriers in the arts training sector including financial and capacity issues as well as a lack of visibility in the arts sector

The literature review revealed several barriers that ethnocultural and racialized organizations faced in the arts training sector. Specifically, financial needs (e.g. lack of access to unrestricted, multiyear funding that is not project based); organizational capacity needs (e.g. lack of capital for staffing, marketing, succession planning, space, etc.); lack of access to leadership or mentorship opportunities, and lack of visibility in Canada of works by ethno-cultural artists. ^{Footnote 69}

4.4. Efficiency: demonstration of efficiency

This section provides the evaluation findings regarding the efficiency of CATF program including more efficient alternatives to achieve the same results.

Consistent and rigorous implementation of eligibility and assessment criteria, site visits and expert assessments were used to validate or alter funding priorities

CATF undertook consistent and rigorous implementation of eligibility and assessment criteria, site visits and expert assessments to validate funding priorities. According to the NRC meeting minutes, other sources of data informed funding decisions, including: recipient applications and reporting, Regional and CCA input and/or public opinion surveys.^{Footnote 70} Using this information, CATF altered program priorities and/or program delivery as necessary to achieve the highest impact (e.g. reallocation of funds across different categories of recipients, reallocation of funds from underperforming organizations/organizations that did not demonstrate need to those showing consistent or increased level of excellence; using professional assessments to create new program policy statements to better meet the needs of Indigenous communities and professional community-based arts facilitation).^{Footnote 71},^{Footnote 72} Regarding the latter example, it was identified that a professional assessment led directly to a new program policy. Specifically, the assessment stated: "This program would prepare them for a professional community-engaged multi-arts career, much sought after by Aboriginal students who wish to train and then return to their communities" which led to a policy statement in the NRC Minutes stating: "CATF accepts that careers in community-based arts facilitation are significant and a legitimate sub-sector of the arts in Aboriginal Canada." ^{Footnote 73}

Measures to ensure program compliance

CATF undertook various measures to ensure program compliance, such as regular communication of reporting requirements to small and emerging training organizations with low administrative capacity to ensure all organizations had an equal understanding of requirements, deadlines and application procedures. The team also monitored activities through site visits and risk-based recipient audits to minimize the risk of inappropriate use of funding.^{Footnote 74}

Average administrative costs lower than previous evaluation period

The administrative costs incurred by PCH for the management and delivery of CATF program totaled $5,365,176 for the period 2012-13 to 2017-18 (Table 15). This represents a ratio of 3.8% of total CATF expenditures during this period. CATF administrative costs were lower than reported in the previous 2014 evaluation (5.6%).

Table 15: program administrative costs, by fiscal year

Resources	2012-13	2013-14	2014-15	2015-16	2016-17	2017-18	Total
O&M expenditures	749,048	1,077,185	923,739	832,916	879,314	902,975	5,365,176
G&C expenditures	22,720,000	22,620,000	22,750,000	22,719,000	22,721,000	22,879,000	136,409,000
Total expenditures	23,469,048	23,697,185	23,673,739	23,551,916	23,600,314	23,781,974	141,774,176
Administrative ratio	3.2%	4.5%	3.9%	3.5%	3.7%	3.8%	3.8%

Source: PCH's Finance data

Funded organizations reported using the money efficiently to maximize their training programs and are using CATF to leverage funding from other sources to diversify their revenues

Stakeholders indicated that funded organizations are using CATF contributions efficiently to fund their training programs and are often leveraging the funding to diversify their revenues. However, they also identified that regardless of efficient use of resources, more funding is needed for organizations to grow and for new organizations to receive CATF funding. As mentioned previously, CATF's budget has been fixed since 2008-09, which limited its capacity to provide funding to new eligible applicants or increase the level of funding to strong, currently funded organizations.^{Footnote 75}

Onerous application and reporting processes

According to stakeholders, applications and reporting requirements were onerous and time consuming, particularly for smaller organizations. While multi-year funding helped to mitigate this issue, not all organizations were eligible as multi-year funding agreements were signed with applicants with high scores for institutional stability and low risk scores. On average, about half of CATF recipients were approved for multi-year funding over the evaluation period, from a high of 16 in 2012–2013 to a low of 13 in 2016-17. Only 3 multiyear funding agreements were signed in 2017-18 but data may be incomplete. Further, stakeholders identified that budget reporting requirements were still too onerous, despite the multi-year model.

CATF met service standards for acknowledging receipt of applications but not for notification of the funding decision

CATF program has established a service standard for each of an acknowledgement of receipt of an application, and a notification of the funding decision. The standard was 15 days for the acknowledgement of receipt of an application, and 29 weeks for notification of the funding decision. PCH's goal was that 80% of the applications received by the Department would meet the published service standards for each program and/or program component. The service standard for acknowledging receipt of applications was met to a high degree (83%-100%) by CATF in the five years covered by evaluation period and published on-line, 2012-13 to 2016-17 (Figure 3). However, CATF did not meet its service standard for notification of the funding decision between 2012-13 and 2016-17.

Figure 3: service standard compliance, by fiscal year

As demonstrated in the graph above, performance in meeting funding decision service standards declined significantly from 69% in 2012-13 to 15% and 21% in 2013-14 and 2014-15. Prior to 2015-16, data was captured through a different system and is not comparable. In 2015-16, CATF experienced delays in the approval of funding during the fiscal year covered by the change in government with a compliance rate of 47%. Compliance improved again to 56% in 2016-17. Encouragingly, the 2017-18 preliminary results indicate a significant improvement in compliance. According to the Grants and Contributions Centre of Expertise, the average number of weeks to process a CATF application in 2017-18 was 24.7 weeks, as compared to 28 weeks in 2015-16. Due to this improvement, in consultation with the program, the service standard for notification of funding decision was reduced, effective October 1st, 2017 to 28 weeks from 29 weeks.

New funding strategies

Stakeholders agreed that there is an ongoing need to develop new strategies to disseminate funding to large versus small organizations while also including new organizations. One suggestion was to have a two-stream approach where small and large organizations are assessed and funded separately. Without new money, there may be a tension between different stakeholders' views of what this could accomplish. Some stakeholders wish to reduce the struggle for small or new or Indigenous programs to compete with prominent large organizations; other stakeholders hope to reduce the recent funding reallocations from large organizations to find funding for new or smaller programs.

Culturally appropriate approaches to funding Indigenous arts organizations and cultures

The Aboriginal Trainer's Caucus identified that the models used by the Ontario and Quebec Arts Councils should be reviewed and considered for this program to improve transparency in the terms of conditions each organization is expected to meet. The literature review revealed that they have developed specific funding opportunities for Indigenous clients which focus on Indigenous culture and way of life as well as access and equity to their programs. During design and delivery, the councils utilized approaches such as conducting consultations with communities regarding appropriate funding programs, providing consistent feedback opportunities and accommodating alternative applications (e.g. verbal applications).^{Footnote 76},^{Footnote 77} Two other surveyed respondents also suggested culturally assessment criteria to increase number of Indigenous clients.

Addition of new performance indicators for CATF-funded organizations and professional assessors to provide more robust reporting

1. Provincial Ministries of Education require standard performance indicators which allow comparison of courses of different lengths. A suggestion to increase the comparability of programs funded by CATF would be to introduce a measure common in education such as: hours of instruction.
2. The evaluation found "administrative stability" to be an appropriate measure of non-profit CATF-funded organizations. It suggested that CATF require recipients to report on one or two indicators of administrative stability common among non-profits such as: staff turnover and Board turnover.
3. The evaluation found that the current performance measure of satisfaction among graduates is an appropriate measure of CATF-funded schools and programs. It is suggested that CATF require recipients to periodically survey their graduates on their satisfaction with their professional training and subsequent employment status and provide the report to the Program.
4. There was strong evidence that the periodic reports produced for CATF by prominent independent professionals in the discipline are useful and even decisive in assisting the NRC to allocate scarce resources to the highest quality programs and to refine program policy. The NRC was also critical of assessors' conclusions, when appropriate.^{Footnote 78} However, some changes are suggested to shorten the report template and to increase the utility of the reports to the NRC when making funding decisions. These include:
   • Replace question on uniqueness with assessor's description of ecosystem of the discipline
   • Ask assessor to define excellence and to rank schools in a discipline as appropriate, and to detail methodology
   • Drop questions on administrative stability or that can be answered through recipients' applications or final reports^{Footnote 79}

Addressing interdisciplinarity and new technologies

Most arts funding programs across Canadian provinces, territories and municipalities have been organized around selected areas of focus. For example, organizations may provide project and operating grants in visual arts, performative arts, dance, literature, music, etc. This current way of working, however, has limits, as interdisciplinarity and new technologies have disrupted these borders and increase needs.^{Footnote 80}, ^{Footnote 81} For example, the Manitoba Arts Council (MAC) had been adding new programs each year to keep up with these new realities, ending up in 2016 with more than fifty distinctive programs. The situation had become complex for both artists and organizations seeking funding, and it was cumbersome and costly for the administration. Based on these realities, the MAC decided to reorganize its program structure into five broad categories: create, share, support, learn, and Aboriginal arts. The MAC also created a plan to use new information technologies to facilitate application processes, communication with artists in the province, and reduce administrative costs. ^{Footnote 82}

This approach is reminiscent of the CCA, which, from 2015 to 2017, overhauled 147 arts-based programs into 6 new programs (with various components) focused on objectives:

• Explore and create
• Inspire and root
• Create, Know and Share First Nations, Inuit and Métis Arts and Culture
• Support artistic practice
• Radiate in Canada
• To radiate internationally^{Footnote 83}

Single suggestions made by surveyed recipients

31% (6) of surveyed CATF recipients expressed gratitude for the funding received and expressed how vital CATF funding is, including to the quality of their training. 2 surveyed CATF recipients and 1 recipient of another Arts Branch program requested that more funds be made available through CATF. The last single suggestion from a CATF recipient was on the difficulty contacting PCH staff in person by a new recipient, and value of that personalized contact.

Survey respondents identified as primarily recipients of different Arts Branch programs made 3 other single suggestions for CATF:

• Reinstate funding for non-performing theatre professionals
• Accept reporting on basis of recipient's own fiscal year
• Allow section of the on-line application to be completed in any order.

5. Conclusions

This section outlines the key conclusions of the evaluation of the CATF program and is presented in compliance with the Policy on Results. It is meant to provide a link between the key findings and the recommendations.

5.1. Relevance

There is a need to contribute to the development of Canadian creators and future cultural leaders of the Canadian arts sector by supporting the training of artists with high potential through organizations that offer training of the highest calibre

There has been a high demand for CATF as it is the only program to provide cohesive, multi-year operational funding for professional arts training organizations and has been essential to the arts ecology and future sustainability of arts in Canada.

CATF was responsive to the needs of funded arts training organizations

CATF supported the development of high-quality training organizations in Canada which produced and prepared high-quality professional artists for their careers. However, due to high program demand and fixed budgets, certain qualified organizations were not being funded and a few high-quality organizations had their funding reduced.

There were demonstrated benefits for both Canadians and Arts organizations for the federal government's role in this field

Culture and artistic performances have been significant contributors to GDP, generators of economic growth, jobs and wealth in Canadian communities, and have contributed to the cultural identify of Canada. Without public funding, some training organizations would not have been viable which may have prevented access for Canadians to strong artistic experiences and Arts organizations to a strong pool of artists within Canada.

5.2. Effectiveness

CATF assessment processes were competitive ensuring only high-quality training organizations were funded

Given CATF's fixed budget, the assessment process was highly competitive and ensured only organizations which best exemplified program objectives were funded, regardless of previous contributions received. However, demand outweighed supply, and certain high quality institutions were not funded. Further, the definition of excellence may not have been appropriate for Indigenous art forms. Nearly three quarters of surveyed recipients agreed "to a great extent" that CATF funding had: contributed to improving/updating their arts training curriculum, increased the scope and depth of graduates' skills, and increased the quality of arts professional training.

CATF-funded organizations had diverse revenue streams to support financial stability

Organizations had a combination of public, earned and private funding. While the amount of funding from each source fluctuated, the proportion from each source remained mostly consistent. Importantly, public funding such as CATF assisted training organizations to be financially stable. This was particularly true for Indigenous, ethnocultural and racialized organizations who experienced challenges accessing private funding. Only 10% of recipients surveyed stated that a lack of CATF funding would endanger their program or school.

CATF supported the development of high-quality training institutions and successful professional artists

Between 2012-13 and 2017-18, CATF funded a total of 41 training organizations which produced 13,100 graduates. Overall, 8,705 of these graduates were working professionally in their field to perform and create for Canadian and international audiences (average of 66.5%). Further, CATF-funded institutions were recognized by industry as leaders in their fields, with 93% of employers indicating graduates of CATF-funded organizations were better prepared for professional careers. 90% of surveyed recipients agreed that CATF funding had a great or moderate impact on the recognition of excellence among their graduates; 95% agreed with the impact on their graduates' subsequent professional careers.

5.3. Government-wide policy considerations

CATF met the official language requirements of section 41 of the Official Languages Act and undertook measures to ensure that both official languages were represented in the funded organizations.

CATF required funded organizations to have a bilingual website, information material and (with a few appropriate exceptions) audition process.

CATF personnel were instructed to apply GBA+ to identify systemic barriers in their terms and conditions and guidelines and targeted specific minority communities.

Several years ago CATF expanded the program scope to include Indigenous and culturally diverse arts forms. More recently, PCH created a Departmental Plan and Framework for GBA+. However, external stakeholders and survey respondents identified that there are ongoing arts training access issues for most of the populations and organizations listed under GBA+.

Stakeholders and survey respondents identified that there is a need to further examine the systemic barriers built into the program which prevents access for specific segments of the population, particularly Indigenous organizations.

5.4. Efficiency

CATF utilized consistent and rigorous implementation of eligibility and assessment criteria, site visits and expert assessments to ensure resources dedicated to the program were used efficiently.

Based on this data, CATF altered the program priorities and program delivery as necessary to achieve a greater impact (e.g. reallocation of funds across different categories of recipients; reallocation of funds from underperforming organizations/organizations that did not demonstrate need to those showing consistent or increased level of excellence). Further, CATF met service standards for acknowledging receipt of applications (80% of applications within 15 days). The program did not meet service standards for notification of the funding decision (80% of applications within 29 weeks). However, program data for 2017-18, suggests that timelines have improved with regards to notification of the funding decision.

Alternative approaches or innovations to achieve programs outcomes were identified throughout the evaluation

The key approaches identified included:

• A two-stream funding model (i.e. one for large organizations and one for small organizations).
• Culturally appropriate approaches to funding Indigenous arts organizations and cultures (e.g. conducting consultations with communities regarding appropriate funding programs, providing consistent feedback opportunities and accommodating alternative applications such as verbal applications).
• Shortened Professional Assessors' Report template to increase consistency and utility to NRC (i.e. replace question on uniqueness with assessor's description of ecosystem of the discipline, ask assessor to define excellence and to rank schools in a discipline as appropriate, to detail methodology, and drop questions on administrative stability or that can be answered through applications or final reports).

6. Recommendations, management response and action plan

Table 16: recommendation 1—action plan

Measure	Deliverables	Timeline	Responsible
1.1 Review and analyze the current needs, differences and strengths of National Arts Training organizations	In-house report	April 1, 2019 to March 31, 2020	Director General, Arts Branch
1.2 Develop options on feasibility and impact of implementing a two-stream funding model	Presentation to senior management	April 1, 2020 to March 31, 2022	Director General, Arts Branch
1.3 Implementation plan of approved option(s)	ADM-approved Action Plan	April 1, 2022 to March 31, 2023	Director General, Arts Branch
Full implementation date: April 1, 2023

Table 17: recommendation 2—action plan

Measure	Deliverables	Timeline	Responsible
1.1 Delivery of the experimentation project	Reports	January 2019 to March 31, 2021	Director General, Arts Branch
1.2 Consultations with schools and other stakeholders on the application and assessment process	Report to senior management	April 1, 2019 to March 31, 2021	Director General, Arts Branch
1.3 Analysis of findings and develop options based on consultations and results of the experimentation project	Presentation to senior management	April 1, 2021 to March 31, 2022	Director General, Arts Branch
1.4 Implementation plan of approved option	ADM-approved Action Plan	April 1, 2022 to March 31, 2023	Director General, Arts Branch
Full implementation date: April 1, 2023

Annex A: evaluation framework

Relevance—issue #1: continued need for program

Questions	Indicators	Methods of data collection
To what extent is there a need to contribute to the development of Canadian creators and future cultural leaders of the Canadian arts sector by supporting the training of artists with high potential through organizations that offer training of the highest calibre?	Evidence and views of key informants on the extent to which the different components of the CATF addressed the demonstrated needs of Canadians. Current state of the needs that gave rise to the programs New conditions that augment or diminish need for the programs Needs of Indigenous and ethnocultural minority clients	Literature Review Document review Administrative data review Key informant interviews Survey of applicants (funded and unfunded) Focus Group
To what extent was the CATF program responsive to the demonstrated needs of Canadians?	Evidence and views of key informants on the extent to which CATF was responsive to the demonstrated needs Number of applications received and total amount of funds requested, by component Number of applications funded and total amount of funds awarded, by component	Literature Review Document review Key informant interviews Focus Group Case studies Administrative data review Survey of applicants (funded and unfunded)

Relevance—issue #2: alignment with government priorities

Questions	Indicators	Methods of data collection
To what extent was the CATF programs aligned with PCH priorities and federal government priorities?	Extent to which each program assisted the Department to reach its priorities Extent to which each program was aligned with federal government priorities	Document review Key informant interviews

Relevance—issue #3: alignment with federal roles and responsibilities

Questions	Indicators	Methods of data collection
Is supporting the training of artists with high potential through organizations that offer training of the highest calibre an appropriate responsibility for the federal government?	Demonstrated benefits (to Canadians and to arts & heritage organizations) of the federal government's roles in this field Degree of complementarity between the federal government's role and the roles played by other stakeholders in the cultural sector, among them the restructured Canada Council for the Arts.	Document review Administrative data review Literature Review Key informant interviews Focus Group

Performance—issue #4: achievement of expected outcomes—immediate outcome

Questions	Indicators	Methods of data collection
To what extent do nationally significant Canadian organizations offer arts training of the highest calibre?	Number of applications made by students to funded organizations versus the number of places available Number of graduates and number of students Number of awards and professional achievements received by instructors/artistic directors	Administrative data review Document review Key Informant Interviews

Performance—issue #4: achievement of expected outcomes—intermediate outcomes

Questions	Indicators	Methods of data collection
Are Arts training organizations of the highest calibre financially and administratively stable?	Amount and percentage of funding sources by type (earned, public, private) Financial viability analysis (FVA)	Administrative data review Document review Key informant interviews
Are graduates recognized for excellence in Canada and internationally?	Level of industry satisfaction with graduates' professional training Number of awards and professional achievements received by students/graduates Percentage of graduates employed professionally in their field in Canada or abroad	Survey of Employers in the Performing Arts Administrative data review Key informant interviews Literature Review

Performance—issue #4: achievement of expected outcomes—ultimate outcomes

Questions	Indicators	Methods of data collection
Did Canadians value the work of Canadian professional artists?	Level of appreciation of Canadians	AHAA Key informant interviews Survey of applicants (funded and unfunded)

Performance—issue #5: government-wide policy considerations

Questions	Indicators	Methods of data collection
Were all official language requirements met?	Extent to which each program supported official languages and met the requirements of section 41 of the Official Languages Act Distribution of funding to official language minority communities (OLMCs) Perspectives of recipients in OLMCs	Document review Administrative data review Key informant interviews Survey of applicants (funded and unfunded)
Did the programs have unintended barriers and issues related to gender-based analysis (GBA+)?	Extent to which GBA+ was taken into account by the programs, specifically; considerations of how the program may impact differently certain segments of the population (such as women-men, ethnocultural communities, Indigenous communities, official language minorities, persons with disabilities, people living in remote areas, youth, etc.)	Document review Administrative data review Literature review Key informant interviews Survey of applicants (funded and unfunded)

Performance—issue #6: efficiency and economy

Questions	Indicators	Methods of data collection
Were the resources dedicated to the program used efficiently to maximize the achievement of outcomes?	Average number of weeks to assess a complete application to the Canada Arts Training Fund	Administrative data review Document review Key informant interviews
Are there more efficient alternatives to achieve the same results?	Evidence and views of key informants on more efficient alternative approaches or innovations (including program design and delivery) to achieve programs outcomes	Literature review Key informant interviews

Annex B: bibliography