Evaluation of the Aboriginal Peoples’ Program 2009-10 to 2013-14

Table 6: Overview of evaluation issues and questions

Preliminary consultation

Before undertaking the evaluation, preliminary discussions were held with APP employees. This led to the development of the Terms of Reference for the evaluation, which included a description of the evaluation scope and issues, the methodological approach, and the detailed evaluation matrix. The Terms of Reference were approved by PCH's IPPMEC on February 24, 2015.

Lines of evidence

The evaluation benefited from both qualitative and quantitative methods, and included a balance of both primary and secondary data sources. The findings from all lines of evidence was organized and analyzed by core issue, evaluation question and indicator and consolidated into an overall evidence matrix. The use of multiple lines of evidence provided the opportunity to develop findings based on a triangulation of evidence, contributing to the strength of conclusions. Data collection and reporting of preliminary evaluation results were undertaken between April 2015 and December 2015.

The evaluation methodology included the following four lines of evidence:

• Literature review – The primary goal of the literature review was to gather, analyze and summarize findings linked to information and documentation from recently published literature, reports, articles, websites, academic journal databases, publications, census (National Household Survey (NHS)) and other sources at the national level and international level on subject matters related to a variety of questions identified in the project's evaluation matrix.
• Document, file and administrative data review - The purpose of the document, file and administrative data review was to systematically extract previously collected relevant secondary data from identified documents, files and databases that provide evidence for a specific evaluation indicator. Documents and files reviewed included key government documents (for example: Throne Speeches and federal Budgets), departmental documents (for example: Departmental Performance Reports, Reports on Plans and Priorities) and program-related documents such as audit and evaluation reports, contribution agreements, the Performance Measurement, Evaluation and Risk Strategy (PMERS), website content, program criteria and review materials, communication products, guidance and information products, presentation decks, reporting templates, and project files. Documents/files were reviewed using a template to capture key statements and summaries, which were then compiled into an evidence matrix.
• Interviews with key informants - Interviews gathered in-depth information, including opinions, explanations, examples and factual information on evaluation issues and questions. The list of potential key informants was developed with input from APP staff. In-depth interviews were conducted with 24 key informants: PCH officials (7), funding recipients (10), non-funded applicants for NAB (1), representatives of territorial governments (2) and experts (4).
• Case studies– Three organization-level case studies were conducted for NAB, and one project-level case study was conducted for NAD. Each case study consisted of interviews, document reviews and file reviews. The NAD case study was designed to highlight details on objectives, activities, outcomes, and challenges of the festival. The NABcase studies were conducted to:
  • contribute to understanding the extent to which Aboriginal community radio broadcasting contributes to preserving and revitalizing Aboriginal languages and cultures and encourages the full participation of Aboriginal peoples in the Canadian society; and
  • determine the extent to which Aboriginal broadcasting related activities are integrated within the communities served by NAB funding recipients.

Methodological limitations

Overall, there were a few methodological challenges encountered when conducting the evaluation. These included:

• Challenges with conducting case studies – there were considerable challenges encountered in recruiting community members for interviews, and obtaining documentation on project outcomes. As a result, the case studies were not able to provide the level of detail and understanding of community level outcomes as originally anticipated.
• Administrative data and final reports have limitations for evaluation purposes – There was an expectation that the administrative data and final reports from projects, in particular the TLA reports, would provide some evidence of outcomes, particularly with respect to the contribution of the NAB funding to outcomes such as "Aboriginal individuals and groups engaged in activities that strengthen Aboriginal languages and culture". The review of administrative data and a further review of a sample of project files and annual reports found that reporting is largely on outputs (for example: broadcast hours, products) rather than measuring the impacts/changes that these outputs are having on individuals and groups (outcomes). In addition, changes in reporting requirements/definitions, combined with some gaps in available data, contributed to challenges in analyzing any trends or changes across the period covered by the evaluation. As well, some of the information for proposed indicators are contained in the narrative final reports that match the contribution agreements. Unfortunately, this makes it challenging to compile and analyze in a consistent manner across projects.

Contextual considerations that impact the current evaluation

In addition to the methodological limitations, there were a number of contextual considerations that also had a direct impact on the evaluation.

• Significant changes in structure of APP – The APP as it existed at the beginning of the evaluation period (2009-10) is radically different from the APP structure that remained as of the end of the period covered by the evaluation (2013-14). As well, there are currently ongoing transferring occurring of some program elements:
  • The termination of one program element, combined with the transfer of multiple program elements from PCH to AANDC, have created multiple challenges for not only the program, but also the evaluation.
  • The considerable changes made to the APP during this period combined with the ongoing transfers have resulted in a large decrease in the program budget (approximately 1/3 of original budget remained as of 2013-14), workforce reduction, and a significant transfer of many experienced employees out of the Directorate. This has contributed to an increased reliance on short-term human resource assignments, and a lack of clarity on the expected outcomes for the remaining elements of the APP resulting in a relatively fractured, fragile program structure remaining at PCH.
  • This situation presents considerable challenges to accurately evaluate the relevance, effectiveness and efficiency of the APP over a five-year period during which all of these changes were taking place. What may have been achieved in 2009-10 and 2010-11 are likely not at all reflective of what the APP was able to achieve in 2013-14. Similarly, the expectations of what was a cohesive, multi-component program can achieve with respect to outcomes is not necessarily a simple addition or subtraction of program elements, particularly when they are designed to address complex, multifactor issues and areas such as Indigenous cultures and languages.
• Splintered approach to evaluation of APP – Circumstances related to the timing of information needed to support the ALI renewal, the planned horizontal evaluation of the two scholarships elements of the program (SYI, PSSP) and the transfer of multiple components from PCH to APP have resulted in a necessary but relatively splintered approach to evaluating APP. While all APP program elements will be covered by evaluation at some point, the ideal cohesive, multi-element approach to evaluation was not possible. The three program elements covered in the current evaluation are all concentrated within one APP component (Aboriginal Living Communities). To be able to conclude on the effectiveness of the program, findings from the recent ALI evaluation have been integrated. This splintered approach makes it more challenging to attribute results to specific encompassing outcomes, to effectively draw conclusions and make recommendations at the program level, and even the sub-program level.
• Challenge identifying trends due to significant administrative changes that occurred – During the period covered by the evaluation, NAB changed its funding allocation process significantly. As of 2012-13, NAB moved from predetermined allocations to a merit-based competitive process. This change in process during the period covered by the evaluation makes it challenging to identify and report on trends in outputs and outcomes for the APP across the five -year period.
• Shifting priorities, needs and programming efficiency – The NAB program appeared to be impacted to some extent by priorities that would shift from year to year. The advisement on priorities would occur late in the proposal and recommendation process, resulting in challenges for the program to maintain service standards and effective communications with applicant organizations. These circumstances also presented some challenges for the evaluation in the areas of accurately assessing alignment between needs, priorities, and programming given all of the shifting from year to year. More specific challenges for the evaluation occurred in assessing the efficiency of the APP processes, as many of the delays noted were likely due to the approval process.
• Defining language and culture –Many of the APP components covered by the evaluation indicate objective and anticipated outcomes with respect to "language and culture". While this notion is a dual concept, the outcomes assessed by the program are mostly limited to the language aspects.
• Diversity and differential impacts - Given the diversity of Indigenous cultures and communities, it is expected that there will be large differential impacts of technology on Indigenous culture and language which are likely important to APP outcomes, but challenging to determine (for example: lack of internet, impact of social media, use of smart phone). Similarly, given the diversity we would anticipate large differences between status/use of Indigenous languages (for example: Inuit vs First Nations vs Metis, on reserve vs off reserve, rural / urban, etc.).

Need for continued support of Indigenous languages and culture

All lines of evidence demonstrated that there continues to be a strong need to support Indigenous languages and culture. The literature and document reviews along with interviews with key informants noted the following points that indicate a strong need for continued support of Indigenous languages and culture:

• Decreasing number of people who have an Indigenous language as a mother tongue or first language – Many Indigenous languages are experiencing long-term and short-term declines in rates of people who claim an Indigenous language as a mother tongue or first language. The impact of the long-term erosion of Indigenous languages from generation to generation is evident today: 202,500 or just 14.5% of Canada's 1.4 million Indigenous population are reporting an Indigenous language as their mother tongue in the 2011 NHS. ^{Footnote 11} The share of the Indigenous population reporting an Indigenous mother tongue declined from 26% in 1996 to 21% in 2001 and 19% in 2006, followed by a declined to 15% in 2011. ^{Footnote 12}
• Challenges with intergenerational transmission of Indigenous languages and culture – UNESCO notes that a key determinant for assessing the vitality of a language is the extent to which there is intergenerational transmission.^{Footnote 13} Intergenerational transmission of Indigenous languages has declined steadily over the past 30 years, from a 1981 index of 76 persons speaking an Indigenous language most often at home per 100 individuals with an Indigenous language mother tongue, to 58 by 2011. ^{Footnote 14} This means that even among those who do have an Indigenous language as their mother tongue (approximately 15% in 2011), only 58% are likely to use it as the language spoken most often in their home. Challenges with intergenerational transmission include the direct impact that residential schools have had on multiple generations, as well as the trend of increased urbanization among Indigenous populations in Canada .^{Footnote 15}, ^{Footnote 16} As Indigenous diversity becomes centralized within urban areas, it is becoming increasingly difficult to identify and target specific languages and cultures upon which to focus revitalization efforts.^{Footnote 17}
• The "critical mass" of speakers needed to maintain a language is not available for many Indigenous languages -The Indigenous languages with strongest likelihood for continued survival are those with the largest numbers of speakers. As one of the key determinants of language vitality, the total number of speakers is crucial to predicting the "health" of a language. ^{Footnote 18} According to the 2011 NHS, there were only three Indigenous languages identified with more than 20,000 speakers (Cree, Inuktitut and Ojibway).^{Footnote 19} In contrast, a 2010 United Nations Educational, Scientific and Cultural Organization (UNESCO) study identified over 60 languages in Canada that it considered to be endangered, in large part due to the low number of speakers of these languages.^{Footnote 20}
• Revitalization ^{Footnote 21} of some Indigenous languages may be occurring through increasing the number of second language speakers –There appears to be a trend over the past 20 years towards greater rates of acquisition of Indigenous languages as a second language. ^{Footnote 22} According to the 2011 NHS, more than 52,000 people are able to converse in an Indigenous language different from their non-Indigenous mother tongue, suggesting second-language acquisition. As noted, in the literature, immersion programs and second-language learning programs may provide alternatives for the revitalization of Indigenous languages in some cases. ^{Footnote 23}Key informants noted that while there are challenges in revitalizing many of the Indigenous languages to the status of mother tongue, there are considerable opportunities available upon which to focus with respect to revitalizing some Indigenous languages via second language learning.
• Impacts of assimilation-based policy and programming on loss of Indigenous language and culture -The legacy resulting from these policies and programming, most notably the Residential School system, have been clearly identified through the Truth and Reconciliation Commission (TRC). As noted in the Commission's report: ^{Footnote 24}

  Canada's residential schools took Aboriginal children away from their parents, their families, and their communities for the purpose of destroying their connection to their traditional cultures and languages. (p.103).…the Survivors explain how the loss of languages led to a loss of identity and ultimately brought Aboriginal people face to face with the destruction of their cultures. (p.109)

Need for continued support of Indigenous broadcasting

The evaluation also found the need for continued support for Indigenous broadcasting and northern Indigenous broadcasting more specifically. Findings from the case studies and key informant interviews were that Indigenous populations in remote locations require culturally relevant information to be broadcasted, particularly with respect to public safety, weather conditions, public health and general community news. This is of particular importance in the context of Indigenous people who traditionally used oral transmission of culture. Most key informants noted that the sustainability of funded Indigenous broadcasting organizations in the North in the absence of APP would be extremely challenging, given that the federal government is the main financial contributor to programming. Other sources of revenue common to broadcasting such as contributions from advertising or directly from the private sector are challenging in the North given the economic structure and resources of northern communities.

Emerging issues

A number of emerging issues were identified from the key informant interviews and case studies. They will be important to consider how they might be addressed in the future. The emerging issues identified included:

• Preservation and archiving of Indigenous culture analog content. NAB recipients in key informant interviews and case studies noted that there is an increasingly urgent need to preserve and archive much of the content related to Indigenous culture that has been produced under NAB and other initiatives. Much of the audio and video content is currently stored only on analog magnetic tapes which are coming to the end of the storage life within the next few years. The content is at risk of being lost without support for appropriate preservation and archiving.
• Broadcasting infrastructure. Most PCH and external interviewees agree with the fact that it is urgent to renew and upgrade the current equipment of the television and radio installation, because it is outdated. This limits the ability of broadcasters to produce quality programming and thereby reduces the ability to obtain new financing and innovate.
• New technologies. Many new digital approaches and platforms have been developed in the last decades to produce and broadcast audio and video content that appeal to the younger generation. However, the costs for such technologies are not covered under the NAB while this transition is costly in capital for issuers' operators.

Alignment with government of Canada priorities

In federal government announcements throughout the evaluation period, commitments made show that APP objectives are aligned with Canada priorities. As outlined in key documents these include:

• Government of Canada's "a vibrant Canadian culture and heritage" strategic outcome, as outlined under the Social Affairs spending area of the whole-of-government framework. ^{Footnote 25}
• Budget Speech 2014 which stated that, in an effort to bring Canadians together by sharing and respecting other cultures the Government of Canada will "continue to support efforts to preserve Aboriginal languages." ^{Footnote 26}

A strong connection can be demonstrated between the current priorities of the government as outlined in key documents and the APP objectives:

• The 2015 Speech from the Throne presents the approach the Government of Canada "will undertake to renew, nation-to-nation, the relationship between Canada and Indigenous peoples, one based on recognition of rights, respect, co-operation and partnership". Among its stated priorities, the government mentions as a priority the response to the calls to action from the TRC's: "the Government will work co-operatively to implement recommendations of the Truth and Reconciliation Commission of Canada". ^{Footnote 27}
• More specifically, the Minister of Canadian Heritage Mandate Letter included the statement that a top priority will be: "Work in collaboration with the Minister of Indigenous and Northern Affairs to provide new funding to promote, preserve and enhance Indigenous languages and cultures". ^{Footnote 28}

Alignment with PCH priorities and strategic outcomes

The 2013-14 Report on Plans and Priorities (RPP) ^{Footnote 29} indicates that the APP aligns with the following two PCH priorities:

• Celebrating our history and heritage. PCH "has a role in promoting a strong national identity that is based on attachment, shared values and, knowledge and experiences of Canada that emphasize the legacy of its history and heritage"; and
• Investing in our communities. "Through supporting these activities, grounded in local and community realities, the Department also encourages the sharing of our diverse cultural expressions and understanding of our history and heritage. These help to connect communities, contribute to healthy and vibrant communities and to a strong sense of Canadian identity".

The 2013-14 Report on Plans and Priorities (RPP) indicates that the Department's legislative mandate is set out in the Department of Canadian Heritage Act and other statutes for which the Minister of Canadian Heritage is responsible and presents a wide-ranging list of responsibilities under the heading of "Canadian identity and values, cultural development, and heritage."

Supporting this mandate is the mission for PCH to promote an environment in which all Canadians take full advantage of dynamic cultural experiences, celebrate our history and heritage, and participate in building creative communities. The PCH vision is one of "a Canada where all Canadians can celebrate our rich cultural diversity and our shared experiences and values, and where all can gain a greater understanding and appreciation of our history, heritage and communities." The APP sub-program is located within the program of "Engagement and Community Participation" supporting the strategic outcome of "Canadians share, express and appreciate their Canadian identity."

Departmental performance documents show that the APP has contributed to departmental priorities. The most recent PAA articulated in PCH's 2013-14 Departmental Performance Report (DPR) ^{Footnote 30} confirms that the activities associated with the APP's objectives support the achievement of Strategic Outcome 2: Canadians share, express and appreciate their Canadian identity and more specifically under Program 2.2: Engagement and Community Participation. through the achievement of results such as "Engaged as an integral part of Canadian society, Aboriginal peoples embrace and share their languages and cultures with other Canadians" and "Aboriginal individuals and groups are engaged in activities that strengthen Aboriginal languages, and cultures, and support community engagement."

The TLA component with the goal of supporting the preservation, development and enhancement of Indigenous languages aligns with the concepts of engagement and community participation in order to effectively achieve success. Language as a key component of identity also aligns with supporting the outcome of sharing, expressing and appreciating identity.

The NAB component with the goal of providing support to produce and distribute radio and television programming in the North aligns with the concepts of engagement and community participation and the sharing, expression and appreciation of identity. The activities under the NAB are very focused on communities, and incorporate various aspects of cultural identity into programming.

The NAD component supports opportunities for Canadians to become better acquainted with cultural diversity of Inuit, Métis, and First Nations people, discover the unique accomplishments of Aboriginal peoples, and celebrate their significant contribution to Canadian society. This aligns closely with the PCH vision of having Canadians celebrate rich cultural diversity and gaining a greater understanding and appreciation of history, heritage and communities.

Table 7 Overall APP objectives and outcomes

Achievement of immediate outcome: Access to resources to deliver projects that incorporate Aboriginal languages and cultures

The evaluation found that under the three APP program elements covered in the current evaluation, Indigenous communities are gaining access to resources to deliver projects that incorporate Indigenous languages and cultures. This demonstrates support towards various program elements' objectives as outlined in Table 2 previously such as "supporting the production of culturally-relevant Aboriginal programming (NAB)", and "support the preservation, use and promotion of the Inuit Language at the community level (TLA)".

TLA: Under the accords, during the period covered by the evaluation, Nunavut has had an annual allocation of $1.1 million for 13 to 24 community-based Inuktitut language initiatives; while Northwest Territories annually receives$1.9 million to support 9 Indigenous languages and the efforts of 7 regional Indigenous Governments. These funding amounts have remained the same for Nunavut since 2005-06, and for the Northwest Territories since 2003-04 (See Figure 1). This trend in constant funding amounts is noted to have occurred during a period when inflation in Canada between 2006 and 2014 was at approximately 14% ^{Footnote 33}.

Figure 1: TLA Annual allocation by territories ($)

NAD: The funding allocated to NAD recipients has remained constant across the five years covered by the evaluation at $120,000 per year. From a review of final reports, there is considerable diversity with respect to performers, workshops, themes for educational days and community projects funded. One organization receives funding annually but works with the National Aboriginal Day Committee (NADC) composed of six of the primary National Aboriginal Organizations (NAO) based in Ottawa to develop an education day for students in the National Capital Region, organize and implement a festival that includes multiple performances, workshops and demonstrations. In addition, 5 to 7 smaller community events are funded each year. The activities reported align with the overall objective of demonstrating the cultural diversity of Aboriginal peoples to NCR residents and visitors.

Figure 2: NAD Annual allocation to NRC ($)

NAB: During the period covered by the evaluation, the same 13 organizations were funded annually by NAB. Total funding amounts ranged from $8.0 million in 2009-10 to $6.8 million in 2013-14. With the exception of the decreasing funding levels in 2012-13 and 2013-14, the funding amounts available for NAB have remained constant for the past 15 years at approximately $8 million since 1998. On average, NAB applicants requested 40% of their global operational budget across the five years covered by the evaluation. This varied between 11% (Org-1) and 64% (Org-11) (See Figure 3).

Figure 3: Ratio of APP funding to recipient organizations' overall operational budget

Seven of the organizations funded produce exclusively radio programming, two produce exclusively television programming and four produce for both radio and television. Of the 65,000 to 85,000 hours of annual radio programming by the NAB funded communication societies, approximately between 12 and 21% of overall radio broadcasting by the communication societies funded under NAB is completed using Aboriginal languages during peak hours. Significantly smaller numbers of television programming were reported (approximately 90 to 120 hours per year), but these were exclusively using Aboriginal languages (100%). This is likely, in large part, due to the higher costs associated with content production and programming for television compared to radio.

ALI: The recent evaluation found that ALI is making progress toward the achievement of the expected immediate outcome, "Aboriginal communities are able to access resources to deliver projects that incorporate Aboriginal languages and cultures through ALI." The evaluation reported that 550 projects were funded over the five-year period of the evaluation for a total Gs&Cs expenditure of $18.6 million.

Gaps in resources

The evaluation found that while there are projects being funded, there remain considerable gaps between the need for resources and the amounts made available under the APP.

TLA: Key informants noted that there are large gaps between the needs identified for Aboriginal languages in the two Territories, and the resources available. The funding levels under the two accords have remained constant over the five-year period ^{Footnote 34} covered by the evaluation. Significant additional territorial resources are being added to the budgets for many of the Aboriginal languages and cultures initiatives in the Northwest Territories and Nunavut, therefore requiring the two territories to bear the main financial weight of the preservation and revitalisation of the languages of their Indigenous population. Key informants noted that funding territorial agreements has not progressed as rapidly as the needs on the ground.

NAD: Funding has remained constant across five years covered by the evaluation at $120 thousand per year, despite increased participation and attendance, and increases in costs associated with the festival and education day. An interesting observation is that the amount of planned additional resources provided by various public institutions and in-kind contributions according to the Contribution Agreements has steadily increased across the five-year period covered by the evaluation (with the exception of 2010-11) from $99 thousand (2009-10) to $285 thousand (2013-14) with an approximately 2:1 leveraging ratio in 2013-14 for planned raised dollars to NAD contributions.

NAB: Key informants noted that there were a number of challenges with respect to repeat programming, limited Aboriginal content, and significant issues with broadcasting infrastructure and equipment. Lack of resources was highlighted as directly contributing to these challenges which an explanation that other potential sources of revenues available to the communication societies funded under NAB are limited (for example: advertising revenue), and are not able to address these gaps. As noted in section 5.2, the majority of NAB funding is allocated to salaries with just a very small proportion of funding (1 to 2%) allocated in 2010-11 and 2013-14 to equipment purchase, rental or leasing.

While it is noted by key informants and experts that many public and civil stakeholders will need to be involved in order to address overall needs with respect to language and culture, there do seem to be some gaps in responsiveness in the APP as it applies specifically to northern Aboriginal broadcasting needs. The review of the administrative data found that while the number of funded organizations for NAB has remained stable at 13, the amount of funding requested has grown significantly higher than resources available, which have actually decreased over the same period (see Figure 4). This increasing gap is likely the result of two opposing trends, notably (i) an increase in funds requested and (ii) a decrease in total funding approved (a 13% decrease over the four-year period). These trends are connected to the observation that while reference levels for NAB have essentially remained constant between 1998 and 2014, inflation in Canada during that same period is determined at approximately 38%. ^{Footnote 35}

Figure 4: Gap between requested and approved funding for NAB (2009-10 to 2013-14)

ALI: The ALI evaluation also found a resource gap largely due to the high demand for funding relative to available funds. The evaluation reported that only 28% of applicants to PCH received ALI funding.

Achievement of intermediate outcome: Engaged in activities that strengthen Aboriginal languages and culture

As noted previously, for the intermediate and ultimate outcomes identified in the logic model for the APP, the actual influence and specific contribution of the three program elements are difficult to ascertain, and perhaps unrealistic to expect large gains given the level of investments during this period, and significant reductions to the APP program structure. The nature of these outcomes (strengthening Aboriginal languages and culture; engagement of Aboriginal people as integral part of Canadian society) is such that they are very broad, extremely challenging to effectively measure, and likely require significant involvement, collaboration and participation from multiple stakeholders and groups in Canadian society to achieve. That being said, the evaluation did find a few proxy indications that the program elements may be making contributions to the outcome of engagement in activities that strengthen Aboriginal languages and cultures. Contributions towards outcomes identified in these areas would be considered supportive of various program elements' objectives identified in Table such as "contribute to the protection and enhancement of Aboriginal languages and cultures (NAB)", or "provide programs that support the revitalization, maintenance and enhancement of the NWT's official Aboriginal Languages (TLA)".

The evaluation found that some activities are contributing to outcomes that have been identified in the literature as key determinants of language vitality. ^{Footnote 36} These include:

• The use of language in different public and private domains: The APP focuses on moving the use and exposure of Aboriginal languages into multiple public spheres beyond just the family setting (for example: broadcasting, program/content development, community projects). NAB provides funding to organizations to produce and broadcast Aboriginal content into public domains. The key informants noted that NAB contributes to helping Aboriginal peoples stay informed on community decision-making consultation and governance process and keeping people abreast of cultural, social, political events and gatherings taking place in their community. Case studies and interviews noted that some broadcasts in aboriginal languages are used multiple times in different settings and formats (for example: schools, podcasts, streaming). TLA funds community projects that include Aboriginal language content being provided in multiple domains including families, schools, community centres, health care settings, public broadcasting, etc. Similarly, NAD funding supports the presentation of various Aboriginal languages and culture through festival performances, workshops and educational days.
• Integration of the language into new domains and media – The evaluation found evidence of TLA community projects and NAD festivities being somewhat responsive in this area with an emphasis on social media, and projects emphasizing different and new areas in the visual and performing arts. The NAB program may be less responsive in this area with its continued emphasis on more traditional broadcasting television and radio (although it is noted that some of the organizations funded under NAB are working in internet based media such as interactive streaming, podcasts, etc., but it is unclear to what extent NAB funding is being used directly in these areas). The key informants interviews and the document Review suggest that, while traditional broadcasting (radio, television) remain important to a segment of this population, there is increasingly other preferred technological modes of delivery (for example: internet streaming, social media) for other segments. Under its current Terms and Conditions, NAB does not currently support these popular alternatives for younger segments of the population(many of whom are increasingly moving to urban centres).
• Integration of language with teaching materials - The integration of Aboriginal languages into teaching materials were noted in all three program elements: TLA through community projects, NAB broadcasts being re-used at schools, and some of the original content produced which serves educational purposes; and NAD during its educational days.

NAB: The case studies, key informant interviews, document review and a file review of funding received by five organizations under NAB (total of 25 projects) provided examples and evidence of where the APP funded projects had contributed to Aboriginal individuals and communities becoming engaged in activities that would likely contribute to strengthening Aboriginal languages and cultures. Administrative data and file review for NAB projects noted that programming is quite diverse including areas such as Aboriginal culture and languages, children and youth programming, radio and television shows centered on community events, cooking shows, documentaries, educational content, entertainment information, health, music, news, sports and weather forecasts. The type of programming most often mentioned includes community events, and local news (see Table E-1 in Appendix E).

Examples from NAB projects include:

• Community involvement: broadcasting local interest and news programming; live coverage of community meetings; increased demands and partnering for programming from community organizations (school boards, government, health board); special commercial rates to Aboriginal businesses; free time offered for community services; employment for Aboriginal speakers; music broadcasted is either performed, produced or written by Aboriginal artists, "Word of the day".
• Youth involvement: an active youth audience for broadcasted programming; attendance and participation at several career days at both high schools and post-secondary institutions; contributing to interest among youth to learn Aboriginal languages; expanding to include youth programming to be delivered via webcast; schools using content produced to support education curricula.

TLA: As noted in interviews and through the document and file review (see Table E1 in Annex E), across both agreements there is considerable variety with respect to the products developed, initiatives implemented and audiences targeted. Products range from printed media (books, magazines) to both video and audio programming. As well, there is consistent emphasis across the years covered by the evaluation in producing reference materials and various materials for educational purposes. Similarly, there is considerable diversity across the two agreements with respect to the types of initiatives funded ranging from immersion language classes to cultural camps and workshops. Finally, the diversity is also demonstrated in the audiences targeted (ranging from children to elders). All projects appear to have a strong language component from the brief description reviewed within the final reports. Given the changes in reporting templates over the period covered by the evaluation and the variety in detail provided in the annual reports, unfortunately the number of products cannot be quantified.

Examples from TLA Projects include:

• Products developed: books/magazines, reference materials, video/television programming, radio programming, various learning materials (puzzles, activity books, calendars, etc.) related to indigenous languages and cultures.
• Initiatives implemented: immersion programs, tutoring, cultural camps, cultural days/festivals, workshops, instructor programs, teaching and learning centres.
• Audiences involved: preschool children, school children, youth, parents, adults, elders, teachers/instructors/translators.

NAD: The document review and cases studies noted a diversity in activities that take place for NAD in the National Capital Region. The main groups of activities over the period covered by the evaluation generally included an Education Day during which over 3,000 local students annually participate in activities from larger-scale performances by Aboriginal musicians and entertainers to various more intimate cultural workshops. Another key component of the NAD was a large scale festival celebrating various aspects of vibrant Aboriginal culture during which over 1,000 Aboriginal artists and performers participate. These events are actively publicized and open to the general public and are focused on family programming with multiple stages, performers and workshops available across a two- to three-day period in late June. In addition, NAD funding supports approximately five to seven smaller community events that focus on a specific group or event at the community level.

Examples from NAD projects include:

• workshops with Aboriginal musicians to showcase emerging talent;
• various performances by Aboriginal artists, primarily from Ontario, with a few from Quebec and the Prairies with singers performing both in English and in a variety of Aboriginal languages; and
• various types of family programming that showcase activities designed to engage all ages comprising interactive workshops that provide authentic, hands-on experience in a variety of Aboriginal arts and crafts projects and demonstrations.

ALI: The ALI evaluation also noted progress towards the outcome of "Aboriginal individuals and groups are engaged in activities that strengthen Aboriginal languages and cultures." The evaluation reported that an estimated 29,400 individuals participated in ALI projects over the five-year timeframe and approximately two-thirds of the projects funded were participatory or partly participatory. The vast majority of ALI funding recipients surveyed indicated that Aboriginal individuals and groups were engaged in activities that strengthened their languages and cultures (90% agreeing this has occurred), and that ALI has strengthened the language and culture of participants (92%).

Achievement of ultimate outcome: Engagement of Aboriginal peoples as an integral part of Canadian society

Among those program elements evaluated, NAD provided some proxy indication of the contribution of the APP towards the engagement of Aboriginal peoples as an integral part of Canadian society. NAD supported events in the NCR that included education days, workshops, demonstrations, and performances by Aboriginals for the Canadian public who are the target audience. Findings from the case studies and document review, indicate that the activities funded under NAD contribute to a greater appreciation by the Canadian public of Aboriginal languages and cultures, and would indicate support for the program elements' objectives as outline in Table 2 such as "active citizenship and participation (NAD)" and "highlight the divers Aboriginal cultures and contributions of Aboriginal people in Canada (NAD)".

The evaluation found that the attendance of NAD main events has fluctuated from approximately 23,000 to 40,000 over the period covered by the evaluation. Attendance at community events has ranged from 3,300 to 4,900 across the period covered by the evaluation. Although demographic distributions of attendees are not available, given the target audience, it seems reasonable to assume that the majority of those in attendance would be non-Aboriginal Canadians who are being exposed through the activities to Aboriginal languages and cultures. Case studies concluded that the NAD project reviewed was a valuable way to engage Aboriginal and non-Aboriginal individuals and groups from the National Capital Region in educational and artistic activities that contribute to Aboriginal peoples embracing and sharing their languages and cultures with other Canadians. The case study noted that the activities attract an increasing number of visitors as non-Aboriginal Canadians are more and more aware of Aboriginal cultures and issues and, for Aboriginal people living in urban communities, the festival represents one of the few opportunities to engage in their community.

Budget and actual expenditures

Over the period under evaluation, reference levels for grants and contributions amount to approximately $207 million, although four-fifths of these funds were spent between 2009-10 and 2011-12 under other APP sub-sub components that have since been transferred to other departments.

The total reference level for this program has decreased from approximately $65 million annually in 2009-10 to $19 million in 2013-14. This important change is explained by the transfer of responsibilities for some components to other departments.

Table 8: Annual program budgeted and actuals ($)

Source: Resource Management Directorate – Aboriginal Peoples' Program

As outlined in Figure 5, the share of administrative costs over budget total remained relatively constant (between 8% and 9%) for three years of the program (2009-10 to 2011-2012). During 2012-13, APP delivered the remaining program elements Gs&Cs while, at the same time, restructuring and supporting the movement of components to AANDC. That year, the proportion of administrative costs nearly doubled to 16% before decreasing back to 12% in 2013-14. The trend to decrease has continued with administrative costs at 10.8% in 2014-15. The challenges encountered with administrative costs in 2012-13 were considered by program staff to be exceptional. These included aspects such as 12 FTE in PCH regional offices being transferred to INAC regional offices, 11 FTE at HQ were declared surplus, and in-year reductions of nearly $2 million were required for the Long Term Financial Strategy.

Figure 5: App administrative costs as proportion of total budget

As illustrated in Figure 6, an analysis of two years of NAB project funding (2010-11; 2013-14) found that the majority of funding received is allocated to salaries (57% and 69%). The remaining funds are spread across numerous categories with production costs (1% and 15%), rent and utilities (6% and 11%) and administration (8% and 10%) being the next largest, although these also demonstrate variability across the two years analyzed. A significant change in program priorities is reflected in Figure 6 with the marked decrease in the funding of production costs (15% in 2010-11 to 1% in 2013-14). As well, the proportion of NAB funding allocated to equipment (1% and 2%) and rent and utilities (6% and 11%) is relatively low overall. This is likely to have a detrimental impact on the NAB's capacity to achieve the specific objective to "facilitate establishment and maintenance of production facilities".

Figure 6: Distribution of NAB funding by expenses categories

Timeliness of NAB project approval processes

The NAB Program tracks the dates at which applications are received by the program (i.e., once an application is received, the date is stamped on the documentation), the date when the program recommended approval, as well as the date when the funding is approved by the minister's office (i.e., date on the approval letter).

The evaluation found that important delays are observed in the time that the approval process takes, but appears to be improving within the past few years. During the period covered by the evaluation, the average time for approval did not meet the service standard of 210 days (although it was quite close in 2009-10). As illustrated in Figure 7, project approvals in 2009-2010 took an average of 214 days, increasing to 265 days on average by 2011-12. The past three years have demonstrated some decrease in this falling to 237 days in 2012-13, 238 in 2013-14 and for the most recent year is down to an average of 181 days (2014-15) which is within the current service standard. As illustrated in Figure 7, a portion of the delay is explained by the number of days that a project waits for approval decision once the program has analyzed the application and provided a recommendation, although for two years (2011-12 and 2012-13), the main contributors to delays appear to be program processing time.

Figure 7: Days elapsed from funding request to approval decision – NAB Projects (standard 210 days)

From the administrative data review, the evaluation found that due to the delays in approval, projects during the period covered by the evaluation were starting on average 131 days (2012-13) to 192 days (2010-11) later than their originally planned start dates. As illustrated in Figure 8, the overall length of projects were impacted with the proportion of fiscal year available to execute the project being reduced on average to 47% to 64%. The most recent year (2014-15), while not within the scope of the evaluation, demonstrates a marked improvement at 87%. The evaluation found that completing the Budget Analysis Tool required a large amount of time through discussions with program staff. It was noted that the NAB competition closure dates range from November through January generally which would allow for approximately 90-150 days to process the applications and make decisions prior to an April 1st start date. This does not fit with the stated 210-day service standard. Assuming projects are aiming to start around the start of the fiscal year, it would be important to align the application date accordingly (for example: August-September time period).

Other impacts as result of these delays noted from the file review included:

• challenges with the timing of project expenses and programming schedule; and
• for some organizations, where the APP contribution counts for more than 50% of their total funding (Figure 5.3), delays can have a significant impact including, as mentioned by a few external key informants, lay-off of staff.

Figure 8: Proportion of fiscal year remaining in which NAB project can be implemented post-approval decision

Overlap and duplication of APP with other programming

According to the literature review and key informant interviews with experts, the number of programs aimed at supporting Aboriginal arts, cultures and languages is large and encompasses programming and resources from the federal, provincial and territorial governments. To determine the extent to which programming and funding is duplicative or complementary is further complicated by the diversity of Aboriginal people, their living situations, and the various jurisdictional issues (for example: on/off reserve; urban/rural; minority/majority situation).

Based on the findings from key informant interviews and the noted gaps between need and resources available, despite the large amount of programming in this area, much of this overlap is likely complementary rather than duplicative. The exceptions may be in broadcasting and with ALI, as it was noted that within the TLAs funding is being made available to northern community projects that focus on television and radio broadcasting in Aboriginal languages as well as use and exposure to Aboriginal cultures and languages. Other areas where potential overlap and duplication may exist is with INAC educational programs that have an Aboriginal language and culture components, and provincial/territorial governments that also support different types of programs for off-reserve/urban Aboriginal groups. In this regard, the Minister's mandate letter, as mentioned in section 4.2, suggest that PCH works collaboratively with INAC to leverage programs, services and funds for a greater impact.

Performance monitoring

The evaluation found that there were some issues with respect to performance measurement and monitoring. Interviews with internal key informants noted challenges with the utility of performance information. The document review noted that the 2011 PMERS for the APP outlined a number of performance elements, indicators and data sources that were to be collected on an annual basis. As of 2011, these were considered "under development" with respect to establishing baselines and targets, and that many of them had not been developed as of 2014. This is not particularly surprising given the large scale changes that occurred with the APP program over this period, making the relevance and applicability of the previous 2011 PMERS perhaps questionable.

The observations from the administrative data and file review are that some of the information for proposed indicators are contained in the narrative final reports that match the contribution agreements. Unfortunately, this makes it challenging to compile and analyze on an ongoing basis. Similarly, there are some challenges with comparability of base units of analyzes for measuring results. For example, "projects" are used throughout the list of indicators, but this appears to vary considerably as to what constitutes a project, potentially resulting in large discrepancies when analyzing at the component or program level. Key informants mentioned in interviews that work has begun on a preliminary analytical framework that would align data from final reports with program expected outcomes. No completion date has yet been confirmed.

Merit-based competitive process for NAB funding

As noted previously, up until 2011-12, the NAB funding had been allocated to the same 13 organizations based on previous historically allocated amounts. As of 2012-13, the process was changed so that the allocations would become merit-based through a competitive proposal-driven process. One particular challenge noted with respect to this change in process included the increased resource requirements on the part of the individual organizations and PCH staff to undertake a competitive process. This increase in resources was required, while the amount of funding remained the same, and the same 13 organizations were funded throughout the five years covered by the evaluation. While there were a few additional applications received from other organizations during this period, none of them were funded.

Adding to this frustration as noted through key informant interviews with external respondents was the seemingly changing annual priorities that were not announced prior to the call for letters, so proposals could not be effectively shaped to address these priorities, the perceived vague eligibility criteria, and lack of clarity with respect to the basis for funding allocations.

The evaluation found that through an analysis of call letters during the period covered by the evaluation, there was no indication of annual priorities relayed to organizations through the letter. It was noted that as the process changed to a competition, the call letter indicated that the funding approved may be less than that requested, and that the process was competitive and applications were to be decided based on merit. This vagueness had an impact on the predictability, the planning and the ability to measure the performance of the program. For recipients, especially when a high ratio of the budget is not stable, it is difficult to do long term planning and to achieve longer term outcomes. For PCH, this short term approach to planning also had an impact on the internal resources. When completing applications, potential recipients must frequently communicate with program officers for clarification. The application process was perceived, by both internal and external informants, as opaque and driven by risk aversion.

Multi-year funding agreements

The evaluation found through the case studies and key informant interviews that there were various challenges that were encountered with the one-year agreements that were further exacerbated with the significant annual delays in funding decisions. Considerable support from both internal and external KIs for multi-year funding arrangements was observed. Potential benefits cited include financial stability for organizations, reduced administrative costs, reduced uncertainty and better planning.

Program design responsiveness and ability to achieve outcomes

As outlined previously in Table 1, during the period covered by the evaluation and to date, the APP has undergone major structural changes. Significant proportions of program elements making up essentially one entire component (Aboriginal communities component) have been transferred from PCH to AANDC which account for approximately 70% of the original funding. Within the remaining smaller component (Aboriginal Living Cultures), there has been a decrease in funding.

In 2009-10, the design of APP was relatively intricate with multiple program elements supporting various sub components which in turn were supporting two distinct overall program components. Target populations, activities, and outputs were numerous and diverse. By 2015-16, the program design structure has been significantly modified in scope, and the integration of multiple program activities involved in contributing to the complex outcomes identified for APP is no longer apparent within the remaining program structure.

As noted in the document and literature review, and supported by the findings from key informant interviews and case studies, the TLA, NAB and NAD program elements provide tangible support for Aboriginal language and culture. Similar findings were noted for ALI. While this support and responsiveness is evident, all lines of evidence and the ALI evaluation noted that the needs are far greater than what the current structure and funding of the remaining APP project elements are able to address. Adding additional demands to the context for APP are a number of emerging issues (as noted in Section 5.1), and an increasingly higher level of priority being assigned by the federal government to address issues of Aboriginal language and culture through partnering and collaboration. It was noted among key informants and case studies that the APP may contribute in part to addressing needs, but considerable efforts and involvement from many other stakeholders are required to fully address the language and cultural needs of Aboriginal people in Canada. In its current state, the ability of the program in achieving its outcomes is questionable.

Relevance

The APP remains relevant. All lines of evidence found that there continues to be a strong, demonstrable need for support to Aboriginal languages and culture. Similarly, the evaluation of ALI concluded that there is a continuing need to support the preservation and revitalization of Aboriginal languages. These needs have been highlighted in recent federal commissions, and by various researchers. Additionally, the evaluation found that there is a strong need for Aboriginal broadcasting organizations to be financially supported in order to be able to broadcast relevant information and content to Aboriginal audiences (particularly in more remote locations).

The APP, by itself and assuming its actual program design structure is maintained, is unlikely to effectively address the decline of Indigenous languages used as a mother tongue given the current status of many of these languages, and the complexity of reversing the decline as well as to respond to emerging needs. The evaluation did find however that the APP has contributed to the promotion of some Aboriginal languages and has likely contributed to revitalizing Aboriginal languages as a second language.

The APP is aligned with federal government priorities as outlined in the 2014 Budget Speech which indicated that the government would continue to support efforts to preserve Aboriginal languages. As well, the APP continues to be aligned with the more recent December 2015 Speech from the Throne which stated the intent of the government to renew the relationship between Canada and Indigenous peoples and to support the implementation of recommendations from the TRC, many of which deal with Aboriginal language and culture.

The APP aligns with Strategic Outcome 2 – Canadians share, express and appreciate their Canadian identity; and is aligned with PCH priorities of the celebration of Canadian history and culture and the investment in communities. The APP also aligns with the current PCH Minister's mandate as outlined in the letter from the Prime Minister.

There is a role for the federal government in supporting the preservation and revitalization of Aboriginal language and culture. This is supported by PCH's responsibility for Canadian identity and values, cultural development and heritage. As well, the result of the TRC strongly acknowledges the Canadian government's role in the Residential Schools system, and the resulting Calls to Action.

Performance - Achievement of expected outcomes

There is a misalignment between the program objectives and its expected outcomes. The APP has made progress on achieving immediate outcomes and making contributions towards intermediate and ultimate outcomes. The APP has achieved the immediate outcome of Aboriginal communities accessing resources to deliver projects that incorporate Aboriginal languages and culture. Across the period covered by the evaluation, approximately $11 million per year were distributed through Gs&Cs to various organizations and communities for projects under TLA, NAB and NAD related to Aboriginal language and culture. An additional $3 million to $4 million of Gs&Cs funding was distributed annually through ALI to community organizations. The types of projects funded and resulting products are quite diverse under the various APP program elements.

Nevertheless, there remain considerable gaps between funding available and needs in this area. Funding from APP has remained constant or decreased over the period covered by the evaluation. During this same period, needs have been increasing, particularly with NAB given challenges with aging equipment and infrastructure. ALI was only able to fund 28% of applicants.

The APP has also contributed to the intermediate outcome of Aboriginal individuals and groups engaging in activities that strengthen some Aboriginal languages and cultures. Even though the APP is limited with respect to the number of recipients involved, the evaluation found examples where APP projects have resulted in tangible outcomes by moving the use and exposure to Aboriginal culture and language into public spheres and delivering language and culture support in an appropriate manner. The majority of ALI project participants reported that ALI has helped them to embrace their Aboriginal culture, language and identity.

The APP has made some limited contributions to ultimate outcomes. APP funds NCR projects in which Aboriginals are sharing their languages and cultures with other Canadians. As well, NAB funded organizations that are broadcasting over large stretches of Canadian territories, reaching non-Aboriginals living in these regions who appreciate news, weather, information and Indigenous content. Most ALI participants report that they are sharing their Aboriginal culture, language and identity with their community, and have become more engaged in their communities.

While the results of the current program activities demonstrate that there has been some achievement of program objectives, the APP's reduced scope and funding limit its capacity to contribute to the preservation, revitalization and promotion of Indigenous languages and culture in Canada.

Performance – Efficiency and economy

The APP has operated relatively well from an economy perspective given the large scale changes in program structure during this period. The APP overall reported slight variances between budget and actuals for four of the five years covered by the evaluation ($0.05 million to $1.29 million). Exceptionally in one year, the variance was larger at $6.02 million, but this was also the year during which the large scale changes in APP structure, and the transfer of components from PCH to AANDC began to occur.

The APP has operated relatively efficiently, particularly for the initial period covered by the evaluation. The proportion of administrative costs to total budget remained relatively constant (between 8% and 9%) for three years of the program (2009-10 to 2011-12). During 2012-13 when much of the restructuring and movement of components to AANDC occurred, the proportion of administrative costs nearly doubled to 16% before decreasing back to 12% in 2013-14. The trend towards decreasing has continued with administrative costs at slightly less than 11% in 2014-15.

The APP has experienced some challenges with respect to timeliness project approvals for the NAB program element. This area was also identified as challenging for ALI. With respect to NAB, the evaluation found that important delays are observed in the time that the approval process takes, but appears to be improving within the past few years. During the period covered by the evaluation, the average time for approval did not meet the service standard of 210 days (although it was quite close in 2009-10). Delayed approval had significant negative impacts on the organizations that receive funding, and ultimately the outcomes achieved by the APP.

Other evaluation questions

Performance monitoring for the APP could be improved. The performance monitoring appears to have been focused primarily on activities and some outputs. This makes it challenging to understand what actual outcomes have occurred as a result of the APP. The use of primarily narrative reports for results with no baselines presents challenges in extracting, analysing and comparing performance information across projects, although work is ongoing in this area.

Multi-year funding is viewed by many as desirable for NAB and ALI funding. The ALI evaluation findings were similar to those for the current evaluation that found there is considerable support from both internal and external KIs for multi-year funding arrangements. Potential benefits cited include financial stability for organizations, reduced administrative costs, reduced uncertainty and better planning.

APP design and ability

The APP is in a fragile state which leads to questions its ability to achieve its outcomes and its responsiveness under the current design. Since 2009-10, the APP program design structure has been significantly modified in scope, and the integration of multiple program activities involved in contributing to the complex objectives identified for APP is no longer apparent within the remaining program structure, conducing to questioning if the program has a reasonable chance of succeeding. All lines of evidence for the current evaluation along with the findings from the ALI evaluation noted that current and emerging needs related to Aboriginal culture and language are far greater than what the current structure and funding of the remaining APP project elements are able to address.