Evaluation of the Exchanges Canada Program

Relevance

The literature identified a number of factors that demonstrated a continued need for young Canadians to increase their knowledge and understanding of Canada, to have opportunities to connect and create linkages with each other and to develop a Canadian identity and a sense of belonging to Canada. By covering travel expenses, the Program supported not-for-profit organizations in providing youth with a range of exchange and forum activities to strengthen their sense of belonging to Canada and, therefore, their sense of Canadian identity. Respondents from all groups indicated that without PCH funding, national exchange and forum activities would not take place at the current scale (e.g., national, diversity of participants) or would not take place at all.

The ECP objectives and expected results aligned with federal government priorities as outlined in speeches from the throne and recent federal budgets. ECP was consistent with a federal government commitment to support communities, culture and heritage and groups that face greater challenges and experience barriers, such as at-risk youth. Similarly, ECP objectives and outcomes were closely aligned with PCH's strategic outcome: Canadians share, express and appreciate their Canadian identity. Overall ECP was aligned with the departmental priority: Invest in our communities.

Key informant interviews showed support for a federal government role in a program such as ECP, given its national scope and PCH's expected outcomes in the areas of attachment to Canada and shared Canadian identity.

Performance – achievement of expected outcomes

ECP made progress toward the achievement of its immediate outcome: Young people have access to a wide variety of exchange experiences, including exchanges related to official languages. ECP provided youth a wide variety of domestic exchange experiences in terms of location, content, duration and design and included exchanges related to official languages. Between 2009-2010 and 2013-2014, ECP had approximately 63,000 participants. The YEC and YFC components accounted for approximately 53% and 40% of ECP participants, respectively. SWSE accounted for 7% of ECP participants.

A variety of organizations were funded by ECP which, in turn, offered a diversity of experiences and a range of activities and program designs. ECP projects provided exposure to both official languages. Results from the post-participation survey also indicated that participation had impacts on participants' knowledge and interest in their second official language.

The two anticipated intermediate outcomes of young people from across Canada participate in youth exchanges and exchange participants reflect the diversity of the Canadian youth population were achieved to some extent. ECP had youth participants from all provinces/territories. However, there were some challenges with obtaining a distribution of ECP participants according to region of origin that was comparable to the distribution of youth in Canada. Overall, the participants in ECP reflected the distribution of the Canadian youth population for all provinces except Ontario. Ontario youth were significantly under-represented in both YEC and YFC. Quebec youth were less likely to participate in YFC events. However, the corollary of this is that the other provinces and territories were over-represented for YFC and YEC. In the case of SWSE, Quebec youth were significantly over-represented, as would be expected as it is a French-English language exchange program. Youth from all other provinces and territories, with the exception of Prince Edward Island, were under-represented.

ECP participants were primarily younger youth (under 19), distributed across a broad demographic profile, including traditionally under-represented youth. There appeared to be some barriers for young men participating in ECP, as a disproportionate number of young women were participants. Across all components of ECP, there was a higher proportion of young women (63%) participating in ECP, compared to young men. This was particularly the case for SWSE, where 73% of participants were young women.

Overall, there was considerable diversity among ECP participants as they included Aboriginal youth, youth from low-income households, visible minority youth, youth with disabilities and rural youth. Participation of rural, low-income and Aboriginal youth exceeded their incidence in the general youth population in Canada. Although the participation of youth with disabilities appears to have improved in the past three years, the broadening of the definition of disability may be contributing to this increase. Overall, visible minority youth were under-represented relative to the general youth population.

Comparing across components:

• YEC exceeded targets for Aboriginal, low-income and rural youth and youth with disabilities, but did not meet targets for visible minority youth.
• SWSE had challenges in achieving representation of Aboriginal and rural youth and youth with disabilities but was more successful at achieving representation of low-income and visible minority youth.
• YFC exceeded targets for Aboriginal youth over all the years. It has improved the participation of rural youth, exceeding the target in the last two years. Similarly, in the last three years, it has exceeded the target for youth with disabilities. Visible minority youth were under-represented relative to the general youth population.

Overall, ECP contributed to the three anticipated ultimate outcomes. A large majority of participants reported that they had enhanced their knowledge and understanding of Canada, connected and created linkages with one another and enhanced their appreciation of the diversity and shared aspects of the Canadian experience (> 80% for all indicators). Parents and organizers perceived an even greater impact. The strongest outcome was in the area of connecting and creating linkages with one another. All key informants noted that the physical, in-person aspect of the exchanges was a contributing factor to the achievement of this outcome. Across the three ultimate outcomes, Aboriginal youth and youth with disabilities were less likely to indicate that these outcomes had occurred for them.

In addition to the anticipated outcomes, ECP contributed significantly to other positive outcomes. Some of these were in the area of personal skills that could be considered as facilitators or levers to obtaining the anticipated outcomes for ECP. As well, they are key considerations for meeting various needs of youth from a developmental perspective including a range of employability/soft skills and increased interest in continuing education. Overall, these were strong results for SWSE participants and visible minority youth. There were also positive results for Aboriginal youth and rural youth with respect to influencing their decision to continue their education.

Another area in which ECP was found to be demonstrating an impact was in increasing levels of attachment to community/region, province/territory and Canada. This is an important outcome given ECP's link to the PCH strategic program outcome—attachment to Canada. This was a very strong result for YFC participants and was an area of considerable positive impact for both Aboriginal and visible minority youth.

Performance: Economy and efficiency

Over the five-year period covered by the evaluation, there was an overall variance of 5% between actual expenditures and reference levels, primarily due to reduced O&M expenditures across the five years studied. Over this period, Gs&Cs expenditures remained fairly stable while O&M expenditures were reduced by half.

The ratio of O&M expenditures to total ECP expenditures over the five years covered by the evaluation was 8.9%. This ratio steadily decreased from 12% in 2009‑2010 to 7% in 2013-2014 providing evidence of improved program efficiency. In comparison with other PCH Gs&Cs programs recently evaluated, the five year ratio was higher than the ratios for the Young Canada Works (YCW) Initiative (6.2%) and the ratio of the Canadian Studies Program/Canada History Fund (CSP/CHF) (5.5%). However, it was lower than the ratio for Youth Take Charge (YTC) (14%).

The PCH contribution per participant varied considerably by component and subcomponent, but was more comparable once the duration of activities was considered. A more appropriate metric, for comparison purposes, is the PCH contribution per participant per day of activity. Using five-year averages, the calculations showed that the PCH contribution per participant per day was $129 for SWSE, $90 for YEC and $152 for YFC.

With respect to the PCH contribution as a proportion of total project costs, over the five years covered by the evaluation PCH contributions covered on average 94% of the annual project costs for SWSE and 68% of the annual project costs for YEC. In the case of YFC, there was considerable leveraging of PCH contributions, with the PCH contribution as a portion of annual project costs averaging 23% for the five years.

The ECP design and model provided adequate flexibility and guidance for effective delivery of the program. The physical exchange experience appeared to be a contributing factor to achieving outcomes. Funding recipients identified opportunities to improve the timeliness of the funding decision, which had an impact on the delivery of projects, and to reduce the complexity of the application process.

PCH invests about $75.6 million annually in programs that benefit youth. There are some programs, including the CHF and the Canada Arts Training Fund that are not exclusively aimed at youth but have a broad target group that includes youth. An analysis of documents from these other PCH programs found that when detailed objectives, results, activities and delivery mechanisms were examined within the context of the evaluation, there was limited evidence of overlap with other programming. ECP shares some aspects with YTC (target populations, some theme areas and some similar outcomes); however, the activities and overall approaches were relatively distinct when closely examined. One indication of this was that there was little overlap in funding recipients for the delivery of the two programs.

Virtual exchanges were explored as an alternative approach to deliver ECP. While they may have some benefits, they are unlikely to achieve the same anticipated outcomes as physical exchanges. Virtual exchanges may be a beneficial complement to the physical exchange activities being undertaken, but are not a replacement.

An examination of other domestic exchange programs found that ECP is unique in supporting domestic Canada-wide exchanges and forums by providing funding that is used almost entirely to support travel. Other programs examined that offered domestic exchanges tended to focus only between specific regions of Canada or target specific groups. ECP, however, encourages exchanges between all provinces and territories and provides opportunities for all Canadian youth to go on exchanges.

Performance - performance measurement and monitoring

A Performance Measurement, Evaluation and Risk Strategy (PMERS) was developed and implemented for ECP. The Program has identified performance indicators for each planned outcome of the Program corresponding to the logic model. These outcomes were tracked and analyzed annually. PCH officials considered the current performance measurement and other monitoring mechanisms as effective in capturing results.

However, there are opportunities to improve the analysis and use of performance data collected from funding recipients for decision-making and program improvements. The program produced annual reports on project participation, project costs and performance relative to target populations. The Policy Research Group (PRG) at PCH produced a report annually on the results of the post-participation surveys. A one-time trend analysis report on participation numbers was completed for the period 2000 to 2012 but this analysis has not been repeated. The information collected from funding recipients should be analyzed over a longer period to assess trends in the achievement of outcomes by component and by target group and to compare projects costs relative to outcomes to determine which projects are most cost-effective. This information could inform program improvements and project selection.

The final project reports are another potential source of information to identify common issues across projects, trends and potential areas for improvement. In interviews, PCH officials indicated that there were opportunities to use more of the information collected in recipient reports. However, they noted that this information is not currently being analyzed because of a lack of staff resources. Consideration should be given to supplementing the quantitative results of the post-participation surveys of participants, organizers and parents with the qualitative information obtained from funding recipients, specifically their perceptions of the achievement of project outcomes, challenges encountered, lessons learned and opportunities for improvement.

Funding recipients identified issues with the content and administration of the participant survey. Among the issues raised by funding recipients were that the survey is long, the language of the survey is difficult for youth to understand and it is difficult to administer the survey. Some funding recipients suggested that the survey be made available online. A pilot of an online survey was undertaken with six organizations in 2012-2013 and 2013-2014. Response rates for the online survey varied considerably, ranging between 9.6% and 96.5%. The four pilot organizations with the largest number of participants had the lowest response rates, which had a significant impact on the overall response rate for the participant survey in the two years of the pilot. However, the two pilot organizations with fewer participants had a response rate of over 90%.

Recommendation #1

To improve the efficiency of the funding application, the Assistant Deputy Minister of Citizenship, Heritage and Regions should streamline the application process and monitor the funding decision process to ensure timeliness of the decision to release funds and reduce the complexity of the application process.

Recommendation #2

To strengthen the collection and use of outcome data for management decision-making and program improvements, the Assistant Deputy Minister of Citizenship, Heritage and Regions should:

• 2.1 consult with funding recipients to modify the language and content of the post-participation survey instruments, particularly for participants;
• 2.2 provide funding recipients with the option to conduct on-line surveys and consult with recipients to identify barriers to administering the surveys on-line; and
• 2.3 collect and analyze the qualitative data in the final progress reports submitted by funding recipients to supplement the quantitative outcome data collected through post-participation surveys with funding recipient perceptions of project outcomes, best practices and lessons learned.

Recommendation #3

While overall, ECP program participants generally reflect the demographic and provincial/territorial diversity of the Canadian youth population, certain components of the program are under-subscribed by certain segments of the population (such as young men, Aboriginal, rural and visible minority youth and youth with disabilities). Therefore, the Assistant Deputy Minister of Citizenship, Heritage and Regions should work with funded organizations to identify the barriers to participation and to develop and implement approaches to attract participation of target groups and provinces/territories, as applicable.

Program Management and Governance

ECP is managed by the Citizen Participation Branch, Citizenship, Heritage and Regions Sector, in the Youth Participation Directorate. The Program is delivered centrally at Headquarters in the National Capital Region, through grants and contribution agreements with funding recipients.

PCH program officers oversee the funding agreements to ensure that there is adequate performance monitoring and that all program financial and activity reporting requirements are met.

YEC is delivered through contribution agreements with four pan-Canadian not-for-profit organizations with a proven network of youth contacts: Society for Educational Visits and Exchanges in Canada (SEVEC), YMCA, 4-H and Canada Sports Friendship Exchange Program (CSFEP). SWSE, is delivered through the YMCA.

YFC has two multi-year funding recipients (Encounters with Canada and Forum for Young Canadians), but also hosts an annual funding competition. A general call for applications is put out annually to possible funding recipients to plan and convene a forum event for Canadian youth. In the last five years, between 8 and 15 organizations have received funding annually through the Youth Forums funding competition.

Funding recipients are responsible for reviewing applications for exchanges and forums, matching exchange groups, meeting participation targets, organizing the exchanges and forums, ensuring the safety and security of participants and reporting on results.

Target groups, key stakeholders and delivery partners

ECP benefits, either directly or indirectly, communities, parents, teachers, group leaders and others. However, the primary clients are the exchange participants themselves: Canadian youth, generally between the ages of 12 and 25.

Table 1: Budgeted and actual expenditures 2009-2010 to 2013-2014

Source: STAR

Preliminary consultation

Before undertaking the evaluation, preliminary discussions were held with the ECP evaluation working group, leading to the development of the terms of reference for the evaluation. The terms of reference included a description of the evaluation scope and issues, the methodological approach and a detailed evaluation matrix. The evaluation matrix (Appendix B) identifies the evaluation questions, associated indicators and the lines of evidence to respond to each question.

Lines of evidence

The evaluation featured important strengths including a mix of qualitative and quantitative lines of evidence, a mix of primary and secondary data sources and multiple lines of evidence to allow for the triangulation of evidence. The evidence from all lines was organized and analyzed by core issue, evaluation question and indicator and consolidated into an overall evidence matrix.

The evaluation methodology incorporated five lines of evidence:

• Document review
• Literature review
• Administrative data review
• Interviews with key informants
• Case studies of four organizations that had received ECP funding

Document review

Program and government documents were reviewed to assess the relevance and performance of ECP, particularly its alignment with federal government and departmental priorities and strategic outcomes. Government of Canada (GC) documents reviewed for the evaluation included budget documents and speeches from the throne. Departmental documents included departmental performance reports (DPRs) and reports on plans and priorities (RPPs). A range of program documents were reviewed, among them the Program's administrative reports, terms and conditions, PMERS, previous audit and evaluation reports, contribution agreements with recipients and recipient financial reports.

Literature review

A literature review provided evidence for the relevance of ECP, including the continuing need and responsiveness of ECP. The sources consulted for the literature review were derived from academic publications, monographs, government and non-governmental research reports and websites.

Administrative data review

The administrative data review included a review of quantitative information on ECP activities and results and was an important source of evidence for evaluation questions pertaining to performance (including effectiveness and efficiency).

The administrative data review included the review of a sample of contribution agreements, final reports and financial statements from funded organizations, results from participant surveys and various compilations of program-level administrative data, including financial and performance measurement information.

It also included an analysis of participant, organizer and parent feedback questionnaires, which had been collected by the program between 2009-2010 and 2013-2014. The response rates for the participant and parent surveys are presented in Table 2 below.

Table 2: Feedback surveys completed by year and respondent group

Source: Post-participation survey reports (2009-2010 – 2012-2013); raw data for 2012-2014

The response rate is not available for organizers. The Program provides funding recipients with surveys to distribute to their organizers; however, the total number of surveys actually distributed is not tracked. Between 648 and 856 surveys were returned annually, with an annual average of 728.

Financial data for ECP were also analyzed for the period of the evaluation, including reference levels and expenditures for ECP O&M and Gs&Cs.

Interviews with key informants

The purpose of the key informant interviews was to gather in-depth information, including opinions, explanations, examples and factual information for all evaluation issues and questions. Key informants were selected based on their involvement, knowledge and experience with ECP. A total of 20 interviews were conducted with the following:

• representatives from organizations that received ECP funding (n=11 organizations out of 31);
• organizers (e.g., teachers and group leaders who work with funding recipients) (n=5); and
• PCH officials (n=4).

Case studies

Four case studies of organizations funded by ECP were conducted as part of the evaluation. The objectives of the case studies were to gather information on the relevance of the program and to seek perspectives from former participants themselves on the longer-term impact of participation in the Program. Two case studies were selected from the four YEC projects, one of which examined the YMCA's SWSE initiative, a sub-component of YEC. Two additional case studies were selected from YFC projects. All of the case studies had received funding in each of the five years of the evaluation period.

The methodology for each case included a document review, administrative data review, interviews with funding recipients and interviews with previous youth participants in three of the four cases. Across the four case studies, interviews were conducted with 11 representatives from 4 organizations and with 17 former ECP participants.

The following guidelines were used to report the findings of the surveys and key informant and case study interviews:

Methodological limitations

While the methodology offered a number of important strengths, including the mix of qualitative and quantitative lines of evidence and a mix of primary and secondary data sources, the evaluation encountered some challenges and there were some limitations to the methodology, including the following:

• Reliance on self-reports by participants to assess achievement of intermediate and ultimate outcomes. The evaluation relied heavily on the results of participant surveys to report on intermediate and ultimate outcomes for the projects. Long-term outcomes were difficult to determine as participants usually complete the survey very soon after their participation and there is no comparison group with other youth. The extent to which ECP activities contribute to a particular outcome versus other potential contributors (e.g., natural maturation process and youth development stages) is difficult to determine. To ascertain the longer-term impacts of the Program, the evaluation included interviews with former ECP participants as part of the case studies.
• Restricted economy/efficiency analyses. Some of the planned analyses to assess economy and efficiency could not be undertaken as the financial information was not sufficiently granular (e.g., O&M divided by ECP component), or the project level financial information did not have consistent definitions or collection over multiple years (e.g., in-kind contributions).
• Participant interviews for case studies. One of the case studies did not include interviews with former participants. The evaluation team was unable to contact participants due to outdated contact information and the short time frame in which to conduct the case studies.
• Challenges with reporting on participant survey data over the years covered by the evaluation. The reporting on participant surveys is done on an annual basis. There were some challenges when using these reports over the years as some of the questions were not consistent over the years. To ensure consistency in breakdowns and cross tabulations of data across the years, the evaluation team obtained raw data from multiple years and combined the information in one database to enable a comparative analysis over the five years covered by the evaluation.

Continued need for ECP

The literature identified a number of factors that demonstrated a continued need for young Canadians to increase their knowledge and understanding of Canada, to have opportunities to connect and create linkages with each other and to develop a Canadian identity and a sense of belonging to Canada.

• Knowledge and understanding of Canada. The literature review indicated that concerns remain about the extent to which young Canadians are knowledgeable about Canada (Cowan and Landon, 2011). ^{Footnote 1} The literature review also found that Canadian youth, particularly younger youth, are less likely when compared to older cohorts to be interested in Canadian history, to feel that it is important to national identity or to feel that it would improve their attachment to Canada. ^{Footnote 2}, ^{Footnote 3}, ^{Footnote 4} Further, a round table report by Deloitte (2012) indicated there is a strong case for improved "citizenship" training and awareness among Canadian youth. ^{Footnote 5} The literature review also identified studies that have linked Canadian youth's lack of knowledge about Canada's civic system and political issues with their lower rates of voting and lower levels of participation in more formal types of community and civic engagement. ^{Footnote 6}, ^{Footnote 7} It should be noted that there is no consensus in the literature with respect to this link, as there continues to be debate among researchers about the strength of this link, competing factors and potential moderators of this relationship.

• Diversity and shared aspect of the Canadian experience. Canadians aged 18 to 30 are more supportive of multiculturalism compared to older Canadians and are also more comfortable around visible minorities. Young Quebecers in particular are more accepting of minorities and immigration than older generations of Quebecers. However, paradoxically, younger Canadians are more likely than older Canadians to ask that fewer immigrants be admitted to Canada. The study attributed this paradox to economic and unemployment-related concerns. Younger Canadians were also generally more supportive of diversity in other areas such as lesbian, gay, bisexual and transgender Canadians. Compared to older Canadians, youth are generally more likely to be proud of Canada's diversity or of official protections for minority groups (e.g., the Canadian Charter of Rights and Freedoms, multiculturalism and bilingualism policies). ^{Footnote 8} When asked the question "What keeps Canada united?" Canadians aged 18 to 24 were much more likely to choose 'multiculturalism' compared to the general population (16% versus 8%). ^{Footnote 9} Youth support for official languages and multiculturalism and respect for Aboriginal culture suggests that youth programming has the greatest potential for amplification and for future impact and to solidify their understanding of diversity and build shared experiences of Canada.

• Canadian identity and sense of attachment. According to the literature younger Canadians are less likely to feel pride in Canada/being Canadian and are less likely to feel attached to Canada than their older cohorts. The 2013 General Social Survey on Social Identity found that 56% of youth aged 15-14 described their sense of belonging to Canada as being very strong, compared to 63% of Canadians generally, and 77% of seniors aged 75 years and older. ^{Footnote 10} While some social scientists consider this to be a developmental issue with civic pride and attachment generally increasing with age, ^{Footnote 11}, ^{Footnote 12} others have argued that youth are more likely to feel civic pride and attachment if they feel like they have input into decisions that affect their communities. ^{Footnote 13} Related to this is the finding that having input during youth exchanges or at youth conferences has been shown to have a positive impact on youth engagement. ^{Footnote 14} Studies on young people and civic engagement have demonstrated that young people who have input into the activities they are involved in are more likely to feel engaged and accomplished. This is noted as being particularly important at the developmental stage of their lives when it is important to begin making independent decisions. ^{Footnote 15} The review also found that ensuring that young people feel they have input and that their "voices are being heard" often correlates positively with involvement and engagement. ^{Footnote 16}, ^{Footnote 17} Findings from the literature review also indicated that there is a strong link between learning the other official language and feeling an attachment to Canada. One study found that over half of 18 to 24 year-olds and 25 to 34 year-olds (58% and 54% respectively) indicated that learning the other official language would strengthen their attachment to Canada. In addition, these groups (25% and 16% respectively) indicated that learning the other official language would have the largest impact on their attachment to Canada when compared with other potential factors. ^{Footnote 18}

• Inclusion of youth from under-represented groups. Under-represented youth (economically disadvantaged youth, Aboriginal youth, youth with disabilities, immigrant youth, religiously, ethnically or linguistically diverse youth, Francophone youth and youth in rural or remote communities) generally exhibit specific limitations to involvement. Common to all mentioned groups is limited access to sufficient financial resources, as well as a feeling of isolation from mainstream society. ^{Footnote 19}, ^{Footnote 20}, ^{Footnote 21}

Findings from the interviews with PCH officials, funding recipients and organizers and the case studies further supported the findings of the literature review. In interviews, PCH officials described needs from the perspective that youth have lower rates of attachment and engagement in civic institutions and the political process.

During interviews, funding recipients and organizers highlighted several needs of Canadian youth, including the need to better understand Canadian history, its diversity and the role of the federal government; the opportunity to combine information gathering through secondary media (e.g., books, internet, television) with experiential opportunities outside their own community; and the opportunity to share their appreciation for their community, their history and their culture with others. Key informants all highlighted the need for a program such as ECP, in particular for youth from under-represented groups and from low-income families.

Youth interviewed as part of the case studies were unanimous that there is a need for a program like ECP. Among the reasons provided were that it gave them an opportunity to expand their knowledge and horizons through exposure to new cultures and other parts of Canada and to improve their second language skills. It also gave them a chance to challenge themselves and become more independent. In addition, youth indicated that ECP responded to their need for professional growth and second language skills.

An area that was noted by many key informants was the need for youth to have opportunities for physical, face-to-face/experiential opportunities. Funding recipients noted that increased virtual communications have brought new dynamics to youth issues. Youth are savvier with, and increasingly attached to, virtual communication devices. PCH officials noted that this has resulted in increased online activity and civic engagement via online means (e.g., petitions, social media campaigns), while at the same time there is a declining interest in civic institutions and traditional means of participation. Organizers of exchanges highlighted that technology creates both new opportunities and new challenges for exchanges programming. While technology makes it easier for youth to get connected and stay connected both before and after an exchange, programs need to find new ways to successfully engage youth and, at times, to focus on the experiential opportunities over virtual/digital ones.

Despite the predominance of virtual communications in the lives of youth, in case studies and interviews, all groups noted that youth have an interest in experiential learning and that learning beyond the textbook approach of the education system was beneficial to them.

Youth who were interviewed as part of the case studies also highlighted the importance of the physical exchange experience and were unanimous that it was a key strength of the program, indicating that the Program would not have had the same impact if it did not involve an in-person experience.

The continued need for ECP is further illustrated by the fact that respondents from all groups indicated that without PCH funding, national level exchange and forum activities either would not take place at the current scale (e.g., national, diversity of participants), or would not take place at all.

ECP's responsiveness to the needs of youth

The evidence indicated that the design and implementation of ECP responded to the needs of youth in the following ways:

• Focusing on the physical exchange experience and the skills acquisition, learning and connections associated with this approach. In interviews, respondents identified the need for youth to have opportunities to combine information gathered through secondary media with experiential opportunities outside their own community. Similarly, some respondents noted that youth reliance on social media has, in part, contributed to decreased face-to face/experiential opportunities.
• Requiring projects to have history and heritage, as well as official language, components. Projects provide opportunities for youth to learn first-hand about the history, geography, industry, institutions, cultures, communities, languages and other facets of their country. A significant proportion of ECP projects focus on exposure to participants' second official language.
• Setting specific targets for participation by members of traditionally under-represented groups who may not otherwise be able to participate without PCH funding. Targets exist for Aboriginal, visible minority, low-income and rural youth and youth with disabilities. Targets are based on Census 2006 (for 2009-2010–2012‑2013) and National Household Survey (NHS) (for 2013-2014) data on the representation of these groups in the Canadian population. Many projects make adjustments in delivery and recruitment to ensure inclusion.
• Promoting a better understanding of diversity, including linguistic duality, Aboriginal culture and Canada's multicultural character. Considerable efforts are made to ensure that participants are representative of the Canadian youth population. Case studies indicated that many of the opportunities are designed to expose youth to greater multicultural diversity both through forums and exchanges.
• Enabling youth to have direct input during youth exchanges or at youth conferences. Participants play an active role in planning the exchange. For exchanges, this includes communicating with their exchange "twin", conducting research on their host community and participating in fundraising activities.

Funding recipients and PCH officials indicated that ECP is responsive to current and emerging needs and priorities of youth. The main reason provided by key informants from both groups is that ECP's design is flexible and allows those that understand youth best—the delivery organizations—to develop the project design. PCH officials added that ECP is responsive to the needs of stakeholders, by providing programming that is relevant to teachers, youth-serving organizations, youth and the Canadian public through a core group of well-established stakeholder organizations.

Alignment with Government of Canada priorities

The document review and key informant interviews demonstrated alignment between ECP objectives and GC priorities. The 2011 federal budget indicated support for communities, heritage and culture and steps to eliminate barriers for those groups that face greater challenges and experience barriers such as at-risk youth. ^{Footnote 22} This aligns with ECP efforts to reduce barriers to participation in exchanges among traditionally under-represented groups of youth.

Government priorities linked to ECP are also identified in the 2012 budget speech which outlines significant investment ($335 million) in culture and the arts. Similarly, the Government's 2012 Economic Action Plan outlined the priorities of "giving our young people the opportunities they deserve and we will achieve that by funding programs that benefit large numbers of young people at a reasonable cost rather than concentrating available funding on a very small number of participants at an excessive per-person cost. Our Government is proud to continue to invest in affordable, effective programming that engages youth, including Encounters with Canada, Forum for Young Canadians, and organizations that support youth, like the YMCA. Canadian Heritage will continue to invest over $105 million in youth programming to allow almost 100,000 young people to learn about their country." ^{Footnote 23}

The Speech from the Throne for the 41st Session of Parliament outlined the government priority of recognizing the importance of celebrating heritage and promoting values, concepts closely aligned with ECP outcomes. ^{Footnote 24}

All PCH officials provided examples of how ECP aligns with federal government priorities. ECP aligns with the 2013 Speech from the Throne, which put a focus on Canadians understanding their heritage and history on the "road to 2017." ^{Footnote 25} The SWSE subcomponent of YEC is also indirectly aligned with GC economic priorities and its focus on creating jobs.

Alignment with PCH priorities and strategic outcomes

PCH's DPRs and RPPs indicated alignment with PCH's priorities and strategic outcomes. According to the 2013-2014 DPR, ECP falls under PCH Program: Attachment to Canada. PCH uses ECP to create opportunities to increase knowledge of Canada and offer experiences leading to a greater understanding of the country. ^{Footnote 26}

ECP contributed to PCH Strategic Outcome #2: Canadians share, express and appreciate their Canadian identity by fostering among youth a stronger Canadian identity through active, engaged, inclusive citizenship and recognition of the importance of both linguistic duality and a shared civic identity.

Based on the 2013-2014 DPR, ECP aligned with PCH Priority #3: Investing in our communities which is focused on helping to connect communities, contribute to healthy and vibrant communities and develop a strong sense of Canadian identity. ^{Footnote 27}

In 2013, the GC announced measures to improve access to funding for local organizations that wished to promote Canadian history in their communities, including local communities and youth groups. ECP allocated $3.6 million per year to "provide young Canadians with more opportunities to take part in history-themed events" and to support "enhanced historical content during all reciprocal exchanges and forums attended by youth so they can discover the people, places and events that make our country unique." ^{Footnote 28}

In interviews, all PCH officials indicated that ECP is aligned with PCH priorities and strategic outcomes. They also noted that when history and heritage became a departmental priority, ECP contributed to this priority by providing young Canadians with more opportunities to take part in history-themed events and by supporting enhanced historical content during all reciprocal exchanges and forums attended by youth.

Access to a variety of exchange experiences

The immediate outcome for ECP is that "young people have access to a wide variety of exchange experiences, including exchanges related to official languages." Evidence indicated that during the period from 2009-2010 to 2013-2014 approximately 63,000 youth participated in a broad variety of exchange experiences.

Figure 1 provides an overview of the number of participants by Program component and subcomponent between 2009-2010 and 2013-2014. Over this period, participation in ECP‑funded projects fluctuated between 12,000 and 13,000. The total number of participants over the five-year period was 62,811. The average number of ECP participants per year was 12,576.

Figure 1: Number of participants by YEC and YFC component and SWSE subcomponent

Figure 2 shows that over the period of the evaluation, YEC projects accounted for slightly over half (33,852 or 53%) of participants. Its sub-component, SWSE, accounted for an additional 4,218 or 7% of participants. YFC projects accounted for approximately 25,079 or 40% of participants during this period. Within the YEC component, SEVEC exchanges accounted for approximately two-thirds of participants (65%). Similarly, Encounters with Canada accounted for two-thirds of YFC participants (65%).

Figure 2: Percentage of participants by component/subcomponent

Overall, participation in ECP-funded projects declined slightly (1%) over the five-year period, with a 6% decrease in participation in the YEC component (excluding SWSE), offset by increased participation in the SWSE subcomponent of YEC (4%) and in the YFC component (11%). According to program respondents, static resource levels, coupled with increasing travel costs, make it difficult for organizations to retain constant numbers of participants.

Findings from the administrative data indicated that over the period of the evaluation YFC events were held in a variety of locations in all regions and the territories. However, most events were held in Ontario (56%). Of the events held in Ontario, 87% were held in the National Capital Region. The remaining forums were generally equally distributed among the other regions (see Figure 3).

Figure 3: Number of YFC events per region

Figure 4: Region visited by participants (%)

Key informant interviews and the case studies indicated that ECP provided young Canadians with a multitude of exchange experiences, ranging from reciprocal group exchanges among younger students to longer-term employment-related exchanges among senior high school students supported by post-secondary students.

Youth were exposed to a variety of subject matter including Canadian politics (trips to Ottawa to meet with Members of Parliament, francophone culture (Les Jeux de la Francophonie) and northern Canada (Students on Ice). The two case studies on forums also demonstrated a wide variety of topics and themes ranging from Canada's role in the Commonwealth to culture and the arts, mental health, security and sciences. All case studies included specific activities related to history and heritage.

Second official language exposure

According to the terms and conditions of ECP, the Program must ensure that a significant proportion of exchanges (25% to 30%) are related to official languages (i.e. exchanges that encourage second-language use/exposure or allow official language minority groups to meet youth from other regions of the country who speak their language).

The document review, case studies and key informant interviews indicated that ECP has offered opportunities to participants to be exposed to both official languages. Exposure to both official languages was highlighted by project representatives as a key component in all four case studies. Similarly, in interviews all PCH officials indicated that ECP has met or exceeded its stated outcome of participants using or being exposed to their second language, namely through experiences like the SWSE and organizations that offer language-based exchanges (e.g., SEVEC).

In interviews with youth who participated in SWSE, all respondents reported that they had improved their second language abilities as a result of the exchange. Some indicated that they improved their language test scores after their participation. For most participants interviewed, participation in an exchange or forum contributed to a greater interest in learning Canada's other official language or an awareness of the importance of speaking Canada's other official language.

Most funding recipients described their projects as officially bilingual, with bilingual events, staff and communication tools (such as a bilingual website, printed materials and simultaneous translation). For example, the Encounters with Canada project included sessions that focused on promoting the benefits of bilingualism.

Each year ECP staff estimate participation in exchanges and/or events with an official language component. ^{Footnote 31} Although the total number of participants in ECP remained relatively stable over the five years covered by the evaluation (between 12,000 and 13,000 participants), the estimated representation of participants in a project with an official languages component fluctuated between one-third (37% in 2009-2010; 35% in 2013-2014) and just under two-thirds (62% in 2010-2011 and 2011-2012; 63% in 2012-2013) of total participation. This variation may be explained by differences in project offerings from year to year.

Although PCH estimates range from 35% to 63% for the number of participants involved in an ECP project with an official languages component, the actual exposure may be higher. Post-participation survey ^{Footnote 32} results found that as a result of their participation in the ECP project 71% of participants in 2011-2012, 76% in 2012-2013 and 74% in 2013‑2014 indicated that they had enhanced their knowledge and awareness of the other official language either moderately (20-22%), significantly (30-32%) or extremely (19‑23%). This may suggest that although a project may not have a specific official language component, the experience of travelling, meeting new people and being in a new environment may be contributing to an increased interest in the other official language.

Similarly, about three-quarters of parents and organizers completing a post-participation questionnaire between 2010-2011 and 2013-2014 indicated that participation in ECP enhanced participants' knowledge and awareness of Canada's other official language.

Evaluation questions

To what extent did ECP achieve its intermediate outcomes?

• Young people from across Canada participate in youth exchanges; and
• Exchange participants reflect the diversity of the Canadian youth population.

Participation of young people from across Canada

An intermediate outcome of ECP is that participants from across Canada participate in exchanges. Applications are assessed on the extent to which they ensure that participation is representative of provincial/territorial youth populations. Over the period of the evaluation, the Program established participation targets based on 2006 Census demographic data (for 2009‑2010 to 2012-2013) and the 2011 NHS for 2013-2014 in accordance with the age group of participants. Figure 5 on the next page shows the results for 2009-2010 to 2012‑2013. An analysis of 2013-2014 participation against NHS targets obtained similar results.

The evaluation found that ECP attracted participants from each region in Canada, including the territories. A comparison of ECP participants from 2009-2010 to 2013-2014 reveals that overall most regions, with the exception of Ontario, are well represented or over-represented among ECP participants.

Each case study demonstrated distribution of participants from across Canada. While some YFC events are located in one region (e.g., Encounters with Canada, National Commonwealth Student Forum), the participants are from all regions of Canada.

A comparison of the distribution of ECP participants with the provincial/territorial distribution of youth, based on the 2006 Census for 2009-2010 to 2012-2013 and the 2011 NHS for 2013-2014, found that while overall ECP has participation from all regions, when compared across components, the distribution of participants is not representative of the provincial and territorial distribution of youth in Canada:

• For YEC, youth from all provinces and territories are well represented except in the case of Quebec and Ontario. There was an over-representation of Quebec youth and under-representation of Ontario youth. The over-representation from Quebec may be explained, in part, by the emphasis on official language exchanges.
• SWSE also had an over-representation of Quebec youth. Youth from the other provinces and territories, with the exception of Prince Edward Island, were under-represented. This may be explained by the fact that SWSE is a French-English language exchange, where families who send their young participants to another region of the country to work and learn their second language are also expected to host other young participants in their first language.
• The YFC component has significant under-representation of Ontario and Quebec youth and a disproportionate number of youth from the Atlantic Provinces, British Columbia, the Prairie Provinces and the territories. This was highlighted by the case studies, where there was an indication that forums encounter challenges in attracting and recruiting proportional numbers of youth, particularly from Ontario. In interviews this was attributed, in part, to the fact that many forum events are held in Ontario and may therefore be of less interest to Ontario youth.

The 2010 evaluation of the ECP made similar observations with respect to the under-representation of Ontario youth in ECP.

Performance – achievement of expected outcomes

ECP provided youth opportunities with a wide variety of domestic exchange experiences in terms of location, content, duration and design. Exposure to the second official language appeared to be high among participants. There were some challenges with obtaining a distribution of ECP participants according to region of origin that was comparable to the distribution of youth in Canada. Ontario youth were significantly underrepresented in both YEC and YFC. Quebec youth were less likely to participate in YFC events.

ECP participants were primarily younger youth under 19 years of age, distributed across a broad demographic profile including traditionally underrepresented youth. There appeared to be some barriers for young men participating in ECP opportunities, as a disproportionate number of young women were participants.

Overall, there was considerable diversity among ECP participants with participation of Aboriginal youth, youth from low-income households, visible minority youth, youth with disabilities and rural youth. ECP participation of rural, low-income households and Aboriginal youth exceeded their incidence in the general youth population in Canada. Although the participation of youth with a disability appeared to have improved in the last three years covered by the evaluation, changes to the definition of disability may have contributed to this increase. Overall, visible minority youth were under-represented relative to the general youth population.

Comparing across components, YEC exceeded targets for Aboriginal, youth from low-income households, rural youth and for youth with disabilities, but did not meet targets for visible minority youth. SWSE had challenges in achieving representation of Aboriginal and rural youth and youth with disabilities but was more successful at achieving representation of youth from low-income households and visible minority youth. YFC exceeded targets for Aboriginal youth across all years. It has improved the participation of rural youth, exceeding the target in the last two years. Similarly, in the last three years, it has exceeded the target for youth with disabilities. Visible minority youth were under-represented relative to their representation in the general youth population.

Overall, ECP contributed to some extent to the three anticipated ultimate outcomes (understanding, connecting, appreciating diversity/shared experiences) for the large majority of participants (>80%).

The strongest outcome was in the area of connecting and creating linkages with one another. Across the three ultimate outcomes, Aboriginal youth and youth with disabilities were less likely to indicate that these outcomes had occurred for them.

In addition to the anticipated outcomes, ECP contributed significantly to other positive outcomes. Some of these were in the area of personal skills that could be considered as facilitators or levers to obtaining the anticipated outcomes for ECP. As well, they were key considerations for meeting various needs of youth from a developmental perspective including a range of employability/soft skills and increased interest in continuing education. Overall, these were strong results for SWSE participants and visible minority youth. Also there were positive results for Aboriginal youth and rural youth with respect to influencing their decision to continue their education.

The other area in which ECP was found to be demonstrating an impact was with respect to increasing levels of attachment to community/region, province/territory and Canada. This was an important outcome given ECP's link to PCH strategic outcome—attachment to Canada. This was a very strong result for YFC participants and was an area of considerable positive impact for Aboriginal youth and visible minority youth.

Performance – efficiency and economy

Actual expenditures compared with reference values were relatively close with the difference primarily due to increased levels of efficiency as reflected in reduced O&M expenditures across the five years studied. There was strong evidence of improved efficiency of the program from a PCH perspective with a considerable decline in expenditures of O&M while Gs&Cs expenditures remained constant.

The ratio of O&M over total PCH expenditures on average over the five years was approximately 8.9%. In the final year studied (2013-2014), the Program attained a 7% ratio. In comparison to other recently evaluated Gs&Cs programs, the five year ratio was higher than the ratios for YCW Initiative (6.2%) and the ratio of the CSP/CHF (5.5%). However, it was lower than the ratio for the YTC (14%).

The PCH contribution per participant varied considerably by component and subcomponent, but was much more comparable once the duration of activities was considered. A more appropriate metric, for comparison purposes, is the PCH contribution per participant per day of activity.

The ratio of PCH contributions to total project costs is an indication of the extent to which the PCH contribution is being leveraged. The evaluation found that over the five years PCH contributions covered on average 94% of the total annual project costs for SWSE and 68% of the total annual project costs for YEC. In the case of YFC, there was considerable leveraging of PCH contributions with the PCH contribution covering 23% of the total annual project costs.

The ECP design and model provided adequate flexibility and guidance for effective delivery of the program. The physical exchange experience appeared to be a contributing factor to achieving outcomes. Funding recipients identified opportunities for improvement by increasing the timeliness of the funding decision and by reducing the complexity of application process.

The evaluation examined virtual exchanges as an alternative approach to deliver ECP. Multiple lines of evidence (literature review, case studies and interviews) confirmed that virtual exchanges may have some benefits, but are unlikely to achieve the same anticipated outcomes as physical exchanges. There was evidence that virtual exchanges may be a beneficial complement to the physical exchange activities being undertaken, but are not a replacement.

An examination of other domestic exchange programs found that ECP is unique in supporting domestic Canada-wide exchanges and forums by providing funding that is used almost entirely to support travel. Other programs examined, that offered domestic exchanges, tended to focus only between specific regions of Canada or targeted specific groups. ECP, however, encourages exchanges between all provinces and territories and provides opportunities for all Canadian youth to go on exchanges.

PCH invests about $75.6 million annually in programs that benefit youth. There are some programs, including the CHF and the Canada Arts Training Fund that are not exclusively aimed at youth but have a broad target group that includes youth. An analysis of documents from these other PCH programs found that, when detailed objectives, results, activities and delivery mechanisms were examined within the context of the evaluation, there was limited evidence of overlap with other programming. ECP shares some aspects with YTC (target populations, some theme areas, some similar outcomes); however, the activities and overall approaches are relatively distinct when closely examined. One indication of this is that there was little overlap in funding recipients for the delivery of the two programs.

Performance - performance measurement and monitoring

A PMERS was developed and implemented for ECP. The Program has identified performance indicators for each planned outcome of the Program corresponding to the logic model. These outcomes were tracked. PCH officials have deemed the PMERS as adequate and supporting many of their needs for monitoring the performance of the program. However, there are opportunities to improve the performance data collected and used for program decision-making and program improvements. There is a reliance on quantitative information collected through the post-participation surveys of participants, parents and organizers to measure intermediate and ultimate outcomes. Surveys are administered immediately after participation in the exchange or forum and there is no follow-up with participants or funding recipients to assess lasting program impacts.

While the program produced annual performance spreadsheets and PCH's PRG produced a report annually on the results of the post-participation survey, the evaluation did not find evidence of trend analysis across the years assessing outcomes by component and by target group which could inform program improvements.

The evaluation also did not find evidence that observations from individual project reports are being consolidated, with a view to identifying common issues across projects, trends and potential areas for improvement. In interviews, PCH officials indicated that there are opportunities to use more of this information. Consideration should be given, therefore, to supplementing the quantitative results of the post-participation surveys of participants, organizers and parents with the qualitative information obtained from funding recipients, specifically their perceptions of the achievement of project outcomes, challenges encountered, lessons learned and opportunities for improvement.

Feedback from funding recipients about the post-participation survey for youth was that it was long and difficult to administer widely and in a way that ensured quality responses. Some funding recipients suggested that the survey should be made available online. A pilot of an online survey was undertaken with six organizations in 2012-2013 and 2013-2014. The four pilot organizations with the largest number of participants had the lowest response rates which had a significant impact on the overall response rate for the participant survey in the two years of the pilot. However, two pilot organizations with the smallest number of participants had a response rate of over 90%.

The survey instrument for participants was also considered to be difficult for youth to understand and some questions were found to be unclear (e.g., how a disability is defined).

6.2 Recommendations and management response

The following three recommendations emerged from the evaluation findings. It was noted that some of challenges and issues identified in the 2010 evaluation of ECP continued to be challenges and issues during the period covered by this evaluation, including issues with respect to the timeliness of the funding decision process and performance measurement and challenges with respect to the achievement of targets for certain groups and regions. There continue to be opportunities for improvement in these areas.