Evaluation of the Exchanges Canada Program

2009-10 to 2013-14

Evaluation Services Directorate

February 5, 2016

Cette publication est également disponible en français.

© Her Majesty the Queen in Right of Canada, 2016

Catalogue No. CH7-33/2016E-PDF

ISBN: 978-0-660-04407-1

A PDF version is also available [PDF Version, 1.1 MB]

Table of contents

List of tables

List of figures

List of acronyms and abbreviations

CA
Contribution Agreement
CSP/CHF
Canadian Studies Program/Canada History Fund
DPR
Departmental Performance Report
ECP
Exchanges Canada Program
ESD
Evaluation Services Directorate
GC
Government of Canada
Gs&Cs
Grants and Contributions
IPPMEC
Integrated Planning, Performance Measurement and Evaluation Committee
KI
Key Informants
NHS
National Household Survey
O&M
Operations and Maintenance
PAA
Program Alignment Architecture
PCH
Department of Canadian Heritage
PMERS
Performance Measurement, Evaluation and Risk Strategy
PRG
Policy Research Group
RPP
Report on Plans and Priorities
SEVEC
Society for Educational Visits and Exchanges in Canada
SWSE
Summer Work Student Exchange
TBS
Treasury Board Secretariat
YCW
Young Canada Works
YEC
Youth Exchanges Canada
YFC
Youth Forums Canada
YTC
Youth Take Charge

Executive Summary

Overview of the program

The Exchanges Canada Program (ECP) was launched in April 2000. ECP provides financial support to not-for-profit organizations to offer youth a range of exchange and forum activities to strengthen their sense of belonging to Canada and, therefore, their sense of Canadian identity.

The Program's objectives are to:

  • Enhance knowledge and understanding of Canada among Canadian youth;
  • Connect young Canadians with one another; and
  • Help youth develop a sense of Canadian identity and attachment to Canada by helping them appreciate the diversity and the shared aspects of the Canadian experience.

The target population is youth between the ages of 12 and 25. Also, key stakeholders include Canadian youth, delivery organizations, schools and other groups wishing to organize exchanges.

ECP has two main components:

  • Youth Exchanges Canada (YEC), which provides reciprocal homestay exchanges for young Canadians generally between the ages of 12 and 17. YEC is delivered through contribution agreements with four pan-Canadian not-for-profit organizations with a proven network of youth contacts.
  • Youth Forums Canada (YFC), which enables young Canadians, generally between the ages of 14 and 25, to connect with one another through youth forums, post-secondary study sessions in another area of Canada and projects such as workshops and thematic exchanges of interest to young Canadians. YFC has two additional funding recipients and hosts an annual funding competition to plan and convene a forum event for Canadian youth.

YEC has a subcomponent, Summer Work Student Exchange (SWSE), which provides six‑week summer work opportunities for 16 and 17 year-olds. SWSE is delivered through the YMCA.

The total grants and contributions (Gs&Cs) issued over the period covered by the evaluation (2009-2010 to 2013-2014) amounted to $88.9 million. Over this period, direct operating costs averaged approximately $1.7 million per year (8.9%) and approximately 63,000 youth participated in ECP activities.

Evaluation approach and methodology

The objective of the evaluation was to provide credible and neutral information on the ongoing relevance and performance (effectiveness, efficiency and economy) of ECP for the period from 2009-2010 to 2013-2014. The evaluation questions addressed the five core issues of relevance and performance as outlined in the Treasury Board Secretariat (TBS) Directive on the Evaluation Function (2009).

The following methods were used for the evaluation: document review, literature review, administrative data review, case studies and interviews with PCH officials and stakeholders.

There were several challenges and limitations associated with this evaluation, including: reliance on self-reporting in surveys filled out by participants shortly after taking part in the program to assess achievement of outcomes; restrictions to the economy and efficiency analysis; and changes to the participant survey questions, limiting the analysis to three years for some outcomes. Finally, because of dated contact information, the evaluation team was unable to reach former participants for one of the case studies. As a result, participant interview data is unavailable for this case study.

To the extent possible within the available budget and time frame, the evaluation methodology incorporated multiple methods and data from primary and secondary sources. Opinions and observations expressed by stakeholders were corroborated, where possible, with evidence from other sources such as program documentation, data and the literature.

Conclusions

Relevance

The literature identified a number of factors that demonstrated a continued need for young Canadians to increase their knowledge and understanding of Canada, to have opportunities to connect and create linkages with each other and to develop a Canadian identity and a sense of belonging to Canada. By covering travel expenses, the Program supported not-for-profit organizations in providing youth with a range of exchange and forum activities to strengthen their sense of belonging to Canada and, therefore, their sense of Canadian identity. Respondents from all groups indicated that without PCH funding, national exchange and forum activities would not take place at the current scale (e.g., national, diversity of participants) or would not take place at all.

The ECP objectives and expected results aligned with federal government priorities as outlined in speeches from the throne and recent federal budgets. ECP was consistent with a federal government commitment to support communities, culture and heritage and groups that face greater challenges and experience barriers, such as at-risk youth. Similarly, ECP objectives and outcomes were closely aligned with PCH's strategic outcome: Canadians share, express and appreciate their Canadian identity. Overall ECP was aligned with the departmental priority: Invest in our communities.

Key informant interviews showed support for a federal government role in a program such as ECP, given its national scope and PCH's expected outcomes in the areas of attachment to Canada and shared Canadian identity.

Performance – achievement of expected outcomes

ECP made progress toward the achievement of its immediate outcome: Young people have access to a wide variety of exchange experiences, including exchanges related to official languages. ECP provided youth a wide variety of domestic exchange experiences in terms of location, content, duration and design and included exchanges related to official languages. Between 2009-2010 and 2013-2014, ECP had approximately 63,000 participants. The YEC and YFC components accounted for approximately 53% and 40% of ECP participants, respectively. SWSE accounted for 7% of ECP participants.

A variety of organizations were funded by ECP which, in turn, offered a diversity of experiences and a range of activities and program designs. ECP projects provided exposure to both official languages. Results from the post-participation survey also indicated that participation had impacts on participants' knowledge and interest in their second official language.

The two anticipated intermediate outcomes of young people from across Canada participate in youth exchanges and exchange participants reflect the diversity of the Canadian youth population were achieved to some extent. ECP had youth participants from all provinces/territories. However, there were some challenges with obtaining a distribution of ECP participants according to region of origin that was comparable to the distribution of youth in Canada. Overall, the participants in ECP reflected the distribution of the Canadian youth population for all provinces except Ontario. Ontario youth were significantly under-represented in both YEC and YFC. Quebec youth were less likely to participate in YFC events. However, the corollary of this is that the other provinces and territories were over-represented for YFC and YEC. In the case of SWSE, Quebec youth were significantly over-represented, as would be expected as it is a French-English language exchange program. Youth from all other provinces and territories, with the exception of Prince Edward Island, were under-represented.

ECP participants were primarily younger youth (under 19), distributed across a broad demographic profile, including traditionally under-represented youth. There appeared to be some barriers for young men participating in ECP, as a disproportionate number of young women were participants. Across all components of ECP, there was a higher proportion of young women (63%) participating in ECP, compared to young men. This was particularly the case for SWSE, where 73% of participants were young women.

Overall, there was considerable diversity among ECP participants as they included Aboriginal youth, youth from low-income households, visible minority youth, youth with disabilities and rural youth. Participation of rural, low-income and Aboriginal youth exceeded their incidence in the general youth population in Canada. Although the participation of youth with disabilities appears to have improved in the past three years, the broadening of the definition of disability may be contributing to this increase. Overall, visible minority youth were under-represented relative to the general youth population.

Comparing across components:

  • YEC exceeded targets for Aboriginal, low-income and rural youth and youth with disabilities, but did not meet targets for visible minority youth.
  • SWSE had challenges in achieving representation of Aboriginal and rural youth and youth with disabilities but was more successful at achieving representation of low-income and visible minority youth.
  • YFC exceeded targets for Aboriginal youth over all the years. It has improved the participation of rural youth, exceeding the target in the last two years. Similarly, in the last three years, it has exceeded the target for youth with disabilities. Visible minority youth were under-represented relative to the general youth population.

Overall, ECP contributed to the three anticipated ultimate outcomes. A large majority of participants reported that they had enhanced their knowledge and understanding of Canada, connected and created linkages with one another and enhanced their appreciation of the diversity and shared aspects of the Canadian experience (> 80% for all indicators). Parents and organizers perceived an even greater impact. The strongest outcome was in the area of connecting and creating linkages with one another. All key informants noted that the physical, in-person aspect of the exchanges was a contributing factor to the achievement of this outcome. Across the three ultimate outcomes, Aboriginal youth and youth with disabilities were less likely to indicate that these outcomes had occurred for them.

In addition to the anticipated outcomes, ECP contributed significantly to other positive outcomes. Some of these were in the area of personal skills that could be considered as facilitators or levers to obtaining the anticipated outcomes for ECP. As well, they are key considerations for meeting various needs of youth from a developmental perspective including a range of employability/soft skills and increased interest in continuing education. Overall, these were strong results for SWSE participants and visible minority youth. There were also positive results for Aboriginal youth and rural youth with respect to influencing their decision to continue their education.

Another area in which ECP was found to be demonstrating an impact was in increasing levels of attachment to community/region, province/territory and Canada. This is an important outcome given ECP's link to the PCH strategic program outcome—attachment to Canada. This was a very strong result for YFC participants and was an area of considerable positive impact for both Aboriginal and visible minority youth.

Performance: Economy and efficiency

Over the five-year period covered by the evaluation, there was an overall variance of 5% between actual expenditures and reference levels, primarily due to reduced O&M expenditures across the five years studied. Over this period, Gs&Cs expenditures remained fairly stable while O&M expenditures were reduced by half.

The ratio of O&M expenditures to total ECP expenditures over the five years covered by the evaluation was 8.9%. This ratio steadily decreased from 12% in 2009‑2010 to 7% in 2013-2014 providing evidence of improved program efficiency. In comparison with other PCH Gs&Cs programs recently evaluated, the five year ratio was higher than the ratios for the Young Canada Works (YCW) Initiative (6.2%) and the ratio of the Canadian Studies Program/Canada History Fund (CSP/CHF) (5.5%). However, it was lower than the ratio for Youth Take Charge (YTC) (14%).

The PCH contribution per participant varied considerably by component and subcomponent, but was more comparable once the duration of activities was considered. A more appropriate metric, for comparison purposes, is the PCH contribution per participant per day of activity. Using five-year averages, the calculations showed that the PCH contribution per participant per day was $129 for SWSE, $90 for YEC and $152 for YFC.

With respect to the PCH contribution as a proportion of total project costs, over the five years covered by the evaluation PCH contributions covered on average 94% of the annual project costs for SWSE and 68% of the annual project costs for YEC. In the case of YFC, there was considerable leveraging of PCH contributions, with the PCH contribution as a portion of annual project costs averaging 23% for the five years.

The ECP design and model provided adequate flexibility and guidance for effective delivery of the program. The physical exchange experience appeared to be a contributing factor to achieving outcomes. Funding recipients identified opportunities to improve the timeliness of the funding decision, which had an impact on the delivery of projects, and to reduce the complexity of the application process.

PCH invests about $75.6 million annually in programs that benefit youth. There are some programs, including the CHF and the Canada Arts Training Fund that are not exclusively aimed at youth but have a broad target group that includes youth. An analysis of documents from these other PCH programs found that when detailed objectives, results, activities and delivery mechanisms were examined within the context of the evaluation, there was limited evidence of overlap with other programming. ECP shares some aspects with YTC (target populations, some theme areas and some similar outcomes); however, the activities and overall approaches were relatively distinct when closely examined. One indication of this was that there was little overlap in funding recipients for the delivery of the two programs.

Virtual exchanges were explored as an alternative approach to deliver ECP. While they may have some benefits, they are unlikely to achieve the same anticipated outcomes as physical exchanges. Virtual exchanges may be a beneficial complement to the physical exchange activities being undertaken, but are not a replacement.

An examination of other domestic exchange programs found that ECP is unique in supporting domestic Canada-wide exchanges and forums by providing funding that is used almost entirely to support travel. Other programs examined that offered domestic exchanges tended to focus only between specific regions of Canada or target specific groups. ECP, however, encourages exchanges between all provinces and territories and provides opportunities for all Canadian youth to go on exchanges.

Performance - performance measurement and monitoring

A Performance Measurement, Evaluation and Risk Strategy (PMERS) was developed and implemented for ECP. The Program has identified performance indicators for each planned outcome of the Program corresponding to the logic model. These outcomes were tracked and analyzed annually. PCH officials considered the current performance measurement and other monitoring mechanisms as effective in capturing results.

However, there are opportunities to improve the analysis and use of performance data collected from funding recipients for decision-making and program improvements. The program produced annual reports on project participation, project costs and performance relative to target populations. The Policy Research Group (PRG) at PCH produced a report annually on the results of the post-participation surveys. A one-time trend analysis report on participation numbers was completed for the period 2000 to 2012 but this analysis has not been repeated. The information collected from funding recipients should be analyzed over a longer period to assess trends in the achievement of outcomes by component and by target group and to compare projects costs relative to outcomes to determine which projects are most cost-effective. This information could inform program improvements and project selection.

The final project reports are another potential source of information to identify common issues across projects, trends and potential areas for improvement. In interviews, PCH officials indicated that there were opportunities to use more of the information collected in recipient reports. However, they noted that this information is not currently being analyzed because of a lack of staff resources. Consideration should be given to supplementing the quantitative results of the post-participation surveys of participants, organizers and parents with the qualitative information obtained from funding recipients, specifically their perceptions of the achievement of project outcomes, challenges encountered, lessons learned and opportunities for improvement.

Funding recipients identified issues with the content and administration of the participant survey. Among the issues raised by funding recipients were that the survey is long, the language of the survey is difficult for youth to understand and it is difficult to administer the survey. Some funding recipients suggested that the survey be made available online. A pilot of an online survey was undertaken with six organizations in 2012-2013 and 2013-2014. Response rates for the online survey varied considerably, ranging between 9.6% and 96.5%. The four pilot organizations with the largest number of participants had the lowest response rates, which had a significant impact on the overall response rate for the participant survey in the two years of the pilot. However, the two pilot organizations with fewer participants had a response rate of over 90%.

Recommendations

The following three recommendations emerged from the evaluation findings. It was noted that some of challenges and issues identified in the 2010 evaluation of ECP continued to be challenges and issues during the period covered by this evaluation, including issues with respect to the timeliness of the funding decision process and performance measurement and challenges with respect to the achievement of targets for certain groups and regions. There continue to be opportunities for improvement in these areas.

Recommendation #1

To improve the efficiency of the funding application, the Assistant Deputy Minister of Citizenship, Heritage and Regions should streamline the application process and monitor the funding decision process to ensure timeliness of the decision to release funds and reduce the complexity of the application process.

Recommendation #2

To strengthen the collection and use of outcome data for management decision-making and program improvements, the Assistant Deputy Minister of Citizenship, Heritage and Regions should:

  • 2.1 consult with funding recipients to modify the language and content of the post-participation survey instruments, particularly for participants;
  • 2.2 provide funding recipients with the option to conduct on-line surveys and consult with recipients to identify barriers to administering the surveys on-line; and
  • 2.3 collect and analyze the qualitative data in the final progress reports submitted by funding recipients to supplement the quantitative outcome data collected through post-participation surveys with funding recipient perceptions of project outcomes, best practices and lessons learned.

Recommendation #3

While overall, ECP program participants generally reflect the demographic and provincial/territorial diversity of the Canadian youth population, certain components of the program are under-subscribed by certain segments of the population (such as young men, Aboriginal, rural and visible minority youth and youth with disabilities). Therefore, the Assistant Deputy Minister of Citizenship, Heritage and Regions should work with funded organizations to identify the barriers to participation and to develop and implement approaches to attract participation of target groups and provinces/territories, as applicable.


1. Introduction

1.1 Purpose

The report presents the findings and recommendations from the 2014‑2015 evaluation of ECP. The evaluation of ECP was conducted to fulfill the requirements of the Financial Administration Act (1985) and the Treasury Board Secretariat (TBS) Policy on Evaluation (2009) to evaluate all ongoing grant and contribution programs every five years.

The purpose of the evaluation was to provide comprehensive and reliable evidence to support decisions regarding continued implementation of the program. The evaluation report provides information on ECP, the evaluation methodology and the findings for each of the evaluation questions, as well as overall conclusions and recommendations. The evaluation was led by the Evaluation Services Directorate (ESD) of PCH.

In accordance with the TBS Directive on the Evaluation Function (2009), the evaluation addresses the five core evaluation issues relating to the relevance and performance (effectiveness, efficiency and economy) of ECP.

The report is structured as follows:

  • Section 2 presents an overview of ECP.
  • Section 3 presents the methodology employed for the evaluation and the associated limitations.
  • Section 4 presents the findings related to the evaluation issue of relevance.
  • Section 5 presents the findings for performance (including those related to achievement of outcomes and efficiency/economy).
  • Section 6 presents the conclusions and recommendations.

2. Program Profile

2.1 Background and context

ECP was launched in April 2000. It provides Gs&Cs in support of youth participation initiatives that allow young Canadians across the country to learn about Canada, to create linkages with each other and to better appreciate the diversity and shared aspects of the Canadian reality. By essentially covering travel expenses, the Program supports not-for-profit organizations to provide youth with a range of exchange and forum activities to strengthen their sense of belonging to Canada and therefore their sense of Canadian identity.

ECP has two main components:

  • Component I - Youth Exchanges Canada
  • Component II - Youth Forums Canada

Youth Exchanges Canada provides reciprocal homestay exchanges for young Canadians generally between the ages of 12 and 17. Groups of 10 to 30 youth from various areas of the country are paired based on age and interests. Exchanges are normally between communities from different provinces and territories, facilitated by non-governmental youth organizations. The participants play an active role in planning the exchange. The pairs communicate regularly, conduct research into their host community and participate in fundraising activities.

  • Summer Work / Student Exchange (a sub-component of YEC) provides six-week summer work opportunities (from late June to mid-August) for 16 and 17 year-olds. The program finds jobs for participants in communities in another province. Communities are paired so that students work in their second official language. During their stay in the host community, each student lives with the family of the student with whom they have been paired. Coordinators (post-secondary students) supervise the students participating in the program and run a program of activities.

Youth Forums Canada enables young Canadians, generally between the ages of 14 and 25, to connect with one another through various means, including national or international youth forums, post-secondary study sessions in another area of Canada and projects such as workshops, thematic exchanges of interest to young Canadians or other exchange-related activities that meet the objectives of ECP.

Examples of YFC projects funded during the period of the evaluation include:

  • Forum for Young Canadians, offered by the Foundation for the Study of the Processes of Government in Canada, brings high-school and CÉGEP (Quebec) students aged 15 to 19 years from across Canada to Ottawa where they spend a week on Parliament Hill and experience first-hand how Canadian government works.
  • 4H Seminar, offered by the Canadian 4-H Council, brings youth together annually to take part in a national citizenship seminar on topical political issues related to citizenship. Canadian youth learn about Canada and its political processes and systems, share opinions on issues of national importance and learn about their roles and responsibilities as Canadian citizens.
  • Activate 2013 National Youth Leadership Forum, offered by Motivate Canada, brought together youth aged 16 to 22 and monitors from across Canada to Ottawa to learn about themselves and their respective communities, while also learning about communities from across Canada, through youth-led workshops, discussions and youth-led group activities. In recognition of Canada's 150th anniversary in 2017, there was a focus on the Canada Games and its 50th anniversary in 2017. As a result of the forum, participants returned home to play an active role in their community, through the planning and execution of a physical activity or event specifically designed for their communities' needs.
  • The Girls Action Foundation's ELLE for Leadership National Youth Forum (October 2012) brought together young women between the ages of 16 and 25 for a five‑day forum centered on increasing participants' knowledge of Canada's history and empowering young women to be self-sufficient and good leaders. The event was intended to build their capacity to influence other youth in their communities and to increase their civic knowledge and responsibilities.

2.2 Objectives and outcomes

ECP's objectives are to:

  • contribute to increased knowledge and understanding of Canada among Canadian youth, by enabling them to learn first-hand about the history, geography, industry, institutions, cultures, communities, languages and other facets of their country;
  • help young Canadians connect to one another and create linkages across the country and between groups, thereby helping to strengthen the fabric of Canadian society; and
  • develop Canadian identity and a sense of belonging to Canada among youth by enhancing their appreciation of both the diversity and the shared aspects of the Canadian experience.

A Performance Measurement, Evaluation and Risk Strategy (PMERS) was developed in 2010 and includes the Program's Logic Model, which reflects the activities, outputs and outcomes of the program (see Appendix A for the ECP Logic Model).

The key activities of the program are managing the funding process, providing Canadians with information on exchanges and undertaking targeted promotion. The outputs of these activities are grants and contribution agreements with organizations in support of youth exchanges and information about youth exchanges.

ECP has the following expected outcomes:

Immediate outcomes

  • Young people have access to a wide variety of exchange experiences, including exchanges related to official languages.

Intermediate outcomes

  • Young people from across Canada participate in youth exchanges.
  • Exchange participants reflect the diversity of the Canadian youth population.

Ultimate outcomes

  • Young participants enhance their knowledge and understanding of Canada.
  • Young participants connect and create linkages with one another.
  • Young participants enhance their appreciation of the diversity and shared aspects of the Canadian experience.

Program Management and Governance

ECP is managed by the Citizen Participation Branch, Citizenship, Heritage and Regions Sector, in the Youth Participation Directorate. The Program is delivered centrally at Headquarters in the National Capital Region, through grants and contribution agreements with funding recipients.

PCH program officers oversee the funding agreements to ensure that there is adequate performance monitoring and that all program financial and activity reporting requirements are met.

YEC is delivered through contribution agreements with four pan-Canadian not-for-profit organizations with a proven network of youth contacts: Society for Educational Visits and Exchanges in Canada (SEVEC), YMCA, 4-H and Canada Sports Friendship Exchange Program (CSFEP). SWSE, is delivered through the YMCA.

YFC has two multi-year funding recipients (Encounters with Canada and Forum for Young Canadians), but also hosts an annual funding competition. A general call for applications is put out annually to possible funding recipients to plan and convene a forum event for Canadian youth. In the last five years, between 8 and 15 organizations have received funding annually through the Youth Forums funding competition.

Funding recipients are responsible for reviewing applications for exchanges and forums, matching exchange groups, meeting participation targets, organizing the exchanges and forums, ensuring the safety and security of participants and reporting on results.

Target groups, key stakeholders and delivery partners

ECP benefits, either directly or indirectly, communities, parents, teachers, group leaders and others. However, the primary clients are the exchange participants themselves: Canadian youth, generally between the ages of 12 and 25.

Target population

Canadian youth between the ages of 12 and 25

Key stakeholders

Canadian youth

Delivery organizations

Schools and other groups wishing to organize exchanges

Delivery partners (recipients)

Youth-serving organizations, as well as other groups wishing to organize exchanges and forums

2.3 Program resources

Table 1 presents the budgeted resources and actual expenditures for ECP for the period covered by the evaluation. Total resources for 2009-2010 to 2013-2014 were $103,069,558. Total expenditures for this period were $97,551,135, which included $8,669,430 in Operations and Maintenance expenditures and $88,881,705 in Grants and Contributions expenditures.

Table 1: Budgeted and actual expenditures 2009-2010 to 2013-2014

Resources

2009-2010

2010-2011

2011-2012

2012-2013

2013-2014

Total

Budgeted Resources

O&M

$3,775,302

$3,270,774

$2,922,049

$2,526,333

$1,643,305

$14,137,763

Actual expenditures

O&M

$2,523,266

$1,895,878

$1,663,367

$1,316,310

$1,270,609

$8,669,430

Budgeted resources

Gs&Cs

$17,786,359

$17,786,359

$17,786,359

$17,786,359

$17,786,359

$88,931,795

Actual expenditures Gs&Cs

$17,882,489

$18,133,151

$17,458,496

$17,562,353

$17,845,216

$88,881,705

Source: STAR

3. Evaluation methodology

3.1 Evaluation scope, timing and quality control

The evaluation's objective is to provide credible and neutral information on the ongoing relevance and performance (effectiveness, efficiency and economy) of ECP for the period from 2009-2010 to 2013-2014.

The evaluation was designed and conducted in accordance with the TBS Policy on Evaluation (2009) and other components of the TBS evaluation policy suite. It meets the Financial Administration Act and the TBS Policy on Evaluation accountability requirements that all direct program spending be evaluated every five years. It also provides PCH management with analysis and recommendations to inform program improvements.

As the program has existed since 2000 and has been evaluated on two previous occasions, the evaluation minimized the level of effort on evaluating the design and delivery aspects of the program. The approach to the evaluation took into account the results of an ESD calibration exercise which sought to ensure that the evaluation was conducted in a cost-effective manner while maintaining the credibility and utility of evaluation results.

The quality of the evaluation was ensured through senior-level ESD planning of the evaluation, including approval of the terms of reference for the evaluation by PCH's Integrated Planning, Performance Measurement and Evaluation Committee (IPPMEC). The draft evaluation report was reviewed by senior level ESD and program staff.

3.2 Evaluation questions by issue area

The evaluation addresses the five core issues of relevance and performance as outlined in the TBS Directive on the Evaluation Function (2009). The questions addressed include:

Relevance

  • Issue #1: Ongoing need for the program
  • Issue #2: Alignment with government priorities
  • Issue #3: Alignment with federal roles and responsibilities

Performance (effectiveness, efficiency and economy)

  • Issue #4: Achievement of expected outcomes
  • Issue #5: Demonstration of efficiency and economy

3.3 Evaluation methods

Preliminary consultation

Before undertaking the evaluation, preliminary discussions were held with the ECP evaluation working group, leading to the development of the terms of reference for the evaluation. The terms of reference included a description of the evaluation scope and issues, the methodological approach and a detailed evaluation matrix. The evaluation matrix (Appendix B) identifies the evaluation questions, associated indicators and the lines of evidence to respond to each question.

Lines of evidence

The evaluation featured important strengths including a mix of qualitative and quantitative lines of evidence, a mix of primary and secondary data sources and multiple lines of evidence to allow for the triangulation of evidence. The evidence from all lines was organized and analyzed by core issue, evaluation question and indicator and consolidated into an overall evidence matrix.

The evaluation methodology incorporated five lines of evidence:

  • Document review
  • Literature review
  • Administrative data review
  • Interviews with key informants
  • Case studies of four organizations that had received ECP funding

Document review

Program and government documents were reviewed to assess the relevance and performance of ECP, particularly its alignment with federal government and departmental priorities and strategic outcomes. Government of Canada (GC) documents reviewed for the evaluation included budget documents and speeches from the throne. Departmental documents included departmental performance reports (DPRs) and reports on plans and priorities (RPPs). A range of program documents were reviewed, among them the Program's administrative reports, terms and conditions, PMERS, previous audit and evaluation reports, contribution agreements with recipients and recipient financial reports.

Literature review

A literature review provided evidence for the relevance of ECP, including the continuing need and responsiveness of ECP. The sources consulted for the literature review were derived from academic publications, monographs, government and non-governmental research reports and websites.

Administrative data review

The administrative data review included a review of quantitative information on ECP activities and results and was an important source of evidence for evaluation questions pertaining to performance (including effectiveness and efficiency).

The administrative data review included the review of a sample of contribution agreements, final reports and financial statements from funded organizations, results from participant surveys and various compilations of program-level administrative data, including financial and performance measurement information.

It also included an analysis of participant, organizer and parent feedback questionnaires, which had been collected by the program between 2009-2010 and 2013-2014. The response rates for the participant and parent surveys are presented in Table 2 below.

Table 2: Feedback surveys completed by year and respondent group

Participants

2009-2010

2010-2011

2011-2012

2012-2013

2013-2014

5-year average

# of surveys sent

12,755

12,977

12,576

12,006

12,567

62,881

# of responses

10,305

10,325

9,973

6,540

6,307

43,450

Response rate

80%

79.6%

79.3%

54.5%

50.2%

69.1%

Parents2009-20102010-20112011-20122012-20132013-20145-year average

# of surveys sent

12,755

12,977

12,576

12,006

12,567

62,881

# of responses

1,281

836

1,756

1,814

1,875

7,562

Response rate

10.0%

6.4%

14.0%

15.1%

14.9

12.0%

Source: Post-participation survey reports (2009-2010 – 2012-2013); raw data for 2012-2014

The response rate is not available for organizers. The Program provides funding recipients with surveys to distribute to their organizers; however, the total number of surveys actually distributed is not tracked. Between 648 and 856 surveys were returned annually, with an annual average of 728.

Financial data for ECP were also analyzed for the period of the evaluation, including reference levels and expenditures for ECP O&M and Gs&Cs.

Interviews with key informants

The purpose of the key informant interviews was to gather in-depth information, including opinions, explanations, examples and factual information for all evaluation issues and questions. Key informants were selected based on their involvement, knowledge and experience with ECP. A total of 20 interviews were conducted with the following:

  • representatives from organizations that received ECP funding (n=11 organizations out of 31);
  • organizers (e.g., teachers and group leaders who work with funding recipients) (n=5); and
  • PCH officials (n=4).

Case studies

Four case studies of organizations funded by ECP were conducted as part of the evaluation. The objectives of the case studies were to gather information on the relevance of the program and to seek perspectives from former participants themselves on the longer-term impact of participation in the Program. Two case studies were selected from the four YEC projects, one of which examined the YMCA's SWSE initiative, a sub-component of YEC. Two additional case studies were selected from YFC projects. All of the case studies had received funding in each of the five years of the evaluation period.

The methodology for each case included a document review, administrative data review, interviews with funding recipients and interviews with previous youth participants in three of the four cases. Across the four case studies, interviews were conducted with 11 representatives from 4 organizations and with 17 former ECP participants.

The following guidelines were used to report the findings of the surveys and key informant and case study interviews:

Quantifiers

% of Respondents / Key Informants

All/almost all

Findings reflect 90% or more of the observations

Large majority/most

Findings reflect 75% or more, but less than 90% of the observations

Majority

Findings reflect at least 50%, but less than 75% of the observations

Some

Findings reflect at least 25%, but less than 50% of the observations

A few

Findings reflect less than 25% of the observations


Methodological limitations

While the methodology offered a number of important strengths, including the mix of qualitative and quantitative lines of evidence and a mix of primary and secondary data sources, the evaluation encountered some challenges and there were some limitations to the methodology, including the following:

  • Reliance on self-reports by participants to assess achievement of intermediate and ultimate outcomes. The evaluation relied heavily on the results of participant surveys to report on intermediate and ultimate outcomes for the projects. Long-term outcomes were difficult to determine as participants usually complete the survey very soon after their participation and there is no comparison group with other youth. The extent to which ECP activities contribute to a particular outcome versus other potential contributors (e.g., natural maturation process and youth development stages) is difficult to determine. To ascertain the longer-term impacts of the Program, the evaluation included interviews with former ECP participants as part of the case studies.
  • Restricted economy/efficiency analyses. Some of the planned analyses to assess economy and efficiency could not be undertaken as the financial information was not sufficiently granular (e.g., O&M divided by ECP component), or the project level financial information did not have consistent definitions or collection over multiple years (e.g., in-kind contributions).
  • Participant interviews for case studies. One of the case studies did not include interviews with former participants. The evaluation team was unable to contact participants due to outdated contact information and the short time frame in which to conduct the case studies.
  • Challenges with reporting on participant survey data over the years covered by the evaluation. The reporting on participant surveys is done on an annual basis. There were some challenges when using these reports over the years as some of the questions were not consistent over the years. To ensure consistency in breakdowns and cross tabulations of data across the years, the evaluation team obtained raw data from multiple years and combined the information in one database to enable a comparative analysis over the five years covered by the evaluation.

4. Findings - relevance

The following sections present the findings related to the relevance of ECP, including the ongoing need for the program and its alignment with federal government and PCH priorities and with federal roles and responsibilities.

4.1 Core issue 1: Continued need for the program

Evaluation question:

To what extent is there a demonstrated need for ECP?

Is ECP responsive to the needs of Canadian youth?

Key findings

The evidence demonstrated a continued need for ECP, based on the literature showing that:

  • Youth are less likely, compared to older cohorts, to be interested in Canadian history, to feel that it is important to national identity or to feel that it would improve their attachment to Canada.
  • Youth, compared to older cohorts, are less likely to feel pride in Canada/being Canadian, are less likely to feel attached to Canada or to have a sense of belonging on either a national, provincial or local community level. Results from the 2013 General Social Survey indicated that 56% of young Canadians 15-24 reported that they had a strong sense of belonging to Canada, compared to 63% of Canadians generally and 77% of seniors aged 75 years and older.
  • Under-represented youth often encounter limitations to involvement—access to sufficient financial resources, geographic remoteness, as well as a feeling of isolation from the mainstream of their society.
  • While Canadian youth, compared to older cohorts, are more supportive of multiculturalism, more comfortable around visible minorities, more supportive of diversity in other areas and are proud of Canada's diversity and of official protections for minority groups, there is an opportunity to solidify their understanding of diversity and build shared experiences of Canada.
  • There is a need for youth to have opportunities for physical, face-to-face/experiential opportunities. A greater reliance on technology and social media has contributed to less in-person interaction/experiential opportunities.

Interviews with recipients provided further evidence for the continued need for ECP. Respondents from all groups indicated that national level exchange and forum activities either would not take place at the current scale or would not take place at all without ECP funding.

The evidence indicated that ECP responds to these needs by:

  • including a history and heritage component in all funded projects;
  • ensuring inclusion of under-represented groups and making adjustments in delivery and recruitment to ensure inclusion;
  • promoting a better understanding of diversity, including linguistic duality, Aboriginal culture and Canada's multicultural character;
  • providing opportunities for youth to have exposure to their second official language; and
  • providing opportunities for interpersonal, experiential opportunities through the physical exchange experience.

Continued need for ECP

The literature identified a number of factors that demonstrated a continued need for young Canadians to increase their knowledge and understanding of Canada, to have opportunities to connect and create linkages with each other and to develop a Canadian identity and a sense of belonging to Canada.

  • Knowledge and understanding of Canada. The literature review indicated that concerns remain about the extent to which young Canadians are knowledgeable about Canada (Cowan and Landon, 2011). Footnote 1 The literature review also found that Canadian youth, particularly younger youth, are less likely when compared to older cohorts to be interested in Canadian history, to feel that it is important to national identity or to feel that it would improve their attachment to Canada. Footnote 2, Footnote 3, Footnote 4 Further, a round table report by Deloitte (2012) indicated there is a strong case for improved "citizenship" training and awareness among Canadian youth. Footnote 5 The literature review also identified studies that have linked Canadian youth's lack of knowledge about Canada's civic system and political issues with their lower rates of voting and lower levels of participation in more formal types of community and civic engagement. Footnote 6, Footnote 7 It should be noted that there is no consensus in the literature with respect to this link, as there continues to be debate among researchers about the strength of this link, competing factors and potential moderators of this relationship.
  • Diversity and shared aspect of the Canadian experience. Canadians aged 18 to 30 are more supportive of multiculturalism compared to older Canadians and are also more comfortable around visible minorities. Young Quebecers in particular are more accepting of minorities and immigration than older generations of Quebecers. However, paradoxically, younger Canadians are more likely than older Canadians to ask that fewer immigrants be admitted to Canada. The study attributed this paradox to economic and unemployment-related concerns. Younger Canadians were also generally more supportive of diversity in other areas such as lesbian, gay, bisexual and transgender Canadians. Compared to older Canadians, youth are generally more likely to be proud of Canada's diversity or of official protections for minority groups (e.g., the Canadian Charter of Rights and Freedoms, multiculturalism and bilingualism policies). Footnote 8 When asked the question "What keeps Canada united?" Canadians aged 18 to 24 were much more likely to choose 'multiculturalism' compared to the general population (16% versus 8%). Footnote 9 Youth support for official languages and multiculturalism and respect for Aboriginal culture suggests that youth programming has the greatest potential for amplification and for future impact and to solidify their understanding of diversity and build shared experiences of Canada.
  • Canadian identity and sense of attachment. According to the literature younger Canadians are less likely to feel pride in Canada/being Canadian and are less likely to feel attached to Canada than their older cohorts. The 2013 General Social Survey on Social Identity found that 56% of youth aged 15-14 described their sense of belonging to Canada as being very strong, compared to 63% of Canadians generally, and 77% of seniors aged 75 years and older. Footnote 10 While some social scientists consider this to be a developmental issue with civic pride and attachment generally increasing with age, Footnote 11, Footnote 12 others have argued that youth are more likely to feel civic pride and attachment if they feel like they have input into decisions that affect their communities. Footnote 13 Related to this is the finding that having input during youth exchanges or at youth conferences has been shown to have a positive impact on youth engagement. Footnote 14 Studies on young people and civic engagement have demonstrated that young people who have input into the activities they are involved in are more likely to feel engaged and accomplished. This is noted as being particularly important at the developmental stage of their lives when it is important to begin making independent decisions. Footnote 15 The review also found that ensuring that young people feel they have input and that their "voices are being heard" often correlates positively with involvement and engagement. Footnote 16, Footnote 17 Findings from the literature review also indicated that there is a strong link between learning the other official language and feeling an attachment to Canada. One study found that over half of 18 to 24 year-olds and 25 to 34 year-olds (58% and 54% respectively) indicated that learning the other official language would strengthen their attachment to Canada. In addition, these groups (25% and 16% respectively) indicated that learning the other official language would have the largest impact on their attachment to Canada when compared with other potential factors. Footnote 18
  • Inclusion of youth from under-represented groups. Under-represented youth (economically disadvantaged youth, Aboriginal youth, youth with disabilities, immigrant youth, religiously, ethnically or linguistically diverse youth, Francophone youth and youth in rural or remote communities) generally exhibit specific limitations to involvement. Common to all mentioned groups is limited access to sufficient financial resources, as well as a feeling of isolation from mainstream society. Footnote 19, Footnote 20, Footnote 21

Findings from the interviews with PCH officials, funding recipients and organizers and the case studies further supported the findings of the literature review. In interviews, PCH officials described needs from the perspective that youth have lower rates of attachment and engagement in civic institutions and the political process.

During interviews, funding recipients and organizers highlighted several needs of Canadian youth, including the need to better understand Canadian history, its diversity and the role of the federal government; the opportunity to combine information gathering through secondary media (e.g., books, internet, television) with experiential opportunities outside their own community; and the opportunity to share their appreciation for their community, their history and their culture with others. Key informants all highlighted the need for a program such as ECP, in particular for youth from under-represented groups and from low-income families.

Youth interviewed as part of the case studies were unanimous that there is a need for a program like ECP. Among the reasons provided were that it gave them an opportunity to expand their knowledge and horizons through exposure to new cultures and other parts of Canada and to improve their second language skills. It also gave them a chance to challenge themselves and become more independent. In addition, youth indicated that ECP responded to their need for professional growth and second language skills.

An area that was noted by many key informants was the need for youth to have opportunities for physical, face-to-face/experiential opportunities. Funding recipients noted that increased virtual communications have brought new dynamics to youth issues. Youth are savvier with, and increasingly attached to, virtual communication devices. PCH officials noted that this has resulted in increased online activity and civic engagement via online means (e.g., petitions, social media campaigns), while at the same time there is a declining interest in civic institutions and traditional means of participation. Organizers of exchanges highlighted that technology creates both new opportunities and new challenges for exchanges programming. While technology makes it easier for youth to get connected and stay connected both before and after an exchange, programs need to find new ways to successfully engage youth and, at times, to focus on the experiential opportunities over virtual/digital ones.

Despite the predominance of virtual communications in the lives of youth, in case studies and interviews, all groups noted that youth have an interest in experiential learning and that learning beyond the textbook approach of the education system was beneficial to them.

Youth who were interviewed as part of the case studies also highlighted the importance of the physical exchange experience and were unanimous that it was a key strength of the program, indicating that the Program would not have had the same impact if it did not involve an in-person experience.

The continued need for ECP is further illustrated by the fact that respondents from all groups indicated that without PCH funding, national level exchange and forum activities either would not take place at the current scale (e.g., national, diversity of participants), or would not take place at all.

ECP's responsiveness to the needs of youth

The evidence indicated that the design and implementation of ECP responded to the needs of youth in the following ways:

  • Focusing on the physical exchange experience and the skills acquisition, learning and connections associated with this approach. In interviews, respondents identified the need for youth to have opportunities to combine information gathered through secondary media with experiential opportunities outside their own community. Similarly, some respondents noted that youth reliance on social media has, in part, contributed to decreased face-to face/experiential opportunities.
  • Requiring projects to have history and heritage, as well as official language, components. Projects provide opportunities for youth to learn first-hand about the history, geography, industry, institutions, cultures, communities, languages and other facets of their country. A significant proportion of ECP projects focus on exposure to participants' second official language.
  • Setting specific targets for participation by members of traditionally under-represented groups who may not otherwise be able to participate without PCH funding. Targets exist for Aboriginal, visible minority, low-income and rural youth and youth with disabilities. Targets are based on Census 2006 (for 2009-2010–2012‑2013) and National Household Survey (NHS) (for 2013-2014) data on the representation of these groups in the Canadian population. Many projects make adjustments in delivery and recruitment to ensure inclusion.
  • Promoting a better understanding of diversity, including linguistic duality, Aboriginal culture and Canada's multicultural character. Considerable efforts are made to ensure that participants are representative of the Canadian youth population. Case studies indicated that many of the opportunities are designed to expose youth to greater multicultural diversity both through forums and exchanges.
  • Enabling youth to have direct input during youth exchanges or at youth conferences. Participants play an active role in planning the exchange. For exchanges, this includes communicating with their exchange "twin", conducting research on their host community and participating in fundraising activities.

Funding recipients and PCH officials indicated that ECP is responsive to current and emerging needs and priorities of youth. The main reason provided by key informants from both groups is that ECP's design is flexible and allows those that understand youth best—the delivery organizations—to develop the project design. PCH officials added that ECP is responsive to the needs of stakeholders, by providing programming that is relevant to teachers, youth-serving organizations, youth and the Canadian public through a core group of well-established stakeholder organizations.

4.2 Core issue 2: Alignment with government priorities

Evaluation questions

To what extent are ECP objectives and expected results aligned with federal government priorities?

To what extent are ECP objectives and expected results aligned with the priorities and strategic outcomes of PCH?

Key findings

ECP objectives and expected results aligned with federal government priorities. ECP is consistent with a commitment of the federal government to support communities, culture and heritage and groups that face greater challenges and experience barriers such as at-risk youth.

ECP aligned with PCH priorities and strategic outcomes. It falls under the Program Activity: Attachment to Canada. PCH uses ECP to create opportunities and offer experiences for youth leading to a greater knowledge and understanding of Canada. ECP results aligned with the PCH Strategic Outcome: Canadians share, express and appreciate their Canadian identity. Overall, this program is aligned with the Department's priority: Investing in our communities.

In interviews, PCH staff noted the flexibility of the Program design in responding to changing departmental priorities. The document review provided evidence of this flexibility. When history and heritage became a departmental priority, ECP was able to update its funding criteria to select projects with this focus.

Alignment with Government of Canada priorities

The document review and key informant interviews demonstrated alignment between ECP objectives and GC priorities. The 2011 federal budget indicated support for communities, heritage and culture and steps to eliminate barriers for those groups that face greater challenges and experience barriers such as at-risk youth. Footnote 22 This aligns with ECP efforts to reduce barriers to participation in exchanges among traditionally under-represented groups of youth.

Government priorities linked to ECP are also identified in the 2012 budget speech which outlines significant investment ($335 million) in culture and the arts. Similarly, the Government's 2012 Economic Action Plan outlined the priorities of "giving our young people the opportunities they deserve and we will achieve that by funding programs that benefit large numbers of young people at a reasonable cost rather than concentrating available funding on a very small number of participants at an excessive per-person cost. Our Government is proud to continue to invest in affordable, effective programming that engages youth, including Encounters with Canada, Forum for Young Canadians, and organizations that support youth, like the YMCA. Canadian Heritage will continue to invest over $105 million in youth programming to allow almost 100,000 young people to learn about their country." Footnote 23

The Speech from the Throne for the 41st Session of Parliament outlined the government priority of recognizing the importance of celebrating heritage and promoting values, concepts closely aligned with ECP outcomes. Footnote 24

All PCH officials provided examples of how ECP aligns with federal government priorities. ECP aligns with the 2013 Speech from the Throne, which put a focus on Canadians understanding their heritage and history on the "road to 2017." Footnote 25 The SWSE subcomponent of YEC is also indirectly aligned with GC economic priorities and its focus on creating jobs.

Alignment with PCH priorities and strategic outcomes

PCH's DPRs and RPPs indicated alignment with PCH's priorities and strategic outcomes. According to the 2013-2014 DPR, ECP falls under PCH Program: Attachment to Canada. PCH uses ECP to create opportunities to increase knowledge of Canada and offer experiences leading to a greater understanding of the country. Footnote 26

ECP contributed to PCH Strategic Outcome #2: Canadians share, express and appreciate their Canadian identity by fostering among youth a stronger Canadian identity through active, engaged, inclusive citizenship and recognition of the importance of both linguistic duality and a shared civic identity.

Based on the 2013-2014 DPR, ECP aligned with PCH Priority #3: Investing in our communities which is focused on helping to connect communities, contribute to healthy and vibrant communities and develop a strong sense of Canadian identity. Footnote 27

In 2013, the GC announced measures to improve access to funding for local organizations that wished to promote Canadian history in their communities, including local communities and youth groups. ECP allocated $3.6 million per year to "provide young Canadians with more opportunities to take part in history-themed events" and to support "enhanced historical content during all reciprocal exchanges and forums attended by youth so they can discover the people, places and events that make our country unique." Footnote 28

In interviews, all PCH officials indicated that ECP is aligned with PCH priorities and strategic outcomes. They also noted that when history and heritage became a departmental priority, ECP contributed to this priority by providing young Canadians with more opportunities to take part in history-themed events and by supporting enhanced historical content during all reciprocal exchanges and forums attended by youth.

4.3 Core issue 3: alignment with federal roles and responsibilities

Evaluation question

Is the delivery of ECP an appropriate role or responsibility for the federal government?

Key findings

ECP objectives and expected results aligned with some of the key responsibilities of PCH, including providing programs and policies that help all Canadians participate in their shared cultural and civic life and additional responsibilities with respect to Canadian identity and values, cultural development and heritage.

ECP aligned with some of the key responsibilities of PCH. As noted in the 2012-2013 PCH RPP:

"The Department of Canadian Heritage is responsible for programs and policies that help all Canadians participate in their shared cultural and civic life. The Department's legislative mandate is set out in the Department of Canadian Heritage Act and other statutes for which the Minister of Canadian Heritage and Official Languages is responsible and presents a wide-ranging list of responsibilities for the Minister under the heading of "Canadian identity and values, cultural development and heritage." Footnote 29

By providing funding, primarily to cover travel expenses, ECP helps young Canadians participate in their shared cultural and civic life through youth exchanges and forums that focus on themes of interest such as heritage, history, culture and official languages.

All funding recipients and PCH officials agreed that it is necessary and/or legitimate for the federal government and PCH to invest in programs such as ECP. They highlighted the appropriateness of the federal government leading on a national-scale program like ECP that aims to foster narratives of an inclusive Canadian identity.


5. Findings – performance

The following sections present the major evaluation findings related to the effectiveness as well as the efficiency and economy of the Program. Footnote 30

5.1 Core issue 4: Achievement of expected outcomes

Evaluation question

To what extent did ECP achieve its immediate outcomes?

  • Young people have access to a wide variety of exchange experiences, including exchanges related to official languages.

Key findings

ECP met its outcome of providing participants with access to a wide variety of exchange experiences. Between 2009-2010 and 2013-2014, a large number of young Canadians participated in a variety of exchange experiences, including exchanges related to official languages. ECP had approximately 63,000 participants. Annual participation in ECP‑funded projects during this period was relatively constant, fluctuating between 12,000 and 13,000. YEC accounted for approximately 53% of participants, YFC 40% and SWSE 7%.

There was considerable diversity in both project size and activities within YEC and YFC. The evaluation found that ECP provided young Canadians with experiences in a variety of locations, of varying durations, covering a wide range of subject matter and of various designs ranging from reciprocal group exchanges among younger students to longer-term employment-related exchanges among senior high school students, supported by post-secondary students.

The exposure to official languages through participation in ECP projects was high among participants. Depending on the year, the Program estimated that 35% to 63% of participants participated in projects with an official languages component. However, exposure to a second language may be higher. Approximately three-quarters of participants (71% to 76%) indicated that their participation in the ECP project had enhanced their knowledge or interest in their second official language.


Access to a variety of exchange experiences

The immediate outcome for ECP is that "young people have access to a wide variety of exchange experiences, including exchanges related to official languages." Evidence indicated that during the period from 2009-2010 to 2013-2014 approximately 63,000 youth participated in a broad variety of exchange experiences.

Figure 1 provides an overview of the number of participants by Program component and subcomponent between 2009-2010 and 2013-2014. Over this period, participation in ECP‑funded projects fluctuated between 12,000 and 13,000. The total number of participants over the five-year period was 62,811. The average number of ECP participants per year was 12,576.

Figure 1: Number of participants by YEC and YFC component and SWSE subcomponent

Source: Administrative data 2009-2014

Figure 1: Number of participants by YEC and YFC component and SWSE subcomponent – text version

SWSE

YEC

YFC

Total

2009-10

827

6968

4960

12755

2010-11

853

7172

4952

12977

2011-12

811

6760

5005

12576

2012-13

869

6431

4706

12006

2013-14

858

6253

5456

12567

Figure 2 shows that over the period of the evaluation, YEC projects accounted for slightly over half (33,852 or 53%) of participants. Its sub-component, SWSE, accounted for an additional 4,218 or 7% of participants. YFC projects accounted for approximately 25,079 or 40% of participants during this period. Within the YEC component, SEVEC exchanges accounted for approximately two-thirds of participants (65%). Similarly, Encounters with Canada accounted for two-thirds of YFC participants (65%).

Figure 2: Percentage of participants by component/subcomponent

Source: Administrative data 2009-2010 – 2013-2014

Figure 2: Percentage of participants by component/subcomponent – text version

SWSE

7%

YEC

53%

YFC

40%

Overall, participation in ECP-funded projects declined slightly (1%) over the five-year period, with a 6% decrease in participation in the YEC component (excluding SWSE), offset by increased participation in the SWSE subcomponent of YEC (4%) and in the YFC component (11%). According to program respondents, static resource levels, coupled with increasing travel costs, make it difficult for organizations to retain constant numbers of participants.

Findings from the administrative data indicated that over the period of the evaluation YFC events were held in a variety of locations in all regions and the territories. However, most events were held in Ontario (56%). Of the events held in Ontario, 87% were held in the National Capital Region. The remaining forums were generally equally distributed among the other regions (see Figure 3).

Figure 3: Number of YFC events per region

Source: Administrative data 2009-2010 – 2013-2014

Similarly, a review of participant survey results identified considerable diversity among the regions visited by participants. As illustrated in Figure 4, over the five year period of the evaluation, as anticipated given that SWSE is an official languages exchange, SWSE had a large proportion of visits to Quebec (49%). The YFC had a large proportion of visits to Ontario (71%). This can be explained by the fact that YFC events occur predominantly in Ontario and that one of the YFC projects with the largest number of participants (Encounters with Canada) is delivered in Ottawa to capitalize on the various national institutions and activities available only in the National Capital Region.

Figure 3: Number of YFC events per region – text version

British Columbia

4

Prairies

7

Ontario

38

Quebec

6

Atlantic

7

Territories

6

Figure 4: Region visited by participants (%)

Source: Participant surveys 2009-2010 - 2013-2014

Figure 4: Region visited by participants (%) – text version
BCPrairiesOntarioQuebecAtlanticTerritories

YFC

1%

3%

71%

8%

11%

7%

YEC

17%

20%

22%

29%

9%

4%

SWSE

11%

7%

27%

49%

6%

0%

Key informant interviews and the case studies indicated that ECP provided young Canadians with a multitude of exchange experiences, ranging from reciprocal group exchanges among younger students to longer-term employment-related exchanges among senior high school students supported by post-secondary students.

Youth were exposed to a variety of subject matter including Canadian politics (trips to Ottawa to meet with Members of Parliament, francophone culture (Les Jeux de la Francophonie) and northern Canada (Students on Ice). The two case studies on forums also demonstrated a wide variety of topics and themes ranging from Canada's role in the Commonwealth to culture and the arts, mental health, security and sciences. All case studies included specific activities related to history and heritage.

Second official language exposure

According to the terms and conditions of ECP, the Program must ensure that a significant proportion of exchanges (25% to 30%) are related to official languages (i.e. exchanges that encourage second-language use/exposure or allow official language minority groups to meet youth from other regions of the country who speak their language).

The document review, case studies and key informant interviews indicated that ECP has offered opportunities to participants to be exposed to both official languages. Exposure to both official languages was highlighted by project representatives as a key component in all four case studies. Similarly, in interviews all PCH officials indicated that ECP has met or exceeded its stated outcome of participants using or being exposed to their second language, namely through experiences like the SWSE and organizations that offer language-based exchanges (e.g., SEVEC).

In interviews with youth who participated in SWSE, all respondents reported that they had improved their second language abilities as a result of the exchange. Some indicated that they improved their language test scores after their participation. For most participants interviewed, participation in an exchange or forum contributed to a greater interest in learning Canada's other official language or an awareness of the importance of speaking Canada's other official language.

Most funding recipients described their projects as officially bilingual, with bilingual events, staff and communication tools (such as a bilingual website, printed materials and simultaneous translation). For example, the Encounters with Canada project included sessions that focused on promoting the benefits of bilingualism.

Each year ECP staff estimate participation in exchanges and/or events with an official language component. Footnote 31 Although the total number of participants in ECP remained relatively stable over the five years covered by the evaluation (between 12,000 and 13,000 participants), the estimated representation of participants in a project with an official languages component fluctuated between one-third (37% in 2009-2010; 35% in 2013-2014) and just under two-thirds (62% in 2010-2011 and 2011-2012; 63% in 2012-2013) of total participation. This variation may be explained by differences in project offerings from year to year.

Although PCH estimates range from 35% to 63% for the number of participants involved in an ECP project with an official languages component, the actual exposure may be higher. Post-participation survey Footnote 32 results found that as a result of their participation in the ECP project 71% of participants in 2011-2012, 76% in 2012-2013 and 74% in 2013‑2014 indicated that they had enhanced their knowledge and awareness of the other official language either moderately (20-22%), significantly (30-32%) or extremely (19‑23%). This may suggest that although a project may not have a specific official language component, the experience of travelling, meeting new people and being in a new environment may be contributing to an increased interest in the other official language.

Similarly, about three-quarters of parents and organizers completing a post-participation questionnaire between 2010-2011 and 2013-2014 indicated that participation in ECP enhanced participants' knowledge and awareness of Canada's other official language.

Evaluation questions

To what extent did ECP achieve its intermediate outcomes?

  • Young people from across Canada participate in youth exchanges; and
  • Exchange participants reflect the diversity of the Canadian youth population.

Key findings

While ECP overall had participation from all provinces/territories, it had challenges attracting and recruiting proportionally representative youth from Ontario. When compared across program components, Ontario youth were under-represented in both YEC and YFC. Quebec youth were less likely to participate in YFC events. As a result, there was an over-representation of youth from the other provinces and territories in YFC and YEC. In the case of SWSE, as expected for a French-English language exchange, Quebec youth were significantly over-represented, while other provinces and territories, with the exception of Prince Edward Island, were under-represented.

Across all components of ECP, there was a higher proportion of young women (63%) participants compared to young men. This was particularly the case for SWSE, where 73% of participants were young women.

Overall, there was considerable diversity among ECP participants with participation of Aboriginal youth, visible minority youth, youth with disabilities and low-income and rural youth. When compared across Program components, ECP participation rates were higher than the general youth population for rural, low-income and Aboriginal youth. Although the participation of youth with disabilities appears to have improved in the past three years, a broadening of the definition of disability may be contributing to the increase in participation. Participation rates were lower than the general youth population for visible minority youth.

Comparing across components:

  • SWSE had challenges in achieving representation of Aboriginal and rural youth and youth with disabilities but was more successful at achieving representation of low-income and visible minority youth.
  • YEC was successful at attracting Aboriginal, low-income and rural youth and youth with disabilities, with participation rates that met or exceeded the representation in the general youth population. However, it did not meet targets for visible minority youth.
  • YFC exceeded targets for Aboriginal youth across all years. It has improved the participation of rural youth, exceeding the target in the last two years. Similarly, in the last three years, it has exceeded the target for youth with disabilities. Visible minority youth are under-represented relative to their representation in the general youth population.

The lower participation of Ontario youth and visible minority youth, as well as the over-representation of young women, were also identified in the 2010 evaluation of ECP.

Participation of young people from across Canada

An intermediate outcome of ECP is that participants from across Canada participate in exchanges. Applications are assessed on the extent to which they ensure that participation is representative of provincial/territorial youth populations. Over the period of the evaluation, the Program established participation targets based on 2006 Census demographic data (for 2009‑2010 to 2012-2013) and the 2011 NHS for 2013-2014 in accordance with the age group of participants. Figure 5 on the next page shows the results for 2009-2010 to 2012‑2013. An analysis of 2013-2014 participation against NHS targets obtained similar results.

The evaluation found that ECP attracted participants from each region in Canada, including the territories. A comparison of ECP participants from 2009-2010 to 2013-2014 reveals that overall most regions, with the exception of Ontario, are well represented or over-represented among ECP participants.

Each case study demonstrated distribution of participants from across Canada. While some YFC events are located in one region (e.g., Encounters with Canada, National Commonwealth Student Forum), the participants are from all regions of Canada.

A comparison of the distribution of ECP participants with the provincial/territorial distribution of youth, based on the 2006 Census for 2009-2010 to 2012-2013 and the 2011 NHS for 2013-2014, found that while overall ECP has participation from all regions, when compared across components, the distribution of participants is not representative of the provincial and territorial distribution of youth in Canada:

  • For YEC, youth from all provinces and territories are well represented except in the case of Quebec and Ontario. There was an over-representation of Quebec youth and under-representation of Ontario youth. The over-representation from Quebec may be explained, in part, by the emphasis on official language exchanges.
  • SWSE also had an over-representation of Quebec youth. Youth from the other provinces and territories, with the exception of Prince Edward Island, were under-represented. This may be explained by the fact that SWSE is a French-English language exchange, where families who send their young participants to another region of the country to work and learn their second language are also expected to host other young participants in their first language.
  • The YFC component has significant under-representation of Ontario and Quebec youth and a disproportionate number of youth from the Atlantic Provinces, British Columbia, the Prairie Provinces and the territories. This was highlighted by the case studies, where there was an indication that forums encounter challenges in attracting and recruiting proportional numbers of youth, particularly from Ontario. In interviews this was attributed, in part, to the fact that many forum events are held in Ontario and may therefore be of less interest to Ontario youth.

The 2010 evaluation of the ECP made similar observations with respect to the under-representation of Ontario youth in ECP.


Figure 5: Percentage distribution of participants by province of origin (2009-2010 – 2012-2013) compared to 2006 Census distribution

Source: Administrative data 2009-2010 - 2012-2013; Census 2006

Figure 5: Percentage distribution of participants by province of origin (2009-2010 – 2012-2013) compared to 2006 Census distribution – text version

NL

PE

NS

NB

QC

ON

MB

SK

AB

BC.

YT

NT

NU

SWSE

1.0%

0.6%

2.0%

1.4%

49.3%

27.3%

1.9%

1.3%

3.4%

11.7%

0.0%

0.0%

0.0%

YEC

2.0%

1.2%

2.8%

2.4%

29.2%

23.9%

5.0%

3.8%

11.3%

15.5%

0.8%

0.5%

1.5%

YFC

5.9%

4.5%

7.3%

7.8%

11.1%

10.4%

7.1%

7.7%

12.9%

22.2%

1.1%

1.1%

1.0%

Overall

3.5%

2.5%

4.6%

4.5%

23.1%

18.7%

5.7%

5.2%

11.6%

17.9%

0.9%

0.7%

1.2%

2006 Census

1.5%

0.4%

2.8%

2.2%

22.4%

38.6%

3.8%

3.4%

11.6%

12.8%

0.1%

0.2%

0.1%


Diversity of participants

An intermediate outcome of ECP is that "exchange participants reflect the diversity of the Canadian youth population." This diversity is reflected by participation of both genders, by participation of all ages and by participation of traditionally under-represented groups. Through its project selection criteria, ECP encourages the participation of traditionally under-represented groups, including Aboriginal youth, youth from rural or isolated communities, visible minority youth, youth with disabilities and, in the case of YEC and SWSE, youth from low-income families.

In interviews and case studies, funding recipients highlighted the importance of ECP for supporting the participation of youth from a variety of socio-demographic groups. Key informants reported that by providing funding for travel costs, ECP enabled the participation of youth who would otherwise not have the financial means to participate, in particular youth from remote geographies and low-income families.

In interviews, PCH officials perceived that ECP met or exceeded its stated outcome of having participants that reflect the diversity of the Canadian youth population. PCH officials described the specific measures they have taken to improve inclusion of traditionally under-represented youth. This included setting targets for under-represented groups according to their representation in the general population and providing an incentive to meet the targets by basing funding decisions on past performance, supporting organizations that focus specifically on disadvantaged youth (e.g., Boys and Girls Club) and providing suggestions to organizations on how to reach under-represented groups.

In case studies and interviews, funding recipients and organizers indicated that their projects reached under-represented youth, including Aboriginal youth, youth from low-income backgrounds, rural youth and youth living in official language minority communities. Most organizations reported having targeted outreach initiatives, including connecting with other organizations (e.g., Aboriginal community groups, schools) and via social media. A few funding recipients indicated that to make progress in these areas, they have introduced an Aboriginal outreach program, or funded new accessibility options for participants with disabilities.

Some funding recipients indicated challenges in obtaining demographic representation of some groups. For instance, youth with disabilities do not always self-identify. Also, recruitment is a challenge for some groups, Aboriginal youth in particular.

Gender

The review of the 2009-2010 to 2013-2014 administrative data found that across all Program components, ECP attracted a larger proportion of female participants (63%) than male participants (37%). This over-representation was particularly noticeable among SWSE participants (73% female) and YFC participants (67% female) (see Figure 6).

This over-representation of female participants, particularly in SWSE, continues a stable trend identified in the 2010 summative evaluation of ECP. The 2010 evaluation found that across all components of ECP female participants represented 62% of youth participants compared to 38% males and the SWSE subcomponent had a somewhat higher percentage of female youth to male youth (76% to 24% respectively). Footnote 33

Figure 6: Percentage of participants by gender

Source: Administrative Data (2009-10 – 2013-14); Census 2006; NHS 2011

Figure 6: Percentage of participants by gender – text version

Female

Male

Census 2006 & NHS 2011 Males 15-24 (51%)

Census 2006 & NHS 2011 Females 15-24 (49%)

SWSE

73%

27%

51%

49%

YEC

58%

42%

51%

49%

YFC

67%

33%

51%

49%

Overall

63%

37%

51%

49%

Age

The majority of participants in ECP are 13 to 18 years of age. As illustrated in Figure 7, as expected, the participants in SWSE were ages 16 to 18. YFC also generally attracted participants aged 16 to 18. The YEC component included younger participants 13 to 15 years of age.

Figure 7: Percentage of participants by age group

Source: Administrative Data (2009-10 – 2013-14)

Figure 7: Percentage of participants by age group – text version

Component

≤12

13 to 15

16 to 18

19 to 21

22 to 26

SWSE

100%

YEC

17%

61%

22%

YFC

26%

69%

3%

2%

Overall

9%

43%

46%

1%

1%

Targets for aboriginal, rural and visible minority youth and youth with disabilities

ECP sets targets for representation of Aboriginal, rural and visible minority youth and youth with disabilities in proportion to their representation in the Canadian youth population. For 2009-2010 to 2012-2013 targets are based on Census 2006. For 2013-2014, targets are based on the NHS 2011. Targets for the period of the evaluation are presented in Table 3.

Table 3: Diversity targets

2006 Census

(2009-2010 to 2012-2013)

NHS

(2013-2014)

YECYFCSWSEYECYFCSWSE

Aboriginal

4.9%

4.9%

4.9%

6.7%

5.8%

6.7%

Rural

18.4%

18.4%

18.4%

20.4%

17%

20.4%

Visible Minority

18.4%

18.4%

18.4%

22.6%

21.9

22.6%

Youth with Disabilities

7.6%

7.6%

7.6%

4%

3.7%

4%

Low-Income

14.9%

N/A

14.9

15.1%

N/A

15.1

Figure 8 shows ECP's results in achieving its diversity targets relative to Census 2006 for the period 2009-2010 to 2012-2013. An analysis of 2013-2014 against the 2011 NHS resulted in similar findings.

Figure 8: Diversity results for the period 2009-2010 to 2012-2013

Source: Administrative Data (2009-10 – 2013-14

Figure 8: Diversity results for the period 2009-2010 to 2012-2013 – text version

SWSE

Rural

Low Income

Aboriginal

Youth with Disability

Visible minority

Average 2009-2020 to 2012-2013

6.7%

6.9%

2.1%

4.0%

16.8%

Census 2006

18.4%

14.9%

4.9%

7.6%

18.4%

YEC

Rural

Low Income

Aboriginal

Youth with Disability

Visible minority

Average 2009-2020 to 2012-2013

38.9%

22.8%

11.2%

8.6%

13.6%

Census 2006

18.4%

14.9%

4.9%

7.6%

18.4%


YFC

Rural

Low Income

Aboriginal

Youth with Disability

Visible minority

Average 2009-2020 to 2012-2013

23.4%

na

7.4%

10.7%

13.3%

Census 2006

18.4%

na

4.9%

7.6%

18.4%

Overall ECP

Rural

Low Income (SWSE and YEC)

Aboriginal

Youth with Disability

Visible minority

Average 2009-2020 to 2012-2013

30.0%

21.0%

9.1%

9.1%

13.7%

Census 2006

18.4%

14.9%

4.9%

7.6%

18.4%

Aboriginal youth (targets: Census 2006 - 4.9%; NHS 2011 - YEC and SWSE 6.7% and YFC - 5.8%)

Overall, ECP had a higher proportion of Aboriginal participants than the Census 2006 and NHS 2011 Aboriginal youth population. In the five years covered by the evaluation, the proportion of Aboriginal participants ranged from 8% to 10%:

  • YEC surpassed this target consistently over the five years (10 to 12%).
  • YFC met the target, ranging from 5% to 9%.
  • SWSE consistently did not meet the target (1 to 4%).

From the case studies, key informants highlighted that the challenge with SWSE meeting this target is in large part due to the programming challenges in arranging for employers in sufficiently close proximity to implement the program in Aboriginal communities.

Rural youth (Targets: Census 2006 - 18.4%; NHS 2011 - 20.4% for YEC and SWSE and 17% for YFC)

Overall, ECP was able to attract rural youth to participate in its projects and has exceeded its targets in all years covered by the evaluation. Approximately 25% to 37% of ECP participants were from rural communities:

  • YEC consistently exceeded targets.
  • YFC exceeded targets in the past two years.
  • SWSE encountered challenges achieving representation of rural youth. Evidence from the case studies indicated that this challenge was related to the nature of the programming associated with SWSE and the requirement to have sufficient numbers of employers within a community to have a group of SWSE participants.

Youth with disabilities (Targets: Census 2006 - 7.6%; NHS 2011 - 4% for YEC and SWSE and 3.7% YFC)

Participation of youth with disabilities increased significantly in the last three years covered by the evaluation period. As illustrated in Figure 8, above, overall ECP exceeded its targets with respect to representation of youth with disabilities:

  • YEC and YFC exceeded the targets in each of the last three years.
  • In the case of SWSE, youth with disabilities were consistently under-represented in SWSE in all years covered by the evaluation.

Findings from the case studies and interviews with funding recipients indicated that caution should be exercised in interpreting the data with respect to youth with disabilities. One challenge that was noted in the case studies was that the definition of disability for youth has broadened in the past few years to include learning disabilities and attention deficit disorder which may not necessarily align with the definitions used for the Census. This may account for the increase in the number of participants self-reporting a disability. Conversely, some funding recipients noted that the number may not be accurate because youth with disabilities do not always self-identify.

Youth from low-income families (targets: Census 2006 - 14.9%, NHS 2011 - 15.1%)

For YEC and SWSE, ECP collects data from funding recipients on the participation of youth from low-income families. This data is based on a subjective assessment by funding recipients. The administrative data indicated that participation rates for low-income youth exceeded their incidence in the general youth population.

Visible minority youth (targets: Census 2006 - 18.4%; NHS 2011 - 22.6% for YEC and SWSE and 21.9% YFC)

Overall, ECP was less successful in attracting visible minority youth. Although the representation of visible minority youth has improved since the 2010 evaluation of ECP (10% of participants self-identified as visible minority), the percentage of youth that self-identified as belonging to a visible minority was lower than their representation in the general youth population. Between 12% and 15% of ECP participants self-reported as being a member of a visible minority group. Compared to YFC and YEC, SWSE has been somewhat more successful at attracting visible minority youth, with the exception of 2013-2014.

Evaluation questions

To what extent did ECP achieve its ultimate outcomes?

  1. Young participants enhance their knowledge and understanding of Canada;
  2. Young participants connect and create linkages with one another; and
  3. Young participants enhance their appreciation of the diversity and shared aspects of the Canadian experience.

Key findings

Evidence from the post-participation surveys, case studies and interviews with funding recipients indicated that participation in ECP has enhanced youths' knowledge and understanding of Canada. YFC participants tended to have the highest indication that they had learned new things about Canada. Visible minority youth were more likely to agree that they had learned new things about Canada and Aboriginal youth and youth with disabilities were less likely to agree.

Creating linkages and connections with other youth was a key area of strength for ECP across all components. Overall, participants rated this outcome the highest of all outcomes measured on the post-participation survey. Compared to males, females had statistically significant higher levels of agreement that they created new ties with people from other communities as result of their participation in ECP. Aboriginal youth, youth with disabilities and visible minority youth were statistically less likely to indicate agreement.

Participants' appreciation of the diversity and shared aspects of the Canadian experience, as measured by having a better understanding of what Canadians have in common and in how they differ, was enhanced through their participation in ECP. Female and rural participants were more likely to agree that they had a greater appreciation of how diverse Canada is. Aboriginal participants and youth with disabilities were less likely to agree. On both measures (commonalities and diversity), YFC participants tended to rate these outcomes higher when compared with SWSE and YEC.

The physical, in-person aspect of the exchanges was a contributing factor to the achievement of these outcomes.

A) Knowledge and understanding of Canada

Analysis of the post-participation surveys from 2009-2010 to 2013-2014 found that almost all parents (92% to 94%) and organizers (95% to 99%) agreed "moderately," "significantly" or "extremely" that ECP participants had learned new things about Canada.

Participants' agreement levels with the statement that as a result of their participation in ECP, they had learned new things about Canada, were slightly lower, ranging between 83% and 86% with a five year average for the top three responses at 84% (Moderately (26%), Significantly (40%) and Extremely (18%)).

Figure 9 presents the overall results for the five years covered by the evaluation and by component and subcomponent.

Figure 9: Percentage of participants agreeing that they had learned new things about Canada

Source: Post-participation Survey – participants (2009-2014)

Figure 9: Percentage of participants agreeing that they had learned new things about Canada – text version

YEC

YFC

SWSE

Total ECP

Not at all

2%

1%

4%

2%

A little

17%

11%

15%

14%

Moderately

29%

23%

26%

26%

Significantly

39%

43%

32%

40%

Extremely

14%

22%

24%

18%

Comparing the top three responses across components/subcomponent, a larger proportion of the YFC participants indicated that they had learned new things about Canada (88%) compared with SWSE (82%) and YEC (82%).

Comparing target groups, there were statistically significant differences among target groups. There were statistically significant higher levels of agreement among visible minority youth that they had learned new things about Canada as a result of their participation in ECP (86%) compared with non-visible minority participants (84%). Footnote 34 Aboriginal youth were less likely to indicate agreement (80%) compared to non-Aboriginal youth (84%). Similarly, youth with disabilities were less likely to indicate agreement (80%) compared to 84% for youth without a disability.

Interviews and case studies found specific activities, such as formal learning activities, as well as interactions with a diversity of youth across the country, contributed to the enhancement of knowledge and understanding of Canada.

For example, the case studies found that the enhancement of participants' knowledge and understanding of Canada was a strong outcome for all case study projects. In interviews, the majority of participants agreed that they had learned new things about Canada. Activities that contributed towards a greater knowledge and understanding of Canada included visits to museums and galleries, participation in debates and discussions and listening to speakers. In addition to the more formal activities that were planned, this outcome was also achieved through the physical aspects of visiting other communities, seeing historic institutions such as the Supreme Court or the Parliament buildings, meeting different people, traveling, working with organizations and staying with host families.

B) Connections and linkages with one another

The outcome of young participants creating linkages and connections with other youth was the highest rated of the outcomes measured through the post-participation survey. The survey asked participants, organizers and parents their opinion as to whether participants "created new ties with people from other communities" after participation in the event or exchange. The three groups rated achievement of this very high. On average over the five years, 95% of participants, 93% of parents and 99% of organizers agreed "moderately", "significantly" or "extremely" that as a result of participation in ECP, youth had created new ties with people from other communities.

As demonstrated in Figure 10, the majority of participants provided the highest rating of "extremely" to this outcome. Although all groups rate this outcome highly, over the five years the YFC participants consistently gave this outcome a higher rating when compared with SWSE and YEC participants.

Figure 10: Percentage of participants indicating that they have created new ties with people from other communities

Source: Post-participation Survey – participants (2009-2014)

Figure 10: Percentage of participants indicating that they have created new ties with people from other communities – text version
YECYFCSWSETotal ECP

Not at all

1%

0%

1%

1%

A little

5%

3%

4%

4%

Moderately

11%

8%

12%

10%

Significantly

30%

25%

29%

28%

Extremely

53%

64%

53%

57%

Statistically significant differences were found among groups. Footnote 35 Post-participation survey results over the five years showed that female participants had statistically significant higher levels of agreement (95%) compared with male participants (94%) that they had created new ties with people from other communities as a result of their participation in ECP. Aboriginal youth were less likely to indicate agreement (91%) versus non-Aboriginal youth. Similarly, youth with disabilities were less likely to indicate agreement (90%) relative to youth without a disability (95%), as were visible minority youth (94%) relative to non-visible minority youth (95%).

In interviews and case studies, respondents indicated that creating linkages and connections with other youth was a key area of strength for ECP across all components. The structure of the various exchanges and activities allow enough time and opportunities for these connections and linkages to occur. All exchange participants interviewed as part of the case studies indicated that they had created close ties with their host family and were still in contact with them. Similarly, YFC participants interviewed were also unanimous in stating that they had created enduring ties with people from communities across Canada, including their own community.

Funding recipients reported that their participants have created new ties with other young Canadians as a result of the exchange. This is particularly the case in the last few years where social media has made it easier to stay connected. Some organizations also host alumni activities.

C) Appreciation of the diversity and shared aspects of the Canadian experience

Questions to measure appreciation of the diversity and shared aspects of the Canadian experience were added to the post-participation survey questionnaires starting in 2011-2012.

Results of the post-participation surveys, interviews and case studies demonstrated that participants' appreciation of both the diversity and the shared aspects of the Canadian experience was enhanced through their participation in ECP. The physical, in-person aspect of an exchange was a contributing factor to the achievement of this outcome. On both measures (appreciation of how diverse Canada is and better understanding of what Canadians have in common), YFC participants rated these outcomes higher when compared with SWSE and YEC over the three years measured.

Post-participation survey responses showed that high proportions of participants agreed "moderately", "significantly" or "extremely" that as a result of participation in ECP, participants have a greater appreciation of the diversity of Canada (participants (89%), parents (95%) and organizers (99%)).

As shown in Figure 11, overall approximately one-third of participants (33%) agreed "extremely" that as a result of participation in ECP, they had a greater appreciation of the diversity of Canada. Using the three top scale categories (moderately, significantly, extremely) female participants (90%) relative to male participants (88%) and rural participants (90%) relative to non-rural participants (89%) were more likely to endorse this outcome. Aboriginal participant were less likely to endorse this outcome relative to non-Aboriginal participants (86% relative to 90%).

Figure 11: Percentage of participants agreeing that they have a greater appreciation of how diverse Canada is

Source: Post-participation surveys (2011-2012 - 2013-2014)

Figure 11: Percentage of participants agreeing that they have a greater appreciation of how diverse Canada is – text version

YEC

YFC

SWSE

Total ECP

Not at all

2%

1%

3%

2%

A little

11%

5%

9%

9%

Moderately

25%

15%

20%

21%

Significantly

36%

35%

31%

35%

Extremely

26%

44%

37%

33%

Switching to the question of commonality, high proportions of parents (90 to 91%) and organizers (96 to 99%) agreed "moderately", "significantly" or "extremely" that as a result of participation in ECP, participants had a better understanding of what Canadians have in common. Participants' responses ranged between 85 and 87% with a five year average for the top three responses at 85%.

Approximately one quarter (23%) of participants agreed "extremely" that as a result of participation in ECP, they have a better understanding of what Canadians have in common (Figure 12).

Using these three top scale categories, youth with disabilities were less likely to endorse this outcome relative to youth with no disabilities (82% versus 86%).

Figure 12: Percentage of participants agreeing that they have a better appreciation for what Canadians have in common

Source: Post-participation Surveys (2009-2010-2013-2014)

Figure 12: Percentage of participants agreeing that they have a better appreciation for what Canadians have in common – text version

YEC

YFC

SWSE

Total ECP

Not at all

3%

1%

8%

3%

A little

14%

6%

16%

11%

Moderately

30%

20%

29%

26%

Significantly

35%

40%

28%

36%

Extremely

17%

33%

20%

23%

Interviews and case studies also demonstrated that ECP contributed to participants' appreciation of the diversity and shared aspects of the Canadian experience. In interviews, funding recipients and organizers reported that their participants increased their appreciation of Canada's diversities and shared aspects by coming together with diverse youth from across Canada. One strong contributing factor to the achievement of this outcome outlined by both participants and project representatives was the physical, in-person aspect of the exchanges. Examples of how this was achieved was ensuring diversity of participants, placement of participants with host families that are culturally and socially different than their own and exposing participants to different groups in communities.

5.2 Other outcomes

A number of other positive outcomes were identified by the evaluation. Information collected through the post-participation questionnaires identified positive outcomes with respect to personal skills (e.g., self-confidence, managing difficult situations and solving problems), impact on education choices and levels of attachment to community/region, province/territory and Canada. In particular, there were positive results for Aboriginal youth and rural youth with respect to influencing their decision to continue their education. Of further interest are the strengths of these outcomes for sub-groups that do not seem to be rating the logic model outcomes as highly (e.g., Aboriginal youth, visible minority youth).

In interviews, respondents from all groups reported that ECP projects resulted in additional positive outcomes. These included: participants developing leadership skills, a desire/confidence to pursue academic, professional or personal goals and a desire/confidence to become more involved in their community. PCH officials also noted anecdotal evidence of how projects have had an influence on decisions to return to school, career decisions and other life-changing experiences. Interviews with participants of the case study projects highlighted positive outcomes for participants such as increased skill levels, influence on education and career decisions, greater appreciation for travel, increased confidence and independence and increased involvement/engagement in communities by youth. These positive outcomes were confirmed by project representatives.

Another area of participant outcome measured by the post-participation survey was the impact to the levels of attachment to community/region, province/territory and Canada. In the ECP logic model promotion of and attachment to Canada are shown as the link between the ultimate outcomes and the PCH strategic outcome: Canadians share, express and appreciate their Canadian identity (effective April 1, 2010). The findings from the analysis of participant survey results demonstrated the positive effects of the ECP program on youth, as findings showed a stronger attachment to Canada than the local community/region. Taking into account the top three responses (extremely, significantly and moderately), survey results over the five years indicated that as a result of participating in ECP:

  • 69% to 72% of youth reported a greater sense of attachment to community/region (extremely (17% to 21%), significantly (23% to 27%) and moderately (23% to 25%).
  • 73% to 75% of youth reported a greater sense of attachment to province/territory (extremely (20% to 24%), significantly (25% to 28%) and moderately (16% to 19%).
  • 80% to 81% of youth reported a greater sense of attachment to Canada (extremely (31% to 34%), significantly (25% to 28%) and moderately (19% in all three years).

5.3 Core issue 5: Demonstration of efficiency and economy

Evaluation questions

  • Are the resources dedicated to ECP being used economically and efficiently to maximize achievement of outcomes?
  • Does ECP duplicate or overlap existing programs/initiatives?

Key findings

Over the period covered by the evaluation, overall there was a 5% variance between actual expenditures and reference levels. O&M expenditures were significantly lower than reference levels while Gs&Cs expenditures were almost identical to reference levels. Over the five year period, Gs&Cs expenditures remained fairly stable, while O&M expenditures were reduced by half.

The ratio of O&M expenditures to total ECP expenditures for the five years covered by the evaluation was 8.9%. This ratio steadily decreased over the past five years from 12% in 2009-2010 to 7% in 2013-2014. In this regard, between 2009-2010 and 2013-2014 the average PCH O&M expenditure per project was reduced by half ($126,000 to $64,000).

The average PCH contribution per participant varied considerably across ECP components; however, once calculated as contribution per participant per day, there was greater similarity across components. Using five year averages, the calculations found the following:

  • SWSE: 840 participants - $5,962 per participant ($129 per participant per day).
  • YEC: 6,717 participants - $1,213 per participant ($90 per participant per day).
  • YFC: 5,016 participants - $877 per participant ($152 per participant per day).

With respect to the PCH contribution as a proportion of total project costs, over the five years PCH contributions covered on average 94% of the annual project costs for SWSE and 68% of the annual project costs for YEC. In the case of YFC, there was considerable leveraging of PCH contributions, with the PCH contribution as a portion of annual project costs averaging 23% for the five years.

Although there was general satisfaction with the ECP application process, interviews with case study project representatives and funding recipients identified potential areas for improvement, including simplifying the application process, particularly for past funding recipients; reducing the review times for funding proposals; and greater use of multi-year funding.

Funding recipients considered the time between submitting an application and receipt of a decision too long. ECP has not met its service standard of 24 weeks for all applications in the three years reviewed. The timing of the funding decision also created challenges from a planning and delivery perspective for some recipients.

The evaluation examined virtual exchanges as an alternative approach to the delivery of ECP, but found that they are not a viable replacement for physical exchanges with respect to achieving similar results. While social media and information technology connections can enhance the outcomes of physical exchanges, they should be viewed as complementing rather that replacing physical exchanges.

Although there are other PCH programs similar to ECP, when detailed objectives, results, activities and delivery mechanisms were examined within the context of the evaluation, there was limited evidence of overlap with other programming. ECP shares some aspects with YTC (target populations, some theme areas and some similar outcomes); however, the activities and overall approaches are relatively distinct when examined closely. One indication of this was that there was limited overlap in funding recipients for the delivery of the two programs.

An examination of other domestic exchange programs found that ECP is unique in supporting domestic Canada-wide exchanges and forums by providing funding that is used almost entirely to support travel. Other programs examined that offered domestic exchanges tended to be between specific regions of Canada or targeted to specific groups. ECP, however, encourages exchanges between all provinces and territories and provides opportunities for all Canadian youth to go on exchanges.

The economy and efficiency of ECP were assessed using a number of indicators such as variance between actual expenditures and reference levels, the ratio of O&M expenditures to total expenditures, PCH O&M costs per project and PCH contributions per participant.

Interviews with PCH officials noted a number of areas where ECP demonstrated delivery efficiencies. These included:

  • encouraging interested applicants to consult a program officer before submitting an application to ensure eligibility and good fit. This saved PCH time during the review process and unprofitable work for applicants; and
  • changes to the level of approvals, leading to greater efficiencies.

The departmental Grants and Contributions Modernization Initiative (GCMI) will offer online funding applications through the Enterprise Online System, which is expected to increase efficiency going forward.

Variance between Actual Expenditures and Reference Levels

As illustrated in Table 4, over the period covered by the evaluation (2009-2010 to 2013‑2014), as a result of departmental budget reductions, O&M expenditures were approximately 39% lower than reference values (i.e. $8.67 million versus $14.14 million). Gs&Cs expenditures were essentially identical to reference values (i.e. $88.88 million versus. $88.93 million). When the two types of expenditures are combined, total expenditures were approximately 5% lower than reference values ($97.55 million versus $103.07 million).

Table 4: Variance between budgeted and actual expenditures

Resources

2009-2010

2010-2011

2011-2012

2012-2013

2013-2014

Total

Budgeted resources

O&M

$3,775,302

$3,270,774

$2,922,049

$2,526,333

$1,643,305

$14,137,763

Actual expenditures

O&M

$2,523,266

$1,895,878

$1,663,367

$1,316,310

$1,270,609

$8,669,430

Variance

O&M

$1,252,036

$1,374,896

$1,258,682

$1,210,023

$372,696

$5,468,333

Budgeted resources

Gs&Cs

$17,786,359

$17,786,359

$17,786,359

$17,786,359

$17,786,359

$88,931,795

Actual expenditures

Gs&Cs

$17,882,489

$18,133,151

$17,458,496

$17,562,353

$17,845,216

$88,881,705

Variance

Gs&Cs

$96,130

$346,792

($327,863)

($224,006)

$58,857

$50,090

Source: STAR system

Ratio of O&M expenditures to total expenditures

One indicator of efficiency is the ratio of O&M expenditures to total expenditures. Across the period of the evaluation, the ratio of O&M expenditures to total expenditures was 8.9% (excluding the administration costs of the delivery organizations). Over the five years covered by the evaluation, this ratio steadily decreased from approximately 12% in 2009-2010 to approximately 7% for the last two years. Over the five years, O&M expenditures were reduced by half; meanwhile, the same number of projects was processed, indicating that program delivery has become more efficient in recent years.

In comparison to other Gs&Cs programs, the five year ratio was higher than the ratios found in some of the recent evaluations of PCH programs with a Gs&Cs component. For example, the ratio for YCW Initiative was 6.2%and the ratio of the CSP/CHF was 5.5%. However, it was lower than the ratio for the YTC Program, which was approximately 14%.

O&M costs per project

Regarding Gs&Cs agreements that support ECP projects, the average PCH O&M expenditure per project funded gives a sense of the efficiency with which this output is attained. The average O&M expenditure per project over the five years (2009-2010 to 2013-2014) ranged from $120,000 in 2009-2010 to 60,000 in 2013-2014, reflecting that O&M expenditures were reduced by half.

Table 5: O&M costs per project

2009-2010

2010-2011

2011-2012

2012-2013

2013-2014

A) # of Funded Projects

21

22

15

15

21

B) O&M expenditures

$2,523,266

$1,895,878

$1,663,367

$1,316,310

$1,270,609

Average O&M expenditures per project

$120,155

$86,176

$110,891

$87,754

$60,505

Source: STAR system

PCH contribution per participant

The average ECP contributions per participant, as well as the average ECP contributions per participant per event day, were calculated by program component for the 2009-2010 to 2013‑2014 period. The total PCH contributions per participant by project ranged from approximately $100 for one event, to approximately $6,000 in the SWSE component.

Given that events and activities have different lengths, a better comparator is the contribution per participant per event day. As shown in Table 6, the average contribution per event per day was $129 for SWSE, $90 for YEC and $152 for YFC. This puts the SWSE subcomponent into perspective, which had the higher cost per participant before taking into consideration the length of the activity. In general, costs for YEC programs were lower, ranging between $80 and $73 per day per participant. Events organized by YFC were, on average, higher than exchanges and SWSE.

Table 6: Average ECP contribution per participant and per participant per event day, five-year average, by component and subcomponent

ECP component/ subcomponent

Average number participants

Contribution per participant

Contribution per participant per day

SWSE

840

$5,962

$129

YEC

6717

$1,213

$90

YFC

5016

$877

$152

Source: Administrative data (2009-2014)

The average ECP contribution per participant in relation to other youth-focused programming at PCH was examined. Similar calculations resulted in an average PCH contribution per participant of $4,530 for the YCW Initiative and $14 to $51 for YTC participants. Footnote 36 PCH officials noted the challenges in making cost comparisons between youth programs as some programs involve a higher contribution per participant due to the lower number of overall participants reached but provide a greater in-depth experience. In addition, programs that cover travel costs for participants tend to have a higher PCH contribution per participant.

PCH officials noted that ECP has a wider reach than the approximately 12,500 youth that participate in ECP annually. All noted the "ripple effect" of ECP that occurs because "projects empower youth to take action in their community." Many forums requested/required participants to take follow-up action in their communities and youth were taught techniques on how to engage with different types of stakeholders in their community.

PCH contribution as proportion of project costs

The ratio of PCH contributions to total project costs is an indication of extent to which the PCH contribution is being leveraged. The evaluation found that over the five years PCH contributions covered most of the annual project costs for SWSE and two thirds of the annual project costs for YEC. In the case of YFC, there was considerable leveraging of PCH contributions:

  • SWSE: The portion of the total annual project costs covered by the contribution from PCH ranged between 84% and 99%, with a five year average of 94%.
  • YEC: The portion of the total annual project costs covered by contributions from PCH ranged from 64% to 75%, with a five year average of 68%.
  • YFC: The portion of total annual project costs covered by contributions from PCH ranged from 14% to 42%, with a five year average of 23%.

Key informant interviews with funding recipients, indicated that YEC and YFC projects received support (financial and in-kind) from other sources to a varying degree, namely from private companies (e.g., banks and insurance companies), foundations and non-profits, as well as provinces, local community/municipal organizations and individual donors. Participants also paid part of the costs in some cases. According to funding recipients, it was difficult to find other funders willing to support travel costs.

Opportunities for improvement

Interviews with case study project representatives and funding recipients identified challenges with the application process and the timelines associated with the review of funding proposals.

Evidence from key informant interviews indicated that most funding recipients were generally satisfied with the application process. However, about a third of funding recipients interviewed commented on challenges they faced with the application process. Among the issues raised was that that the process was long, detailed and time-consuming. A few funding recipients suggested that the application process could be simplified for past funding recipients via an abbreviated renewal application form. PCH officials noted that the online application process which is to be implemented in 2015-2016, is expected to make the application process easier and more efficient for funding recipients.

All case study project representatives and most funding recipients indicated that the length of time between application submission and notification of funding was too long. This was particularly the case for applications to YFC. For some projects, confirmation of funding came late, which puts stress on planning and delivery. Another issue related to the lengthy timeline is that recipients often have to submit their applications before they have a solid idea of the fee for the following year (for example online registration and how much they should charge). This can result in difficulties with "clawbacks" if their estimates are incorrect. As a result too many revisions need to be made to the application after it has been submitted. Some funding recipients suggested that multi-year funding would provide stability and resolve some of these issues and allow for improved project planning and delivery.

ECP's service standard is to issue official written notification of the funding decision within 24 weeks. As shown in Figure 13, in the case of the Youth Forums component, in the last three years covered by the evaluation, ECP did not meet its service standard with respect to notifying applicants of the funding decision in all cases.

Figure 13: Percentage of applications meeting service standard for notification of funding decision (YFC)

Source: Service Standards for Canadian Heritage Funding Programs

Figure 13: Percentage of applications meeting service standard for notification of funding decision (YFC) – text version

% Met

% Not Met

2011-12

29%

71%

2012-13

63%

37%

2013-14

50%

50%

Alternative approaches to achieve similar results

The evaluation examined alternative and potentially more economical or efficient approaches to deliver ECP that would produce similar results. The evaluation examined the viability of virtual exchanges to achieve similar results as ECP. According to the literature review, virtual exchanges are receiving increased attention as an alternative, but few actual steps are being taken in this direction with the exception of some initiatives in the United States. The US Department of State has created a few programs that use virtual exchanges such as the BRIDGE program, Global Connections and Exchanges and the Chris Stevens Youth Network, which have been specifically created for building positive relationships and increasing cultural understanding between youth in the United States and youth in the Middle East. Of these programs, the ones which have been evaluated have received positive feedback, suggesting that depending on the context and if properly implemented, virtual exchanges could be an innovative alternative. The literature review also found that some studies have taken the viewpoint that while virtual civic engagement might not be a full replacement for a face-to-face engagement opportunity, a virtual connection is increasingly important for facilitating one. Footnote 37 This is supported to some extent by the finding from the literature review that the use of social media to connect and remain connected is becoming increasingly popular among youth (and other generations). Footnote 38

While there was some indication from the literature review that there could be positive benefits from virtual exchanges (although virtual exchanges were often complemented with a physical exchange component), the findings from the interviews and case studies showed that the physical exchange component was perceived as essential to obtaining the outcomes for ECP. Respondents from all groups indicated that the in-person exchange is important for achieving the Program's results and that virtual exchanges would not be as effective. According to funding recipients and organizers, the physical experience of a different Canadian community is essential and makes ECP a unique and particularly transformative experience for participants. They highlighted that, relative to other methods, in-person experiences are most effective for creating understanding and memorable learning and for fostering stronger bonds among people. Funding recipients explained that virtual communications are an important complementary tool to the in-person experience, but that removing the latter would reduce the impact of the program on participants.

Youth interviewed as part of the case studies were unanimous in expressing the importance of the in-person/physical exchange experience, indicating that the physical exchange has important advantages over virtual exchanges for language learning, learning about a new community, creating ties with other Canadians and personal growth.

The evaluation also explored potential alternative delivery approaches for ECP. ECP was consistently identified in interviews and cases studies as unique in that it essentially funds national travel. According to PCH officials and almost all funding recipients other levels of government and the private and non-profit sectors are less likely to provide funding for national travel.

Duplication or complementarity with other programs

The literature review indicated that ECP is relatively unique in supporting domestic Canada-wide exchanges by providing funding that is used almost entirely to support travel. Many of the other programs that were reviewed, offered by other federal departments and provincial governments, focus on international exchanges designed to build "global competencies". The programs reviewed that offered domestic exchanges tended to focus only between specific regions of Canada, or were targeted at specific groups of under-represented youth. In contrast, ECP encourages exchanges between all provinces and territories and provides opportunities for all Canadian youth to go on exchanges.

Key informants were not aware of any programs that duplicate ECP. Funding recipients noted that ECP is unique in supporting exchanges at the national level and in providing funding that can be used almost entirely to support travel. Key informants mentioned similar programs to ECP, such as: the Cadet movement, Imagine Canada, Rotary Exchanges (are international), Scouts Canada, Canada World Youth, programs offered at the provincial level (e.g., debate competitions, conferences), inter-provincial exchanges (e.g., Quebec, BC used to have a program) and exchanges offered by for profit companies (e.g., EC Tours). PCH officials noted that these programs do not have as large of a scope or the same objectives as ECP.

PCH invests about $75.6 million annually in programs that benefit youth. A general analysis of these PCH programs (ECP and components of other programs, including the YCW Initiative, CHF and initiatives funded through the Official Languages Support Program (Explore, Destination Clic and Odyssey) was undertaken for the purposes of the evaluation (See Appendix C). An analysis of documents from these other PCH programs found that, when detailed objectives, results, activities and delivery mechanisms were examined within the context of the evaluation, there was limited evidence of overlap with other programming.

The document review found that several PCH programs share with ECP the common characteristics of serving youth; however, they differ in their specific objectives. While some contribute to the development of official languages, other programs funded by PCH through the Official Languages Support Programs (Explore, Destination Clic and Odyssey) focus on this as their main mission. For example:

  • Explore offers students aged 16 and older an opportunity to receive a bursary to study English or French as a second language, for five weeks, at one of a number of affiliated Canadian educational institutions to strengthen their bilingualism and learn about the culture of the other linguistic community in Canada.
  • Destination Clic is a summer bursary program for francophone students in Grades 8 and Grades 9 attending a francophone school outside of Quebec. This program allows them to take an intensive French course in a Francophone institution in Ottawa, the province of Quebec or Moncton, thereby enhancing their knowledge of their mother tongue and Francophone culture.
  • Odyssey is a full-time work experience as a language assistant that provides post-secondary students the opportunity to travel to another part of Canada and share their language and culture with another community.

Others (SWSE, YEC, YFC and CHF) aim primarily to enhance knowledge of Canada. Others yet focus on engagement (YTC) or labour market skills (YCW). Even within these clusters, which have the most in common, it is clear that variants or components are necessary either to address specific subgroups (e.g., Explore for second language learning versus Destination Clic to strengthen French as a first language) or entice youth participation by different means (the three components of ECP for example). This level of variety is required to reach as many youth as possible within the overall common threads.

Another indication of complementarity of ECP with other PCH programs is that the organizations that received funding through the various programs revealed very few common funding recipients.

5.4 Other Evaluation questions

Evaluation questions

  • Is an adequate performance measurement strategy in place to report on program results?
  • What, if any, changes to performance measurement are required?

Key findings

PCH officials considered the current performance measurement and other monitoring mechanisms effective in capturing results.

However, there are opportunities to improve the performance data collected and used for decision-making and program improvements. While the program produced annual reports and monitored participation numbers, the evaluation did not find evidence of trend analysis across the years assessing outcomes by component and by target group which could inform program improvements. The evaluation also did not find evidence that the qualitative data collected from funding recipients through the final project reports was being consolidated and analyzed to identify trends, best practices or areas for improvement to supplement the quantitative data being collected through the post-participation surveys.

According to feedback from funding recipients about the post-participation survey for youth, it is long and difficult to administer widely and in a way that ensures quality responses. Some funding recipients suggested that the survey be made available online. A pilot of an online survey with six organizations was undertaken in 2012-2013 and 2013‑2014. The four organizations with the largest number of participants had the lowest response rates. However, the two smallest pilot organizations had a response rate of over 90%.

The survey instrument for participants was also considered difficult for youth to understand and some questions were found to be unclear (e.g., how a disability is defined).

Performance measurement

ECP's Performance Measurement, Evaluation and Risk Strategy (2010) outlines the main components of a PMERS, including program profile, logic model, performance measurement strategy, evaluation strategy and a section on risk assessment and management. Expected results, indicators, data sources, timing and reporting requirements are clearly presented.

The 2010 terms and conditions outline key points of the ECP Performance Strategy, indicating expected results and performance indicators. A review of sample contribution agreements identified clear indications of implementation of the PMERS through the agreement via the reporting requirements for each agreement. Contribution agreement signatories must report on project/programming activities, outcomes, official languages, heritage and history content, participation, distribution, outreach activities and special conditions.

Performance data were reported in the DPR and RPP, as well as through the Official Languages Roadmap. PCH officials indicated that they use performance data to inform funding decisions, namely whether to continue supporting a project and with how much funding.

Opportunities for improvement

ECP relies on self-reported post-participation survey results from participants and also collects survey results from parents and organizers of exchanges, to measure intermediate and ultimate outcomes. Surveys are administered immediately or shortly after participation in the exchange or forum and there is no follow-up with participants or with funding recipients to assess lasting program impacts.

ESD did not conduct a survey of participants as part of this evaluation to assess ECP's longer-term impacts. To do so would have required the funding recipients to maintain up-to-date contact information for participants and for ECP to collect this information (email addresses and phone numbers) from funding recipients as well as confirmation that participants' consent had been obtained. For privacy and security reasons, the Program does not collect this information.

ECP compiled annual performance reports based on administrative data and reports received from funding recipients and which included information on target population, participation and financial data, including cost per participant. PRG produced annual reports on the results of the post-participation surveys. A one-time trend analysis report on participation numbers was done for the period 2000 to 2012 but this analysis has not been updated. In conducting the evaluation, the annual format of reports, as well as variations in survey questions and how survey results were reported across the years covered by the evaluation made it difficult to undertake trend analysis across components and target groups and necessitated the creation of a database that enabled such analysis to be undertaken for the period covered by the evaluation. Consideration could be given, therefore, to compiling the information in a form that would enable more efficient analysis of trends across components and years.

There are opportunities to collect, analyze and make use of information being collected from funding recipients through their progress reports on their perceptions of program outcomes, best practices and lessons learned.

The program collects additional information from funding recipients and participants which could be analyzed over a longer period to assess trends in the achievement of outcomes by component and by target group and to compare projects costs relative to outcomes to determine which projects are most cost-effective. This qualitative data could supplement the quantitative data collected through the post-participation surveys of participants, parents and organizers. This information could inform program improvements and project selection.

Observations from individual project reports are another source of information to identify common issues across projects, trends and potential areas for improvement. While project final activity reports, including the qualitative information in the reports, are analyzed by a program officer on a project-by-project basis and a report entered in GCIMS, the evaluation did not find evidence that observations from individual project reports are being consolidated, with a view to identifying common issues across projects, trends and potential areas for improvement. In interviews, PCH officials did indicate that there are opportunities to use more of the information collected in recipient reports, including trend analysis and a roll-up of learnings from final reports. According to PHC officials, a roll-up of learnings from final project narrative and activity reports does not happen due to lack of staff resources. To strengthen and confirm the quantitative data collected through the surveys, however, the Program should consider supplementing these quantitative results with qualitative information from funding recipients on their perceptions of the achievement of project outcomes, challenges encountered, lessons learned and opportunities for improvement.

More than half the funding recipients interviewed identified issues with the content and administration of the participant survey. Among the issues raised is that the survey is long and difficult for youth to understand, noting also that some questions are unclear (e.g., how a 'disability' is defined). Also, the survey is difficult to administer widely in a way that ensures quality responses. A few PCH officials and funding recipients suggested that the level of effort of administering the participant survey could be reduced by moving it online (some questionnaires are already online) or removing questions about outcomes that are not directly related to the program's stated outcomes (e.g., independence, skills built, plans to return to school).

The Program piloted an online survey in 2012-2013 and in 2013-2014. Administrative data indicated that response rates ranged from a low of 9.5% to a high of 96.5%, with results for the latter comparable to the previous year's results for the paper survey. The four pilot organizations with the largest number of participants had the lowest response rates. As a result, the overall participant survey response rate went to 54% in 2012‑2013 and 50% in 2014‑2015 from approximately 80% in the three previous years. However, two organizations with the smallest number of participants had a response rate of over 90%. The Program should offer the survey online to those organizations that would prefer the online version and investigate the barriers to the successful implementation of an online survey more broadly with a view to finding ways to resolve the barriers.


6. Conclusions and Recommendations

6.1 Conclusions

Relevance

All lines of evidence demonstrated the continued relevance of ECP. The evidence from the literature and key informant interviews indicated that there is a need to address concerns that youth are not knowledgeable about Canada and its civic systems and political issues; are less interested than older cohorts in Canadian history; feel less pride in Canada/being Canadian; and are less likely to feel attached to Canada. ECP was responsive in that it directly addressed some of these concerns with respect to understanding Canada, developing a Canadian identity and attachment by providing opportunities for youth to learn about their history and heritage; promoting a better understanding of diversity, including linguistic duality, Aboriginal culture, Canada's multicultural character; ensuring inclusion of under-represented groups; and by making adjustments in delivery and recruitment to ensure inclusion and by providing experiential opportunities during key developmental periods, characteristic of adolescence and early adulthood.

ECP objectives and expected results aligned with federal government priorities as outlined in speeches from the throne and recent GC Budgets.

ECP objectives and outcomes of the program were closely aligned with PCH's strategic outcome: Canadians share, express and appreciate their Canadian identity and with the PCH priority (#3): investing in our communities.

Delivery of ECP was appropriate for the federal government given its national scope and delivery and expected results in the areas of attachment to Canada and shared Canadian identity.

Performance – achievement of expected outcomes

ECP provided youth opportunities with a wide variety of domestic exchange experiences in terms of location, content, duration and design. Exposure to the second official language appeared to be high among participants. There were some challenges with obtaining a distribution of ECP participants according to region of origin that was comparable to the distribution of youth in Canada. Ontario youth were significantly underrepresented in both YEC and YFC. Quebec youth were less likely to participate in YFC events.

ECP participants were primarily younger youth under 19 years of age, distributed across a broad demographic profile including traditionally underrepresented youth. There appeared to be some barriers for young men participating in ECP opportunities, as a disproportionate number of young women were participants.

Overall, there was considerable diversity among ECP participants with participation of Aboriginal youth, youth from low-income households, visible minority youth, youth with disabilities and rural youth. ECP participation of rural, low-income households and Aboriginal youth exceeded their incidence in the general youth population in Canada. Although the participation of youth with a disability appeared to have improved in the last three years covered by the evaluation, changes to the definition of disability may have contributed to this increase. Overall, visible minority youth were under-represented relative to the general youth population.

Comparing across components, YEC exceeded targets for Aboriginal, youth from low-income households, rural youth and for youth with disabilities, but did not meet targets for visible minority youth. SWSE had challenges in achieving representation of Aboriginal and rural youth and youth with disabilities but was more successful at achieving representation of youth from low-income households and visible minority youth. YFC exceeded targets for Aboriginal youth across all years. It has improved the participation of rural youth, exceeding the target in the last two years. Similarly, in the last three years, it has exceeded the target for youth with disabilities. Visible minority youth were under-represented relative to their representation in the general youth population.

Overall, ECP contributed to some extent to the three anticipated ultimate outcomes (understanding, connecting, appreciating diversity/shared experiences) for the large majority of participants (>80%).

The strongest outcome was in the area of connecting and creating linkages with one another. Across the three ultimate outcomes, Aboriginal youth and youth with disabilities were less likely to indicate that these outcomes had occurred for them.

In addition to the anticipated outcomes, ECP contributed significantly to other positive outcomes. Some of these were in the area of personal skills that could be considered as facilitators or levers to obtaining the anticipated outcomes for ECP. As well, they were key considerations for meeting various needs of youth from a developmental perspective including a range of employability/soft skills and increased interest in continuing education. Overall, these were strong results for SWSE participants and visible minority youth. Also there were positive results for Aboriginal youth and rural youth with respect to influencing their decision to continue their education.

The other area in which ECP was found to be demonstrating an impact was with respect to increasing levels of attachment to community/region, province/territory and Canada. This was an important outcome given ECP's link to PCH strategic outcome—attachment to Canada. This was a very strong result for YFC participants and was an area of considerable positive impact for Aboriginal youth and visible minority youth.

Performance – efficiency and economy

Actual expenditures compared with reference values were relatively close with the difference primarily due to increased levels of efficiency as reflected in reduced O&M expenditures across the five years studied. There was strong evidence of improved efficiency of the program from a PCH perspective with a considerable decline in expenditures of O&M while Gs&Cs expenditures remained constant.

The ratio of O&M over total PCH expenditures on average over the five years was approximately 8.9%. In the final year studied (2013-2014), the Program attained a 7% ratio. In comparison to other recently evaluated Gs&Cs programs, the five year ratio was higher than the ratios for YCW Initiative (6.2%) and the ratio of the CSP/CHF (5.5%). However, it was lower than the ratio for the YTC (14%).

The PCH contribution per participant varied considerably by component and subcomponent, but was much more comparable once the duration of activities was considered. A more appropriate metric, for comparison purposes, is the PCH contribution per participant per day of activity.

The ratio of PCH contributions to total project costs is an indication of the extent to which the PCH contribution is being leveraged. The evaluation found that over the five years PCH contributions covered on average 94% of the total annual project costs for SWSE and 68% of the total annual project costs for YEC. In the case of YFC, there was considerable leveraging of PCH contributions with the PCH contribution covering 23% of the total annual project costs.

The ECP design and model provided adequate flexibility and guidance for effective delivery of the program. The physical exchange experience appeared to be a contributing factor to achieving outcomes. Funding recipients identified opportunities for improvement by increasing the timeliness of the funding decision and by reducing the complexity of application process.

The evaluation examined virtual exchanges as an alternative approach to deliver ECP. Multiple lines of evidence (literature review, case studies and interviews) confirmed that virtual exchanges may have some benefits, but are unlikely to achieve the same anticipated outcomes as physical exchanges. There was evidence that virtual exchanges may be a beneficial complement to the physical exchange activities being undertaken, but are not a replacement.

An examination of other domestic exchange programs found that ECP is unique in supporting domestic Canada-wide exchanges and forums by providing funding that is used almost entirely to support travel. Other programs examined, that offered domestic exchanges, tended to focus only between specific regions of Canada or targeted specific groups. ECP, however, encourages exchanges between all provinces and territories and provides opportunities for all Canadian youth to go on exchanges.

PCH invests about $75.6 million annually in programs that benefit youth. There are some programs, including the CHF and the Canada Arts Training Fund that are not exclusively aimed at youth but have a broad target group that includes youth. An analysis of documents from these other PCH programs found that, when detailed objectives, results, activities and delivery mechanisms were examined within the context of the evaluation, there was limited evidence of overlap with other programming. ECP shares some aspects with YTC (target populations, some theme areas, some similar outcomes); however, the activities and overall approaches are relatively distinct when closely examined. One indication of this is that there was little overlap in funding recipients for the delivery of the two programs.

Performance - performance measurement and monitoring

A PMERS was developed and implemented for ECP. The Program has identified performance indicators for each planned outcome of the Program corresponding to the logic model. These outcomes were tracked. PCH officials have deemed the PMERS as adequate and supporting many of their needs for monitoring the performance of the program. However, there are opportunities to improve the performance data collected and used for program decision-making and program improvements. There is a reliance on quantitative information collected through the post-participation surveys of participants, parents and organizers to measure intermediate and ultimate outcomes. Surveys are administered immediately after participation in the exchange or forum and there is no follow-up with participants or funding recipients to assess lasting program impacts.

While the program produced annual performance spreadsheets and PCH's PRG produced a report annually on the results of the post-participation survey, the evaluation did not find evidence of trend analysis across the years assessing outcomes by component and by target group which could inform program improvements.

The evaluation also did not find evidence that observations from individual project reports are being consolidated, with a view to identifying common issues across projects, trends and potential areas for improvement. In interviews, PCH officials indicated that there are opportunities to use more of this information. Consideration should be given, therefore, to supplementing the quantitative results of the post-participation surveys of participants, organizers and parents with the qualitative information obtained from funding recipients, specifically their perceptions of the achievement of project outcomes, challenges encountered, lessons learned and opportunities for improvement.

Feedback from funding recipients about the post-participation survey for youth was that it was long and difficult to administer widely and in a way that ensured quality responses. Some funding recipients suggested that the survey should be made available online. A pilot of an online survey was undertaken with six organizations in 2012-2013 and 2013-2014. The four pilot organizations with the largest number of participants had the lowest response rates which had a significant impact on the overall response rate for the participant survey in the two years of the pilot. However, two pilot organizations with the smallest number of participants had a response rate of over 90%.

The survey instrument for participants was also considered to be difficult for youth to understand and some questions were found to be unclear (e.g., how a disability is defined).

6.2 Recommendations and management response

The following three recommendations emerged from the evaluation findings. It was noted that some of challenges and issues identified in the 2010 evaluation of ECP continued to be challenges and issues during the period covered by this evaluation, including issues with respect to the timeliness of the funding decision process and performance measurement and challenges with respect to the achievement of targets for certain groups and regions. There continue to be opportunities for improvement in these areas.

Recommendation 1

To improve the efficiency of the funding application, the Assistant Deputy Minister of Citizenship, Heritage and Regions should streamline the application process and monitor the funding decision process to ensure timeliness of the decision to release funds and reduce the complexity of the application process.

Statement of agreement /disagreement

Management agrees with this recommendation.

Management response

The Program will update its intake mapping document to demonstrate and monitor compliance with the established service standards (24 weeks) through its intake and assessment process in fall 2015, for funding of projects in 2016-2017. The Program met its service standards for funding decisions for projects starting in 2015-2016 and will continue to strive to meet them in the future. Furthermore, the Program will increase access to multi-year funding for Program recipients.

In terms of the complexity of the application process, the Program will review its application forms with the objective to streamline them for applicants. It will also work in collaboration with the Chief Information Officer's Branch (CIOB) to make its application process available on-line as part of the Program's transition to the Enterprise On-line System. A simplified application form and on-line access will contribute to reducing the complexity of the ECP application process.

Deliverable(s)TimelinesOPI

Report on recruitment barriers consultation with ECP recipients.

March 31, 2016

Director, Youth Participation

Develop recruitment options paper and feasibility analysis.

March 31, 2016

Director, Youth Participation

Share recruitment options with recipients and require implementation of suggested recommendations in Contribution Agreements.

January 31, 2017

Director, Youth Participation

ECP Participant Highlights Report.

November 30, 2017

Director, Youth Participation

Recommendation 2

To strengthen the collection and use of outcome data for management decision-making and program improvements, the Assistant Deputy Minister of Citizenship, Heritage and Regions should:

2.1 consult with funding recipients to modify the language and content of the post-participation survey instruments, particularly for participants;

2.2 provide funding recipients with the option to conduct on-line surveys and consult with recipients to identify barriers to administering the surveys on-line; and

2.3 collect and analyze the qualitative data in the final progress reports submitted by funding recipients to supplement the quantitative outcome data collected through post-participation surveys with funding recipient perceptions of project outcomes, best practices and lessons learned.

Statement of agreement /disagreement

Management agrees with this recommendation.

Management response

2.1 The Program will revise its participant questionnaires for the 2016-2017 programming cycle. It will work with organizations to make clarifications to the language and content of the questionnaires. The Program will also collaborate with the Department's Strategic Policy, Planning and Research Branch to ensure that the questionnaires continue to provide the appropriate data to support decision-making and reporting on program outcomes.

2.2 The Program will consult with recipients to understand barriers and will expand access to on-line questionnaires. The program will also continue to monitor response rates and ensure reliable results for each funded project.

2.3 The Program will use the final report analyses conducted by program officers to identify qualitative findings on outcomes, best practices and lessons learned. These qualitative findings will be added to the annual ECP Program Participant Highlights report, to better inform program decision making.

Deliverable(s)TimelinesOPI

Consultation and revised ECP participant questionnaires.

March 31, 2016

Director, Youth Participation

Report on results of on-line survey consultation with ECP recipients.

June 30, 2016

Director, Youth Participation

Online survey implementation plan with ECP recipients.

September 30, 2016

Director, Youth Participation

Results of online survey implementation

September 30, 2017

Director, Youth Participation

Qualitative Findings presented in ECP Program Participant Highlights Report.

November 30, 2016

Director, Youth Participation

Recommendation 3

While overall, ECP program participants generally reflect the demographic and provincial/territorial diversity of the Canadian youth population, certain components of the program are under-subscribed by certain segments of the population (such as young men, Aboriginal, rural and visible minority youth and youth with disabilities). Therefore, the Assistant Deputy Minister of Citizenship, Heritage and Regions should work with funded organizations to identify the barriers to participation and to develop and implement approaches to attract participation of target groups and provinces/territories, as applicable.

Statement of agreement /disagreement

Management agrees with this recommendation.

Management response

The Program will monitor overall participant diversity representation for its funded projects and will continue to assess organizations based on proposed participant distributions identified in their applications, as well as their track record in achieving targets.

The Program will consult with recipients to better understand barriers to participation and then develop options to better reflect the diversity of certain segments of participants that have been previously under-represented in ECP components.

The Program will also consult with the Department's Strategic Policy, Planning and Research Branch to better understand factors which may have an impact on participants identifying or not as visible minority, disabled, or Aboriginal.

Taking into account these factors, the Program will work with organizations to meet the target levels identified in their Contribution Agreements through the use of specific measures, including targeted outreach plans, organizations supplementing participant registration fees, holdbacks and other measures.

Finally, the annual ECP Participant Highlights report will present a report card on each funded projects' ability to meet its targets (as well as the Program overall) and it will function as the tool for program management to chart recipient performance and make further adjustments.

Deliverable(s)TimelinesOPI

Updated program intake mapping document

October 30, 2015

Director, Youth Participation

Access to multi-year funding available for next Program cycle

March 31, 2016

Director, Youth Participation

Revised program application forms ready for approval

December 31, 2016

Director, Youth Participation

Online availability of ECP Program's revised application forms

December 31, 2017

Director, Youth Participation; CIOB


Appendices

Appendix A: Logic model

Exchanges Canada program logic model – Text version


PCH Activities:

  • Management of funding process
  • Providing Canadians with information on exchanges/undertaking targeted promotion

PCH Outputs

  • Grants and contribution agreements in support of youth exchanges
  • Information about youth exchanges is provided to Canadians

Immediate Outcomes:

  • Young people have access to a wide variety of exchange experiences, including exchanges related to official languages
  • Exchange participants reflect the diversity of the Canadian youth population

Ultimate Outcomes

  • Young participants enhance their knowledge and understanding of Canada
  • Young participants connect and create linkages with one another
  • Young participants enhance their appreciation of the diversity and shared aspects of the Canadian experience

Link to PCH Strategic Outcome

  • Promotion of and Attachment to Canada
  • Canadians share, express and appreciate their Canadian identity (Effective April 1, 2010)

Appendix B: Evaluation framework

Evaluation questions

Indicators

Data sources

Methods of collection

  1. To what extent is there a demonstrated need for ECP?
  1. Is ECP responsive to the needs of Canadian youth?
  • Evidence and perceptions, academic discussion and key stakeholder testimonial as to the extent to which there is a need for Canadian youth to:
  • Enhance their knowledge and understanding of Canada
  • Connect and create linkages with one another
  • Enhance their appreciation of the diversity and shared aspects of the Canadian experience
  • Published articles and research on issues related to program's objectives
  • Post-program participants survey
  • PCH officials
  • Representatives of Delivery Organizations
  • Organizers of exchanges
  • Literature review
  • Document and file review
  • Key Informant interviews with PCH officials, Delivery Organizations, organizers of exchanges
  1. To what extent are the program objectives and expected results aligned with federal government priorities?
  1. To what extent are the program objectives and expected results aligned the priorities and strategic outcomes of PCH?
  • Evidence and perceptions as to the extent to which program objectives and expected results are aligned with federal government priorities.
  • Evidence and perceptions as to the extent to which program objectives and expected results are aligned with the priorities and strategic outcomes of PCH.
  • Program documents such as Terms & Conditions, program guidelines, contribution agreement, final reports and official government documents such as Speeches from the Throne and federal budgets.
  • PCH officials
  • Document and file review
  • Key informant interviews with PCH officials
  1. To what extent is ECP aligned with departmental and federal roles and responsibilities?
  • Evidence and perceptions as to the extent to which ECP is aligned with departmental and federal roles and responsibilities.
  • Published articles and research on the federal role in programming aimed at young Canadians
  • Program documents such as Terms & Conditions, program guidelines, contribution agreements, final reports and official government documents such as, Speeches from the Throne and federal budgets.
  • PCH officials
  • Literature review
  • Document and file review
  • Key informant interviews with PCH officials

Immediate outcome:

  1. To what extent has the program provided young Canadians with an access to a wide variety of exchange experiences, including exchanges related to official languages?
  • Number of funded Delivery Organizations by province/territory
  • Number and type of funded exchanges by province/territory
  • Percentage of exchanges with an OL component
  • Perceptions of key informants on the extent to which the program provided young Canadians with a wide variety of exchange experiences.
  • Program final activity reports
  • Youth participants, parents and organizers questionnaires
  • PCH officials
  • Representatives of delivery organizations
  • Organizers of exchange
  • Document and file review
  • Databases review
  • Key informant interviews with PCH official, delivery organizations and exchanges organizers

Intermediate outcomes:

  1. To what extent has the program contributed to the participation of young people from across Canada in youth exchanges?
  • Number of participants by province/territory
  • Perceptions of key informants on the program contribution to the participation of young people from across Canada
  • Delivery organization program reports, demographics table provided by Delivery Organizations, participants highlights decks prepared by the program
  • Post- program questionnaires (youth, parents and organizers).
  • PCH officials, representatives of delivery organizations, organizers of exchanges
  • Document and file review
  • Databases review
  • Post-program questionnaires (youth, parents, organizers).
  • Key informant interviews with PCH official, delivery organizations and exchanges organizers
  1. To what extent do exchange participants reflect the diversity of the Canadian youth population?
  • Demographic profile of participants (e.g., age, sex, Aboriginal youth, youth from rural regions, visible minority youth, youth from low-income families Footnote 39, youth with disabilities, etc.)
  • Perceptions of the extent to which exchange participants reflect the diversity of the Canadian youth population
  • Post-program questionnaires (youth, parent, organizers)
  • Published articles and research on Canada demographics
  • PCH officials, representatives of Delivery Organizations, organizers of exchanges
  • Delivery organization program reports
  • Document and file review
  • Databases review
  • Post-program questionnaires (youth, parent, organizers)
  • Literature review
  • Key informant interviews with PCH official, Delivery Organizations and exchanges organizers

Ultimate outcomes:

  1. To what extent has the program contributed to the enhancement of young participants' knowledge and understanding of Canada?

Percentage of participants who report enhanced knowledge and understanding of Canada

Percentage of parents who report that their child enhanced his/her knowledge and understanding of Canada

Percentage of organizers who report that their participants enhanced their knowledge and understanding of Canada

Percentage of participants who report having enhanced their knowledge of their other official language during their exchange

Percentage of parents who report that their child enhanced his/her knowledge of their other official language during the exchange

Percentage of organizers who report that their participants enhanced knowledge of their other official language during the exchange

  • Delivery organization program reports
  • Post-program questionnaires (youth, parents, organizers)
  • PCH officials, representatives of Delivery Organizations, organizers of exchanges
  • Focus group
  • Document and review
  • Databases review
  • Post-program questionnaires (youth, parents, organizers). s
  • Key informant interviews with PCH official, Delivery Organizations and exchanges organizers
  1. To what extent have exchange participants created connections and linkages with one another?

Percentage of participants who report having created new ties with other young Canadians as a result of the exchange

Percentage of parents who report that their child created new ties with other young Canadians as a result of the exchange

Percentage of organizers who report that their participants created new ties with other young Canadians as a result of the exchange

  • Post- program questionnaires (youth, parents, organizers) questionnaires
  • PCH officials, representatives of Delivery Organizations, organizers of exchanges
  • Delivery organization program reports
  • Focus group
  • Document and file review
  • Databases review
  • Post-program questionnaires (youth, parents organizers)
  • Key informant interviews with PCH official, Delivery Organizations and exchanges organizers
  1. To what extent have exchange participants enhanced their appreciation of the diversity and shared aspects of the Canadian experience?

Percentage of participants who report enhanced appreciation for diversity and commonalities of the Canadian experience

Percentage of parents who report that their child enhanced his/her appreciation for diversity and commonalities of the Canadian experience

Percentage of organizers who report that their participants increased their appreciation for diversity and commonalities of the Canadian experience

  • Post participation questionnaires (youth, parents, organizers)
  • PCH officials, representatives of Delivery Organizations,
    organizers of exchanges
  • Delivery organization program reports
  • Focus group
  • Document and file review
  • Databases review
  • Post participation questionnaires (youth, parents, organizers)
  • Key informant interviews with PCH officials, Delivery Organizations and exchanges organizers
  1. Is an adequate performance measurement strategy in place to report on program results?
  2. What, if any, changes to performance measurement are required?
  • Effectiveness of current monitoring processes
  • Adequacy of performance measurement mechanisms and systems in place:
    • Extent to which the performance indicators accurately reflect outputs and results (accuracy)
    • Extent to which program data capture and reporting capacity (including external measurement project reporting) corresponds to expectations outlined in the performance measurement framework (accuracy)
    • Extent to which the performance data being collected is accurate and complete (quality)
  • Extent to which the performance data supports decision-making and departmental accountability requirements (usefulness)
  • Identification of potential changes and improvements to performance measurement
  • Unexpected outcomes (benefits, negative aspects) of program delivery
  • Program documents
  • Databases on participation, demographics of participants, etc.
  • PCH officials
  • Document and file review
  • Databases review
  • Key informant interviews with PCH officials
  1. Are the resources dedicated to ECP being used economically and efficiently to maximize achievement of outcomes?
  • Planned vs. utilized financial and human resources
  • Program operational costs in relation to overall budget
  • Program costs per participant
  • Program delivery costs compared to other similar programs
  • Program administrative data
  • Delivery organization program reports
  • Document and file review
  • Key informant interviews with PCH officials
  1. Are there alternative approaches to achieving similar results?
  • Evidence of more cost-effective alternative approaches to meet program's objectives (programs, delivery mechanisms at PCH or at other federal departments or at the municipal, provincial/territorial, national or international levels)
  • Program administrative data, including financial data
  • Scan of published literature on alternative approaches
  • Document and file review
  • Key informant interviews with PCH officials
  • Literature review
  1. Does ECP duplicate or complement existing programs/initiatives?
  • Extent to which program and/or program components duplicate or complement one another or other programs delivered through other organizations or levels of government
  • Scan of youth programs at the federal/territorial/ /provincial and international levels
  • PCH officials
  • Document and file review
  • Key informant interviews with PCH officials
  • Literature review

Appendix C: PCH programs and initiatives that benefit youth

Target Groups

Target Age Groups

Delivery mechanism (2014-15)

Main outcomes

Average Reach/

Participants per year

Support for official languages

Annual Budget (2014-15)

Strategic Outcome (2014-15) Footnote 40

PAA

Youth Take Charge: Strengthens youth attachment to Canada and supports youth-led projects that help young people to become active and engaged citizens.

Youth

7-30

Contribution agreements with delivery organizations

Civic engagement and attachment to Canada

300,000

Funding applicants are asked about measures to include youth from official language minority communities

$1.5M

#2

2.1.6

Exchanges Canada Program: Provides youth with opportunities to learn about Canada, connect with one another and appreciate the diversity and shared aspects of the Canadian experience.

Youth

12-17

(YEC)

Contribution agreements with delivery organizations

Knowledge of Canada, connection to others, appreciation of Canadian experience

6,700

(YEC)

Activities funded by ECP offer opportunities to promote English and French in Canadian society.

$8.1M

(YEC)

#2

2.1.5

14-25

(YFC)

5,000

(YFC)

$4.6M

(YFC)

16-18

(SWSE)

850

(SWSE)

$5.1M

(SWSE)

Young Canada Works Initiative: Provides work experiences and internships in Department of Canadian Heritage priority areas (official languages and heritage institutions)

Youth

16-30

Grant and Contribution agreements with delivery organizations

Employability

2,300

Funding recipients are selected to ensure that services and funding are available to heritage employers and youth from both official language minority communities.

$11.3M

#1 and #2

N/A

Canada History Fund – Encourages Canadians to learn about Canada's history, civic life and public policy by supporting the development of learning materials, activities and experiences. Some projects supported are targeted specifically at youth, including the Government of Canada History Awards for high school students and teachers.

All Canadians

no age limit

Grant and Contribution agreements with delivery organizations

Knowledge and dissemination of Canadian history

tens of thousands of young Canadians

No

$1M Footnote 41

#2

2.1.4

Official Languages Support Programs: Offers language-learning initiatives, work experiences and internships for youth, approximately 7,200 summer language bursaries and 300 official language monitor jobs.

Youth

Grade 8 and above

Initiatives are delivered by the Council of Ministers of Education Footnote 42

Knowledge of official languages

7,500

Yes

$24M

#2

2.3

Queen Elizabeth II Silver Jubilee Endowment Fund for Study in a Second Official Language Award: Encourages young Canadians who wish to improve their proficiency in their second official language to pursue studies, on a full-time basis, at a university which functions in the other official language and in a milieu in which that language predominates.

Youth

University students (Bachelor's degree)

Awards ($7K)

Knowledge of official languages

Up to 3

Yes

$21K

#2

2.3

Canada Arts Training Fund: Contributes to the development of Canadian creators and future cultural leaders by supporting the training of artists with high potential through organizations that offer training of the highest calibre.

Artists with high potential

No age limit

Operational funding for training organizations

Canadian institutions offer arts training of the highest calibre

4,000

(3,500 youth)

No

$20M Footnote 43

#1

1.1.4


Appendix D: Bibliography

Bartlett, Jamie, Jonathan Birdwell, and Louis Reynolds. Like, Share, Vote. London: Demos, 2014.

Brown, Randall for Deloitte & Touche LLP and Learning for a Sustainable Future. Ready or not? Preparing youth for 21st century responsible citizenship. Toronto: Learning for a Sustainable Future, 2012.

Canada. Department of Canadian Heritage. Canadian Heritage 2012-13 Report on Plans and Priorities. Gatineau: Department of Canadian Heritage, 2012.

Canada. Department of Canadian Heritage. Canadian Heritage 2013-14 Departmental Performance Report. Gatineau: Department of Canadian Heritage, 2014.

Canada. Department of Canadian Heritage. Service Standards for Canadian Heritage Funding Programs.Retrieved from http://canada.pch.gc.ca/eng/1435587065455/1435587168990.

Canada. Department of Canadian Heritage. Summative Evaluation of the Exchanges Canada Program. Gatineau: Department of Canadian Heritage, March 2010.

Canada. Department of Finance. The Next Phase of Canada's Economic Action: A Low Tax Plan for Jobs and Growth. Ottawa: Public Works and Government Services Canada, June 6, 2011.

Canada. Department of Finance. Jobs, Growth and Prosperity: Economic Action Plan 2012. Ottawa: Public Works and Government Services Canada, March 29, 2012.

Canada. Speech from the Throne to Open the First Session of the 41st Parliament of Canada. Ottawa: June 3, 2011.

Canada. Speech from the Throne to Open the 2nd session of the 41st Canadian Parliament. Ottawa: October 16, 2013.

Canada, Statistics Canada. "Individual Internet use and E-commerce." The Daily. Ottawa: Statistics Canada, 12 October 2011.

Canada. Statistics Canada. "Sense of Belonging to Canada, the province of residence and the local community, 2013." The Daily. Ottawa: Statistics Canada, 2015-06-29.

Canadian News Wire. "Minister Moore Announces New Programs to Support Canada's History." Retrieved from http://www.newswire.ca/en/story/1181737/minister-moore-announces-new-programs-to-support-canada-s-history, June 11, 2013.

Cowan, Mairi, and Christopher Landon. "The Missing Links in History Education." Canadian Journal for Social Research 4, 1 (Spring 2011), pp. 20-30.

Franke, Sandra. Current Realities and Emerging Issues Facing Youth in Canada: An Analytical Framework for Public Policy Research, Development and Evaluation. Ottawa: Government of Canada Policy Research Initiative, 2010.

Gerhards, Jurgen, and Silke Hans. "Transnational Human Capital, Education, and Social Inequality. Analyses of International Student Exchange." Zeitschrift fü Soziologie 42, 2 (2013), pp. 99-117.

Hewitt, Judith Marie. Engaging First Nations Youth through Reciprocal Intercommunity Exchange. Masters of Education Thesis. Queens University, 2011.

Hooghe, Marc, and Dietlind Stolle. "Age Matters: Life cycle and Cohort differences in the Socialisation Effect of Voluntary Participation." European Political Science 3, 2 (2003), pp. 49-56.

Ipsos-Reid for the Vimy Foundation. One in Five (18%) Canadians Don't Know What Vimy Ridge Is. Montreal: Vimy Foundation, 2014.

Jedwab, Jack. Attaching Canadians: What Things Do Canadians Believe Most Reinforce Attachment to their Country? Montreal: Association for Canadian Studies, 2011.

Jedwab, Jack. What Keeps Canada Together? Montreal: Association for Canadian Studies, 2014.

Larzén-Östermark, Eva. "Intercultural Sojourns as Educational Experiences: A Narrative Study of Finnish Student Teachers' Language-Practice Periods in Britain." Scandinavian Journal of Educational Research 55, 5 (2011), pp. 455-473.

Lawford, Heather L. et al. "Predictors of adolescent successful development after an exchange: The importance of activity qualities and youth input." Journal of Adolescence 35, 5 (2012), pp. 1381-1391.

Menard, Marion. « La participation electorale des jeunes au Canada 2 – Determinants et inventions.» Ottawa: Library of Parliament, 2010.

Northrup, David. "Canadians and Their Pasts." Canadian Journal for Social Research 4, 1 (2011), pp. 6-19.

O'Neill, Brenda. "Indifferent or Just Different? The Political and Civic Engagement of Young People in Canada." Ottawa: Canadian Policy Research Networks, 2007.

Pancer, S. Mark, Linda Rose-Krasnor, and Lisa D. Loiselle. "Youth conferences as a context for engagement." New Directions for Youth Development 96 (2002), pp.47-64.

Pearson, Sarah S. and Heather M. Voke. Building an effective citizenry: Lessons Learned from Initiatives in Youth Engagement. Washington: American Youth Policy Forum, 2003.

Soroka, Stuart N., Richard Johnston, and Keith Banting. "Ties that Bind? Social Cohesion and Diversity in Canada." In Belonging? Diversity, Recognition and Shared Citizenship in Canada. Keith G. Banting, Thomas J. Courchene and F. Leslie Seidle. Montreal: Institute for Research on Public Policy, 2006.

Page details

Date modified: