Evaluation of the Treasury Board of Canada Secretariat's Centralized Language Training Program 

Executive summary

This evaluation of the Centralized Language Training (CLT) Program was conducted between and by the Treasury Board of Canada Secretariat (TBS) to assess the relevance and performance of the program from the 2011 to 2012 fiscal year to the 2015 to 2016 fiscal year.

Conclusions

The evaluation found that there is a continued need and desire for language training at the Secretariat in order to support access to employment opportunities.

  • The program is aligned with federal roles, responsibilities and priorities.
  • Employees’ and managers’ objectives differ from the expected outcomes of the CLT Program.
  • The vast majority of Secretariat employees meet the language profile of their position. Despite this, Public Service Employee Survey (PSES) results revealed that employees do not always feel that they can work in their official language of choice.

The evaluation found that the CLT Program is largely meeting its expected outcomes:

  • With very few exceptions, employees appointed with non-imperative language status attain their language levels.
  • The CLT Program is accessible to most employees, with a caveat that access is variable across the Secretariat. As managers balance pressure to complete work while supporting employees to acquire or maintain required language levels, optional training may not be accessible across the Secretariat.

The centralization of CLT Program delivery was an efficient, effective and responsive way to deliver language training in the Secretariat:

  • Adopting a group training approach was as effective as individual training. The majority of Secretariat employees who took their Public Service Commission of Canada (PSC) second language evaluations (SLEs) after training either maintained or improved their results.
  • Considering both direct and opportunity costs, group classes cost significantly less than individual training and in most circumstances provide equivalent benefit.
  • The procurement vehicle had both strengths and limitations. Key strengths were having training offsite as well as the responsiveness and flexibility afforded by the use of Task Authorizations. However, the assessment criteria in the contract did not give sufficient weighting to ensure quality instruction and a quality learning environment.

The program is making progress toward attaining its long-term outcome; however, there are barriers to full realization.

Recommendations

It is recommended that:

  1. The Secretariat continue offering the CLT Program through a centralized model
  2. The CLT Program place increased emphasis on language maintenance, including the responsibility of employees in this matter, and review ways to support ongoing use of the employee’s second language
  3. The Secretariat’s Human Resources Division (HRD) work with the Procurement and Contracting Unit of the Corporate Services Sector to ensure a balance between quality and cost rating criteria used to assess the CLT Request for Proposals (RFP) to ensure higher-quality instruction and to optimize learning
  4. A departmental requirement be established that would define the conditions in which each sector would offer full- and part-time training to ensure policy compliance and access to training
  5. A departmental mechanism (for example, a training contract) be developed for managers and learners for both full- and part-time training that would:
    • Respect the recommended length of training time, to the degree possible
    • Minimize unnecessary test-taking
    • Commit employers to giving employees the time to attend their training course
    • Commit to support employees to practise skills in the workplace

Introduction

The evaluation of the Centralized Language Training (CLT) Program is part of the Treasury Board of Canada Secretariat’s (TBS’s) approved five-year evaluation plan. It was carried out by the Internal Audit and Evaluation Bureau with the assistance of Goss Gilroy Inc. between and . The evaluation assessed the relevance and performance of the program from the 2011 to 2012 fiscal year to the 2015 to 2016 fiscal year.

Context and program background

Context

Language training in Canada is a substantial and highly competitive business. The Government of Canada (GC) spends approximately $52.5 million a year on English and French language training. Footnote 1

The fields of English as second language (ESL) and French as second language (FSL) are considerably different from one another. The ESL industry is supported by a variety of credentials, curricula and assessment tools, whereas the FSL industry is smaller and lacks ESL’s credentialing and volume of assessment tools. However, it is the latter that is of particular interest to the Secretariat for two reasons:

  • The National Capital Region’s (NCR’s) designation as a federal bilingual region
  • FSL training commands 90% of the NCR’s language service providers’ business

The 2012 GC Policy on Official Languages is a driver of the demand for language training. It stipulates that “deputy heads must ensure that, in bilingual regions, all measures are taken to enable the institution’s employees to work…as well as [receive] training and professional development in the official language of their choice, pursuant to the Directive on Official Languages for People Management.” The directive goes on to say that “deputy heads…are responsible for offering language training to employees who wish to develop their second-language skills in order to advance in their career and possibly hold bilingual positions in the future.”

A second driver is the nature of career progression and opportunity in the GC. The GC strives to be an employer of choice, and one of the elements that contributes to this is the diversity of career opportunities across departments and agencies. However, maximizing career progression (for example, advancement to the executive cadre) and career opportunities (for example, building and transferring skills through “at-level” opportunities) requires competency in both official languages in the NCR.

Since the 2013 to 2014 fiscal year, there has been a 0.4% increase in GC positions that require Level C proficiency in oral interaction. PSC informants noted that with each increase in proficiency requirements, there is an increase in employees who fail the SLE. The potential impact on employee movement within the public service is significant. If this trend continues, there may be an increased need for training.

The Secretariat in particular has high expectations for employee bilingualism relative to the core public administration. In 2016, 86.7% of positions within the Secretariat were designated bilingual as compared with 67.7% of NCR positions within the core public administration. The number of Secretariat employees who meet the language requirements of their positions increased from 92% in the 2009 to 2010 fiscal year to 98% in the 2012 to 2013 fiscal year. Footnote 2 Currently, 97% of Secretariat employees meet the language profile of their position, including 96% of executives. Footnote 3

The CLT Program

The Secretariat’s CLT Program was established in April 2011. Previously, second language training was delivered in a decentralized manner across the Secretariat. The decision to develop the program, that is, centralize second language training, came in 2010 following an in-depth analysis of second language training costs and training delivery methods (group or individual). The new program was to have two key features:

  • Group training rather than individual training would be standard
  • All language training would be managed and contracted by the Secretariat’s HRD

Appendix A shows the CLT Program delivery process map.

Expected outcomes

The expected outcomes of the CLT Program, as outlined in its logic model (see Appendix B), are as follows:

Immediate outcomes

  • Employees receive training as outlined in the Public Service Official Language Exclusion Approval OrderFootnote 4
  • Employees have timely access to second language training and various linguistic services
  • Effective delivery of second language training

Intermediate outcomes

  • Employees meet the linguistic profile of their positions
  • Employees improve and maintain their second language abilities
  • Cost efficiency is attained and maintained

Long-term outcomes

  • Enable colleagues to work in the language of their choice

Evaluation context and methodology

The evaluation approach is consistent with the 2016 Treasury Board Policy on Results. The evaluation looked at the five core issues stated in Box A (below) and calibrated the scope to focus more heavily on the assessment of immediate and intermediate outcomes, given that this is the program’s first evaluation. The evaluation does not compare the CLT to other federal government language training programs; the point of comparison is internal to the current TBS program.

The assessment of relevance determined the extent to which the program meets an ongoing need for language training at the Secretariat. The assessment of performance examined the extent to which the program outcomes were achieved. It also identified program delivery changes that would better support language training. Footnote 5 Finally, the evaluation did a costing analysis to determine whether there was a difference in efficiency between group and individual training.

The evaluation included both formative and summative approaches, with a participatory emphasis. It was formative in that it provided results on program administration that could be implemented as the evaluation was progressing. It was a summative evaluation in that it assessed program outcomes. And it was participative because the program stakeholders had a high degree of engagement, assisting in data cleaning and discussing formative questions.

Box A: core evaluation issues

Relevance
Issue 1: continued need for the program
Issue 2: alignment with government priorities
Issue 3: alignment with federal roles and responsibilities

Performance (effectiveness, efficiency and economy)
Issue 4: achievement of expected outcomes
Issue 5: demonstration of efficiency and economy

The evaluators used a quasi-experimental approach, relying on seven qualitative and quantitative lines of evidence:

  1. Administrative data
  2. Document review
  3. Key informant interviews
  4. Focus groups with service providers
  5. Employee survey
  6. Costing analysis
  7. Business process mapping and analysis

Limitations of the evaluation

There is one limitation to be considered when interpreting the evaluation findings: a census approach rather than a random sample was used to distribute the online survey. This approach may have resulted in some selection bias. However, given the multiple lines of evidence, the response rate of 25.7% and the broad range of respondents, Footnote 6 the limitation was mitigated and should not impact on the conclusions.

Relevance

Conclusion

The evaluation found that there is a continued need and desire for language training at the Secretariat in order to support access to employment opportunities.

  • The program is aligned with federal roles, responsibilities and priorities.
  • Employees’ and managers’ objectives differ from the expected outcomes of the CLT Program.
  • The administrative data review showed that the vast majority of employees meet the language profile of their position. Despite this, PSES results revealed that employees do not always feel that they can work in their official language of choice.

Continued need for the program

The 2014 PSES results indicate a disparity between Francophones and Anglophones feeling free to work in their official language of choice, specifically, their ability to write in the language of choice (20% Footnote 7). The Secretariat has a significant percentage of employees for whom French is their first official language (40.6%); even so, CLT employee survey respondents indicated that English is the predominant working language.

The CLT employee survey showed an ongoing need for training in maintenance and acquisition of second official languages. Of those respondents who have taken language training since 2011 (58% of survey respondents), 76% said it would have been difficult or very difficult to maintain their second language without the CLT Program, and 89% said that it would have been difficult or very difficult to improve.

Alignment with federal priorities, roles and responsibilities

The long-term outcome of the CLT Program is to “Enable colleagues to work in the language of their choice.” Footnote 8 This outcome is aligned with both legislation and the TBS Official Languages Action Plan for 2015 to 2018.

The Official Languages Act, section 39(1)(a)(b), stipulates that language should not be a barrier to employment in the federal government. The document review revealed that there are public servants, future and current, who do not have the requisite skill level in their second official language to ensure equitable access to bilingual imperative federal positions. For example, as noted earlier, organizations with bilingual requirements such as the Secretariat’s are largely not accessible to some Canadians as a result of language requirements. This has recruitment implications. Those who live in unilingual regions or who are newly arrived to Canada are essentially excluded from employment at the Secretariat.

Language training remains a priority for the Secretariat. The document TBS: Our People, Our Culture 2015–18 Everyone, Everywhere, Every Day focuses on the Secretariat’s role as a department in driving and modelling excellence in public sector management, including the development and implementation of the official languages plan. The TBS Official Languages Action Plan for 2015 to 2018 promotes a work environment that encourages the use of both official languages (Part V of the Official Languages Act) by ensuring that a “language training, maintenance program, and development opportunities are provided to employees and managers.”

Performance

Achievement of immediate outcomes

Conclusions

The evaluation found that the CLT Program is largely meeting its immediate outcomes.

  • With few exceptions, employees appointed with non-imperative language status start their language training within the prescribed timeline.
  • The CLT Program is accessible to most employees, with a caveat that access is variable across the Secretariat. As managers balance timelines and workloads while supporting employees to maintain required language levels, optional training is not accessible across the Secretariat.
  • The program is effective and responsive in its delivery.
Immediate outcome 1: employees receive training as outlined in the Public Service Official Language Exclusion Approval Order

The Public Service Official Language Exclusion Approval Order applies only to public servants appointed to a position with a non-imperative language requirement. Since the 2011 to 2012 fiscal year, 79 employees have been appointed to positions with non-imperative language status.Footnote 9 According to the program’s administrative data, the number of employees appointed with this flexibility has been declining, from 28 in the 2011 to 2012 fiscal year to 9 in the 2015 to 2016 fiscal year, giving way to an increase in bilingual imperative positions (see Figure 1). The majority of the appointments were for FIs Footnote 10 in the Office of the Comptroller General for the Chartered Accountant Student Training Program, Footnote 11 making this recruitment mechanism accessible to a broad pool of Canadians.

Figure 1: declining non-imperative appointments over time
Figure 1: declining non-imperative appointments over time. Text version below:
Figure 1 - Text version

The figure is a line graph. The vertical axis (y-axis) shows the number of appointments, and the horizontal axis (x axis) shows fiscal years. The vertical axis starts at 0 and ends at 30, with ticks every 10 points. The horizontal axis starts at the 2011 to 2012 fiscal year and ends at the 2015 to 2016 fiscal year.

In the 2011 to 2012 fiscal year, 28 employees were appointed to positions with non-imperative language status; in the 2012 to 2013 fiscal year, 13 were appointed; in the 2013 to 2014 fiscal year, 22 were appointed; in the 2014 to 2015 fiscal year, 7 were appointed; and in the 2015 to 2016 fiscal year, 9 were appointed.

Source: Administrative data, attendance reports

The Secretariat’s practice of sending employees with non-imperative language requirements to training within six months of signing letters of offer directly supports the CLT Program’s immediate outcome. Of note, 38% of those surveyed who took language training said the training opportunity has influenced their decision to work at the Secretariat.

Immediate outcome 2: employees have timely access to second language training and various linguistic services

Program administrative data show that language training is by far the most highly subscribed training within the Secretariat. Approximately 470 (26% to 28%) employees register annually, with the majority attending part-time in groups (see Figure 2). Only a minority take individual training; between the 2011 to 2012 fiscal year and the 2015 to 2016 fiscal year, individual training was provided 32 times.Footnote 12

Figure 2: trends in full-time ( line) and part-time (dotted line) program registration with mean (solid line)
Figure 2: trends in full-time (dashed line) and part-time (dotted line) program registration with mean (solid line). Text version below:
Figure 2 - Text version

The figure is a line graph of the individual and group registrations for language training. The vertical axis (y-axis) shows the number of learners, and the horizontal axis (x-axis) shows fiscal years. The vertical axis starts at 0 and ends at 500, with ticks every 100 points. The horizontal axis (x-axis) starts at the 2011 to 2012 fiscal year and ends at the 2015 to 2016 fiscal year.

The dashed line shows an increase in group training learners: 99 in the 2011 to 2012 fiscal year; 409 in the 2012 to 2013 fiscal year; 326 in the 2013 to 2014 fiscal year; 393 in the 2014 to 2015 fiscal year; and 470 in the 2015 to 2016 fiscal year.

The dotted line shows the number of persons taking individual training: 88 in the 2011 to 2012 fiscal year, 49 in the 2012 to 2013 fiscal year, 73 in the 2013 to 2014 fiscal year, 77 in the 2014 to 2015 fiscal year, and 78 in the 2015 to 2016 fiscal year.

Source: Administrative data, attendance reports

The Secretariat’s Official Languages Action Plan directs the CLT Program to be available to all indeterminate and term employees. Participation is determined by the Secretariat’s sectors and managers.

The 2014 PSES revealed that since 2008, 70% to 73% of Secretariat employees feel that they obtain the language training they need. The evaluation survey revealed that the majority of Secretariat employees (91%) know that the Secretariat offers language training to its employees. They learn about the program from the internal bulletin TBS In-Brief (78%), but they also learn from their colleagues (44%) and managers (38%). However, of the 464 survey respondents, 36% were not aware of the CLT Program’s additional language training tools.

The evaluation revealed that the CLT Program is less accessible to some employees than others. A few survey respondents noted that they were not approved for language training because they either occupied a unilingual position or they met the linguistic level of their position. As one respondent expressed, “[our] directorate does not support during-hours language training if the employees meets the language requirements of the position. [If the] position is English, [there is] no support to learn the French language during working hours.”

Considering accessibility more broadly, there are two key issues: the first is access to full-time training, and the second is workload and attendance. According to key informant and focus group evidence, some employees become frustrated when they cannot access full-time training to improve their language levels. In response, they take part-time or maintenance training with the hope that they can achieve improved levels. However, the part-time and maintenance courses are not designed for acquisition, so these employees are unlikely to meet their goals and as such may become further frustrated and discouraged. Key informant and focus group evidence shows that this frustration and discouragement are linked to absenteeism.

Key informants and focus group participants both reported that managers do not always approve the recommended length of time for training. Rather, they may be approved for semi-full-time training Footnote 13 or for fewer weeks than was recommended. The risk of this approach is expressed in the following survey quote: “My objective was to achieve Bs. The amount of time was not enough to fulfill this goal. Additionally, work often got in the way of training. In the end, too much time was missed.”

Immediate outcome 3: effective delivery of second language training

Evidence shows that the CLT Program is responsive to employee needs.

  • The CLT Program proactively pilots new tools and gathers learner input at the end of every session in order to improve service delivery.
  • The CLT Program uses feedback to evolve and to meet the needs of clients when possible, for example, offering courses in the evening.
  • Opting out of the Public Services and Procurement Canada linguistic services National Master Standing Offer has allowed for a more flexible array of options for Secretariat employees, for example, smaller classes.

The administrative data supported the evidence from key informants: in most cases, group classes are as effective as individual training. Program data indicate that, on average, group classes produce slightly faster results (296 hours per group versus 310 hours per individual) to attain required language levels in all three competencies. When comparing language test results of group and individual training, success rates were similar for both, with an 85% rate of success for reading, writing and oral.

The key informants noted the conditions under which group classes would be as effective as (or even more effective than) individual lessons. Better results are achieved when:

  • group classes are not too big (for example, 4 to 6 when working toward a Level C)
  • teachers are skilled
  • the learners are all at the same level and working toward the same goal

While individual courses can meet specific needs, such as very tight deadlines for oral exams or tailored instruction for those with learning disabilities, the focus groups confirmed that a group setting can be conducive to efficient learning when there are positive group dynamics.

Barriers to achieving immediate outcomes

There are three key factors that result in reduced outcomes for adult second language learners:

  • Underestimation by learners of the time needed to attain required proficiency levels
  • A tendency to test too soon and too frequently
  • Management of many competing time pressures

These factors are exacerbated by the stress of testing in a high-stakes environment (one in which significant decisions, such as career progression, are made), which key informants indicated is a significant issue.

With the focus on passing the PSC tests as quickly as possible, key informants noted that managers will often approve less time for training and encourage frequent testing so that employees return as soon as possible. This may increase the learner’s test-taking skills without a commensurate increase in language skill.

Data show that instead of expediting learning, this approach often resulted in extensions, repeated testing and increased stress. Key informants agreed that adults most often underestimate the time required to become proficient in a second language. This is particularly relevant for those studying for their Level C.

Finally, the language assessment expert and focus group participants noted that frequent testing can be discouraging to the learner. Related to this, the PSC’s SLEs are broad measurement tools of language ability that do not reflect the gradual progress a learner may achieve. For example, a learner may make significant progress in their written expression skills and advance from 32/55 to 42/55 on the test but will still have a Level B.

According to literature and key informants, the stakes and stress can be mitigated by multiple forms of assessment, that is, skill and ability are evaluated in various ways as opposed to in a single performance instance. Because the GC assesses language performance in a single instance, it is not surprising that learners seek to manage and mitigate the high stakes tests by learning “how” to take the test. This results in multiple requests by learners for details on the test content and strategies for how to take the test. As a result, a training objective to improve second official language skills has been superseded by an objective to succeed on test performance. Focus groups held with service providers revealed that students commonly request that instructors skip the content of their learning curriculum and just teach what will be on the PSC assessment.

However, not all students invite this approach. A Secretariat survey respondent wrote, “It would be great if there was a way to make the training seem like you are learning a second language as opposed to being prepared to pass a test. The stress can be overwhelming.”

Achievement of intermediate outcomes

Conclusion

The evidence indicates that the CLT Program is meeting its intermediate outcomes.

  • Employees are attaining, improving and maintaining their language levels.
  • The evaluation found that centralizing language training delivery was an efficient and effective way to deliver language training at the Secretariat.
  • Adopting a group training approach was as effective as individual training. The majority of Secretariat employees who took their PSC SLEs after training either maintained or improved their results.
  • Considering both direct and opportunity costs, group classes cost less than individual training and in most circumstances provide equivalent benefit.
  • The procurement vehicle had both strengths and limitations. Key strengths were having training offsite and the use of Task Authorizations. The key weakness was the administration of the CLT contract as an unskilled service contract rather than an educational contract. As such, the assessment criteria in the contract did not give sufficient weighting to ensure quality instruction and a quality learning environment.
Intermediate outcome 1: employees meet the linguistic profile of their positions

Program administrative data show that Secretariat employees appointed non-imperatively receive language training as required by the Public Service Official Languages Exclusion Approval Order, and 89% attain their language levels within the two-year time frame. For example, of the 79 non-imperative appointments since the 2011 to 2012 fiscal year, only two did not meet the required language levels within the two years allocated and were granted extensions by the Secretary of the Treasury Board. Five employees were deployed into positions with a lower language level.

Intermediate outcome 2: employees improve and maintain their second language abilities
Second language improvement

Of survey respondents who took language training, 62% said that it met their learning objectives. For those survey respondents who took the PSC SLEs after their training:

  • 48% said that the CLT Program adequately prepared them for the assessments
  • 26% said that the CLT Program did not adequately prepare them for the PSC SLEs
  • 45% said that their language results improved
  • 52% said that their language results stayed the same
Second language maintenance

Second language maintenance is heavily reliant on language use. The employee survey, focus groups and the key informants quite clearly demonstrated that employees see the employer as having a primary responsibility for enabling them to acquire and maintain their language levels. That some learners assume that training will always be provided is evident in the remarks shared with instructors as they complete their assessments, such as “See you in five years.”

However, a shared responsibility is implied by the 2010 Departmental Policy on Official Languages, which states that employees “use the official language of their choice and are encouraged to maintain acquired second language skills by actively using them in their workplace.” For managers, it states they “are responsible for promoting and implementing the [policy] by ensuring the use of both English and French in daily operations.... They must also ensure English- and French-speaking employees have equal access to employment and advancement opportunities.”

Further to this, the 2013 report called Challenges: The New Environment for Language Training in the Federal Public Service by the Office of the Commissioner of Official Languages states that “it is reasonable that an employee’s prior commitment to his or her language retention be considered as a criterion in making funding available. In other words, employers must encourage employee language retention through various means, but employees must also demonstrate their own efforts in this regard.”

The CLT Program offers multiple options for second language maintenance. Administrative data show that attendance rates vary vastly by the type of class, with attendance in maintenance classes being poorest (58%) and one-week workshops being highest (94%) (see Figure 3). The survey respondents noted that the factors contributing to absenteeism (aside from illness) are workload, which is the most common reason provided for learner absences, and unreasonable language acquisition expectations (discussed earlier).

Figure 3: attendance by training type
Figure 3: attendance by training type. Text version below:
Figure 3 - Text version

The figure is a horizontal bar graph. Categories are on the vertical axis (y-axis), and values on the horizontal axis (x-axis). The vertical axis shows four types of attendance plus the total for overall attendance, and the horizontal axis shows the rates of absenteeism.

The total overall attendance rate was 79%, the maintenance class attendance rate was 58%, the group class attendance rate was 82%, the individual attendance rate was 91%, and the workshop attendance rate was 94%.

Note: for individual attendance rates, absences are sometimes hidden by rescheduling.

Source: Administrative data, attendance reports

*For individual attendance, absences are sometimes hidden by rescheduling.

Intermediate outcome 3: efficient delivery of second language training

The centralized administration of the program was universally praised by senior management, learners and the service provider. The single-window access point for language training was felt to be quick, efficient, transparent and responsive for a more effective delivery of the program.

In an effort to be more responsive to employees, HRD implemented continuous enrolment in 2013 and removed the two-person minimum for groups. While this change to program delivery offered more flexibility to Secretariat employees, it disrupted the classrooms and often resulted in classes of one to two students.Footnote 14 This approach was discontinued in 2016.

As part of its contract under the program, the service provider was required to deliver the majority of the training off-site. As the data show, this was a program strength, given that adults learn better when they attend language training away from the workplace. There are two distinct reasons for this:

  • Learning a second language in the workplace, which may result in the appearance of lower competence, can stifle the learner’s willingness to speak and make mistakes.
  • Leaving the workplace can also result in better attention to the class and less disruption by work matters.

Another effective feature of the contract was that it provided the Secretariat the flexibility to establish Task Authorizations (mini-contracts) on an ongoing basis, as well as to qualify teachers as required. The service provider would try (usually with success) to switch teachers and classes if requested by the learner, which helped them respond to learners’ needs.

The contract did have some limitations. There is evidence that over the five years of the program, the procurement vehicle was not adequately responsive to the evolving opportunities for language training options. For example, when employees asked for e-learning options, the contract restrictions precluded these offerings being available to the Secretariat.

One important finding related to procurement was that focus group participants indicated that they no longer bid on federal contracts, as they did not allow schools to adequately remunerate their teachers and thereby retain them. As noted earlier, the GC demand for FSL training has created a localized teacher shortage in the NCR. This gap, in combination with a propensity to award language contracts to lowest-cost bidders, can result in service providers who cannot hire and retain those instructors with a high level of teaching experience or familiarity with government vocabulary. This suggests that there may be some issues in balancing quality and price in the RFP rating criteria. This is worth noting, given the issue of teacher turnover and the importance of quality teachers for quality learning.

Group versus individual training

Efficiency was assessed by comparing the costs of individual versus group language training. The evaluation compared the costs of achieving language levels for those in individual training versus those in group training (that is, successfully passing evaluations in comprehension, grammar and oral skill needed to meet job requirements). Only the EX category had sufficient numbers in both types of training to be compared.Footnote 15  A group of seventeen EXs was compared to a group of 76 EXs; the advantage of this comparison was that the candidates were well matched from a profile perspective.

According to the administrative data, direct training costs (contracted service delivery) for language training totalled $4.2 million for the period evaluated. Opportunity costs (staff salaries paid while on training) were much higher ($11.1 million in total, excluding benefits).

The performance data showed that the average training cost for those studying in a group and who passed all three tests (as required for their position) was nearly half the cost per person than that of one-on-one training (see Figure 4).

Figure 4: comparison of average cost per person of one-on-one training and group training
Figure 4: comparison of average cost per person of one-on-one training and group training. Text version below:
Figure 4 - Text version

This figure is a pictorial description of the costs of individual versus group training. The figure consists of two small circles. Inside the first circle is a single stick person, with $10,544 written above the circle. Inside the second circle are four stick people, with $6,075 written above the circle. The amount of $6,075 is the average training cost per person for those studying in a group and who passed all three tests (as required for their position). The amount of $10,544 is the average training cost per person for those in individual training.

Opportunity costs (that is, salary costs for learners while on training) were similar in both groups in achieving their levels. In terms of training hours, learning for both groups and individuals took a similar amount of time to achieve the levels sought, 296 hours for individuals in a group and 310 hours for individuals.

Achievement of long-term outcome

Conclusion

The program is making progress toward attaining its long-term outcome; however, there are barriers to full realization.

Long-term outcome: enable colleagues to work in the language of their choice

Given the level of bilingualism at the Secretariat, at almost 87%, it might be assumed that employees would feel able to work in their language of choice. However, the relationship between increasing rates of bilingualism and an increase in employees’ ability to work in the official language of their choice is not clear.

The Secretariat’s 2014 Survey on Language of Work found a large discrepancy between the perceptions of Anglophone and Francophone employees regarding the use of both official languages. The survey indicated that although 76% of Anglophones felt that Secretariat meetings were conducted in both official languages and felt comfortable using their preferred official language, only 41% of Francophones felt the same way.

Of CLT survey respondents, 67% of those who participated in the CLT Program reported that after training they are more comfortable working in their second language, and 64% said that they had the opportunity to use their second language at work after their training. However, only 42% said that the training increased the actual frequency of their use of their second language.

This is in line with a reality raised in the 2013 Office of the Commissioner of Official Language report: Challenges: The New Environment for Language Training in the Federal Public Service: “language training enables employees to prepare for a test, but that the training is not enough to make them comfortable using the acquired skills in their work.”

The interview with the language assessment expert and focus group participants raised several points for consideration, which may be barriers to a full realization of the long-term outcome. Three points are as follows:

  1. Employees aspire to being, and also appearing, competent in the workplace; working in one’s second language may make communicating genuinely more difficult and create an appearance of lower competence.
  2. With added time pressure, communicating in one’s second language often takes longer, and the workplace environment may not support taking additional time. Again, requiring more time may call into question the competence of the individual.
  3. The skills and vocabulary required in some professions or in some situations require a high degree of precision and accuracy; this degree of second language proficiency can take years to attain.

These views were reinforced through comments in the Secretariat’s 2014 Survey on Language of Work in which 72% of respondents said they switch to the first official language of a colleague when they sense that he or she is struggling in his or her second official language. CLT survey respondents indicated similar sentiments. As one summarized: “Most have no patience to allow Anglophone to practice or use their French second language and would prefer to change into English in the [interest] of time/efficiency.”

The evidence revealed a disparity between the official languages policy objectives, public servants working in their language of choice, and the goals of survey respondents. The survey results show that 45% of employees take training primarily to progress in their careers. The second and third most commonly cited reasons were to improve second language and maintenance. This indicates that the achievement of desired language levels does not necessarily lead to employees being enabled to work in their language of choice.

Research completed by the Office of the Commissioner of Official Languages of Canada suggests that as public servants' second official language competence and comfort levels increase, they will increasingly be able to use their language of choice.Footnote 16 In other words, an employee is more likely to use their official language of choice with colleagues if they are confident it won't impede mutual understanding. However, the findings of the evaluation do not necessarily support this.

In conclusion, there is no strong evidence to show that at the Secretariat an individual's improved second language proficiency positively impacts others using their language of choice.

Recommendations

It is recommended that:

  1. The Secretariat continue to offer the CLT Program through a centralized model
  2. The CLT Program place increased emphasis on language maintenance, including the responsibility of employees in this matter, and review ways to support ongoing use of the employee’s second language
  3. The Secretariat’s Human Resources Division work with the procurement unit of the Corporate Services Sector to ensure a balance between quality and cost rating criteria used to assess the CLT Request for Proposals to ensure higher-quality instruction and to optimize learning
  4. A departmental requirement be established that would define the conditions in which each sector would offer full- and part-time training to ensure policy compliance and training access
  5. A departmental mechanism (for example, a training contract) be developed for managers and learners for both full- and part-time training that would:
    • Respect the recommended length of training time, to the degree possible
    • Minimize unnecessary test-taking
    • Commit employers to giving employees the time to attend their training course
    • Commit to support employees to practise skills in the workplace

Appendix A: The Business Process Map of the Centralized Language Training Program

  1. Advertisements are sent to Strategic Communications and Ministerial Affairs for distribution to employees via TBS In-brief with a link to the registration form
    • Emails sent monthly
    • Email approved by Director only if there are program changes being announced
  2. Employees complete paper registration forms including supervisor approval
    • Each sector may implement its own additional levels of approval
  3. Registration forms are sent to Human Resources Division (HRD).
    • Sectors have individual protocols for sending registrations to HRD, e.g. individually by employee in a bundle for the sector, scanned and emailed or hard copies.
  4. HRD receives the forms and manually enters them into a spread sheet
    • information is verified in PeopleSoft
    • individual training history is checked to determine if new assessment is required
  5. HRD completes multiple Task Authorizations (TA).  This is a task that is ongoing through the year.
    • TAs are for individual, part-time and full time classes, maintenance sessions and workshops
    • Each TA includes dates of training, the course, the hours of training and cost, name of each student and the learning level
    • TA amendments are done when new students are added to pre-existing classes if training dates and cost are changed.
    • If the TA is over $100, 000. 00 PWGSC approval  is required as a first step
  6. The TAs are reviewed by the TBS Procurement group and sent to Centre de langues internationales Charpentier (CLIC)
  7. CLIC reviews, adds the name of the teachers if for full-time group, signs and sends back to TBS Procurement
  8. HRD receives TA and signs
  9. For six weeks prior to the fall and winter part-times classes and maintenance  sessions starting, HRD sends assessment sheet to CLIC
    • CLIC books phone or in-person assessments with employees
    • On occasion these assessments are booked by HRD
  10. CLIC sends assessment results back to HRD
  11. HRD forms classes
    • class lists are developed and Outlook invitations are sent to each participant for the session
    • Participants who at this point request accommodation to their schedule have their request reviewed
  12. First class visit by HRD happens after week 4 for part-time and maintenance.  
    • Class visits are bi-weekly for full-time classes.
  13. Attendance is taken at each class
    • Attendance is returned to TBS by CLIC monthly for full-time and workshops and distributed to managers
    • Attendance is returned to TBS by CLIC monthly and distributed to managers by session for all part-time classes and maintenance sessions 
  14. Administrative procedures for Invoicing (and TAs.)
    • confirm accuracy of invoice and enters information on TA report and attendance report and budget report; track through HRD
    • invoice sent to the Executive Director’s (ED) office
    • the ED sends the invoice to Director for approval
    • Invoice payment approved by director
    • Approved invoiced returned to ED
    • approved invoice returned to CLT unit
    • CLT forwards approved invoice to Finance for payment
  15. At the end of each session the CLT unit sends a feedback form to each participant.
    • HRD documents major issues and bring them to the attention of CLIC

Appendix B: Centralized Language Training Program logic model

Centralized Language Training Program logic model
Appendix B: Centralized Language Training Program logic model. Text version below:
Text version

The main components of the Centralized Language Training Program are:

  • inputs
  • activities
  • outputs
  • immediate outcomes
  • intermediate outcomes
  • long-term outcome

The program inputs are:

  • resources
  • time
  • money
  • partners
  • research tools
  • curriculum

The program activities are:

  • provide official languages assessments and second language training to TBS employees appointed non-imperatively
  • connect employees with second language training for career development and maintenance
  • linguistic Services Unit develops and delivers tools
  • program administration and monitoring

The program outputs are:

  • employee training recommendations
  • TBS In-Brief advertisements
  • tools: Phonetics course, Qs&As, Pairing Program
  • task authorizations, paid invoices, budget reports, monitoring reports, School Action Plan

The immediate program outcomes are:

  • employees receive required training as per the Public Service Official Languages Exclusion Approval Order
  • employees have timely access to second language training and various linguistic services
  • effective delivery of second language training

The intermediate program outcomes are:

  • employees meet the bilingual linguistic profile of the positions
  • employees improve and maintain their second language abilities
  • cost efficiency is attained and maintained

The long-term outcome of the program is to enable colleagues to work in the language of their choice

Appendix C: definitions

Public Service Official Languages Exclusion Approval Order
Intends to provide an appropriate balance between: ensuring that persons appointed meet the official language proficiency requirements of bilingual positions and fostering the access of unilingual Canadians to bilingual positions in the federal public service. For more information, see the Public Service Commission of Canada website.
full-time language training
A program that runs 35 hours per week.
non-imperative designations
A non-imperative designation that allows the position to be filled by a candidate who undertakes to pursue language training in order to meet the requirements within two years.
part-time training
A program that runs days or evenings for 3 to 6 hours per week.
second language evaluation tests preparation workshops
One-week workshops for each linguistic ability that provide employees with 35 hours to prepare for the second language evaluation tests.
second language maintenance sessions
Weekly sessions of 1 or 2 hours per week for employees who have already obtained Level B or Level C in the second language evaluation tests and who wish to maintain their acquired skills.
semi-full-time second language training
A program that runs 15 hours per week.
Task Authorizations
A contract with Task Authorizations is a method of supply for services under which all of the work or a portion of the work will be performed on an “as and when requested basis” through predetermined conditions, including an administrative process involving task authorizations see Contracts with Task Authorizations under “Contracts” in the Supply Manual, Chapter 3.

Appendix D: Management Response and Action Plan

The Treasury Board of Canada Secretariat’s Human Resources Division (HRD) has reviewed the evaluation and has provided the following comments regarding the report’s recommendations.

Recommendations Proposed Action Start Date Targeted Completion Date Office of Primary Interest

Recommendation 1

The Secretariat continue offering the Centralized Language Training (CLT) Program through a centralized model

HRD agrees with the recommendation.

HRD will continue to offer language training through a centralized model.

Ongoing Ongoing HRD

Recommendation 2

The CLT Program place increased emphasis on language maintenance, including the responsibility of employees in this matter, and review ways to support ongoing use of the employee’s second language

HRD agrees with the recommendation.

HRD will research and promote established and emerging best practices to support employees with maintenance of their second language, such as through the use of language training tent cards and through facilitated lunchtime discussions on work-related topics.

HRD will collaborate with the Secretariat’s Strategic Communications and Ministerial Affairs (SCMA) to develop communications products to raise awareness of employees’ responsibilities regarding maintenance of their second official language, as well as of management’s role in providing a work environment to enable employees to practise their second language skills in the workplace.

Ongoing HRD

Recommendation 3

HRD work with the Procurement and Contracting Unit  of the Secretariat’s Corporate Services Sector to ensure a balance between criteria for quality and cost rating used to assess the CLT Request for Proposals (RFP) to ensure higher-quality instruction and to optimize learning

HRD agrees with the recommendation.

HRD is currently developing an RFP in collaboration with TBS’s Procurement and Contracting Unit and Public Services and Procurement Canada. The rating criteria will balance the assessment of technical criteria and program cost with a consideration for the conditions that support quality instruction.

HRD

Recommendation 4

A departmental requirement be established that would define the conditions in which each sector would offer full- and part-time training to ensure policy compliance and access to training

HRD agrees with the recommendation.

HRD will engage the Departmental Advisory Committee on Official Languages to develop a tool to guide sectors in determining what criteria should be used to allocate language training equitably within sectors.

HRD will collaborate with SCMA to develop a communications strategy to raise awareness of stakeholders’ roles and responsibilities regarding language training.

 

 

HRD

Recommendation 5

A departmental mechanism (for example, a training contract) be developed for managers and learners for both full- and part-time training that would:

  • respect the recommended length of training time, to the degree possible
  • minimize unnecessary test-taking
  • commit employers to giving employees the time to attend their training course
  • Commit to support employees to practise their language skills in the workplace
HRD agrees with the recommendation. HRD will engage the Departmental Advisory Committee on Official Languages to discuss what can be developed and implemented for managers and employees to:
  • respect the recommended language training time, to the degree possible
  • be tested only when required according to the learner’s training plan
  • commit to employees’ training plans
  • provide a work environment where employees can practise their acquired skills (refer to the actions set out in recommendation 2)
HRD
Report a problem or mistake on this page
Please select all that apply:

Privacy statement

Thank you for your help!

You will not receive a reply. For enquiries, contact us.

Date modified: