Evaluation of the Treasury Board of Canada Secretariat's Centralized Language Training Program 

Conclusions

The evaluation found that there is a continued need and desire for language training at the Secretariat in order to support access to employment opportunities.

• The program is aligned with federal roles, responsibilities and priorities.
• Employees’ and managers’ objectives differ from the expected outcomes of the CLT Program.
• The vast majority of Secretariat employees meet the language profile of their position. Despite this, Public Service Employee Survey (PSES) results revealed that employees do not always feel that they can work in their official language of choice.

The evaluation found that the CLT Program is largely meeting its expected outcomes:

• With very few exceptions, employees appointed with non-imperative language status attain their language levels.
• The CLT Program is accessible to most employees, with a caveat that access is variable across the Secretariat. As managers balance pressure to complete work while supporting employees to acquire or maintain required language levels, optional training may not be accessible across the Secretariat.

The centralization of CLT Program delivery was an efficient, effective and responsive way to deliver language training in the Secretariat:

• Adopting a group training approach was as effective as individual training. The majority of Secretariat employees who took their Public Service Commission of Canada (PSC) second language evaluations (SLEs) after training either maintained or improved their results.
• Considering both direct and opportunity costs, group classes cost significantly less than individual training and in most circumstances provide equivalent benefit.
• The procurement vehicle had both strengths and limitations. Key strengths were having training offsite as well as the responsiveness and flexibility afforded by the use of Task Authorizations. However, the assessment criteria in the contract did not give sufficient weighting to ensure quality instruction and a quality learning environment.

The program is making progress toward attaining its long-term outcome; however, there are barriers to full realization.

Recommendations

It is recommended that:

1. The Secretariat continue offering the CLT Program through a centralized model
2. The CLT Program place increased emphasis on language maintenance, including the responsibility of employees in this matter, and review ways to support ongoing use of the employee’s second language
3. The Secretariat’s Human Resources Division (HRD) work with the Procurement and Contracting Unit of the Corporate Services Sector to ensure a balance between quality and cost rating criteria used to assess the CLT Request for Proposals (RFP) to ensure higher-quality instruction and to optimize learning
4. A departmental requirement be established that would define the conditions in which each sector would offer full- and part-time training to ensure policy compliance and access to training
5. A departmental mechanism (for example, a training contract) be developed for managers and learners for both full- and part-time training that would:
   • Respect the recommended length of training time, to the degree possible
   • Minimize unnecessary test-taking
   • Commit employers to giving employees the time to attend their training course
   • Commit to support employees to practise skills in the workplace

Context

Language training in Canada is a substantial and highly competitive business. The Government of Canada (GC) spends approximately $52.5 million a year on English and French language training. ^{Footnote 1}

The fields of English as second language (ESL) and French as second language (FSL) are considerably different from one another. The ESL industry is supported by a variety of credentials, curricula and assessment tools, whereas the FSL industry is smaller and lacks ESL’s credentialing and volume of assessment tools. However, it is the latter that is of particular interest to the Secretariat for two reasons:

• The National Capital Region’s (NCR’s) designation as a federal bilingual region
• FSL training commands 90% of the NCR’s language service providers’ business

The 2012 GC Policy on Official Languages is a driver of the demand for language training. It stipulates that “deputy heads must ensure that, in bilingual regions, all measures are taken to enable the institution’s employees to work…as well as [receive] training and professional development in the official language of their choice, pursuant to the Directive on Official Languages for People Management.” The directive goes on to say that “deputy heads…are responsible for offering language training to employees who wish to develop their second-language skills in order to advance in their career and possibly hold bilingual positions in the future.”

A second driver is the nature of career progression and opportunity in the GC. The GC strives to be an employer of choice, and one of the elements that contributes to this is the diversity of career opportunities across departments and agencies. However, maximizing career progression (for example, advancement to the executive cadre) and career opportunities (for example, building and transferring skills through “at-level” opportunities) requires competency in both official languages in the NCR.

Since the 2013 to 2014 fiscal year, there has been a 0.4% increase in GC positions that require Level C proficiency in oral interaction. PSC informants noted that with each increase in proficiency requirements, there is an increase in employees who fail the SLE. The potential impact on employee movement within the public service is significant. If this trend continues, there may be an increased need for training.

The Secretariat in particular has high expectations for employee bilingualism relative to the core public administration. In 2016, 86.7% of positions within the Secretariat were designated bilingual as compared with 67.7% of NCR positions within the core public administration. The number of Secretariat employees who meet the language requirements of their positions increased from 92% in the 2009 to 2010 fiscal year to 98% in the 2012 to 2013 fiscal year. ^{Footnote 2} Currently, 97% of Secretariat employees meet the language profile of their position, including 96% of executives. ^{Footnote 3}

The CLT Program

The Secretariat’s CLT Program was established in April 2011. Previously, second language training was delivered in a decentralized manner across the Secretariat. The decision to develop the program, that is, centralize second language training, came in 2010 following an in-depth analysis of second language training costs and training delivery methods (group or individual). The new program was to have two key features:

• Group training rather than individual training would be standard
• All language training would be managed and contracted by the Secretariat’s HRD

Appendix A shows the CLT Program delivery process map.

Expected outcomes

The expected outcomes of the CLT Program, as outlined in its logic model (see Appendix B), are as follows:

Conclusion

The evaluation found that there is a continued need and desire for language training at the Secretariat in order to support access to employment opportunities.

• The program is aligned with federal roles, responsibilities and priorities.
• Employees’ and managers’ objectives differ from the expected outcomes of the CLT Program.
• The administrative data review showed that the vast majority of employees meet the language profile of their position. Despite this, PSES results revealed that employees do not always feel that they can work in their official language of choice.

Continued need for the program

The 2014 PSES results indicate a disparity between Francophones and Anglophones feeling free to work in their official language of choice, specifically, their ability to write in the language of choice (20% ^{Footnote 7}). The Secretariat has a significant percentage of employees for whom French is their first official language (40.6%); even so, CLT employee survey respondents indicated that English is the predominant working language.

The CLT employee survey showed an ongoing need for training in maintenance and acquisition of second official languages. Of those respondents who have taken language training since 2011 (58% of survey respondents), 76% said it would have been difficult or very difficult to maintain their second language without the CLT Program, and 89% said that it would have been difficult or very difficult to improve.

Alignment with federal priorities, roles and responsibilities

The long-term outcome of the CLT Program is to “Enable colleagues to work in the language of their choice.” ^{Footnote 8} This outcome is aligned with both legislation and the TBS Official Languages Action Plan for 2015 to 2018.

The Official Languages Act, section 39(1)(a)(b), stipulates that language should not be a barrier to employment in the federal government. The document review revealed that there are public servants, future and current, who do not have the requisite skill level in their second official language to ensure equitable access to bilingual imperative federal positions. For example, as noted earlier, organizations with bilingual requirements such as the Secretariat’s are largely not accessible to some Canadians as a result of language requirements. This has recruitment implications. Those who live in unilingual regions or who are newly arrived to Canada are essentially excluded from employment at the Secretariat.

Language training remains a priority for the Secretariat. The document TBS: Our People, Our Culture 2015–18 Everyone, Everywhere, Every Day focuses on the Secretariat’s role as a department in driving and modelling excellence in public sector management, including the development and implementation of the official languages plan. The TBS Official Languages Action Plan for 2015 to 2018 promotes a work environment that encourages the use of both official languages (Part V of the Official Languages Act) by ensuring that a “language training, maintenance program, and development opportunities are provided to employees and managers.”

Achievement of immediate outcomes

Conclusions

The evaluation found that the CLT Program is largely meeting its immediate outcomes.

• With few exceptions, employees appointed with non-imperative language status start their language training within the prescribed timeline.
• The CLT Program is accessible to most employees, with a caveat that access is variable across the Secretariat. As managers balance timelines and workloads while supporting employees to maintain required language levels, optional training is not accessible across the Secretariat.
• The program is effective and responsive in its delivery.

Immediate outcome 1: employees receive training as outlined in the Public Service Official Language Exclusion Approval Order

The Public Service Official Language Exclusion Approval Order applies only to public servants appointed to a position with a non-imperative language requirement. Since the 2011 to 2012 fiscal year, 79 employees have been appointed to positions with non-imperative language status.^{Footnote 9} According to the program’s administrative data, the number of employees appointed with this flexibility has been declining, from 28 in the 2011 to 2012 fiscal year to 9 in the 2015 to 2016 fiscal year, giving way to an increase in bilingual imperative positions (see Figure 1). The majority of the appointments were for FIs ^{Footnote 10} in the Office of the Comptroller General for the Chartered Accountant Student Training Program, ^{Footnote 11} making this recruitment mechanism accessible to a broad pool of Canadians.

Figure 1 - Text version

The figure is a line graph. The vertical axis (y-axis) shows the number of appointments, and the horizontal axis (x axis) shows fiscal years. The vertical axis starts at 0 and ends at 30, with ticks every 10 points. The horizontal axis starts at the 2011 to 2012 fiscal year and ends at the 2015 to 2016 fiscal year.

In the 2011 to 2012 fiscal year, 28 employees were appointed to positions with non-imperative language status; in the 2012 to 2013 fiscal year, 13 were appointed; in the 2013 to 2014 fiscal year, 22 were appointed; in the 2014 to 2015 fiscal year, 7 were appointed; and in the 2015 to 2016 fiscal year, 9 were appointed.

Source: Administrative data, attendance reports

The Secretariat’s practice of sending employees with non-imperative language requirements to training within six months of signing letters of offer directly supports the CLT Program’s immediate outcome. Of note, 38% of those surveyed who took language training said the training opportunity has influenced their decision to work at the Secretariat.

Immediate outcome 2: employees have timely access to second language training and various linguistic services

Program administrative data show that language training is by far the most highly subscribed training within the Secretariat. Approximately 470 (26% to 28%) employees register annually, with the majority attending part-time in groups (see Figure 2). Only a minority take individual training; between the 2011 to 2012 fiscal year and the 2015 to 2016 fiscal year, individual training was provided 32 times.^{Footnote 12}

Figure 2 - Text version

The figure is a line graph of the individual and group registrations for language training. The vertical axis (y-axis) shows the number of learners, and the horizontal axis (x-axis) shows fiscal years. The vertical axis starts at 0 and ends at 500, with ticks every 100 points. The horizontal axis (x-axis) starts at the 2011 to 2012 fiscal year and ends at the 2015 to 2016 fiscal year.

The dashed line shows an increase in group training learners: 99 in the 2011 to 2012 fiscal year; 409 in the 2012 to 2013 fiscal year; 326 in the 2013 to 2014 fiscal year; 393 in the 2014 to 2015 fiscal year; and 470 in the 2015 to 2016 fiscal year.

The dotted line shows the number of persons taking individual training: 88 in the 2011 to 2012 fiscal year, 49 in the 2012 to 2013 fiscal year, 73 in the 2013 to 2014 fiscal year, 77 in the 2014 to 2015 fiscal year, and 78 in the 2015 to 2016 fiscal year.

Source: Administrative data, attendance reports

The Secretariat’s Official Languages Action Plan directs the CLT Program to be available to all indeterminate and term employees. Participation is determined by the Secretariat’s sectors and managers.

The 2014 PSES revealed that since 2008, 70% to 73% of Secretariat employees feel that they obtain the language training they need. The evaluation survey revealed that the majority of Secretariat employees (91%) know that the Secretariat offers language training to its employees. They learn about the program from the internal bulletin TBS In-Brief (78%), but they also learn from their colleagues (44%) and managers (38%). However, of the 464 survey respondents, 36% were not aware of the CLT Program’s additional language training tools.

The evaluation revealed that the CLT Program is less accessible to some employees than others. A few survey respondents noted that they were not approved for language training because they either occupied a unilingual position or they met the linguistic level of their position. As one respondent expressed, “[our] directorate does not support during-hours language training if the employees meets the language requirements of the position. [If the] position is English, [there is] no support to learn the French language during working hours.”

Considering accessibility more broadly, there are two key issues: the first is access to full-time training, and the second is workload and attendance. According to key informant and focus group evidence, some employees become frustrated when they cannot access full-time training to improve their language levels. In response, they take part-time or maintenance training with the hope that they can achieve improved levels. However, the part-time and maintenance courses are not designed for acquisition, so these employees are unlikely to meet their goals and as such may become further frustrated and discouraged. Key informant and focus group evidence shows that this frustration and discouragement are linked to absenteeism.

Key informants and focus group participants both reported that managers do not always approve the recommended length of time for training. Rather, they may be approved for semi-full-time training ^{Footnote 13} or for fewer weeks than was recommended. The risk of this approach is expressed in the following survey quote: “My objective was to achieve Bs. The amount of time was not enough to fulfill this goal. Additionally, work often got in the way of training. In the end, too much time was missed.”

Immediate outcome 3: effective delivery of second language training

Evidence shows that the CLT Program is responsive to employee needs.

• The CLT Program proactively pilots new tools and gathers learner input at the end of every session in order to improve service delivery.
• The CLT Program uses feedback to evolve and to meet the needs of clients when possible, for example, offering courses in the evening.
• Opting out of the Public Services and Procurement Canada linguistic services National Master Standing Offer has allowed for a more flexible array of options for Secretariat employees, for example, smaller classes.

The administrative data supported the evidence from key informants: in most cases, group classes are as effective as individual training. Program data indicate that, on average, group classes produce slightly faster results (296 hours per group versus 310 hours per individual) to attain required language levels in all three competencies. When comparing language test results of group and individual training, success rates were similar for both, with an 85% rate of success for reading, writing and oral.

The key informants noted the conditions under which group classes would be as effective as (or even more effective than) individual lessons. Better results are achieved when:

• group classes are not too big (for example, 4 to 6 when working toward a Level C)
• teachers are skilled
• the learners are all at the same level and working toward the same goal

While individual courses can meet specific needs, such as very tight deadlines for oral exams or tailored instruction for those with learning disabilities, the focus groups confirmed that a group setting can be conducive to efficient learning when there are positive group dynamics.

Achievement of intermediate outcomes

Conclusion

The evidence indicates that the CLT Program is meeting its intermediate outcomes.

• Employees are attaining, improving and maintaining their language levels.
• The evaluation found that centralizing language training delivery was an efficient and effective way to deliver language training at the Secretariat.
• Adopting a group training approach was as effective as individual training. The majority of Secretariat employees who took their PSC SLEs after training either maintained or improved their results.
• Considering both direct and opportunity costs, group classes cost less than individual training and in most circumstances provide equivalent benefit.
• The procurement vehicle had both strengths and limitations. Key strengths were having training offsite and the use of Task Authorizations. The key weakness was the administration of the CLT contract as an unskilled service contract rather than an educational contract. As such, the assessment criteria in the contract did not give sufficient weighting to ensure quality instruction and a quality learning environment.

Intermediate outcome 1: employees meet the linguistic profile of their positions

Program administrative data show that Secretariat employees appointed non-imperatively receive language training as required by the Public Service Official Languages Exclusion Approval Order, and 89% attain their language levels within the two-year time frame. For example, of the 79 non-imperative appointments since the 2011 to 2012 fiscal year, only two did not meet the required language levels within the two years allocated and were granted extensions by the Secretary of the Treasury Board. Five employees were deployed into positions with a lower language level.

Intermediate outcome 2: employees improve and maintain their second language abilities

Second language improvement

Of survey respondents who took language training, 62% said that it met their learning objectives. For those survey respondents who took the PSC SLEs after their training:

• 48% said that the CLT Program adequately prepared them for the assessments
• 26% said that the CLT Program did not adequately prepare them for the PSC SLEs
• 45% said that their language results improved
• 52% said that their language results stayed the same

Second language maintenance

Second language maintenance is heavily reliant on language use. The employee survey, focus groups and the key informants quite clearly demonstrated that employees see the employer as having a primary responsibility for enabling them to acquire and maintain their language levels. That some learners assume that training will always be provided is evident in the remarks shared with instructors as they complete their assessments, such as “See you in five years.”

However, a shared responsibility is implied by the 2010 Departmental Policy on Official Languages, which states that employees “use the official language of their choice and are encouraged to maintain acquired second language skills by actively using them in their workplace.” For managers, it states they “are responsible for promoting and implementing the [policy] by ensuring the use of both English and French in daily operations.... They must also ensure English- and French-speaking employees have equal access to employment and advancement opportunities.”

Further to this, the 2013 report called Challenges: The New Environment for Language Training in the Federal Public Service by the Office of the Commissioner of Official Languages states that “it is reasonable that an employee’s prior commitment to his or her language retention be considered as a criterion in making funding available. In other words, employers must encourage employee language retention through various means, but employees must also demonstrate their own efforts in this regard.”

The CLT Program offers multiple options for second language maintenance. Administrative data show that attendance rates vary vastly by the type of class, with attendance in maintenance classes being poorest (58%) and one-week workshops being highest (94%) (see Figure 3). The survey respondents noted that the factors contributing to absenteeism (aside from illness) are workload, which is the most common reason provided for learner absences, and unreasonable language acquisition expectations (discussed earlier).

Figure 3 - Text version

The figure is a horizontal bar graph. Categories are on the vertical axis (y-axis), and values on the horizontal axis (x-axis). The vertical axis shows four types of attendance plus the total for overall attendance, and the horizontal axis shows the rates of absenteeism.

The total overall attendance rate was 79%, the maintenance class attendance rate was 58%, the group class attendance rate was 82%, the individual attendance rate was 91%, and the workshop attendance rate was 94%.

Note: for individual attendance rates, absences are sometimes hidden by rescheduling.

Source: Administrative data, attendance reports

*For individual attendance, absences are sometimes hidden by rescheduling.

Intermediate outcome 3: efficient delivery of second language training

The centralized administration of the program was universally praised by senior management, learners and the service provider. The single-window access point for language training was felt to be quick, efficient, transparent and responsive for a more effective delivery of the program.

In an effort to be more responsive to employees, HRD implemented continuous enrolment in 2013 and removed the two-person minimum for groups. While this change to program delivery offered more flexibility to Secretariat employees, it disrupted the classrooms and often resulted in classes of one to two students.^{Footnote 14} This approach was discontinued in 2016.

As part of its contract under the program, the service provider was required to deliver the majority of the training off-site. As the data show, this was a program strength, given that adults learn better when they attend language training away from the workplace. There are two distinct reasons for this:

• Learning a second language in the workplace, which may result in the appearance of lower competence, can stifle the learner’s willingness to speak and make mistakes.
• Leaving the workplace can also result in better attention to the class and less disruption by work matters.

Another effective feature of the contract was that it provided the Secretariat the flexibility to establish Task Authorizations (mini-contracts) on an ongoing basis, as well as to qualify teachers as required. The service provider would try (usually with success) to switch teachers and classes if requested by the learner, which helped them respond to learners’ needs.

The contract did have some limitations. There is evidence that over the five years of the program, the procurement vehicle was not adequately responsive to the evolving opportunities for language training options. For example, when employees asked for e-learning options, the contract restrictions precluded these offerings being available to the Secretariat.

One important finding related to procurement was that focus group participants indicated that they no longer bid on federal contracts, as they did not allow schools to adequately remunerate their teachers and thereby retain them. As noted earlier, the GC demand for FSL training has created a localized teacher shortage in the NCR. This gap, in combination with a propensity to award language contracts to lowest-cost bidders, can result in service providers who cannot hire and retain those instructors with a high level of teaching experience or familiarity with government vocabulary. This suggests that there may be some issues in balancing quality and price in the RFP rating criteria. This is worth noting, given the issue of teacher turnover and the importance of quality teachers for quality learning.

Group versus individual training

Efficiency was assessed by comparing the costs of individual versus group language training. The evaluation compared the costs of achieving language levels for those in individual training versus those in group training (that is, successfully passing evaluations in comprehension, grammar and oral skill needed to meet job requirements). Only the EX category had sufficient numbers in both types of training to be compared.^{Footnote 15}  A group of seventeen EXs was compared to a group of 76 EXs; the advantage of this comparison was that the candidates were well matched from a profile perspective.

According to the administrative data, direct training costs (contracted service delivery) for language training totalled $4.2 million for the period evaluated. Opportunity costs (staff salaries paid while on training) were much higher ($11.1 million in total, excluding benefits).

The performance data showed that the average training cost for those studying in a group and who passed all three tests (as required for their position) was nearly half the cost per person than that of one-on-one training (see Figure 4).

Figure 4 - Text version

This figure is a pictorial description of the costs of individual versus group training. The figure consists of two small circles. Inside the first circle is a single stick person, with $10,544 written above the circle. Inside the second circle are four stick people, with $6,075 written above the circle. The amount of $6,075 is the average training cost per person for those studying in a group and who passed all three tests (as required for their position). The amount of $10,544 is the average training cost per person for those in individual training.

Opportunity costs (that is, salary costs for learners while on training) were similar in both groups in achieving their levels. In terms of training hours, learning for both groups and individuals took a similar amount of time to achieve the levels sought, 296 hours for individuals in a group and 310 hours for individuals.

Achievement of long-term outcome

Conclusion

The program is making progress toward attaining its long-term outcome; however, there are barriers to full realization.

Long-term outcome: enable colleagues to work in the language of their choice

Given the level of bilingualism at the Secretariat, at almost 87%, it might be assumed that employees would feel able to work in their language of choice. However, the relationship between increasing rates of bilingualism and an increase in employees’ ability to work in the official language of their choice is not clear.

The Secretariat’s 2014 Survey on Language of Work found a large discrepancy between the perceptions of Anglophone and Francophone employees regarding the use of both official languages. The survey indicated that although 76% of Anglophones felt that Secretariat meetings were conducted in both official languages and felt comfortable using their preferred official language, only 41% of Francophones felt the same way.

Of CLT survey respondents, 67% of those who participated in the CLT Program reported that after training they are more comfortable working in their second language, and 64% said that they had the opportunity to use their second language at work after their training. However, only 42% said that the training increased the actual frequency of their use of their second language.

This is in line with a reality raised in the 2013 Office of the Commissioner of Official Language report: Challenges: The New Environment for Language Training in the Federal Public Service: “language training enables employees to prepare for a test, but that the training is not enough to make them comfortable using the acquired skills in their work.”

The interview with the language assessment expert and focus group participants raised several points for consideration, which may be barriers to a full realization of the long-term outcome. Three points are as follows:

1. Employees aspire to being, and also appearing, competent in the workplace; working in one’s second language may make communicating genuinely more difficult and create an appearance of lower competence.
2. With added time pressure, communicating in one’s second language often takes longer, and the workplace environment may not support taking additional time. Again, requiring more time may call into question the competence of the individual.
3. The skills and vocabulary required in some professions or in some situations require a high degree of precision and accuracy; this degree of second language proficiency can take years to attain.

These views were reinforced through comments in the Secretariat’s 2014 Survey on Language of Work in which 72% of respondents said they switch to the first official language of a colleague when they sense that he or she is struggling in his or her second official language. CLT survey respondents indicated similar sentiments. As one summarized: “Most have no patience to allow Anglophone to practice or use their French second language and would prefer to change into English in the [interest] of time/efficiency.”

The evidence revealed a disparity between the official languages policy objectives, public servants working in their language of choice, and the goals of survey respondents. The survey results show that 45% of employees take training primarily to progress in their careers. The second and third most commonly cited reasons were to improve second language and maintenance. This indicates that the achievement of desired language levels does not necessarily lead to employees being enabled to work in their language of choice.

Research completed by the Office of the Commissioner of Official Languages of Canada suggests that as public servants' second official language competence and comfort levels increase, they will increasingly be able to use their language of choice.^{Footnote 16} In other words, an employee is more likely to use their official language of choice with colleagues if they are confident it won't impede mutual understanding. However, the findings of the evaluation do not necessarily support this.

In conclusion, there is no strong evidence to show that at the Secretariat an individual's improved second language proficiency positively impacts others using their language of choice.

Recommendations

It is recommended that:

1. The Secretariat continue to offer the CLT Program through a centralized model
2. The CLT Program place increased emphasis on language maintenance, including the responsibility of employees in this matter, and review ways to support ongoing use of the employee’s second language
3. The Secretariat’s Human Resources Division work with the procurement unit of the Corporate Services Sector to ensure a balance between quality and cost rating criteria used to assess the CLT Request for Proposals to ensure higher-quality instruction and to optimize learning
4. A departmental requirement be established that would define the conditions in which each sector would offer full- and part-time training to ensure policy compliance and training access
5. A departmental mechanism (for example, a training contract) be developed for managers and learners for both full- and part-time training that would:
   • Respect the recommended length of training time, to the degree possible
   • Minimize unnecessary test-taking
   • Commit employers to giving employees the time to attend their training course
   • Commit to support employees to practise skills in the workplace