Review of Lessons Learned - Accommodation Measures for Unsupervised Online Second Language Evaluation Tests

Table of Contents

• Introduction
• 1.1. Background
• 1.2 Review objective, scope and approach
• 2. Lessons learned and best practices
• 2.1 Project planning and design
• 2.2 Project governance and coordination
• 2.3 Agile method as a project management approach
• 3. Conclusion
• Appendices

Approved version of the report, June 11, 2021

Introduction

1.1. Background

1. Under normal circumstances, the Appointment Policy requires federal departments and agencies to use Public Service Commission of Canada (PSC) tests to assess second language proficiency for all appointments to bilingual positions.^{Footnote 1} These tests are administered in PSC test centres across the country, as well as in federal departments and agencies.

2. The COVID-19 pandemic affected government operations in many areas, including the ability of the PSC and departments to conduct in-person testing. More importantly, the pandemic has increased the demands and expectations of federal organizations and Canadians for access to modern assessment and staffing solutions and approaches that meet the needs of all users in remote work situations.^{Footnote 2}

3. On March 24, 2020, the PSC quickly implemented 2 temporary measures and supporting directives to provide more flexibility to departments and agencies in assessing second language requirements for bilingual positions. These measures, which were originally intended to remain in force until March 31, 2021, have been extended until September 30, 2021. They first exempt deputy heads from the requirement to use PSC’s tests to evaluate second language proficiency (reading, writing and oral). Second, these temporary measures allow departments and agencies to use an alternative assessment method.

4. However, the use of this alternative assessment method must respect the PSC's guiding principles for second language evaluation alternatives. ^{Footnote 3} It must be emphasized that, for any appointment of more than 12 months, the appointee’s proficiency in their second official language must be confirmed using the PSC’s second language evaluation tests within 24 months of the appointment.

5. As part of the special measures related to the COVID-19 pandemic, the PSC launched unsupervised online second language evaluation tests for reading comprehension and written expression. Unsupervised online second language evaluation tests allow departments and agencies to fill bilingual positions during the pandemic. ^{Footnote 4} These tests are performed online and remotely. They are available for all new job ads posted on the GC Jobs recruitment platform.

6. The number of second language evaluation tests conducted by the PSC at its test centres across the country, as well as by departments, decreased significantly in 2020–21 compared to 2019–20 (see Tables 2 and 3 in the Appendix). It is important to note that the tests associated with the temporary special measures replaced the regular tests in fiscal year 2020–21. As for unsupervised internet tests, 6 315 reading tests and 7 491 writing tests were administered between January 16 and March 31, 2021 (see Table 4 in Appendix).

7. However, in the summer of 2020, a risk was identified with respect to the administration of unsupervised online tests, as these assessments were not meeting the expectations of candidates who required accommodation. To mitigate this risk, the PSC established a team to work on a project on accommodation measures for unsupervised online testing. This team included staff from the Services and Business Development Sector and the Information Technology Services Directorate of the Corporate Affairs Sector.

8. On September 17, the Executive Management Committee was informed that the cost to complete the project on accommodation measures for unsupervised online testing would be approximately $150,000 for the Information Technology Services Directorate and $50,000 for the Services and Business Development Sector. A plan outlining the project was presented. It emphasized the importance of ensuring that dedicated resources would be fully allocated to carry out this project. The Vice-President of Corporate Affairs committed to providing regular updates on the project.

9. Despite the challenges reported in early October 2020 in obtaining the servers needed to complete the project, the PSC was able to obtain the required servers from Shared Services Canada; most importantly, the PSC was successful in obtaining them in the demonstration phase before the end of October 2020. It was noted, however, that the project timeline may be compromised due to competing priorities being worked on by the Information Technology Services Directorate and the Personnel Psychology Centre.

10. It should also be noted that the unsupervised online testing accommodation project was required because these tests were hosted through the Public Service Resourcing System, on the existing unsupervised online testing platform. However, this platform did not offer the capacity to meet the expectations of candidates who require accommodation for taking second language tests, including more time, a larger font, or a reading service (reading test questions aloud).

11. These limitations of the existing online testing platform led the project team to conclude that unsupervised online testing for second language evaluations was not fully accessible. Before the pandemic, PSC tests met accessibility standards. To address this issue, the project team found that the new candidate assessment tool, which previously hosted the "Management Inbox – Electronic" developed in collaboration with Code for Canada ^{Footnote 5}, not only offered common functionality for accommodation requests, such as changes to the test schedule or duration (more time, breaks), but would also make other types of accommodation possible, such as multiple-choice tests.

12. Given the shortcomings of the old unsupervised online platform for second language evaluation tests, the project team decided to host the tests on the new assessment tool platform, especially for candidates who require accommodation. This new platform, which helps meet the needs of applicants with disabilities, officially went into production on February 9, 2021, and has been in use since March 12, 2021.

1.2 Review objective, scope and approach

13. The project on accommodation measures for unsupervised online second language evaluation tests was designed to facilitate access to these tests for persons requiring accommodation, thereby allowing departments and agencies to fill bilingual positions during the pandemic. Accommodation measures are changes made to the procedure, format or content of a second language evaluation tool. They are intended to remove barriers to fair assessment by allowing candidates to fully demonstrate their skills.^{Footnote 6} It is important to note that accommodation measures do not change the nature or level of the qualification being assessed .^{Footnote 7}

14. The purpose of this review was to examine lessons learned about PSC activities related to planning, designing and implementing accommodation measures for unsupervised online second language evaluation tests. This is a priority project for the PSC in response to the COVID-19 pandemic. It is included in the Internal Audit and Evaluation Plan 2021–23.

15. The review identified best practices and lessons learned that can be applied to planning, designing and implementing similar projects in the future. The Internal Audit and Evaluation Directorate was able to carry out this work through close collaboration with the Information Technology Services Directorate and the Services and Business Development Sector.

16. The scope of the review included activities undertaken between September 2020 and March 2021 to contribute to advancing this priority project, which was put in place to provide accommodation for candidates who request to take an unsupervised online second language evaluation test for the Test of Reading Comprehension and Test of Written Expression.

17. The key aspects identified for this review are grouped into 3 areas that are important to the success of a project of this magnitude: planning and design, governance structure and project management approach. These aspects were examined to identify lessons learned that could be applied to planning and implementing similar projects at the PSC.

18. The approach to this review was based on the following methods:

• Document review: The review included documents from the project on accommodation measures for unsupervised online second language evaluation tests that were provided by some of the key stakeholders, as well as PSC departmental policy and strategy documents.
• Interviews: In total, we conducted 20 interviews with key stakeholders involved in designing and implementing the accommodation project to learn about their experiences with the project. Stakeholders consulted included representatives from the Information Technology Services Directorate and the Services and Business Development Sector, including staff from the Personnel Psychology Centre and the National Recruitment Directorate.
• Literature review:The literature review focused on publications that offer other views and perspectives on issues related to project management and the use of agile methodology.

2. Lessons learned and best practices

2.1 Project planning and design

19. Generally, the purpose of project planning is to determine in advance how the output will be produced and delivered within project constraints. It is therefore necessary to determine at the planning stage what needs to be done, how, by whom, at what cost and within what timeline.

Definition of project work and requirements

20. The review team found a divergence of views among those consulted as to whether the project scope, requirements, deliverables and timeline were clearly defined.

21. Some of those consulted noted that the scope of the project was defined in a minimalist manner, using the concept of “minimum viable product.” The review team was informed that due to the exceptional circumstances of the pandemic and the urgency of addressing the needs of applicants with disabilities, a minimal definition of the project scope was required. A “minimum viable product” is one that addresses the most important user needs, and allows for the planning and prioritization of the most expected functionalities of an application.^{Footnote 8} The concept of “minimum viable product” is often put into practice in agile project management, especially for the development of a computer application or software.

22. According to these respondents, most of whom are employees of the Information Technology Services Directorate, it would not have been realistic to clearly define all requirements and needs before starting the project. These respondents felt that, despite a minimal definition of the project scope and plan, the project team had a good level of understanding and knowledge of the deliverables expected for each work cycle. In other words, the project team had a vision, a budget already established and a definition of the work or project requirements that had been broadly assessed, but which focused on the minimum viable product.

23. In contrast, some of the other persons consulted felt that the scope of the project was defined in broad terms only, with very little detail on the deliverables and the project requirements and timeline. According to these individuals, most of whom work in the Personnel Psychology Centre, this has created some ambiguity about the project. They believe that the implementation of the project would have been more effective if there had been a minimally defined work plan, so that they could better understand their role in the project.

24. This difference in views between the stakeholders regarding the planning and definition of requirements and deliverables could be explained, on the one hand, by the fundamental differences between the project management method used for this project, known as “agile management,” and the traditional approach to project management, and on the other hand, by the way the agile method was used in this project. This distinction will be discussed in the last part of the report (see section 2.3).

25. At this point, however, it should be noted that beyond this divergence of views on the project management approach, some interviewees suggested that for future similar initiatives, it would be necessary to ensure from the planning stage that all components of the project team work together to share a common understanding of the project management approach and its relevance to the project.

Establishment of a project team and allocation of resources

26. The term “project team” refers collectively to “those individuals who work on the project and are assigned to it either full time for a significant portion of the implementation phase or part time for most of the implementation phase”^{Footnote 9} [translation]. Overall, the review found that the development of the accommodation platform was successful because of the approach used in planning the project: building the project team, allocating resources and providing a reasonable timeline for developing the platform.

27. With respect to the approach used to assemble the project team, almost all interviewees noted that having people dedicated to the project, as well as the required expertise in application development, quality assurance and accessibility, contributed greatly to the success of the project. This team included members from the Services and Business Development Sector and the Information Technology Services Directorate.

28. Another frequently cited success factor is the commitment of all team members to work hard to move the project forward, especially the Information Technology team members. Several respondents mentioned that without this commitment from the project team, it would have been difficult to meet the deadline and achieve such good results.

29. However, the review team noted that some aspects of team building and resource allocation could have been better planned before the project started. For example, a fundamental aspect of project planning is the selection and availability of resource persons. This aspect helps to identify the right people, assign them to tasks, and obtain their commitment. Several interviewees acknowledged that the project team members showed determination and commitment to moving the project forward even though they had to manage competing priorities, which posed a challenge to implementing the project in a timely manner. It should be noted that employees of the Information Technology Services Directorate were working on other priority projects at the same time, and that Personnel Psychology Centre employees were working on 5 other related projects to improve the delivery of second language evaluation services.

30. The review found that the Personnel Psychology Centre, like the Information Technology Services Directorate, is a very busy work environment. It is difficult for stakeholders involved in a project to make a commitment until they know the list of tasks to be completed, the expected timeline and the budget available to them. The selection and assignment of resource persons is an important aspect of project planning that could be reviewed and improved. As such, the Information Technology Services Directorate and the Personnel Psychology Centre should continue to coordinate their efforts and work together at the project planning stage to optimize the process of identifying the resources required to deliver projects. This would increase the effectiveness of the project teams once they are formed.

31. To optimize planning efforts and support better functioning of the project team, interviewees made a few suggestions about the need to have the right resources in place before the project starts. The first suggestion emphasizes the importance of having as complete a team as possible at the beginning of the project. Several respondents indicated that a project of this magnitude, requiring collaboration among various sectors of the PSC, would require a project manager capable of ensuring proper project planning, identifying project risks and mitigation measures, assessing progress, and most importantly, linking the various project stakeholders from the planning stage through to full project completion. The importance of the project manager is supported by the project management literature which states that it is the project manager who drives the planning as much as possible.^{Footnote 10}

32. The review team found that the lack of a project manager had a relatively negative impact on the cohesiveness and effectiveness of the team, especially in the early stages, as a project manager was appointed shortly after the project began. Almost everyone consulted emphasized the critical role that the project manager played in ensuring that everything was in place and that efforts were coordinated to complete the project.

33. Similarly, respondents indicated that it is crucial to have a business analyst at the beginning of the project who can translate the client's requirements into computer language. Several interviewees felt that this role had been somewhat lacking at the beginning of the project. The analysis revealed that this role was indeed assumed by a member of the project team, due to the urgency and tight schedule for developing new applications, but that this person also had to assume several other roles at the same time. Given the importance of having a dedicated business analyst for the project, one should clearly have been identified and their name communicated to the entire project team. In short, the success of a project with a strong digital component depends on a project team that, from the outset, includes a project manager and a business analyst. The importance of these 2 key players will be discussed later in section 2.3.

Application development (platform)

34. As previously mentioned, the purpose of the project on accommodation measures for unsupervised online second language evaluation tests was to host the latest tools developed by the PSC for second language evaluation on a new accessible online testing platform. This platform aims to ensure that language testing is accessible to all Canadians, including those who require accommodation. To do this, the project team, including the PSC's application development experts, used the new Candidate Assessment Tool platform that was already being developed in collaboration with Code for Canada.

35. The review demonstrated that the development of the accommodation platform for unsupervised online second language evaluation tests was a great success, given the tight timeline and exceptional circumstances in which it was developed. Indeed, the project was launched only in September 2020, while the platform has already been in use since April 2021. The new platform offers features common to many accommodation requests, such as the ability to change the test schedule or duration, or to take breaks.^{Footnote 11}

36. A large majority of interviewees noted that the greatest success of this project was the fact that the PSC was able to develop a platform to administer these tests in only a few months, in order to meet the expectations of those requiring accommodation.

37. Finally, interviewees noted that the platform was developed according to the highest industry standards while considering PSC future needs. For example, respondents indicated that the platform was developed using modern digital tools that meet cloud standards. According to these interviewees, the transfer of the platform developed for accommodation measures to the cloud-based platform should be easier once the PSC's cloud-based structure is in place.

2.2 Project governance and coordination

38. The review found that, in general, an appropriate governance structure was in place to support project planning, design and implementation. This governance structure included the key stakeholders involved in the project as well as experts who could understand the related issues. It consisted primarily of the Project Management Committee, the Personnel Psychology Centre Steering Committee for unsupervised online testing, in addition to the Project Team and the PSC Executive Management Committee.

39. Project Management Committee: This committee served as the main decision-making body for the project. It was composed of representatives from the Services and Business Development Sector and the Corporate Affairs Sector, including the 2 Vice-Presidents, the Acting Director General of the Personnel Psychology Centre, the Director General of the Information Technology Services Directorate, and the directors and managers of these 2 directorates who were also contributing to the project. This committee met weekly to review the scope and progress of the project, to make quick decisions when necessary to complete the project, and to provide guidance to ensure that the various components of the project were progressing as planned.

40. Steering Committee, Personnel Psychology Centre, Unsupervised Online Testing: This committee was established in the Services and Business Development Sector to support the Project Management Committee. It included all the stakeholders in this sector associated with the project and met once a week.

41. Project team: This team included operational level managers and employees from the Services and Business Development Sector and the Information Technology Services Directorate. Short meetings of approximately 15 minutes were held each day to allow team members to collaborate, identify and resolve emerging issues, and ensure the smooth functioning of the team.

42. Executive Management Committee: Regular presentations were made to the Executive Management Committee to update them on the progress of all projects related to second language evaluation, including the accommodation project.

What worked well

43. Less strict and more flexible governance structure: Several interviewees mentioned that having established lines of communication based on a looser governance structure for this project really helped to speed up the decision-making process, as it did not require strict adherence to the usual approval procedures. For example, it was not necessary to follow all the normal approval procedures except for the Information Management and Technology Committee, which was already aware that this project was under way.

44. Senior management commitment and support: Several respondents said they particularly appreciated the commitment of senior management at Project Management Committee follow-up meetings. These respondents explained that the director generals and directors who participated in the Management Committee meetings facilitated by the project manager were truly engaged, which allowed for the early resolution of project challenges and issues. Similarly, some interviewees stated that they got all the support they needed from the Services and Business Development Sector. According to these respondents, access to senior management was easy and contributed to timely and effective decision-making because it was a priority project.

45. Engaging key stakeholders: The review found that having key stakeholders and all those directly involved in the project at the same table each week facilitated the decision-making process. The mobilization and active participation of key players contributed to timely project management decisions that were communicated to all interested stakeholders. Based on their experiences in project management at the PSC, some stakeholders in the Information Technology Services Directorate suggested that such a governance structure ensuring the engagement and participation of key players able to make timely and informed decisions should be put in place for all digital technology projects. These respondents informed the review team that there was not the same level of engagement of key stakeholders in the past, and that the governance structure in place was not appropriate for moving forward with projects that have a digital component.

46. Good collaboration between the various players: Most respondents highlighted the collaboration between the different players as a factor that contributed to the success of the project. They felt that the different teams worked together, listened to each other, and worked effectively even when requirements and some roles were not clearly defined. They mentioned that in the past, good collaboration and listening skills were challenges, and that it was often difficult for units involved in a project to clearly understand the needs of their counterparts.

47. Daily project team meetings: Most interviewees considered the daily meetings to be one of the key elements that contributed to the success of the project. Several respondents stated that these daily meetings provided a forum for all stakeholders to discuss project issues and problems that arose in order to find immediate solutions. In their opinion, this contributed greatly to the progress of the project. However, a few respondents felt that, at times, some key players were unable to attend these meetings due to competing priorities and gaps in resource allocation. For this reason, these respondents stressed that for future similar projects, it would be important to ensure a better presence and representation of the entities involved in these projects so that related information is communicated in a consistent and timely manner to key stakeholders.

What didn’t work so well

48. Definition of roles and responsibilities: Several interviewees indicated that roles and responsibilities could be better defined, especially at the beginning of the project. They mentioned, for example, that there were sub-projects that were very loosely defined and for which no one was responsible, which created some ambiguity. This is particularly true for the translation and verification of labels used in the platform's functionalities. Respondents stated that with the appointment of the project manager, roles and responsibilities were better defined.

49. Communication and information sharing: Many interviewees stated that they sometimes felt that information could have been better shared in a timelier manner, even though communication among the various stakeholders and with the client went relatively smoothly. These respondents explained that because of the large number of meetings at different levels, decisions and minutes of meetings sometimes do not effectively reach all employees at the operational level. Other respondents mentioned that at times they felt there was too much information, sometimes felt to be contradictory, from different project stakeholders.

2.3 Agile method as a project management approach

50. The review team found that the adoption of an agile project management method was generally welcomed by a large proportion of interviewees who felt that this approach contributed significantly to the project's success. However, this view was not shared by all those consulted. Before discussing in detail the impact of the agile method on the project process, it is necessary to present what this method consists of and how it differs from the traditional approach to project management.

Differences between the agile method and the traditional project management method

51. The “agile method” is a structured, iterative decision-making process that allows an organization to adapt to and cope with uncertainty. It is also referred to as “adaptive project management” because it is an intentional approach that allows decisions and adjustments to be made in response to new information or a change in context.^{Footnote 12}

52. While there are many definitions, it should be noted that experimentation is at the heart of agile management, including assumptions, controls and replication opportunities. The agile method differs essentially from the traditional project management method in the decision-making process, and above all, in the ability of the organization or project team to adapt and respond effectively to dynamics and uncertainties. Table 1 briefly presents some examples that illustrate the difference between the 2 approaches.

Table 1: Differences between the traditional approach and the agile project management method

Traditional approach	Agile approach
We propose this solution to meet our objective.	The solutions are not known in advance.
We try to plan as much as possible from the start.	We plan as we go along, throughout the intervention.
We strive to follow the plan and adjust it as needed.	The plan will change, perhaps even completely.
We track project activities and costs first, and then the effects.	We track project effects first, then activities and costs.
We learn at the end of the project for future interventions.	We are learning throughout the project for this and future interventions.
We must try to achieve everything.	We can only achieve so much. Failure is part of an intervention.

Impact of the agile method on project planning

53. As we noted earlier, the review revealed a divergence of views among respondents with respect to the minimum definition of project requirements, deliverables and timelines. It should be noted that these are the principles advocated by the agile method, which was the business model on which the project on accommodations for unsupervised online second language evaluation tests was based.

54. According to interviewees who advocated the agile method, this approach allowed the project team to be efficient by allowing to do several things at once, to not put too much emphasis on long-term planning, as it is impossible to plan a near-perfect list of requirements in advance, and above all, it offered more flexibility and speed in decision making. For example, these respondents noted that the project was completed in a timely manner because the focus was on developing the container rather than the content. In other words, more emphasis was placed on the pre-design of the platform without necessarily waiting for all the testing to be completed before proceeding with the other tasks planned for this project.

55. However, for some respondents, the lack of a clear definition of deliverables or requirements at the outset of the project is one of the biggest shortcomings observed in the management of the project. According to these respondents, this approach to project management created ambiguity around the project and resulted in delays that could have been avoided if the project team had taken more time to plan the project in advance.

56. It must be said that this divergence of views refers to a fundamental difference between the traditional project management approach and the agile approach in terms of the important factors that need to be considered in the planning stage of a project. In the traditional approach, project planning is a decision to determine in advance the means by which the agent will produce and deliver the project output, with the resources available, within the constraints.^{Footnote 13} According to this traditional approach, project planning must be carried out with the utmost care, as it is a prerequisite for the execution of the project.^{Footnote 14}

57. The agile management approach is at the other end of the project management continuum. In this project management model, the project team does not plan everything at once. Instead, it plans a little bit, delivers, learns, and then plans a little bit more in a continuous cycle. Instead of drawing a precise, predetermined path to reach its goal, agile planning considers intermediate outcomes (called milestones) and organizes shorter activity-planning cycles around these milestones.^{Footnote 15}

58. For example, some interviewees mentioned that clients expect to get a perfect product all at once. Such an expectation was not entirely realistic in this case because of the nature of the project. Some of the respondents explained that in the digital world, it is often difficult for work teams to come up with accurate estimates, because these teams are usually working with new technologies or on projects that seem vague at first. Therefore, a definition of deliverables at each stage seemed more realistic.

59. Beyond this difference of opinion, we have found that certain factors are crucial to the success of a project. These are factors already raised in section 2.1 and focus on the availability of required resources before the start of a project, including the importance of having a project manager and business analyst and ensuring that key stakeholders have a common understanding of the project deliverables.

60. Based on their experience, respondents raised other considerations that must also be taken into account to ensure the success of a project. They include a common understanding of the project management approach used and a clear list of minimum requirements understood by all stakeholders at the outset of the project. For these respondents, it is difficult to effectively implement an agile method if the various elements of the project team do not share the same understanding of this approach and do not agree on its relevance.

61. In the case of the project on accommodation measures for unsupervised online second language evaluation tests, some respondents stated that sufficient and common understanding among all elements of the project team about the project management approach and expectations could have been improved at the planning stage. For example, since the minimum viable product is a key element of the agile method, a consensus should have been reached among the various stakeholders, not only with regard to the minimum functionalities required for the project, but also with regard to IT security requirements and the minimum controls required to be made while implementing the project.

62. In this regard, some interviewees felt that data management, as well as IT security and reporting for communication purposes, should be considered as fundamental requirements in the digital solutions implemented at the PSC, starting at the planning phase.

63. With respect to data management, the review team was informed that this component was discussed during the project planning phase, but was not included in the project scope due to the urgency to act. This is an essential dimension of the optimal implementation of projects with a digital component. As concluded in the formative evaluation of the implementation of the PSC Data Management Strategy approved in 2020, ^{Footnote 16} consideration should be given to developing and documenting data quality assurance procedures up front to support improved data management practices once new or updated solutions are in place.

64. Given that the platform was developed with the PSC's future digital solution needs in mind, greater consideration could have been given to data management and quality concerns. To this end, the Information Technology Services Directorate could collaborate with the Data Services and Analysis Directorate to ensure better integration of data management requirements into the planning of projects with a digital component. This would promote optimal implementation of digital solutions at the PSC.

65. Improved communication with the client during the project planning phase would have reduced the variances observed with respect to the functionalities identified as part of the minimum viable product, thereby promoting greater efficiency in project delivery. Similarly, it was suggested that brief terms of reference (2 to 3 pages maximum) be developed to ensure better planning for similar projects in the future. This document could contain a general overview of the work plan, a timeline with expected deadlines, and a brief description of the roles and responsibilities of each party.

Impact of the agile method on project delivery

66. During the implementation of the project, activities were prioritized in accordance with the agile method, that is, this took place throughout the intervention, as the context, needs and resources evolved. The review found differing views on this approach to priority setting. This divergence is due, as noted above, to a lack of common understanding of the basis of the approach used to manage this project.

67. For some respondents, the prioritization of activities was based on the overall project objective and the target for each of the key activities. The prioritized list of tasks for each team was updated at each short-planning cycle. Other respondents felt that this prioritization could have been improved. For the latter, the competing priorities being worked on by the Information Technology Services Directorate and the Personnel Psychology Centre posed a challenge to the effective implementation of the project. For these respondents, proper prioritization is necessary to optimize project timelines and ensure that the human resources dedicated to the project are available in a timely manner.

68. However, several interviewees felt that the project was well managed in terms of risk. Indeed, potential risks were identified in a timely manner and measures were taken to mitigate them. The review team also found that the project's governance structure ensured that the project schedule was met and that the scope of the project was adjusted as it was implemented to focus on the essential work to be done.

69. In terms of improvements that could be considered for similar projects, a large majority of interviewees suggested, first, that there should be increased client availability to test functionality as it is developed by the Information Technology Services Directorate. Increased client involvement should be ensured throughout the execution of the project. This would ensure that the features developed correspond to the characteristics desired by the client. Respondents also stressed the importance of ensuring that the content for each feature developed is translated in a timely manner.

70. Secondly, interviewees stated that due to the urgency of developing the platform as quickly as possible, there were gaps in the documentation of deliverables, particularly in relation to the new features developed. In the future, time should be set aside to properly document each new feature as it is created so as to avoid ambiguity in project execution.

71. Similarly, the review team found that there was some lack of documentation or cost control for the project, as the project team did not have the necessary computer tools and programs to track the expenditures incurred for this project. This is an important aspect of project management which, according to the information gathered, should be taken into consideration in future projects in order to identify and correct any potential cost overruns in a timely manner. In project management, documenting or estimating costs as accurately as possible is not just a matter of accounting accuracy, but it is also a management concern.

3. Conclusion

72. According to the majority of those consulted for this review, the project on accommodation measures for unsupervised online second language evaluation tests is a success, because the PSC was able to put in place a test administration platform in a few months to meet the expectations of those requiring accommodation. It is also a success given the tight timelines and exceptional circumstances under which the PSC developed this platform.

73. The review team learned that the platform was developed according to the highest industry standards while taking into account the PSC's future needs for digital solutions. For example, the platform was developed using modern tools that respect cloud standards, which will facilitate its transfer to a cloud platform when the PSC is ready for this shift.

74. The review also identified best practices and lessons learned in the planning, governance and management of a project with a digital component. The review team determined that before starting such a project, it is essential to ensure that the project team includes a project manager and a business analyst. It is also important to ensure, at the planning stage, that key stakeholders have a common understanding of the project management approach, how this method will be implemented and the expected deliverables. The review team learned that in order to optimize the prioritization and allocation of resources to a project, it is important that all sectors involved work together and coordinate their efforts from the planning stage. Similarly, for digital solutions, it is important in the planning phase to consider a way to develop and document key requirements such as data management and quality, IT security and reporting for communication purposes. This would promote optimal implementation of digital solutions at the PSC.

75. The governance structure was critical to the success of this project. The review team confirmed that this project was successful because the PSC quickly put in place a team of experts with the commitment and support of senior management, with a flexible and effective governance structure. According to interviewees, this collaboration at all levels of the organization allowed for timely and effective decision-making to address issues and challenges that arose during the course of the project.

76. With respect to the adoption of the agile method as an approach to project management, it should be noted that many interviewees considered this method to be one of the factors that contributed significantly to the success of the project. However, the review team found that it is difficult to implement this approach effectively if the various elements of the project team do not share a common understanding of the “agile” approach and how it will be implemented.

Appendices