Lessons Learned NDSi

Project Sponsor (OPI)

Co-Chair

DG, Business Development and Systems Directorate

Director, IT Services 

A/Director, Office of Renewal

Jason McGuire

1

Project planning

The project charter was drafted collaboratively with, and approved by, the project sponsor and all impacted functional leads. Estimated project resource requirements (types and time allocation) were broken down by functional group and these estimates were embedded in the document. A clear project RASCI (responsible, accountable, supportive, consulted, and informed) matrix was embedded in the document to entrench decision making authority and governance roles and responsibilities from the outset of the project. This helped keep the project on track and within scope.

Draft charters collaboratively with input from all impacted functional leads prior to project kick-off in order to ensure alignment with the project vision and awareness of resourcing requirements.

Entrench decision making authority directly in the charter.

2

Project Governance

The early establishment of the steering committee with senior management representation from all impacted functional groups built a strong decision-making body that facilitated project direction and implementation.

Attempt to represent all functions with resources dedicated to the project on the steering committee within the PSC to ensure that they are aware of what their employees are doing and so that they can have input on the project’s direction.

3

Project Governance

Project governance was based on the establishment and use of multidisciplinary sub-working groups comprised of all impacted areas at PSC. These sub-working groups (including a 3rd party service provider) had the expertise and knowledge required to support a major transformation initiative. They provided input, support, guidance, improved alignment within the PSC, and supported the project executive and steering committee in reporting decisions and actions back to their respective units/team to ensure consistent messaging.

On certain (larger) projects, the creation of multi-disciplinary sub-working groups comprised of all impacted areas at PSC to provide input, support, guidance, and to report back to their respective units/team to ensure consistent messaging is recommended.
 
On smaller projects, the core project team should be comprised of multi-disciplinary members representing all impacted areas at PSC.

Make sure that you have the expertise and knowledge required to successfully undertake your project.

Include policy representatives on the core project team from the outset of the project.

4

Project Leadership

The project sponsor’s vision was clear and motivated the team to innovate and experiment resulting in a very real drive to positively impact the NDSi project outcomes.

The project sponsor’s vision should be clearly articulated and understood by all stakeholders in order to ensure alignment with project goals and objectives and to increase engagement. 

5

Project Leadership

The project steering committee showed an openness to experimentation that allowed the project team to try many new approaches, processes, and tools which increased project team engagement, increased team member professional development, and improved the final project outputs. 

When possible, project governance should be open to trying new approaches, processes, and tools to achieve project objectives. Allowing project teams to fail fast and then forgive them encourages innovation within organizations.

Have the responsible team lead/SME present current outputs to the steering committee to provide governance with real time, relevant updates.

6

Decision making

The strength of the project executive (product owner) role was demonstrated by the fact that there was (relatively) easy access to rapid decision making and guidance with respect to what was required in order to attain specific goals while meeting project expectations.

Future iterative projects require easy access to a product ‘owner’ or a strong proxy in order to ensure the project vision is being respected and accelerate decision making under tight timelines.

7

Decision Making

The core project team was comprised of key stakeholders responsible for monitoring the progress of each key deliverable and for making decisions about course corrections that do not require escalation to the project governance.

Leveraging the core project team to monitor the progress of key deliverables and for making decisions about course corrections that do not require escalation to the project governance is highly recommended.

8

Project Methodology

The creation of a multi-functional core project team comprised of sub-working group team leads serving as the single point of contact with their sub-working groups greatly increased cross-organizational collaboration and fostered representative input from the onset of the project.

The creation of a core project team made up of the sub-working group team leads is highly recommended. These leads should serve as the single point of contact with their sub-working groups and should support the project executive and Steering Committee in sharing decisions and actions.

On smaller projects, the core project team should be comprised of members representing of all impacted areas at PSC.

The inclusion of policy from the outset of any project is strongly recommended.

9

Project Methodology

Scheduling regular, mandatory stand up meetings with core project team members (and ideally the product owner or a strong proxy) provided a regular connection point for the project team which was very valuable. The team was able to address and discuss issues in a more spontaneous manner than other more formal meetings allowed for. The project moved so quickly and intensely at some points that it was beneficial to have the opportunity to touch base with the team to ensure alignment and integration across sub-working groups.

Schedule regular mandatory stand up meetings to track project progress, team alignment, and identify risks.

The purpose of the daily stand up should be re-iterated regularly to ensure that the objective of the meeting is met in a timely manner.

Attendance at daily stand-ups should be mandatory to all team leads and the product owner or proxy. The audience for daily stand-ups should be limited to the above, unless by invitation to ensure the objective is achieved efficiently and effectively.

Project Methodology

The use of a standard, repeatable process (sprints) greatly aided the planning of research, design, user engagement, and testing.

Schedule regular meetings (e.g. design sessions, user engagement sessions, analysis sessions, etc.) based on a standard, repeatable cycle.

Make meetings virtual; if an attendee can’t make it in person, perhaps they can still attend remotely to maintain momentum. Having a virtual meeting option for all sessions can also help promote and sustain remote and regional representation and participation.

Establish a back-up plan for technology; if the technology required to make a meeting virtual does not work, the session must go on. Test the technology well in advance of every session, have alternative methods of delivery (i.e. video-conference if WebEx fails), and have electronic versions of documentation available to distribute electronically if all else fails.

Identify proxies for subject matter experts (SMEs) who can fill in for their absent colleagues.

10

Project Engagement

The project team consulted directly with members of the (external) job seeker community and hiring manager, and HR user groups across the federal public service. Over 120 sessions were held with several hundred users. 

Tremendous insights were obtained from direct user engagement. Avoid the use of proxies and deal directly with users whenever possible to ascertain needs, requirements, and to solicit feedback. It is important to get users involved and engaged early and often.

11

Project Engagement

Several types of awareness and engagement activities were delivered by the NDSi project team (e.g. presentations, demos, elicitation sessions, moderated usability testing, and on-line usability testing) to refine, test, and validate requirements in a variety of ways to ensure an accurate reflection of the user needs. These sessions resulted in valuable user feedback which informed the project’s outputs.

Determine what the project team is trying to elicit from a user group and why prior to planning an engagement session. Also, think carefully about what the project team is going to do with the results and how they are going to use them.

12

Project Engagement

Mock-ups and prototypes were tested with all job seekers and hiring managers via a moderator to garner additional feedback to refine requirements and influence the project direction.

Don’t just talk about the project’s ideas, concepts, and prototypes; get users to test them to see how they actually work! The use of a moderator allows for insights as to what is working (or not) and why.

13

Project Engagement

Follow-up questionnaires were sent to test takers and participants to measure perceived improvements of efficiency, effectiveness, and satisfaction.

Performance measurement is an important aspect of demonstrating project progress and success. Determine the elements the project wishes to measure ahead of time and track the results.  Use different methods to measure efficiency, effectiveness, and satisfaction. Questionnaires are relatively easy to create and administer. Note: don’t forget about translation! It takes time.

14

Project Engagement

Interactive prototypes were tested online with all user groups to measure time on task and task completion rates. Questionnaires were embedded in the online testing to measure perceived improvements of efficiency, effectiveness, and satisfaction.

Performance measurement is an important aspect of demonstrating project success. Interactive prototypes can be used to measure actual time on task rather than perceived improvements. They take longer to create, change, and maintain, but they can garner valuable information in the right circumstances. Done right, they can also measure actual and perceived improvements as well (i.e. the inclusion of questionnaires in the prototype itself.)

15

Project Engagement

The use of a design thinking approach (using images to convey ideas) to engagement sessions guide the exploration and co-creation of refined stakeholder requirements. Design thinking simplified and made ‘real’ the sharing of complex concepts which led to tangible feedback from users.

A picture really IS worth a thousand words; the use of UI flows, wireframes and/or mock-ups, and interactive prototypes greatly improved the communication and understanding of project concepts and ideas that the project team wanted to explore with users. Use images to explore ideas, validate concepts and identify gaps or missing information.

Measure the potential return on investment and value of design-thinking artifacts (wireframes, mock-ups, and prototypes) prior to their use. Wireframes can be created in real time with users using paper and pens or with an online tool such as myBalsamiq for future reference. Wireframes are cheap, interactive, and immediately capture user requirements. Mock-ups are more refined and allow for a more show and tell discussion and feedback elicitation, as well as being a good tool for usability testing in a moderated setting and online. Mock-ups take longer to produce and definitely make users think more of an actual ‘system’ than idea. Interactive prototypes are time-consuming and costly and often do not provide much more value than mock-ups. Interactive prototypes should be used if they introduce value to the project (e.g. measurable performance results, proof of concept validation, etc.)

16

Project Engagement

The creation of ‘rapid elicitation sessions’ (RES) prior to testing mock-ups yielded tremendous insights and improved the mock-ups that were going to be tested.

Schedule short, focussed engagement sessions with a small number of participants (6-10, ideally 8) to explore ideas and concepts and improve mock-ups prior to undertaking more expensive moderated usability testing.

17

Business Requirements

Over 1500 items of feedback resulting from the engagement sessions were analyzed, aligned to a feedback architecture, and prioritized. This feedback was used to further refine requirements and design in a rapid manner.

Get feedback directly from users and then take the time required to analyze the feedback. Then make sure the project team uses those results!

1

Project Expectation

Ensure understanding of the project’s outputs throughout the project.

NDSi was based on a design thinking approach to requirements definition and validation. This was one of the first times the PSC undertook such an approach. From the onset of the project, an interactive prototype was developed for the purpose of actively exploring the feasibility of NDSi concepts and ideas and to capture performance measurement results. Clarity about its intended end use based on interpretation of project outputs resulted in some temporary confusion and some redirection of focus for a brief period of time.

The project team, including the project executive, would have benefitted from training on the approach and methods that would be used from the outset of the project. Build time to train the team into the project schedule for future similar projects.

The intended use of project outputs should be clearly described and agreed to at the beginning of any project. If disagreement or confusion about the intended use arises, this should be addressed rapidly via established governance.

Project outputs should be aligned with PSC standards to ensure ease of adoption, understanding, and minimize re-work.

2

Project Methodology

Ensure that the velocity of the sprint cycle is maintainable.

Based on the exploratory nature of this project, an iterative and repeatable process was implemented to ensure that the project team could explore all of the tasks identified as within scope for NDSi in the time allotted and to ensure a high level of consistency throughout the various phases of the project. Direct user engagement resulted in new information entering into the project regularly. This process could have been evaluated with more frequency to ensure scope, as defined by users, was being respected and task completion quality wasn’t being impacted and that time was being allocated accordingly.

Hold retrospectives at the end of each ‘sprint’; honestly evaluate what was learned and what was accomplished in the time allotted and, if required, propose changes to the approved scope and/or sprint cycle via established governance.

Stay consistent with the methodology and avoid reverting back to traditional approaches to project management.

Honour resource commitments; NDSi’s scope (i.e. area of focus) grew based on user feedback, but the project team did not have the resources required to deal with user-identified requirements.

3

Project Outputs

Establish a clearer understanding of project output ownership, purpose, and the roles and responsibilities of sub-working groups to contribute to their development.

Sustained communication regarding the project outputs would have increased the awareness of project sub-working members. Similarly a more frequent review of the roles and responsibilities of other sub-working groups may have improved inter-working group collaboration.

Future projects should ensure that the project team has a common understanding of the final outputs, the methodology that will lead to their successful completion, and the vocabulary that will be used to describe those outputs. In order to accomplish this, the project team should have an onboarding plan regarding the chosen project methodology and outputs.

• Take the time at the beginning of the project to get the core (and preferably extended) project team together to ensure there is agreement and understanding of what the project outputs are.
• Have the subject matter experts or sub-working group leads responsible for their delivery educate the rest of the team about the output. Introduce the artifact, its use, the associated vocabulary, and any other relevant information so that the entire team can start the project in alignment.

4

Project Outputs

Ensure outputs are aligned with the project’s vision.

Empowered sub-working groups created project outputs independently and often presented them to the core team for review in advanced states. If they were not aligned to the project vision, there was a requirement for re-work.

Present early versions of project outputs to the core project team to ensure alignment with the project vision, identify early course correction, or escalation.

Have the product owner or a strong proxy present at project output research or creation sessions, when feasible, to minimize this issue.

5

Project Outputs

Document ownership and control.

Many key project outputs were worked on collaboratively by team members. New versions or copies would be created to ensure that work was completed by the required due date. If multiple team members had multiple copies, this could cause confusion about which was the authoritative document and which version was the most up to date.

The technical capacity to amalgamate multiple versions of a document into an authoritative version was beneficial to NDSi.

The creation and maintenance of key project outputs should be limited; single versions should be created with edit access granted to only key personnel in order to reduce errors, improve consistency, and enable the provision of real-time accurate information as required.

6

Project Methodology

The culture shift required to implement empowered sub-working groups was underestimated and the organization was not sufficiently equipped to support the roles and responsibilities required in this context.

Once the project was underway, certain sub-working groups were not fully resourced and/or did not align with the realities of the project which led to confusion about roles and responsibilities.

Sub-working group lead roles and responsibilities must be clearly articulated and understood at the beginning of the project.

Each sub-working group needs to establish clear directions and objectives at the beginning of the project.

Sub-working groups should be evaluated at regular intervals throughout a project to ensure that they are aligned with the project vision, sufficiently resourced, delivering upon expected outcomes, and non-duplicative.

Project priorities should be respected. Once the charter was signed and resources were officially committed to the project, these resources should have been made available to the sub-working groups. Branch priorities occasionally superseded project priorities and resources were not made available as committed to in the charter.

7

Feedback

Consistent and thorough documentation of feedback from elicitation sessions.

The NDSi team gathered user feedback from engagement sessions.  Rapid note taking led to feedback transcription that was occasionally incomplete which required extra revision.

The use of a dedicated ‘scribe’ at elicitation sessions was incredibly valuable. Ideally, this individual will have a background in UX or business analysis in order to bring an additional element of expertise to the role.

If the project team requires an attendee to elaborate upon their feedback, ask them to elaborate then and there. It’s better to ensure that the user was heard, understood, and recorded accurately than to have incomplete notes that take time and effort to interpret and categorize, yet may not be an accurate reflection of the user’s feedback.  

8

Feedback

Efficient and effective categorization of user feedback.

The NDSi project team experimented with the feedback analysis and prioritization matrix at the outset of the project. It took several iterations of the matrix prior to identifying the version that would be best used to analyse the feedback.

As well, sub-working group ‘ownership’ of the feedback was not clearly established which led to confusion about who was responsible for establishing the analysis process.

Policy advice related to feedback also took longer than expected to obtain.

Establish feedback collection and management guidelines and an associated categorization structure at the beginning of the project that includes identification of roles, responsibilities, the different types of feedback to capture, and their purpose. That said, don’t take too much time establishing this information; your team will not likely start with the right matrix; plan to iterate and evolve your matrix over time – the quality will improve.

Plan for enough time to analyze user feedback.

Analyse the feedback early and often and keep the analysis simple. Aligning the feedback to a simple feedback architecture will ensure that feedback is addressed in a timely manner and that all team members understand the analysis criteria. If the feedback architecture is deemed insufficient as the project progresses, it can be elaborated upon as required and prior feedback can be reviewed.
Determine at the onset of the project who owns the feedback and have the owner(s) of the feedback provide training or information sessions to all members of the team to ensure that there is a common understanding of how the feedback management process will work.

Establish a structure at the outset, within the policy team, the guidance sub-working group and the project team, setting out clear parameters around points such as:

• Management of policy advice – for example, logging of informal and formal advice;
• Identification of when formal interpretations are needed;
• Formal policy consideration documentation as required.

If further analysis is required, it should be undertaken systematically and result in a formal policy brief or policy recommendation

9

Project Engagement

Improve administration, communication, and engagement with users.

The creation and maintenance of the user engagement participant list required more time and resources than originally projected.

Ensure that the outreach team is sufficiently equipped to manage the master participants list and schedule the engagement sessions. This should include technology set-up and technology testing.
 
Leveraging technology to provide ‘stock’ project information to interested participants reduces the associated administrative burden. The NDSi team developed and posted information (including project FAQs) to GCPedia; this efficiently and effectively answered the majority of participants’ questions.

Keep the project’s list of participants informed and updated about the project’s status. These participants are primary spokespeople for the project within their organizations.

10

Project Engagement

Use ‘good enough’ presentation materials to engage with users.

In some cases, the search for perfection in material development resulted in some inefficient use of time. This was particularly notable during the stages of early engagement with users.

The most important part of this project was engaging with users; the designs and presentation material do not have to be perfect. Get actual users in a room, discuss their needs, show them what the project has, gather feedback, adjust the project’s concepts and the associated designs, and get back in front of them to collect more feedback.

11

User Engagement

Plan to let users talk about information beyond the planed scope of an engagement session

When users attended NDSi elicitation sessions, they often wanted to explore elements beyond the targeted scope of the session. It was a challenge to achieve the goal of the session in the time period allotted.

The fifth

When engaging directly with users, have a clear goal. Draft an agenda that clearly identifies the goal of the meeting, the expected outputs, and a schedule to accomplish this.

Establish a ‘parking lot’ or other mechanism to capture ideas about items that are out of scope.

Make the sessions longer! Plan for more time than required to discuss ‘out of scope’ issues with participants at the end of the session.

Having the product owner or a strong proxy present at the engagement sessions helped ensure alignment with the project vision.