Developing Simulations

Situational Exercises: An Added Degree of Realism

This guide provides practical information on the development of situational exercises simulations. It deals with the selection of the content of the simulation exercise, the presentation of problems to candidates and the evaluation of candidates' performance.

Situational exercises or simulations place candidates in realistic situations which require them to demonstrate job-relevant qualification(s). The content of the simulation exercise may be derived directly from actual work settings or may be designed in a more general manner by presenting situations and problems which are not specific to a particular work situation.

A notable characteristic of situational exercises is that candidates must demonstrate, at the time of the selection process, actual performance of the job and job tasks.

Situational exercises may be used to assess simple as well as complex qualifications. For example, simple qualifications, such as the ability to operate a given piece of machinery, are readily assessed by requiring the candidates to operate the machinery. More complex qualifications, such as the ability to solve administrative problems, can be assessed by presenting candidates with a realistic administrative scenario which requires them to solve relevant problems.

Situational exercises may also be used to assess more than one qualification at the same time. For example, a situational exercise designed to assess the ability to solve administrative problems could require candidates to make a presentation of their proposed solutions to a given administrative problem. This would allow for the assessment of their oral communication skills in addition to their ability to solve administrative problems.

Developing Situational Exercises: 3 Basic Steps

The information derived from situational exercises must be relevant to the qualification(s) being assessed. In following the 3 basic steps outlined below, you ensure that the situational exercise developed will provide you with reliable and valid assessment information.

Step #1

Identify relevant job situations.

Step #2

Developing the content of the simulation.

Step #3

Assess candidates' performance.

The ability to provide professional advice in a context of consultation services will be used to illustrate these 3 basic steps.

Step #1: Identify relevant job situations

Get the information you will need to develop the simulation exercise.

As a first step in the development of simulations, it is best to start by identifying job situations that are relevant to the qualification(s) that are to be assessed. From this information, the content of the simulation exercise will be developed.

From qualifications to relevant job situations

For each qualification to be assessed, ask yourself:

1. How do incumbents demonstrate that they possess some degree of this qualification? What do incumbents who are strong on this qualification do? What do incumbents who are weak on this qualification do? Are there different aspects or components to effective performance?
2. What kind of problems elicit the required qualifications? In what job situations is the qualification demonstrated? What aspects of the situation are essential, crucial, or important? What is the difference between easy situations and difficult ones?
3. What is the impact of the different actions that could be taken in the various situations? What is the difference between effective and ineffective actions?

An example

Let's consider the ability to provide professional advice in a context of consultation services. In asking the questions above, the following information could be elicited:

• Consultants who are good in providing professional advice in this position

1. ask relevant questions,
2. listen well to their clients' concerns and
3. are persuasive in presenting their ideas.

• An important situation is the face-to-face meeting with the clients; and that these meetings are not usually one-on-one but with groups of clients.
• Client satisfaction is the most important result of effective consultation and professional advice. Clients feel that their concerns were understood and taken into account in the proposed solution; they find it easy to support the proposed solution.

The above information provides relevant job situations which may be used to develop the content of a situational exercise.

Step #2: Developing the content of the simulation

At this step, the context of the simulation is identified and the items to be included are chosen or developed. As well, decisions are made about how the simulation exercise is to be presented to the candidates, and what the candidates will be required to do.

How similar to the actual job situations must the items and context of the simulation be?

The items and the context of the simulation can be taken directly from the job without any modification. Alternatively, scenarios and contents can appear quite different on the surface from those found on the job but still tap the same qualification(s).

The essential requirement with respect to the simulation exercise is that it provides an opportunity to observe the required qualification(s) in action. This requirement is more important than the similarity of the assessment material to the actual job content. The following points may be useful in deciding how similar the scenarios and the content should be to the actual situations.

1. When you are interested in generic abilities, the simulation content should not be too similar to that found on the job. Consider, for example, the assessment of problem-solving ability. If the problems presented to the candidates are too similar to that found on the job, it would not be clear whether a successful approach would be attributable to problem-solving ability or familiarity with how such problems have been dealt with in the past. As a general rule, the more generic the qualification the less similar the scenario and content needs to be to that found on the job.

"Double jeopardy": A problem to avoid

Double jeopardy refers to the situation where the demonstration of a qualification you wish to assess depends on another qualification. In such a situation, it is impossible to assess one qualification independently from the other qualification. For instance, when a situational exercise relies heavily on technical knowledge, performance on other qualifications (e.g., abilities) which are not necessarily related to the technical knowledge may be adversely affected. One solution is to assess knowledge qualifications using another assessment method and to develop the simulation in such a way as to minimize the influence of technical knowledge (e.g., providing candidates with background information containing the knowledge required for handling the simulation exercise; placing candidates in a neutral scenario which does not require technical knowledge; etc.).

2. When some candidates may be familiar with duties and responsibilities of the position or with current management thinking on a problem, the content of the simulation exercise should not be too similar to that found on the job. Otherwise, this kind of "insider information" might give an unfair advantage to some candidates.

Whether or not the items and the context of the simulation are similar to that found on the job, they should be of the same degree of complexity or difficulty as those found on the job.

In what types of formats can the items of the simulation be presented?

The formats which can be used for the purposes of situational exercises are virtually endless; but usually they fall into one of the three following types:

Written: Letters, memoranda, reports, program proposals, calendars of events, fiscal forecasts, variance reports, briefing notes, workplans, organizational charts, etc.

Apparatus: computer terminals, engines, tools, etc.

Audio-visual: Videotaped discussions, films, etc.

Each type of format has its own advantages. Written materials, for instance, are quite versatile, easy to use, and easily reproducible. Other types of formats are less versatile but more appropriate than written materials for certain qualifications.

When the simulation involves interacting with people, actors can be used to elicit the desired skill (e.g. interviewing, negotiating, selling, etc.) to ensure all candidates are presented with equivalent challenges.

Standardize your exercise

All candidates should be provided with the same initial information, instructions and timeframes.

When exercises are "interactive," as in the assessment of interviewing or negotiating skills, it is appropriate to react in a natural way to the behaviour of the candidate. In these circumstances, a "flexible script" may be of use. For example, if the candidates are expected to seek out information by asking questions, then the information given to candidates would depend on the questions asked. Candidates who ask the right kind of questions will get more information. However, for the same question, all candidates would get the same answer. The answers that would be given to the different questions constitute a "flexible script." Some additional suggestions are given below:

1. Write down your instructions. This makes it easier to verify that your instructions are clear and that nothing was forgotten. It also ensures that all candidates are given the same instructions. Ensure that candidates know all they need to know about their task: how much time they have; what they must produce, if it is a written text - how many pages; if it is a presentation - to whom and for how long; will they be interacting with the board members? with other candidates?; etc.
2. Some documentation may be given ahead of time. For more involved simulations, candidates may be given a package of information to study beforehand. For instance, such a package could be mailed to them a week before they are called in for the exercise.
3. Set appropriate time limits. Unless the speed of work is the aspect being assessed, time limits should allow enough time for most candidates to complete the task.
4. Inform the candidates about the qualifications which will be assessed and the criteria which will be used to evaluate their performance (e.g., speed, accuracy, quality, precision, number of units, style, or a combination of the above).

The information provided should be useful to the candidate without compromising the assessment tool.

Returning to our example

To assess the ability to provide professional advice in a context of consultation services, candidates could be provided with background documentation on a policy issue. The problem need not involve specialized or technical knowledge. Candidates could be given the simulation background documentation one week ahead of time to review and prepare for the next step. The next step could involve two 1-hour meetings. In the first meeting, scheduled at 9:00 am, the candidate could meet the board to ask questions and gather further information related to the policy issue. In the second meeting, scheduled at 1:00 pm, the candidate could present to the board his or her proposed solution and attempt to get an agreement on a plan of action.

Step #3: Assess candidates' performance

Define what you're looking for ahead of time.

In looking at the job situations relevant to the qualification(s) being assessed (step 1), you have already identified the impact of the different actions and the difference between effective and ineffective actions. This information will be useful in assessing the performance of candidates.

It is often the case that qualifications are broken down into assessment criteria. For example, on the basis of our analysis of the position, it may be decided to assess:

1. the quality of the questions asked,
2. the extent to which the candidates listen to the client's concerns, and
3. their persuasiveness in presenting their ideas.

Rating scales

Rating scales are useful in assessing the candidate's performance in situational exercises. Whether a numerical or narrative rating scheme is used, it is recommended that each rating be described as clearly as possible, and in advance of the actual assessment. One useful method is to list behaviours that would be expected to earn a given rating. These behaviours are sometimes called "benchmarks.". Candidates should always be assessed against these established benchmarks rather than against each other.

These benchmarks are not necessarily specific, as often there is no single best course of action. In such cases, a list of the characteristics of an effective action could be developed. For example, consider the assessment of the quality of the questions asked by a candidate. The benchmark description for a rating of "excellent," or "5 out of 5", would be "all of the important questions asked, relatively few irrelevant questions asked, questions are clear and to the point."

Multiple assessors

The assessment of candidate performance in situational exercises usually involves integrative judgements. It is precisely in these circumstances that the combined judgement of multiple assessors is best. The ratings of the assessors should be integrated by consensus. Each assessor should rate each candidate's performance independently. The ratings of the different assessors are then compared. Discrepancies are discussed and the reasons for the divergent ratings are given. Afterwards, a second set of ratings are made, and consensus is attempted again. The process is repeated until consensus is reached.

As presented in this guide, simulations are useful assessment exercises. They are valid and reliable assessment tools. The advantage of such assessment is that it is easily developed by managers and staffing officers. Although it is more time consuming than some traditional approaches to assessment, it provides the selection board with an opportunity to observe the candidate perform the qualifications required for the position being staffed in a very realistic manner.

For guidance and advice on developing situational exercises, please contact the Consultation Services:

Telephone: 819-420-8671

Fax: 819-420-8594

E-mail : CFP.CPP-PPC.PSC@cfp-psc.gc.ca