Paper-and-pencil instruments: An efficient method of assessment

An efficient method of assessment

Paper-and-pencil instruments refer to a general group of assessment tools in which candidates read questions and respond in writing. This includes tests, such as knowledge and ability tests, and inventories, such as personality and interest inventories.

Paper-and-pencil tests can be used to assess job-related knowledge and ability or skill qualifications. The possible range of qualifications which can be assessed using paper-and-pencil tests is quite broad. For example, such tests can assess anything from knowledge of office procedures to knowledge of federal legislation, and from the ability to follow directions to the ability to solve numerical problems.

Because many candidates can be assessed at the same time with a paper-and-pencil test, such tests are an efficient method of assessment.

Developing paper-and-pencil tests: 4 basic steps.

All assessment methods must provide information that is relevant to the qualification(s) being assessed. The following 4 steps ensure that paper-and-pencil tests provide this information.

Step #1

Listing topic areas/tasks

Step #2

Specifying the response format, number of questions, the time limit and difficulty level

Step #3

Writing the questions and developing the scoring guide

Step #4

Reviewing the questions and scoring guide

Step #1

Listing topic areas/tasks

For each knowledge/ability qualification that will be assessed by the test, list the topic areas/tasks to be covered. Check off any critical topic areas/tasks that are particularly important to the job.

For example, the topic areas that will be covered for the qualification «knowledge of office procedures» might be knowledge of correspondence, knowledge of filing and knowledge of making travel arrangements. Or, for example, the tasks to be assessed for the qualification «ability to solve numerical problems» might be the ability to add, subtract, multiply and divide.

Step #2

Specifying the response format, number of questions, the time limit and difficulty level

Prior to writing the questions for your test, you should decide on such things as the response format, the number of questions, the time limit and the difficulty level.

What type of response format should I choose?

The three most common response formats are:

• (a) multiple-choice (see Example 1);
• (b) short answer (see Example 2); and
• (c) essay.

(a) Multiple-choice

With a multiple-choice response format, a large number of different topic areas/tasks can be covered within the same test and the questions are easy to score. However, because all potential answers must be chosen by some candidates, it is time-consuming to write good questions.

With a short-answer response format, as in multiple choice, a large number of different topic areas/tasks can be covered within the same test and these questions are easy to score. In addition, less time is required to write these questions compared to multiple-choice ones.

(b) Short-answer

With a short-answer response format, a large number of different topic areas/tasks can be covered within the same test and these questions are easy to score. In addition, less time ie required to write these questions compared to multiple-choice ones.

(c) Essay

With an essay response format, only a few topic areas/tasks can be covered due to the amount of time it takes to answer questions; however, the content can be covered in greater detail. Essay questions require little time to write but they are very time-consuming to score.

Although at first glance a multiple-choice format may seem a relatively easy and logical choice if breadth of coverage is emphasized, don't be fooled. It is hard to write good multiple-choice questions and you should only choose this type of response format if you are willing to devote a lot of time to editing, reviewing, and revising the questions. If depth of coverage is emphasized, use an essay response format.

How many questions do I need?

The number of questions you need depends on the breadth and depth of coverage required and the importance of each topic area/task. Generally, the more important a topic area/task, the more questions you should have. You should initially write several questions from each topic area or task so that you can choose the best ones for the final version of the test.

With a multiple-choice response format, 30 questions are often sufficient. If you are using a short-answer or essay format, fewer questions will be required.

How much time should I allow?

Unless speed of work is being assessed, the time limit should be set to allow the majority of candidates to finish within the allotted time. A general guide for multiple-choice questions is to allow about a minute per question. The time needed for short-answer and essay questions depends on the complexity of the questions. If you are in doubt, try your test and time limit out in advance.

How difficult should the questions be?

You should consider both the number of possible candidates and the level of job performance required for the position. The questions should have a range of difficulty levels. You should try to make all questions differentiate between poorly-qualified and well-qualified candidates.

Step #3

Writing the questions and developing the scoring guide.

All questions should tap meaningful information. Also, the level of language used for the questions should be appropriate for the requirements of the position. The questions do not have to always be expressed verbally. Diagrams, graphs, or tables may be incorporated into a question where useful.

When you are writing each question, you should prepare the answers, designate the marks to be allotted to each item, and decide on the rules for scoring. This ensures that there is a clear-cut answer for each question. It also allows you to indicate the value of each question so that candidates can decide for themselves the amount of time they should spend on each question. The marks assigned to each question should reflect the relative importance of the question.

(a) Multiple-choice

The scoring guide for multiple-choice questions must include a scoring key indicating the correct answer and it may also include a rationale for or explanation of the correct answer. If marks are to be deducted for guessing, this must be determined and stated in the instructions to candidates.

(b) Short-answer

The scoring guide for short-answer questions should include predetermined scoring procedures and mark allocations. Each required point in the answer should be listed with its relative mark allocation.

(c) Essay

The scoring guide for essay questions should include predetermined scoring procedures and mark allocations. The major points of the answer should be listed with their relative mark allocation. If marks are to be deducted for incorrect grammar, spelling and punctuation, this must be stated in the instructions to candidates.

The scoring guides for short-answer and essay questions should be clear enough so that scorers can judge whether or not marks should be given to a variation of the answer.

Step #4

Reviewing questions and scoring guide.

Have the questions and scoring guide reviewed by individuals familiar with the position in question. Typically, this would involve other managers and/or job incumbents. Have them ensure that each question is answerable, relevant to the job, technically and grammatically correct and clear and understandable. Any symbols, words, phrases or content found to be offensive, ambiguous, inappropriate, or misleading should be revised or replaced. The reviewers should also assess the appropriateness of the time limit and difficulty level of the questions. It may be helpful to have the reviewers actually take the test under testing conditions. You should only include in the final version of the paper-and-pencil test those questions that your reviewers endorsed.

The development of your own paper-and-pencil tests can be difficult and time consuming. However, if these 4 steps are followed, the end result should be a quality assessment instrument.

Example 1:

Below is an example of a multiple-choice question and scoring guide (with accompanying rationale) designed to assess the qualification, ability to solve numerical problems.

Question:

1. You are responsible for purchasing equipment and furniture. How many filing cabinets can you purchase with a budget of $6,000, if 50 filing cabinets cost $10,000? Choose the correct answer from among the four choices provided. No marks will be deducted for guessing. (Please note that the use of calculators is not permitted.)

1. 15
2. 20
3. 25
4. 30

Scoring guide rationale:

d.30

$10,000/50=$6,000/X
X=$300,000/10,000
X=30

Example 2:

Below is an example of a short answer question and scoring guide designed to assess the qualification which is called "knowledge concerning immigration".

Question:

1. What are 5 major factors that are considered in a request to immigrate to Canada? (5 marks)

• circumstances in home country (i.e., is persecution likely?)
• presence of support in Canada (i.e., relatives)
• skills and experiences that are required in or advantageous to Canada
• health
• financial status

Scoring guide: 1 point each