Copyright © 1998, Nanette Miner
Permission is granted to reproduce or excerpt this article if proper credit is given in this format:
Reprinted/excerpted with permission. The Training Doctor. Author: Nanette Miner. www.trainingdr.com

Level 2 Evaluation Primer

By Nanette Miner, EdD

Did you know that improperly constructed test questions can result in legal cases? It's true. If someone is denied a raise or promotion due to flunking a test you created, you may have to defend the validity of your test questions.

To paraphrase Odin Westgaard, the guru of test creation:

Your test should be easy for those who understand what you've taught, difficult for those who don't understand, and a total mystery to the uninitiated.

The number one rule in evaluating and testing is DO NOT TRICK THEM. If you have not taught it in the training, it should not be on the test. Likewise, you should state your test questions in the same manner that you stated/taught them in the class. For example: if you teach the three characteristics of steel, don't ask in your test question: Which one of these is NOT a characteristic of steel - it's just tricky and mean. Also, for some reason it's hard for most people to have success with "null" answers.

The number two rule is: stick to the facts. Do not include trivial information - the only intention of which is to confuse the test taker. For instance: Bob and Ed left their office on "K" Street in Washington DC at 4:45 pm to travel to BWI airport for a 9:00 pm flight - how far is the airport from their office? The times given have nothing to do with the answer you are seeking - in fact, we don't even have to know that Bob and Ed are leaving, do we?

Here are a number of other "miscellaneous" rules:

• " The more important it is that an item be done correctly - the more test questions should be created for it. You'll want to know - backwards and forwards - whether the trainee knows his stuff.
• " Do NOT provide new information in a test question. The test taker is in a search and retrieve mode - he can't be asked to deal with new information.
• " Use the key words: who, what, where, when, how or why to begin questions. They assist the test-taker in figuring out the answer.
  Who triggers the respondent to look for a person or position
  What triggers the test taker to look for a thing or a process
  Where triggers them to look for a place or location
  When will trigger them to look for a date or a period of time
  and Why will signal them to look for reasoning
• Never, never, never ask a question that requires a subjective (opinion) response. For instance:
  How important is it that the cabin doors are secured before an aircraft takes off??
  __ Very important
  __ Important
  __ Unimportant
  __ Not at all important
  
  What would be the purpose of asking such a question? Instead, phrase the question as:
  In order to ensure cabin pressure upon take-off, the cabin doors should be:
  __ secured
  __ locked
  __ secured and locked
• If you are providing a "scale" of choices, make sure the scales start and end at distinct points. For example:

YES	NO
A baby should be fed every: __ 2-3 hours __ 4-5 hours __ 6-7 hours	A baby should be fed every: __ 2-3 hours __ 3-4 hours __ 4-5 hours

If the correct answer is every 3 hours - then there are technically two correct answers in the "NO" example- which response does the test taker check off?

INSTRUCTIONS

Instructions are critical. You must do everything you can to make sure test takers know WHAT to do, and WHEN and HOW to do it. Examples:

• If there is a time requirement, state it (you must finish this section in 30 minutes).
• If a tool or resource is allowed, state it (you may use a calculator for questions 11 - 20); the opposite is true as well - you may NOT use a calculator to complete this section.
• For each item in column A there is ONLY ONE correct answer in column B.
• If you do not know an answer, it is better to guess than to leave it blank.

It's also quite helpful to read the instructions out loud at the start of the test - even when they are clearly written on the test. It ensures everyone hears, sees, and interprets the directions the same way and allows you to ask for questions before anyone begins.

SCORING OR WEIGHTING

It is my personal preference that all test questions have the same value. It makes it a LOT easier to grade the test in the end. However, if you feel that certain areas of information are more critical to on-the-job success than others, then you may choose to weight those items more heavily.

Let's look at the different types of evaluation items and go over some hints and "rules" for each.

MULTIPLE CHOICE:

Typically, when someone is not skilled at writing multiple choice questions, the correct answer will always be the longest answer - try to be aware of this and keep your answers to the same length.

It's MUCH easier to provide 3 alternatives than 4; for some reason the 4th one is unbelievably hard to come up with.

Each choice must be logically consistent. You cannot have:
Check off all answers that are primary colors:
__ Red
__ Blue
__ White
__ Pony

By the same token, the right answer shouldn't jump out from the list either. You'll want all choices to be somewhat feasible so that the test-taker has to put some critical thought in to their answer.

Your list of possible choices don't have to be terribly difficult - remember you are testing people who have new knowledge and limited skill/experience - don't come up with answers that would stump you, come up with answers that would stump the unskilled person.

If your answer begins with a vowel, the preceding question should end with a(n); if it ends with an - which is grammatically correct - the question itself automatically triggers the trainee to choose the answer that begins with the vowel. For example;
The Interstate is also known as an:
__ Expressway
__ Highway
__ Route
Do you see how ending the question with an forces you to choose Expressway?

Parallel structure: all your choices should begin with similar phrasing/sentence structure; for example:

YES	NO
The time to cross the street is: A. When the light turns red B. After you have looked both ways C. If the road is clear	The best time for fly fishing is: A. Morning B. Afternoon C. April and May, after the ice pack has melted

If you're going to provide "none of the above" (or "all of the above") as one of your choices, you should include it in ALL of your choices. If you have 15 multiple choice questions and only 1 or 2 have the option of "none of the above," it's a clear signal that the right answer is "none of the above."

TRUE / FALSE

In true/false questions, make sure your options are entirely true or entirely false - do not include answers that might be true under certain conditions.

Avoid what are known as "specific determiners" which signal the correct answer - most obvious are ALWAYS and NEVER. Rarely is anything always true or never true. The use of either of these words is usually a clue to the test-taker to choose the other option. (For instance: "i" always comes before "e.")

We tend to write more true statements than false ones, because it's easier; so be aware of that and try to create an equal number of each.

MATCHING

Matching questions are somewhat easier to create because you only have to come up with one right answer for the "B" column (as opposed to multiple choice which requires you to come up with 3+ "right" answers).

Never provide more than 10 matching questions; if you DO want to provide more, break them up into chunks of 10 (perhaps that all deal with the same concept). More than 10 really bogs down the process for the test taker. Pretty soon all the letters and numbers are just swimming in front of their eyes.

You may choose to provide more "B" column answers than "A" column questions. This makes the matching process a bit more difficult for the test taker. Do not provide more than three extra "B" column options, however.

Alternatively, you may have less options in the "B" column and more questions in the "A" column. In this case you would instruct the test-taker to use the "B" column responses as many times as appropriate.

For instance:
In column "A" identify what type of store layout each named store utilizes. You may use the answers in column "B" more than once.

A	B
1. ____ Home Depot 2. ____ CVS 3. ____ JC Penney 4. ____ Sears 5. ____ Blockbuster 6. ____ Shop Rite / Publix	A. Grid B. Racetrack C. Boutique

If you're going to create a one-of-a-kind question, be sure to provide a logical alternative, even if it doesn't fit any other possibility. For instance, A compass always points _______, should have the options of at least North and South in the "B" column. If only North is supplied, and you only have one question in the "A" column that has to do with direction, the correct answer is obvious to the test taker.

OBSERVATIONS

You may be creating observation sheets that will go back to the job site and be administered by someone else there. When creating observation sheets:

Include instructions for the observer - what should be the setting? Should the trainee be told in advance they will be observed? How long should the observation last? Should the observer give comments/feedback during the performance or wait until the end? Should the trainee be telling the observer what he is doing, or just do it (e.g. next, I will check the depth of…)?

Your observation sheet will more than likely be a checkoff list with yes/no possibilities. The hardest part is breaking the activity down into all of its components. Let's analyze the task of changing a car's oil. You will have to decide how detailed you want the checkoff item to be - is it enough to say "locate the oil cap" or should you start with "open the hood?" The deciding factor will be how important it is to have the interim steps done correctly or in a specific order. If my ultimate goal is to make a salad - it doesn't really matter which order I cut the vegetables in, so long as they all get in the salad. Likewise I might chop the lettuce or hand-tear it - there is no "correct" method as it all ends up making a salad in the end. But if I'm changing a car's oil it IS important to remove the old oil before putting the new in. If I am to relieve the pressure on a gasket ONLY if it reaches a certain point, it is important that I correctly read the gauge, first.

ORDERING THE QUESTIONS
It is really up to you to decide if you want to group your evaluation questions by topic or to mix them up. Personally, I like to mix things up because back on the job the work they encounter won't show up in any kind of logical sequence - so I like for them to have to think "out of sequence" on the exam as well. The benefit to keeping your questions grouped is that you'll easily spot if a trainee just "didn't get" a particular topic. If all the questions on a similar topic are grouped together and the trainee gets all or almost all of them wrong - it's apparent he needs retraining in this area. If your same-topic-area questions are interspersed throughout the exam, it might be harder for you to spot a problem.

USE OF RESOURCES
By resources I mean calculators, policy and procedure manuals, job aids, etc. - the rule of thumb here is, if they are allowed to use it on the job, they should be allowed to use it in the evaluation as well. Why make them memorize something they will never need to commit to memory (other than to pass your test)?

Nanette Miner, EdD, is President of The Training Doctor, LLC, a CT-based firm which specializes in the custom design of training to increase the performance of an organization's personnel assets. Reprinting of all or part of this article is prohibited without the express of the author. Please contact Dr. Miner at 1-800-282-5474 or nanette@trainingdr.com. Visit www.trainingdr.com for more information.