Book/Recommendations for Creating Test Items

Item writing is a item that requires imagination and creativity, but it also requires considerable work discipline and knowledge of learning objectives. The creation of items must be based on a clear idea of the learning objectives. A test should measure a single cognitive area.

Before you start creating items

Before it comes to the actual creation of items, it is necessary to return to the question of what we actually want to test and why. It is not enough just to take the textbook, flip through the chapters that cover the content of the upcoming test, and create items for the text that catches your eye.

Ideally, we have a prepared test plan (blueprint) at our disposal. If not, the test plan should be created before we start writing items. If even that is not possible for any reason, we should at least have the learning objectives written down in as much detail as possible (i.e., not just the thematic areas that the learning covers). It is advisable to add to the learning objectives an idea of how big a part of the test should deal with them.

A test plan or a detailed list of learning objectives gives us a clear indication of what items we need for the test and in what quantity. At the same time, already at this stage we should have a basic idea of the types of items that we will use in the test.

Selection of Item Types

When compiling a summative test, we will not go wrong if most of the items are single best answer items, and, if necessary, we supplement them with open-ended questions with a short form answer. We should use other types of items judiciously, or according to the customs of a specific field. Single best answer items provide the best value for money. The time required to answer them is relatively short, so that a sufficient number of such items can be included in the test, and at the same time they allow sufficiently sensitive recognition of students' skills. Items with a short form answer are more difficult to evaluate, but on the other hand, they are a suitable addition, as they provide better feedback to the teacher.

In formative tests, especially if the group of test takers is small, the ratio can be reversed – the majority of the test can be made up of short-answer questions supplemented by multiple-choice questions with a single best answer. In formative tests, there is no need to be afraid of other item formats, if their use is expedient. But the test should never combine too many different formats (no more than three or four), otherwise it will be confusing and students will spend a lot of time researching what is actually being asked of them and how they are supposed to answer which item.

If the test combines multiple item types, students should be clearly told what is expected of them. The instruction must be very specific.

Example:

Instruction for single best answer problems:

Appropriate: Circle the best answer.

Unclear: Choose the best answer.

Instruction for short formed answer problems:

Appropriate: Answer with a single word or phrase.

Recommendations for Creating Multiple Choice Items

Recommendations for the creation of closed-ended (multiple choice) items will be given here for items with a single best answer (SBA), i.e. the type of items that should be the basis of most tests. For the most part, the same recommendations can also be applied to other multiple choice formats.

Ambiguity is a frequently-occurring problem in multiple choice items. It mostly results from the author of the item having a specific situation in mind when compiling it, but then trying to write the item as succinctly as possible. This results in the loss of details and assumptions from the text, which are important for answering it. The student taking the test must first guess what the author of the test actually had in mind, and only then can they answer. The item usually measures the student's ability to guess what the teacher wanted to ask, rather than the actual knowledge and skills in the tested area.

Therefore, the basis of a good-quality multiple choice item is a well-written stem. Problems with a single correct answer tend to have a relatively long stem, several lines long (sometimes referred to as the statement). The stem should tell a story – describe a simple but real situation, or perhaps an experiment. A clear description of the situation the author had in mind will avoid most ambiguities. Moreover, students will find items compiled in this way to be motivating – stories that remind them of real experience, remind them that they are learning something practical that they will need in their future employment.

For novice test writers, writing statements is sometimes difficult. The basic recommendation is that they should write items as if they were designing research to answer specific (but of course simple) questions in their field.

The statement is followed by the actual question. That question should be short, unambiguous, and it must ask only one thing. In single-best-answer items, it must be clear from the question that the student is really being asked to mark only one option that is a better answer than all the others. This is especially important in the event that other options offered would make sense, at least under certain conditions. Properly constructed questions tend to take the form of, for example, “What is the most likely cause?” or “What is the most appropriate next course of action?” “What will the events described most likely lead to?”

Another simple principle follows from the above: Good quality items usually have a relatively long stem (text) followed by a short but clear question. This is followed by a menu of options, which should also be short.

Regarding the options offered, it is usually easy to create the correct answer (the so-called key). The rule here is that the expert should give the correct answer after reading the stem text and the question, even without an offer of options. Choosing from several options actually just makes it easier to evaluate the test and provides some help to the student who is not yet as proficient as an expert in the given area.

It is more difficult for teachers to suggest incorrect answers – distractors. The authors of the items may no longer fully remember what the problem was when they were learning the tested topic, and then propose distractors that are irrelevant for the students. It may therefore be advantageous to first give the item to students as an open-ended question, preferably in a formative test. Distractors are then created from incorrect responses. At the same time, the teacher gets an idea of how difficult the created item will be. If the teacher combines this kind of procedure with a discussion of the answers, or if the teacher invites the students to “think aloud” during the solution, it will provide the teacher with additional valuable information both for teaching and for further modifications to the test items.

More experienced test writers are often helped by a quintet recommendations of how an item with a single best answer should look. These five recommendations will also help reviewers during opposition proceedings for new test items.

1. Focus on an important problem.

   Higher education should prepare students for work in the real world, and therefore test items should relate to problems encountered in practice. Don't waste time with trivial or overly complex questions. There is no point in testing little essential, marginal knowledge – such items usually do not give any indication of readiness for practice, rather they test how well a student “knows how to take tests”. Do not use “trick questions”, and avoid negatively worded items. Test knowledge and understanding, not concentration.

2. Test for use of knowledge, not recall of a concept or an isolated fact.

   A longer text of the item requires the student to somehow evaluate the described situation and interpret it. In summative tests, avoid testing definitions and classifications – test for the student’s understanding of the content of the concepts and ability handle them, or whether the inclusion of a concept in a certain category is also connected with an understanding of the properties of that category. Whether this recommendation has been fulfilled can often be easily verified: copy the entire item stem into an Internet search engine – the correct answer should not appear in front of you.

3. It must be possible to answer the item even with the answer options hidden.

   Have your peers review the question. First, give it to them without the options provided – they should be able to answer it correctly. If not, rewrite the item.

4. Do not use relativistic nor absolute terms.

   The text of the stem or the anwers choices offered should not contain relativizing terms such as often, rarely, exceptionally, mostly, etc. The use of such words will make the item unclear, whereas the relativizing term conceals the author’s specific point of view, which the student may not guess correctly. For example, a certain situation may be rare from the point of view of the general population, but at the same time it is relatively common from the point of view of an expert who professionally deals with the solution of such situations.

   “Forbidden words”, especially in the multiple-choice answers, also include terms expressing the absolute, such as always, never, etc. Only rarely does something really apply 100%, so the use of such a word in one of the offered choices usually means that it is a distractor – and students can easily recognize these.

5. All answers offered must be homogeneous.

   All the options offered must appear similar – they should be written in a similar style and should be of a similar length. They must fall into a single category – for example, all the options offered are possible causes of an event, parts of speech, biological taxonomy, work activities, chemical substances, etc. It must be possible to rank the answers from best to worst.

Fig. 3.4.1 In the example on the left, the options offered for the question can be ranked from worst (option C) to best (option D). This confirms the homogeneity of the options offered. If you pose the same question to several experts, they will all rank the offered answers in the same order. On the other hand, in the case on the right, it is not possible to sort the offered options – each option falls into a different category, answers the question from a different aspect, so we would be “comparing apples with oranges” when trying to sort them.

An example of a question with non-homogeneous options

Select the best statement about Marfan syndrome:

A. It affects men more often

B. It is a disorder of the collagen ligament

C. It is treated with hyaluronic acid

D. It is often associated with oligophrenia

E. It is manifested by conspicuously short limbs

The options offered should be ordered randomly or alphabetically. If the options contain a numeric value, they should be sorted by that number.

Recommendations for Creating Open-ended Items

Follow these guidelines when creating short-answer questions for written tests:

• Formulate questions simply and clearly, avoid linguistic tricks and catches.

A good short answer question tests knowledge of specific facts or the ability to analyze and interpret a scenario. It is not appropriate to simultaneously, in the same item, test the student’s ability to understand a complexly constructed question – the assessment results would then be practically uninterpretable.

• Try to answer the question from different points of view.

A question that asks about one specific fact should have only one correct answer. Conversely, a question that asks about possible variants (e.g. differential diagnoses) will have several correct solutions. Keep in mind that even a question that seems obvious to you can be understood differently by different readers. It is always advisable to have the questions checked by a reviewer.

• State the length you expect the answer to be, e.g., Answer in one word or phrase., or Sketch the graph of a function.... Also write how the item will be graded.

  Use negatively worded questions with caution.

Positively worded questions ("What is the best course of action...", "What is the most likely cause...") have more didactic value than negative questions ("What is the wrong course of action"). If you are already using a negatively worded question, emphasize the negative, for example by using capitals ("Which antibiotic is NOT appropriate in this situation?").

• Make sure that the answer is not dictated by, for example, the size of the space for entering it.

  Prepare evaluator instructions carefully and have them checked by your colleagues along with the item.