Book/Test Items (Questions)

Items, or questions, are the basic building block of every test. We will deliberately avoid the frequently used term question, which, as we will see below, can also have a different, narrower meaning.

In general, skills can be assessed directly or indirectly. During direct examination, the student is given the task of directly performing a certain activity. Direct testing often uses practical testing techniques (workplace-based assessment), but written tests can also include some direct questions.

Examples:

• Create your structured resume in English.
• In the R programming language, create a function that...

Direct items make it possible to assess the extent to which the candidate has achieved the target competencies and how prepared they are, for example, for the performance of a certain work activity. Their disadvantage is difficult grading. The candidate's performance must be evaluated by several evaluators, and usually several areas or aspects of the performance are evaluated. Evaluators must be trained in advance, and they use a structured evaluation form for assessment.

More common are indirect testing methods. The student's skills are not tested directly but are assessed on the basis of knowledge and skills that are a prerequisite for a particular ability.

A typical written test tries to assess the achievement of target competencies indirectly. The authors of the test create a certain construct of which knowledge and skills are essential for the achievement of competence. By testing for these, they try to estimate whether the examinee could achieve the target competence. From this perspective, we are not examining whether the examinee can actually do a certain thing, but whether he or she has the prerequisites to do it.

It is not possible using indirect testing methods alone to reliably decide whether a candidate is capable of independently performing a certain activity, and it is not possible to completely replace direct methods with these. However, these methods are much faster, easier to evaluate, cheaper, and can more readily achieve reproducibility.

There are basically two types of indirect test items:

1. Open-ended Items

   The examinee must create an answer – write a text, perform a calculation, draw a picture, etc. The answer can have a range of formats and lengths – on the one hand, it can be about filling in the missing letter, on the other about writing a multi-page essay. Since the examinee creates their answer, these items can also be characterized as productive.

1. Closed-ended Items

   The examinee chooses a solution from a finite range of options that have been offered to him. He or she does not create the solution – their item is only to select and mark it. Most often, they choose between several possible answers to a certain question. However, this also includes other types of items in which the examinee chooses from several predefined alternatives, e.g. they have to match related concepts to each other, arrange items in a certain order, fill in i or y in the missing places in the text, or decide whether in a quantity will decrease, increase or remain unchanged in a certain situation.

The line between open-ended and closed-ended items can sometimes be blurred. In some cases, the examinee must choose an answer from a practically unlimited range of possibilities, without it being classified as a productive item. An example could be items in which the student has to mark (i.e. select) a certain place on a photograph of a microscopic preparation.

Both open-ended and closed-ended items have their irreplaceable place in testing and in university learning, while each is suitable for something different:

Open-ended items make it possible to evaluate more complex skills, especially skills of a productive, creative nature (Schindler 2006). To formulate an answer, students are forced to actively use professional terminology. It is often possible to follow the thought process that led the test taker to the solution. During the evaluation, it is possible to recognize how well the test takers understood the assignment and whether the item is not poorly formulated. Open-ended items are an invaluable tool for continuous formative testing during learning, which is primarily intended to provide feedback to students and teachers. They provide feedback more effectively than multiple choice items and are well suited for use as a starting point for discussion of the topics covered. Open-ended items can also be part of the final summative tests, in which they are used to verify the achievement of not only component knowledge and understanding of the learned facts, but also the use and incorporation of this knowledge in more complex items. However, the preparation of open-ended items for summative tests is demanding, as is their evaluation.

Open-end items cannot be graded automatically – the answers must be assessed by qualified evaluators. It may thus happen that the assessment is plagued by subjective error. Each open-ended item is successively evaluated by several mutually independent evaluators, who receive them anonymized if possible. Very detailed instructions are prepared in advance for the evaluators. Nevertheless, examinees may challenge the objectivity of the test and it may be more difficult to justify the evaluators' decisions beyond doubt. Therefore, if open-ended items are to be used in a test of fundamental importance, these items and the rules according to which they are scored need to be prepared very carefully. Open-ended items are often evaluated on a wider scale than just “correct/incorrect” and all evaluators must equally assign partial points for partially correct solutions. In general, the more open-ended the item, the more difficult it is to ensure its objective assessment. It is also necessary to have procedures in place in the event that the evaluators' opinions on a specific solution differ. Therefore, the preparation of open-ended items tends to be time consuming and its difficulty increases with the importance of the test.

Open-ended items can place students with communication impairments at a disadvantage, since the formulation of an answer often affects the evaluation.

Closed-ended items include multiple choice, pairing and arranging items (Schindler 2006). They are easier for processing, tests can often be evaluated automatically by computer, or possibly they can even be graded by a less qualified worker. In most situations, closed-ended items are the fastest and most effective tool to find out how much knowledge the student has acquired and how well they have understood the subject matter. To a limited degree, these items can be used to evaluate even the mastering some simple skills.

The big advantage of closed-ended items is that it is easy to decide whether the examinee responded correctly. As a result, the test evaluation is reproducible. Scoring the test is also very fast. The answers do not depend on the formulation skills of the examinee, their graphomotor skills, their typing speed, etc. On the other hand, closed-ended items do not allow testing of many types of skills. Closed-ended items place students who are less attentive or function less accurately under stress at a disadvantage.