Book (2022)/Angoff Method

The Angoff method, or its modification pursuant to Hambleton and Plake, is based on the concept of a minimally competent candidate. This means a model candidate whose knowledge and skills are just on the lower edge of the permissible minimum. In other words, he or she is the weakest student that should still pass the test.

The test items are assessed by a group of 4-20 experts who have an idea of ​​both the topic and the actual competences of the students, i.e. most often teachers of the given field. As part of the preparatory meeting, the group should be trained in the methodology and familiarized with the required performance standards corresponding to the curriculum, in order to unify the idea of ​​the required competencies. In the next step, the panelists go through one item after another of the test, separately, and write down their estimate on the prepared forms with what probability the least competent candidate should answer it correctly. It is recommended that the first few items be assessed together for training purposes. The experts' results are then entered into a common table. If the estimates for an item differ by more than a pre-agreed maximum allowable deviation between estimates (usually 15%), such a question is discussed with the whole group and consensus, i.e. agreement on the assessment, is sought. Items for which consensus cannot be reached are dropped from the test, as differing expert opinions usually indicate a problem with the item itself.

The required percentage score for passing the test is then determined as the average of the probabilities of successfully answering all the questions on the test. The advantage of this procedure is its objectivity and independence from personal preferences. The disadvantage is the time-consuming and professionally demanding nature of the procedure.

In order to set the correct cutoff for passing the test, it is essential in the Angoff method that experts must know the target group well and be able to estimate how difficult specific items will be for that group. Despite the fact that the Angoff method appears to work in practice, some authors debate whether the idea of ​​a minimally competent student is a sufficient anchor for setting the standard. Some argue that it might be more appropriate to consider what is important to achieve (rather than how difficult it is to achieve) and what all candidates should achieve (rather than just what a group of successful candidates would achieve) in determining the required performance standards. The suspicion arises, therefore, that experts will be more likely to imagine an average examinee and speculate how such an examinee would pass the test, rather than looking for a minimally acceptable level of competence in relation to the desired learning objectives.

Using the Angoff method

Test items are judged by a group of experts, and for each item each expert estimates what percentage of minimally competent students would answer the given question correctly. Experts work independently so as not to influence each other. The results are entered in a table in which the rows represent the items from the test and the columns contain the estimates of individual experts.

Table 5.4.1 Table of expert estimates of the probability of a minimally competent student answering the question correctly.

Item number	Expert 1	Expert 2	Expert 3	Expert 4	Expert 5	Expert 6	Expert 7	Average
1	0,70	0,70	0,65	0,65	0,80	0,60	0,70	0,69
2	0,50	0,50	0,60	0,60	0,55	0,50	0,60	0,55
3	0,80	0,75	0,70	0,70	0,70	0,80	0,70	0,74
4	0,70	0,60	0,70	0,70	0,75	0,60	0,60	0,66
...	...	...	...	...	...	...	...	...
Average								0,66, tj. 66 %

Note: If there were to be TRUE/FALSE questions on the test, the lowest possible estimate of success would of course be 0.5, because even a student who does not know the answer has a 50% chance of answering correctly. Analogously, for a multiple-choice question with five options and only one correct answer, the minimum will be 0.2.

After filling in the table, it is usually assessed whether the experts agreed in their estimates. Items where the estimate variance is greater than a pre-agreed percentage (typically 15%) should be discussed; often, this reveals ambiguous wording or another problem.

Finally, the average of all estimates in the table is calculated. This average tells what percentage of the total possible number of points a competent student should achieve. In other words, this average indicates the threshold of success for the given test - i.e. the line between “passed” and “failed”. The success threshold divides the set of test-takers into successful and unsuccessful.

Fig. 5.4.1 The success limit determined using the Angoff method divides the set of test takers into successful and unsuccessful (pass/fail). This method is also supported by some testing programs, for example Rogō (which will be discussed in a separate chapter), from which you can receive this very chart.

Support for Angoff's method is included not only in the Rogō test program, but is also offered free (for registration) including instructions on YouTube by Assessment Systems, a manufacturer of programs for testing and test analysis,.

Conditions for using the Angoff method

For the successful use of the Angoff method, it is necessary that the participating experts have sufficient experience in the given field and agree fairly precisely on the idea of what the students must be able to do in the given course. Experts must therefore be able to imagine what at least a competent student can do, or should be able to do.