Book (2022)/Cohen Method

The Cohen method is a very elegant compromise method for determining the cut-off score, combining criterion and relative judgment. It assumes that there are always a few relatively good students in the group of test subjects, who have essentially comparable performance across the groups. They are not the very best, whose variability can be considerable, but the “second best” – excellent students whose results are in the 95th percentile of the given group. It turns out that the scores of these excellent students are a very good and reliable measure of the difficulty of the test. Thus, the first step of the method according to Cohen is to determine the difficulty of the test more or less by methods of relative standardization – by ranking students according to their results and determining how many points correspond to the 95th percentile (i.e. above what point gain the top 5% of students placed).

In the second step, we determine the cutoff score that must be achieved to pass the test. Cohen and Van der Vleuten tracked students' performance on various tests for nine years and proposed that this cutoff score be 60% of the score achieved by students in the 95th percentile. A correction for guessing is then added to the calculation, so its final form is:

PM = 0.6 · (P – C) + C

PM threshold score (passing mark)

P is the score achieved in the 95th percentile (i.e., how many points above the top 5% of students scored)

C is the score that can be achieved by guessing

Example: The test contained 30 “single best answer” items. Each question could be scored 0 or 1, for a total of 0 to 30 points. Each question offered 5 options. It was therefore possible to get a fifth of the points by guessing, i.e. 30 ÷ 5 = 6 points. 80 students took the test. The top 5% of them, i.e. the top four students, scored 27, 26, 26 and 24 points. The score achieved at the 95th percentile is therefore 24 points. Using Cohen's method, we will propose a PM cut-off score:

PM = 0.6 · (24 – 6) + 6 = 16.8

A student who has scored 16 points still does not pass. A student who scored 17 or more points passed the test successfully.

Fig. 5.5.2 Finding the cut-off score according to the simplified Cohen method (i.e. without correction for guessing).
On the histogram showing the distribution of test scores, the first dashed line (right) shows the score values ​​of outstanding students who placed in the 95th percentile. The middle dashed line shows the cut-off for passing the test, which is at 60% of the performance of excellent students. The last dashed line (left) shows the value of the score that can be achieved by simple guessing.

The Cohen method was validated on a large set of tests and it turned out to have very good results and excellent agreement, especially with the Angoff method. It overcomes the disadvantages of widely used criterion and relative cutoff scoring methods. Its advantages are simplicity, speed and low cost. A disadvantage may be that the point limit for passing the test cannot be announced in advance, but only after the test has been given and scored. This can cause doubts both among students and especially among people responsible for teaching, even though the algorithm by which the cut-off score is determined is clearly and unambiguously announced in advance.