Book/Test Analysis Software

Test analysis and its reporting are important steps in the testing process. During item and test analysis, we can find out how our test behaves as a whole and what the properties of individual items are. Thanks to this feedback, we can correct and improve the test for the next rounds. There are dozens of commercial tools for analyzing tests and items. There are significantly fewer freely available tools. Some analytical tool modules can be included directly in testing programs (Rogō) or learning management systems (Moodle), but for a really thorough analysis, you need to use specialized tools or a statistical environment with relevant libraries.

Specialized commercial programs tend to be user-friendly, very sophisticated and relatively expensive. Freely available non-commercial solutions usually have a high difficulty threshold.

Since we assume that our readers hail mostly from the academic sphere, where high quality analysis is required and mental capacity is more readily available than financial resources, we will start with an excellent analytical tool – the statistical software R. Those who do not have a problem with entering commands from the command line can use one of the many packages in CRAN's “R library” focused on the area of “Psychometric Models and Methods”. For those who prefer a more user-friendly solution, there is the ShinyItemAnalysis web application derived from the package of the same name in the R library.

ShinyItemAnalysis

The ShinyItemAnalysis application, freely available on the web by Patrícia Martinková and her colleagues, was originally created for the analysis of university entrance tests. It also now offers a wide range of other analyzes in the field of didactic and psychological measurements. It allows you to perform test and item analyses including graphic outputs (distractor analysis, two-color DD graphs, ...). You can use the pre-loaded data to test the analyses on sample data. Many methods are available, though the associated help is relatively terse. However, if you know what you need, it's not that much of a problem.

Uploading data to the system is not completely limitless. If you have a data format error, you won't get a message about the reason for the failure. You can spend a lot of time this way before you discover the problem, but don't be discouraged, it will be better the second time around. Otherwise, it's a really unique tool. You can find it at http://www.shinyitemanalysis.org/

jMetric

jMetrik is free and open source psychometric software. It was developed by J. Patrick Meyer at the University of Virginia. Psychometric methods offered include classical item analysis, reliability estimation, test scaling, differential item functioning, item response theory, Rasch models, and more. There is comprehensive help for the program. However, according to some authors, jMetrik is somewhat cumbersome. A separate IRT illustrator module is now available that allows you to plot various Item Response Theory (IRT) functions. Both jMetrik and IRT illustrator are pure Java applications, working on all operating systems that have a current version of Java. Readers can find more information and the software itself at https://itemanalysis.com/

We would like to mention three of the commercially available analysis tools – Lertap, Iteman and Xcalibre, which we had the opportunity to try.

Lertap

Based in Australia, psychometrician Larry Nelson has developed a number of test analysis programs. The latest in this series, LERTAP5, is a comprehensive software package for test analysis, using Microsoft Excel. It computes analysis of test results, items, including graphic outputs. It offers cheating detection tools. Also, Lertap5 is more oriented towards classical test theory (CTT) methods, it also offers basic Rasch analyses for dichotomous test items. The calculations are not the fastest, which is due to the Excel environment in which they take place. The free version includes all the features but will not process more than 250 data records. A perpetual license (bound to a computer) costs $78, placing this product on the borderline between commercial and non-commercial tools.

Iteman

Iteman is an interesting commercial software program for analyzing items and tests using classical test theory (CTT). It is unique in that it produces comprehensive and professionally processed reports in Microsoft Word format on the quality of the test items, on the test as a whole and on its psychometric properties, including embedded graphics and tables. A description of one of the older versions is provided by Byčkovský. Iteman is now available (in version 4) either as a cloud version or as an application for Windows. The cloud version allows you to use the software anywhere. The version for Windows, on the other hand, allows you to store all (potentially sensitive) data on one computer. A non-commercial (academic) license for Iteman costs $1,295 per year. The trial version is limited to 100 students and 100 items. The reader can find more information, a description of the current version and licensing terms on the developer's website http://www.assess.com/.

Xcalibre

Xcalibre 4, from the same manufacturer, is a powerful test analysis tool based on item response theory (IRT). The program has a very user-friendly interface. It provides professional reports summarizing analysis results, including embedded tables and graphs. It allows you to analyze large data sets, perform comparative studies, or refine items using distractor analysis. Control is point-and-click, you don't need to write any code. The non-commercial version of Xcalibre costs $1,495 per year.