Book/Testing Software

There are dozens of tools to support formative and summative testing. New ones are constantly being created, some are losing their importance and are being abandoned. The existing ones are part of intensive development, especially through shifting towards mobile platforms. In this chapter, focused on test tools, we will not attempt a comprehensive overview of available products, as the static format of this publication does not allow us to capture dynamic developments, but will instead limit ourselves to conveying our experience with those tools that have worked for us, or on the contrary, have not.

Fig. 10.1.1 Infographic of coverage of the test cycle by Rogō, Moolde and Remark Office tools.

Rogō

upravit Among the tools for electronic testing, the Rogō program occupies a prominent place. The extraordinary position of this system lies in the fact that it is a high-quality, secure and easy-to-use test tool that is also freely distributable.

Advantages (quality)

• Each item has its own record of changes and use in tests.
• Invited external collaborators have easy and secure access to reviews.
• During remote testing, it is possible to set the end of the test after the time allowance has been exhausted.
• The items used in the test will be locked.
• Items cannot be added or removed from a test that is currently in progress.
• Possibility to set the time limit of the test.
• Student responses can be analyzed and item properties can be easily evaluated.
• The grading can be adjusted according to the performance of the cohort after the end of the test using the Hofstee method.
• Modified Angoff and Ebel methods are also available for setting test scores.

Security

• Only HTTPS protocol is used for connection. Data is encrypted using 256-bit SSL.
• Seamlessly and securely share materials across staff teams to work together on assessments.
• Customizable checks and weights to ensure fair grading for all users.
• They can use established local authentication systems for student and teacher access.
• In the event of an Internet connection failure, only the answer on the currently open test page is lost, not the entire test.
• Access to the test can be limited to selected IP addresses.

Applicability

• It can run on both Windows and Linux servers.
• LDAP compatibility – no need to create additional credentials for users.
• Support for language mutations.
• Tailored help systems - separate help for users by role.
• Rogō is a web application that runs on all major browsers - Chrome, Edge, Firefox, Safari, Internet Explorer.
• Adepts with special needs can adjust both the appearance of the test and the time allowance.

License

As far as license is concerned, the Rogō online testing web application is a freely distributable open source program, released under the GPL version 3.0. It is therefore possible to change the code, extend it and thus contribute to the project. In practice, it works in such a way that requests for code modifications, whether related to reporting problems or proposals for new functionality, are written into the request queue and are gradually addressed by the community.

History

The Rogō test system has been developed since 2003 at the University of Nottingham Medical School under the name “TouchStone”. The Rogō system originated at the medical school and is very well adapted especially for learning medicine. Among other things, it allows the use of interactive images in items, on which the student marks the desired object with the mouse. The system then evaluates whether the student marked the desired structure with the required accuracy.

After its success in its home faculty, it was expanded to the entire university, converted to open source software, and on that occasion renamed to avoid confusion with other systems. Rogō means “I ask” in Latin. The Rogō development community received financial support from JISC, which enabled further development of the system, including the ability to translate into national languages. In the Czech Republic, the Rogō system is installed (on the servers of the First Faculty of Medicine of the Charles University) at the address https://www.rogo.cz/ and, in addition to the First Faculty of Medicine of the Charles University, it also serves other faculties of the Charles University. The First Faculty of Medicine of the Charles University has prepared a Czech translation of the environment and is continuously working on the localization of the help section. Thanks to LDAP support, students are automatically imported into the system from SIS (Study Information System of the Charles University) and can authenticate with their CAS account (Central Authentication System of the Charles University).

Specific Properties

Unlike other programs, Rogō covers and supports many steps of the test development cycle, from collaborating on the preparation of test items, to challenging them in terms of difficulty and relevance, creating a test plan, standardization, and evaluating the quality of questions. This kind of comprehensive solution brings significant advantages. For example, it is advisable to invite a number of in-house and external experts to supervise test questions, which is usually time- and organizationally demanding. At the same time, if item proposals circulate among a large number of people, it is very difficult to ensure their secrecy. In Rogō, on the other hand, opponents are prompted to join the system, so the test items never leave the system. Again, comments and suggestions are entered directly into Rogō and item authors can respond to them immediately. After the test has taken place, it is possible to display the descriptive characteristics, a histogram of the total scores of all students, or the difficulty of the items. Rogō automatically calculates discrimination indices for each item of the test, which makes it possible to identify poorly constructed items and exclude them from further use. From the point of view of applying modern procedures in testing, the system is completely unique and its introduction supports the extension of good test practices into the field.

The system makes it possible to distribute both paper and online tests, both for self-assessment and for secure summative testing.

Rogō includes tools to automatically import students and courses they have enrolled into the system. Thanks to the support of the LDAP directory service, which is used by the entire Charles University, students can log in directly with their CAS account and are assigned to all their courses in Rogō.

The system allows educators to create a many types of tests and surveys:

• Formative assessment
• Summative assessment
• Progress tests
• Surveys (questionnaires)
• e-OSCE (clinical trial)
• Offline tests
• Peer assessment (student)

Each of these types of tests and surveys can use a variety of item forms:

• Area delimitation
• Dichotomous items
• Multiple choice questions
• Extended Matching
• Multiple true false
• Complete the text
• Mark points in the image
• Likert scale
• Scenario matching test
• Text fields

Advantages

Table. X.X Advantages of the Rogō system

Low cost of acquisition Support for the entire testing process Support for teamwork High level of security A large selection of types of test items, including multimedia ones

Disadvantages

Table X.X Disadvantages of the Rogō System

A smaller community maintaining and developing the system Need for local support and administration Not quite intuitive operation Time consuming when learning the program

Moodle

upravit The Moodle learning management system is a globally widespread online learning environment. More than 250 million students use this open-source platform. Moodle is probably the most widespread system for learning management at universities today. It was created in 2002 and is continuously updated. With its open source code, security and privacy, it is an attractive solution for many colleges and universities. A large and active community collaborates on the development of Moodle. Services that go beyond the capabilities of individual administrators are offered by specialized companies with Moodle partner qualifications. These are mainly services such as hosting, customization, support, training, or even comprehensive management of entire projects in Moodle.

Advantages

• Price
• Extension
• Community of developers
• Flexibility thanks to many modules
• Integration options
• Mobile and PC interfaces
• LDAP support

Disadvantages

• Unclear arrangement
• Chaotic user interface
• Too many modules
• Absence of leadership and unified concept
• Server load when multiple users work simultaneously
• Upgrades may invalidate previous work
• From the point of view of lean modern mobile applications, this is a behemoth

The main advantage of Moodle is its extensions and flexibility. On the other hand, it is necessary to keep in mind that with a large number of simultaneous accesses (a typical situation for testing), the system may become overwhelmed. The solution lies in the appropriate dimensioning of the infrastructure, e.g. distributing the load across multiple servers.

Advantages for testing

LMS Moodle is not specifically focused on testing, but it still provides some interesting options. If you use it for proctored testing, it is possible to use the Safe Exam Browser. An Ada Quiz module is available for adaptive testing. The adaptive quiz guides the student through an itinerary of questions adapted to his knowledge.

• Multiple item types
• Item mixing
• Test timeout
• Automatic grading
• Localization
• Good security

Disadvantages for testing

From the point of view of item banking, it is a shame that Moodle does not collect information on items from previous uses. This would allow Moodle to be used more in the role of an item bank. It is strange that the statistics in test analyses have their own nomenclature, so the teacher sometimes has to experimentally find out what the Moodle authors meant by which designation.

• Little support for teamwork
• Necessity of teacher training
• Many other functionalities outside of testing
• Unintuitive interface
• Laboriousness of test preparation

However, despite these caveats, Moodle is a widely-used tool, excellent for testing. Test items can be prepared outside the Moodle environment, or they can be taken from the item bank. Moodle supports QTI interoperability standards and a range of import formats.

Remark Office

upravit Testing large groups of students is often done using “pencil and paper”. The advantage is independence from technical means and very good provability in the event of a dispute. However, the evaluation of submitted response forms can be a bottleneck when using this technology. Especially when it comes to the case of a large number of test takers and a high importance of the test, it is necessary to ensure that the process is fast, as far as possible error-free, demonstrable and reproducible. Grading using a translucency does not achieve these parameters. Therefore, optical mark recognition (OMR) programs are used.

One such program is Remark Office OMR. This program is used to recognize scanned answer forms, questionnaires or tests and convert them into electronic form.

The program allows you to load data directly from the scanner or from saved files. Collected data can be exported to various data formats or processed directly. The program also enables the creation of reports on the collected data. A list of available statistics and types of reports can be found on the manufacturer's website (www.gravic.com).

A blank form must first be loaded – a template on which places for optical reading are marked. The template defines the variable type, variable name and description, etc. The software recognizes a blacked-out circle, an empty circle, and even a blacked out but later crossed out circle. It can handle incomplete answers, smudged and damaged forms, multiple marks and other anomalies. Remark reads barcodes and can recognize text (OCR). For handwritten text, it can save the field as a graphic file for further processing.

If the automation encounters a case where it doesn't know what to do, it will provide the operator with a zoomed-in view of the relevant part of the form and ask for help. It is very user-friendly and unrecognized cases are minimal. With a slight exaggeration, the company coined the slogan: If you can read it, we can read it too.

The software is very useful not only for evaluating tests and surveys, but for example for academic elections and other occasions when a large number of paper forms need to be processed.

A permanent license costs approx. CZK 30,000 and the price annual support is around 20% of this price. Considering the functionality, the price does not seem excessive. Fixing a license to a specific computer may seem impractical.

Socrative

Socrative (https://www.socrative.com) is a tool for online tests, quizzes and surveys. Originally, it was mainly used for frontal teaching, in which it replaced the previously used “voting buttons”. Socrative allows you to create multiple choice quizzes (true/false, single best answer and multiple true-false) as well as short answer questions. Students answer using mobile phones, and the teacher can run the quiz from a computer or from a mobile phone.

The advantage of Socrative is a well-crafted interface for teachers, which on the one hand allows for a relatively large variability of tests and quizzes and their easy adaptation to what is needed at a given moment, but on the other hand is simple and clear. Teachers are well versed in it, and working with Socrative does not unduly distract attention from the presentation or communication with students.

After the quiz has been completed, items can be scored automatically, the teacher can adjust the assessment of items with a short answer or score such items completely manually. Visualizations of how the students answered are then available, which again can be easily used in further work with students. It is also possible to download several forms of reports, from a summary of answers for each student to tables with results for teachers.

In addition to complete quizzes, it is possible to ask students individual questions. An interesting tool is available for constructed response questions where the teacher can first have students write their answers and then in a second step have them vote on which answer is best. Other features of Socrative include a feedback questionnaire at the end of the lesson.

The basic version of Socrative is free. Quizzes can be used repeatedly, new versions can be created and saved in different folders. The paid version allows you to create multiple “rooms”, which is useful if one teacher uses the tool for several different courses and has a large number of quizzes in it.

Kahoot!

Kahoot! is an application for creating quizzes, which is mainly used in lower levels of education. It provides a playful, stimulating environment that allows you to create engaging competitions. Again, students answer questions using a mobile phone. In addition to individual quizzes, Kahoot! and competitions between teams. The quizzes you create can be shared and you can find many different competitions, tests and quizzes online in Kahoot!, covering a wide range of topics.

Mentimeter

Another tool in which the teacher asks the students questions and lets them answer using their mobile phone or laptop is Mentimeter. Unlike the previous tools, it is not suitable for testing, nor is it possible to conveniently monitor which answer was given by which student. Mentimeter is particularly useful for asking attitude questions and for stimulating discussion on a certain topic. It contains a number of options to visualize the answers. The world cloud is most often used. Listeners are invited to respond with individual words or phrases (it is possible to set the number of responses that one student can submit). The result is displayed as a “cloud”, in which the word or phrase is larger the more often it occurs in the answers. Surveys using different types of scales, such as Likert scales, are also used, the results of which are again clearly visualized.

The free version has a limited number of questions that can be used in one presentation. However, a larger number of presentations can be created, already created surveys can be used repeatedly, survey results can be downloaded and further worked with.