Book/Item Bank

The item bank (or “test question bank”) is a repository of tests and test questions and their metadata.

Test item banks are closely related to the test cycle described above, as an item bank in a broader sense can include tools for creating, reviewing and managing items, for planning, creating and delivering tests, and finally for evaluating responses, including analyzing tests and test items. So, they can be used to support the entire test cycle.

Along with tests and questions, metadata is also stored, including psychometric characteristics of items from previous test runs, which can be used as feedback for improving further test runs. If the system also maintains information about the author of the item and the reviewers, it can provide them with feedback on how the item performed in the test, thereby contributing to their education.

Items are usually in a “multiple choice” format, but any format can be used. Items from the bank are used to construct tests distributed in both classic and electronic form.

David Vale described an item bank as “an organized collection of test items”. The simplest item bank may be a shoe box with cards containing test problems. When item statistics were available, they were often written on the back of the cards, along with the date the test took place. If the test writer wanted to create a new test, they manually searched the item box, checked the contents of the items and their statistics, and selected the ones to use in the test.

The difference between an item bank and a simple set of test items is that item banks allow you to track an item throughout its life cycle, in which the item can be used in a series of tests. Information about the behavior of the item in one cycle is stored and used to construct better tests in subsequent cycles. This “reusability” and the improvement of quality due to the evaluation of previous uses of items are among the basic features of item banks.

At the turn of the century, “reusable education objects” were a research topic. In the case of learning materials, the effort to store them in repositories and with metadata for further use did not succeed. Description using metadata was too laborious for busy educators, and its complexity suppressed the effect of savings from reuse. In the case of item banks, the situation is different. Working with metadata is useful and mostly fully automated. The idea of using “reusable education objects” is thus coming to fruition decades later, albeit in a different context than originally planned.

Item banks are an essential part of the quality assessment process. In addition to supporting the creation of test questions, they can do much more: store metadata about items, store psychometric properties of items in tests, track their usage, handle user management, security management, and enforce good workflows that help maintain quality standards. You don't need an item bank if you are doing a small number of tests that are of more of a formative nature. But you can't do without an item bank if you're preparing large-scale and important assessments of student performance.

Items and Their Metadata

Item banks contain not only the text of each item, but also a range of information regarding its origin, incorporation, use, and psychometric characteristics. Examples of such metadata include:

• Item text
• Creation date
• Correct answer
• Item format
• Assignment to topics
• The author of the item
• Item reviewers
• Reviewers' statements (for the Angoff Method)
• Review status (for review, done, rejected, for revision, ...)
• Item status (e.g. new, reviewed, active, archived, replaced by a new version, ...)
• Characteristics according to Classical Test Theory
• Characteristics according to Item Response Theory
• Item incorporation in test plans
• Relationships to other items

When building an item bank, it is necessary to consider the requirements of the specific field and already adapt item classification to it, for example. The item can be stored in the database as a whole, or in individual parts – as a stem (short text), the question itself and the offered answers. After this, it is easy to generate derived versions of the items (e.g., with different data for calculation), but it is more difficult to keep the information in order, and keep track of what form the item was used and in which test. The storage method must be chosen at the beginning, since a number of other properties of the item bank depend on it.

Types of Relationships Between Items on Tests

Various metadata is stored on items. This data includes relationships between items, describing whether and under what circumstances two items can be together on one test.

Companions	Items that must occur together because they rest on a common foundation or use common supporting material.
Close Friends	Closely related items must occur together. If one appears, the other must also be present.
Snobs	Items of the “Close Friends” type that can only be used in a certain order to make them understandable.
Dependent	Items that can only occur with the support of a “Supporter”
Supporters	Items with no separate interaction that provide context for subsequent “Dependent” items. For example, this can be a text that will be referred to in several items.
Antagonists	Items that must not appear close to each other in the test, because one provides a hint for the other.
Enemies	Items that cannot be in the same test because they ask the same thing.
Clone (offspring)	An item that was derived by changing (fixing, improving) the parent item.

Functionality of Item Banks

We typically expect an Item Bank to ensure and support:

• Access authentication
• Creating items
• Saving items
• Ordering items
• Item review (including review process management)
• Creation of test plans (blueprint)
• Item management
• Test administration
• Management of learning objectives for item categorization
• Print or take the test electronically
• Loading of results forms
• Automatic psychometric analyses
• Activity logging
• Records of item and test exports
• Data retention of psychometric data from previous item uses

The essential feature that distinguishes an item bank from an item repository is precisely that the item bank preserves psychometric characteristics from previous rounds of testing. It thus enables their use for the creation of better items and the construction of better tests.

As the reader must have noticed, automated test generation is not explicitly mentioned in the list of properties. It turns out that the design of the test requires the author to take into account a large number of often unexpressed item parameters and balance their representation in the test. So, even if the system can generate a draft of a set of test items, it is still assumed that this draft will go through a proofreading by the “author” of the test, who is responsible for its balance. The item bank offers the author a number of tools for this, for example, the ability to monitor how items cover the tested area (blueprinting), how many computational items are in the test, and so on.

Comprehensive item banks have other useful mechanisms built in, such as monitoring changes to dependent item parameters. After reworking the item, they can change the “review status” from the value “to be revised” to the value “to be reviewed” etc.

A note on the influence deciding on the necessity of forensic analyses of a test on the structure of the item bank. The item bank itself does not need to store information on the identity of the tested person. If the creators of the item bank consider it necessary in the given context, they can store information about year, gender and other attributes in the item bank, but there is usually no reason to store a specific identity. The situation changes when it becomes clear that there is a need to use statistical tools to check whether there is any illegal conduct during testing. In that case, it is necessary to work with the identity of the test takers. It is a conceptual question, because then we have to deal not only with the statistics of the test items, but also with the statistics of the examinees. When designing an item bank, it is advisable to remember this circumstance and take it into account in advance.

Advantages of Item Banks

An item bank should be at the core of any serious testing system. Its use brings a number of advantages:

1. It enables repeated creation of tests with predictable properties.
2. It provides an opportunity to objectively determine the specifics of individual authors of items. It may turn out that some item authors systematically prepare easier or, conversely, more difficult items. Some give preference to one thematic area, or one type of item (e.g. computing). Therefore if, for example, you need to supplement the test set with a specific item, you know who to contact.
3. Regularly working with authors allows you to train them and increase their skills in creating items.
4. The bank forces a standardized procedure for the preparation of test items (content review, language proofreading, typographic checks, determination of difficulty, post-test evaluation...), which is a guarantee of systematic quality improvement.
5. Order is maintained in different versions of items. When an item needs to be modified or corrected, either a new version of it or just a modified version is created depending on the extent of the change (e.g., if it's just an issue of correcting typos or modifying typography).
6. All items are assigned to specific topics and the system allows you to search them according to a number of criteria. This ensures better coverage of the test substance when creating tests, makes it easier to follow the test plan, avoids repetition and prevents problems with items that are of an unknown or unnecessary focus.
7. Item banks make it possible to assign the results of psychometric analysis of completed tests to items. It is thus possible to sort low-quality items, or to monitor whether an item was not leaked between two tests.
8. In the item bank, the permissions of individual users are set based on roles. At the same time, all activities are logged, especially changes to items, mass exports and access to finalized tests. All this contributes to increasing the security of the test.
9. The item bank should allow for easy identification of duplicate items and items that have some type of relationship (enemies, friends, close friends, ...)
10. An item bank increases testing efficiency and quality by viewing items as reusable objects and supporting the entire test development cycle.

Examples of Item Banks

An item bank is essentially a simple database, so it can be stored in a database system or even in a spreadsheet environment.

An example of an item bank created in Excel was presented at the Association for Educational Assessment – Europe 2016 conference. The solution was relatively simple and yet completely functional, and we can definitely recommend this step to those potentially interested in item banks. Even if it is only a temporary solution – it will help you clarify what your needs are and what you require from a possible future, more comprehensive, solution.

Most large companies dealing with testing have developed their own item banks. In the Czech Republic, for example, there is no doubt that SCIO or Cermat have some form of item banks. At some universities, parts of the item bank are integrated directly into the school's information system.

Item Bank of the First Faculty of Medicine of the Charles University

The First Faculty of Medicine of the Charles University has also developed its own separate item bank. It is a comprehensive item bank in the Multiple True/False item format. Similar to the Rogō test system, it is a web application that runs on all major browsers. The bank, whose development dates back to 2014, supports all steps of the test cycle, from blueprinting to item creation, review and version management. The bank enables the creation of tests, their review, print preparation, import of results, itemized analysis and reporting of test results. There are about 10 thousand items in the bank, including metadata about the results of use in previous rounds. Due to its use in important exams, the bank is heavily secured and every access to it is logged. The properties of this item bank are the basis of the general description of item banks given above. And conversely – this bank has all the properties required in the general model.

Many people interested in testing are looking for an economically acceptable way to purchase a commercial item bank. A number of options are available, but their licensing model is usually based on an environment where significantly more resources are allocated for these purposes, which makes these solutions practically unavailable for potential domestic customers.

Experience with TAO of Testing

At the boundaries of commercial products stands the “TAO of testing” system, which caught our attention with the availability of its free version. As it might also attract other potential interested parties so we consider it useful to share our experience.

TAO is, in Chinese philosophy, an expression for the basic principle of the universe, but also a French abbreviation for Testing Assisté par Ordinateur (computer-assisted testing). The TAO platform has been developed at the University of Luxembourg as an open source project. It provides participants in the computer-based testing process with a comprehensive set of features that support the creation, management, and administration of electronic tests. In particular, it covers:

• item development and management,
• management of the test takers,
• creating and managing tests,
• management of authors and reviewers,
• delivery of tests,
• results management.

TAO is an open and modular system based on the assumption that no one solution can fit all, so users are expected to adapt it to their needs. It is a web application that runs on a server and does not require anything to be installed on the user's computer. It supports translations into national languages. It offers authors a WYSIWYG editor for intuitive item creation, including multimedia integration.

Although the system is open source, it is only seemingly “free”. You have two options to use it. Either you use the provider's paid cloud installation, which costs the same as other item banks on the market, or you install TAO on your servers yourself. However, the documentation is insufficient and installation and updates are poorly described. The installation scripts contain errors and the manuals contain references to non-existent scripts and other resources. The developer does offer support, but it is expensive.

The system lacks support for directory services (LDAP), which makes it impossible to use its existing identity verification (and user name and password management) at a specific institution. This is very impractical when deployed in large institutions, which would have to maintain several directories for testing purposes. The TAO system also does not include test and item analysis tools (only export to QTI 2.2. or CSV formats), so you need to use third-party software for test and item analysis.

We implemented and experimentally ran this item bank, but problems with documentation and updates, as well as the need to handle student usernames and passwords separately outside of the existing directories (LDAP), led to the termination of this experiment.

Extensive Banks of Test Items

In some cases, the normal work with the item bank has exceeded the usual dimensions. Let us mention two such interesting cases. The most interesting of these collaborations is the British Medical Schools Council Assessment Alliance (MSA-AA), which unites all 31 medical schools in Great Britain.

Medical Schools Council Assessment Alliance

This alliance operates a common item bank for them. The common objective of all participants is to improve the evaluation of learning results at medical faculties.

The alliance follows the activities of the Association of Medical Faculties – Universities Medical Assessment Partnership (UMAP), which was founded in 2003 for the purpose of cooperation in the creation and sharing of test items. The association gradually expanded to include other schools and after 2009 was transformed into MSC-AA.

The common item bank mainly contains items in the single-best-answer (SBA) format, but items for OSCE stations and “multiple-mini-interviews” are also being added. The bank is accessible to all participating schools. Questions are created collaboratively and undergo extensive quality testing and standardization. All medical colleges in the UK have agreed to include an agreed proportion of shared questions in the final exams, thereby ensuring psychometrically valid comparability of the “state” exams.

Item Management System

In German-speaking countries, the Item Management System (IMS) item bank is widely used in higher education. It was established in 2006 as a result of the cooperation of the medical faculties of the universities in Heidelberg, Berlin and Munich. The group gradually grew to 77 institutions, mainly in Germany and Switzerland and is covered by the Umbrella Consortium for Assessment Networks (UCAN).

The Item Management System is a database of items that can cooperate with computer testing applications, paper or mobile testing, and in analyzing results and so on. As of March 2021, 700,000 items (without format resolution) were stored in the system, of which 125,000 were shared,.

The Umbrella Consortium for Assessment Networks (UCAN), which maintains the item bank, is declared as a non-profit organization, but outwardly it functions as a commercial entity. The license is calculated according to the number of test takers. Prices are set for Western European conditions.