Book/Test Security Plan

The test security plan describes the values that need to be protected, the known risks, and the procedures to reduce the risks.

A test security plan is a comprehensive set of policies, procedures, and documents that outline and govern actions related to test security. From the development of the test plan to the recapitulation of the results in the security audit, “security” applies to almost every step. The use of test scores to assess candidate performance presupposes confidence in the integrity and objectivity of the test. Without trust, even credibility would be compromised.

What needs to be done to make test scores reliable and interpretable? At a minimum, this requires having and following a reliable test security plan.

Most of the policies and procedures in a test security plan are based on common sense. For example, it is essential to have a communication channel for clear and unambiguous sending of messages to applicants. How else can participants be expected to follow the rules if those rules are not explained, along with the corresponding consequences? Adopted procedures must make sense, fit well with the given testing program, be enforceable, and be legally defensible. The proposed procedures should align with the threats specific to your programs and be tailored to specific needs. While in one case the main problem may be the leaking of items between exam dates, in another arrangement the greatest threat may be identity fraud. Threats vary from program to program, and security plans should address protection against these threats.

Roles, Items, Responsibilities

Preparing for important tests is a collective effort. Even from the point of view of security, credibility would be difficult to ensure if all powers were concentrated in the hands of a single person.

A role-based approach helps to limit these risks. All personnel collaborating on testing should have specified roles and work only within the scope of these roles. Someone may have the role of “item author”, someone the role of “item reviewer”, another “test author”, or “test administrator”. Role security limits will help ensure that, for example, whoever is in charge of managing the list of tested students may never see any test items.

Responsibilities that the security team must provide include, but are not limited to, the protection of internal information from disclosure. As part of the so-called “soft security”, we take care of this confidentiality by selecting responsible employees whose moral integrity indicates that they will not spread classified information unnecessarily or succumb to the temptation to provide this information to someone for a bribe. In the concept of “hard security” (for example, when we do not have enough information about the personnel involved), a test security agreement, sometimes referred to as a non-disclosure agreement (NDA), is used to protect classified information. It is usually a unilateral, legally binding contract between the institution developing the test (or content owner) and another party solving a component item. A non-disclosure agreement usually specifies what information or materials are considered confidential and/or proprietary, what the period of confidentiality is, and what the consequences are for violating the agreement.

Recurring activities include updating legal policies and procedures and training staff on test security. At a time when most tests are produced and stored in an electronic environment, the test team’s job is to secure the test data on local or cloud servers. Access to these servers must be limited to entrusted and vetted workers, and monitored and logged.

It is good practice to require that anyone who has access to test content or other proprietary information be trained and sign a non-disclosure agreement. This includes professionals involved in test development, staff who monitor test administration, staff who process test materials and results, teachers who receive or store test materials, etc. Nondisclosure agreements should be updated annually and kept on file after the period specified in the test security plan (usually at least three years).

Preparation and Administration of Tests

There are a number of security precautions that should be taken before the test itself is given. This includes not only the secure preparation of test content, but also the monitoring of websites and social media. The danger of leaking items is magnified by content sharing technologies. There are specialized sites that collect items for certifications and exams that they have captured from individuals, accumulate them by category, and then offer them to interested parties for a fee. These pages can be found under the keyword “brain dump”. Security preparation therefore assumes that the team preparing the test will monitor social networks, try targeted queries to web search engines in search of leaked items, and monitor blogs commenting on the given exam or certification in order to identify leaked items in time.

When illegal practices are suspected, a technique known as “mystery shopping” can be used. This type of exam security verification assumes that an agreed collaborator of the test team registers as a student to take the exam and reports on the security of the test from the examinee's point of view. This form of security monitoring is expensive, but in case of doubt it can provide very valuable and otherwise unavailable data.

Given the importance, attention must also be paid to the distribution of sensitive materials and access to them. The test plan should therefore describe the procedures for how protected materials are distributed, collected and archived and who has access to them. Viewing and interventions in sensitive materials (e.g. in the wording of sharp tests) must either be recorded by technical means (logging, camera recordings) or carried out by committee (at least two people) and a record should be made of the action. An unchecked risk are individual accesses and interventions for which there is no retrospective evidence.

Finally, we need to address the issue of training. Everyone involved in the test cycle should be trained in test security. Training may cover a variety of topics, including, but not limited to, proper handling of test materials, establishing or maintaining a secure testing environment, critical aspects of a confidentiality agreement, examiner rights and responsibilities, and acceptable test supervision practices. Training may also include “what if” scenarios. Training should be tailored to align with the roles of different team members, including subject matter experts, supervisors, test administrators and coordinators, content developers, psychometricians and management. Third-party workers who collaborate on testing should also complete the training. Ensuring testing security requires the cooperation of the entire team.

Test Day Policy

Another important aspect of security is the so-called test day policy. Is the testing environment secure? Are supervisors sufficiently trained in test security? What are the registration requirements? How do participants identify themselves? How many forms of identification are needed? Is the course of the test recorded by a camera system? Is there a predetermined seating order for the test takers in the room? Is there a safe place to store personal items such as cell phones and study materials? Are calculators allowed? Is a response form supplied? If so, is the form individualized? Are the forms collected at the end of the test? Are screen protectors used for computer monitors? Are workstations separated? Are breaks or restroom breaks allowed during the test?

Communication with examinees starts well in advance of the exam date and continues until the moment the results are announced. The rules must be clearly established and distributed to interested parties and stakeholders. In addition, the consequences of breaking the rules must be clearly defined and announced. Before testing, examiners may be required to confirm that they have read, understood and agreed to follow the required rules.

For regular testing of great importance, we cannot do without some form of comprehensive test system (item bank). This, of course, brings a new kind of risk, because valuable information (the wording of items, but also the wording of prepared live tests) is concentrated here in its final form in one place for a long time, which increases the risk of unwanted exposure. Important security measures related to item banks include the technical provision of permanent logging of risk events, especially associated with test exports, or displaying a larger number of test items, or even entire tests.

Security of results

Another component of security are the procedures for the storage and distribution of sensitive materials (e.g. test assignment) and the retention of test results. This procedure determines how protected materials are distributed, collected and archived. The names and functions of the persons responsible for carrying out these procedures are also stored. Data and signatures from each person involved in testing and supervision are collected and archived as part of the test history. In general, when working with sensitive data, access must either be logged, or done under the checking of two pairs eyes, and logs must be kept of the actions performed.

Rapid Analysis of Test Data

Rapid analysis of test data is an extremely valuable tool for detecting signs of irregularities in tests that have just been completed (but not yet scored). It allows, for example, to detect possible ambiguities in the wording of items, errors in the key determining the correct answers, and the like, even before classifying students. Suspicious items, e.g. with very high or low difficulty, or with very low discriminating ability, are subjected to a content check, and in case of errors, ambiguities or inaccuracies, such an item is excluded from scoring, or the key for its scoring is modified. Authors and reviewers are also notified of problematic items to correct before further use. Similarly, a quick analysis can also pick up some non-standard patterns of behavior pointing to potential security issues.

Evaluation of the Course of the Test Round

To ensure the credibility of tests, the testing organization should have a procedure for reporting incidents and irregularities in test administration and security. Test takers, teaching supervisors, and other testing personnel should be aware of the mechanism for reporting incidents, anomalies, or potential rule violations. The form offered should range from an anonymous notification to a formal message.

Incident Response

The test security plan should set out how incidents will be logged, processed and investigated. It should be clear under what circumstances the achieved score will be invalidated and when any sanctions will be applied.

Test Security Audit

During the security audit of the test, the security team recapitulates the preventive measures that were taken, their effectiveness, the threats that were noted, how they were resolved, and what adjustments to the security rules need to be made before administering subsequent tests.