The relationship between test performance and test mode is an important one to consider, especially in situations of high-stakes testing. This study investigated relationships between test mode, gender, and computer familiarity and test performance, on a practice Graduate Record Exam (GRE). The practice GRE contained three sections (Analytical, Quantitative, and Verbal). Self-reported GPA and SAT scores were employed as covariates. There were three levels of test mode: paper-and-pencil, computerized with editorial control, and computerized without editorial control. Both computer forms were non-adaptive, and included three minutes more than the paper-and-pencil form for each timed section. There were three levels in the computer familiarity variable (lower, moderate, higher), determined by examinee responses to a survey. The survey also included items that asked students to provide feedback on the test experience. The test was administered to 222 undergraduate students who were stratified by gender, then randomly assigned to a test mode group. MANCOVA revealed that examinees in the paper-and-pencil group outperformed the computerized, no editorial control test group on all subtests. The computerized, with editorial control group outperformed the computerized without editorial control group on the Analytical section only. Significant main effects of computer familiarity on the Analytical and Quantitative subtests were found, but significant interactions were found between computer familiarity and test mode on the Quantitative subtest, and between gender and test mode on the Analytical subtest. Item-level analysis indicated items accompanied by graphics were more difficult in computerized form, and, the subtests were dramatically more speeded in the computerized forms. Examinees reported finding aspects of the computerized forms (i.e., individually paced testing, no pencils and bubble sheets) an improvement over the paper-and-pencil test, but other aspects more inconvenient (i.e., transferring item information back and forth between screen and scratch paper, Inability to skip/review test items). Results emphasize the importance of evaluating time constraints when converting exams from paper-and-pencil to computer-delivery, and suggest that computerized formats may have prevented examinees from using traditional test-taking strategies, affecting test performance.
