Abstract

Computer Adaptive Sequential Testing (CAST) is a test delivery model that combines features of the traditional conventional paper-and-pencil testing and item-based computerized adaptive testing (CAT). The basic structure of CAST is a panel composed of multiple testlets adaptively administered to examinees at different stages. Current applications of CAST reply on the item response theory (IRT) and assume a unidimensional IRT model for scoring.

This study evaluated the robustness of CAST when tests were constructed, administered, and scored by a unidimensional IRT model but item responses were multidimensional. Various conditions of multidimensionality were simulated in item pools, as well as different levels of content misclassification through manipulation of the correspondence between content area and dimension of items. An automated test assembly (ATA) process constructed CAST panels from the item pools, each representing a unique combination of multidimensionality and content misclassification. Administration of the panels was simulated and multidimensional response data were scored by the unidimensional IRT model. The ability scores, routing decisions, and pass-fail decisions were evaluated against "true" ability scores and decisions to assess the impacts of multidimensionality and content misclassification.

Results showed that, when multidimensionality was mild as measured by the angle distance between item clusters, unidimensional ability estimates and routing decisions were not sensitive to the level of content misclassification in item pools. Only when multidimensionality was severe, panels without content misclassification yielded more accurate ability estimates and routing decisions. However, type I errors of pass-fail decisions (defined as passing unqualified examinees) were the smallest for panels without content misclassification, regardless of the severity of multidimensionality. The conclusion was that, although content classification, affecting dimensional structure during test assembly, might not be a serious concern for unidimensional ability estimation and routing decisions for CAST with multidimensional data, it played a significant role in the accuracy of pass-fail decisions.