Abstract

College foreign language (FL) educators, like other educators, are faced with the imposing challenge to evaluate and ensure the qualities of their assessments. However, for most language educators, current validation standards and methods may offer little in the way of utility, feasibility, and meaningfulness for meeting this challenge. In this dissertation, the status of assessment and its validation within U.S. college FL education was first reviewed, and it was found to prioritize measurement qualities over assessment uses. This focus on measurement was then traced to psychometric traditions of validation practice in educational measurement, and shortcomings were identified. In response, it was proposed that assessment validation be reconceptualized as validity evaluation, and a framework was outlined for applying the methods of program evaluation to the validation of educational assessments in use. A three-year study then explored what happened when validity evaluation, following a utilization-focused model, was implemented in conjunction with the development and use of a placement assessment program in one college FL setting, the Georgetown University German Department. Examined were the questions, methods, findings, and uses that resulted from validity evaluation at each of four assessment program stages. Outcomes indicated that the process led to extensive use of findings by local language educators for understanding whether the assessment was meeting its intended uses, adjudicating continued use of instruments and procedures, communicating about assessment to a range of stakeholders, and revising assessment practices. In conclusion, the implications of validity evaluation were explored in terms of improving FL assessment practice and, more generally, transforming assessment validation into an educationally relevant endeavor.