Although psychologists often believe that all assessment information collected on a client should be integrated to reach a conclusion, statistical analyses and empirical evidence show that this may not be the optimal procedure for ensuring diagnostic decision-making accuracy. As a result, we are left with the question: How should psychologists determine the optimal combination of variables, or the single variable, to use in diagnosis or prediction? Proper assessment of incremental validity addresses these issues by allowing one to identify the most valid combination of assessment information beyond which either additional variables do not add information in a clinically significant way, or the cost/benefit ratio of additional information is unnecessarily high.
In order to study this issue in the field of school psychology, hypothetical case scenarios were provided to school psychologists, via the Internet or postal mail. These scenarios were designed (a) to assess respondents' use of statistical information when deciding which measures to use in a diagnostic process, and (b) to assess if a generic diagnostic label (e.g., Disorder X) and assessment instrument (e.g., Measure 1) versus a specific label (e.g., ADHD) and instrument (e.g., Parent Rating Scale) was associated with the use of statistical information during this process.
No differences were identified between the Internet and postal-mail methods of participation. Results, however, showed that the school psychologists made sub-optimal statistical choices about measures even when only two measures were provided. In more complex scenarios (i.e., four measures), respondents continued to make sub-optimal choices. When diagnostic labels and names of assessment instruments were included, a number of inconsistencies were observed, indicating that statistical information was not properly incorporated in the decision-making process. Results are discussed in terms of a failure to attend to issues associated with incremental validity by school psychologists during diagnostic decision-making tasks, and implications for training, clinical practice, and future research are described.
