Validity and Judgment Bias in Visual Analysis of Single-Case Data

Abstract

Analysis of data from single-case intervention studies commonly involves visual analysis. Previous research indicates that visual analysis may suffer from low reliability and unpromising error rates. We investigated the reliability and validity of visual analysis and explored to what extent data trends affect judgments. We administered a within-subject experiment in which 186 teacher-education students visually analyzed specifically constructed single-case graphs that included either an intervention effect, a trend effect, both effects, or no effect. Participants identified intervention effects in 75% of the graphs, regardless of the existence of a trend. Type I error rates were low (5%) in graphs without a trend but increased fivefold (25%) for graphs with a trend. Inter- and intra-rater reliability was low, particularly when a trend was present in the data.

Publication
International Journal for Research in Learning Disabilities, 5, 13-24
Timo Lüke
Timo Lüke
Professor of Inclusion-Oriented Diagnostics

Interested in assessment and instruction, especially for learners with disabilities or at risk. Special educator with a focus on learning & intellectual disabilities, and enthusiastic user of single-case research methods and Open Science practices.

Related