Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Autor/inn/en	Schroeders, Ulrich; Schmidt, Christoph; Gnambs, Timo
Titel	Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting
Quelle	In: Educational and Psychological Measurement, 82 (2022) 1, S.29-56 (28 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Schroeders, Ulrich) ORCID (Gnambs, Timo)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0013-1644
DOI	10.1177/00131644211004708
Schlagwörter	Response Style (Tests); Surveys; Artificial Intelligence; Identification; Statistical Analysis; Foreign Countries; Germany + Suchen Sie Ihr Suchwort? Antwortverhalten; Survey; Umfrage; Befragung; Künstliche Intelligenz; Identifikation; Identifizierung; Statistische Analyse; Ausland; Deutschland
Abstract	Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01