The Impact of Anonymization for Automated Essay Scoring

Autor/inn/en	Shermis, Mark D.; Lottridge, Sue; Mayfield, Elijah
Titel	The Impact of Anonymization for Automated Essay Scoring
Quelle	In: Journal of Educational Measurement, 52 (2015) 4, S.419-436 (18 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0022-0655
DOI	10.1111/jedm.12093
Schlagwörter	Scoring; Essays; Computer Assisted Testing; Natural Language Processing; Prediction; Scoring Rubrics; Comparative Analysis; Privacy + Suchen Sie Ihr Suchwort? Bewertung; Essay; Aufsatzunterricht; Natürliche Sprache; Vorhersage; Scoring formulas; Auswertungsbogen; Privatsphäre
Abstract	This study investigated the impact of anonymizing text on predicted scores made by two kinds of automated scoring engines: one that incorporates elements of natural language processing (NLP) and one that does not. Eight data sets (N = 22,029) were used to form both training and test sets in which the scoring engines had access to both text and human rater scores for training, but only the text for the test set. Machine ratings were applied under three conditions: (a) both the training and test were conducted with the original data, (b) the training was modeled on the anonymized data, but the predictions were made on the original data, and (c) both the training and test were conducted on the anonymized text. The first condition served as the baseline for subsequent comparisons on the mean, standard deviation, and quadratic weighted kappa. With one exception, results on scoring scales in the range of 1-6 were not significantly different. The results on scales that were much wider did show significant differences. The conclusion was that anonymizing text for operational use may have a differential impact on machine score predictions for both NLP and non-NLP applications. (As Provided).
Anmerkungen	Wiley-Blackwell. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01