Automated Essay Scoring: Psychometric Guidelines and Practices

Autor/inn/en	Ramineni, Chaitanya; Williamson, David M.
Titel	Automated Essay Scoring: Psychometric Guidelines and Practices
Quelle	In: Assessing Writing, 18 (2013) 1, S.25-39 (15 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1075-2935
DOI	10.1016/j.asw.2012.10.004
Schlagwörter	Educational Testing; Guidelines; Scoring; Psychometrics; Evaluation Criteria; Program Descriptions; Vendors; Writing Evaluation; Essay Tests; Computer Assisted Testing; Program Evaluation; Evaluation Methods; Measurement + Suchen Sie Ihr Suchwort? Richtlinien; Bewertung; Psychometry; Psychometrie; Vendor; Anbieter; Schriftlicher Sprachgebrauch; Programme evaluation; Programmevaluation; Messverfahren
Abstract	In this paper, we provide an overview of psychometric procedures and guidelines Educational Testing Service (ETS) uses to evaluate automated essay scoring for operational use. We briefly describe the e-rater system, the procedures and criteria used to evaluate e-rater, implications for a range of potential uses of e-rater, and directions for future research. The description of e-rater includes a summary of characteristics of writing covered by e-rater, variations in modeling techniques available, and the regression-based model building procedure. The evaluation procedures cover multiple criteria, including association with human scores, distributional differences, subgroup differences and association with external variables of interest. Expected levels of performance for each evaluation are provided. We conclude that the "a priori" establishment of performance expectations and the evaluation of performance of e-rater against these expectations help to ensure that automated scoring provides a positive contribution to the large-scale assessment of writing. We call for continuing transparency in the design of automated scoring systems and clear and consistent expectations of performance of automated scoring before using such systems operationally. (Contains 1 figure and 1 table.) (As Provided).
Anmerkungen	Elsevier. 6277 Sea Harbor Drive, Orlando, FL 32887-4800. Tel: 877-839-7126; Tel: 407-345-4020; Fax: 407-363-1354; e-mail: usjcs@elsevier.com; Web site: http://www.elsevier.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10