Evaluating the Advisory Flags and Machine Scoring Difficulty in the "e-rater"® Automated Scoring Engine. Research Report. ETS RR-16-30

Autor/inn/en	Zhang, Mo; Chen, Jing; Ruan, Chunyi
Titel	Evaluating the Advisory Flags and Machine Scoring Difficulty in the "e-rater"® Automated Scoring Engine. Research Report. ETS RR-16-30
Quelle	In: ETS Research Report Series, (2016), (16 Seiten)Infoseite zur Zeitschrift PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	2330-8516
Schlagwörter	Essays; Scoring; Automation; Test Scoring Machines; Responses; Probability; Error Patterns + Suchen Sie Ihr Suchwort? Essay; Aufsatzunterricht; Bewertung; Wahrscheinlichkeitsrechnung; Wahrscheinlichkeitstheorie; Fehlertyp
Abstract	Successful detection of unusual responses is critical for using machine scoring in the assessment context. This study evaluated the utility of approaches to detecting unusual responses in automated essay scoring. Two research questions were pursued. One question concerned the performance of various prescreening advisory flags, and the other related to the degree of machine scoring difficulty and whether the size of the human-machine discrepancy could be predicted. The results suggested that some advisory flags operated more consistently across measures and tasks in detecting responses that the machine was likely to score differently from human raters than did other flags, and relatively little scoring difficulty was found for three of the four tasks examined in this study, with the relationship between machine and human scores being reasonably strong. Limitations and future studies are also discussed. (As Provided).
Anmerkungen	Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01