Designing, Evaluating, and Deploying Automated Scoring Systems with Validity in Mind: Methodological Design Decisions

Autor/in	Rupp, André A.
Titel	Designing, Evaluating, and Deploying Automated Scoring Systems with Validity in Mind: Methodological Design Decisions
Quelle	In: Applied Measurement in Education, 31 (2018) 3, S.191-214 (24 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0895-7347
DOI	10.1080/08957347.2018.1464448
Schlagwörter	Design; Automation; Scoring; Test Scoring Machines; Computer Assisted Testing; Test Validity; Data Collection; Data Interpretation; Test Interpretation; Essay Tests; Decision Making; Test Construction; Best Practices; Interdisciplinary Approach; Generalizability Theory; Information Management; Quality Control + Suchen Sie Ihr Suchwort? Bewertung; Testvalidität; Data capture; Datensammlung; Data evaluation; Datenauswertung; Test analysis; Testauswertung; Schriftlicher Sprachgebrauch; Decision-making; Entscheidungsfindung; Testaufbau; Fächerübergreifender Unterricht; Fächerverbindender Unterricht; Interdisziplinarität; Procurement of information; Informationsbeschaffung; Qualitätskontrolle
Abstract	This article discusses critical methodological design decisions for collecting, interpreting, and synthesizing empirical evidence during the design, deployment, and operational quality-control phases for automated scoring systems. The discussion is inspired by work on operational large-scale systems for automated essay scoring but many of the principles have implications for principled reasoning and workflow management for other use contexts. The overall workflow is described as a series of five phases, each one having two critical sub-phases with a large number of associated methodological design decisions. These phases involve assessment design, linguistic component design, model design, model validation, and operational deployment. Through brief examples, the various considerations for these design decisions are illustrated, which have to be carefully weighed in the overall decision-making process for the system in order to unveil the complexities that underlie this work. The article closes with reflections on resource demands as well as recommendations for best practices of interdisciplinary teams who engage in this work, underscoring how this work is a blend of scientific rigor and artful practice. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01