Improving Content Validation Studies Using an Asymmetric Confidence Interval for the Mean of Expert Ratings

Autor/inn/en	Penfield, Randall D.; Miller, Jeffrey M.
Titel	Improving Content Validation Studies Using an Asymmetric Confidence Interval for the Mean of Expert Ratings
Quelle	In: Applied Measurement in Education, 17 (2004) 4, S.359-370 (12 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0895-7347
Schlagwörter	Student Evaluation; Evaluation Methods; Content Validity; Scoring; Scores; Automation; Test Scoring Machines + Suchen Sie Ihr Suchwort? Schulnote; Studentische Bewertung; Bewertung
Abstract	As automated scoring of complex constructed-response examinations reaches operational status, the process of evaluating the quality of resultant scores, particularly in contrast to scores of expert human graders, becomes as complex as the data itself. Using a vignette from the Architectural Registration Examination (ARE), this article explores the potential utility of Classification and Regression Trees (CART) and Kohonen Self-Organizing Maps (SOM) as tools to facilitate subject matter expert (SME) examination of the fine-grained (feature level) quality of automated scores for complex data, with implications for the validity of resultant scores. This article explores both supervised and unsupervised learning techniques, with the former being represented by CART (Breiman, Friedman, Olshen, & Stone, 1984) and the latter by SOM (Kohonen, 1989). Three applications comprise this investigation, the first of which suggests that CART can facilitate efficient and economical identification of specific elements of complex responses that contribute to automated and human score discrepancies. The second application builds on the first by exploring CART for efficiently and accurately automating case selection for human intervention to ensure score validity. The final application explores the potential for SOM to reduce the need for SMEs in evaluating automated scoring. Although both supervised and unsupervised methodologies examined were found to be promising tools for facilitating SME roles in maintaining and improving the quality of automated scoring, such applications remain unproven, and further studies are necessary to establish the reliability of these techniques. (Author).
Anmerkungen	Lawrence Erlbaum Associates, Inc., Journal Subscription Department, 10 Industrial Avenue, Mahwah, NJ 07430-2262. Tel: 800-926-6579 (Toll Free); e-mail: journals@erlbaum.com.
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10