A Differential Word Use Measure for Content Analysis in Automated Essay Scoring. Research Report. ETS RR-11-36

Autor/in	Attali, Yigal
Institution	Educational Testing Service
Titel	A Differential Word Use Measure for Content Analysis in Automated Essay Scoring. Research Report. ETS RR-11-36
Quelle	(2011), (23 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Monographie
Schlagwörter	Scoring; Essay Tests; Word Frequency; Content Analysis; Test Scoring Machines; Automation; College Entrance Examinations; Language Tests; Factor Analysis; Regression (Statistics); Graduate Record Examinations; Test of English as a Foreign Language + Suchen Sie Ihr Suchwort? Bewertung; Schriftlicher Sprachgebrauch; Word analysis; Frequency; Wortanalyse; Häufigkeit; Inhaltsanalyse; Aufnahmeprüfung; Language test; Sprachtest; Faktorenanalyse; Regression; Regressionsanalyse
Abstract	This paper proposes an alternative content measure for essay scoring, based on the "difference" in the relative frequency of a word in high-scored versus low-scored essays. The "differential word use" (DWU) measure is the average of these differences across all words in the essay. A positive value indicates the essay is using vocabulary more typical of high-scoring essays in this task, and vice versa. In addition to the traditional prompt level, this measure is also computed at the generic task level, where the content of an essay is evaluated in the context of the general vocabulary of the writing task (e.g., GRE[R] issue), "across" different prompts. Evaluation results across four GRE and TOEFL[R] tasks are presented. Factor analyses show that both the prompt and task DWU measures load on the same factor as the prompt-specific content analysis measures of e-rater[R]. Regression results on the human essay scores show that the generic task DWU measure is a strong predictor of human scores, second only to essay length among noncontent e-rater features. This measure provides a way to conduct a task-level content analysis that is related to the prompt-specific content analysis of e-rater but does not require prompt-specific training. Regression results for the prompt-level DWU show that it can be used as a replacement to prompt-specific content-vector analysis (CVA) features. Appended are: (1) Examples of Extreme d Values for the GRE Issue Task; and (2) Examples of Extreme d Values for the GRE Argument Task. (Contains 8 tables.) (As Provided).
Anmerkungen	Educational Testing Service. Rosedale Road Mailstop 19R, Princeton, NJ 08541-0001. Tel: 609-921-9000; Fax: 609-734-5410; Web site: http://www.ets.org
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10