How Accurate Are ESL Students' Holistic Writing Scores on Large-Scale Assessments?--A Generalizability Theory Approach

Autor/in	Huang, Jinyan
Titel	How Accurate Are ESL Students' Holistic Writing Scores on Large-Scale Assessments?--A Generalizability Theory Approach
Quelle	In: Assessing Writing, 13 (2008) 3, S.201-218 (18 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1075-2935
DOI	10.1016/j.asw.2008.10.002
Schlagwörter	Generalizability Theory; Foreign Countries; English (Second Language); Writing Tests; Scores; Student Evaluation; Second Language Instruction; Second Language Learning; Culture Fair Tests; Test Bias; Testing Problems; Reliability; Measurement; Elementary Secondary Education; Comparative Analysis; Native Speakers; Canada + Suchen Sie Ihr Suchwort? Ausland; English as second language; English; Second Language; Englisch als Zweitsprache; Writing test; Schreibtest; Schulnote; Studentische Bewertung; Fremdsprachenunterricht; Zweitsprachenerwerb; Testkritik; Reliabilität; Messverfahren; Muttersprachler; Kanada
Abstract	Using generalizability theory, this study examined both the rating variability and reliability of ESL students' writing in the provincial English examinations in Canada. Three years' data were used in order to complete the analyses and examine the stability of the results. The major research question that guided this study was: Are there any differences between the rating variability and reliability of the writing scores assigned to ESL students and to Native English (NE) students in the writing components of the provincial examinations across three years? A series of generalizability studies and decision studies was conducted. Results showed that differences in score variation did exist between ESL and NE students when adjudicated scores were used. First, there was a large effect for both language group and person within language-by-task interaction. Second, the unwanted residual variance component was significantly larger for ESL students than for NE students in all three years. Finally, the desired variance associated with the object of measurement was significantly smaller for ESL students than for NE students in one year. Consequently, the observed generalizability coefficient for ESL students was significantly lower than that for NE students in that year. These findings raise a potential question about the fairness of the writing scores assigned to ESL students. (Contains 8 tables.) (As Provided).
Anmerkungen	Elsevier. 6277 Sea Harbor Drive, Orlando, FL 32887-4800. Tel: 877-839-7126; Tel: 407-345-4020; Fax: 407-363-1354; e-mail: usjcs@elsevier.com; Web site: http://www.elsevier.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10