Using an Innovative Standard-Setting Approach to Align Integrated and Independent Writing Tasks to the CEFR

Autor/inn/en	Harsch, Claudia; Kanistra, Voula Paraskevi
Titel	Using an Innovative Standard-Setting Approach to Align Integrated and Independent Writing Tasks to the CEFR
Quelle	In: Language Assessment Quarterly, 17 (2020) 3, S.262-281 (20 Seiten) PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1543-4303
DOI	10.1080/15434303.2020.1754828
Schlagwörter	Standard Setting; Guidelines; Rating Scales; Definitions; Decision Making; Writing Evaluation; Second Language Learning; Foreign Countries; Benchmarking; Test Items; Language Tests; Task Analysis; Cutting Scores; Validity; Specialists; English (Second Language); Item Response Theory; European Union; United Kingdom (London) + Suchen Sie Ihr Suchwort? Standardisierung; Richtlinien; Rating-Skala; Begriffsbestimmung; Decision-making; Entscheidungsfindung; Zweitsprachenerwerb; Ausland; Test content; Testaufgabe; Language test; Sprachtest; Aufgabenanalyse; Gültigkeit; English as second language; English; Second Language; Englisch als Zweitsprache; Item-Response-Theorie
Abstract	We report on a standard-setting project in which the Item-Descriptor-Matching Method (IDM) and a complementary benchmarking approach were employed to align a suite of English language proficiency exams to the "Common European Framework of Reference" (CEFR), with a particular focus on the integrated and independent writing exams. Judges' ratings on eight writing tasks and 48 test taker scripts were collected online via SmartSurvey. The judges gave CEFR-level jkudgements for tasks and scripts, they stated which CEFR descriptors they matched against the task demands and scripts, and they evaluated the combined approach and outcomes. Thus, it was possible to monitor how judges applied and interpreted the CEFR descriptors, a prerequisite for establishing alignment validity. Analyses of judgement consistency revealed a high level of consistency on the task judgements and overall performance ratings, yet they also revealed some variations in the selected CEFR descriptors underpinning the judgements. Making these variations transparent facilitated a targeted discussion with an explicit focus on the CEFR, i.e. the framework to which the tests were to be aligned. Overall, the judges reported confidence in using the combined approaches, in their judgements and in the recommended CEFR cut-scores, thus corroborating procedural validity. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01