The Effect of Response Order on Candidate Viewing Behaviour and Item Difficulty in a Multiple-Choice Listening Test

Autor/inn/en	Holzknecht, Franz; McCray, Gareth; Eberharter, Kathrin; Kremmel, Benjamin; Zehentner, Matthias; Spiby, Richard; Dunlea, Jamie
Titel	The Effect of Response Order on Candidate Viewing Behaviour and Item Difficulty in a Multiple-Choice Listening Test
Quelle	In: Language Testing, 38 (2021) 1, S.41-61 (21 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Holzknecht, Franz)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0265-5322
DOI	10.1177/0265532220917316
Schlagwörter	Language Tests; Test Items; Multiple Choice Tests; Listening Comprehension Tests; Eye Movements; Item Analysis; Difficulty Level; Language Processing; German; Second Language Learning; Native Language; Rating Scales; Guidelines; Foreign Countries; Europe + Suchen Sie Ihr Suchwort? Language test; Sprachtest; Test content; Testaufgabe; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Hörverstehensübung; Augenbewegung; Itemanalyse; Schwierigkeitsgrad; Sprachverarbeitung; Deutscher; Zweitsprachenerwerb; Rating-Skala; Richtlinien; Ausland; Europa
Abstract	Studies from various disciplines have reported that spatial location of options in relation to processing order impacts the ultimate choice of the option. A large number of studies have found a primacy effect, that is, the tendency to prefer the first option. In this paper we report on evidence that position of the key in four-option multiple-choice (MC) listening test items may affect item difficulty and thereby potentially introduce construct-irrelevant variance. Two sets of analyses were undertaken. With Study 1 we explored 30 test takers' processing via eye-tracking on listening items from the Aptis Test. An unexpected finding concerned the amount of processing undertaken on different response options on the MC questions, given their order. Based on this, in Study 2 we looked at the direct effect of key position on item difficulty in a sample of 200 live Aptis items and around 6000 test takers per item. The results suggest that the spatial location of the key in MC listening tests affects the amount of processing it receives and the item's difficulty. Given the widespread use of MC tasks in language assessments, these findings seem crucial, particularly for tests that randomize response order. Candidates who by chance have many keys in last position might be significantly disadvantaged. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01