How Do Judges in Comparative Judgement Exercises Make Their Judgements?

Autor/inn/en	Leech, Tony; Chambers, Lucy
Titel	How Do Judges in Comparative Judgement Exercises Make Their Judgements?
Quelle	In: Research Matters, (2022) 33, S.31-47 (17 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1755-6031
Schlagwörter	Comparative Analysis; Decision Making; Evaluators; Reliability; Observation; Surveys; Protocol Analysis; Preferences; Responses; Task Analysis; Standards; Computer Software; Physical Education; Scripts; Grading; Evaluation Methods + Suchen Sie Ihr Suchwort? Decision-making; Entscheidungsfindung; Reliabilität; Beobachtung; Survey; Umfrage; Befragung; Aufgabenanalyse; Standard; Körpererziehung; Sportunterricht; Skript; Notengebung; Schulnote
Abstract	Two of the central issues in comparative judgement (CJ), which are perhaps underexplored compared to questions of the method's reliability and technical quality, are "what processes do judges use to make their decisions" and "what features do they focus on when making their decisions?" This article discusses both, in the context of CJ for standard maintaining, by reporting the results of both a study into the processes used by judges when making CJ judgements, and the outcomes of surveys of judges who have used CJ. In the first instance, using insights from observations of judges and their being asked to think aloud while they judged, we highlight the variety of processes used when making their decisions, including comparative reference, re-marking and question-by-question evaluation. We then develop a four dimension model to explore what impacts what judges attend to, and explore through survey responses the distinctive ways in which the structure of the question paper, different elements of candidate responses, judges' own preferences and the CJ task itself affect decision-making. We conclude by discussing, in the light of these factors, whether the judgements made in CJ (or in the judgemental element of current standard maintaining procedures) are meaningfully holistic, and whether judges can properly take into account differences in difficulty between different papers. (As Provided).
Anmerkungen	Cambridge University Press & Assessment. Shaftesbury Road Cambridge CB2 8EA. Tel: 44-1223-553311; e-mail: directcs@cambridge.org; Web site: https://www.cambridgeassessment.org.uk/our-research/all-published-resources/research-matters/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01