A Meta-Analysis on the Reliability of Comparative Judgement

Autor/inn/en	Verhavert, San; Bouwer, Renske; Donche, Vincent; De Maeyer, Sven
Titel	A Meta-Analysis on the Reliability of Comparative Judgement
Quelle	In: Assessment in Education: Principles, Policy & Practice, 26 (2019) 5, S.541-562 (22 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Verhavert, San) ORCID (Donche, Vincent)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0969-594X
DOI	10.1080/0969594X.2019.1602027
Schlagwörter	Meta Analysis; Reliability; Comparative Analysis; Value Judgment; Bayesian Statistics; Probability; Performance Based Assessment; Models + Suchen Sie Ihr Suchwort? Meta-analysis; Metaanalyse; Reliabilität; Werturteil; Wahrscheinlichkeitsrechnung; Wahrscheinlichkeitstheorie; Leistungsermittlung; Analogiemodell
Abstract	Comparative Judgement (CJ) aims to improve the quality of performance-based assessments by letting multiple assessors judge pairs of performances. CJ is generally associated with high levels of reliability, but there is also a large variation in reliability between assessments. This study investigates which assessment characteristics influence the level of reliability. A meta-analysis was performed on the results of 49 CJ assessments. Results show that there was an effect of the number of comparisons on the level of reliability. In addition, the probability of reaching an asymptote in the reliability, i.e., the point where large effort is needed to only slightly increase the reliability, was larger for experts and peers than for novices. For reliability levels of 0.70 between 10 and 14 comparisons per performance are needed. This rises to 26 to 37 comparisons for a reliability of 0.90. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01