The Effect of Adaptivity on the Reliability Coefficient in Adaptive Comparative Judgement

Autor/inn/en	Bramley, Tom; Vitello, Sylvia
Titel	The Effect of Adaptivity on the Reliability Coefficient in Adaptive Comparative Judgement
Quelle	In: Assessment in Education: Principles, Policy & Practice, 26 (2019) 1, S.43-58 (16 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0969-594X
DOI	10.1080/0969594X.2017.1418734
Schlagwörter	Reliability; Evaluation Methods; Comparative Analysis; Essay Tests; Foreign Countries; Judges; United Kingdom + Suchen Sie Ihr Suchwort? Reliabilität; Schriftlicher Sprachgebrauch; Ausland; Judge; Richter; Großbritannien
Abstract	Comparative Judgement (CJ) is an increasingly widely investigated method in assessment for creating a scale, for example of the quality of essays. One area that has attracted attention in CJ studies is the optimisation of the selection of pairs of objects for judgement. One approach is known as adaptive comparative judgement (ACJ). It has been claimed in the literature that ACJ produces very high reliability, often higher than can be obtained by conventional marking. Bramley showed by simulation that adaptivity can substantially inflate the apparent reliability in ACJ. The empirical study described here compared an adaptive with a non-adaptive CJ study using GCSE English essays. An all-play-all set of comparisons of a subset of the essays allowed the extent of scale inflation to be quantified: the reported adaptive reliability was 0.97 whereas the deflated value was 0.84. The value from the non-adaptive study was 0.72. However, the scale from the non-adaptive study correlated slightly higher with external variables, suggesting the non-adaptive study was no less valid than the adaptive one. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01