Re-Conceptualising and Accounting for Examiner (Cut-Score) Stringency in a 'High Frequency, Small Cohort' Performance Test

Autor/in	Homer, Matt
Titel	Re-Conceptualising and Accounting for Examiner (Cut-Score) Stringency in a 'High Frequency, Small Cohort' Performance Test
Quelle	In: Advances in Health Sciences Education, 26 (2021) 2, S.369-383 (15 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Homer, Matt)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1382-4996
DOI	10.1007/s10459-020-09990-x
Schlagwörter	Examiners; Experimenter Characteristics; Cutting Scores; Performance Based Assessment; College Graduates; Medical Education; Foreign Countries; Error of Measurement; United Kingdom + Suchen Sie Ihr Suchwort? Examiner; Prüfer; Leistungsermittlung; Hochschulabsolvent; Hochschulabsolventin; Medizinische Ausbildung; Ausland; Messfehler; Großbritannien
Abstract	Variation in examiner stringency is an ongoing problem in many performance settings such as in OSCEs, and usually is conceptualised and measured based on scores/grades examiners award. Under borderline regression, the standard within a station is set using checklist/domain scores and global grades acting in combination. This complexity requires a more nuanced view of what stringency might mean when considering sources of variation of cut-scores in stations. This study uses data from 349 administrations of an 18-station, 36 candidate single circuit OSCE for international medical graduates wanting to practice in the UK (PLAB2). The station-level data was gathered over a 34-month period up to July 2019. Linear mixed models are used to estimate and then separate out examiner (n = 547), station (n = 330) and examination (n = 349) effects on borderline regression cut-scores. Examiners are the largest source of variation in cut-scores accounting for 56% of variance in cut-scores, compared to 6% for stations, < 1% for exam and 37% residual. Aggregating to the exam level tends to ameliorate this effect. For 96% of examinations, a 'fair' cut-score, equalising out variation in examiner stringency that candidates experience, is within one standard error of measurement (SEM) of the actual cut-score. The addition of the SEM to produce the final pass mark generally ensures the public is protected from almost all false positives in the examination caused by examiner cut-score stringency acting in candidates' favour. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01