Undesired Variance Due to Examiner Stringency/Leniency Effect in Communication Skill Scores Assessed in OSCEs

Autor/inn/en	Harasym, Peter H.; Woloschuk, Wayne; Cunning, Leslie
Titel	Undesired Variance Due to Examiner Stringency/Leniency Effect in Communication Skill Scores Assessed in OSCEs
Quelle	In: Advances in Health Sciences Education, 13 (2008) 5, S.617-632 (16 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1382-4996
DOI	10.1007/s10459-007-9068-0
Schlagwörter	Medical Schools; Family Practice (Medicine); Examiners; Error of Measurement; Raw Scores; Communication Skills; Item Response Theory; Patients; Physicians; Satisfaction; Evaluation; Reliability; True Scores + Suchen Sie Ihr Suchwort? Medizinische Ausbildung; Examiner; Prüfer; Messfehler; Kommunikationsstil; Item-Response-Theorie; Patient; Physician; Doctor; Arzt; Zufriedenheit; Evaluierung; Reliabilität
Abstract	Physician-patient communication is a clinical skill that can be learned and has a positive impact on patient satisfaction and health outcomes. A concerted effort at all medical schools is now directed at teaching and evaluating this core skill. Student communication skills are often assessed by an Objective Structure Clinical Examination (OSCE). However, it is unknown what sources of error variance are introduced into examinee communication scores by various OSCE components. This study primarily examined the effect different examiners had on the evaluation of students' communication skills assessed at the end of a family medicine clerkship rotation. The communication performance of clinical clerks from Classes 2005 and 2006 were assessed using six OSCE stations. Performance was rated at each station using the 28-item Calgary-Cambridge guide. Item Response Theory analysis using a Multifaceted Rasch model was used to partition the various sources of error variance and generate a "true" communication score where the effects of examiner, case, and items are removed. Variance and reliability of scores were as follows: communication scores (0.20 and 0.87), examiner stringency/leniency (0.86 and 0.91), case (0.03 and 0.96), and item (0.86 and 0.99), respectively. All facet scores were reliable (0.87-0.99). Examiner variance (0.86) was more than four times the examinee variance (0.20). About 11% of the clerks' outcome status shifted using "true" rather than observed/raw scores. There was large variability in examinee scores due to variation in examiner stringency/leniency behaviors that may impact pass-fail decisions. Exploring the benefits of examiner training and employing "true" scores generated using Item Response Theory analyses prior to making pass/fail decisions are recommended. (As Provided).
Anmerkungen	Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: service-ny@springer.com; Web site: http://www.springerlink.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10