Hidden Item Variance in Multiple Mini-Interview Scores

Autor/inn/en	Zaidi, Nikki L.; Swoboda, Christopher M.; Kelcey, Benjamin M.; Manuel, R. Stephen
Titel	Hidden Item Variance in Multiple Mini-Interview Scores
Quelle	In: Advances in Health Sciences Education, 22 (2017) 2, S.337-363 (27 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1382-4996
DOI	10.1007/s10459-016-9706-5
Schlagwörter	Interviews; Scores; Generalizability Theory; Monte Carlo Methods; Test Bias; Test Items; Models; Test Reliability; Comparative Analysis; College Admission; Medical Schools + Suchen Sie Ihr Suchwort? Interviewing; Interviewtechnik; Monte-Carlo-Methode; Testkritik; Test content; Testaufgabe; Analogiemodell; Testreliabilität; Hochschulzugang; Hochschulzulassung; Zulassung; Medizinische Ausbildung
Abstract	The extant literature has largely ignored a potentially significant source of variance in multiple mini-interview (MMI) scores by "hiding" the variance attributable to the sample of attributes used on an evaluation form. This potential source of hidden variance can be defined as rating items, which typically comprise an MMI evaluation form. Due to its multi-faceted, repeated measures format, reliability for the MMI has been primarily evaluated using generalizability (G) theory. A key assumption of G theory is that G studies model the most important sources of variance to which a researcher plans to generalize. Because G studies can only attribute variance to the facets that are modeled in a G study, failure to model potentially substantial sources of variation in MMI scores can result in biased estimates of variance components. This study demonstrates the implications of hiding the item facet in MMI studies when true item-level effects exist. An extensive Monte Carlo simulation study was conducted to examine whether a commonly used hidden item, person-by-station (p × s\|i) G study design results in biased estimated variance components. Estimates from this hidden item model were compared with estimates from a more complete person-by-station-by-item (p × s × i) model. Results suggest that when true item-level effects exist, the hidden item model (p × s\|i) will result in biased variance components which can bias reliability estimates; therefore, researchers should consider using the more complete person-by-station-by-item model (p × s × i) when evaluating generalizability of MMI scores. (As Provided).
Anmerkungen	Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: service-ny@springer.com; Web site: http://www.springerlink.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01