Do Subject Matter Experts' Judgments of Multiple-Choice Format Suitability Predict Item Quality?

Autor/inn/en	Berenbon, Rebecca F.; McHugh, Bridget C.
Titel	Do Subject Matter Experts' Judgments of Multiple-Choice Format Suitability Predict Item Quality?
Quelle	In: Educational Measurement: Issues and Practice, 42 (2023) 3, S.13-21 (9 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Berenbon, Rebecca F.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0731-1745
DOI	10.1111/emip.12570
Schlagwörter	Test Items; Multiple Choice Tests; Standards; Difficulty Level; Prediction; Psychometrics; Test Format + Suchen Sie Ihr Suchwort? Test content; Testaufgabe; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Standard; Schwierigkeitsgrad; Vorhersage; Psychometry; Psychometrie; Testentwicklung
Abstract	To assemble a high-quality test, psychometricians rely on subject matter experts (SMEs) to write high-quality items. However, SMEs are not typically given the opportunity to provide input on which content standards are most suitable for multiple-choice questions (MCQs). In the present study, we explored the relationship between perceived MCQ suitability for a given content standard and the associated item characteristics. Prior to item writing, we surveyed SMEs on MCQ suitability for each content standard. Following field testing, we then used SMEs' average ratings for each content standard to predict item characteristics for the tests. We analyzed multilevel models predicting item difficulty (p value), discrimination, and nonfunctioning distractor presence. Items were nested within courses and content standards. There was a curvilinear relationship between SMEs' ratings and item difficulty such that very low MCQ suitability ratings were predictive of easier items. After controlling for item difficulty, items with higher MCQ suitability ratings had higher discrimination and were less likely to have one or more nonfunctioning distractors. This research has practical implications for optimizing test blueprints. Additionally, psychometricians may use these ratings to better prepare for coaching SMEs during item writing. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01