Are Faculty Predictions or Item Taxonomies Useful for Estimating the Outcome of Multiple-Choice Examinations?

Autor/inn/en	Kibble, Jonathan D.; Johnson, Teresa
Titel	Are Faculty Predictions or Item Taxonomies Useful for Estimating the Outcome of Multiple-Choice Examinations?
Quelle	In: Advances in Physiology Education, 35 (2011) 4, S.396-401 (6 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1043-4046
DOI	10.1152/advan.00062.2011
Schlagwörter	Test Items; Hidden Curriculum; Reliability; Physiology; Faculty; Prediction; Classification; Multiple Choice Tests; Evaluation; Medical Education; Correlation; Scores; Difficulty Level; Validity; Undergraduate Study; Pretests Posttests + Suchen Sie Ihr Suchwort? Test content; Testaufgabe; Heimlicher Lehrplan; Reliabilität; Physiologie; Academic Staff; Lehrkörper; Vorhersage; Classification system; Klassifikation; Klassifikationssystem; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Evaluierung; Medizinische Ausbildung; Korrelation; Schwierigkeitsgrad; Gültigkeit; Grundstudium
Abstract	The purpose of this study was to evaluate whether multiple-choice item difficulty could be predicted either by a subjective judgment by the question author or by applying a learning taxonomy to the items. Eight physiology faculty members teaching an upper-level undergraduate human physiology course consented to participate in the study. The faculty members annotated questions before exams with the descriptors "easy," "moderate," or "hard" and classified them according to whether they tested knowledge, comprehension, or application. Overall analysis showed a statistically significant, but relatively low, correlation between the intended item difficulty and actual student scores ([rho] = -0.19, P less than 0.01), indicating that, as intended item difficulty increased, the resulting student scores on items tended to decrease. Although this expected inverse relationship was detected, faculty members were correct only 48% of the time when estimating difficulty. There was also significant individual variation among faculty members in the ability to predict item difficulty ([chi][superscript 2] = 16.84, P = 0.02). With regard to the cognitive level of items, no significant correlation was found between the item cognitive level and either actual student scores ([rho] = -0.09, P = 0.14) or item discrimination ([rho] = 0.05, P = 0.42). Despite the inability of faculty members to accurately predict item difficulty, the examinations were of high quality, as evidenced by reliability coefficients (Cronbach's [alpha]) of 0.70-0.92, the rejection of only 4 of 300 items in the postexamination review, and a mean item discrimination (point biserial) of 0.37. In conclusion, the effort of assigning annotations describing intended difficulty and cognitive levels to multiple-choice items is of doubtful value in terms of controlling examination difficulty. However, we also report that the process of annotating questions may enhance examination validity and can reveal aspects of the hidden curriculum. (Contains 3 tables and 3 figures.) (As Provided).
Anmerkungen	American Physiological Society. 9650 Rockville Pike, Bethesda, MD 20814-3991. Tel: 301-634-7164; Fax: 301-634-7241; e-mail: webmaster@the-aps.org; Web site: http://advan.physiology.org/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10