Leveraging Machine Learning Technology to Improve Accuracy and Efficiency of Identification of Enemy Item Pairs

Autor/inn/en	Micir, Ian; Swygert, Kimberly; D'Angelo, Jean
Titel	Leveraging Machine Learning Technology to Improve Accuracy and Efficiency of Identification of Enemy Item Pairs
Quelle	In: Journal of Applied Testing Technology, 23 (2022), S.30-40 (11 Seiten) PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
Schlagwörter	Artificial Intelligence; Man Machine Systems; Accuracy; Efficiency; Identification; Item Banks; Natural Language Processing; Prediction; Test Construction; Test Interpretation + Suchen Sie Ihr Suchwort? Künstliche Intelligenz; Mensch-Maschine-System; Effectiveness; Effektivität; Wirkungsgrad; Identifikation; Identifizierung; Natürliche Sprache; Vorhersage; Testaufbau; Test analysis; Testauswertung
Abstract	The interpretations of test scores in secure, high-stakes environments are dependent on several assumptions, one of which is that examinee responses to items are independent and no enemy items are included on the same forms. This paper documents the development and implementation of a C#-based application that uses Natural Language Processing (NLP) and Machine Learning (ML) techniques to produce prioritized predictions of item enemy statuses within a large item bank, which can then be followed by medical editor review of the prioritized predictions as part of an iterative process. An item bank of 4130 items from a large-scale healthcare specialist certification exam was used, in which it was assumed that many unidentified enemy pairs existed. For each pair of items, cosine similarities using TF-IDF weights were computed for the stem and answer text separately, with additional dichotomous classification variables added indicating content and existing enemy relationships. Each item pairs' existing enemy status (enemy or non-enemy) was the dependent variable for the supervised ML model, the coefficients of which were then used to generate probabilities that a given pair of items were enemies. Medical editors reviewed prioritized lists of the actual versus predicted enemy relationships in an iterative fashion. Of the 700 untagged enemy item pairs reviewed, 666 were confirmed and tagged by editors as enemies (95.1% accuracy). Thus, this application was successful in allowing editors to efficiently identify the most egregious uncoded enemy item pairs in a large item bank. The ultimate goal of this research is to inform discussion about the potential for NLP and ML applications to greatly improve accuracy and efficiency of human expert work in test construction. (As Provided).
Anmerkungen	Association of Test Publishers. 601 Pennsylvania Avenue NW, South Building Suite 900, Washington DC 20004. Tel: 866-240-7909; Fax: 717-755-8962; e-mail: wgh.atp@att.net; Web site: https://jattjournal.net/index.php/atp/index
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01