Use of Automated Scoring Features to Generate Hypotheses Regarding Language-Based DIF

Autor/inn/en	Shermis, Mark D.; Mao, Liyang; Mulholland, Matthew; Kieftenbeld, Vincent
Titel	Use of Automated Scoring Features to Generate Hypotheses Regarding Language-Based DIF
Quelle	In: International Journal of Testing, 17 (2017) 4, S.351-371 (21 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1530-5058
DOI	10.1080/15305058.2017.1308949
Schlagwörter	Computer Assisted Testing; Scoring; Hypothesis Testing; Essays; Test Items; Item Response Theory; Test Bias; Secondary School Students; Languages; Language Usage; Language Variation; Gender Differences; Native Language; Socioeconomic Status; Scoring Rubrics; Earth Science; Formative Evaluation; Multiple Choice Tests; Likert Scales; Prediction; Computer Use + Suchen Sie Ihr Suchwort? Bewertung; Hypothesenprüfung; Hypothesentest; Essay; Aufsatzunterricht; Test content; Testaufgabe; Item-Response-Theorie; Testkritik; Sekundarschüler; Language; Sprache; Sprachgebrauch; Sprachenvielfalt; Geschlechterkonflikt; Socio-economic status; Sozioökonomischer Status; Scoring formulas; Auswertungsbogen; Earth sciences; Geowissenschaften; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Likert-Skala; Vorhersage
Abstract	This study uses the feature sets employed by two automated scoring engines to determine if a "linguistic profile" could be formulated that would help identify items that are likely to exhibit differential item functioning (DIF) based on linguistic features. Sixteen items were administered to 1200 students where demographic information was collected on gender and socioeconomic status (SES). Textual features were extracted and analyzed using Differential Item Functioning Analysis System (DIFAS) and the Mantel-Haenszel chi-square, the Liu-Agresti cumulative common log-odds ratio, and Cox's noncentrality parameter estimator to assess the probability that the focal groups (e.g., females) differed significantly from the reference groups (e.g., males). Two of the 14 items were flagged for possible DIF on gender and four were flagged for possible DIF on SES. The responses from the focal and reference groups were then analyzed using nine machine learning algorithms to determine if there were significant differences on specific linguistic features. For gender, the procedure did not reliably identify items manifesting DIF. The results for SES were more encouraging with a reliable prediction of the presence of DIF, but not its direction. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01