What Makes Children's Responses to Creativity Assessments Difficult to Judge Reliably?

Autor/inn/en	Dumas, Denis; Acar, Selcuk; Berthiaume, Kelly; Organisciak, Peter; Eby, David; Grajzel, Katalin; Vlaamster, Theadora; Newman, Michele; Carrera, Melanie
Titel	What Makes Children's Responses to Creativity Assessments Difficult to Judge Reliably?
Quelle	In: Journal of Creative Behavior, 57 (2023) 3, S.419-438 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Dumas, Denis) ORCID (Grajzel, Katalin) Weitere Informationen
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0022-0175
DOI	10.1002/jocb.588
Schlagwörter	Elementary School Students; Creativity; Evaluation; Reliability; Responses; Judges; Gifted; Academically Gifted; Talent; Latin Americans; Hispanic American Students; English Language Learners; Asian American Students; Asians + Suchen Sie Ihr Suchwort? Kreativität; Evaluierung; Reliabilität; Judge; Richter; Begabter, Hoch Begabter; Begabung; Hochbegabung; Latin America; People; Lateinamerika; Bevölkerung; Volk; Hispanic; Hispanic Americans; Student; Students; Hispanoamerikaner; Schüler; Schülerin; Studentin; Asian immigrant; United States; Asiatischer Einwanderer; USA; Asian; Asiat; Asiatin; Asiaten; Asiate
Abstract	Open-ended verbal creativity assessments are commonly administered in psychological research and in educational practice to elementary-aged children. Children's responses are then typically rated by teams of judges who are trained to identify original ideas, hopefully with a degree of inter-rater agreement. Even in cases where the judges are reliable, some residual disagreement on the originality of the responses is inevitable. Here, we modeled the predictors of inter-rater disagreement in a large (i.e., 387 elementary school students and 10,449 individual item responses) dataset of children's creativity assessment responses. Our five trained judges rated the responses with a high degree of consistency reliability ([alpha] = 0.844), but we undertook this study to predict the residual disagreement. We used an adaptive LASSO model to predict 72% of the variance in our judges' residual disagreement and found that there were certain types of responses on which our judges tended to disagree more. The main effects in our model showed that responses that were less original, more elaborate, prompted by a Uses task, from younger children, or from male students, were all more difficult for the judges to rate reliably. Among the interaction effects, we found that our judges were also more likely to disagree on highly original responses from Gifted/Talented students, responses from Latinx students who were identified as English Language Learners, or responses from Asian students who took a lot of time on the task. Given that human judgments such as these are currently being used to train artificial intelligence systems to rate responses to creativity assessments, we believe understanding their nuances is important. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01