Machine Learning in Systematic Reviews: Comparing Automated Text Clustering with Lingo3G and Human Researcher Categorization in a Rapid Review

Autor/inn/en	Muller, Ashley Elizabeth; Ames, Heather Melanie R.; Jardim, Patricia Sofia Jacobsen; Rose, Christopher James
Titel	Machine Learning in Systematic Reviews: Comparing Automated Text Clustering with Lingo3G and Human Researcher Categorization in a Rapid Review
Quelle	In: Research Synthesis Methods, 13 (2022) 2, S.229-241 (13 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Muller, Ashley Elizabeth) ORCID (Ames, Heather Melanie R.) ORCID (Rose, Christopher James)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1759-2879
DOI	10.1002/jrsm.1541
Schlagwörter	Artificial Intelligence; Man Machine Systems; Automation; Literature Reviews; Classification; Mathematics; Accuracy + Suchen Sie Ihr Suchwort? Künstliche Intelligenz; Mensch-Maschine-System; Classification system; Klassifikation; Klassifikationssystem; Mathematik
Abstract	Systematic reviews are resource-intensive. The machine learning tools being developed mostly focus on the study identification process, but tools to assist in analysis and categorization are also needed. One possibility is to use unsupervised automatic text clustering, in which each study is automatically assigned to one or more meaningful clusters. Our main aim was to assess the usefulness of an automated clustering method, Lingo3G, in categorizing studies in a simplified rapid review, then compare performance (precision and recall) of this method compared to manual categorization. We randomly assigned all 128 studies in a review to be coded by a human researcher blinded to cluster assignment (mimicking two independent researchers) or by a human researcher non-blinded to cluster assignment (mimicking one researcher checking another's work). We compared time use, precision and recall of manual categorization versus automated clustering. Automated clustering and manual categorization organized studies by population and intervention/context. Automated clustering failed to identify two manually identified categories but identified one additional category not identified by the human researcher. We estimate that automated clustering has similar precision to both blinded and non-blinded researchers (e.g., 88% vs. 89%), but higher recall (e.g., 89% vs. 84%). Manual categorization required 49% more time than automated clustering. Using a specific clustering algorithm, automated clustering can be helpful with categorization of and identifying patterns across studies in simpler systematic reviews. We found that the clustering was sensitive enough to group studies according to linguistic differences that often corresponded to the manual categories. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01