Prompt Text Classifications with Transformer Models! An Exemplary Introduction to Prompt-Based Learning with Large Language Models

Autor/inn/en	Mayer, Christian W. F.; Ludwig, Sabrina; Brandt, Steffen
Titel	Prompt Text Classifications with Transformer Models! An Exemplary Introduction to Prompt-Based Learning with Large Language Models
Quelle	In: Journal of Research on Technology in Education, 55 (2023) 1, S.125-141 (17 Seiten) PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Mayer, Christian W. F.) ORCID (Ludwig, Sabrina) ORCID (Brandt, Steffen)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1539-1523
DOI	10.1080/15391523.2022.2142872
Schlagwörter	Prompting; Classification; Artificial Intelligence; Natural Language Processing; Prediction; Language Usage; Electronic Mail; Coding; Algorithms; Technology Uses in Education; Reliability; Evaluation Methods + Suchen Sie Ihr Suchwort? Benutzerführung; Classification system; Klassifikation; Klassifikationssystem; Künstliche Intelligenz; Natürliche Sprache; Vorhersage; Sprachgebrauch; Elektronischer Briefkasten; Codierung; Programmierung; Algorithm; Algorithmus; Technology enhanced learning; Technology aided learning; Technologieunterstütztes Lernen; Reliabilität
Abstract	This study investigates the potential of automated classification using prompt-based learning approaches with transformer models (large language models trained in an unsupervised manner) for a domain-specific classification task. Prompt-based learning with zero or few shots has the potential to (1) make use of artificial intelligence without sophisticated programming skills and (2) make use of artificial intelligence without fine-tuning models with large amounts of labeled training data. We apply this novel method to perform an experiment using so-called zero-shot classification as a baseline model and a few-shot approach for classification. For comparison, we also fine-tune a language model on the given classification task and conducted a second independent human rating to compare it with the given human ratings from the original study. The used dataset consists of 2,088 email responses to a domain-specific problem-solving task that were manually labeled for their professional communication style. With the novel prompt-based learning approach, we achieved a Cohen's kappa of 0.40, while the fine-tuning approach yields a kappa of 0.59, and the new human rating achieved a kappa of 0.58 with the original human ratings. However, the classifications from the machine learning models have the advantage that each prediction is provided with a reliability estimate allowing us to identify responses that are difficult to score. We, therefore, argue that response ratings should be based on a reciprocal workflow of machine raters and human raters, where the machine rates easy-to-classify responses and the human raters focus and agree on the responses that are difficult to classify. Further, we believe that this new, more intuitive, prompt-based learning approach will enable more people to use artificial intelligence. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01