Developing Competency Frameworks Using Natural Language Processing: An Exploratory Study

Autor/inn/en	Garman, Andrew N.; Erwin, Taylor S.; Garman, Tyler R.; Kim, Dae Hyun
Titel	Developing Competency Frameworks Using Natural Language Processing: An Exploratory Study
Quelle	In: Journal of Competency-Based Education, 6 (2021) 3, (4 Seiten) PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Garman, Andrew N.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	2379-6154
DOI	10.1002/cbe2.1256
Schlagwörter	Natural Language Processing; Automation; Guidelines; Leadership Effectiveness; Health Occupations; Health Personnel; Evaluators; Models; Evaluation Methods; Correlation; Generalization; Competence + Suchen Sie Ihr Suchwort? Natürliche Sprache; Richtlinien; Führungseffizienz; Gesundheitsberuf; Medizinisches Personal; Analogiemodell; Korrelation; Kompetenz
Abstract	Background: Competency models provide useful frameworks for organizing learning and assessment programs, but their construction is both time intensive and subject to perceptual biases. Some aspects of model development may be particularly well-suited to automation, specifically natural language processing (NLP), which could also help make them more generalizable and thus more learner-centric. Aims: In this study, we sought to evaluate the potential for NLP techniques be applied to competency framework development. Materials & Methods: Using NLP, we developed a set of new competency frameworks from a sample of existing leadership competency models from the health professions (e.g. nursing, medicine, healthcare management, social work, spiritual care). We then arranged for a human reviewer who was blind to the frameworks' sources to evaluate their relative coherence. Results: The human-developed frameworks tended to be viewed as more coherent than the NLP-generated frameworks, however the coherence advantage was greatest for the least complex models, and there was no apparent advantage in the most complex model we tested. Discussion: Although NLP did not consistently outperform the human-developed model structures, the pattern of results suggested directions for further model refinement and future study. Conclusion: Replicating this research with a broader sample of competency models will be important for establishing whether the observed relationship between NLP performance and model size is a more widely generalizable principle. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01