Applying Kane's Validity Framework to a Simulation Based Assessment of Clinical Competence

Autor/inn/en	Tavares, Walter; Brydges, Ryan; Myre, Paul; Prpic, Jason; Turner, Linda; Yelle, Richard; Huiskamp, Maud
Titel	Applying Kane's Validity Framework to a Simulation Based Assessment of Clinical Competence
Quelle	In: Advances in Health Sciences Education, 23 (2018) 2, S.323-338 (16 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Tavares, Walter)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1382-4996
DOI	10.1007/s10459-017-9800-3
Schlagwörter	Competence; Simulation; Allied Health Personnel; Certification; Inferences; Evaluation Methods; Student Evaluation; Test Validity; Scoring; Generalization; Correlation; Error Patterns; Interrater Reliability + Suchen Sie Ihr Suchwort? Kompetenz; Simulation program; Simulationsprogramm; Abschlusszeugnis; Zertifizierung; Inference; Inferenz; Schulnote; Studentische Bewertung; Testvalidität; Bewertung; Korrelation; Fehlertyp; Interrater-Reliabilität
Abstract	Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particularly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane's validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assessment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assessment data. Data evaluating "Scoring" included: (a) desirable tool characteristics, with acceptable inter-item correlations; (b) strong item-total correlations; (c) low error variance for items and raters; and (d) strong inter-rater reliability. Data evaluating "Generalizability" included: a robust sampling strategy capturing the majority of relevant medical directives, skills and national competencies, and good overall and inter-station reliability. Data evaluating "Extrapolation" included: low correlations between assessment scores by dimension and clinical errors in practice. Data evaluating "Implications" included low error rates in practice. Interpreting our findings according to Kane's framework, we suggest the evidence for scoring, generalization and implications supports use of our simulation-based paramedic assessment strategy as a certifying exam; however, the extrapolation evidence was weak, suggesting exam scores did not predict clinical error rates. Our analysis represents a worked example others can follow when using Kane's validity framework to evaluate, and iteratively develop and refine assessment strategies. (As Provided).
Anmerkungen	Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: service-ny@springer.com; Web site: http://www.springerlink.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01