From Aggregation to Interpretation: How Assessors Judge Complex Data in a Competency-Based Portfolio

Autor/inn/en	Oudkerk Pool, Andrea; Govaerts, Marjan J. B.; Jaarsma, Debbie A. D. C.; Driessen, Erik W.
Titel	From Aggregation to Interpretation: How Assessors Judge Complex Data in a Competency-Based Portfolio
Quelle	In: Advances in Health Sciences Education, 23 (2018) 2, S.275-287 (13 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Oudkerk Pool, Andrea)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1382-4996
DOI	10.1007/s10459-017-9793-y
Schlagwörter	Undergraduate Students; Medical Students; Medical Education; Competency Based Education; Portfolio Assessment; Expertise; Judges; Decision Making; Student Evaluation; Evaluative Thinking; Protocol Analysis; Foreign Countries; Interrater Reliability; Semi Structured Interviews; Netherlands + Suchen Sie Ihr Suchwort? Medizinische Ausbildung; Education; Competence; Competency; Competency-based education; Unterricht; Kompetenzorientierte Methode; Portfoliobeurteilung; Expert appraisal; Judge; Richter; Decision-making; Entscheidungsfindung; Schulnote; Studentische Bewertung; Ausland; Interrater-Reliabilität; Niederlande
Abstract	While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass-fail) judgments of students' professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors' divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors' opinions about what kind of evidence gives the most valuable and trustworthy information about the student's competence, whereas assessors' performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors' reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence. (As Provided).
Anmerkungen	Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: service-ny@springer.com; Web site: http://www.springerlink.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01