Kappa and Rater Accuracy: Paradigms and Parameters

Autor/in	Conger, Anthony J.
Titel	Kappa and Rater Accuracy: Paradigms and Parameters
Quelle	In: Educational and Psychological Measurement, 77 (2017) 6, S.1019-1047 (29 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0013-1644
DOI	10.1177/0013164416663277
Schlagwörter	Interrater Reliability; Evaluators; Accuracy; Statistical Analysis; Standards; Probability; Models; Classification; Error of Measurement; Mathematical Formulas + Suchen Sie Ihr Suchwort? Interrater-Reliabilität; Statistische Analyse; Standard; Wahrscheinlichkeitsrechnung; Wahrscheinlichkeitstheorie; Analogiemodell; Classification system; Klassifikation; Klassifikationssystem; Messfehler; Mathematische Formel
Abstract	Drawing parallels to classical test theory, this article clarifies the difference between rater accuracy and reliability and demonstrates how category marginal frequencies affect rater agreement and Cohen's kappa. Category assignment paradigms are developed: comparing raters to a standard (index) versus comparing two raters to one another (concordance), using both nonstochastic and stochastic category membership. Using a probability model to express category assignments in terms of rater accuracy and random error, it is shown that observed agreement (Po) depends only on rater accuracy and number of categories; however, expected agreement (Pe) and kappa depend additionally on category frequencies. Moreover, category frequencies affect Pe and kappa solely through the variance of the category proportions, regardless of the specific frequencies underlying the variance. Paradoxically, some judgment paradigms involving stochastic categories are shown to yield higher kappa values than their nonstochastic counterparts. Using the stated probability model, assignments to categories were generated for 552 combinations of paradigms, rater and category parameters, category frequencies, and number of stimuli. Observed means and standard errors for Po, Pe, and kappa were fully consistent with theory expectations. Guidelines for interpretation of rater accuracy and reliability are offered, along with a discussion of alternatives to the basic model. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01