A Bayesian Approach to Ranking and Rater Evaluation: An Application to Grant Reviews

Autor/inn/en	Cao, Jing; Stokes, S. Lynne; Zhang, Song
Titel	A Bayesian Approach to Ranking and Rater Evaluation: An Application to Grant Reviews
Quelle	In: Journal of Educational and Behavioral Statistics, 35 (2010) 2, S.194-214 (21 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1076-9986
DOI	10.3102/1076998609353116
Schlagwörter	Bayesian Statistics; Data Analysis; Bias; Measurement; Error of Measurement; Grants; Simulation; Computation + Suchen Sie Ihr Suchwort? Auswertung; Messverfahren; Messfehler; Grant; Finanzielle Beihilfe; Simulation program; Simulationsprogramm
Abstract	We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented. (Contains 2 tables and 7 figures.) (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10