Evaluating the Effects of Analytical Decisions in Large-Scale Assessments: Analyzing PISA Mathematics 2003-2012

Autor/inn/en	Heine, Jörg-Henrik; Robitzsch, Alexander
Titel	Evaluating the Effects of Analytical Decisions in Large-Scale Assessments: Analyzing PISA Mathematics 2003-2012
Quelle	In: Large-scale Assessments in Education, 10 (2022), Artikel 10 (29 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Heine, Jörg-Henrik)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
DOI	10.1186/s40536-022-00129-5
Schlagwörter	Foreign Countries; International Assessment; Achievement Tests; Secondary School Students; Mathematics Achievement; Educational Trends; Mathematics Tests; Test Items; Statistical Analysis; Sampling; Trend Analysis; Evaluation Methods; Decision Making; Program for International Student Assessment + Suchen Sie Ihr Suchwort? Ausland; Achievement test; Achievement; Testing; Test; Tests; Leistungsbeurteilung; Leistungsüberprüfung; Leistung; Testdurchführung; Testen; Sekundarschüler; Mathmatics sikills; Mathmatics achievement; Mathematical ability; Mathematische Kompetenz; Bildungsentwicklung; Test content; Testaufgabe; Statistische Analyse; Trendanalyse; Decision-making; Entscheidungsfindung
Abstract	Research Question: This paper examines the overarching question of to what extent different analytic choices may influence the inference about country-specific cross-sectional and trend estimates in international large-scale assessments. We take data from the assessment of PISA mathematics proficiency from the four rounds from 2003 to 2012 as a case study. Methods: In particular, four key methodological factors are considered as analytical choices in the rescaling and analysis of the data--(1) The selection of country sub-samples for item calibration differing at three factor levels. (2) The item sample referring to two sets of mathematics items used within PISA. (3) The estimation method used for item calibration: marginal maximum likelihood estimation method as implemented in R package TAM or an pairwise row averaging approach as implemented in the R package pairwise. (4) The type of linking method: concurrent calibration or separate calibration with successive chain linking. Findings: It turned out that analytical decisions for scaling did affect the PISA outcomes. The factors of choosing different calibration samples, estimation method and linking method tend to show only small effects on the country-specific cross-sectional and trend estimates. However, the selection of different link items seems to have a decisive influence on country ranking and development trends between and within countries. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01