Literaturnachweis - Detailanzeige
| Autor/inn/en | Oddis, Kyle; Burstein, Jill; McCaffrey, Daniel F.; Holtzman, Steven L. |
|---|---|
| Titel | A Framework for Analyzing Features of Writing Curriculum in Studies of Student Writing Achievement |
| Quelle | 6 (2022), S.95-144 (50 Seiten)Infoseite zur Zeitschrift
PDF als Volltext (1); PDF als Volltext kostenfreie Datei (2) |
| Zusatzinformation | Weitere Informationen |
| Sprache | englisch |
| Dokumenttyp | gedruckt; online; Zeitschriftenaufsatz |
| Schlagwörter | Writing Instruction; Writing Achievement; Curriculum Evaluation; College Instruction; Scoring Rubrics; Writing Evaluation; Writing Exercises; Course Descriptions |
| Abstract | Background: Researchers interested in quantitative measures of student "success" in writing cannot control completely for contextual factors which are local and site-based (i.e., in context of a specific instructor's writing classroom at a specific institution). (In)ability to control for curriculum in studies of student writing achievement complicates interpretation of features measured in student writing. This article demonstrates how identifying and analyzing features of writing curriculum can provide dimensions of local context not captured in analysis of student-generated texts alone. Using a dataset of 48 curricular texts collected from 21 instructors teaching in five disciplines across six four-year public universities in the United States, this article: (1) presents a set of curriculum scoring rubrics developed through qualitative analysis, (2) describes a protocol for training raters to use the rubrics to score curricular texts to achieve rater agreement and generate quantitative data, and (3) explores how this framework might be amended to more deeply consider feature relationships between curriculum and student writing in studies of student writing achievement. Literature Review: The literature review provides an overview of existing studies that our research expands upon; grounds rubric development in genre theory, threshold concepts in writing studies, and design thinking; and explores how conducting curriculum analysis in tandem with feature analysis of student writing can benefit writing analytics research programs. Research Questions: (RQ1) What identifiable features of writing curriculum might affect how students approach situated writing tasks? (RQ2) How can we categorize features of writing curriculum to help us better understand its role in student writing achievement within and across disciplines? (RQ3) Can we produce a quantitative measure of curricular features that can be used in conjunction with natural language processing (NLP) data gathered on features of student writing? (RQ4) How can a set of usable, theoretically grounded rubrics offer insight into what research teams interested in studies of writing achievement might consider going forward? Methodology: The first phase of this study involved qualitative analysis of curricular texts as a guide for creating scoring rubrics. The scoring task in the second phase of the study consisted of three components: (1) Development of scoring rubrics. Rubric development was based on observations from two exploratory rounds of qualitative coding of texts in our dataset. Rubrics addressed five features of writing curriculum: "accessibility," "applicability," "actionability," "situational clarity," and "overall quality." (2) Annotation protocol training. Three research assistants with experience annotating linguistic features in texts served as raters and were trained to annotate and score the curricular texts according to the rubrics. (3) Application of scoring rubrics. Each trained rater scored the curricular texts in batches, and the study lead (first author) served as an "expert rater" who also scored the texts so final rater agreements (quadratic weighted kappa) could be calculated. Results: Qualitative analysis revealed that features of writing curriculum varied widely across learning sites, attesting to a lack of standardization or consistency of writing curriculum at and across institutions. Quantitative results speak to challenges in producing "usable" statistical data with a limited dataset. Discussion: Our study illustrates the challenges of applying rubrics to curricular datasets which offer only a partial picture of the realities of teaching and learning writing in multiple disciplines at various institutions. The potential to observe relationships between features of curricular texts and features measured in student writing requires collecting more robust datasets that include assignment grading rubrics, assignment sheets/instructions, and syllabi across disciplines in local contexts where writing happens. Future studies would need to include a sufficiently large number of courses where faculty provide a complete set of relevant curricular materials to allow for course-level analysis. Conclusions: This study's design is promising for application to larger datasets which may be drawn from single and multi-institutional contexts. Our limited dataset offers inconclusive results for demonstrating relationships between student writing features and features of associated writing curriculum (e.g., student writing motivation and applicability of curriculum). However, insights from this process suggest that in order to understand student writing achievement more comprehensively, we must develop more diversified data collection and analysis practices. This would afford deeper insight into the complexities of teaching and learning writing, specifically in terms of how students orient themselves to writing tasks delivered in curriculum. Future approaches to similar kinds of research can offer more insight into how curriculum affects student writing achievement and broader outcomes (e.g., college GPA). (As Provided). |
| Erfasst von | ERIC (Education Resources Information Center), Washington, DC |
| Update | 2024/1/01 |
Literaturbeschaffung und Bestandsnachweise in Bibliotheken prüfen
Permalink als QR-Code
Inhalt auf sozialen Plattformen teilen (nur vorhanden, wenn Javascript eingeschaltet ist)
Teile diese Seite:
