Research Project IV DFG-Project Q+ASA

General aim

The DFG research project Q+ASA (02/26 – 02/27; No. GI1972/2-1) contributes to the evidence-based investigation of argumentative writing skills and to the further development of computer-assisted analysis in an educational context. The central question is how argumentative structures in texts written by school and university students can be reliably recorded, analysed and automatically evaluated. In doing so, the project combines qualitative and quantitative research approaches with current methods of argument mining and machine learning.

Current status of the project

The project builds on discussion texts from secondary school students (grade 9) that have already been collected. These originate largely from Research Project I, Fair Debating and Written Argumentation, and have already been holistically and analytically evaluated within that framework, thus preparing them for the DFG project. 

Building on this, the first work package involves the annotation of argumentative structures. This is based on established annotation schemes from argumentation research. In addition, suitable software tools for the graphical annotation of tree structures will be evaluated.

In the second work package, the annotated texts will be examined using qualitative structural tree analyses. The aim is to highlight correlations between the quality of argumentative structures and the overall text quality. In particular, the analysis will focus on identifying the characteristics that distinguish coherent and high-quality arguments.

The third work package is dedicated to developing a proficiency level model for written argumentation. To this end, quantitative methods such as factor and cluster analyses will be used to empirically model different proficiency levels. The model will be validated through external peer review.

Finally, the fourth work package focuses on automated argument mining. To this end, various language models, including BERT, RoBERTa and freely available large language models such as LLaMA, will be trained and evaluated for tasks such as segmentation, classification of argumentative units and the prediction of support/attack relations. Particular attention will be paid to the question of how well models can be transferred to new datasets.

Future work

In the long term, the project is aimed to provide the foundations for a prolonged follow up project in which the developed analysis tool and the evidence-based competence level model will be tested and further developed in school contexts in collaboration with teachers. 

Furthermore, research into automated argument analysis is to be deepened. In particular, the use of large language models for complex argument mining tasks offers great potential for future applications in research and teaching. In the long term, this could lead to the development of methods that support teachers in analysing argumentative texts and open up new approaches to fostering argumentative writing skills.