Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments

Marc Feger, Katarina Boland, Stefan Dietze · 2025

Read Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments by Marc Feger, Katarina Boland, Stefan Dietze (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22137.txt.

Introduction

Marc Feger, Katarina Boland, and Stefan Dietze re-tested leading argument mining models in twenty twenty-five and found they generalize poorly. Argument mining finds claims and reasons in text, a base skill for analyzing debates and online talk. Transformer models top the leaderboards on standard data sets, which suggested they had learned the task itself.

The team ran four transformers across seventeen English data sets at sentence level. Scores were strong on familiar data but fell sharply on unseen sets. Probing showed the models lean on content-word shortcuts tied to each data set's habits. Extra task-specific pre-training and joint training across benchmarks did help robustness. The study warns that leaderboard gains may reflect dataset learning more than argument understanding.

Catalog record

CREATORMarc Feger, Katarina Boland, Stefan Dietze
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22137

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments Marc Feger, Katarina Boland, Stefan Dietze Abstract Identifying arguments is a necessary prerequisite for various tasks in automated discourse analysis, particularly within contexts such as political debates, online discussions, and scientific reasoning. In addition to theoretical advances in understanding the constitution of arguments, a significant body of research has emerged around practical argument mining, supported by a growing number of publicly available datasets. On these benchmarks, BERT-like transformers have consistently performed best, reinforcing the belief that such models are broadly applicable across diverse contexts of debate. This study offers the first large-scale re-evaluation of such state-of-the-art models, with a specific focus on their ability to generalize in identifying arguments. We evaluate four transformers, three standard and one enhanced with contrastive pre-training for better generalization, on 17 English sentence-level datasets as most relevant to the task. Our findings show that, to varying degrees, these models tend to rely on lexical shortcuts tied to content words, suggesting that apparent progress may often be driven by dataset-specific cues rather than true task alignment. While the models achieve strong results on familiar benchmarks, their performance drops markedly when applied to unseen datasets. Nonetheless, incorporating both task-specific pre-training and joint benchmark training proves effective in enhancing both robustness and generalization. 1 Introduction Undeniably, discourse gives people the opportunity to express and discuss their beliefs on any topic. Argument mining, in this sense, is the automatic identification of the structure of inference and reasoning expressed as arguments presented in natural language Lawrence and Reed (2019) . Although there is no one-size-fits-all answer to What is an argument? Stab et al. (2018) , the idea suggests itself that arguments are latent yet observable and revolve around how they are constituted in terms of their logical scaffolding of argument discourse units, rather than what specific subject they address. In practice, these elements, whether sentences or sub-sentence segments, are pragmatically assigned functional roles, most commonly claims and premises, and form the fundamental building blocks of an argument Stab and Gurevych (2014) ; Daxenberger et al. (2017) ; Lawrence and Reed (2019) ; Lopes Cardoso et al. (2023) . Consider the example X should Y, because Z , such as Students should study, because it improves grades or We should reduce plastic use, because it minimizes ocean pollution , which illustrates that the manifestation of an argument should ideally rely on structural components conveyed through functional patterns, while remaining agnostic of certain topics or other content-specific elements. For this reason, one might assert that argument mining, in theory, is applicable across different corpora if the structural signals defining arguments are reliably identifiable from appropriately labeled data. Conversely, in practice, any inability to apply these signals to diverse datasets may expose systematic biases in the field, an issue that has long been informally discussed over coffee breaks. Generalizability, in this regard, takes high priority, especially at leading NLP conferences such as ACL 2025, as it allows models to make reliable and reasonable predictions on data that does not correspond to their training data. This is especially true for real-world models, which should mimic human-like generalization abilities, where emerging evidence indicates that such models are often fine-tuned to the specifics of established benchmark datasets, leading to unfounded optimism about their improvements Saphra et al. (2024) . Consequently, concerns about vulnerability to shortcut learning Geirhos et al. (2020) highlight the broader challenge of evaluating baselines beyond isolated benchmarks Rendle et al. (2019) . Argument mining is one such area of natural language processing applications in which the ability to generalize is key. Hence, we ask for: Q1 : How comparable are the existing benchmark datasets for argument mining? Q2 : Do state-of-the-art argument mining models generalize to out-of-distribution data from other benchmarks? Q3 : Do these models acquire a generalizable concept of arguments? In this context, there has been speculation that BERT Devlin et al. (2019) , known to pay great attention to basic syntax, nouns, and co-references Clark et al. (2019) , is prone to learning shortcuts when mining arguments Geirhos et al. (2020) , where its generalization is limited to within-topic signals in datasets sharing similar argument and topic structures Thorn Jakobsen et al. (2021) . Our aim is not to propose a new formalism for arguments or to pinpoint the best-performing argument mining model, but to use data from previous work in which different theories have been applied to see whether individual efforts and perspectives converge in terms of identifying arguments. With this being said, we perform the first large-scale experimental assessment of benchmarks, systematically evaluating generalization across diverse argument mining datasets following a comprehensive review of datasets spanning 2008 to 2024. For our study, we selected BERT Devlin et al. (2019) , RoBERTa Liu et al. (2019) , and DistilBERT Sanh et al. (2019) as exemplary BERT-like models, widely recognized as standard baselines in various areas of natural language processing Rogers et al. (2020) , including recent research on argument mining Shnarch et al. (2020) ; Mayer et al. (2020a) ; Fromm et al. (2021a) ; Alhamzeh et al. (2022) ; Feger and Dietze (2024b) . We also examine WRAP Feger and Dietze (2024a) , the only transformer whose language representation pre-training is extended by leveraging contrasts of inference and information signals to generalize argument components. Although originally designed for cross-topic generalization on Twitter ( \mathbb{X} ), WRAP does not rely on tweet- or topic-specific features to enhance its generalizability, distinguishing it from the others and making it particularly interesting for research. In this study, we start by detailing our process of finding argument mining benchmark datasets and explain the selection criteria and justifications in Section 2 . The core characteristics of these datasets, addressing research question Q1 , are then examined in Section 3 . Next, we describe our experimental setup in Section 4 , covering both result generation and the implementation of best practices for significance testing, which form the basis for answering Q2 - Q3 in Section 5 . The results of this paper are then discussed in Section 6 and concluded in Section 7 . In order not only to elucidate the process but also to foster discussion that may inspire new approaches for novel datasets and broader generalization of argument mining methods, we contribute: 1. A survey of argument mining datasets between 2008 and 2024, primarily from the ACL Anthology, that identified 52 relevant papers with datasets from leading NLP conferences. 2. The first large-scale re-assessment that combines benchmark evaluations for 17 selected argument mining datasets, including controlled manipulation experiments to determine whether the reported state-of-the-art models (BERT, RoBERTa, DistilBERT, WRAP) actually learn generalizable argument concepts. 3. Statistical evidence that shortcut learning undermines generalization in argument mining. Although each of the examined transformers delivers strong results on benchmarks, all struggle to varying degrees when applied to other datasets, with WRAP generally performing slightly better. These challenges are compounded by divergent argument definitions and inconsistent annotations across datasets. 2 Argument Mining Benchmark Datasets This section outlines the dataset collection and selection process, emphasizing the rationale behind our choice of benchmark datasets for argument mining. The decisions for all 52 datasets reviewed are present in

Read the complete text