ArgInstruct: Specialized Instruction Fine-Tuning for Computational Argumentation Equal contribution. Equal advising.
Maja Stahl, Timon Ziegenbein, Joonsuk Park, Henning Wachsmuth · 2025
Read ArgInstruct: Specialized Instruction Fine-Tuning for Computational Argumentation Equal contribution. Equal advising. by Maja Stahl, Timon Ziegenbein, Joonsuk Park, Henning Wachsmuth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22076.txt.
Introduction
Stahl, Ziegenbein, Park, and Wachsmuth present a study from twenty twenty-five on training language models for argumentation tasks. General instruction tuning leaves models weak on tasks that need domain knowledge. Argumentation, with tasks like claim and fallacy checks, is one such domain. The team craft plain instructions for one hundred five argumentation tasks from past research. They build a benchmark to grade models across those tasks. They then synthesize tens of thousands of training instructions by a self-instruct method.
Fine-tuning on that set lifts scores on both seen and unseen argumentation tasks. General language task scores stay stable in parallel checks. The authors say domain instruction sets can specialize models safely. Their benchmark gives the field a shared yardstick.
Catalog record
| CREATOR | Maja Stahl, Timon Ziegenbein, Joonsuk Park, Henning Wachsmuth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22076
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ArgInstruct: Specialized Instruction Fine-Tuning for Computational Argumentation Equal contribution. Equal advising. Maja Stahl, Timon Ziegenbein, Joonsuk Park, Henning Wachsmuth Abstract Training large language models (LLMs) to follow instructions has significantly enhanced their ability to tackle unseen tasks. However, despite their strong generalization capabilities, instruction-following LLMs encounter difficulties when dealing with tasks that require domain knowledge. This work introduces a specialized instruction fine-tuning for the domain of computational argumentation (CA). The goal is to enable an LLM to effectively tackle any unseen CA tasks while preserving its generalization capabilities. Reviewing existing CA research, we crafted natural language instructions for 105 CA tasks to this end. On this basis, we developed a CA-specific benchmark for LLMs that allows for a comprehensive evaluation of LLMs’ capabilities in solving various CA tasks. We synthesized 52k CA-related instructions, adapting the self-instruct process to train a CA-specialized instruction-following LLM. Our experiments suggest that CA-specialized instruction fine-tuning significantly enhances the LLM on both seen and unseen CA tasks. At the same time, performance on the general NLP tasks of the SuperNI benchmark remains stable. 1 Introduction Figure 1: Comparison of fine-tuning methods: (a) Optimizing an LLM for a CA task on input-output pairs. (b) Making an LLM instruction-following on highly diverse tasks. (c) Our method: Making an LLM an instruction-following CA specialist on diverse CA-specific tasks. Large language models (LLMs) have proven effective for various NLP tasks, including several tasks from computational argumentation (CA), the computational analysis and synthesis of natural language arguments Chen et al. 2024a . Initially, it was common to fine-tune pretrained LLMs on input-output pairs for a task Devlin et al. 2019 ; Radford et al. 2019 . Figure 1 (a) illustrates such task-specific fine-tuning for the mining of claims and premises from student essays. In contrast, recent LLMs are often instruction fine-tuned by exposing them to highly diverse tasks 1 1 1 Here, the term task refers to a natural language instruction along with one or more input-output pairs that provide contextual guidance Mishra et al. 2022 ; Wang et al. 2022 . Ouyang et al. 2022 ; Taori et al. 2023 , as shown in Figure 1 (b). This enables them to generate responses aligned with specific task requirements described in the instruction Wang et al. 2022 ; Wang et al. 2023 . However, despite their strong generalization abilities, instruction-following LLMs often struggle to solve tasks that require domain knowledge Lecler et al. 2023 ; Castro Nascimento and Pimentel 2023 ; Yang et al. 2023 . This limitation results from the principle of general instruction fine-tuning to prioritize generalizability over specialization. It affects CA tasks in particular, as they often center around sophisticated context-related concepts from argumentation theory Wachsmuth et al. 2024 . Specifically, CA research in NLP focuses on the mining, assessment, and generation of natural language arguments Stede and Schneider 2018 . Despite recent advancements in LLMs, tackling CA tasks remains challenging Chen et al. 2024a due to their context-dependent specificities (e.g., in newspaper articles vs. social media posts) Habernal et al. 2014 and their subjectivity (e.g., in assessing argument quality) Wachsmuth et al. 2017 ; Romberg 2022 . In fact, providing context- and argumentation-specific knowledge, such as details about the debate setting and definitions of argumentative concepts, has been stressed to be important for task performance Lauscher et al. 2022 . In this paper, we study the impact of conflating the two learning paradigms of task-specific fine-tuning and general instruction fine-tuning. That is, we introduce the idea of specialized instruction fine-tuning by combining a highly diverse set of general tasks with a diverse set of tasks specific to a given task domain, CA in our case. The goal is to obtain an LLM that is highly proficient in CA while being agnostic to the particular CA task it encounters and maintaining generalization capabilities. We hypothesize that providing an LLM with argumentation-specific knowledge during instruction fine-tuning in a way that enables joint learning of representations across tasks is key to addressing the limitations of both general-purpose and task-specific LLMs in this domain. Through CA-specialized instruction fine-tuning (Figure 1 (c)), we enhance the LLM’s ability to mine argument structure, assess argument quality, and generate arguments across CA contexts, ensuring both accuracy and versatility for the full spectrum of argumentation. We expect this to require many and diverse CA tasks, exceeding what can be achieved by merely combining existing tasks. Towards the outlined goal, we create a large CA-specific instruction fine-tuning dataset. Starting from 105 seed tasks, derived from a total of 30 argumentation corpora, we follow the self-instruct process of Wang et al. 2023 to automatically generate a diverse set of 52k CA-specific tasks (instructions plus input-output pairs). By combining these tasks with general instruction fine-tuning data in various ways, we train instruction-following Gemma Gemma Team et al. 2024 variants for CA. The seed tasks serve as a new CA benchmark. Our experiments suggest that our specialized instruction fine-tuning method (dubbed ArgInstruct ) successfully generalizes toward unseen CA tasks, outperforming a wide range of competitive instruction-following LLMs in a zero-shot setting. Moreover, we demonstrate on SuperNI Wang et al. 2022 that the LLM’s general instruction-following abilities remain despite specialization. Altogether, this paper’s main contributions are: • A general method for specialized instruction fine-tuning, instantiated for CA • An extensive dataset for CA-specific instruction fine-tuning and benchmarking of LLMs • Empirical evidence that our CA-specialized instruction-finetuning effectively enhances an LLM’s generalizability for unseen CA tasks 2 2 2 Our dataset and experiment code can be found under: https://github.com/webis-de/ACL-25 2 Related Work The computational analysis and synthesis of arguments in natural language, often referred to as computational argumentation (CA), has its roots in a long history of philosophical research Aristotle ca. 350 B.C.E./ translated 2007 , which has gained significant attention from the NLP community in recent years. The three main CA research areas frequently covered are argument mining Park and Cardie 2014 ; Boltužić and Šnajder 2014 ; Stab and Gurevych 2017a , argument assessment Persing and Ng 2015 ; Wachsmuth et al. 2017 ; Gretz et al. 2020 , and argument generation Syed et al. 2021 ; Schiller et al. 2021 ; Wachsmuth et al. 2018a . Figure 2: Overview of our methodology: We manually craft CA-specific seed tasks and prompt an LLM to generate new CA-specific tasks in a loop by (1) generating new instructions, (2) filtering them for CA relevance and novelty, and (3) generating corresponding instances. (4) After postprocessing, the generated CA-specific tasks from the task pool are combined with existing general tasks to specialize an LLM for CA using instruction fine-tuning. Although the contributions to each area are plentiful, most works focus on one or a few tasks within or across the areas. Many methods rely on supervised learning and single-domain datasets, limiting generalizability Waldis et al. 2024 . Recently, Chen et al. 2024a , Elaraby et al. 2024 , Rescala et al. 2024 , and Cabessa et al. 2025 studied the potential of LLMs to tackle a selection of mining, assessment, and generation tasks. They obtained promising results, showing that LLMs can address multiple CA tasks, sometimes even without explicit training. Beyond that, Wachsmuth et al. 2024 propose to systematically instruct LLMs for argument quality