Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification
Eleni Papadopulos, Firoj Alam Giovanni Da San Martino · 2026
Read Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification by Eleni Papadopulos, Firoj Alam Giovanni Da San Martino (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26698.txt.
Introduction
Eleni Papadopulos, Firoj Alam, and Giovanni Da San Martino improve automatic detection of logical fallacies. A fallacy is a defective pattern of reasoning that misleads. Fallacies appear in subtle, varied forms that resist fixed templates. The authors ask whether combining abstract logical structure with everyday language cues classifies them better. Their framework extracts such patterns directly from examples.
A large language model reads flawed texts with their explanations. It draws out pattern descriptions that merge logical form with everyday wording. Those patterns then guide classification in zero-shot and one-shot settings, meaning with no example or one example given. Results improved significantly over zero-shot baselines and beat competing approaches. Tests on a second data set confirmed the method generalises. The authors present pattern mining from data as a good way to build logical descriptions.
Catalog record
| CREATOR | Eleni Papadopulos, Firoj Alam Giovanni Da San Martino |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26698
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification Eleni Papadopulos, Firoj Alam Giovanni Da San Martino Abstract In today’s fast-paced information era, logical fallacies, defined as defective patterns of reasoning, inevitably contribute to the growth of information disorder. However, often fallacies appear in nuanced forms that complicate automated classification. In this study, we investigate whether merging abstract logical structures with context-level linguistic cues proves beneficial for fallacy classification, developing a framework that inductively extracts such patterns from fallacious examples and their explanations using Large Language Models (LLMs). We evaluate the impact of these patterns across different LLMs and experimental zero- and one-shot configurations, showing statistically significant improvements over zero-shot baselines and outperforming competing approaches. Cross-dataset experiments validate generalization, establishing data-driven pattern extraction as an effective method for generating logical representations. 1 Introduction A logical fallacy is a common thinking error, especially one apt to mislead ( 9 ) . These arguments often appear rational and logically coherent on the surface, but deeper analysis reveals they are not ( 4 ) . Fallacies are traditionally classified into formal and informal types: formal fallacies violate the rules of logical structure regardless of content, while informal fallacies are patterns of mistakes that are made in the everyday uses of language and are related to contextual meaning 13 ; 2 . To evaluate the quality of an argument, it is helpful to reconstruct it into what is known as logical form, the structure that emerges when the specific content of a statement is replaced by variables ( 18 ) . For example, the argument If it rains, then the ground will be wet. It is raining. Therefore, the ground is wet has the logical form If P, then Q. P. Therefore, Q . Building on this formalization framework, 17 developed a structure-aware model for fallacy detection on the Logic dataset that compares arguments’ and fallacies’ logical forms. However, in their approach a single logical form is assigned to each fallacy, which might fail to capture the full spectrum of ways a fallacy can manifest in natural discourse. Another challenge is related to informal fallacies, where reasoning is often more nuanced and context-dependent than abstract representations suggest. These limitations motivate the need to go beyond purely abstract representations, incorporating linguistic elements, such as lexical markers or rhetorical devices, to provide a more comprehensive characterization of how fallacies manifest in natural language. We argue that LLMs can inductively extract such representations from fallacious examples, capturing both the logical structure and linguistic cues that reveal the underlying mechanisms of deception. Unlike prior work that formalized fallacy logic through hand-crafted templates ( 27 ) , our approach is data-driven and not restricted to a limited number of fallacies. Hereafter, we refer to these extracted structures collectively as structural patterns . Our goal is to investigate whether context-aware structural information is valuable for automated fallacy detection. While existing supervised approaches require heavy computational resources for fine-tuning ( 36 ; 19 ; 32 ; 33 ; 1 ) , to our knowledge, no prior work has explored fallacy classification from a structural perspective without any additional fine-tuning. Although we use labeled data for pattern extraction (resulting in a weakly supervised approach), our framework avoids fine-tuning costs and produces generalizable patterns that allow classification through prompting alone, enabling comparison with unsupervised methods. We evaluate multiple prompting configurations to determine which components enhance performance and examine the impact of demonstrations on detection capabilities. Our approach, incorporating generated patterns, achieves noteworthy results among unsupervised methods on the dataset Logic . Finally, to validate the robustness and transferability of our patterns, we assess their performance across two further datasets spanning diverse domains and argumentative styles. In summary, our contributions are threefold: • We leverage LLMs to automatically extract patterns from fallacious examples and their explanations, which are then employed in inference-only classification. • We evaluate different LLMs with various prompt designs outperforming competing approaches on the LOGIC dataset. • We validate generalizability by testing our patterns on two different datasets with different domains and structures. 2 Related Work Recent advances in fallacy detection have increasingly turned to LLMs, though few studies have relied exclusively on prompting-based techniques. Several works have employed fallacy detection to probe LLMs’ logical reasoning abilities ( 35 ; 15 ; 20 ; 40 ) . Among these, 15 investigated self-verification capabilities and showed that LLMs face more challenges with structure-based (formal) fallacies with respect to content-based (informal) ones, and that fallacy definitions provide minimal improvements. 40 has shown that reasoning models have better performances with respect to non-reasoning ones for fallacy classification. Among studies relying exclusively on prompting techniques, 24 designed single-round and multi-round prompting schemes for zero-shot detection, while 16 introduced contextual prompting incorporating counterarguments, explanations, and goals with confidence-based ranking, showing that explanations particularly enhance performance. 21 assessed detection abilities on the Logic dataset using few-shot prompting, though their different taxonomy limits direct comparison with our work. Other research has examined the logical structure of argumentation. Most notably, 17 developed a structure-aware model based on Electra that distills arguments into logical forms and compares them against fallacy patterns sourced from logicallyfallacious.com . Another prominent framework in this field is Walton’s theory of argumentation ( 37 ) , consisting in about 60 templates that capture common argument types, each associated with a set of critical questions to evaluate their validity. These schemes have been adopted in computational approaches for disinformation ( 11 ) , misinformation ( 29 ) and fake news ( 38 ) detection. Regarding fallacies specifically, 28 introduced a dataset of sentences grounded in Walton’s argumentation schemes, labeling them as fallacies when the associated critical questions could not be successfully answered. However, Walton’s schemes do not hold a one-to-one correspondence with fallacies and cover only a limited number of them, limiting their direct applicability to fallacy classification. Of particular relevance is the work of 27 , who re-designed four of 37 and 26 ’s schemes with the goal of expliciting fallacies’ implicit logic, introducing formal logical schemas with explicit variables and relationships. While this represents a meaningful step toward structural formalization, their patterns are manually designed and leave a large portion of fallacy types unrepresented. Our patterns, by contrast, are extracted automatically from data, allowing to adapt to intra-class variation. 3 Datasets The Logic dataset is a collection of 2,449 examples across 13 fallacy types ( 17 ) . Instances are sourced from educational platforms about fallacies such as Quizziz and study.com. The dataset consists of brief dialogues and short statements. Given the educational intent behind these examples, sentences tend to have relatively straightforward syntactic structures, making the dataset particularly well-suited for the extraction of the patterns. Although it contains 13 distinct classes, a thorough analysis revealed that some of the classes actually contain instances of different fallacies, that were grouped together. For instance, the class Hasty Generalization contains examples of actual Hasty Generalization as well as Slippery Slope (Table 1 ). While these grouped fallacies share common logical flaws and thus belong to the same conceptual group, they manifest through different structural patterns. Class Fallacies included Intentional Fallacy Intentional Fallacy Shifting the Burden of Proof Moving the Goalposts No True Scotsman False Cause Post Hoc False