A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models

Zhengqing Zang, Yuqi Ding, Yanmei Gu, Changkai Song, Zhengkai Yang, Guoping Du, Junbo Zhao, Haobo Wang · 2026

Read A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models by Zhengqing Zang, Yuqi Ding, Yanmei Gu, Changkai Song, Zhengkai Yang, Guoping Du, Junbo Zhao, Haobo Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17426.txt.

Introduction

Zhengqing Zang and seven co-authors probe logic in large language models with syllogisms. A syllogism is a short formal argument with two premises and a claim. The probe centers on existential import. That is the old question of whether a general term implies that things of that kind exist. Traditional and modern logic answer it in unlike ways.

Human logic moved from gut feel to formal systems over time. The team asks if models show a like shift. Their tests trace how model answers change across setups. The probe gives a clean lens on machine reasoning. It can show if models hold one logic or mix two. That matters for law and science aids.

Catalog record

CREATORZhengqing Zang, Yuqi Ding, Yanmei Gu, Changkai Song, Zhengkai Yang, Guoping Du, Junbo Zhao, Haobo Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17426

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Syllogistic Probe: Tracing the Evolution of Logic Reasoning in Large Language Models Zhengqing Zang, Yuqi Ding, Yanmei Gu, Changkai Song, Zhengkai Yang, Guoping Du, Junbo Zhao, Haobo Wang Abstract Human logic has gradually shifted from intuition-driven inference to rigorous formal systems. Motivated by recent advances in large language models (LLMs), we explore whether LLMs exhibit a similar evolution in the underlying logical framework. Using existential import as a probe, we for evaluate syllogism under traditional and modern logic. Through extensive experiments of testing SOTA LLMs on a new syllogism dataset, we have some interesting findings: (i) Model size scaling promotes the shift toward modern logic; (ii) Thinking serves as an efficient accelerator beyond parameter scaling; (iii) the Base model plays a crucial role in determining how easily and stably this shift can emerge. Beyond these core factors, we conduct additional experiments for in-depth analysis of properties of current LLMs on syllogistic reasoning. † † footnotetext: * These authors contributed equally. † † footnotetext: \dagger Corresponding Authors. 1 Introduction Human logic has evolved from earlier, more intuition-driven accounts of valid inference 2 to increasingly rigorous formal frameworks 14 . In particular, the development of symbolic logic clarified the semantics of quantification and enabled precise validity checking under explicit model-theoretic interpretations, laying the foundation for contemporary logical analysis. Recently, neural networks have evolved from early, relatively simple architectures with limited capacity for logical reasoning to today’s large language models (LLMs), which have achieved remarkable progress across natural language processing tasks. State-of-the-art models such as GPT-5 20 and Gemini-3-Pro-Preview 11 , often rival human experts in complex reasoning tasks ranging from commonsense reasoning 3 ; 5 to mathematical problem-solving 24 ; 30 . These advances raise a natural question: do LLMs exhibit an analogous evolution in their underlying logical framework? If so, what changes, and how does this change emerge? Figure 1: The illustration of existential import problem and the trace of model logic. Figure 2: Overall performance of auto-regressive models under traditional logic and modern logic. The upper figure shows model performance under the traditional logic criterion, while the lower panel reports performance under the modern logic criterion. Point size is proportional to model scale, and color denotes model family. Qwen-T indicates Qwen Thinking models/mode. For closed-source models, we use a fixed medium point size for visualization only, which does not reflect their true parameter counts. The horizontal dashed line marks the dividing line between the traditional and modern logic. Existing reasoning benchmarks 16 increasingly target first-order logic(modern logic), examining whether they can follow this more rigorous, formal style of reasoning. However, in syllogism reasoning, existing datasets 1 ; 18 ; 31 typically treat traditional logic as the implicit default. This creates a systematic bias. A model may score high simply because it has learned dataset-specific shortcuts in traditional syllogisms, not because it truly has rigorous reasoning ability that can be transferred to new settings. On the other hand, a model may score low because it takes a modern logic view and therefore refuses to infer existence from a statement like “All unicorns are hairy animals”, which then gets marked as wrong. Even worse, these unstated rules mix up a model’s reasoning ability with how well it matches the evaluation convention, making the scores hard to interpret. In this study, we focus on syllogism 2 , a classic and well-studied form of deductive reasoning. The evaluation of syllogisms differs between two frameworks: Traditional Logic (Aristotelian) and Modern Logic (Boolean interpretation). The key difference between them lies in existential import (EI) 22 , where traditional logic typically assumes the relevant terms are non-empty while modern logic makes existential commitments only when explicitly stated. As shown in Figure 1 , this syllogism is typically treated as "valid" in traditional logic because the universal statement about unicorns (Premise2: All unicorns are hairy animals) is taken to presuppose that unicorns exist. While in modern logic, the conclusion does not follow unless existence of unicorns is separately asserted ("Some unicorns exist") , since the premise can be true even when there are no unicorns. To trace the evolution of logic reasoning in LLMs, we use existential import as a probe and conduct a series of investigations on a new syllogism dataset, which can be summarized in following key findings: (1) Controlled evidence across open-source model families and scales. We run systematic evaluations on Qwen 3 32 , Llama 3 15 , and Gemma 3 28 series across model sizes and training variants. We find that as model size increases, acc_{m} rises across all models. Models in Llama 3 and Gemma 3 series largely retain a traditional-logic reasoning style. However, in the Qwen series, we observe a clear shift in its logical paradigm from traditional logic to modern logic. We also identify a turning point where consistency fluctuates during the transition. (2) Thinking as an efficient driver beyond parameter scaling. By comparing matched-size models, we show that RL-trained thinking variants can strongly accelerate the shift toward modern logic and improve consistency. Prompted chain-of-thought yields only a partial shift, and distillation alone does not reliably produce strict modern logic behavior, suggesting that the transition is driven more by post-training optimization of reasoning policies than by scale or imitation learning alone. (3) Base-model constraints on learnability and stability. We evaluate Base models and show that they set the starting point for post-training. When the Base model already shows signals aligned with modern logic, post-training shifts are easier and more stable. Otherwise, the shift is harder and less stable. We further report the experiments of different prompts, the emptiness of minor terms, cross-lingual gaps, and architecture effects including diffusion-based LLMs, conducting an in-depth analysis of properties of current LLMs on syllogistic reasoning. 2 Background and Dataset Construction 2.1 Syllogism and Existential Import Aristotle characterizes a syllogism as consisting of two premises and a conclusion 2 , where each statement is a categorical proposition relating a subject term ( S ) to a predicate term ( P ). Within the syllogism’s structure, the conclusion’s subject ( S ) is called the minor term, and its predicate ( P ) the major term. In standard form, there are four categorical proposition types (A/E/I/O): A(universal affirmative): All S are P E(universal negative): No S are P I(particular affirmative): Some S are P O(particular negative): Some S are not P In this paper, we use traditional logic to denote the Aristotelian syllogistic framework, and modern logic to denote the Boolean interpretation of categorical propositions 6 . For reference, under modern logic these four forms are typically rendered as: A: \displaystyle\ \forall x\,(Sx\rightarrow Px) E: \displaystyle\ \forall x\,(Sx\rightarrow\neg Px)\ \text{(equiv.\ }\neg\exists x\,(Sx\land Px)\text{)} I: \displaystyle\ \exists x\,(Sx\land Px) O: \displaystyle\ \exists x\,(Sx\land\neg Px) The core distinction is existential import (EI) : whether a proposition is taken to imply that its subject class is non-empty 22 . \bullet In traditional logic , universal propositions (A/E) are typically assumed to have EI: for instance, "All S are P " is read as implying that the class S is not empty. \bullet In modern logic , universal propositions lack EI. "All S are P " is formalized as a conditional, \forall x\,(Sx\rightarrow Px) , which can remain true

Read the complete text