The Scaling Properties of Implicit Deductive Reasoning in Transformers
Enrico Vompa, Tanel Tammet · 2026
Read The Scaling Properties of Implicit Deductive Reasoning in Transformers by Enrico Vompa, Tanel Tammet (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04330.txt.
Introduction
Enrico Vompa and Tanel Tammet investigate whether transformer models can reason deductively inside a single pass, without writing out step-by-step chains. Their testbed is logical deduction over Horn clauses, a clean setting where conclusions follow from rules and facts.
They discourage shortcut learning with counterfactual training data and encourage shared reasoning skills across direct and step-by-step modes. In deep enough models with a suitable attention mask, hidden reasoning matched explicit step-by-step performance across graph shapes and problem sizes. Writing steps out was still needed when problems grew deeper than those seen in training, marking a clear limit.
Catalog record
| CREATOR | Enrico Vompa, Tanel Tammet |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Scaling Properties of Implicit Deductive Reasoning in Transformers Enrico Vompa, Tanel Tammet Abstract We investigate the scaling properties of implicit deductive reasoning over Horn clauses in depth-bounded Transformers. By systematically decorrelating provability from spurious features and enforcing algorithmic alignment, we find that in sufficiently deep models with a bidirectional prefix mask, implicit reasoning approaches explicit CoT performance across graph topologies and problem widths, though CoT remains necessary for depth extrapolation. 1 Introduction Formal logic provides a foundation for systematic reasoning, offering the guarantees of truth-preservation and compositionality. To enforce such guarantees, classical logic solvers often rely on unbounded search and global planning to navigate complex or undecidable spaces 56 ; 57 ; 6 . Early work suggested that Transformers could simulate logical algorithms directly. 12 and 63 provided the first empirical demonstration that Transformers could operate as soft theorem provers, successfully generalizing to reasoning depths unseen during training, and 64 expanded this scope by bridging explicit logical rules with implicit, pre-trained world knowledge. However, subsequent work revealed that these models often relied on spurious features specific to their training distributions, failing to achieve true out-of-distribution generalization 79 . Fundamentally, rather than faithfully executing sequential algorithms, where the causal structure mirrors the underlying logical structure of the problem 13 , neural networks tend to learn simple, low-complexity approximations 31 ; 67 . This bias manifests as shortcut learning 24 , where models exploit spurious features or rely on parallelizable computations to bypass sequential execution 37 ; 35 ; 73 . To address these shortcut-inducing biases, we introduce three complementary interventions: the r2 heuristic to decorrelate spurious features from provability; bidirectional prefix masking for problem-wide visibility; and a single-sequence corrective training objective to align the reasoning primitives shared by direct and chain of thought (CoT) 74 predictions. Under standard complexity assumptions 40 , depth-bounded models cannot perfectly solve Horn-satisfiability or do AI planning directly 41 , yet the specific limiting factors remain unclear. We investigate these limits on deductive reasoning over Horn clauses and identify complexity- and information-theoretic scaling properties for faithful approximations as a function of attention 68 layer count and head dimensionality. Understanding these limitations is important for assessing how these models might extend to first-order logic or faithful natural language reasoning 66 ; 36 ; 61 . Motivated by evidence that attention layers can recover latent causal structures 47 , we find that scaling model depth is essential for achieving robust generalization to unseen distributions. While directly learning functions that require serial computation in transformers is computationally intractable for gradient descent (requiring exponential iterations) 33 ; 21 , they can be learned efficiently with CoT 34 ; 33 . In our setting, we observe that the corrective objective makes this optimization tractable for direct prediction, allowing CoT to act as an empirical upper bound. Our contributions are as follows: - We identify empirical scaling trends motivated by complexity- and information-theoretic concepts for faithful approximation to deductive reasoning over Horn clauses in depth-bounded Transformers. - We systematically mitigate shortcut-inducing biases, and improve out-of-distribution performance through the r2 heuristic, bidirectional prefix masking, and a corrective objective. - Within this setting, we find increasing model depth closes the implicit–explicit reasoning gap across graph topologies and problem widths, though CoT remains necessary for depth extrapolation. 2 Related work A complementary line of research delegates deduction to external theorem provers by using LLMs as semantic parsers 48 ; 49 . While effective, these approaches do not directly improve the model’s implicit deductive competence. Meanwhile, modern open-source models 51 ; 45 , driven by advancements in reinforcement learning, achieve strong deductive accuracy with CoT. Yet, their direct performance continues to lag behind in symbolic reasoning 59 , which we also observe (Appendix C ). Furthermore, as 77 showed, reinforcement learning primarily preserves rather than expands the base model’s reasoning coverage, which we also observe (Appendix S ). Methodologically, we build on counterfactually-augmented data 32 and factual–counterfactual balancing 10 , which is interpreted through the lens of information bottleneck that argues models achieve robustness by compressing their hidden representations to filter out spurious mutual information. Instead, we show that by decorrelating such features from labels, these near-collisions start competing with standard regularization. We differentiate between causal 52 ; 53 and non-causal decoders 54 ; 18 . It has been shown that the latter, which introduces a bidirectional prefix mask, outperforms the former after masked language modeling and subsequent multitask fine-tuning 72 . However, we find that in our setting, an autoregressive objective is sufficient for a non-causal decoder to outperform a causal decoder. Traditionally, multitask setting is a common method for jointly training implicit and explicit reasoning capabilities 46 ; 29 . Inspired by recent advances in structured attention mechanisms designed to isolate concurrent reasoning pathways 80 , we propose a single-sequence formulation, which unifies both reasoning modes while mitigating information leakage and reducing embedding collapse. To understand the internal dynamics of language models, various probing techniques have been developed 1 ; 65 . However, a primary limitation of standard probes is that their accuracy degrades across network depth; a probe trained on representations at layer l often fails to generalize to layer l+1 due to continuous transformations of the hidden state 4 . To address this, we develop an alignment method using Procrustes 71 ; 55 that accounts for the orthogonal transformation of the representation space. By doing so, we find that linearly separable information remains topologically consistent and decodable across all layers in our setting, allowing us to track the evolution of our models’ logical state. 3 Methodology To systematically mitigate shortcut-inducing biases, we require an environment where graph complexity and topology can be manipulated independently, allowing us to evaluate model robustness under distribution shifts. 3.1 Model architecture We train a Llama 3-based 44 decoder-only Transformer ( L=8 , d_{\text{model}}=256 , full hyperparameters in Appendix L ). We use a learned, compositional type embedding system to decouple logical roles from token identity, where the final input representation is constructed by summing the token embedding with active semantic role embeddings (e.g., tagging a token as both rule and conclusion , see Appendix B ). This explicit grounding provides the model with a more direct handle on topology, reducing parsing overhead. During training, we find that applying weight decay to normalization parameters is essential for convergence in this domain (Appendix M ). 3.2 Logic datasets The dataset generators proposed by 79 allow us to control graph complexity and topology. The Rule-Priority (RP) algorithm generates problems with an entangled structure by randomly sampling facts, rules, and a query. Ground truth is computed via forward-chaining , a process where rules are iteratively applied to facts to derive all reachable conclusions. Within this dependency graph, we define the proof depth ( \delta ) as the index of the forward breadth-first search (BFS) layer in which the query is first derived. In contrast, the Label-Priority (LP) algorithm constructs problems with a hierarchical backbone. It assigns truth values to predicates (propositional variables) in ordered levels and subsequently obscures this structure by injecting random distractor rules. A third variant, LP* , modifies LP by increasing the number of cyclic dependencies. Figure 1 : Proof depth \delta on RP and LP problems.