Dynamics Reveals Structure: Challenging the Linear Propagation Assumption

Hoyeon Chang, Bálint Mucsányi, Seong Joon Oh · 2026

Read Dynamics Reveals Structure: Challenging the Linear Propagation Assumption by Hoyeon Chang, Bálint Mucsányi, Seong Joon Oh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21601.txt.

Introduction

Hoyeon Chang, Bálint Mucsányi, and Seong Joon Oh test whether gradient updates keep a network's logic coherent. Networks learn by small first-order steps in their parameters. It is often assumed that a small local step spreads sensibly to all its logical consequences.

The authors name that premise the linear propagation assumption and probe its limits. Using relation algebra, they track three basic operations: negation, converse, and composition of relations. The study shows where the premise breaks down as changes propagate. Knowing those limits helps explain odd failures where a model learns a fact but misses its obvious flip side.

Catalog record

CREATORHoyeon Chang, Bálint Mucsányi, Seong Joon Oh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21601

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamics Reveals Structure: Challenging the Linear Propagation Assumption Hoyeon Chang, Bálint Mucsányi, Seong Joon Oh Abstract Neural networks adapt through first-order parameter updates, yet it remains unclear whether such updates preserve logical coherence. We investigate the geometric limits of the Linear Propagation Assumption (LPA), the premise that local updates coherently propagate to logical consequences. To formalize this, we adopt relation algebra and study three core operations on relations: negation flips truth values, converse swaps argument order, and composition chains relations. For negation and converse, we prove that guaranteeing direction-agnostic first-order propagation necessitates a tensor factorization separating entity-pair context from relation content. However, for composition, we identify a fundamental obstruction. We show that composition reduces to conjunction, and prove that any conjunction well-defined on linear features must be bilinear. Since bilinearity is incompatible with negation, this forces the feature map to collapse. These results suggest that failures in knowledge editing, the reversal curse, and multi-hop reasoning may stem from common structural limitations inherent to the LPA. 1 Introduction Modern machine learning systems evolve through a long trajectory, spanning pretraining, continual learning ( 42 ) , and unlearning ( 21 ) . Throughout this lifecycle, the central operation for adapting to new information is the first-order parameter update. Ideally, this adaptation should be rational: when a model revises its belief about a fact, its logically related beliefs should update accordingly to maintain coherence. However, maximizing the likelihood of a target fact is fundamentally an optimization process, distinct from rational belief revision ( 18 ) . Consequently, it remains unclear whether the local geometry of gradient-based updates can inherently guarantee such logical consistency without inducing contradictions. Encouraged by the impressive capabilities of current Large Language Models (LLMs) in logical reasoning during inference ( 23 ; 1 ) , it is tempting to assume that such coherence is preserved under local, first-order parameter updates. This premise, which we term the Linear Propagation Assumption (LPA) , often serves as a foundational design principle in current techniques. For instance, prominent knowledge editing methods explicitly formulate the update as a constrained linear least-squares problem, treating network layers as linear associative memories ( 3 ; 28 ; 29 ) . Beyond editing, this implicit assumption also appears in continual learning strategies that aim to add tasks without forgetting ( 27 ) and unlearning techniques designed to erase specific knowledge ( 21 ) . However, the validity of the LPA is questionable, as empirical failures of LLMs on logical coherence show. For instance, LLMs exhibit the “reversal curse,” failing to generalize to reverse relationship ( 4 ) , and struggle with compositional reasoning tasks ( 11 ) . Since these representations are constructed through first-order updates, such persistent failures suggest that the update mechanism may not reliably imprint the necessary logical structure. This limitation becomes explicitly visible in knowledge editing, where even carefully targeted updates consistently fail to propagate to logical consequences such as negations or implications ( 44 ; 8 ; 26 ) . These phenomena suggest a shared structural issue: the geometry of first-order updates imposes structural constraints that are inherently ill-suited for systematic logical operations. In this work, we rigorously investigate this hypothesis by asking: What structural constraints are imposed on a model’s representation if we demand that local linear updates respect the logical structure of relational knowledge? To answer this, we formalize relational knowledge using relation algebra ( 15 ) , which builds relational knowledge via three fundamental operations: negation (flipping truth values), converse (swapping argument order), and composition (chaining relations). Following Tarski’s invariance criterion for logical notions ( 38 ; 35 ) , we treat entity renamings as symmetries of the query space and ask what constraints such symmetries impose on the linearized geometry. This provides a principled way to test whether first-order parameter updates can support systematic propagation of logical operations. Parameter Manifold After update: \Delta s(\neg q)=-\Delta s(q) Tangent Space \theta \phi_{q} =\nabla s_{\theta}(q) \phi_{\neg q}\approx-\phi_{q} \theta+\Delta\theta Query ( q ) “T-Rex exists” Negated Query ( \neg q ) “T-Rex does not exist” Gradient ( \phi_{q} ) [ \rightarrow ] Inverted Gradient ( -\phi_{q} ) [ \leftarrow ] Log. Not \neg Linearize \phi Linearize \phi Sign Flip -I Figure 1 : Geometric interpretation of logical equivariance. (Left) A query q is associated with a score s_{\theta}(q) (e.g., log probability), and its gradient \phi_{q}=\nabla s_{\theta}(q) as a feature. Under the LPA, a local parameter update \Delta\theta induces a score change approximated by the inner product: \Delta s(q)=s_{\theta+\Delta\theta}(q)-s_{\theta}(q)\approx\langle\phi_{q},\Delta\theta\rangle . Logical consistency under direction-agnostic first-order propagation requires that any local parameter change enhancing q suppresses \neg q (i.e., \Delta s(\neg q)=-\Delta s(q) ), which necessitates that the gradient vectors be anti-aligned ( \phi_{\neg q}\approx-\phi_{q} ). (Right) This geometric requirement induces a commutative diagram where symbolic negation \neg in the query space corresponds to a linear inversion -I in the gradient feature space. Geometrically, satisfying these criteria requires navigating a fundamental trade-off between total superposition (uncontrollable interference) and perfect decoupling (lookup tables). Instead, we seek a structured coupling : for example, an update to p must automatically adjust \neg p ( Figure 1 ), while remaining linearly independent from unrelated facts. We translate this requirement into the geometric structure of gradients to formalize Systematic Linear Propagation (SLP) . SLP imposes strict logical equivariance on coupled facts, ensuring that linear updates systematically track unary logical transformations (negation and converse). Our theoretical analysis reveals several necessary conditions on the required geometry of first-order updates. For negation, we prove that guaranteeing direction-agnostic first-order propagation necessitates a tensor factorization separating entity-pair context from relation content ( Theorem 1 ), reminiscent of 37 . Moreover, converse requires its further decomposition into symmetric and antisymmetric components, constraining how argument order is represented ( Theorem 2 ). In contrast, we identify a fundamental obstruction when extending this systematicity to composition ( Theorem 3 ). We show that a minimal form of systematic composition reduces to conjunction, where a conjunction well-defined on linear features necessitates a bilinear structure. However, we demonstrate that this bilinearity is incompatible with the geometry enforced by negation equivariance. This conflict forces the feature map to collapse, suggesting that the failure to propagate updates to compositional consequences in the first-order regime may stem from a fundamental geometric mismatch. More broadly, our results support the view that dynamics reveals structure : analyzing how representations transform under updates exposes structural necessities that are invisible to static function approximation. This perspective contributes to the classical systematicity debate ( 13 ) by deriving binding-compatible block structure not as an architectural choice, but as a geometric necessity for logical coherence under LPA. Ultimately, this opens a path toward logical geometric deep learning ( 5 ) , treating logical operations not merely as static rules, but as dynamic symmetries to be preserved throughout learning. 2 Problem Formulation This section formalizes the problem. We first show empirically that current LLMs violate a basic requirement for negation consistency under LPA ( Section 2.1 ), then introduce relation algebra as our formalism ( Section 2.2 ) and linearized features as our geometric tool ( Section 2.3 ). These lead to our central definition: Systematic Linear Propagation (SLP) ( Section 2.4 ), which grounds the theorems in Section 3 .

Read the complete text