GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models
Tianyu Bell Pan, Damon L. Woodard, [email protected] · 2026
Read GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models by Tianyu Bell Pan, Damon L. Woodard, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19460.txt.
Introduction
Pan and Woodard train language models to keep their inner token paths spread and well shaped. Their GeoLAN treats token states as geometric paths through the model. Two new regularizers, drawn from Kakeya-style math, push for isotropy and varied attention. Tests span Gemma-3 at three sizes and Llama-3 at 8 billion weights. Accuracy mostly holds while geometric scores improve under the training. Some fairness biases shrink as the geometry improves. Gains peak in mid-sized models, which shows a scale trade-off.
Catalog record
| CREATOR | Tianyu Bell Pan, Damon L. Woodard, [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 115546 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19460
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models Tianyu Bell Pan, Damon L. Woodard, [email protected] Abstract Large language models (LLMs) demonstrate strong performance, but they often lack transparency. We introduce GeoLAN, a training framework that treats token representations as geometric trajectories and applies stickiness conditions inspired by recent developments related to the Kakeya Conjecture. We have developed two differentiable regularizers, Katz-Tao Convex Wolff (KT-CW) and Katz-Tao Attention (KT-Attn), that promote isotropy and encourage diverse attention. Our experiments with Gemma-3 (1B, 4B, 12B) and Llama-3-8B show that GeoLAN frequently maintains task accuracy while improving geometric metrics and reducing certain fairness biases. These benefits are most significant in mid-sized models. Our findings reveal scale-dependent trade-offs between geometric precision and performance, suggesting that geometry-aware training is a promising approach to enhance mechanistic interpretability. 1 Introduction The deployment of large language models (LLMs) in critical domains, such as legal adjudication, medical diagnostics, and autonomous code generation, has precipitated an urgent interpretability crisis. While these models perform exceptionally well on benchmarks such as MMLU ( 10 ) and GSM8K ( 4 ) , their internal logic operates as a black box, obscuring the causal relationships between inputs and outputs ( 16 ) . This lack of transparency is not just an academic issue; it poses systemic risks regarding inspection, accountability, and safety ( 35 ; 21 ) . When a model fabricates a legal precedent or misdiagnoses a patient, users currently lack the means to audit the reasons behind these errors and often have to rely on the model’s potentially fabricated explanations ( 8 ) . A key challenge in tackling this crisis is the inherent limitations of existing interpretability methods. Techniques such as attention visualization, gradient-based saliency maps, and causal mediation analysis provide post hoc explanations ( 18 ; 3 ) . These methods attach plausible human rationales to model outputs without ensuring fidelity to the model’s internal mechanisms. More advanced mechanistic interpretability methods, such as sparse autoencoders (SAEs) and dictionary learning, aim to decompose representations into monosemantic features ( 20 ; 24 ; 5 ; 31 ) . However, these methods face a challenge known as superposition, in which neural networks can represent more features than they have neurons by encoding them in non-orthogonal directions ( 11 ) . This phenomenon results in individual neurons becoming polysemantic, responding to multiple concepts (e.g., a neuron may activate to both “cats” and “biblical verses”), complicating efforts to achieve clear decomposition. In addition to these interpretability challenges, there is the issue of representation degradation, particularly anisotropy ( 11 ; 17 ) . Research has consistently demonstrated that the embedding spaces of standard Transformers are highly anisotropic, meaning that most of the variance is concentrated in a few dominant directions, effectively collapsing the representation into a narrow cone ( 26 ) . This geometric collapse has two harmful effects: it limits the model’s effective capacity by underutilizing available dimensional space, and it worsens entanglement, forcing distinct semantic concepts to share a limited subspace. Recent studies using metrics such as IsoScore have confirmed that this anisotropy correlates with a diminished downstream performance and model brittleness ( 27 ) . Although some architectures, such as Pythia ( 1 ) , have shown improved isotropy, the issue persists across most optimized causal language models. More related work is presented in Appendix B . To address these combined challenges, we propose GeoLAN, a framework that treats interpretability as a geometric constraint to be enforced during training rather than discovered post hoc. We hypothesize that preventing representation collapse and compelling the model to use its full geometric space through regularization can yield a disentangled latent structure. Our approach draws inspiration from the Kakeya needle problem in geometric measure theory, particularly the recent resolution of the Kakeya conjecture in \mathbb{R}^{3} , which introduced the concept of sticky Kakeya sets ( 32 ) . By viewing token representations not as static points but as continuous trajectories or tubes moving through the network layers, we adapt the mathematical framework of Kakeya sets to the linguistic domain. The following core questions guide our research: RQ1: Can we mathematically formalize the “messiness” of latent spaces using geometric measure theory? RQ2: Can we derive differentiable loss functions that enforce “stickiness” (non-clustering of features) without destroying model performance? RQ3: Does this geometric intervention actually yield more interpretable, stable, and disentangled representations in practice? The key contributions of this paper are as follows: (1) A geometric theory for internal explainability: By formalizing layer-wise token representations as geometric trajectories, we propose a “stickiness” condition that allows the latent space to be decomposed into low-interference components. This approach provides a mechanism-level perspective on explainability, focusing on the model’s internal workings. (2) Training-time geometric regularization: We introduce two complementary, differentiable losses that operationalize the theory: one that penalizes hidden-state collapse via correlation structure, and one that discourages low-rank attention routing. (3) Explainability-align evaluation protocol: We propose a comprehensive evaluation suite that connects geometry to explainability goals (decomposability into stable directions, sensitivity under controlled perturbations, and causal intervention behavior), addressing faithfulness and plausibility. (4) Empirical characterization across scales: We present evidence from analyzing multiple modern LLMs that geometric constraints can often maintain task performance while enhancing specific geometric and explainability metrics. Additionally, we identify scale-dependent scenarios, specifically a scale floor at which geometric constraints conflict with the necessary superposition in small models. 2 Theoretical Framework To thoroughly examine the geometry of latent space, we move beyond a discrete, layer-level perspective and regard information flow as a continuous geometric process. We introduce a formalism that maps the internal states of a Transformer to tubes in a high-dimensional manifold, enabling us to apply theorems from geometric measure theory. 2.1 Preliminaries: Token Trajectories and Representation Fields Let V be a finite vocabulary and S=\left(w_{1},\ldots,w_{N}\right) be an input sequence of length N . A Transformer model M maps S to a sequence of hidden representations Z^{(l)}\in\mathbb{R}^{N\times d} across layers l=0,\ldots,L . While layers are discrete, the residual connection structure z^{(l+1)}=z^{(l)}+f\left(z^{(l)}\right) mimics a discretized ordinary differential equation (ODE). We treat the layer index as a discrete time parameter t_{l}=\frac{l}{L}\in[0,1] . Definition 1 (Token Trajectory). The evolution of the i -th token is modeled as a differentiable curve \gamma_{i}:[0,1]\to\mathbb{R}^{d} such that \gamma_{i}(t_{l})\approx z_{i}^{(l)} . The derivative \gamma^{\prime}_{i}(t) represents the semantic velocity, or the instantaneous update vector applied by attention and feed-forward networks at depth t . Ideally, distinct tokens should follow distinct trajectories that do not collapse into a single manifold. Definition 2 (Token Tube). To capture the volume of space occupied by a token’s path and its immediate semantic neighborhood, we define the \delta -tube of token i as the set of points in spacetime within a radius \delta of its trajectory: \displaystyle T_{i}^{\delta}=\{(x,t)\in\mathbb{R}^{d}\times:\|x-\gamma_{i}(t)\|<\delta\}. (1) At any fixed layer l , the cross-section of this tube is the instantaneous tube T(z_{i}^{(l)}) (within angular threshold \Delta ). This geometric object represents the semantic footprint of a token. Definition 3 (Representation Field). The representation field at layer l for context S is the collection of all instantaneous tubes: \displaystyle\Phi_{l}(S)=\{T(z_{i}^{(l)}):i=1,\dots,N\}. (2) This field represents the instantaneous semantic geometry. A clean field consists of well-separated tubes (disentanglement), while a cluttered field features significant overlap (entanglement/anisotropy) where multiple tokens occupy the