Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
Sridhar Mahadevan · 2026
Read Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning by Sridhar Mahadevan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27259.txt.
Introduction
Mahadevan offers a categorical language for Transformer builds. A layer is an extension operator: attention draws on token neighborhoods, geometric mixing on sparse links, and Kan extension on simplicial sources. It counts as a left Kan extension only when source values are functorial, weights representable, and the sum step realizes the coend. If not, Kan-style is a reading, not a proof. Predict-detach blocks gradients through a carrier, but alone does not make a noncausal update autoregressive, since carrier used at a target must be seen in its prefix.
Catalog record
| CREATOR | Sridhar Mahadevan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92432 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27259
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning Sridhar Mahadevan Abstract We propose Kan Extension Transformers (KETs) as a unifying categorical framework for a diverse group of Transformer implementations. The core claim is that a Transformer layer can be viewed as a weighted structured extension operator: standard attention is the singleton-neighborhood case, Geometric Transformer style incidence mixing is a sparse edge-restricted case, and KET is the higher-order simplicial case. This lens also clarifies a bridge to diffusion-style completion. When the extension operator acts on detached predictive carriers instead of teacher-forced hidden states, it becomes a valid self-conditioning mechanism that exposes noncausal structure without leaking gold future tokens. We include a comprehensive experimental validation of 12 12 different Transformer implementations varying across strict-causal and predict-detach regimes on Penn Treebank, WikiText-2, and WikiText-103. In the strict-causal setting, quadratic KET is the strongest model among the compared causal architectures on WikiText-2 and WikiText-103. Across all datasets, however, the largest gains come from the predict-detach regime rather than from changing the neighborhood family alone. 1 Introduction A foundational problem in machine learning, already visible in Gold’s formalization of language identification in the limit ( Gold 1967 ) , is generalization : extending information from a small or local domain to a larger structured domain. Category theory reframes this as the extension of a functor , a structured map on both objects and arrows, along a structural map ( Mac Lane 1971 ; Riehl 2017 ; Richter 2020 ) . Once source values and target structure are organized functorially, left and right Kan extensions provide principled extension operations. Attention extends token evidence into contextualized representations ( Vaswani et al. 2017 ) , while diffusion-style models refine partial or noisy structure through denoising ( Ho et al. 2020 ) . KETs give a common categorical language for both: once we choose source neighborhoods and values on them, a contextual update is a weighted extension from local objects to token positions. Attention uses singleton neighborhoods, Geometric Transformer style mixing uses sparse local incidences, and KET uses simplices and their incidences. This viewpoint matters most when we ask what values are extended. Ordinary teacher-forced hidden states can leak gold future information through noncausal neighborhoods. Detached predictive carriers computed from causal prefix states instead make the same operator a valid self-conditioning mechanism: the operator is unchanged, but observed hidden states are replaced by predicted-and-detached surrogates. We focus the paper around three claims. 1. KET is best understood as a weighted left-Kan-style extension over simplicial neighborhoods, with attention and incidence mixing as special cases. 2. Predict-detach gives a clean causal boundary: richer noncausal neighborhoods become usable only when the transported values are detached predictions rather than teacher-forced future states. 3. We provide a comprehensive experimental validation of 12 different Transformer models on three canonical benchmark domains: Penn Tree Bank (PTB), WikiText-2, and WikiText-103, strict-causal quadratic KET is competitive and strongest on the two larger corpora, but the dominant empirical effect comes from the predict-detach regime itself. 1.1 Background Materials This paper builds on six years of previous research on a categorical foundation for Artificial General Intelligence (AGI) – now available as a 600 -page book complete with a Lean-4 verification of the theoretical results – which contains a deeper theoretical study of Kan Extension Transformers ( Mahadevan 2025a ) . There is a chatbot CLIFF implemented using a novel deep learning language called Functorflow that provides implementation details of the Kan Extension Transformers (KETs) described in this paper ( Mahadevan 2025b ) . KET models were used in two state of the art deep causal research systems, Democritus ( Mahadevan 2025c ) and Prometheus ( Mahadevan 2026 ) . Supplementary Materials give the categorical details (Section 9 ), information-regime analysis for GT (Section 16 ), and reproducibility details. 2 Neighborhood Systems and Kan-Style Aggregation The model families studied in this paper differ less in the form of their update rule than in the structure of the neighborhoods over which they aggregate. Standard attention aggregates over tokens. TopoCoend aggregates over learned topological neighborhoods. Kan Extension Transformers aggregate over simplicial objects such as tokens, edges, and higher-order motifs. This section gives a common categorical language for these choices. Figure 1 illustrates the distinction: standard attention follows token-level neighborhoods, TopoCoend learns geometric adjacency in a latent space, and KET adds higher-order simplicial source objects such as edges and faces. t_{1} t_{2} t_{3} t_{4} (a) Token neighborhood: causal chain t_{1} t_{2} t_{3} t_{4} (b) Topological neighborhood: learned geometric adjacency t_{1} t_{2} t_{3} t_{4} e_{12} e_{24} \sigma_{123} (c) Simplicial neighborhood: higher-order objects Figure 1 : Three neighborhood systems. (a) Attention uses token-level neighborhoods. (b) TopoCoend adds learned geometric edges in a latent space. (c) KET adds higher-order simplices, so aggregation may occur over edges, faces, and motifs rather than only tokens. 2.1 A Common Setup Let \mathcal{T}=\{0,\dots,S-1\} denote the target token positions in a sequence window of length S . A model update at token t is built from a source neighborhood system \mathcal{N} , whose objects may be tokens, edges, simplices, or learned geometric neighbors. We write X:\mathcal{N}\to\mathbf{Vect} for the assignment of feature vectors to source objects. Here X(\sigma) may be a token state, edge embedding, or higher-order simplex value, and its contribution to target token t is determined by a structural weight W(t,\sigma) arising from attention scores, topological kernels, or simplicial incidence. This yields a common weighted aggregation form h^{\prime}_{t}\;\approx\;\int^{\sigma\in\mathcal{N}}W(t,\sigma)\otimes X(\sigma)\;\approx\;\sum_{\sigma\in\mathcal{N}}w(t,\sigma)\,V(\sigma), (1) where the coend notation emphasizes that the update is a structured weighted extension rather than an arbitrary sum (see Section 9 for more details on coend calculus). Equation ( 1 ) is the computational form shared by attention, TopoCoend, and KET. What changes from one model family to another is the choice of \mathcal{N} and the origin of the weights W(t,\sigma) . 2.2 Token Neighborhoods: Attention For standard self-attention, the source neighborhood system contains only token objects: \mathcal{N}_{\mathrm{attn}}=\{0,\dots,S-1\}. Each source object is a token s , and the update at target token t takes the familiar form h^{\prime}_{t}=\sum_{s}w(t,s)\,V_{s}. In categorical language, this is a weighted left-Kan-style extension in which source and target are both token positions; KET and TopoCoend generalize it by enriching the source neighborhood system. 2.3 Topological Neighborhoods: TopoCoend TopoCoend keeps the source objects at the token level but changes the neighborhood relation. Instead of using only positional adjacency or dense token-to-token attention, it learns a low-dimensional geometric representation z_{t}=\pi(v_{t}), where v_{t} is either a hidden state or a detached predictive carrier. A fuzzy k -nearest-neighbor graph is then constructed in the latent space of the z_{t} ’s. The resulting \mathcal{N}_{\mathrm{topo}} is a learned geometric graph whose weights W_{\mathrm{topo}}(t,s) are induced by topological proximity rather than sequence position alone. The corresponding update is h^{\prime}_{t}=h_{t}+\sum_{s}w_{\mathrm{topo}}(t,s)\,V_{s}. (2) TopoCoend is thus weighted aggregation over a learned topological neighborhood category. 2.4 Simplicial Neighborhoods: KET KET changes not just the weights but the source objects themselves: • 0 -simplices: tokens t , • 1 -simplices: edges such as (t-1,t) , • optional higher simplices: faces, motifs, or larger spans. Let \mathcal{N}_{\mathrm{simp}} denote this simplicial indexing category. Each simplex \sigma has value X(\sigma) , and its contribution to token t is controlled by W_{\mathrm{simp}}(t,\sigma) . The generic KET update is therefore