A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Models

Sridhar Mahadevan · 2025

Read A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Models by Sridhar Mahadevan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10018.txt.

Introduction

Sridhar Mahadevan's 2025 paper takes up a basic flaw in large language models. Two sentences can mean the same thing, such as one saying Darwin wrote a book and one calling him its author. A model should give both the same odds for what comes next, yet in practice it often does not.

Mahadevan builds a math framework for this problem using category theory and homotopy theory. He models the odds a model assigns to word sequences, then marks which unlike forms should count as equal in meaning. The paper maps out how tools from higher algebra could give models a firmer grip on sameness of meaning.

Catalog record

CREATORSridhar Mahadevan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10018

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Models Thanks: Draft under review. Sridhar Mahadevan Abstract Natural language is replete with superficially different statements, such as “Charles Darwin wrote" and “Charles Darwin is the author of", which carry the same meaning. Large language models (LLMs) should generate the same next-token probabilities in such cases, but usually do not. Empirical workarounds have been explored, such as using k k -NN estimates of sentence similarity to produce smoothed estimates. In this paper, we tackle this problem more abstractly, introducing a categorical homotopy framework for LLMs. We introduce an LLM Markov category to represent probability distributions in language generated by an LLM, where the probability of a sentence, such as “Charles Darwin wrote" is defined by an arrow in a Markov category. However, this approach runs into difficulties as language is full of equivalent rephrases, and each generates a non-isomorphic arrow in the LLM Markov category. To address this fundamental problem, we use categorical homotopy techniques to capture “weak equivalences" in an LLM Markov category. We present a detailed overview of application of categorical homotopy to LLMs, from higher algebraic K-theory to model categories, building on powerful theoretical results developed over the past half a century. K eywords Large Language Models \cdot Category Theory \cdot Homotopy Theory \cdot AI \cdot \cdot Machine Learning 1 Introduction In recent years, large language models (LLMs) have revolutionized AI, building on the success of the Transformer model [ Vaswani et al. 2017 ] and related structured state space sequence models [ Gu et al. 2022 ] . LLMs have enabled building large foundation models using massive amounts of data [ Bommasani et al. 2022 ] , which has implications for whether artificial general intelligence (AGI) [ Morris et al. 2023 ] may be achievable in the near term. Recent advances, such as DeepSeek-R1 [ DeepSeek-AI et al. 2025 ] show that significant gains in the computational cost of building these models can be achieved by structuring the problem using trial-and-error search based on reinforcement learning [ Sutton and Barto 1998 ] . However, much remains unclear about exactly what these systems are learning. Also, existing theoretical results regarding LLMs, on the one hand show they cannot recognize Dyck languages or compute the parity function [ Hahn 2020 , Merrill et al. 2022 ] , and on the other hand show that they are a universal function approximator over sequences [ Yun et al. 2020 ] Our approach is inspired by categorical models of language [ Asudeh and Giorgolo 2020 , Bradley et al. 2022 , Coecke 2019 ] , however these approaches do not address semantic homotopy: when do two seemingly different sentences in language “mean" the same thing? For example, Bradley et al. 2022 define a syntactic category (see Definition 4 ) and a semantic category (see Definition 5 ) for LLMs where objects are fragments of sentences x and morphisms x\rightarrow y represent possible extensions of fragment x by y . The set of possible completions is represented as an object in an enriched monoidal preorder category of the unit interval [0,1] , where logical connectives such as implication are defined as an adjoint functors. Merrill et al. 2024 investigate whether implications are indeed learnable from co-occurrence patterns in LLMs. Neither of these studies focus on homotopy. A large body of work exists around modeling language with symmetric monoidal categories in quantum NLP [ Coecke 2020 , Coecke et al. 2010 ] , which converts natural language into circuits for use with quantum computation [ Coecke and Kissinger 2017 ] . A Python package exists called lambeq [ Kartsaklis et al. 2021 ] .The quantum NLP work has not addressed the issue of homotopy. Their approach builds on a compositional theory of language defined by Lambek pregroup grammars, a type of category defined by a partially ordered monoid [ Lambek 2006 ] . LLMs encode sentence prefixes to a fixed-sized representations and predict the next word in the text using this encoding. But, such next-token predictions are challenging for natural language paraphrases: for example, “Charles Dickens is the author of" should generate the same next-token distribution as “Charles Dickens wrote". Khandelwal et al. 2020 use nearest-neighbor methods to improve an LLM’s performance at this type of next-token prediction problem. Our work proposes a rigorous homotopy theory to analyze similar text fragments. Our homotopic framework is also related to research on LLM model distillation [ Hsieh et al. 2023 ] , but these approaches usually rely on supervised training to construct smaller task-specific models, whereas we are proposing an internal analysis of lower-dimensional manifolds that the model parameters lie on. Finally, our work is related to work that uses logprob or logit information from commercial LLM implementations to reveal internal architectural details [ Finlayson et al. 2024 ] by finding linear lower-dimensional matrix manifolds that the logit scores lie on, but our approach uses more powerful nonlinear homotopic methods. To explain the main idea, we propose to model LLMs using the framework of a Markov category [ Fritz 2020 ] , a symmetric monoidal category with a comonoidal “copy-delete" structure on each object. Such Markov categories have been shown recently to be capable of a wide range of intricate reasoning in probability and statistics, and a diagrammatic foundation that compares favorably in ease of use to the alternative “assembly language" measure theory foundation [ Halmos 1974 ] . A probability distribution over a natural language phrase is encoded in a Markov category as the arrow f:\psi\rightarrow\text{Charles Darwin wrote} However, an immediate dilemma that an LLM Markov category by itself is incapable of recognizing that the paraphrase g:\mu\rightarrow\text{Charles Darwin is the author of} should be treated as “isomorphic" to the previous arrow. In other words, in an LLM Markov category, many objects exist that appear syntactically different, but are semantically identical. We address this fundamental problem by introducing categorical homotopy techniques into LLMs, particularly the use of categorical techniques for modeling them. 2 Homotopy in LLMs Figure 1: A lifting diagram for LLMs: any such commutative diagram has a “lift" h^{\prime} if its composed morphisms g\circ i=p\circ f . Applied to LLMs, it asks whether the similarity between text sentences, i.e. is one fragment an equivalent rephrasing of another, can be ascribed topological “meaning" in terms of how a circle fits into a cylinder when moved over the unit interval path [0,1] . Homotopy means deciding if two arbitrary objects are in some sense equivalent. For example, two real-valued functions f,g:X\rightarrow\mathbb{R} are considered homotopic if there is a continuous mapping h:[0,1]\times X\rightarrow\mathbb{R} such that h(0,x)=f(x) and h(1,x)=g(x) . If you imagine plotting f and g , then the mental picture is of smoothly perturbing the graph of f to align it with the graph of g . We want to define homotopy over LLM categories that capture “syntax" and “semantics" (Definition 4 to Definition 6 ). Quillen 1967 proposed model categories as a way to do abstract homotopy in any category, which we will review in Section 7.3 . We prove in Section 7 that LLMs define model categories. The concept of homotopy originally arose in algebraic topology as

Read the complete text