Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures
Emma Leonhart · 2026
Read Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures by Emma Leonhart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20919.txt.
Introduction
Emma Leonhart presents Sutra, a typed, fully functional programming language whose built forward pass is a PyTorch neural net. The compiler beta-shrinks the whole program, primitives, control flow, and string input-output, to one merged tensor-op graph over a fixed embedding substrate. Rotation binding, unbind, pack, polynomial Kleene three-valued rules, and tail-recursive loops all drop to tensor operations. Validation is one fact tested two ways, with the same program running on four frozen embeddings spanning two modalities.
Catalog record
| CREATOR | Emma Leonhart |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41412 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20919
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures Emma Leonhart Abstract Abstract Sutra is a typed, purely functional programming language whose compiled forward pass is a PyTorch neural network. The compiler beta-reduces the whole program — primitives, control flow, string I/O — to one fused tensor-op graph over a frozen embedding substrate. Rotation binding, unbind, bundle, polynomial Kleene three-valued logic, and tail-recursive loops all lower to tensor operations; the Kleene connectives are Lagrange-interpolated polynomials exact on the {−1, 0, +1} truth grid. Validation is one fact tested two ways. (1) The same program runs on four frozen embeddings spanning two modalities — three text encoders (nomic-embed-text, all-minilm, mxbai-embed-large) and one protein language model (ESM-2) — and decodes bundles at 100% accuracy through width k=8 on every substrate, where the textbook Hadamard product has already collapsed (2.5% on mxbai-embed-large, 7.5% on all-minilm). (2) PyTorch autograd flows through the actually compiled graph: a fuzzy-rule classifier written in .su trains from random init (18.7 ± 9.5%; chance = 20%, five classes) to 100.0 ± 0.0% (three seeds) by backpropagating through the emitted graph, the symbolic source unmodified. A weighted variant additionally trains a scalar cosine gain and writes it back into the .su source as a numeric literal; recompiling reproduces the trained behaviour to ≈ 2×10⁻⁷ per logit, so the trained model is itself legible, recompilable code. The same artifact is therefore both a logic program and a trainable neural network. 1 Introduction A frozen embedding model maps strings (or amino-acid sequences, or any other input the model was trained on) into a deterministic continuous vector space. Given such a substrate, two technical questions follow: 1. Which operations on these embeddings are reliable enough to be used as primitives of a compositional algebra over the substrate's vector space? 2. What is the correct binding operation? Hyperdimensional computing's textbook bind operators (Hadamard product, circular convolution) were derived assuming hypervectors drawn from a controlled random distribution. Frozen LLM embeddings are not such a distribution: they are strongly anisotropic , concentrating in a narrow cone so cosine similarity is compressed and inflated even between unrelated items (Ethayarajh 2019). §3.2 measures four substrates and reports that rotation binding decodes at 100% accuracy through bundle widths where Hadamard has already collapsed. This paper answers both questions in the form of a working programming language, Sutra , whose primitives are these consolidated operations and whose compiled forward pass is a PyTorch neural network. The naming: Sutra is the Sanskrit sūtra (thread, rule, aphorism), the term for Pāṇini's foundational Sanskrit grammar. 1.1 Contributions The four core technical contributions of this paper are: 1. Polynomial fuzzy logic via Lagrange interpolation of Kleene's three-valued truth tables. The truth axis encodes T=+1 , U=0 , F=-1 . On the discrete \{-1,0,+1\} grid, the Kleene connectives are \mathrm{AND}=\min , \mathrm{OR}=\max , \mathrm{NOT}=-\,\cdot\, . The min/max forms (the standard Gödel t-norm/t-conorm choice; Hájek 1998) are non-differentiable at the diagonal a=b , which breaks gradient flow when connectives compose with the tensor-op graph (van Krieken, Acar & van Harmelen 2022 survey the issue across t-norm-derived neural-symbolic operators). Sutra resolves this by Lagrange-interpolating each connective as a polynomial that is exact on the 3\times 3 Kleene grid and C^{\infty} elsewhere: \displaystyle\mathrm{AND}(a,b) \displaystyle=\tfrac{1}{2}(a+b+ab-a^{2}-b^{2}+a^{2}b^{2}) \displaystyle\mathrm{NAND}(a,b) \displaystyle=\tfrac{1}{2}(-a-b-ab+a^{2}+b^{2}-a^{2}b^{2}) \displaystyle\mathrm{OR}(a,b) \displaystyle=\tfrac{1}{2}(a+b-ab+a^{2}+b^{2}-a^{2}b^{2}) \displaystyle\mathrm{NOR}(a,b) \displaystyle=\tfrac{1}{2}(-a-b+ab-a^{2}-b^{2}+a^{2}b^{2}) \displaystyle\mathrm{NOT}(a) \displaystyle=-a \displaystyle\mathrm{XOR}(a,b) \displaystyle=-ab \displaystyle\mathrm{XNOR}(a,b) \displaystyle=ab {AND, OR, NOT} is functionally complete for the Kleene fragment; \mathrm{NAND} and \mathrm{NOR} are just -\mathrm{AND} and -\mathrm{OR} (negation is -\,\cdot\, ), and XOR/XNOR collapse to a single multiplicative term because their interpolant is zero whenever either input is U and bilinear in the {−1, +1} corners. Every Kleene-valid connective is therefore a polynomial tensor-op-graph fragment, gradient-compatible, branchless, and exact on the discrete-logic regime. A symbolic if-then rule built from these gates is one fused subgraph that PyTorch autograd backprops through end-to-end (§3.6). 2. Beta reduction to a substrate-pure tensor-op graph. The compiler inlines stdlib operator definitions, beta-reduces through bound names, then runs an algebraic-simplification pass over the residual. What's left is a fused tensor-op graph (matmul / element-wise / nonlinear) with no named bindings or function calls. Three concrete moves go beyond standard inlining + constant folding: conditionals lower to soft-mux polynomials ( \tfrac{1+\mathrm{cond}}{2}\,a+\tfrac{1-\mathrm{cond}}{2}\,b ) so the compiled artifact has no if opcodes; Haar-orthogonal binding rotations R_role are materialized at compile time so runtime bind is one matmul against a constant matrix; canonical synthetic axes are assigned compile-time so every primitive-type read/write is a known index, not a hashtable lookup. §4.2 traces this lowering stage-by-stage on a concrete program; the compilation pipeline as a whole is diagrammed in Figure~ 3 . 3. Tail recursion as the loop primitive. Loops are tail-recursive function declarations ( do_while , while_loop , iterative_loop , foreach_loop ) whose body's return NAME(args) becomes the recurrent step. Each loop compiles to a soft-halt RNN cell with substrate-pure halt detection (heaviside → cumulative monotone halt → soft-mux state freeze). The body of every loop tick is one straight-line tensor pipeline with no in-graph branches; a thin Python while True: … break driver wraps the body and terminates when the halt scalar saturates (§3.4). The state vector is fixed-width across iterations, O(1) state, O(N) compute, O(N) gradient tape during training , where N is iterations actually executed. 4. Synthetic-dimension rotation binding as an angular hash map. The compiler reserves a synthetic block of canonical dimensions and uses Haar-orthogonal rotations seeded from the role's content hash to bind keys to slots. We are unaware of any prior use of a high-dimensional rotation pattern as the substrate for a functional hash-map primitive. These four primitives integrate into a single working compiler that lowers .su source to a self-contained PyTorch module on CPU or CUDA. Program inputs and outputs are embeddings in the substrate's vector space; a compile-time codebook (implemented with an embedded vector database, §3.5) handles the convenience of source-level string literals and nearest-string output. 1.2 The substrate is the architecture target A Sutra program is compiled for an embedding-space architecture , the way a C program is compiled for x86 and a CUDA kernel for an NVIDIA SM. The embedding model fixes dimensionality, the geometry of the semantic block, and the meaning of every basis-vector lookup; swap the model and the same source recompiles to a different .sdb codebook against a different geometry. The substrate need not be an LLM, it can be any network producing a dense vector representation, including the hidden state of a trained model. §3.2's ESM-2 protein-LM row demonstrates this substrate-agnostically. 2 Related Work 2.1 Vector Symbolic Architectures VSA is a family of algebraic frameworks for computing with high- dimensional vectors (Kanerva 2009; Plate 1995; Gayler 2003). The standard VSA development assumes hypervectors drawn from a controlled random distribution designed for the algebra; bind is typically Hadamard product or circular convolution. Frozen LLM embedding spaces are not designed for VSA: they are anisotropic (Ethayarajh 2019; Gao et al. 2019; Mu et al. 2018) — representations concentrate in a narrow cone, so cosine similarity has low dynamic range and the textbook bind operations do not transfer cleanly. The same anisotropy makes an unweighted cosine read a weak rule signal, which is why §3.6 trains the embeddings the rules compare against rather than relying on raw cosine.