Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures

Emma Leonhart · 2026

Read Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures by Emma Leonhart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20919.txt.

Introduction

Emma Leonhart presents Sutra, a typed, fully functional programming language whose built forward pass is a PyTorch neural net. The compiler beta-shrinks the whole program, primitives, control flow, and string input-output, to one merged tensor-op graph over a fixed embedding substrate. Rotation binding, unbind, pack, polynomial Kleene three-valued rules, and tail-recursive loops all drop to tensor operations. Validation is one fact tested two ways, with the same program running on four frozen embeddings spanning two modalities.

Catalog record

CREATOREmma Leonhart
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41412
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sutra: Tensor-Op RNNs as a Compilation Target for Vector Symbolic Architectures Emma Leonhart Abstract Abstract Sutra is a typed, purely functional programming language whose compiled forward pass is a PyTorch neural network. The compiler beta-reduces the whole program — primitives, control flow, string I/O — to one fused tensor-op graph over a frozen embedding substrate. Rotation binding, unbind, bundle, polynomial Kleene three-valued logic, and tail-recursive loops all lower to tensor operations; the Kleene connectives are Lagrange-interpolated polynomials exact on the {−1, 0, +1} truth grid. Validation is one fact tested two ways. (1) The same program runs on four frozen embeddings spanning two modalities — three text encoders (nomic-embed-text, all-minilm, mxbai-embed-large) and one protein language model (ESM-2) — and decodes bundles at 100% accuracy through width k=8 on every substrate, where the textbook Hadamard product has already collapsed (2.5% on mxbai-embed-large, 7.5% on all-minilm). (2) PyTorch autograd flows through the actually compiled graph: a fuzzy-rule classifier written in .su trains from random init (18.7 ± 9.5%; chance = 20%, five classes) to 100.0 ± 0.0% (three seeds) by backpropagating through the emitted graph, the symbolic source unmodified. A weighted variant additionally trains a scalar cosine gain and writes it back into the .su source as a numeric literal; recompiling reproduces the trained behaviour to ≈ 2×10⁻⁷ per logit, so the trained model is itself legible, recompilable code. The same artifact is therefore both a logic program and a trainable neural network. 1 Introduction A frozen embedding model maps strings (or amino-acid sequences, or any other input the model was trained on) into a deterministic continuous vector space. Given such a substrate, two technical questions follow: 1. Which operations on these embeddings are reliable enough to be used as primitives of a compositional algebra over the substrate's vector space? 2. What is the correct binding operation? Hyperdimensional computing's textbook bind operators (Hadamard product, circular convolution) were derived assuming hypervectors drawn from a controlled random distribution. Frozen LLM embeddings are not such a distribution: they are strongly anisotropic , concentrating in a narrow cone so cosine similarity is compressed and inflated even between unrelated items (Ethayarajh 2019). §3.2 measures four substrates and reports that rotation binding decodes at 100% accuracy through bundle widths where Hadamard has already collapsed. This paper answers both questions in the form of a working programming language, Sutra , whose primitives are these consolidated operations and whose compiled forward pass is a PyTorch neural network. The naming: Sutra is the Sanskrit sūtra (thread, rule, aphorism), the term for Pāṇini's foundational Sanskrit grammar. 1.1 Contributions The four core technical contributions of this paper are: 1. Polynomial fuzzy logic via Lagrange interpolation of Kleene's three-valued truth tables. The truth axis encodes T=+1 , U=0 , F=-1 . On the discrete \{-1,0,+1\} grid, the Kleene connectives are \mathrm{AND}=\min , \mathrm{OR}=\max , \mathrm{NOT}=-\,\cdot\, . The min/max forms (the standard Gödel t-norm/t-conorm choice; Hájek 1998) are non-differentiable at the diagonal a=b , which breaks gradient flow when connectives compose with the tensor-op graph (van Krieken, Acar & van Harmelen 2022 survey the issue across t-norm-derived neural-symbolic operators). Sutra resolves this by Lagrange-interpolating each connective as a polynomial that is exact on the 3\times 3 Kleene grid and C^{\infty} elsewhere: \displaystyle\mathrm{AND}(a,b) \displaystyle=\tfrac{1}{2}(a+b+ab-a^{2}-b^{2}+a^{2}b^{2}) \displaystyle\mathrm{NAND}(a,b) \displaystyle=\tfrac{1}{2}(-a-b-ab+a^{2}+b^{2}-a^{2}b^{2}) \displaystyle\mathrm{OR}(a,b) \displaystyle=\tfrac{1}{2}(a+b-ab+a^{2}+b^{2}-a^{2}b^{2}) \displaystyle\mathrm{NOR}(a,b) \displaystyle=\tfrac{1}{2}(-a-b+ab-a^{2}-b^{2}+a^{2}b^{2}) \displaystyle\mathrm{NOT}(a) \displaystyle=-a \displaystyle\mathrm{XOR}(a,b) \displaystyle=-ab \displaystyle\mathrm{XNOR}(a,b) \displaystyle=ab {AND, OR, NOT} is functionally complete for the Kleene fragment; \mathrm{NAND} and \mathrm{NOR} are just -\mathrm{AND} and -\mathrm{OR} (negation is -\,\cdot\, ), and XOR/XNOR collapse to a single multiplicative term because their interpolant is zero whenever either input is U and bilinear in the {−1, +1} corners. Every Kleene-valid connective is therefore a polynomial tensor-op-graph fragment, gradient-compatible, branchless, and exact on the discrete-logic regime. A symbolic if-then rule built from these gates is one fused subgraph that PyTorch autograd backprops through end-to-end (§3.6). 2. Beta reduction to a substrate-pure tensor-op graph. The compiler inlines stdlib operator definitions, beta-reduces through bound names, then runs an algebraic-simplification pass over the residual. What's left is a fused tensor-op graph (matmul / element-wise / nonlinear) with no named bindings or function calls. Three concrete moves go beyond standard inlining + constant folding: conditionals lower to soft-mux polynomials ( \tfrac{1+\mathrm{cond}}{2}\,a+\tfrac{1-\mathrm{cond}}{2}\,b ) so the compiled artifact has no if opcodes; Haar-orthogonal binding rotations R_role are materialized at compile time so runtime bind is one matmul against a constant matrix; canonical synthetic axes are assigned compile-time so every primitive-type read/write is a known index, not a hashtable lookup. §4.2 traces this lowering stage-by-stage on a concrete program; the compilation pipeline as a whole is diagrammed in Figure~ 3 . 3. Tail recursion as the loop primitive. Loops are tail-recursive function declarations ( do_while , while_loop , iterative_loop , foreach_loop ) whose body's return NAME(args) becomes the recurrent step. Each loop compiles to a soft-halt RNN cell with substrate-pure halt detection (heaviside → cumulative monotone halt → soft-mux state freeze). The body of every loop tick is one straight-line tensor pipeline with no in-graph branches; a thin Python while True: … break driver wraps the body and terminates when the halt scalar saturates (§3.4). The state vector is fixed-width across iterations, O(1) state, O(N) compute, O(N) gradient tape during training , where N is iterations actually executed. 4. Synthetic-dimension rotation binding as an angular hash map. The compiler reserves a synthetic block of canonical dimensions and uses Haar-orthogonal rotations seeded from the role's content hash to bind keys to slots. We are unaware of any prior use of a high-dimensional rotation pattern as the substrate for a functional hash-map primitive. These four primitives integrate into a single working compiler that lowers .su source to a self-contained PyTorch module on CPU or CUDA. Program inputs and outputs are embeddings in the substrate's vector space; a compile-time codebook (implemented with an embedded vector database, §3.5) handles the convenience of source-level string literals and nearest-string output. 1.2 The substrate is the architecture target A Sutra program is compiled for an embedding-space architecture , the way a C program is compiled for x86 and a CUDA kernel for an NVIDIA SM. The embedding model fixes dimensionality, the geometry of the semantic block, and the meaning of every basis-vector lookup; swap the model and the same source recompiles to a different .sdb codebook against a different geometry. The substrate need not be an LLM, it can be any network producing a dense vector representation, including the hidden state of a trained model. §3.2's ESM-2 protein-LM row demonstrates this substrate-agnostically. 2 Related Work 2.1 Vector Symbolic Architectures VSA is a family of algebraic frameworks for computing with high- dimensional vectors (Kanerva 2009; Plate 1995; Gayler 2003). The standard VSA development assumes hypervectors drawn from a controlled random distribution designed for the algebra; bind is typically Hadamard product or circular convolution. Frozen LLM embedding spaces are not designed for VSA: they are anisotropic (Ethayarajh 2019; Gao et al. 2019; Mu et al. 2018) — representations concentrate in a narrow cone, so cosine similarity has low dynamic range and the textbook bind operations do not transfer cleanly. The same anisotropy makes an unweighted cosine read a weak rule signal, which is why §3.6 trains the embeddings the rules compare against rather than relying on raw cosine.

Read the complete text