Modular Linear Tokenization (MLT)

Tcharlies Schmitz · 2025

Read Modular Linear Tokenization (MLT) by Tcharlies Schmitz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25952.txt.

Introduction

Tcharlies Schmitz, working alone, introduces a reversible way to turn categorical identifiers into compact number vectors. Identifiers with many possible values, such as user or movie IDs, are often encoded by hashing, which can collide, or by one hot vectors, which are large. His method, Modular Linear Tokenization, uses modular arithmetic over finite fields and invertible linear maps to keep a one to one link.

Schmitz reports tests on the MovieLens 20M data set, a large set of movie ratings. The method matched the predictive performance of learned supervised embeddings, which are vectors trained with the task, while using far fewer parameters and lower training cost. Code is released as an open source package for Python users.

Catalog record

CREATORTcharlies Schmitz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25952

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modular Linear Tokenization (MLT) Tcharlies Schmitz Abstract Abstract. This paper introduces Modular Linear Tokenization (MLT), a reversible and deterministic technique for encoding high-cardinality categorical identifiers into compact numerical vectors. Unlike traditional hashing or one-hot encodings, MLT preserves bijective mappings by leveraging modular arithmetic over finite fields and invertible linear transformations. The method offers explicit control of dimensionality and computational scalability while maintaining full reversibility, even for millions of identifiers. Experimental results on the MovieLens 20M dataset show that MLT achieves comparable predictive performance to supervised embeddings while requiring significantly fewer parameters and lower training cost. An open-source implementation of MLT is available on PyPI ( https://pypi.org/project/light-mlt/ ) and GitHub ( https://github.com/tcharliesschmitz/light-mlt ). Keywords: Modular Linear Tokenization, reversible encoding, categorical data, finite fields, efficient preprocessing. 1 Introduction The way we represent categorical and textual data is decisive for the success of machine learning models—especially in deep architectures. In high-cardinality scenarios, such as recommendation systems, user/item classification, or large vocabularies, the tokenization stage often becomes a bottleneck: it consumes large amounts of memory, increases computational cost, and limits generalization capacity. This work presents the Modular Linear Tokenization (MLT), a deterministic technique that transforms integer identifiers into compact vectors while preserving exact reversibility. The proposal combines three key advantages: (i) explicit control over dimensionality, (ii) a significant reduction in output cost for tasks with a large number of labels, and (iii) the absence of collisions—something not guaranteed by methods such as hashing . Historically, the first approaches date back to the 1950s–1960s with one-hot and bag-of-words (BoW) representations [ 1 ] . These encodings are simple but produce extremely large and sparse vectors whose size grows linearly with the vocabulary. In the 1970s, TF–IDF introduced a statistical notion to frequency counts but maintained high dimensionality [ 2 ] . Between the 1980s and 1990s, n-grams and count-based models emerged, capturing local context but suffering from combinatorial explosion for large n [ 3 ] . In the 2000s, within the tabular domain, encodings such as label encoding (fast but prone to artificial ordering) and supervised variants such as target / count encoding (more informative but subject to leakage) appeared. In 2009, the hashing trick offered a way to project categories into a fixed-dimensional space, stabilizing production pipelines but introducing collisions and losing reversibility [ 4 ] . The following decade consolidated learned distributed embeddings, such as word2vec (2013) [ 5 ] and GloVe (2014) [ 6 ] , which captured semantics in dense vectors. fastText (2016–2017) incorporated morphology through character n-grams [ 7 ] . In NLP, tokenization evolved from whole words to subwords: BPE (2015) [ 8 ] and WordPiece (2016) [ 9 ] balanced robustness and coverage, followed by SentencePiece (2018), which unified vocabularies and normalization [ 10 ] . From 2019 onward, byte-level variants (e.g., GPT-2) and tokenizer-free models (ByT5, 2021) sought to simplify vocabularies [ 11 ] . Between 2022 and 2024, engineering practices consolidated unified normalizers, compact multilingual vocabularies, and high-performance tokenizers, while recent research explores hybrid schemes, domain specialization, and even reversible representations. 1 1 1 See recent proceedings from ACL, EMNLP, NAACL conferences (2022-2024) for in-depth studies on state-of-the-art tokenizers and normalizers. Despite these advances, three gaps remain: 1. the lack of reversibility and the risk of collisions in efficient methods such as hashing ; 2. the absence of explicit control over the relationship between dimensionality and cost; 3. the difficulty of reducing the cost of output layers when the number of classes is very large. MLT positions itself precisely in this space. By representing identifiers in base- p (with p prime) and applying invertible modular linear transformations, it guarantees a bijective mapping, direct control of dimensionality, and a natural decomposition of the output into n softmax layers of size p , instead of a single softmax of size V . The objective of this paper is to formalize MLT, present its properties—uniqueness, reversibility, and dimensionality control—and discuss its practical impact on training and inference costs in high-cardinality scenarios. The contributions include: the mathematical formulation and encoding/decoding algorithms, guidelines for selecting p and n according to the vocabulary size, and the analysis of its integration into deep architectures, with emphasis on output cost reduction. 2 Methodology The idea behind Modular Linear Tokenization (MLT) is straightforward: to transform high-cardinality integer identifiers (such as user or contract IDs) into smaller vectors without losing the ability to recover the original value. Unlike hashing techniques, which may produce collisions, MLT is fully reversible. To achieve this, the method relies on three pillars: (i) representation in base- p , (ii) the application of an invertible matrix over this vector, and (iii) the guarantee of uniqueness in representations. The starting point is the conversion of an identifier into base- p . Instead of handling a large integer, the value is decomposed into a fixed-length vector of digits. Each digit takes values between 0 and p-1 , with p chosen as a prime number to facilitate subsequent operations. For this representation to be sufficient, the space of combinations must be larger than the vocabulary size V (i.e., the maximum number of categories to represent): p^{n}>V (1) Next comes the matrix M , defined over the finite field \mathbb{Z}_{p} . This matrix must be invertible, meaning that its determinant is nonzero modulo p : \det(M)\not\equiv 0\pmod{p} (2) The role of the matrix is to densely mix the digits, spreading information across the vector’s positions. This prevents numerically close identifiers from producing similar representations, thus enriching the encoding. In practice, each final component of the vector depends on multiple original positions, ensuring greater robustness. The tokenization process therefore follows a simple sequence: 1. convert the identifier to base- p ; 2. multiply the resulting vector by matrix M ; 3. reduce the result modulo p . The final token vector is defined as: t=(M\cdot v)\bmod p (3) This vector can then be used directly as input to machine learning models. Decoding consists of applying the inverse process. The inverse matrix M^{-1} is computed in \mathbb{Z}_{p} , and the inverse transformation is applied: v=(M^{-1}\cdot t)\bmod p (4) Finally, the vector v is reconverted from base- p into its integer form, recovering the original identifier. In practical terms, encoding can be summarized as: convert \rightarrow multiply \rightarrow reduce modulo p \rightarrow obtain tokens. Decoding follows the reverse path. Both steps involve only basic modular arithmetic operations, making them fast and scalable even in extremely high-cardinality scenarios. Among the most relevant properties of MLT are: • Reversibility : ensures a bijective mapping between IDs and token vectors, with no collisions; • Dimensional control : parameters p and n allow explicit calibration of computational cost as a function of vocabulary size; • Efficiency : encoding and decoding involve only simple operations, enabling large-scale deployment of the technique. Regarding computational complexity, Modular Linear Tokenization (MLT) is linear with respect to the number of digits n in the base- p representation, that is, \mathcal{O}(n) . Each encoding or decoding operation involves only the conversion of the identifier into a fixed-length vector and a vector–matrix multiplication modulo p . Since both steps depend solely on n and not on the total vocabulary size V , the cost remains proportional to the vector’s dimension,

Read the complete text