A Separable Architecture for Continuous Token Representation in Language Models

Reza T. Batley, Sourav Saha · 2026

Read A Separable Architecture for Continuous Token Representation in Language Models by Reza T. Batley, Sourav Saha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22040.txt.

Description

Reza Batley and Sourav Saha question how small language models spend their parameters. Scaling analyses usually treat all parameters as interchangeable, an assumption that predicts loss from compute well. In small models below a billion parameters, though, embedding matrices eat most of the parameter budget.

The authors argue that this allocation is as suboptimal as it is counterintuitive. Their 2026 preprint proposes a separable architecture for continuous token representation, splitting how tokens are represented so parameters go where they earn their keep. Small models run on phones and embedded devices, so squeezing more ability from the same budget has direct practical value.

Catalog record

CREATORReza T. Batley, Sourav Saha
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22040

Browse the full library · About the Tower of Babel