A Separable Architecture for Continuous Token Representation in Language Models
Reza T. Batley, Sourav Saha · 2026
Read A Separable Architecture for Continuous Token Representation in Language Models by Reza T. Batley, Sourav Saha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22040.txt.
Description
Reza Batley and Sourav Saha question how small language models spend their parameters. Scaling analyses usually treat all parameters as interchangeable, an assumption that predicts loss from compute well. In small models below a billion parameters, though, embedding matrices eat most of the parameter budget.
The authors argue that this allocation is as suboptimal as it is counterintuitive. Their 2026 preprint proposes a separable architecture for continuous token representation, splitting how tokens are represented so parameters go where they earn their keep. Small models run on phones and embedded devices, so squeezing more ability from the same budget has direct practical value.
Catalog record
| CREATOR | Reza T. Batley, Sourav Saha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22040