MultiHashFormer: Hash-based Generative Language Models
Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras · 2026
Read MultiHashFormer: Hash-based Generative Language Models by Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28057.txt.
Description
Language models store a vector for every token in the vocabulary. That matrix grows linearly with vocabulary size. Hashing many tokens into one vector saves parameters but causes collisions that break causal language modelling. Huiyin Xue, Atsuki Yamaguchi, and Nikolaos Aletras propose MultiHashFormer in 2026 to make hashing work for generation. Each token gets a unique hash signature. Several independent hash functions produce its short sequence of hash IDs.
A hash encoder compresses the signature into one latent vector for the decoder. A hash decoder generates the next token's signature, which maps back to text. Models were evaluated at three scales, one hundred million, one billion, and three billion parameters. MultiHashFormer consistently outperformed standard transformer language models across benchmarks. Multilingual expansion works with a constant parameter footprint and no modifications. Rare word representation also improves, the conclusion reports.
Catalog record
| CREATOR | Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28057