MultiHashFormer: Hash-based Generative Language Models

Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras · 2026

Read MultiHashFormer: Hash-based Generative Language Models by Huiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28057.txt.

Description

Language models store a vector for every token in the vocabulary. That matrix grows linearly with vocabulary size. Hashing many tokens into one vector saves parameters but causes collisions that break causal language modelling. Huiyin Xue, Atsuki Yamaguchi, and Nikolaos Aletras propose MultiHashFormer in 2026 to make hashing work for generation. Each token gets a unique hash signature. Several independent hash functions produce its short sequence of hash IDs.

A hash encoder compresses the signature into one latent vector for the decoder. A hash decoder generates the next token's signature, which maps back to text. Models were evaluated at three scales, one hundred million, one billion, and three billion parameters. MultiHashFormer consistently outperformed standard transformer language models across benchmarks. Multilingual expansion works with a constant parameter footprint and no modifications. Rare word representation also improves, the conclusion reports.

Catalog record

CREATORHuiyin Xue, Atsuki Yamaguchi, Nikolaos Aletras
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28057

Browse the full library · About the Tower of Babel