Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention
Alon Rozental [email protected] · 2026
Read Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention by Alon Rozental [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21768.txt.
Description
Alon Rozental argues that large language models are held back by the way they chop text into fixed pieces. Tools such as Byte Pair Encoding split words before learning starts, and that split cannot adapt during training. His 2026 preprint presents Zonkey, a model built to remove that fixed step.
Zonkey is a hierarchical diffusion model with a fully trainable path from raw characters up to representations of whole documents. Its tokenization can be learned along with the rest of the model, and its attention works in terms of probabilities. The aim is a system that adapts better to noisy text and to special fields where standard token splits fit poorly.
Catalog record
| CREATOR | Alon Rozental [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21768