Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention

Alon Rozental [email protected] · 2026

Read Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention by Alon Rozental [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21768.txt.

Description

Alon Rozental argues that large language models are held back by the way they chop text into fixed pieces. Tools such as Byte Pair Encoding split words before learning starts, and that split cannot adapt during training. His 2026 preprint presents Zonkey, a model built to remove that fixed step.

Zonkey is a hierarchical diffusion model with a fully trainable path from raw characters up to representations of whole documents. Its tokenization can be learned along with the rest of the model, and its attention works in terms of probabilities. The aim is a system that adapts better to noisy text and to special fields where standard token splits fit poorly.

Catalog record

CREATORAlon Rozental [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21768

Browse the full library · About the Tower of Babel