Esoteric Language Models

Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, Arash Vahdat · 2025

Read Esoteric Language Models by Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, Arash Vahdat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01928.txt.

Description

Subham Sekhar Sahoo and nine co-authors introduce a family of language models that blends two generation styles, in a 2025 paper. Diffusion models write text in parallel and allow control, but the best of them still trail ordinary left-to-right models in quality. They also cannot reuse past computations at inference time, a trick called KV caching that speeds up generation.

The new models, called Eso-LMs, fuse the two styles and slide smoothly between them. The key move is using causal attention, borrowed from ordinary models, inside the diffusion setup. That change lets them compute exact likelihoods for the first time and enables KV caching while keeping parallel generation. With a tuned sampling schedule, the models set a new best trade-off between speed and quality for unconditional text generation. Code and checkpoints are public.

Catalog record

CREATORSubham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, Arash Vahdat
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01928

Browse the full library · About the Tower of Babel