Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective

Siyue Zhang N {}^{\hskip 0.81949pt{{\color[rgb]{0.0078,0.0078,0.7852}\boldsymbol{N}}}} A {}^{\hskip 0.81949pt{{\color[rgb]{1,0.4141,0}\boldsymbol{A}}}} Yilun Zhao Y {}^{\hskip 0.81949pt{\color[rgb]{0.7031,0.6914,0.0156}\boldsymbol{Y}}} Liyuan Geng S {}^{\hskip 0.81949pt{\color[rgb]{0.5273,0.0039,0.6 · 2025

Read Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective by Siyue Zhang N {}^{\hskip 0.81949pt{{\color[rgb]{0.0078,0.0078,0.7852}\boldsymbol{N}}}} A {}^{\hskip 0.81949pt{{\color[rgb]{1,0.4141,0}\boldsymbol{A}}}} Yilun Zhao Y {}^{\hskip 0.81949pt{\color[rgb]{0.7031,0.6914,0.0156}\boldsymbol{Y}}} Liyuan Geng S {}^{\hskip 0.81949pt{\color[rgb]{0.5273,0.0039,0.6 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15045.txt.

Description

Turning text into searchable vectors works better when the reader sees the whole passage at once, and diffusion models read that way by design. Siyue Zhang and co-authors test diffusion models as text embedders in a 2025 paper. Standard large models read left to right, which clashes with the two-way nature of embedding. Their study is billed as the first systematic one of its kind.

Diffusion embedders beat the standard model by a fifth on long-document search. Gains also showed on reasoning-heavy and instruction-based search. Classic embedding tests stayed competitive. Checks confirmed that two-way attention drives the long-text edge. The authors pitch diffusion as a better base for retrieval.

Catalog record

CREATORSiyue Zhang N {}^{\hskip 0.81949pt{{\color[rgb]{0.0078,0.0078,0.7852}\boldsymbol{N}}}} A {}^{\hskip 0.81949pt{{\color[rgb]{1,0.4141,0}\boldsymbol{A}}}} Yilun Zhao Y {}^{\hskip 0.81949pt{\color[rgb]{0.7031,0.6914,0.0156}\boldsymbol{Y}}} Liyuan Geng S {}^{\hskip 0.81949pt{\color[rgb]{0.5273,0.0039,0.6
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15045

Browse the full library · About the Tower of Babel