Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective
Siyue Zhang N {}^{\hskip 0.81949pt{{\color[rgb]{0.0078,0.0078,0.7852}\boldsymbol{N}}}} A {}^{\hskip 0.81949pt{{\color[rgb]{1,0.4141,0}\boldsymbol{A}}}} Yilun Zhao Y {}^{\hskip 0.81949pt{\color[rgb]{0.7031,0.6914,0.0156}\boldsymbol{Y}}} Liyuan Geng S {}^{\hskip 0.81949pt{\color[rgb]{0.5273,0.0039,0.6 · 2025
Read Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective by Siyue Zhang N {}^{\hskip 0.81949pt{{\color[rgb]{0.0078,0.0078,0.7852}\boldsymbol{N}}}} A {}^{\hskip 0.81949pt{{\color[rgb]{1,0.4141,0}\boldsymbol{A}}}} Yilun Zhao Y {}^{\hskip 0.81949pt{\color[rgb]{0.7031,0.6914,0.0156}\boldsymbol{Y}}} Liyuan Geng S {}^{\hskip 0.81949pt{\color[rgb]{0.5273,0.0039,0.6 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15045.txt.
Description
Turning text into searchable vectors works better when the reader sees the whole passage at once, and diffusion models read that way by design. Siyue Zhang and co-authors test diffusion models as text embedders in a 2025 paper. Standard large models read left to right, which clashes with the two-way nature of embedding. Their study is billed as the first systematic one of its kind.
Diffusion embedders beat the standard model by a fifth on long-document search. Gains also showed on reasoning-heavy and instruction-based search. Classic embedding tests stayed competitive. Checks confirmed that two-way attention drives the long-text edge. The authors pitch diffusion as a better base for retrieval.
Catalog record
| CREATOR | Siyue Zhang N {}^{\hskip 0.81949pt{{\color[rgb]{0.0078,0.0078,0.7852}\boldsymbol{N}}}} A {}^{\hskip 0.81949pt{{\color[rgb]{1,0.4141,0}\boldsymbol{A}}}} Yilun Zhao Y {}^{\hskip 0.81949pt{\color[rgb]{0.7031,0.6914,0.0156}\boldsymbol{Y}}} Liyuan Geng S {}^{\hskip 0.81949pt{\color[rgb]{0.5273,0.0039,0.6 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15045