Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval
Matei Benescu, Ivo Pascal de Jong · 2026
Read Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval by Matei Benescu, Ivo Pascal de Jong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08077.txt.
Description
Benescu and colleagues ask whether large language models can judge relevance better than embedding similarity in information retrieval. Similarity gives a short-sighted view of relevance, they argue, while language understanding and reasoning could capture what a passage truly offers. They compare LLM-based relevance judgment systems against neural embedding retrieval systems on the TREC-DL 2019 passage set, and find no noticeable improvement at first. Digging deeper, they analyze how reasoning itself shifts the comparison between the two approaches.
Catalog record
| CREATOR | Matei Benescu, Ivo Pascal de Jong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36820 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08077