Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval

Matei Benescu, Ivo Pascal de Jong · 2026

Read Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval by Matei Benescu, Ivo Pascal de Jong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08077.txt.

Description

Benescu and colleagues ask whether large language models can judge relevance better than embedding similarity in information retrieval. Similarity gives a short-sighted view of relevance, they argue, while language understanding and reasoning could capture what a passage truly offers. They compare LLM-based relevance judgment systems against neural embedding retrieval systems on the TREC-DL 2019 passage set, and find no noticeable improvement at first. Digging deeper, they analyze how reasoning itself shifts the comparison between the two approaches.

Catalog record

CREATORMatei Benescu, Ivo Pascal de Jong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36820
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08077

Browse the full library · About the Tower of Babel