Entities as Retrieval Signals: A Systematic Study of Coverage, Supervision, and Evaluation in Entity-Oriented Ranking

Shubham Chatterjee · 2026

Read Entities as Retrieval Signals: A Systematic Study of Coverage, Supervision, and Evaluation in Entity-Oriented Ranking by Shubham Chatterjee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05204.txt.

Introduction

Chatterjee shows that mixed results for entity search stem from tests, not model failure. On TREC Robust04, 443 systems were tested against BM25. None lifts MAP by more than 0.05 in open-world tests. The best setup matches the top system and beats most rankers, so build is not the limit. The choke is the entity channel: signals cover only 19.7 percent of apt texts. Training works at the idea level and ignores linking, giving valid but not sharp signals.

Catalog record

CREATORShubham Chatterjee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101829
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Entities as Retrieval Signals: A Systematic Study of Coverage, Supervision, and Evaluation in Entity-Oriented Ranking Shubham Chatterjee Abstract . Entity-oriented retrieval is built on the intuition that documents relevant to a query should exhibit entities relevant to the user’s information need. Yet current evaluations give conflicting answers about whether entity signals actually help ranking. We show that the inconsistency is not a model failure — it is an evaluation failure. We conduct a systematic study on TREC Robust04, evaluating six neural entity-oriented rerankers alongside 437 unsupervised entity-oriented configurations and a BM25 baseline. Across 443 configurations, no system improves MAP by more than 0.05 over BM25 under open-world evaluation over the full candidate set, despite much stronger gains under settings restricted to entity-reachable documents. The best open-world configuration matches the official Robust04 best system and outperforms the majority of neural rerankers, confirming that the architecture is not the problem. The bottleneck is the entity channel itself: even under an idealized discriminative criterion, entity selection covers only 19.7% of relevant documents in the BM25 candidate pool, and no configuration across 193 entity selection strategies simultaneously achieves high coverage and high discrimination. To explain this, we distinguish Conceptual Entity Relevance (CER) — whether an entity is semantically related to a query — from Observable Entity Relevance (OER) — whether the presence of that entity, as detected by the entity linker in this collection, actually increases the probability that a document is relevant. Every supervision strategy evaluated operates at the CER level. None accounts for the linking environment that determines observable discriminativeness. This mismatch explains why better supervision does not recover open-world performance: conceptually correct entity labels are not the same as observably discriminative ones, and filtering for stronger signals reduces coverage without recovering discrimination. The core finding is that conditional and open-world evaluation are answering different scientific questions — whether a model can exploit entity evidence when it is available, and whether the full pipeline improves retrieval under realistic linking conditions — and the field has been reporting answers to the first as if they answered the second. Progress requires collections with aligned entity-level discriminativeness annotations and evaluation protocols that report both coverage and effectiveness. Until then, conditional gains should not be interpreted as evidence of open-world effectiveness, and open-world results should not be interpreted as evidence that entity-oriented architectures fail. 1. Introduction Entity-oriented retrieval is motivated by a simple intuition: documents relevant to a query are more likely to exhibit entities that are themselves relevant to the user’s information need, and these entities can serve as evidence of document relevance. For example, for the query Black Bear Attacks , a document mentioning entities such as Parks Highway , known for bear sightings, provides stronger evidence than one mentioning peripheral entities such as National Guard . The goal is thus to leverage query-relevant entities to identify relevant documents beyond surface term matching. This intuition underlies a wide range of entity-oriented ranking models. EQFE ( 5 ) explicitly states it, identifying queries-relevant entities and using them as signals of document relevance. Other approaches adopt the same view implicitly: the Word-Entity Duet ( 19 ) , JointSem ( 21 ) , and EsdRank ( 20 ) model entities as intermediate signals that connect queries and documents, while EDRM ( 11 ) incorporates entity semantics into neural ranking. Recent models ( 2 ; 1 ) make this bridge explicit by ranking entities and using them to guide document ranking. Despite their architectural differences, these approaches share the same premise: entity-level evidence can improve document retrieval. This raises a fundamental question: Can current evaluation settings in entity-oriented retrieval meaningfully assess the usefulness of entity signals for document ranking? In this paper, we argue that they often cannot. More specifically, can these evaluation settings actually test the long-standing hypothesis that relevant documents exhibit entity-level evidence relevant to the query? In this paper, we study document re-ranking over an initial set of retrieved candidates. We find that current benchmarks often fail to show consistent gains from entity-based methods when evaluated over the full candidate set, despite strong improvements under settings restricted to entity-reachable documents. This creates an empirical inconsistency: the same models appear effective under one evaluation regime and ineffective under another. We show that this discrepancy does not arise from the ranking architecture itself, but from the interaction between entity signals and the evaluation setting. Current benchmarks do not provide a setting in which the usefulness of entity signals can be isolated from how the entity channel is constructed, filtered, and observed. As a result, evaluation entangles entity signal quality with coverage, hindering a clear assessment of their contribution. This issue is compounded by the absence of entity-level supervision in standard benchmarks, which forces entity-oriented methods to rely on proxy signals derived from document relevance. To see why, it is important to distinguish the two main questions that current evaluation settings answer. In conditional evaluation, models are assessed over a document set that has already been filtered or aligned through entity evidence. This asks: if relevant entity signals are available, can the model exploit them to rank documents effectively? That is a meaningful question about model capacity. In open-world evaluation, the model is judged over the full candidate pool, so entity selection, linking, and coverage all matter. This asks: does the full entity-oriented pipeline improve retrieval end-to-end over the entire candidate space? That is a meaningful question about practical effectiveness. Both questions are valid, but they are not equivalent, and neither subsumes the other. This distinction matters because entity-oriented methods can perform well under one regime and poorly under the other for reasons that have little to do with the ranking architecture itself. A system may achieve strong conditional performance because, once entity-reachable documents are identified, it has genuinely learned to exploit entity evidence. Yet the same system may show little gain in the open-world setting if those selected entities cover only a small fraction of the relevant documents. Conversely, broadening the entity channel can reduce the gap between conditional and open-world evaluation by increasing coverage, but only at the cost of introducing many weak, generic, or non-discriminative entities. Open-world failure therefore does not automatically imply that earlier conditional results were wrong, nor does conditional success imply end-to-end usefulness. The two settings answer different questions, and current benchmarks do not cleanly reconcile them. We demonstrate this tension through a systematic study on the TREC Robust04 collection, evaluating six entity-oriented neural ranking models—EDRM ( 11 ) , Word-Entity Duet ( 19 ) , EsdRank ( 20 ) , EVA ( 17 ) , DREQ ( 2 ) , and QDER ( 1 ) —alongside 437 unsupervised entity-oriented ranking configurations and a BM25 baseline. Under open-world evaluation over the full BM25 candidate pool, no configuration improves MAP by more than 0.05 over BM25 (0.34 vs. 0.29), despite much stronger gains reported in conditional settings. Crucially, this pattern holds across model families. The architecture is not the primary limitation. Instead, the bottleneck lies in the entity channel itself. Standard ad hoc retrieval collections do not provide entity-level relevance judgments, so entity-oriented methods must rely on proxy supervision to learn which entities matter for a query. At the same time, entity linking

Read the complete text