Reproducing Complex Set-Compositional Information Retrieval Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia ISBN: 979-8-4007-2599-9/2026/07 DOI: 10.1145/3805712.3808555 CCS: Information systems Retrieval effectiveness

Vincent Degenhart, Dewi Timman, Arjen P. de Vries, Faegheh Hasibi, Mohanna Hoveyda · 2026

Read Reproducing Complex Set-Compositional Information Retrieval Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia ISBN: 979-8-4007-2599-9/2026/07 DOI: 10.1145/3805712.3808555 CCS: Information systems Retrieval effectiveness by Vincent Degenhart, Dewi Timman, Arjen P. de Vries, Faegheh Hasibi, Mohanna Hoveyda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03824.txt.

Introduction

Vincent Degenhart and four co-authors test whether modern search systems truly satisfy complex queries built from conjunction, disjunction, and exclusion, or whether they take semantic shortcuts. They rerun earlier work across major retrieval families and reasoning-targeted methods on the QUEST benchmarks. They also introduce LIMIT+, a controlled benchmark where relevance depends on arbitrary attribute predicates rather than pre-trained knowledge.

On QUEST, the best neural retrievers more than doubled the recall of the classic BM25 method, though reasoning-targeted systems did not win uniformly. On LIMIT+, the strongest QUEST method collapsed to near-zero recall while lexical retrieval surged. Grouping by compositional depth showed consistent degradation for all methods, with sparse and lexical approaches more stable than dense ones.

Catalog record

CREATORVincent Degenhart, Dewi Timman, Arjen P. de Vries, Faegheh Hasibi, Mohanna Hoveyda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03824

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reproducing Complex Set-Compositional Information Retrieval Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia ISBN: 979-8-4007-2599-9/2026/07 DOI: 10.1145/3805712.3808555 CCS: Information systems Retrieval effectiveness Vincent Degenhart, Dewi Timman, Arjen P. de Vries, Faegheh Hasibi, Mohanna Hoveyda Abstract . Complex information needs may involve set-compositional queries using conjunction, disjunction, and exclusion, yet it remains unclear whether current retrieval paradigms genuinely satisfy such constraints or exploit ‘semantic shortcuts’. We conduct a reproducibility study to benchmark major retrieval families and reasoning-targeted methods on QUEST and QUEST+Variants, and introduce LIMIT+, a controlled benchmark where relevance depends on arbitrary attribute predicates and constraint satisfaction, and less on pretrained knowledge. Our findings show that (i) on QUEST, the best neural retrievers achieve an effectiveness that is more than double what can be achieved with BM25 (Recall@100 > {>} 0.41 vs. 0.20), but reasoning-targeted methods like ReasonIR and Search-R1 do not outperform general-purpose retrievers uniformly; (ii) on LIMIT+, gains fail to transfer, where the strongest QUEST method collapses from Recall@100 ≈ {\approx} 0.42 to below 0.02, while classic lexical retrieval gains to ∼ {\sim} 0.96. Lastly, (iii) stratifying by compositional depth reveals a consistent degradation across all methods, where algebraic sparse and lexical methods show more stable performance while dense approaches collapse. We release code and LIMIT+ data generation scripts to support future reproducibility and controlled evaluation. † † cc-license: by 1. Introduction In many real-world deployments, information needs involve more than topical or semantic similarity, and need to satisfy several conditions, avoiding results that violate these constraints ( 25 ) . A user may seek entities that match several properties, or match one property while excluding another ( 29 ; 11 ) . These information needs can be formalized as set operations; intersection (AND), union (OR), and set difference (NOT). Figure 1. The transition from context-rich narrative to atomic datasets. Datasets example. Figure to illustrate the datasets used. The urgency of addressing these operations is underscored by two trends: Deep Research:: The frequency of constraint-heavy queries increases as users engage in more specialized, deep-research tasks that move beyond simple keyword matching ( 31 ) . Agentic Query Rewriting:: In agentic workflows, autonomous systems frequently rewrite user prompts into multi-step logical queries, requiring the underlying retrieval engine to be more than just semantically aware ( 26 ) . Yet most modern retrieval systems are trained and evaluated on benchmarks where semantic relatedness is a dominant signal ( 1 ; 4 ) . Consequently, improvements on complex-query retrieval benchmarks can be difficult to interpret: strong aggregate performance does not, by itself, demonstrate reliable enforcement of compositional requirements, and may instead reflect semantic heuristics and pretrained world knowledge that correlate with the query constraints. Recent benchmarks such as QUEST ( 11 ; 17 ) operationalize set-compositional retrieval by instantiating entity-seeking queries whose templates imply set operations over a large Wikipedia corpus. In parallel, retrieval research now spans a broad range of paradigms, from lexical and dense bi-encoders to late-interaction, learned sparse retrieval, and LLM-based rerankers, alongside recent methods proposed specifically for reasoning-intensive retrieval via LLMs or enhanced constraint representation learning. This motivates two empirical questions. Firstly, it is unclear how major retrieval families compare under a consistent, reproducible evaluation protocol in complex set-compositional retrieval. Second, even when aggregate scores improve, it is unclear whether observed gains reflect improved adherence to compositional requirements or improvements driven by correlations in the underlying data and model priors. To address these questions, we conduct a reproducibility study with a two-part evaluation. We first benchmark a broad suite of retrievers and rerankers on QUEST and QUEST+Variants ( 17 ) under a unified pipeline that standardizes indexing, retrieval, and evaluation. We then introduce LIMIT+ , a new benchmark designed to isolate constraint satisfaction from semantic shortcut effects. LIMIT+ extends LIMIT ( 23 ) by composing atomic attribute predicates into QUEST -style templates, so that relevance is determined by exact predicate satisfaction rather than topical relatedness. This construction yields exact gold sets by design and associates each query instance with template and attribute metadata, enabling stratified analyses by constraint type and compositional depth. Research Questions Concretely, we study the following: RQ1. How do major retrieval families and reasoning-targeted methods (e.g., Search-R1 ( 7 ) , ReasonIR ( 16 ) ) perform on set-compo- sitional queries in QUEST and QUEST+Variants ? RQ2. Do gains observed on QUEST persist in a controlled setting in LIMIT+ (Section 3 ) where semantic similarity provides limited signal and relevance is driven by constraint satisfaction? RQ3. How does effectiveness change with increasing compositional depth and across constraint types? Summary of main findings Our experiments reveal 3 key results. (RQ1) On QUEST and QUEST+Variants , retrieval families separate clearly: the best neural retrievers demonstrate a twofold improvement over BM25 (Recall@100 {>} 0.41 vs. 0.20), yet methods explicitly designed for reasoning in retrieval, such as ReasonIR and Search-R1, do not outperform strong general-purpose retrievers; suggesting that a reasoning specific objective is not sufficient to perform well on set compositional retrieval tasks. (RQ2) When semantic shortcuts are minimized in LIMIT+ , the performance gains observed on simpler benchmarks fail to generalize: the strongest enhanced method on QUEST (LogiCol; Recall@100 {\approx} 0.42) collapses below 0.02, and all dense single-vector models fall below 0.10, while lexical retrieval ( {\approx} 0.96) and token-level methods remain robust, suggesting that much of the observed progress on QUEST reflects learned correlations with semantic cues rather than genuine constraint satisfaction. Under a controlled re-ranking setup where all gold documents are guaranteed to appear in the candidate set, the LLM-based re-ranker achieves near-ceiling effectiveness (nDCG@5 up to 0.97). The neuro-symbolic approach is a close second, trailing by only a marginal gap, while the cross-encoder fails at successfully positioning relevant documents on the top ( {\approx} 0.16). This localizes the primary bottleneck in set-compositional complex queries to first-stage candidate generation. (RQ3) Lastly, stratifying by constraint type and compositional depth reveals that aggregate scores mask heterogeneity: templates with negations, in isolation or combined with other constraints, are markedly harder than atomic or disjunctive queries, and performance degrades monotonically with depth. On LIMIT+ , algebraic sparse methods ( ( 9 ) ) remain stable (nDCG@20 {\approx} 0.22) where learned dense approaches collapse ( {\approx} 0.01). Contributions (1) A reproducibility study benchmarking lexical, dense, late-interaction, learned sparse, cross-encoder, LLM and neuro-symbolic reranking pipelines, alongside reasoning-oriented methods (ReasonIR, Search-R1, LogiCol ( 17 ) , Set-Comp ( 9 ) ), under a unified evaluation protocol for set-compositional retrieval. (2) LIMIT+ , a controlled set-compositional benchmark derived from LIMIT by composing QUEST -style constraint templates with controlled depth and constraint types, enabling stratified analyses that disentangle semantic relatedness from constraint satisfaction. (3) Diagnostic ablations over compositional depth and constraint types, exposing failure modes invisible in aggregate metrics, showing how learned dense representations and some reasoning-enhanced methods collapse under complex composition. The remainder of the paper is organized as follows. Section 2 surveys related work on complex-query retrieval, retrieval paradigms, and reasoning-targeted methods. Section 3 describes the construction of the LIMIT+ benchmark. Section 4 details the

Read the complete text