Formalized Information Needs Improve Large-Language-Model Relevance Judgments Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809561 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results CCS: Information systems Relevance assessment CCS: Information systems Test collections

Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, Philipp Schaer · 2026

Read Formalized Information Needs Improve Large-Language-Model Relevance Judgments Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809561 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results CCS: Information systems Relevance assessment CCS: Information systems Test collections by Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, Philipp Schaer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04140.txt.

Description

Jüri Keller and team asked whether language-model relevance judges work better when given formalized information needs instead of bare queries. They generated topics with descriptions and narratives in the style of past human assessments, then tested assessors on the 2019 and 2020 editions of TREC Deep Learning and Robust04. Query-only judges marked far more documents relevant and agreed less with each other, weakening evaluation reliability. Formalized topics improved agreement between human and model judgments, even when the topics themselves were machine-made.

Catalog record

CREATORJüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, Philipp Schaer
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59142
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04140

Browse the full library · About the Tower of Babel