Formalized Information Needs Improve Large-Language-Model Relevance Judgments Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809561 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results CCS: Information systems Relevance assessment CCS: Information systems Test collections
Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, Philipp Schaer · 2026
Read Formalized Information Needs Improve Large-Language-Model Relevance Judgments Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809561 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results CCS: Information systems Relevance assessment CCS: Information systems Test collections by Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, Philipp Schaer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04140.txt.
Description
Jüri Keller and team asked whether language-model relevance judges work better when given formalized information needs instead of bare queries. They generated topics with descriptions and narratives in the style of past human assessments, then tested assessors on the 2019 and 2020 editions of TREC Deep Learning and Robust04. Query-only judges marked far more documents relevant and agreed less with each other, weakening evaluation reliability. Formalized topics improved agreement between human and model judgments, even when the topics themselves were machine-made.
Catalog record
| CREATOR | Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, Philipp Schaer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59142 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04140