IfIR : A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval

Tingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Zhejiang University, Yale University https://github.com/SighingSnow/IFIR · 2025

Read IfIR : A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval by Tingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Zhejiang University, Yale University https://github.com/SighingSnow/IFIR (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04644.txt.

Description

Search systems in expert fields must follow detailed instructions, not just match keywords. Tingyu Song, Guo Gan, Mingsheng Shang, and Yilun Zhao build a test for that skill in this 2025 paper. Their benchmark, IFIR, spans finance, law, health care, and science writing, where a loose match can be a wrong answer.

The set holds 2,426 examples across eight subsets, each tied to real tasks in its field. Instructions come at several levels of complexity, so the test can show where models start to fail. The team also proposes grading by a language model judge, checked for trustworthiness. Fifteen leading search models, including ones built on language models, were tried. All struggled with complex, field-specific instructions, and the paper maps those limits to guide better designs.

Catalog record

CREATORTingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Zhejiang University, Yale University https://github.com/SighingSnow/IFIR
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04644

Browse the full library · About the Tower of Babel