IfIR : A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval
Tingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Zhejiang University, Yale University https://github.com/SighingSnow/IFIR · 2025
Read IfIR : A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval by Tingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Zhejiang University, Yale University https://github.com/SighingSnow/IFIR (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04644.txt.
Description
Search systems in expert fields must follow detailed instructions, not just match keywords. Tingyu Song, Guo Gan, Mingsheng Shang, and Yilun Zhao build a test for that skill in this 2025 paper. Their benchmark, IFIR, spans finance, law, health care, and science writing, where a loose match can be a wrong answer.
The set holds 2,426 examples across eight subsets, each tied to real tasks in its field. Instructions come at several levels of complexity, so the test can show where models start to fail. The team also proposes grading by a language model judge, checked for trustworthiness. Fifteen leading search models, including ones built on language models, were tried. All struggled with complex, field-specific instructions, and the paper maps those limits to guide better designs.
Catalog record
| CREATOR | Tingyu Song, Guo Gan, Mingsheng Shang, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences, Zhejiang University, Yale University https://github.com/SighingSnow/IFIR |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04644