ScholarSearch: Benchmarking Scholar Searching Ability of LLMs
Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang · 2025
Read ScholarSearch: Benchmarking Scholar Searching Ability of LLMs by Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13784.txt.
Description
Junting Zhou and co-authors (2025) build ScholarSearch, a benchmark that tests how well large language models search for scholarly information. General search benchmarks exist, but academic search is harder in specific ways. It demands tracing literature chains, using scholarly databases properly, finding obscure long-tail knowledge, and keeping academic rigor. A model that browses the open web well may still fail a literature hunt. ScholarSearch is, the authors say, the first dataset aimed at these skills.
Its questions mirror real study and research tasks rather than traps for the model. Answers are hard even for strong single systems, including dedicated deep-search products. The benchmark checks whether a model can plan a search, judge sources, and assemble findings a researcher could trust. By scoring models on this kind of work, the authors give developers a target closer to what scholars actually need from AI search.
Catalog record
| CREATOR | Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13784