ScholarSearch: Benchmarking Scholar Searching Ability of LLMs

Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang · 2025

Read ScholarSearch: Benchmarking Scholar Searching Ability of LLMs by Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13784.txt.

Description

Junting Zhou and co-authors (2025) build ScholarSearch, a benchmark that tests how well large language models search for scholarly information. General search benchmarks exist, but academic search is harder in specific ways. It demands tracing literature chains, using scholarly databases properly, finding obscure long-tail knowledge, and keeping academic rigor. A model that browses the open web well may still fail a literature hunt. ScholarSearch is, the authors say, the first dataset aimed at these skills.

Its questions mirror real study and research tasks rather than traps for the model. Answers are hard even for strong single systems, including dedicated deep-search products. The benchmark checks whether a model can plan a search, judge sources, and assemble findings a researcher could trust. By scoring models on this kind of work, the authors give developers a target closer to what scholars actually need from AI search.

Catalog record

CREATORJunting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang, Tingjia Miao, Zhihui Qi, Yuhan Wu, Tong Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13784

Browse the full library · About the Tower of Babel