Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles

Miao Li, Alexander Gurung, Irina Saparina & Mirella Lapata · 2025

Read Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles by Miao Li, Alexander Gurung, Irina Saparina & Mirella Lapata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21028.txt.

Description

Answering a question that needs arithmetic over dozens of full scientific papers is a stern test for language models. Miao Li and three co-authors build SciTrek, a 2025 dataset for long-context numerical reasoning. Existing long-context tests mostly check simple retrieval or use artificial contexts.

SciTrek questions require counting, sorting, aggregation, and comparison over collections of real articles. The questions are generated automatically by turning them into database queries over article metadata, such as titles, authors, and references. Ground-truth answers come from executing those queries, so they are exact. Inputs exceed sixty-four thousand tokens. The dataset both evaluates current models and supplies training material to improve their numerical reasoning over long documents.

Catalog record

CREATORMiao Li, Alexander Gurung, Irina Saparina & Mirella Lapata
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-21028

Browse the full library · About the Tower of Babel