AcademicEval: Live Long-Context LLM Benchmark

Haozhen Zhang [email protected], [email protected], Tao Feng [email protected], Pengrui Han [email protected], Jiaxuan You [email protected] · 2025

Read AcademicEval: Live Long-Context LLM Benchmark by Haozhen Zhang [email protected], [email protected], Tao Feng [email protected], Pengrui Han [email protected], Jiaxuan You [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17725.txt.

Introduction

Haozhen Zhang and three co-authors built a new test for long-context language models. Their benchmark, AcademicEval, checks how well models write with very long inputs. Older tests use fixed lengths and costly hand labeling. They also risk leaking answers into model training data.

AcademicEval draws its tasks from real papers posted on arXiv. Models must draft titles, abstracts, opening sections, and related-work text. Sample answers come from a graph of co-author links. Fresh papers keep the test live and free of leaks. Results show models still struggle as tasks grow more layered.

Catalog record

CREATORHaozhen Zhang [email protected], [email protected], Tao Feng [email protected], Pengrui Han [email protected], Jiaxuan You [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17725

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AcademicEval: Live Long-Context LLM Benchmark Haozhen Zhang [email protected], [email protected], Tao Feng [email protected], Pengrui Han [email protected], Jiaxuan You [email protected] Abstract Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose AcademicEval , a live benchmark for evaluating LLMs over long-context generation tasks. AcademicEval adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, i.e. , Title , Abstract , Introduction , and Related Work , which cover a wide range of abstraction levels and require no manual labeling. Moreover, AcademicEval integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, AcademicEval features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on AcademicEval , and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs’ long-context modeling capabilities. Code is available at https://github.com/ulab-uiuc/AcademicEval . 1 Introduction Large Language Models (LLMs) have recently achieved tremendous success in natural language processing (NLP) tasks ( Achiam et al. 2023 ; AI@Meta 2024 ) . However, when facing long context inputs, LLMs show a sharp decline in performance, which poses a pressing challenge to LLMs in understanding and capturing key information in long texts ( Li et al. 2024 ; Liu et al. 2024 ) . Therefore, several long-context LLM benchmarks are spawned to evaluate LLMs in various settings, including question answering, summarizing, and reasoning ( Shaham et al. 2023 ; An et al. 2023 ; Dong et al. 2023 ; Bai et al. 2023b ; Li et al. 2023 ; Zhang et al. 2024b ) . Despite their success, these benchmarks still suffer from concerns of rigid context length, saturated performance, and being leaked in LLM training. We envision that the next-generation long-context LLM benchmarks should ideally possess three key features. (1) Flexible and potentially unlimited context length: existing benchmarks fix the context for each long-context problem; ideally, the format and length of the context could be flexibly set based on the LLM’s capability, especially given the release of long-context LLMs ( Reid et al. 2024 ) and their capabilities in ingesting multi-modal information, e.g. , graphs ( Dong et al. 2024 ) . (2) High-quality labels derived from real-world data , minimizing human labeling efforts: existing long-context benchmarks often require human labeling ( Bai et al. 2023b ; An et al. 2023 ; Li et al. 2023 ; Dong et al. 2023 ; Zhang et al. 2024b ) , which is costly and limits the size of the benchmarks to about 2000 samples ( Xu et al. 2023 ) (3) Live updates to mitigate information leakage during LLM pretraining and fine-tuning: benchmark data contamination in LLM has gradually become a severe issue ( Sainz et al. 2023 ; Ye et al. 2024 ; Zhu et al. 2024b ; Zhu et al. 2024a ; Xu et al. 2024 ) ; we argue that holding out future data as the val/test set is one of the most effective approaches for open benchmarks. Based on these principles, we propose AcademicEval , a live benchmark to evaluate LLMs over long-context generation tasks. AcademicEval adopts arXiv as its data source and features a suite of academic writing tasks on each paper without labor-intensive annotation: Title , Abstract , Introduction , and Related Work , each of which has long-context input and hierarchical abstraction levels. In particular, we construct a co-author graph via the arXiv API to conveniently obtain co-author papers as high-quality and expert-curated few-shot demonstrations, which also possess AcademicEval flexible context length. Furthermore, AcademicEval introduces efficient live evaluation based on the co-author graph, which utilizes the latest papers on arXiv to update the benchmark data periodically and ensures no label leakage. Moreover, AcademicEval provides in-context few-shot demonstrations for each sample, which is neglected by most existing long-context LLM benchmarks ( Liu et al. 2024 ; Li et al. 2024 ) . In our experiments, we evaluate three categories of baselines on AcademicEval : standard LLMs, long-context LLMs, and retrieval-augmented language models (RALM). Under automatic metrics (BERTScore and ROUGE-L), RALM often attains the strongest results by concentrating salient evidence into shorter retrieved chunks, while long-context LLMs and strong standard models remain competitive in several settings. However, an LLM-as-a-Judge evaluation, which assesses novelty, feasibility, consistency, factuality, and academic style, reveals a more nuanced picture: retrieval is not always preferred ( e.g. , for Title / Abstract ), whereas it is highly beneficial for Related Work . Across both evaluations, performance commonly degrades as the input length grows, and correlated few-shot demonstrations from the co-author graph can provide modest gains for specific model–task pairs. Overall, the results indicate that AcademicEval is a challenging benchmark that exposes complementary facets of long-context modeling: overlap-oriented automatic metrics and higher-level judged quality. Table 1: Comparison with Existing Long-context LLM Benchmarks . Each column indicates the average input length, whether the annotation is human-assisted, whether there are tasks with hierarchical abstraction levels, whether it contains few-shot demonstrations, and whether the benchmark is lively updated, respectively. Benchmark Avg Len Automatic Annotation Hierarchical Abstraction Few-shot Demos Live Update ZeroSCROLLS ( Shaham et al. 2023 ) \sim 10K ✓ ✗ ✗ ✗ L-Eval ( An et al. 2023 ) \sim 8K ✗ ✗ ✗ ✗ BAMBOO ( Dong et al. 2023 ) \sim 16K ✗ ✗ ✗ ✗ LongBench ( Bai et al. 2023b ) \sim 8K ✗ ✗ ✓ ✗ LooGLE ( Li et al. 2023 ) \sim 20K ✗ ✗ ✗ ✗ \infty Bench ( Zhang et al. 2024b ) \sim 200K ✗ ✗ ✗ ✗ AcademicEval (ours) Flexible ✓ ✓ ✓ ✓ We illustrate the comparison with existing long-context LLM benchmarks in Table 1 . Our contributions are summarized as follows: • We propose a live benchmark, AcademicEval , to evaluate LLMs over long-context generation tasks. AcademicEval features four academic writing tasks with hierarchical abstraction levels and requires no manual annotation. • We construct a co-author graph via the arXiv API and draw on the co-author papers as informative few-shot demonstrations, making the context length of AcademicEval flexible and scalable. Especially, AcademicEval conducts periodic data updates on the co-author graph to enable efficient live evaluation, which ensures no label leakage and fair evaluation. • We conduct comprehensive experiments on AcademicEval , and the results demonstrate its challenges and yield potential insights for improving LLMs in long-context modeling. 2 Related Work Long-context Modeling and LLM Benchmarks. LLMs are known to be powerful in language modeling tasks ( Achiam et al. 2023 ; AI@Meta 2024 ) . However, when it comes to long-context inputs, LLMs show a sharp decline in performance, posing a pressing challenge when benchmarking their long-context modeling capabilities ( Liu et al. 2024 ; Li et al. 2024 ; Li et al. 2025 ) . Currently, there are two mainstream technologies for long-context modeling tasks: retrieval-augmented language models (RALM) ( Ram et al. 2023 ; Yu et al. 2023 ; Trivedi et al.

Read the complete text