SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models

Jing Yu, Yuqi Tang, Kehua Feng, Mingyang Rao, Lei Liang, Zhiqiang Zhang, Mengshu Sun, Wen Zhang, Qiang Zhang, Keyan Ding, Huajun Chen · 2025

Read SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models by Jing Yu, Yuqi Tang, Kehua Feng, Mingyang Rao, Lei Liang, Zhiqiang Zhang, Mengshu Sun, Wen Zhang, Qiang Zhang, Keyan Ding, Huajun Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15094.txt.

Introduction

Jing Yu and co-authors build SciCUEval in a 2025 paper, a benchmark for how well language models understand scientific context. General benchmarks miss the density of real science, where answers hide in tables, graphs of facts, and mixed sources.

Ten sub-datasets span biology, chemistry, physics, biomedicine, and materials science. Four skills are probed: spotting relevant information, noticing when information is absent, merging several sources, and drawing context-aware inferences. Question formats vary to stress each skill. The team ran leading models through the full suite and mapped their strengths and blind spots in fine grain. The dataset aims to guide the next round of science-tuned model design.

Catalog record

CREATORJing Yu, Yuqi Tang, Kehua Feng, Mingyang Rao, Lei Liang, Zhiqiang Zhang, Mengshu Sun, Wen Zhang, Qiang Zhang, Keyan Ding, Huajun Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15094

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models Jing Yu, Yuqi Tang, Kehua Feng, Mingyang Rao, Lei Liang, Zhiqiang Zhang, Mengshu Sun, Wen Zhang, Qiang Zhang, Keyan Ding, Huajun Chen Abstract Large Language Models (LLMs) have shown impressive capabilities in contextual understanding and reasoning. However, evaluating their performance across diverse scientific domains remains underexplored, as existing benchmarks primarily focus on general domains and fail to capture the intricate complexity of scientific data. To bridge this gap, we construct SciCUEval , a comprehensive benchmark dataset tailored to assess the scientific context understanding capability of LLMs. It comprises ten domain-specific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, integrating diverse data modalities including structured tables, knowledge graphs, and unstructured texts. SciCUEval systematically evaluates four core competencies: Relevant information identification , Information-absence detection , Multi-source information integration , and Context-aware inference , through a variety of question formats. We conduct extensive evaluations of state-of-the-art LLMs on SciCUEval, providing a fine-grained analysis of their strengths and limitations in scientific context understanding, and offering valuable insights for the future development of scientific-domain LLMs. 1 Introduction Large Language Models (LLMs) have demonstrated strong capabilities in natural language understanding, reasoning, and generation across a wide range of general-domain tasks Bai et al. (2023a) ; OpenAI et al. (2024) ; Dubey et al. (2024) ; Qwen et al. (2025) . However, their application to scientific domains remains challenging due to the unique characteristics of scientific language and knowledge. Scientific texts are often dense with technical terminology, implicit assumptions, multimodal data representations, and tightly interlinked concepts that require deeper contextual comprehension Beltagy et al. (2019) ; Mann et al. (2020) . Existing LLM benchmarks in scientific domains Chen et al. (2025) ; Sun et al. (2024) ; Feng et al. (2024) ; Saikh et al. (2022) ; Pedersen et al. (2020) ; Rubungo et al. (2024) ; Jiang et al. (2025) primarily focus on direct question-answering tasks, offering limited insight into how well LLMs perform in scientific context understanding, particularly for noisy and lengthy contexts. Additionally, they often neglect the heterogeneous and structured nature of scientific data, which can span textual descriptions, relational graphs Talmor and Berant (2018) ; He et al. (2024) , and tabular datasets Fang et al. (2024) . In contrast, robust scientific context understanding demands precise information extraction, the ability to identify gaps or missing elements in context, and the integration of multiple evidence sources to support accurate conclusions. Figure 1: Overview of the SciCUEval dataset. It spans five scientific domains, supports three data modalities (structured tables, knowledge graphs, and unstructured text), and includes four question types. Data are collected from high-quality scientific sources. The dataset enables evaluation across four key competencies: (1) relevant information identification, (2) information-absence detection, (3) multi-source information integration, and (4) context-aware inference. To address this gap, we introduce SciCUEval, a comprehensive benchmark dataset designed to rigorously evaluate the scientific context understanding capabilities of LLMs. As shown in Figure 1 , SciCUEval spans ten domain-specific sub-datasets covering diverse disciplines such as biology, chemistry, physics, biomedicine, and materials science. Each subset incorporates rich external knowledge in multiple forms (structured tables, semi-structured knowledge graphs, and unstructured scientific texts) to represent the data modalities commonly encountered in scientific research. SciCUEval targets four core competencies essential for scientific understanding: (1) Relevant Information Identification that locate and extract relevant information from complex and lengthy inputs; (2) Information-Absence Detection that recognize missing, ambiguous, or incomplete contextual elements; (3) Multi-source Information Integration that aggregate and compare information from diverse sources; and (4) Context-Aware Inference that deduce accurate conclusions grounded in scientific contexts. These competencies are evaluated using diverse question types, including open-ended Q&A, multiple-choice, content completion, and true/false validation. The contributions of this paper are summarized as follows: • Establishing a scientific context understanding benchmark: We establish a benchmark to evaluate context understanding capabilities of LLMs in scientific domains, serving as a standardized evaluation suite for assessing LLMs’ capabilities in identifying, detecting, integrating, and reasoning over scientific contexts. • Constructing a diverse set of domain-specific context understanding datasets: We construct ten sub-datasets across multiple disciplines, encompassing various data modalities and a wide range of question types to ensure comprehensive evaluation. • Extensive evaluation and analysis of LLMs: We systematically evaluate and analyze the performance of various state-of-the-art LLMs on SciCUEval, highlighting their strengths and limitations, and offering insights for improvement. Table 1: Comparison of SciCUEval with existing benchmark datasets. Question Types: QA (Question Answering), MCQ (Multiple Choice Question), T/F (True/False Question), and CC(Cloze Completion). Datasets Contexts Domains Data Modalities Question Types Evaluation Competencies # Nums LongICLBench Li et al. (2024) \checkmark General Text QA Identification 2,618 LongBench Bai et al. (2023b) \checkmark General, Code Text QA Identification, Integration 4,750 LongBench V2 Bai et al. (2024) \checkmark General, Law, Finance Text MCQ Identification, Integration, Inference 503 RGB Chen et al. (2024) \checkmark General Text QA Identification, Detec., Integration, Inference 1,000 ChemLit-QA Wellawatte et al. (2025) \checkmark Chemistry Text QA Identification, Detec., Inference 1,054 CHEMRAG-BENCH Zhong et al. (2025) \times Chemistry Text QA, MCQ Identification, Inference 1,932 SciCUEval \checkmark Comprehensive Science Text, Table, KG QA, MCQ, T/F, CC Identification, Detec., Integration, Inference 11,343 2 Backgrounds and Related Works Context Understanding Tasks Large language models (LLMs) have demonstrated remarkable capabilities in context understanding Huang et al. (2023) ; Dong et al. (2022) . This paradigm allows LLMs to adapt flexibly to new external knowledge without requiring additional fine-tuning or retraining Lewis et al. (2020) ; Liu et al. (2024) . Importantly, LLMs have shown strong performance across a wide range of domains when equipped with additional context. Furthermore, in-context learning strengthens LLMs’ transparency by firmly establishing their arguments in the documents that were obtained Mialon et al. (2023) ; Xiong et al. (2024) . However, despite its potential, context understanding remains sensitive to prompt design and the quality of the provided context. It also demands that the model possess a robust ability to process and comprehend long texts. In this study, we systematically investigate the robustness and effectiveness of LLMs for context understanding in diverse scientific domains. Context Understanding Benchmarks The development of robust and comprehensive benchmarks for evaluating long context understanding has gained increasing attention in recent research Li et al. (2024) ; Liu et al. (2024) ; Bai et al. (2023b) ; Zhang et al. (2024c) ; Wu et al. (2025) ; Chen et al. (2020) ; Chen et al. (2021) . For instance, LongICLBench Li et al. (2024) evaluates the long in-context learning capabilities of large language models across various domains, including emotion classification, intent detection, relation extraction, and named entity recognition. RepoQA Liu et al. (2024) evaluates LLMs’ long-context code understanding ability. LongBench Bai et al. (2023b) and LongBench-V2 Bai et al. (2024) are comprehensive benchmarks for evaluating long-context understanding and reason across domains including law, finance, literature, news, and code. In the scientific domain, ChemLit-QA Wellawatte et al. (2025) is a chemistry dataset of question-answer-context triplets. CHEMRAG-BENCH Zhong et al. (2025) benchmarks retrieval-augmented generation (RAG) in chemistry by dynamically retrieving relevant contexts for each question from shared, large-scale corpora. However, there is still a lack of scientific context understanding datasets that cover multiple disciplines, modalities, and capabilities. To fill this

Read the complete text