SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

Yiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang, PKU-Anker LLM Lab, Peking University Zhongguancun Academy IDEA · 2026

Read SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models by Yiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang, PKU-Anker LLM Lab, Peking University Zhongguancun Academy IDEA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19357.txt.

Introduction

Yiyang Gu and colleagues build SciCustom, a tool for tests of science skill in language models. Standard tests are hand-made or generic, so they miss the skills real science work needs. SciCustom sorts knowledge into ontology-tied units at set detail levels, trains a tagger to place data into that space, and picks apt units by votes across models. Retrieval and subset choice yield tailored tests without expert labels or made-up questions. Tests in chemistry and health care reveal gaps that standard tests overlook.

Catalog record

CREATORYiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang, PKU-Anker LLM Lab, Peking University Zhongguancun Academy IDEA
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56912
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19357

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models Yiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang, PKU-Anker LLM Lab, Peking University Zhongguancun Academy IDEA Abstract Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most benchmarks are manually curated or domain-generic, limiting scalability and alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware benchmark retrieval via binary search, followed by proxy subset selection and data-grounded benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs. The source code is available at https://github.com/yjwtheonly/SciCustom . 1 Introduction Figure 1: Illustrations of SciCustom . ( a ) Comparison between traditional off-the-shelf benchmarking and our ontology-driven framework. ( b, c ) Evaluation of 10 LLMs on different benchmarks, where each dot represents a model. Targeting specific capabilities in Technical Chemistry, ( b ) the general scientific benchmark (GPQA Diamond) aligns poorly with expert ground truth, whereas ( c ) the benchmark constructed by SciCustom demonstrates strong alignment. With the rapid advancement of Large Language Models (LLMs), their applications have significantly broadened across the scientific landscape 9 ; 7 ; 6 ; 28 ; 56 ; 26 ; 54 . For example, an LLM-based system successfully proposed a novel mechanism of gene transfer in bacteria, mirroring conclusions that took years of experimental validation 14 ; 33 . However, as scientists increasingly seek to leverage LLMs for specific scientific applications, a critical challenge emerges: the performance of a given LLM within a particular scientific context remains largely uncertain and difficult to evaluate 8 ; 41 ; 29 ; 4 . While numerous benchmarks exist to evaluate various aspects of model capability, they frequently fail to reflect the requirements of specialized usage 2 ; 23 ; 3 . Empirical observations indicate that performance on widely-used benchmarks often diverges from that in specific scientific tasks ( Figure 1 b ), necessitating the customization of benchmarks tailored to particular scientific applications 41 ; 29 . Given the vast and ever-expanding range of scientific applications, manual curation of benchmarks for each emerging use case is impractical 18 ; 53 . A plausible solution is to automate the customized benchmark construction process 13 ; 22 ; 11 ; 47 . However, developing such an automated framework presents two challenges. First, scientific applications are inherently complex and highly interdisciplinary. A single application often requires knowledge from multiple subfields (e.g., drug discovery intertwines organic chemistry, molecular biology, and pharmacology 27 ), while distinct applications frequently share underlying knowledge (e.g., both drug discovery and clinical decision-making require the knowledge of pharmacology 42 ; 5 ; 31 ). Consequently, constructing benchmarks from scratch for each specific scenario results in massive redundant labor and limits the framework’s scalability 18 ; 21 . Second, scientific evaluation requires grounded validity. High-quality evaluation data are frequently derived from costly wet-lab experiments or computational simulations 10 ; 36 , rendering simple general-purpose data synthesis methods infeasible 32 ; 51 ; 11 ; 55 . To this end, we introduce SciCustom , a framework for custom evaluation of scientific capabilities in LLMs ( Figure 1 a,c ). The key idea is that a complex scientific application can be approximated by a composition of fine-grained knowledge units. To construct these units, we select 642 representative concepts from a comprehensive scientific ontology to serve as unified knowledge units. Leveraging the rich hierarchical information within the ontology, which links specific scientific concepts to knowledge units, we train a tagger to automatically align real-world scientific corpora with the corresponding knowledge units. This offline process performs a once-for-all population of each knowledge unit with grounded scientific data. When a user presents an evaluation requirement, SciCustom identifies the relevant knowledge units and orchestrates a hierarchical retrieval process to construct a tailored benchmark. The combinability of these pre-constructed knowledge units enables dynamic adaptation to new requirements via efficient reuse, ensuring scalability for diverse downstream applications. Experimental results in chemistry and healthcare domains demonstrate that our automatically constructed benchmarks closely align with expert-curated benchmarks. Furthermore, for pericyclic reaction, a novel scientific application lacking prior benchmarks, our framework successfully generates a high-quality benchmark, providing a scalable and reliable solution for assessing LLMs in the ever-expanding landscape of scientific research. Our contributions can be summarized as follows: • We propose SciCustom , a framework that models scientific applications as compositions of reusable knowledge units to automate benchmark construction, enabling efficient adaptation to diverse downstream tasks. • We develop a tagger that leverages ontology structures to map large-scale, real-world scientific corpora into knowledge units, ensuring that evaluations are grounded in verifiable data. • Experiments in chemistry and healthcare confirm SciCustom ’s strong alignment with expert-curated benchmarks and its capability to evaluate novel scientific contexts where no prior benchmarks exist. 2 Methodology Figure 2: Framework of SciCustom . It consists of an offline phase where scientific data is indexed into ontology-grounded knowledge units via a trained tagger, and an online phase where user requirements are parsed by multi-model voting to identify relevant tags. These tags guide the binary search-based selection and proxy selection of data. The problem set of the benchmark is generated based on these data. 2.1 Overview We study the problem of customized scientific capability evaluation for LLMs. Given an evaluation requirement r and a large corpus of scientific data \mathcal{D} , the goal is to construct a benchmark \mathcal{B}_{r} that reflects the capabilities required by r without expert annotation. SciCustom is an automated framework for this problem, operating through offline data indexing and online data-grounded generation phases. In the offline phase, SciCustom represents large-scale scientific data within shared knowledge units using a tagger. This phase organizes scattered scientific content into reusable units. In the online phase, SciCustom first employs a voting-based multi-model consensus to identify knowledge units relevant to the user requirement r . Guided by these units, the framework performs a hierarchical data selection process: retrieving candidate data, filtering via binary search, and extracting a representative proxy subset to generate final questions for efficient evaluation. An overview of SciCustom is depicted in Figure 2 . 2.2 Ontology-Grounded Knowledge Units To enable fine-grained yet reusable modeling of scientific knowledge, SciCustom is built upon a structured scientific ontology that serves as a unified semantic backbone for both knowledge definition and data organization. Following 25 , we integrate multiple authoritative repositories to construct a large-scale scientific

Read the complete text