SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models

Chuan Qin ∗, Xin Chen †, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong, Qingqing Long, Boya Pan, Han Wu, Chengzan Li, Yuanchun Zhou ‡, Hui Xiong, Hengshu Zhu ‡ · 2025

Read SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models by Chuan Qin ∗, Xin Chen †, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong, Qingqing Long, Boya Pan, Han Wu, Chengzan Li, Yuanchun Zhou ‡, Hui Xiong, Hengshu Zhu ‡ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13503.txt.

Introduction

Chuan Qin and co-authors present a framework for judging how ready science is for AI, in a 2025 paper. AI is reshaping discovery, but no shared yardstick measures both the data and the models behind it. Their framework, SciHorizon, scores scientific data on quality, fairness of access, explainability, and compliance, split into fifteen finer dimensions. From journal papers it also builds recommendation lists of AI-ready datasets for Earth, life, and materials sciences.

On the model side, sixteen dimensions test language models across knowledge, understanding, reasoning, multiple input types, and values in five sciences. The team ran over fifty leading open and closed models through the benchmark. All results are posted publicly online. The authors aim to give the AI-for-science field a common readiness standard rather than scattered one-off tests.

Catalog record

CREATORChuan Qin ∗, Xin Chen †, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong, Qingqing Long, Boya Pan, Han Wu, Chengzan Li, Yuanchun Zhou ‡, Hui Xiong, Hengshu Zhu ‡
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13503

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models Chuan Qin ∗, Xin Chen †, Chengrui Wang, Pengmin Wu, Xi Chen, Yihang Cheng, Jingyi Zhao, Meng Xiao, Xiangchao Dong, Qingqing Long, Boya Pan, Han Wu, Chengzan Li, Yuanchun Zhou ‡, Hui Xiong, Hengshu Zhu ‡ Abstract In recent years, the rapid advancement of Artificial Intelligence (AI) technologies, particularly Large Language Models (LLMs), has revolutionized the paradigm of scientific discovery, establishing AI-for-Science (AI4Science) as a dynamic and evolving field. However, there is still a lack of an effective framework for the overall assessment of AI4Science, particularly from a holistic perspective on data quality and model capability. Therefore, in this study, we propose SciHorizon, a comprehensive assessment framework designed to benchmark the readiness of AI4Science from both scientific data and LLM perspectives. First, we introduce a generalizable framework for assessing AI-ready scientific data, encompassing four key dimensions—Quality, FAIRness, Explainability, and Compliance—which are subdivided into 15 sub-dimensions. Drawing on data resource papers published between 2018 and 2023 in peer-reviewed journals, we present recommendation lists of AI-ready datasets for Earth, Life, and Materials Sciences, making a novel and original contribution to the field. Concurrently, to assess the capabilities of LLMs across multiple scientific disciplines, we establish 16 assessment dimensions based on five core indicators—Knowledge, Understanding, Reasoning, Multimodality, and Values—spanning Mathematics, Physics, Chemistry, Life Sciences, and Earth and Space Sciences. Using the developed benchmark datasets, we have conducted a comprehensive evaluation of over 50 representative open-source and closed-source LLMs. All the results are publicly available and can be accessed online at www.scihorizon.cn/en . 1 1 footnotetext: Leading the LLM assessment. 2 2 footnotetext: Leading the scientific data assessment. 3 3 footnotetext: Project Leader. 1 Introduction Scientific data resources serve as the core driver propelling scientific discoveries and as a critical enabler for interdisciplinary research that integrates Artificial Intelligence (AI) with other disciplines. As AI technologies advance rapidly, particularly in Large Language Models (LLMs), AI-for-Science (AI4Science) has emerged as a ‌frontier research hotspot Zhang et al. (2019) ; Zhu et al. (2020) ; Zhang et al. (2021) ; Shen et al. (2021) ; Hu et al. (2025) ; Ji et al. (2025) ; Wang et al. (2025) . For instance, in structural biology, DeepMind’s AlphaFold has revolutionized protein structure prediction by achieving unprecedented accuracy, effectively resolving a challenge that previously demanded decades of experimental effort Jumper et al. (2021) . Likewise, LLMs, such as OpenAI’s ChatGPT, are expanding AI’s role in scientific research by not only enhancing data analysis capabilities Hassani and Silva (2023) ; Morgan (2023) ; Qin et al. (2025) ; Gong and Sun (2024) , but also assisting in literature review Temsah et al. (2023) , hypothesis generation Zhou et al. (2024) ; Park et al. (2024) , and complex reasoning Frieder et al. (2023) ; Guo et al. (2023) . These models facilitate the synthesis of vast scientific knowledge, accelerate discovery processes, and foster interdisciplinary collaboration, thereby reshaping the landscape of modern scientific inquiry. Despite rapid advancements in AI4Science, the field faces persistent challenges stemming from the reliance on large-scale, interdisciplinary datasets and the scarcity of AI-ready, high-quality data‌. Therefore, researchers advocate for systematic assessment frameworks that integrate metrics for data accuracy, completeness, and domain relevance. While traditional tools are instrumental in assessing data quality from different perspectives Schelter et al. (2018) ; Houston et al. (2018) ; Shi et al. (2019) ; Li et al. (2021) or emphasizing specific aspects of data readiness Clarke et al. (2019) ; Holland et al. (2020) ; Arca and Hewett (2020) ; Carlini et al. (2022) , they fall short of addressing the unique requirements of AI applications in the scientific domain. This leads to two challenges in evaluating scientific data in AI4Science: 1) AI researchers struggle to efficiently extract valuable insights from vast, domain-specific datasets, leading to the underutilization or misapplication of high-potential data that could drive significant advances in AI-driven scientific discovery; and 2) Researchers across disciplines lack clear criteria to assess whether their datasets align with AI model requirements, hindering the development and optimization of high-quality, AI-ready data. Meanwhile, after assessing the possibility of high-quality scientific data integration, a critical challenge lies in comprehensively and effectively assessing the application capabilities of AI4Science models at a fine-grained level. Despite the proliferation of diverse assessment frameworks for LLMs Chang et al. (2024) , those specifically designed for scientific applications remain scarce. Recently, several targeted benchmarks have been proposed, however, they exhibit notable limitations: 1) Most of them focus on specific disciplines Cobbe et al. (2021) ; Singhal et al. (2023) , lacking a unified framework that accommodates multiple scientific fields; 2) Their capability assessments are relatively narrow—for instance, JEEBench Arora et al. (2023) primarily evaluates reasoning abilities related to basic computations, while MultiMedQA focuses on assessing clinical knowledge Singhal et al. (2023) , both of which fall short of providing a more comprehensive and fine-grained evaluation; and 3) No existing framework assesses whether LLMs embody the correct scientific research values Wang et al. (2023) , which is crucial for the responsible adoption of AI techniques in scientific tasks. To bridge these critical gaps, here we present SciHorizon—an integrated assessment framework that evaluates AI4Science readiness from both AI-ready data and LLM perspectives. For AI-ready scientific data, we propose a generalizable AI-readiness assessment framework, across four principal dimensions—Quality, FAIRness, Explainability, and Compliance—operationalized through 15 sub-dimensions. To demonstrate its applicability, we analyze approximately 1,600 datasets published between 2018 and 2023, primarily consisting of those published in data resource papers in peer-reviewed journals (e.g., Scientific Data and ESSD ), identifying dataset recommendations for Earth, Life, and Materials Sciences to support AI-driven scientific advancements. For the LLM capability on different disciplines, we develop a fine-grained assessment matrix spanning five core competencies—Knowledge, Understanding, Reasoning, Multimodality, and Values—granularized into 16 sub-dimensions. Our benchmark suite, covering Mathematics, Physics, Chemistry, Life Sciences, and Earth and Space Sciences, enables systematic comparison of more than 50 representative open-source and closed-source LLMs. All the results are publicly available and can be accessed online at www.scihorizon.cn/en 1 1 1 The platform was initially launched on January 22, 2025 scihorizon (2025) . . 2 Related Works The related works of this paper can be grouped into two categories, namely Data AI-Readiness and LLM Benchmarking. 2.1 Data AI-Readiness The assessment of data AI-readiness has drawn more and more attention. Hiniduma et al. proposed a taxonomy of data readiness for AI metrics Hiniduma et al. (2024a) and developed a quantitative assessment framework for data AI readiness Hiniduma et al. (2024b) . They are mainly from a data modality perspective and lack support and applicability for scientific data. Regarding the scientific data, the FAIR principles (Findable, Accessible, Interoperable, Reusable) presented in 2016 have been widely recognized as basic characteristics for scientific data sharing Wilkinson et al. (2016) . The principles are also used for AI-readiness assessment for scientific data Huerta et al. (2023) ; Khan et al. (2023) . Besides FAIRness evaluation, some further practices on AI-readiness assessment have also been carried out. ESIP Data Readiness Cluster published a checklist to examine AI-readiness for open environmental datasets Cluster (2022) , covering data quality, data documentation, data access, and data preparation. The checklist serves more as a guideline than an

Read the complete text