NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction

Zhongmin Li, Runze Ma, Jiahao Tan, Chengzi Tan, Shuangjia Zheng · 2025

Read NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction by Zhongmin Li, Runze Ma, Jiahao Tan, Chengzi Tan, Shuangjia Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02888.txt.

Introduction

Zhongmin Li, Runze Ma, Jiahao Tan, Chengzi Tan, and Shuangjia Zheng built NABench, a large benchmark for models that predict how DNA and RNA changes affect fitness. A single letter change in a sequence can shift function a lot, and new foundation models claim to predict those shifts from sequence alone. Past data sets were too uneven to compare the models fairly.

NABench gathers 162 high-throughput assays and 2.6 million mutated sequences across many DNA and RNA families, with standard splits and rich metadata. The team tested twenty-nine leading models under zero-shot, few-shot, transfer, and supervised settings. Scores varied widely by task and molecule type, exposing clear strengths and failure modes. The open release aims to set firm baselines for nucleic acid modeling.

Catalog record

CREATORZhongmin Li, Runze Ma, Jiahao Tan, Chengzi Tan, Shuangjia Zheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02888

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NABench: Large-Scale Benchmarks of Nucleotide Foundation Models for Fitness Prediction Zhongmin Li, Runze Ma, Jiahao Tan, Chengzi Tan, Shuangjia Zheng Abstract Nucleotide sequence variation can induce significant shifts in functional fitness. Recent nucleotide foundation models promise to predict such fitness effects directly from sequence, yet heterogeneous datasets and inconsistent preprocessing make it difficult to compare methods fairly across DNA and RNA families. Here we introduce NABench, a large-scale, systematic benchmark for nucleic acid fitness prediction. NABench aggregates 162 high-throughput assays and curates 2.6 million mutated sequences spanning diverse DNA and RNA families, with standardized splits and rich metadata. We show that NABench surpasses prior nucleotide fitness benchmarks in scale, diversity, and data quality. Under a unified evaluation suite, we rigorously assess 29 representative foundation models across zero-shot, few-shot prediction, transfer learning, and supervised settings. The results quantify performance heterogeneity across tasks and nucleic-acid types, demonstrating clear strengths and failure modes for different modeling choices and establishing strong, reproducible baselines. We release NABench to advance nucleic acid modeling, supporting downstream applications in RNA/DNA design, synthetic biology, and biochemistry. Our code is available at https://github.com/mrzzmrzz/NABench . \svgsetup inkscapelatex=true, inkscapearea=page 1 Introduction Sequence variation in nucleic acids, such as single-nucleotide substitutions and small insertions or deletions, can profoundly alter molecular structure and function, thereby influencing fitness ( Sanjuán et al. 2004 ; Orr 2009 ; Cuevas et al. 2012 ; Huang et al. 2021 ) . Accurately modeling this complex, high-dimensional sequence–fitness landscape is essential for identifying pathogenic variants ( Riesselman et al. 2018 ; Agarwal et al. 2023 ; Pucci et al. 2024 ; Ito et al. 2025 ) and for guiding the rational engineering of DNA and RNA elements ( Ward et al. 2023 ; Li et al. 2024b ; Ma et al. 2025 ) , with broad implications for synthetic biology and therapeutic development ( Yi & Dean 2019 ; Wagner 2023 ; Gosai et al. 2024 ) . Recently, nucleotide foundation models (NFMs) ( He et al. 2025 ; Dalla-Torre et al. 2025 ; Nguyen 2025 ; Wu et al. 2025 ) have introduced a new paradigm for nucleic acid fitness prediction. Pre-trained in a self-supervised manner on massive collections of DNA and RNA sequences, NFMs learn comprehensive and transferable representations, which capture complex long-range dependencies and subtle evolutionary signals that are often overlooked by traditional methods relying on local features or hand-crafted descriptors. Consequently, these advances have enabled more accurate prediction of fitness directly from raw nucleotide sequences. However, evaluating the effectiveness of these models remains challenging, which in turn constrains the development and application of NFMs. Firstly, existing evaluations are typically conducted on diverse and contrived datasets, making direct and fair comparison difficult. Moreover, prior studies ( Arora et al. 2025 ; Ren et al. 2024 ) have shown that model performance can vary substantially across nucleic acid families and prediction tasks. This heterogeneity hinders researchers from accurately assessing the true capabilities and failure modes of different architectures, and prevents them from selecting the most suitable model for a given task. To address these limitations, we present NABench , a large-scale benchmark specifically designed for nucleic acid fitness prediction. NABench integrates an extensive collection of experimental measurements from deep mutational scanning (DMS) ( Fowler & Fields 2014 ) and systematic evolution of ligands by exponential enrichment (SELEX) ( Tuerk & Gold 1990 ; Ellington & Szostak 1990 ) , comprising over 2.6 million mutated sequences from more than 160 experiments reported in 33 studies. Among these, raw sequencing data from more than 110 experiments are carefully processed through quality assessment, length filtering, paired-end merging, frequency estimation, clustering, and statistical analysis. These steps require considerable manual effort and computational resources to ensure that only valid sequences are retained and that the resulting datasets meet the standards for reliable evaluation. The curated datasets span a wide range of DNA and RNA families, such as mRNA, tRNA, ribozymes, enhancers, promoters, and other functional nucleic acids, and cover diverse functional categories and mutation depths. Overall, NABench is over 8 \times larger than the latest RNAGym ( Arora et al. 2025 ) benchmark in nucleic fitness tasks. Beyond scale, NABench systematically evaluates 29 representative foundation models within a unified and standardized framework to enable fair and robust comparisons. The candidate set covers diverse architectures ( e.g. , BERT, GPT, Hyena). Concretely, NABench incorporates multiple dataset partitioning strategies ( e.g. , random and contiguous splits) to ensure unbiased evaluation, and supports four evaluation settings, including zero-shot, few-shot, supervised, and transfer learning, to comprehensively assess model performance in realistic application scenarios. Through comprehensive evaluation of nucleic acid fitness modeling, NABench enables advances in rational DNA/RNA design, property prediction, and engineering optimization, while offering new insights into next-generation nucleotide foundation models. 2 Related Work 2.1 Biomolecular Fitness Prediction Fitness can be defined as a mapping between biological sequences and a specific property ( Romero & Arnold 2009 ) . Biomolecular fitness prediction mainly relied on evolutionary information before the emergence of deep learning. The functional constraints encoded in homologous sequences served as implicit signals, and methods such as multiple sequence alignments (MSA) and position-specific scoring matrices (PSSM) were used to assess mutational effects ( Schroeder 2009 ; Palmeri et al. 2014 ) . With the advent of large-scale experimental fitness data from techniques like deep mutational scanning (DMS), data-driven learning methods quickly gained prominence. Early models typically employed convolutional or recurrent neural networks to extract local and sequential patterns from protein sequences for mutation effect prediction ( Yang et al. 2019 ; Freschlin et al. 2022 ) . In recent years, large-scale self-supervised pretraining has become the dominant paradigm. Protein language models such as ESM ( Lin et al. 2023 ) and nucleic acid language models such as RNA-FM ( Chen et al. 2022 ) are able to learn broad, transferable representations of sequence regularities and biophysical constraints, demonstrating emergent capability in fitness prediction tasks. Meanwhile, hybrid approaches that integrate sequence and structure information, such as SaProt ( Su et al. 2023 ) S3F ( Zhang et al. 2024 ) , ProtSST ( Li et al. 2024a ) and DPLM ( Wang et al. 2025b ) have also been explored. Given the current limited prediction accuracy in structure prediction for nucleic acids and complex assemblies ( Kretsch et al. 2025 ) , as well as the absence of reliable target structures in many practical scenarios, this work focuses on a systematic evaluation of sequence-only foundation models, enabling fair comparisons of their fitness prediction performance. 2.2 Nucleotide Foundation Models and Fitness Benchmarks Nucleotide foundation models, which are large-scale architectures pretrained on nucleotide sequence data, have advanced significantly in recent years, demonstrating broad transferability and emergent capabilities for various computational biology tasks ( Guo et al. 2025 ; Benegas et al. 2025 ) . These models, such as RNA-FM ( Chen et al. 2022 ) , Evo series ( Merchant 2024 ; Nguyen 2025 ) , LucaOne ( He et al. 2025 ) and Nucleotide Transformer ( Dalla-Torre et al. 2025 ) , leverage self-supervised learning on massive nucleotide sequence corpora to extract generalizable representations, enabling zero-shot and few-shot prediction for diverse biological tasks across DNA and RNA families. However, their fair and comprehensive evaluation has become

Read the complete text