NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations

Ke Ding, Brian Parker, Jiayu Wen · 2025

Read NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations by Ke Ding, Brian Parker, Jiayu Wen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13191.txt.

Introduction

Ke Ding, Brian Parker, and Jiayu Wen pre train a genomic language model that learns from every letter of DNA in a 2025 paper. Masked word methods, common in this field, train on only some positions and cost a lot to run. Their model, NucEL, borrows a rival design where a small generator alters letters and a second network must spot the changes.

That setup gives feedback at all positions and uses single letters as tokens for fine detail. Trained on the human genome, the model topped similar sized rivals on tasks such as finding promoters, enhancers, and protein binding sites. It even rivaled models 25 times larger, and its attention maps picked out known biological motifs.

Catalog record

CREATORKe Ding, Brian Parker, Jiayu Wen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13191

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations Ke Ding, Brian Parker, Jiayu Wen Abstract Pre-training large language models on genomic sequences has become a powerful approach for learning biologically meaningful representations. While masked language modeling (MLM)-based approaches, such as DNABERT and Nucleotide Transformer (NT), achieve strong performance, they are hindered by inefficiencies due to partial token supervision, pre-training/fine-tuning mismatches, and high computational costs. We introduce NucEL, the first ELECTRA-style pre-training framework for genomic foundation models, which overcomes these challenges. Through a discriminator network identifying tokens modified by a generator, NucEL achieves comprehensive token-level supervision across all sequence positions, thereby markedly improving training efficiency relative to the partial supervision of masked positions inherent in MLM frameworks. By integrating ModernBERT’s architectural advancements, including hybrid local-global attention and flash attention mechanisms, NucEL establishes an optimized BERT architecture for genomic sequence modeling. Unlike traditional methods that tokenize genomic sequences into 6-mers, NucEL implements single-nucleotide tokenization, enabling fine-grained resolution and improving both efficiency and interpretability. Pre-trained on the human genome only, NucEL achieves state-of-the-art performance on benchmark datasets across diverse downstream tasks in both human and non-human species, including regulatory element identification (e.g., promoters, enhancers), transcription factor binding prediction in human and mouse, open chromatin region classification, and histone modification profiles, surpassing MLM-based models of similar size and rivaling models 25 times larger, such as NT. Ablation studies provide critical insights into tokenization and masking strategies, optimizing ELECTRA-style pretraining for DNA sequences. Attention analyses reveal NucEL’s superior ability to capture biologically relevant sequence motifs compared to NT, offering valuable insights into its hierarchical learning process and regulatory element modeling capabilities. This work highlights the potential of ELECTRA-style pretraining as an efficient and effective strategy for advancing genomic representation learning with broad implications for future genomic research. 1 Introduction Understanding genomic sequences is a fundamental challenge in biology, as DNA encodes the instructions for cellular function, development, and evolution. Large language models (LLMs), which have transformed natural language processing (NLP), are now being adapted for biological sequences, showing remarkable success across genomics, transcriptomics, proteomics, metagenomics, and epigenomics ( Dalla-Torre et al. 2025 ; Sasse et al. 2024 ; Jumper et al. 2021 ; Abramson et al. 2024 ) . Genomic language models (gLMs) are particularly promising for decoding the functional logic of genomes, revealing how DNA sequences are organized and interact to produce the complexity of biological systems. Traditional deep learning methods for genomics, such as convolutional neural networks (CNNs) ( LeCun et al. 2015 ) and recurrent neural networks (RNNs) ( Hochreiter and Schmidhuber 1997 ) , are limited: CNNs struggle with long-range dependencies in DNA sequences, while RNNs process sequences inefficiently. Transformer-based architectures overcome these shortcomings by leveraging pre-training on vast unlabeled genomic datasets to learn meaningful representations of functional elements such as promoters and enhancers, facilitating effective transfer learning for downstream tasks ( Devlin et al. 2019 ) . However, existing transformer-based models, such as DNABERT ( Ji et al. 2021 ) and Nucleotide Transformer ( Dalla-Torre et al. 2025 ) , rely on masked language modeling (MLM), which is computationally inefficient. MLM learns from only a small subset of the input—typically 15% of masked tokens—underutilizing the majority of the sequence during training ( Clark et al. 2020 ) . Additionally, the pre-training (with masking) and fine-tuning (without masking) mismatch can impair transfer learning efficacy. To address these limitations, we introduce NucEL, the first genomic foundation model to adopt the ELECTRA framework’s replaced token detection (RTD) objective ( Clark et al. 2020 ) . Unlike MLM, RTD employs a generator-discriminator architecture, enabling dense token-level supervision across all input positions and significantly boosting training efficiency. NucEL integrates ModernBERT’s architectural innovations ( Warner et al. 2024 ) , including hybrid local-global attention ( Beltagy et al. 2020 ) and flash attention mechanisms ( Dao 2023 ) , to capture both short and long-range genomic dependencies effectively. Unlike k-mer ( Dalla-Torre et al. 2025 ; Ji et al. 2021 ) or BPE tokenization ( Zhou et al. 2023 ; Sanabria et al. 2024 ) , NucEL uses single-nucleotide tokenization, which may yield more tokens for long sequences but ensures fine-grained resolution and interpretability ( Dotan et al. 2024 ) . Combining single-nucleotide resolution with RTD optimizes efficiency and precision: it preserves base-level detail for tasks like mutation analysis, while dense supervision and hybrid attention mitigate the computational cost of longer sequences and enhance long-range interaction modeling. We evaluate NucEL across a range of benchmark tasks—including promoter classification, transcription factor binding prediction, and epigenetic mark profiling—and demonstrate that it achieves state-of-the-art performance while using significantly fewer parameters than larger MLM-based models. Our contributions include: • Introduced the first ELECTRA-style genomic foundation model, leveraging Replaced Token Detection (RTD) to enhance training efficiency over MLM-based approaches by utilizing all input tokens. • Achieved state-of-the-art performance on genome foundation model benchmarks with significantly fewer parameters than MLM-based models. • Utilized single-nucleotide resolution and fine-grained attention analyses to enable precise genomic modeling while enhancing interpretability by revealing complex biological representations. • Integrated ModernBERT’s hybrid local-global attention and flash attention mechanisms to efficiently capture genomic dependencies while optimizing computational performance. Our results demonstrate that NucEL provides a practical and efficient approach to learning biologically meaningful genomic representations, advancing genomic language modeling while reducing computational demands, thus making advanced genomic models more accessible to the broader research community. 2 Related Work Genomic sequence modeling has evolved rapidly with deep learning, particularly through transformer-based models adapted from natural language processing. Here, we review key developments in genomic foundation models, pretraining objectives and tokenization strategies, positioning NucEL within this landscape. 2.1 Genomic Language Models Transformer-based models have become a cornerstone of genomic representation learning. DNABERT ( Ji et al. 2021 ) pioneered this approach by adapting BERT’s masked language modeling (MLM) to DNA sequences, using k -mer tokenization to predict masked tokens for tasks like promoter prediction. DNABERT-2 ( Zhou et al. 2023 ) refined this with Byte Pair Encoding (BPE) tokenization and architectural improvements, enhancing performance on genomic benchmarks. The Nucleotide Transformer (NT) series ( Dalla-Torre et al. 2025 ) scaled this paradigm to billions of parameters, pretrained on multi-species datasets, demonstrating strong transfer learning capabilities. Additionally, ModernBERT ( Warner et al. 2024 ) introduced architectural innovations such as hybrid local-global attention and flash attention mechanisms, improving transformer efficiency for sequence modeling. However, MLM-based models are limited by partial token supervision (only \sim 15% of tokens) and a pretraining-finetuning mismatch, reducing efficiency. Alternatives to transformers include state-space models (SSMs) like HyenaDNA ( Nguyen et al. 2023 ) and Caduceus ( Schiff et al. 2024 ) , which use single-nucleotide resolution and reverse-complement awareness for efficient long-sequence modeling. Convolutional neural networks (CNNs), such as DeepBind ( Alipanahi et al. 2015 ) and Enformer ( Avsec et al. 2021 ) , remain relevant for capturing local motifs but struggle with long-range dependencies compared to transformers. 2.2 Pretraining Objectives Pretraining objectives shape the efficiency and effectiveness of genomic models. MLM, used in DNABERT ( Ji et al. 2021 ) and NT ( Dalla-Torre et al. 2025 ) , masks a subset of tokens for prediction, but its sparse supervision leaves most of the forward pass unutilized. Next-Sentence Prediction (NSP), used alongside with MLM in

Read the complete text