Gene42 : Long-Range Genomic Foundation Model With Dense Attention

Kirill Vishniakov, Boulbaba Ben Amor †, Engin Tekin, Nancy A. ElNaker, Karthik Viswanathan, Aleksandr Medvedev, Aahan Singh, Maryam Nadeem, Mohammad Amaan Sayeed, Praveenkumar Kanithi, Tiago Magalhaes, Natalia Vassilieva, Dwarikanath Mahapatra, Marco Pimentel, Shadab Khan · 2025

Read Gene42 : Long-Range Genomic Foundation Model With Dense Attention by Kirill Vishniakov, Boulbaba Ben Amor †, Engin Tekin, Nancy A. ElNaker, Karthik Viswanathan, Aleksandr Medvedev, Aahan Singh, Maryam Nadeem, Mohammad Amaan Sayeed, Praveenkumar Kanithi, Tiago Magalhaes, Natalia Vassilieva, Dwarikanath Mahapatra, Marco Pimentel, Shadab Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16565.txt.

Introduction

Kirill Vishniakov and co-authors introduce Gene42, a family of models that read long spans of DNA in two thousand twenty-five. The models work at the level of single letters and can hold spans far longer than most past models. That length is needed to catch patterns that link far-apart parts of the genome.

The design follows a standard decoder layout with full attention, rather than the shortcuts many rivals use. Training began on short spans and then grew in stages to the full length. The models scored well on guess accuracy and on rebuilt sequences. Across genome test tasks, from region typing to variant harm calls, they matched or beat the best known results.

Catalog record

CREATORKirill Vishniakov, Boulbaba Ben Amor †, Engin Tekin, Nancy A. ElNaker, Karthik Viswanathan, Aleksandr Medvedev, Aahan Singh, Maryam Nadeem, Mohammad Amaan Sayeed, Praveenkumar Kanithi, Tiago Magalhaes, Natalia Vassilieva, Dwarikanath Mahapatra, Marco Pimentel, Shadab Khan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16565

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gene42 : Long-Range Genomic Foundation Model With Dense Attention Kirill Vishniakov, Boulbaba Ben Amor †, Engin Tekin, Nancy A. ElNaker, Karthik Viswanathan, Aleksandr Medvedev, Aahan Singh, Maryam Nadeem, Mohammad Amaan Sayeed, Praveenkumar Kanithi, Tiago Magalhaes, Natalia Vassilieva, Dwarikanath Mahapatra, Marco Pimentel, Shadab Khan Abstract We introduce Gene42, a novel family of Genomic Foundation Models (GFMs) designed to manage context lengths of up to 192,000 base pairs (bp) at a single-nucleotide resolution. Gene42 models utilize a decoder-only (LLaMA-style) architecture with a dense self-attention mechanism. Initially trained on fixed-length sequences of 4,096 bp, our models underwent continuous pretraining to extend the context length to 192,000 bp. This iterative extension allowed for the comprehensive processing of large-scale genomic data and the capture of intricate patterns and dependencies within the human genome. Gene42 is the first dense attention model capable of handling such extensive long context lengths in genomics, challenging state-space models that often rely on convolutional operators among other mechanisms. Our pretrained models exhibit notably low perplexity values and high reconstruction accuracy, highlighting their strong ability to model genomic data. Extensive experiments on various genomic benchmarks have demonstrated state-of-the-art performance across multiple tasks, including biotype classification, regulatory region identification, chromatin profiling prediction, variant pathogenicity prediction, and species classification. The models are publicly available at huggingface.co/inceptionai . Gene42 is part of the Omics42 platform which includes also a family of protein LMs and a family of chemical LMs named Prot42 and Chem42, respectively. Refer to Omics42 blog at huggingface.co/inceptionai for further details. 1 Introduction Understanding the intricate details of the human genome is crucial for advancing our comprehension of diseases and developing personalized therapies tailored to an individual’s genetic makeup. This approach improves treatment efficacy and minimizes adverse effects [ BST17 ] , marking a significant shift from the traditional ”one-size-fits-all” model of medicine to a more targeted and effective strategy. The Human Genome Project, completed in 2003, marked a pivotal moment in genomic analysis. It provided a comprehensive reference that revolutionized the understanding of human genetics and laid the foundation for modern genomic research. Since then, advances in DNA sequencing technologies have significantly reduced the cost and time required to sequence genomes, improving access to these technologies and enabling precision medicine initiatives around the world, while fueling innovation in related fields [ SJM+23 ] . Building foundation models for genomic data analysis has been a dynamic area of research [ CDW+23 ] , producing models that are evaluated on various downstream tasks and benchmarks [ GMČ+23 , LLL+24 , DTGMR+23 ] . These models, referred to as Genomic Foundation Models (GFMs), leverage a range of advanced architectures and deep learning techniques to decode the complex language of DNA. By capturing long-range dependencies and contextual information within genomic sequences, they aim to provide more precise predictions and insights into genomic functions and interactions between different elements of the genome [ CDW+23 ] . GFMs are designed to take gene sequencing data as input to facilitate tasks such as predicting gene expression, identifying regulatory elements, and modeling complex biological systems. By improving the understanding of genomic sequences, these models can accelerate discoveries in precision medicine and drug design, leading to more effective solutions. The importance of such models is highlighted by two aspects of genomic data that have influenced the development of GFMs: (i) Single Nucleotide Polymorphisms (SNPs): Changes in a single nucleotide can significantly alter genes by affecting the coding sequence or regulatory elements. These changes can impact protein function or expression levels, potentially leading to the production of non-functional proteins, alterations in the protein activity or stability, or the activation of disease pathways. Such alterations can cause cells to enter disease states, including cancers, genetic disorders, and other health conditions [ Sha09 ] . (ii) Long-range Genomic Interactions: Genomic data exhibits long-range interactions, where regions of the genome that are distant in the linear sequence can physically interact through the three-dimensional folding of DNA. These interactions are crucial for regulating gene expression, as enhancers and other regulatory elements can influence genes located mega-bases away. Understanding these interactions is essential for comprehending complex regulatory networks that control cellular functions and how disruption of these networks can lead to diseases [ SLJD12 ] . Based on the above analysis, in this paper, we present a new family of Genomic Foundation Models capable of handling medium and long-range context lengths. We adopt a decoder-only LLaMA-style architecture, pretrained on genomic sequences. We have built a state-of-the-art (SoTA) family of transformer-based models leveraging dense attention for robust contextual understanding and the ability to capture long-range (up to 192 kbp) dependencies at single-nucleotide resolution, which represents the main contribution of this work. We demonstrate the efficiency of our model family across various short, medium, and long-range genomic tasks. The rest of the paper is structured as follows. In Section 2 , we begin with a brief literature review on Genomic Foundation Models. Section 3 outlines our methodology, detailing the process from genomic data tokenization to the development of our models with extended perceptive field through incremental continuous pretraining. Section 4 presents our comprehensive evaluations and comparisons to SoTA models. Finally, Section 5 offers concluding remarks and discusses the implications of our work for genomic data analysis and related challenges. 2 Related Works BERT-based approaches. Foundation models for genomic data have seen substantial progress with the introduction of DNABERT [ JZLD21 ] and its successor, DNABERT-2 [ ZJL+23 ] , which use BERT (Bidirectional Encoder Representations from Transformers) architecture. DNABERT’s approach of treating k-mers as tokens and utilizing bidirectional transformers enabled it to achieve high accuracy in various genomic prediction tasks. k-mer tokenization breaks the sequence into fixed-length subsequences, which can result in the loss of important contextual information that spans beyond the k-mer length. For instance, motifs or regulatory elements longer than k might be split across multiple k-mers. DNABERT-2, in contrast, builds upon its predecessor by employing an advanced tokenization strategy called Byte Pair Encoding (BPE) [ ZJL+23 ] . The Nucleotide Transformer (NT) [ DTGMR+23 ] combines a BERT-based encoder and a k-mer tokenization strategy. Trained on diverse datasets, including the Human Reference Genome (HRG), 1000 Genomes, and multispecies genome, NT models capture transferable, context-specific representations of k-mer sequences. The NT work also established a new benchmark called the Nucleotide Transformer (NT) Benchmark and a leaderboard to evaluate models on 18 genomic prediction tasks. GENA-LM [ FKP+23 ] , another family of models following a similar approach, uses a BERT architecture and includes a family of transformer-based models designed to process extensive DNA sequences up to 36k base pairs. Utilizing BPE tokenization, GENA-LM can encode long sequence fragments. In addition to the Human Reference Genome (HRG), GENA-LM uses multispecies data and variations curated from the 1000G Project. More recently, [ KSF+24 ] investigated enhancing GENA-LM models using Recurrent Memory Transformers (RMT) [ BKB22 ] to better handle ultra-long DNA sequences. By incorporating memory tokens and processing sequences in segments, RMT allows models to process sequences up to 196 kbp. Decoder-based approaches. HyenaDNA [ NPF+24 ] is a model capable of processing DNA sequences up to 1 million tokens at single nucleotide resolution. It leverages implicit convolutions to capture long-range dependencies, allowing for much longer sequence lengths than

Read the complete text