BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects
Hongyang Li 1,∗ , Sanjoy Dey 1,∗ , Bum Chul Kwon 1 , Michael Danziger 2 , Michal Rosen-Tzvi 2 , Jianying Hu 1 , James Kozloski 1 , Ching-Huei Tsou 1 , Bharath Dandala 1,† , Pablo Meyer 1,† 1 IBM Research, Yorktown Heights, NY, USA 2 IBM Research, Haifa, Israel ∗ These authors contributed equally † T · 2025
Read BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects by Hongyang Li 1,∗ , Sanjoy Dey 1,∗ , Bum Chul Kwon 1 , Michael Danziger 2 , Michal Rosen-Tzvi 2 , Jianying Hu 1 , James Kozloski 1 , Ching-Huei Tsou 1 , Bharath Dandala 1,† , Pablo Meyer 1,† 1 IBM Research, Yorktown Heights, NY, USA 2 IBM Research, Haifa, Israel ∗ These authors contributed equally † T (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05265.txt.
Introduction
Hongyang Li and co-authors present BMFM-DNA, a foundation model for DNA that takes genetic variants into account, in a 2025 paper. Language models have been retrained on DNA, treating base sequences as text. But most such models ignore the small letter changes, called SNPs, that differ between people and shape biological function.
The team pretrained two versions on a modern text model base: one on the plain reference genome, and one using a new scheme that writes variants into the sequence. The variant-aware version did better on every fine-tuning task they tried. They also tested ways to fill in missing variants. The models are released to help build better tests of variant effects.
Catalog record
| CREATOR | Hongyang Li 1,∗ , Sanjoy Dey 1,∗ , Bum Chul Kwon 1 , Michael Danziger 2 , Michal Rosen-Tzvi 2 , Jianying Hu 1 , James Kozloski 1 , Ching-Huei Tsou 1 , Bharath Dandala 1,† , Pablo Meyer 1,† 1 IBM Research, Yorktown Heights, NY, USA 2 IBM Research, Haifa, Israel ∗ These authors contributed equally † T |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects Hongyang Li 1,∗ , Sanjoy Dey 1,∗ , Bum Chul Kwon 1 , Michael Danziger 2 , Michal Rosen-Tzvi 2 , Jianying Hu 1 , James Kozloski 1 , Ching-Huei Tsou 1 , Bharath Dandala 1,† , Pablo Meyer 1,† 1 IBM Research, Yorktown Heights, NY, USA 2 IBM Research, Haifa, Israel ∗ These authors contributed equally † To whom correspondence should be addressed: [email protected], [email protected] Abstract Large language models (LLMs) trained on text demonstrated remarkable results on natural language processing (NLP) tasks. These models have been adapted to decipher the language of DNA, where sequences of nucleotides act as “words” that encode genomic functions. However, the genome differs fundamentally from natural language, as it lacks clearly defined words or a consistent grammar. Although DNA language models (DNALMs) such as DNABERT, GENA-LM have achieved high level of performance on genome-related biological tasks, these models do not encode biological functions in the presence of sequence variations. To address this problem, we pre-train foundation models that effectively integrate sequence variations, in particular Single Nucleotide Polymorphisms (SNPs), as they underlie important biological functions. Specifically, we use ModernBERT to pre-train two different Biomedical Foundation Models (BMFM), namely, BMFM-DNA-REF in which the model is trained with sequences of varying lengths along with their reverse complements derived from the reference genome and BMFM-DNA-SNP in which the model is trained with sequences created using a novel representation scheme that encodes sequence variations. Our findings indicate that integrating sequence variations into DNALMs helps capture the biological functions as seen in improvements on all fine-tuning tasks. To explore the model’s practical utility, we experimented with various strategies for SNP imputation on promoter detection task introduced in DNABERT-2. However, we acknowledge that the current benchmarks are limited in their ability to fully evaluate these models. To enable more comprehensive assessment in the future and encourage community contributions, we release our models through HuggingFace and the code to reproduce the results at https://github.com/BiomedSciAI/biomed-multi-omic . 1 Introduction The advent of high-throughput DNA sequencing technologies has resulted in an unprecedented volume of genomic data, including over 500,000 human genomes, 3,000 genomes from higher organisms, and millions of bacterial genomes. Despite this wealth of data, our understanding of functional genomics remain limited. While the genetic code and non-coding regulatory regions have been well-studied, they account for only about 1.5% international2001 and 5–20% encode2012 , respectively, of the approximately 3 billion base pairs in the human genome. To address the complexity of interpreting this vast sequence space, machine learning has emerged as a powerful tool. It has been successfully applied to tasks such as gene and splice site identification jaganathan2019 , prediction of single nucleotide polymorphism (SNP) locations poplin2018 , functional effect prediction of variants zhou2015 , and the discovery of regulatory DNA motifs relevant to gene expression avsec2021 . Recently, machine learning has experienced a paradigm shift with the rise of general-purpose foundation models, which are trained using self-supervised techniques on large-scale, unlabeled datasets and can be adapted to a wide range of downstream tasks bommasani_opportunities_2022 . These models demonstrated impressive capabilities in solving a wide range of problems in natural language processing (NLP) and natural language understanding (NLU) zhou_comprehensive_2024 . This shift has unlocked new opportunities to study genome function at scale by using models to learn generic representations from genomic data that can be transferred to quantitative biological tasks. Inspired by this paradigm shift, a growing number of foundation methods have recently been proposed to “decode the biological language” by leveraging large-scale genomics, proteomics, transcriptomics, and epigenomics data si_foundation_2024 . Genomic foundation models, often based on transformer architectures vaswani2017 , extend the central dogma metaphor by treating DNA as a language, with its own words, syntax, and grammar that encode the rules of biology. However, decoding the biological language remains a profound challenge, as the genome, unlike natural language, lacks a clearly defined vocabulary or consistent grammar. It is more akin to a vast manuscript compiled over evolutionary time: layered, fragmented, and overwritten, which some have called a biological palimpsest haig2004 . In this view, the role of a DNA language model is not just to read a coherent narrative, but to reconstruct meaning from overlapping, context-dependent signals—fragments of many narratives that span millions of years. Another important aspect of genomic variation lies in single nucleotide polymorphisms (SNPs), which are single-base changes in DNA sequence that are present in a large human population. Indeed, SNPs are the most common type of genetic variation among individuals and can influence everything from physical traits to disease susceptibility and drug response maurano2012 . Understanding the functional impact of SNPs, especially those located in non-coding regions, is crucial for linking genetic variation to phenotypic outcomes and advancing personalized medicine. SNPs in DNA can be compared to polysemous words or tokens in a sentence. Just as each word or token in a sentence carries specific meaning and influences the overall meaning of a sentence, each SNP in a genome represents a small but meaningful variation that can impact biological processes and phenotypic traits. Genomic foundation models such as DNABERT-2 and GENA-LM zhou2023dnabert ; fishman2025 trained on the human genome and genomes from several species, NT-transformer dalla2024nucleotide trained on 3,202 human genomes and 850 genomes from diverse species, and Evo nguyen2024sequence trained on 2.7 million prokaryotic and phage genomes have demonstrated success in predicting several biological properties encoded in DNA such as recognizing transcription factor binding sites, predicting gene expression, or predicting promoter activity. DNABERT-2 zhou2023dnabert provided conceptual and empirical insights into genome tokenization, and demonstrated Byte Pair Encoding (BPE), a statistics-based data compression algorithm that constructs tokens by iteratively merging the most frequent co-occurring genome segment in the corpus can encode the genomic language better than k-mer or fixed-length permutations of A, T, C, and G. Other approaches, such as GROVER, have focused on the biological interpretation of BPE tokens sanabria2024 , demonstrating that token frequencies can reveal insights into genomic patterns. For example, they can highlight a preference for A/T-rich sequences in gene coding regions, which are less prone to mutation, or show a higher GC content near Alu regions. However, these models still require additional pre-training tasks to better capture the broader biological context of genomic sequences. Conversely, SEI chen2022 has adopted a multi-task learning framework to enhance DNA representation learning by combining both self-supervised and supervised objectives. The self-supervised learning objective captures intrinsic relationships between DNA sequences, while the supervised objective leverages available metadata, such as cell type specific DNA accessibility, methylation state, chromatin structure, histone modifications, and external knowledge (e.g. signaling pathway ontology relationships). This dual approach enables the model to encode explicit relationships, facilitating the capture of higher-level biological connections. However, in eukaryotes, questions remain about whether the pre-training strategies of DNA language models (DNALMs) effectively capture key biological properties and consistently outperform traditional approaches tang2024evaluating ; patel2024dart . In particular, the focus of current DNALMs has been modeling the reference genome, while ignoring the encoding of natural genomic variations in the pre-training stage. This is an important omission, because many drivers of complex inherited disease risk lie in regulatory regions of the genome and SNPs occur almost once in every 1,000