Multi-modal single-cell foundation models via dynamic token adaptation
Wenmin Zhao, Ana Solaguren-Beascoa, Grant Neilson, Louwai Muhammed, Liisi Laaniste · 2025
Read Multi-modal single-cell foundation models via dynamic token adaptation by Wenmin Zhao, Ana Solaguren-Beascoa, Grant Neilson, Louwai Muhammed, Liisi Laaniste (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13049.txt.
Introduction
Wenmin Zhao, Ana Solaguren-Beascoa, and colleagues join two kinds of foundation models for single-cell biology in a 2025 study. One model family reads DNA sequence, while another reads single-cell measurements. Each alone misses what the other sees, and gene regulation depends on both sequence and cell context.
Dynamic token adaptation bridges the two, translating DNA embeddings into the token space of the single-cell model. Training an adapter on that bridge lets the combined system predict regulation in varied genetic settings. The authors probe how DNA changes ripple into the learned regulatory networks. The method is general, with the single-cell case as the worked example.
Catalog record
| CREATOR | Wenmin Zhao, Ana Solaguren-Beascoa, Grant Neilson, Louwai Muhammed, Liisi Laaniste |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13049
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-modal single-cell foundation models via dynamic token adaptation Wenmin Zhao, Ana Solaguren-Beascoa, Grant Neilson, Louwai Muhammed, Liisi Laaniste Abstract Recent advances in applying deep learning in genomics include DNA-language and single-cell foundation models. However, these models take only one data type as input. We introduce dynamic token adaptation and demonstrate how it allows combining these models to predict gene regulation at single-cell level in different genetic contexts. Although the method is generalisable, we focus on an illustrative example by training an adapter from DNA-sequence embeddings to a single-cell foundation model’s token embedding space. As qualitative evaluation, we assess the impact of DNA sequence changes on the model’s learned gene regulatory networks by mutating the transcriptional start site of the transcription factor GATA4 in silico , observing predicted expression changes in its target genes in fetal cardiomyocytes. 1 Introduction There have been rapid advances in training single-cell foundation models on large single-cell RNA sequencing (scRNA-seq) datasets ( Theodoris et al. 2023 ; Yang et al. 2022 ; Cui et al. 2023 ) . These models represent each gene in a single-cell transcriptome as an input text token and do not integrate genetic information, making it hard to interpret gene expression predictions under genetic changes. On the other hand, DNA language models have been trained to predict epigenetic signals from the DNA sequence ( Kelley et al. 2018 ; Kelley 2020 ; Avsec et al. 2021b ; Avsec et al. 2021a ) , and can be fine-tuned to predict the expression values of individual genes across cells in a scRNA-seq dataset ( Schwessinger et al. 2023 ) . However, these models do not take cell-level co-regulation into account when predicting a gene’s expression, instead, they focus on predicting epigenetic signals or the expression for each gene separately. Our method of combining the modelling of both DNA sequences and single-cell transcriptomics data is inspired by unified embedding architectures for multi-modal large language models (LLMs), which convert an image into embedding vectors as a set of special tokens that are prepended to the input text tokens ( Cho et al. 2021 ) . Similar approaches using an adapter to provide additional information to the transcriptome have recently been applied to single-cell models ( Maleki et al. 2025 ; Levine et al. 2023 ) , however, these methods have been restricted to a few additional tokens encoding a single entity (e.g. cell-type, disease state, molecule of drug treatment). In this paper, we propose extending the approach to all tokens in the input to allow their embeddings to flexibly encode additional information from a different modality that may change between data samples, which we call dynamic token adaptation (DTA). As an application, we introduce Bio-DTA, a novel multi-modal model that learns from single-cell transcriptomes and DNA sequences jointly. Finally, we demonstrate that the model has learned dynamic co-regulation by assessing the impact of genetic changes to the DNA sequence of the transcription factor GATA4 in silico on the model’s predictions for its targets. 2 Methods Training data We downloaded scRNA-seq data from 33,364,242 unique cells across 265 datasets in the census dataset (version 2023-07-25) from the CellXGene data portal ( CZI Single-Cell Biology Program et al. 2023 ) . Data processing followed Theodoris et al. 2023 , representing each single-cell transcriptome as a sequence of gene names of maximum length 2,048 ordered by their median-normalised expression. We excluded cancer cells and cells with <500 expressed genes. Transcriptional start sites (TSSs) of all protein coding genes were obtained from Ensembl (GRCh38.108). For each gene, 196,608 bp around the TSS of the reference genome was inputted to Enformer to compute the mean embedding of dimension 3,072 across the positions of the pooling convolutional filters. The Enformer checkpoint was obtained from https://github.com/lucidrains/enformer-pytorch . Model architecture and training Bio-DTA combines a DNA language model with a single-cell foundation model via token adapters. Although our method is flexible and can accommodate other architectures, in the experiments presented here, the single-cell foundation model is based on a bidirectional transformer encoder-only architecture (BERT). The model receives a single-cell transcriptome of length 2,048 as an ordered sequence of gene names as input, which are mapped to integer identifiers called token IDs ( Theodoris et al. 2023 ) . In a usual BERT model, each token ID is mapped to a unique and trainable embedding vector that forms the input to the transformer encoder ( Devlin et al. 2019 ) . In contrast, Bio-DTA (Figure 1 ) projects Enformer’s aggregated embeddings for each input gene to the token embedding size using an adapter layer (e.g. a multilayer perceptron followed by a softplus activation). This forms the input to the transformer encoder capturing genetic information. Here, a gene’s token embedding is not unique and may change if the gene’s input DNA sequence changes, and the same adapter is used for all the genes. Bio-DTA outputs the token IDs of the input gene names according to the single-cell foundation model. Bio-DTA was trained end-to-end with a masked language modelling task (masking 15\% of input tokens) for three epochs. For more implementation details as well as hyperparameters used for the final model and training procedure see Appendix A.1 . Figure 1: Schematic of Bio-DTA. Fixed token embeddings are replaced with a projection of aggregated Enformer embeddings of the gene’s DNA sequence. 3 In silico mutagenesis reveals learned connection dynamics in Bio-DTA To assess how changes in the input DNA sequence of one gene affects learned co-regulation networks, we followed Theodoris et al. 2023 and focussed on GATA4 and TBX5 , two known congenital heart disease genes, which are co-expressed during cardiac morphogenesis, physically interact and have co-bound targets ( Misra et al. 2014 ) . We obtained 103 transcriptomes from fetal cardiomyocytes expressing GATA4 ( Knight-Schrijver et al. 2022 ) . We then introduced random mutations in silico in 100 bp around the GATA4 TSS, which reduced Enformer’s predicted expression (Figure 2 ). For each single-cell transcriptome in the dataset, we performed a forward pass and extracted the contextualised embedding for each input gene from the penultimate layer of Bio-DTA, once when using the unchanged GATA4 embedding, and once when using the mutated embedding as input for GATA4 . As the retained gene embeddings are from deeper layers of Bio-DTA’s BERT encoder, they may capture information about other co-regulated genes in the input sequence. To test if these depend on genetic changes, we calculated the cosine similarity between the gene embeddings before and after GATA4 ’s in silico mutagenesis for each transcriptome. Indeed, the embeddings of experimentally identified GATA4 and TBX5 targets (defined based on ChIP-seq data in Theodoris et al. 2023 ) had a significant drop in cosine similarity compared with the remainder of the genome ( p<0.05 , Wilcoxon test), while embeddings of housekeeping genes remained stable (Table 1 ). Figure 2: Enformer predicted Cap analysis of gene expression (CAGE) tracks of gene expression centered around the GATA4 TSS for the reference genome (top), in silico mutated sequence (random sequence replacing 100 bp around TSS) (middle), and the difference in predictions between the in silico mutated and reference sequence showing a large decrease around the TSS in the middle of the track (bottom). P-value FDR adjusted