Mix-Geneformer: Unified Representation Learning for Human and Mouse scRNA-seq Data

Yuki Nishio, Prof. Takayoshi Yamashita, Keita Ito, Tsubasa Hirakawa, Prof. Hironobu Fujiyoshi · 2025

Read Mix-Geneformer: Unified Representation Learning for Human and Mouse scRNA-seq Data by Yuki Nishio, Prof. Takayoshi Yamashita, Keita Ito, Tsubasa Hirakawa, Prof. Hironobu Fujiyoshi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07454.txt.

Introduction

Models that read single-cell gene activity usually train on one species at a time, which limits what mouse studies can say about human biology. Yuki Nishio and co-authors built Mix-Geneformer, a transformer that learns from human and mouse data together. It trains on about fifty million cells with two self-teaching tasks at once, masked-word style prediction and a contrast method, and it encodes genes by rank so strong signals stand out.

On cell-type labeling and simulated gene-perturbation tests, the joint model matched or beat species-specific rivals, reaching 95.8 percent accuracy on mouse kidney data against 94.9 for the best prior model. It also surfaced control genes that animal studies had already confirmed. The authors note heavy computing cost and uneven zero-shot transfer as open limits.

Catalog record

CREATORYuki Nishio, Prof. Takayoshi Yamashita, Keita Ito, Tsubasa Hirakawa, Prof. Hironobu Fujiyoshi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07454

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mix-Geneformer: Unified Representation Learning for Human and Mouse scRNA-seq Data Yuki Nishio, Prof. Takayoshi Yamashita, Keita Ito, Tsubasa Hirakawa, Prof. Hironobu Fujiyoshi Abstract Single-cell RNA sequencing (scRNA-seq) enables single-cell transcriptomic profiling, revealing cellular heterogeneity and rare populations. Recent deep learning models like Geneformer and Mouse-Geneformer perform well on tasks such as cell-type classification and in silico perturbation. However, their species-specific design limits cross-species generalization and translational applications, which are crucial for advancing translational research and drug discovery. We present Mix-Geneformer, a novel Transformer-based model that integrates human and mouse scRNA-seq data into a unified representation via a hybrid self-supervised approach combining Masked Language Modeling (MLM) and SimCSE-based contrastive loss to capture both shared and species-specific gene patterns. A rank-value encoding scheme further emphasizes high-variance gene signals during training. Trained on about 50 million cells from diverse human and mouse organs, Mix-Geneformer matched or outperformed state-of-the-art baselines in cell-type classification and in silico perturbation tasks, achieving 95.8% accuracy on mouse kidney data versus 94.9% from the best existing model. It also successfully identified key regulatory genes validated by in vivo studies. By enabling scalable cross-species transcriptomic modeling, Mix-Geneformer offers a powerful tool for comparative transcriptomics and translational applications. While our results demonstrate strong performance, we also acknowledge limitations, such as the computational cost and variability in zero-shot transfer. 1 Introduction Single-cell RNA sequencing (scRNA-seq) analysis scRNA is a powerful technique that enables the acquisition of gene expression information at the individual cell level, providing unprecedented insights into cellular heterogeneity and rare cell populations. This technology has become indispensable for understanding complex biological processes, including cancer and immune responses, and has generated vast datasets across diverse organisms, such as humans and mice. The increasing availability of these data necessitates the development of advanced analytical methods to fully leverage their potential for driving progress in medicine and the life sciences. Deep learning models, particularly those based on the Transformer architecture transformer , have shown remarkable success in analyzing large-scale and high-dimensional scRNA-seq data. Models like Geneformer geneformer and Mouse-Geneformer mouse-geneformer treat gene expression profiles as "sentences" and genes as "words," using attention mechanisms to capture intricate contextual relationships between genes. These models have proven valuable for various downstream tasks, including cell-type classification and in silico gene perturbation simulations, offering promising applications in drug discovery and regenerative medicine. However, Geneformer is specifically trained on human scRNA-seq data, limiting its generalization to other species. While Mouse-Geneformer addresses this by training on mouse data and achieving high performance in mouse-specific tasks, both models are species-specific and cannot effectively handle data from both humans and mice simultaneously. This limitation hinders their ability to robustly handle domain shifts between species and restricts their translational potential. To address this critical need, we introduce Mix-Geneformer, a novel Transformer-based model designed for joint learning from human and mouse scRNA-seq data to enable effective cross-species analysis. Mix-Geneformer employs a hybrid self-supervised learning strategy, combining Masked Language Modeling (MLM) with SimCSE-based contrastive learning, to learn consistent gene representations across species. This approach enhances analytical efficiency and facilitates real-world biological research. Contributions. The main contributions of this work are: • We propose Mix-Geneformer, a Transformer-based model for joint learning of scRNA-seq data from human and mouse. • We introduce a hybrid self-supervised learning scheme, integrating Masked Language Modeling (MLM) and SimCSE-based contrastive learning, to achieve consistent gene representations across species. • We demonstrate that Mix-Geneformer achieves competitive or superior performance compared to species-specific models in cell-type classification and in silico perturbation tasks, using a large-scale dataset of approximately 50,000,000 cells from human and mouse. • We evaluate zero-shot transfer capabilities in both human → mouse and mouse → human directions, showing the utility of our approach for translational research in drug discovery and disease studies. 2 Related Work Single-cell RNA sequencing (scRNA-seq) has revolutionized the study of gene expression by enabling the acquisition of transcriptomic information at the individual cell level. scRNA-seq offers a significant advantage over traditional bulk RNA-seq methods. This technology reveals cellular heterogeneity and allows for the characterization of rare cell types that are often masked in population-averaged data scRNA . Consequently, scRNA-seq data have become an indispensable resource for investigating diverse biological processes, including cancer development and immune responses, and are now widely available for a broad range of organisms, such as humans and mice. The analysis of large-scale, high-dimensional scRNA-seq datasets has greatly benefited from advancements in deep learning. Various deep learning architectures have been applied to scRNA-seq data for tasks such as dimensionality reduction and imputation scvae . In particular, Transformer-based models have emerged as powerful tools for capturing complex relationships within gene expression profiles enformer . The Transformer architecture transformer , originally developed for natural language processing, has proven to be highly effective in modeling gene–gene interactions in scRNA-seq data. These models treat gene expression profiles as “sentences” and individual genes as “words,” using attention mechanisms to learn contextual relationships between genes. Geneformer geneformer is a prominent example, pre-trained on large-scale human scRNA-seq data using Masked Language Modeling (MLM) and demonstrating strong performance in human-specific downstream tasks, including cell-type classification and in silico perturbation experiments. Mouse-Geneformer mouse-geneformer adopts the same Transformer architecture and pre-training strategy as Geneformer but is trained on mouse scRNA-seq data, achieving high accuracy in mouse-specific analyses. Additionally, scDeepSort scdeepsort employs a weighted graph neural network (GNN) framework to perform cell-type annotation on single-cell transcriptomics. It was pre-trained on two high-quality scRNA-seq atlases comprising 764,741 cells across 88 human and mouse tissues, achieving an overall annotation accuracy of 83.79% on 76 external test datasets without reliance on marker genes or reference profiles scdeepsort . Self-supervised learning techniques, such as Masked Language Modeling (MLM) bert , have been crucial for pre-training Transformer models on unlabeled scRNA-seq data. MLM enables the model to learn context-dependent representations of genes by predicting masked gene expression values, effectively capturing gene co-expression patterns. scBERT scbert is another example of a Transformer-based model for single-cell analysis that utilizes MLM for pre-training, achieving impressive results in cell-type annotation and batch effect correction. MLM has also been successfully applied in genomics, as demonstrated by DNABERT dnabert , which is pre-trained on DNA sequences for various genomic analyses. Pre-trained models like Geneformer and Mouse-Geneformer can be effectively utilized for various downstream tasks through transfer learning. By fine-tuning these models on specific tasks, such as cell-type classification or in silico perturbation, researchers can leverage the learned gene representations to gain insights into biological processes and disease mechanisms. Transfer learning has also been shown to be a valuable approach in network biology, as demonstrated in geneformer . Despite their success, Geneformer and Mouse-Geneformer are inherently species-specific, limiting their ability to directly analyze and compare scRNA-seq data from different species. This limitation hinders cross-species generalization and translational applications, which are crucial for translating findings from model organisms (e.g., mice) to human studies. While human and mouse gene expression patterns share conserved features, they also exhibit significant species-specific regulatory mechanisms and evolutionary differences. Consequently, existing species-specific models struggle to capture both shared and unique features, limiting their effectiveness in integrated cross-species analysis. To address this challenge and accelerate translational research, there is a growing need for general-purpose models capable of jointly analyzing scRNA-seq data from multiple species. 3

Read the complete text