SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention

Jiahao Li Jiayi Dong Peng Ye Xiaochi Zhou Haohai Lu Fei Wang College of Computer Science and Artificial Intelligence, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {lijiahao23, jiayidong21, pye24, xczhou24, hhlu24}@m.fudan.edu.cn, [email protected] · 2026

Read SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention by Jiahao Li Jiayi Dong Peng Ye Xiaochi Zhou Haohai Lu Fei Wang College of Computer Science and Artificial Intelligence, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {lijiahao23, jiayidong21, pye24, xczhou24, hhlu24}@m.fudan.edu.cn, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16776.txt.

Introduction

Li and colleagues open SAVE, a unified generative framework built on conditional Transformers for multi-set single-cell modeling. Old methods treat genes as solo tokens, missing higher-level biological links and doing poorly. SAVE groups meaning-wise related genes into blocks, catching higher-order dependencies among gene modules. Flow Matching and condition masking enable free sim and generalization to unseen condition mixes. Across set making, batch effect fixing and shock guessing, SAVE beats state-of-the-art methods in lean settings.

Catalog record

CREATORJiahao Li Jiayi Dong Peng Ye Xiaochi Zhou Haohai Lu Fei Wang College of Computer Science and Artificial Intelligence, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {lijiahao23, jiayidong21, pye24, xczhou24, hhlu24}@m.fudan.edu.cn, [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55176
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16776

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention Jiahao Li Jiayi Dong Peng Ye Xiaochi Zhou Haohai Lu Fei Wang College of Computer Science and Artificial Intelligence, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {lijiahao23, jiayidong21, pye24, xczhou24, hhlu24}@m.fudan.edu.cn, [email protected] Abstract Modeling single-cell gene expression across diverse biological and technical conditions is crucial for characterizing cellular states and simulating unseen scenarios. Existing methods often treat genes as independent tokens, overlooking their high-level biological relationships and leading to poor performance. We introduce SAVE, a unified generative framework based on conditional Transformers for multi-condition single-cell modeling. SAVE leverages a coarse-grained representation by grouping semantically related genes into blocks, capturing higher-order dependencies among gene modules. A Flow Matching mechanism and condition-masking strategy further enhance flexible simulation and enable generalization to unseen condition combinations. We evaluate SAVE on a range of benchmarks, including conditional generation, batch effect correction, and perturbation prediction. SAVE consistently outperforms state-of-the-art methods in generation fidelity and extrapolative generalization, especially in low-resource or combinatorially held-out settings. Overall, SAVE offers a scalable and generalizable solution for modeling complex single-cell data, with broad utility in virtual cell synthesis and biological interpretation. Our code is publicly available at https://github.com/fdu-wangfeilab/sc-save 1 Introduction The rapid expansion of high-throughput single-cell RNA sequencing (scRNA-seq) technologies provides unprecedented opportunities to computationally model and simulate diverse cellular states under complex experimental conditions 17 ; 31 ; 16 . Generative models that can predict gene expression profiles under unseen combinations of covariates—such as cell type, disease state, and perturbation—can greatly reduce experimental costs and accelerate biological discovery. Variational Autoencoders (VAEs) have become the foundation of many generative frameworks in single-cell omics, with scVI 21 being a notable example. While effective in learning latent representations and enabling batch correction and imputation, such models typically assume simple architectures and are limited in their ability to model complex, combinatorial interactions among multiple external conditions. Addressing this limitation is crucial for simulating realistic cellular responses in multi-conditional settings 25 . To model condition-dependent gene expression patterns at scale, recent approaches have adopted Transformer-based architectures, representing cells as gene token sequences and conditions as condition tokens 9 ; 2 . These so-called “foundation models” rely on masked modeling objectives to capture the relationships between genes and covariates. However, they face several fundamental challenges: (1) they assume a flat, token-level view of genes, neglecting biological structures such as gene modules or pathways; (2) they fail to model the global expression distribution, focusing primarily on non-zero values and ignoring the informative zero inflation inherent in scRNA-seq data 29 ; and (3) they are typically not integrated into a generative framework capable of sampling from the learned conditional distribution. We draw inspiration from masked generative modeling in computer vision, such as 4 , which demonstrates that unordered data can be effectively modeled using coarse-grained representations like image patches. Gene expression data shares key properties with images in this context—it is high-dimensional, sparse, and inherently unordered. This suggests that a similar coarse-graining strategy, modeling at the level of ’gene blocks’ rather than individual genes, could prove highly effective. However, a key challenge arises: unlike pixels, which are grouped by spatial proximity, genes lack a natural local structure. To overcome this, we propose forming blocks based on semantic similarity. We leverage Large Language Models (LLMs), pre-trained on extensive text corpora, to extract rich features from the comprehensive gene descriptions in the NCBI database. Genes exhibiting high semantic similarity are then aggregated into blocks. These blocks serve as coarser-grained tokens, upon which we apply an attention mechanism to learn the complex relationships among them. In this work, we propose SAVE ( S ingle-cell Gene Block A ttention-based V ariational g E nerative framework), a unified framework for conditional single-cell data generation and integration. SAVE integrates the latent space structure of VAEs with a coarse-grained Transformer that attends over meaningful gene blocks, effectively capturing high-order dependencies. To enhance conditional generation, SAVE employs a Flow Matching , enabling simulation under complex, unseen condition combinations. Our main contributions are summarized as follows: • We introduce Gene Block Attention, a attention mechanism that captures high-order relationships among blocks of genes. • We develop masked modeling strategy on Flow Matching and VAE to enhance SAVE’s ability to learn the conditional distribution of cell states. • We demonstrate that SAVE achieves state-of-the-art performance on multiple tasks, including batch alignment, perturbation prediction, and simulation under unseen condition combinations. 2 Related work 2.1 Generative Modeling for Single-Cell Data Variational Autoencoders (VAEs) have been widely adopted for modeling single-cell RNA-seq data. Notably, scVI 21 introduces a zero-inflated negative binomial (ZINB) likelihood and encodes covariates into the latent space for tasks like batch correction and imputation. However, traditional VAE-based models primarily focus on representation learning rather than flexible data generation across complex conditions 36 . To improve generative capabilities, recent models have explored more expressive architectures. CFGen 28 , a flow-based model, applies optimal transport-guided diffusion and classifier-free guidance to model conditional distributions. Similarly, scDiffusion 26 combines latent diffusion with a pre-trained autoencoder 13 and integrates condition labels via gradient-based classifier guidance. While these approaches support conditional generation, they rely on fine-grained diffusion processes and often struggle with interpretability and scalability across diverse biological contexts. 2.2 Conditional Transfer in Single-Cell Analysis Predicting gene expression under unseen conditions—such as novel drug perturbations or disease states—is a central goal in single-cell analysis 35 . scGen 23 addresses this using a latent shift strategy in an autoencoder framework, assuming linear transitions in the latent space. trVAE 22 applies Maximum Mean Discrepancy (MMD) for alignment and decodes data conditioned on state labels. These methods, however, are typically limited to a single type of condition. scDisInFact 40 extends to multi-factor settings by employing multiple encoders to disentangle condition-specific and condition-invariant components. Nonetheless, such designs require predefined factor separation and often lack scalability to combinatorial condition spaces. 2.3 Transformer Models in Single-Cell Learning Transformer-based models have recently gained traction in single-cell omics, mostly for representation learning. scGPT 9 , scBERT 38 , and Geneformer 32 tokenize gene expression using rank or discretized values, often sacrificing fine-grained quantitative information. GeneCompass 39 improves upon this by combining rank and absolute expression with regression loss. Some methods directly project expression values into continuous space, with additional strategies to handle zero inflation. TOSICA 5 uses gene networks to filter unreliable zeros, while scFoundation 12 introduces special tokens to mask them. CellPLM 33 models inter-cell relationships via a VAE-Transformer hybrid, treating entire cells as tokens. Despite this progress, most Transformer-based methods focus on encoding rather than generation, and there is little agreement on how best to represent gene expression in tokenized form. Incorporating biological structure (e.g., gene sets) and enabling conditional generation remain open challenges. 3 Methodology Figure 1: The overall architecture of the SAVE framework. The model consists of three main modules: (1) Gene Block Construction , which partitions gene embeddings into semantic blocks via Optimal Transport clustering; (2)a VAE utilizing Gene Block Attention for latent compression and reconstruction ; and (3) a Conditional Flow Matching Network , which maps prior noise x_{0} to generated latent distributions x_{1} by integrating conditions s and time steps t via Adaptive Layer Normalization (AdaLN). We present SAVE,

Read the complete text