Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training

Haoran Wang, Xuanyi Zhang, Shuangsang Fang, Longke Ran, Ziqing Deng, Yong Zhang, Yuxiang Li, Shaoshuai Li · 2026

Read Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training by Haoran Wang, Xuanyi Zhang, Shuangsang Fang, Longke Ran, Ziqing Deng, Yong Zhang, Yuxiang Li, Shaoshuai Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05648.txt.

Introduction

Haoran Wang and 7 co-authors present OKR-CELL, a 2026 foundation model that joins single-cell gene data with written cell descriptions. Single-cell RNA sequencing reveals how cells differ, but models trained on it often ignore rich individual profiles and suffer from noisy multi-modal data.

Two ideas drive the design. A retrieval-augmented workflow uses large language models to enrich each cell's text description with open-world knowledge. A robust alignment objective then weighs sample reliability, orders training from easier to harder, and uses momentum contrastive learning to resist noise. Pretrained on millions of cell-text pairs, the model reportedly led on six tasks, including clustering, cell-type annotation, and cell-text retrieval.

Catalog record

CREATORHaoran Wang, Xuanyi Zhang, Shuangsang Fang, Longke Ran, Ziqing Deng, Yong Zhang, Yuxiang Li, Shaoshuai Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05648

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Open World Knowledge Aided Single-Cell Foundation Model with Robust Cross-Modal Cell-Language Pre-training Haoran Wang, Xuanyi Zhang, Shuangsang Fang, Longke Ran, Ziqing Deng, Yong Zhang, Yuxiang Li, Shaoshuai Li Abstract Recent advancements in single-cell multi-omics, particularly RNA-seq, have provided profound insights into cellular heterogeneity and gene regulation. While pre-trained language model (PLM) paradigm based single-cell foundation models have shown promise, they remain constrained by insufficient integration of in-depth individual profiles and neglecting the influence of noise within multi-modal data. To address both issues, we propose an Open-world Language Knowledge-Aided Robust Single-Cell Foundation Model (OKR-CELL). It is built based on a cross-modal Cell-Language pre-training framework, which comprises two key innovations: (1) leveraging Large Language Models (LLMs) based workflow with retrieval-augmented generation (RAG) enriches cell textual descriptions using open-world knowledge; (2) devising a Cross-modal Robust Alignment (CRA) objective that incorporates sample reliability assessment, curriculum learning, and coupled momentum contrastive learning to strengthen the model’s resistance to noisy data. After pretraining on 32M cell-text pairs, OKR-CELL obtains cutting-edge results across 6 evaluation tasks. Beyond standard benchmarks such as cell clustering, cell-type annotation, batch-effect correction, and few-shot annotation, the model also demonstrates superior performance in broader multi-modal applications, including zero-shot cell-type annotation and bidirectional cell-text retrieval. keywords Single-cell Foundation Model, Cross-modal Pretraining, Large Language Model, Noise-robust Contrastive Learning 1 Introduction Recent progress in single-cell multi-omics technologies 7 ; 17 ; 2 ; 21 ; 29 , particularly high-resolution RNA-seq, have revolutionized our understanding of cellular heterogeneity and gene regulatory mechanisms, laying the foundation for critical applications such as drug response prediction 35 and disease mechanism dissection. Parallel to this, the success of Pre-trained Language Models (PLMs) 13 ; 39 ; 5 has spurred the development of single-cell foundation models 33 ; 53 ; 44 ; 15 ; 11 , which aim to distill generalizable biological insights from large-scale transcriptomic data. However, most of them rely solely on uni-modal information, failing to capture the complex, multi-faceted biological characteristics inherent to cells. Language serves as a carrier of human biological knowledge, enabling foundation models to decode the ”life language” contained in cells. While early attempts like BioTranslator 52 leveraged text-only pretraining, recent cross-modal Cell-Text Pretraining (CTP) methods 56 ; 4 have advanced by adopting cross-modal alignment frameworks 38 ; 26 ; 47 ; 27 ; 46 . For instance, LangCell 56 integrated cell-text cross-attention and contrastive learning to transfer textual knowledge, and scMULAN 4 constructed ”c-sentences” by fusing transcriptomics with meta-textual attributes. Despite these strides, current CTP approaches suffer from two critical and unresolved limitations: (1) Impoverished and constrained textual corpora : The original textual data primarily consists of isolated attribute descriptors (e.g. cell type labels) that lack contextual dependencies among them. For example, macrophages function as microglia in the brain and Kupffer cells in the liver nuances unconveyed by mere labels, resulting in superficial cellular state understanding compared to gene-level transcriptomic profiles. Furthermore, these specialized corpora are confined to the knowledge scope of their creators, failing to incorporate the vast, evolving open-world biological knowledge essential for comprehensive cellular interpretation. (2) Vulnerability to multi-modal noise : Real-world cell-text data is inherently noisy, yet traditional CTP methods (e.g., CLIP-based frameworks) ignore this problem. In the cellular modality, technical variability across sequencing platforms/protocols introduces gene dropout and batch effects 22 ; 32 ; in the textual modality, phenotypic descriptions are often subjective, incomplete, or inaccurate due to varying biological expertise 34 . Such noise-induced misalignments severely undermine the fidelity of cross-modal learning, restricting model’s robustness and generalization. To address these challenges, we propose an O pen-World Language K nowledge-Aided R obust Single- Cell Foundation Model ( OKR-CELL) , which integrates open-world biological knowledge and robust cross-modal alignment into a unified framework. Our core innovations are two-fold: First, at the data level, we augment textual corpora using Large Language Models (LLMs) enriched with retrieval-augmented generation (RAG). Leveraging LLMs’ open-world knowledge, we generate comprehensive, context-rich textual descriptions for each cell—going beyond static attributes. Meanwhile, to mitigate LLM hallucinations, we retrieve specialized bioinformatics literature databases ( e.g. 6 ) based on original metadata, ensuring the generated text is biologically reliable through RAG. Second, at the method level, we design a novel Cross-modal Robust Alignment (CRA) objective. Unlike conventional contrastive learning that blindly maximizes positive/negative sample discrimination, CRA explicitly accounts for noise from four aspects: (i) For positive sample pairs, we introduce the concept of positive sample pair reliability to emphasize highly reliable positive pairs and suppress low-reliability ones; (ii) For negative pairs, we enhance discriminability across samples of varying reliability by indirectly optimizing their matching probabilities, which prioritizes low-probability (high-confidence) negatives while suppressing the influence of high-probability (potentially false negative) ones. (iii) To alleviate early-stage reliability assessment limitations on negatives, a curriculum learning based strategy is presented to progressively increase the emphasis on negative sample pairs. (iv) We introduce negative sample queues to expand candidate negatives to promote cross-modal contrastive learning. OKR-CELL adopts a multi-task learning framework, which leverages scGPT as backbone to process RNA-seq data and Clinical-LongFormer as the textual encoder. On one hand, we learn intra-modal cellular information using scGPT’s masked gene modeling objective; On the other hand, we align cell and textual representations in a shared embedding space to convey complementary information across modalities. To train this model, we construct a SCxGEN-32M dataset, consisting of 32 million cell-text pairs crawled from CELLXGENE 36 platform, each with 9 key metadata items (cell type, tissue origin, etc.). Extensive experiments demonstrate that OKR-CELL achieves state-of-the-art (SoTA) performance across 6 tasks: In traditional tasks, we achieve superior performance in cell clustering, cell type annotation, and batch effect correction. Furthermore, we extend our evaluation to multi-modal cell understanding tasks. Our model also exhibits superior performance in zero-shot cell type annotation and bidirectional cell–text cross-modal retrieval, underscoring its robust generalization to previously unseen cell classes. Our main contributions can be summarized as follows: • We propose OKR-CELL, the first single-cell foundation model that integrates LLM-derived open-world knowledge to enable deep, individualized cellular understanding. • We devise a novel Cross-modal Robust Alignment (CRA) objective, which leverages complementary learning to improve the robustness of model to the noisy training data. Moreover, a Coupled Momentum Constrastive Learning and Curriculum Learning based strategy are incorporated into CRA loss to enhance cross-modal alignment. • The extensive experiments on multiple benchmark datasets validate the superiority of our model in both traditional biological tasks and multi-modal understanding, advancing its generalization to real-world noisy scenarios. 2 Results 2.1 Method Overview 2.1.1 Model Architecture In this section, we present a comprehensive description about the workflow of our OKR-CELL method, whose framework is illustrated in Figure 1 , OKR-CELL is composed of three trainable components: a cell encoder, a text encoder and a cross-modal projector connecting cellular and textual modalities. 1) For cell encoder, we employ utilize a transformer-based single-cell foundation model, scGPT 11 to handle the gene tokens and expression values that are specific to scRNA-seq data. 2) For the textual branch, a longformer architecture based clinical corpora enriched language model named Clinical-Longformer 28 is utilized, whose maximum input sequence token length is 4,096. It is pre-trained on approximately 2 million clinical notes extracted from the MIMIC-III dataset. 3) Cross-modal projectors. We employ a cell-to-text connector to map cellular representation into a joint space

Read the complete text