Benchmarking and optimizing organism wide single-cell RNA alignment methods
Juan Javier Díaz-Mejía, Elias Williams, Octavian Focsa · 2025
Read Benchmarking and optimizing organism wide single-cell RNA alignment methods by Juan Javier Díaz-Mejía, Elias Williams, Octavian Focsa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20730.txt.
Introduction
Juan Javier Díaz-Mejía and co-authors grade methods that line up single-cell RNA data sets in a 2025 paper. Each study measures gene activity cell by cell, but lab effects shift the numbers, so sets must be aligned before they can be merged. With many methods on offer and scores that mix many factors, picking the best for organism-wide work is hard.
The team built a single score that punishes leftover lab effects while rewarding correct cell-type calls across sets. They also built two test collections from dozens of human studies with cleaned labels. Using the score, they tuned rival methods and offered their own variant, which came out on top. The aligned space kept fine cell groupings intact.
Catalog record
| CREATOR | Juan Javier Díaz-Mejía, Elias Williams, Octavian Focsa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20730
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking and optimizing organism wide single-cell RNA alignment methods Juan Javier Díaz-Mejía, Elias Williams, Octavian Focsa Abstract Many methods have been proposed for removing batch effects and aligning single-cell RNA (scRNA) datasets. However, performance is typically evaluated based on multiple parameters and few datasets, creating challenges in assessing which method is best for aligning data at scale. Here, we introduce the K-Neighbors Intersection (KNI) score, a single score that both penalizes batch effects and measures accuracy at cross-dataset cell-type label prediction alongside carefully curated small (scMARK) and large (scREF) benchmarks comprising 11 and 46 human scRNA studies respectively, where we have standardized author labels. Using the KNI score, we evaluate and optimize approaches for cross-dataset single-cell RNA integration. We introduce Batch Adversarial single-cell Variational Inference (BA-scVI), as a new variant of scVI that uses adversarial training to penalize batch-effects in the encoder and decoder, and show this approach outperforms other methods. In the resulting aligned space, we find that the granularity of cell-type groupings is conserved, supporting the notion that whole-organism cell-type maps can be created by a single model without loss of information. 1 Introduction To build comprehensive organism-wide and inter-species maps of cell types and states, we must build integrated transcriptional atlases that combine studies and patient populations at scale Regev et al. 2017 . The now large number of published scRNA studies creates an opportunity for building a largely aligned scRNA atlas that would enable standardized reference-based analysis and cross-dataset comparison Lotfollahi et al. 2024 . However, the challenge in combining data from disparate scRNA studies remains due to batch effects Lotfollahi et al. 2024 ; Lähnemann et al. 2020 ; Gavish et al. 2023 . While studies have looked at the alignment of batches within datasets or between a handful of datasets focused on a specific tissue type, few have looked at model alignment across studies from different tissue types and instruments, as would be required for the generation of a reference atlas. Meanwhile, those studies that have used models to align datasets across tissue types and studies have used supervised models trained on cell-type labels, such as scBERT, Celltypist, SCimilarity, and SATURN Yang et al. 2022 ; Domínguez Conde et al. 2022 ; Heimberg et al. 2024 ; Rosen et al. 2024 . However, unsupervised alignment is often preferred for cell-type discovery and comparative analysis of cell-type variation Vasighizaker et al. 2022 . Thus, there remains a demand for unsupervised methods that can verifiably align scRNA data at scale Lotfollahi et al. 2024 . In this study, we propose a single-metric the K-Neighbors Intersection (KNI) score that combines the K-bet score Büttner et al. 2019 , with accuracy at author label prediction. We use this score to evaluate and optimize the ability of models to align a small (MNIST-like) and large benchmark (ImageNet-like) dataset that we present with standardized author labels. On this benchmark, we introduce a variant of the popular single-cell Variational Inference (scVI) model Lopez et al. 2018 , Batch Adversarially trained single-cell Variational Inference (BA-scVI), that outperforms other approaches, including newer foundational models. The resulting embedding space maintains the cell-type clusters identified in the studies that comprise the benchmark, supporting the notion that a single unsupervised model can be used for organism-wide scRNA integration. Moreover, the prediction of cell types outside of the benchmark correlates well with benchmark scores, indicating author labels are a valid approximation of ground truth. 2 Related Work 2.1 Metrics and benchmarks for assessment of scRNA alignment quality The most comprehensive assessment of metrics and methods for scRNA alignment performance to date is performed in Luecken et al. 2020 . Here, the authors benchmark 16 commonly used methods in 13 integration tasks. Performance on each task is assessed by a score that integrates five batch correction metrics, with nine metrics focused on the conservation of biological signal. Notably, this assessment includes the kBET score for batch effects Büttner et al. 2019 , and F1 accuracy metric at cell-type prediction. This study has proven extremely valuable in helping authors pick specific models for specific scRNA analysis use cases Luecken et al. 2020 . However, integrating these metrics after averaging of all data points creates a lower bar for success. Namely, a data point can improve the score obtained by being labeled correctly or by being well integrated with other datasets. We propose that top-performing alignment methods must both integrate data points with other studies and enable accurate prediction of cell type. This requires a metric that integrates batch-effect detection with accuracy for individual data points. 2.2 Supervised methods for cross-dataset cell-type prediction Recent studies have looked to use cross-dataset prediction of cell-type labels as a means of automated annotation. Notably, Fischer et al. describe a deep-learning architecture scTAB that performs well at cross-dataset cell-type prediction on datasets generated from 10X technologies Fischer et al. 2024 . Here, the authors leverage the CELLxGENE corpus and schema, with automated filtering of cell-type groups for those having more fine-grained cell-type labels, with over 5000 instances, in at least 30 donors. They benchmark existing approaches and determine that the scTAB architecture performs best at cell type prediction with donor versus study level holdout of cells Fischer et al. 2024 . Ergen et al. also recently described a consensus-based approach to cell-type annotation Ergen et al. 2024 . Here, the authors develop a tool that performs unsupervised / semi-supervised alignment of scRNA datasets using three common approaches, BBKN, Scanorama, scVI, and scANVI, prior to the application of eight supervised models and then selection of the consensus cell-type label from these methods. This meta-approach performs well at unsupervised cell-type classification on a lung-tissue atlas but highlights that no single method has yet emerged that solves the task of new cell-type annotation. Our work here uses cell-type labels to assess performance at unsupervised alignment versus using cell-type labels at train time to maximize prediction accuracy. We propose that the top-performing unsupervised methods are likely better for cell-type, subtype, and state discovery as they are less susceptible to over-fitting. We also hypothesize that a single top-performing model can be identified that supersedes other approaches versus building off of a consensus. 3 Results 3.1 The KNI score is a single-metric that evaluates scRNA alignment quality In scRNA dataset alignment, the task is to reduce gene space for individual datasets into a shared low-dimensional representation of cell types that are aligned between studies. It is important that this space captures underlying biological variation versus noise introduced by the experiment (batch-effects). Previously, the kBET score has been developed as an effective means of testing for the presence of batch effects Büttner et al. 2019 . Accuracy at cross-dataset cell-type prediction of author labels also represents a gold-standard metric for evaluating the preservation of biological signal Domínguez Conde et al. 2021 . We propose the K-Neighbors Intersection (KNI) score as a score that combines these metrics at the level of individual data points Figure 1 a. To calculate the KNI we consider the set C=\{c_{1},c_{2}...c_{n}\} of cells in a low-dimensional cell-type feature space where each cell c_{i} is defined by its coordinates x_{i} , batch identifier b_{i} , and cell-type identifier t_{i} . The distance function D between two cells is