CellCLIP – Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning
Mingyu Lu, Ethan Weinberger, Chanwoo Kim, Su-In Lee · 2025
Read CellCLIP – Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning by Mingyu Lu, Ethan Weinberger, Chanwoo Kim, Su-In Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06290.txt.
Introduction
Mingyu Lu and co-authors present CellCLIP in a 2025 paper on reading how cells respond to treatments. Cell Painting, a microscopy method that stains cell parts and photographs them by the thousands, lets scientists screen drugs and gene knockouts at huge scale.
The hard part is linking each treatment to the shape it leaves behind, since cell images speak a different language from natural photos. CellCLIP learns one shared space where treatments, written as text, sit next to the cell pictures they caused. A new channel-coding scheme helps the model read the stained channels properly. The aligned space lets researchers ask which treatments push cells into similar states, a shortcut for finding drug effects and gene roles.
Catalog record
| CREATOR | Mingyu Lu, Ethan Weinberger, Chanwoo Kim, Su-In Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06290
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CellCLIP – Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning Mingyu Lu, Ethan Weinberger, Chanwoo Kim, Su-In Lee Abstract High-content screening (HCS) assays based on high-throughput microscopy techniques such as Cell Painting have enabled the interrogation of cells’ morphological responses to perturbations at an unprecedented scale. The collection of such data promises to facilitate a better understanding of the relationships between different perturbations and their effects on cellular state. Towards achieving this goal, recent advances in cross-modal contrastive learning could, in theory, be leveraged to learn a unified latent space that aligns perturbations with their corresponding morphological effects. However, the application of such methods to HCS data is not straightforward due to substantial differences in the semantics of Cell Painting images compared to natural images, and the difficulty of representing different classes of perturbations (e.g. small molecule vs CRISPR gene knockout) in a single latent space. In response to these challenges, here we introduce CellCLIP, a cross-modal contrastive learning framework for HCS data. CellCLIP leverages pre-trained image encoders coupled with a novel channel encoding scheme to better capture relationships between different microscopy channels in image embeddings, along with natural language encoders for representing perturbations. Our framework outperforms current open-source models, demonstrating the best performance in both cross-modal retrieval and biologically meaningful downstream tasks while also achieving significant reductions in computation time. Code for our reproducing our experiments is available at https://github.com/suinleelab/CellCLIP/tree/main . 1 Introduction A grand challenge in cellular biology is understanding the impacts of different perturbations, such as exposure to chemical compounds or gene knockouts, on cellular function. In pursuit of this goal, a number of high-content screening (HCS) techniques have been developed that combine image-based deep phenotyping with high-throughput perturbations [ 20 , 29 ] . For example, the Cell Painting protocol [ 4 ] stains cells with six fluorescent dyes highlighting distinct cellular components (e.g. Hoescht staining for nuclear DNA), which are then imaged in five channels via automated microscopy ( Figure 1 a). Cell Painting is now routinely combined with other biotechnological advances to measure cells’ response to thousands of chemical and/or genetic perturbations in parallel at low cost [ 39 , 35 ] . Despite the promise of this data, extracting meaningful representations of cells’ underlying states from their images presents formidable challenges. Traditional HCS data analyses use custom software tools to extract domain-expert-crafted morphological features (e.g. nucleus size) from each image, with the resulting collection of features referred to as a cell’s profile [ 6 , 8 ] . More recent works have found that self-supervised deep learning methods based on DINO [ 7 , 32 ] or masked autoencoder (MAE; [ 21 ] ) architectures can capture more subtle changes in morphology compared to human-designed features, resulting in profiles that better reflect known relationships between perturbations. In order to more explicitly capture the relationships between perturbations’ effects on cellular state, recent work has applied cross-modal contrastive learning (CL) techniques [ 34 ] to learn aligned representations of both perturbation labels and Cell Painting images. In particular, recent work has applied CL to this task by treating images and their corresponding perturbation labels as paired samples from different modalities [ 16 , 37 ] . Post-training, such methods can be applied to systematically identify perturbations with similar effects based on relative proximity in the perturbation encoder’s representation space. Moreover, novel perturbations can be input to the perturbation encoder to predict their morphological effects based on distance to previously seen perturbations. Despite their promise, previous CL methods for Cell Painting come with significant drawbacks. First, unlike natural images, where the standard RGB channels all capture semantically related phenomena, microscopy images are comprised of a larger number of channels each capturing semantically independent information. Second, Cell Painting image-perturbation pairs exhibit a “many-to-one” structure where multiple images are associated with the same perturbation label. Standard CL, however, treats all image pairs from different instances as negatives, even if they share the same label, leading to a substantial number of false negatives where perceptually similar images are incorrectly pushed apart in the embedding space [ 5 , 43 ] . Furthermore, not all Cell Painting images faithfully represent the intended perturbation effect, as some perturbations may be unsuccessful due to technical issues (e.g., poor guide efficiency with CRISPR-mediated gene knockouts), effectively resulting in false positive pairs. Yet, existing CL methods for HCS data fail to account for these properties [ 37 ] or are not openly available [ 16 ] . Finally, previous works [ 16 , 37 ] have exclusively focused on chemical perturbations, relying on graph representation learning techniques applied to chemical structures to obtain perturbation representations. Thus, these methods cannot be applied to other classes of perturbations (e.g. CRISPR gene knockouts), and it is unclear how to represent perturbations from different classes within a unified input space for CL. To address these issues, here we propose CellCLIP ( Figure 1 b), a CL framework designed for the unique challenges in Cell Painting perturbation data. Our framework employs off-the-shelf pretrained vision models alongside a novel channel-aware encoding scheme ( Figure 1 c) to facilitate reasoning across the different sources of information captured in Cell Painting channels; in addition, our image encoding architecture incorporates techniques from the multiple-instance-learning literature [ 24 ] to account for the many-to-one nature of Cell Painting perturbation-image pairs. To represent images’ corresponding perturbations, CellCLIP employs natural language encoders, enabling application to multiple classes of perturbations with a single model. We benchmarked CellCLIP on profile-perturbation retrieval for unseen compounds [ 4 ] and its ability to recover known biological relationships between perturbations [ 10 , 27 ] . We find that CellCLIP achieves strong performance compared to the previous state-of-the-art while taking a fraction of the time to train. Our method also shows promising results in cross-perturbation class matching, thereby providing a scalable and effective solution for biological discovery. Figure 1: Overview of CellCLIP. ( a ) Cells’ responses to a perturbation are measured by the Cell Painting assay, which captures a set of distinct cellular components in five imaging channels. ( b ) CellCLIP aligns perturbation embeddings generated from natural language descriptions with embeddings of corresponding images produced by a new CrossChannelFormer architecture designed to account for the idiosyncracies of Cell Painting. ( c ) For a perturbation i , CrossChannelFormer takes as input images of all cells receiving perturbation i and pools them into a single embedding p_{i} that takes into account the relationship between information contained in the different Cell Painting channels. 2 Background Representation Learning for Cell Painting Self-supervised learning (SSL) deep learning techniques have been successfully applied to experimental microscopy data, with recent studies demonstrating their ability to capture intricate details of cellular morphology [ 39 , 13 ] better than hand-crafted features as implemented in standard software packages like CellProfiler [ 8 ] . Initial applications of SSL methods to Cell Painting relied on architectures designed for natural RGB images, where information across channels exhibits strong correlations [ 2 ] . On the other hand, Cell Painting channels each capture distinct biological structures (e.g. actin via phalloidin, mitochondria via MitoProbe, etc.). To account for this, recent works [ 2 , 26 ] have proposed so-called