CyclePose - Leveraging Cycle-Consistency for Annotation-Free Nuclei Segmentation in Fluorescence Microscopy
Jonas Utz, Stefan Vocht, Anne Tjorven Büssen, Dennis Possart, Fabian Wagner, Mareike Thies, Mingxuan Gu, Stefan Uderhardt, Katharina Breininger · 2025
Read CyclePose - Leveraging Cycle-Consistency for Annotation-Free Nuclei Segmentation in Fluorescence Microscopy by Jonas Utz, Stefan Vocht, Anne Tjorven Büssen, Dennis Possart, Fabian Wagner, Mareike Thies, Mingxuan Gu, Stefan Uderhardt, Katharina Breininger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11266.txt.
Introduction
Jonas Utz and eight co-authors teach a computer to outline cell nuclei without any hand labels in a 2025 paper. Labeling microscope images is slow expert work, and top outlining tools starve without it. Making fake training images first, then training the tool, doubles the cost.
Their system, CyclePose, fuses both jobs. It uses a back-and-forth image translator that swaps photos and outline masks, and slips the outlining model inside the loop. The round trip itself supervises learning: a good outline must survive translation and return. On two public image sets, the label-free system beat other weak or unsupervised methods. The authors post the code openly for labs to use.
Catalog record
| CREATOR | Jonas Utz, Stefan Vocht, Anne Tjorven Büssen, Dennis Possart, Fabian Wagner, Mareike Thies, Mingxuan Gu, Stefan Uderhardt, Katharina Breininger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11266
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CyclePose - Leveraging Cycle-Consistency for Annotation-Free Nuclei Segmentation in Fluorescence Microscopy Jonas Utz, Stefan Vocht, Anne Tjorven Büssen, Dennis Possart, Fabian Wagner, Mareike Thies, Mingxuan Gu, Stefan Uderhardt, Katharina Breininger Abstract In recent years, numerous neural network architectures specifically designed for the instance segmentation of nuclei in microscopic images have been released. These models embed nuclei-specific priors to outperform generic architectures like U-Nets; however, they require large annotated datasets, which are often not available. Generative models (GANs, diffusion models) have been used to compensate for this by synthesizing training data. These two-stage approaches are computationally expensive, as first a generative model and then a segmentation model has to be trained. We propose CyclePose, a hybrid framework integrating synthetic data generation and segmentation training. CyclePose builds on a CycleGAN architecture, which allows unpaired translation between microscopy images and segmentation masks. We embed a segmentation model into CycleGAN and leverage a cycle consistency loss for self-supervision. Without annotated data, CyclePose outperforms other weakly or unsupervised methods on two public datasets. Code is available at https://github.com/jonasutz/CyclePose 1 Introduction Nuclei segmentation is essential in quantitative cell biology and digital pathology, enabling downstream tasks such as cell counting, tracking, and morphological analysis [ 2 ] . As a key structural component, the nucleus is often used to identify individual cells, especially when cell boundaries are unclear. Accurate nuclei segmentation remains challenging owing to variations in imaging conditions, including signal-to-noise ratio, staining intensity, and microscope settings. Densely packed nuclei often have touching or overlapping boundaries, which makes instance separation difficult. While early approaches relied on intensity thresholding and linear filtering [ 14 ] , deep learning now dominates because of its ability to extract complex features and contextual information. Several specialized architectures have been developed for cell and nuclei segmentation. Notable models include StarDist [ 17 ] and Cellpose [ 18 ] , which incorporate domain-specific priors, such as shape and distribution of nuclei, to outperform generic instance segmentation methods such as Mask R-CNN [ 9 ] and U-Net [ 16 ] . However, these models rely on supervised learning and require large, annotated datasets. Nuclei annotation is particularly labor-intensive and requires expert knowledge and training to delineate individual instances consistently. This creates a bottleneck for training deep learning models at scale, necessitating methods that require little or no annotated data. A promising direction is the use of generative models such as CycleGAN to synthesize realistic microscopy images from masks. Conventionally, synthetic data generated through such models can be used to train a segmentation network; however, this two-step approach is computationally expensive and may introduce a domain gap between synthetic and real images. In this study, we tackle these limitations and propose an annotation-free approach that directly integrates a Cellpose-based segmentation model into a CycleGAN-inspired framework. By leveraging the cycle consistency loss, we train the segmentation network without requiring explicitly annotated training data, making our approach effectively unsupervised. This hybrid architecture offers reduced parameter count, faster training times, and allows the segmentation model to interact directly with real microscopy images during training. By avoiding separately trained synthetic data models, our approach elegantly streamlines nuclei segmentation and improves the adaptation to real-world imaging conditions. 2 Related Work Deep learning-based models have largely replaced traditional segmentation methods (e.g., thresholding, linear filtering, and watershed transform) due to their superior performance in cell and nuclei segmentation. Several algorithms have been developed specifically for this task. StarDist [ 17 ] models objects as star-convex polygons, leveraging prior knowledge that nuclei are typically blob-like to predict distances from a central point and define nuclear boundaries. This representation mitigates issues such as cell merging in per-pixel segmentation and inaccuracies in bounding-box-based approaches. Cellpose [ 18 ] predicts a flow field for each image pixel, guiding pixels toward their respective nuclei centers. Pixels that converge to the same point are grouped into an instance. This approach improves segmentation in nuclei-dense regions, where per-pixel classification or bounding-box-based methods often fail. However, both StarDist and Cellpose rely on annotated training data which limits their utility since manual annotation is a labor-intensive and time-consuming process. To circumvent this, various unsupervised nuclei segmentation approaches have been proposed. Many unsupervised nuclei segmentation methods rely on a two-step process where, first, synthetic image-mask pairs are created, and second, these are used to train a segmentation model. A widely used strategy for generating synthetic masks involves deforming ellipses or leveraging masks from a different domain [ 3 ] , as demonstrated in prior studies [ 5 , 19 , 10 , 7 , 21 ] . Generative adversarial networks (GANs), such as CycleGAN, can then be employed to synthesize corresponding images, creating a paired dataset for training [ 7 , 19 , 3 ] . Alternatively, nuclei textures can be synthesized using Perlin noise [ 1 ] , or synthetic images can be obtained through simulation-based methods such as CytoPacq [ 20 ] . Although unsupervised methods eliminate the need for annotations, semi-supervised approaches can reduce annotation effort while still benefiting from annotated data. DenoiSeg [ 4 ] requires only a small set of annotated images by integrating denoising and segmentation into a single network. Denoising simplifies the segmentation task, and by combining both tasks within a single model, a co-learning effect emerges, where the simultaneous learning of both tasks mutually reinforces performance. Unlike prior unsupervised approaches that rely on synthetic image-mask pairs for training, our method directly integrates segmentation within a CycleGAN-inspired framework, eliminating the need for separate synthetic data generation. This reduces computational overhead and minimizes the domain gap, enabling more robust adaptation to real microscopy images. 3 Methods Figure 1: Overview of the cycles in CyclePose: Synthetic masks from rotated ellipses undergo elastic deformation to create training masks. Adding Perlin noise-based texture, Gaussian blur, and Poisson noise produces Perlin images, which serve as auxiliary inputs for Cellpose alongside fake and real images. Discriminators are omitted for brevity. During inference only S is used. Our proposed method, CyclePose , is an unsupervised nuclei segmentation framework that integrates the gradient flow representation of Cellpose within a CycleGAN architecture. The method relies on two key assumptions: (1) synthetic segmentation masks for nuclei images can be easily generated using deformed ellipses, and (2) the cycle consistency loss of a CycleGAN can serve as an effective proxy for supervision in the absence of paired data. CyclePose consists of four primary network components: an image generator G , a segmentation model S (based on Cellpose), and two domain-specific discriminators D_{img},D_{seg} . G translates synthetic segmentation masks into microscopy images, while the S takes microscopy images and predicts the horizontal and vertical gradients of the Cellpose flows and a probability map for nuclei segmentation. Synthetic masks for G can be obtained by applying random elastic deformations to binary masks of ellipses, while S is trained on real microscopy images. Following the CycleGAN paradigm, the network outputs are passed back through the counterpart networks, forming a cycle (Fig. 1 ). This resembles a self-supervised training of S on generated microscopy images using the synthetic masks as ground truth. Hence, our cycle loss \mathcal{L}_{cyc} is a weighted sum of an L2 loss for the flows and a cross-entropy loss for the probabilities as described in [ 18