Segment Anything for Cell Tracking
Zhu Chen(✉), Mert Edgü, Er Jin, Johannes Stegmaier · 2025
Read Segment Anything for Cell Tracking by Zhu Chen(✉), Mert Edgü, Er Jin, Johannes Stegmaier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-09943.txt.
Introduction
Zhu Chen and 3 co-authors track living cells through microscope movies without any hand-labelled training data in a 2025 biomedical imaging paper. Following cells and spotting divisions is essential work, yet footage fights back with low contrast, fuzzy borders, dense crowds, and cells that all look alike. Labelled training sets are expensive, and models raised on one laboratory's data often stumble on another's.
Their answer plugs Segment Anything 2, a general segmentation foundation model, straight into the tracking pipeline as a zero-shot tool. Nothing is fine-tuned, so no single data set's habits are inherited. Accuracy stayed competitive on both flat two-dimensional sequences and large three-dimensional time-lapse volumes, suggesting foundation models can carry specialised science further than expected.
Catalog record
| CREATOR | Zhu Chen(✉), Mert Edgü, Er Jin, Johannes Stegmaier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-09943
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Segment Anything for Cell Tracking Zhu Chen(✉), Mert Edgü, Er Jin, Johannes Stegmaier Abstract Tracking cells and detecting mitotic events in time-lapse microscopy image sequences is a crucial task in biomedical research. However, it remains highly challenging due to dividing objects, low signal-to-noise ratios, indistinct boundaries, dense clusters, and the visually similar appearance of individual cells. Existing deep learning-based methods rely on manually labeled datasets for training, which is both costly and time-consuming. Moreover, their generalizability to unseen datasets remains limited due to the vast diversity of microscopy data. To overcome these limitations, we propose a zero-shot cell tracking framework by integrating Segment Anything 2 (SAM2) , a large foundation model designed for general image and video segmentation, into the tracking pipeline. As a fully-unsupervised approach, our method does not depend on or inherit biases from any specific training dataset, allowing it to generalize across diverse microscopy datasets without fine-tuning. Our approach achieves competitive accuracy in both 2D and large-scale 3D time-lapse microscopy videos while eliminating the need for dataset-specific adaptation. The source code is publicly available at https://github.com/zhuchen96/sam4celltracking . 1 Introduction Cell tracking plays a key role in understanding tissue formation, disease progression, embryonic development, and various biomedical applications [ 6 , 21 ] . The task includes tracking the movement and morphological changes of individual cells, identifying parent-daughter cell pairs during mitosis, detecting cell interactions, and reconstructing full lineage information [ 5 ] . Most current methods follow the tracking-by-detection paradigm, where objects are first detected in individual frames and then linked over time to form trajectories. The linking of segmented masks is typically performed as a post-processing step, which can be solved by simple approaches like nearest-neighbor matching. Alternatively, it can be formulated as a global optimization problem and solved by integer linear programming (ILP) [ 14 , 18 ] or the Viterbi algorithm [ 17 ] . In this case, the performance of tracking heavily depends on the quality of the segmentation results [ 3 ] . Recently, deep learning-based methods have significantly improved both cell segmentation [ 27 , 30 , 12 ] and tracking [ 2 , 7 , 23 , 4 , 8 , 9 ] . In addition, by fine-tuning foundation models like Segment Anything (SAM) [ 11 ] and SAM2 [ 25 ] on diverse microscopy datasets, they demonstrate strong performance in 2D microscopy image segmentation [ 10 , 24 ] . MicroSAM [ 1 ] extends SAM for volumetric segmentation and tracking in microscopy data, but it relies on projecting 2D segmentation masks onto adjacent slices and time frames, which limits accuracy and structural analysis. Additionally, [ 31 ] and [ 15 ] treat different slices of 3D image data as a video sequence, allowing SAM2 to be applied for volumetric segmentation. Among these methods, training with datasets spanning multiple imaging modalities and cell types has been shown to enhance generalizability. However, microscopy image analysis faces a major challenge due to the vast diversity of cell types and imaging techniques. Methods such as phase-contrast, fluorescence, and bright-field microscopy each have unique characteristics [ 28 ] . As a result, existing methods often fail to maintain accuracy when applied to a completely different type of data. To address the challenge, we propose a novel zero-shot 2D and 3D cell tracking pipeline by integrating the pretrained SAM2 model. As a fully-unsupervised approach, our method avoids dataset-specific biases and removes the requirement for manual segmentation and tracking annotations. For 2D and small 3D datasets, we employ a linking-only approach, where we construct a complete lineage graph from a pre-segmented set of masks and correct any failed detections. For large-scale 3D datasets, where detection and segmentation are significantly more challenging, we simultaneously address tracking and segmentation by incorporating SAM-Med3D [ 29 ] , a medical foundation model trained without microscopy images. By fine-tuning it on rough segmentation masks generated by simple methods such as watershed, we enable the model to produce accurate 3D segmentation results during the tracking process. Compared to global optimization-based approaches, our tracking pipeline allows for the targeted tracking of specific subsets of cells, reducing the influence of blurring and noisy areas in large-scale images. Our key contributions are as follows: 1. With appropriate prompt selection, the pre-trained SAM2 model can be applied to segment and link cells based on previous mask information in microscopy image sequences without requiring annotated data. 2. By integrating SAM-Med3D, our pipeline achieves volumetric segmentation and tracking performance comparable to fully-supervised algorithms. 2 Method 2.1 Cell Linking in 2D and 3D Time-lapse Microscopy Images Figure 1: Overview of the cell linking method. Patches are cropped from consecutive images based on the mask center at time point t . Bounding box and point prompts are then generated from the mask at time t . SAM2 predicts the segmentation masks of the same cell in both time frames, and the masks are linked to form tracklets. The depicted example is the Fluo-N2DL-HeLa dataset from Cell Tracking Challenge ( CTC ) [ 19 ] . Cell segmentation masks are typically generated independently for each time frame and must be linked to reconstruct full lineage trees, incorporating cell migration and mitotic information. For a sequence of T raw images I_{1},I_{2},\dots,I_{T}\in\mathbb{R}^{w\times h} , each image I_{t} has an associated segmentation mask set M_{t} , which can be generated using any segmentation approach. A cell lineage L_{k} is defined as a temporally continuous sequence of segments: L_{k}=\{m_{t}^{k,s}\mid t_{1}\leq t\leq t_{2},s\in M_{t}\} , where each segmentation mask m_{t}^{k,s} of tracklet k corresponds to a specific segment s at time t . Temporal consistency ensures that each time point within [t_{1},t_{2}] contains exactly one mask per lineage L_{k} . We initialize tracking at the final time frame T , where a user interface allows refinement of detections (Suppl. Video Part 1), yielding an initial mask set \tilde{M}_{T} . The algorithm then propagates masks backward in time, linking across frames. For each mask m_{t}^{k,s} , the center c_{t}^{k,s}=(x_{t}^{k,s},y_{t}^{k,s}) is computed, and a square patch p_{t}^{k,s} of side length d is extracted: p_{t}^{k,s}=I_{t}\left[x_{t}^{k,s}-\frac{d}{2}:x_{t}^{k,s}+\frac{d}{2},\ y_{t}^{k,s}-\frac{d}{2}:y_{t}^{k,s}+\frac{d}{2}\right]. (1) The corresponding patch p_{t-1}^{k,s} is extracted from I_{t-1} at the same coordinates, as depicted in Fig. 1 . The pair \{p_{t}^{k,s},p_{t-1}^{k,s}\} is provided to SAM2 as a short video sequence. Using the known mask m_{t}^{k,s} , we generate the following prompts: 1. Bounding box bb_{t}^{k,s} of m_{t}^{k,s} . 2. Positive points from m_{t}^{k,s} and negative points from the background. By taking the input image patch sequence \{p_{t}^{k,s},p_{t-1}^{k,s}\} and applying the prompts on p_{t}^{k,s} , we apply the mask decoder to predict the potential mask for frame t-1 . The potential mask is then compared to the pre-segmented masks in M_{t-1} . If a mask sufficiently overlaps, it is labeled as m_{t-1}^{k,s} and linked to L_{k} . If the potential mask is too small or touches the image boundary, tracking is terminated; otherwise, if the predicted mask is valid but missing in M_{t-1} , it is added to L_{k} and M_{t-1} as a recovered detection. Mitosis is detected when two masks in frame t link to the same mask in t-1 . If multiple masks link to one, only the two closest centers are retained. New tracklets are also initialized in intermediate