Colony Grounded SAM2: Zero-shot detection and segmentation of bacterial colonies using foundation models
Daan Korporaal ∗1, Patrick de Kruijf 1, Ralph H.G.M. Litjens 1, Bas H.M. van der Velden 1 · 2026
Read Colony Grounded SAM2: Zero-shot detection and segmentation of bacterial colonies using foundation models by Daan Korporaal ∗1, Patrick de Kruijf 1, Ralph H.G.M. Litjens 1, Bas H.M. van der Velden 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13393.txt.
Introduction
Daan Korporaal and co-authors present Colony Grounded SAM2 for finding bacterial colonies on agar plates. Detection and classification matter in microbiology, but labeled datasets are scarce. Their zero-shot pipeline joins Grounding DINO and Segment Anything Model 2, fine-tuned for the microbiology domain. No further training is needed for new settings, which aids robustness when data change. Mean Average Precision reached 93.1 percent, with a strong Dice score at detection. The full pipeline and model weights are shared open access for annotation work.
Catalog record
| CREATOR | Daan Korporaal ∗1, Patrick de Kruijf 1, Ralph H.G.M. Litjens 1, Bas H.M. van der Velden 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 10899 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13393
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Colony Grounded SAM2: Zero-shot detection and segmentation of bacterial colonies using foundation models Daan Korporaal ∗1, Patrick de Kruijf 1, Ralph H.G.M. Litjens 1, Bas H.M. van der Velden 1 Abstract The detection and classification of bacterial colonies in images of agar-plates is important in microbiology, but is hindered by the lack of labeled datasets. Therefore, we propose Colony Grounded SAM2, a zero-shot inference pipeline to detect and segment bacterial colonies in multiple settings without any further training. By utilizing the pre-trained foundation models Grounding DINO and Segment Anything Model 2, fine-tuned to the microbiological domain, we developed a model that is robust to data changes. Results showed a mean Average Precision of 93.1% and a D i c e @ d e t e c t i o n Dice@detection score of 0.85, showing excellent detection and segmentation capabilities on out-of-distribution datasets. The entire pipeline with model weights are shared open access to aid with annotation- and classification purposes in microbiology. 1 Introduction Bacterial colonies grown on agar plates allow quantification and identification for medical diagnostics, risk assessment, or food safety. Manual quantification is a complex and time-consuming task, sparking the need for automation. Recent advances in bacterial colony detection using artificial intelligence (AI) have shown impressive results [ 10 , 12 , 11 ] . However, to perform well these AI models typically need task-specific manually labeled datasets, which is time- and resource-intensive. To alleviate this need for manually labeled datasets, we propose Colony Grounded SAM2, a general purpose zero-shot detection and segmentation pipeline for bacterial colony images. By adapting large pre-trained foundation models to the microbiological domain, we aim for a robust method to detect and segment bacterial colonies over multiple datasets without needing further training. This allows researchers and domain experts to utilize AI-automated colony detection on data that was not feasible before, supporting and enhancing human observer performance in such tasks. 2 Data We used three open access image datasets of colonies cultured on agar plates ( Table 1 ). The AGAR (Annotated Germs for Automated Recognition) dataset [ 5 ] consists of 5 types of pharmaceutical bacterial colonies on Trypticase Soy Agar plates, over multiple imaging settings. The ADBC (Annotated dataset for deep-learning-based bacterial colony detection) dataset [ 6 ] contains 24 bacteria species of veterinary importance on 4 agar types. Lastly, the Hemolysis segmentation dataset [ 9 ] contains annotated colony segmentation masks from blood agar plates, used to classify hemolysis producing bacterial colonies from urinary infections and throat swab screening. Table 1 : AGAR = Annotated Germs for Automated Recognition, ADBC = Annotated Dataset for deep-learning-based Bacterial Colony detection. Dataset name # photos # colonies Type of ground truth AGAR [ 5 ] 18 000 336 442 Bounding boxes ADBC [ 6 ] 369 56 865 Bounding boxes Hemolysis segmentation [ 9 ] 286 Unknown Segmentations 3 Methods We propose Colony Grounded SAM2 ( Figure 1 ), an AI pipeline for zero-shot inference that consists of sequential detection and segmentation of bacterial colonies. We used Grounded SAM2 [ 8 ] , a combination of Grounding DINO [ 3 ] and Segment Anything Model (SAM) 2 [ 7 ] , and fine-tuned the Grounding DINO to a bacterial colony database. Code and models are available at https://github.com/WFSRDataScience/ColonyGroundedSam2 . The following subsections further explain these models and our evaluation of the zero-shot AI pipeline. Figure 1 : The proposed Colony Grounded SAM2 pipeline for bacterial colonies 3.1 Detection The first task of our pipeline was to detect individual colonies in the images using Grounding DINO. Grounding DINO extends the large pre-trained vision transformer DINO [ 1 ] to an open-vocabulary detection model using grounded pre-training. The open-vocabulary and self-supervised pre-trained vision transformer give Grounding DINO a broad and robust understanding of object detection in any natural image, making Grounding DINO a good choice for zero-shot approaches. Microbiological agar plate images are, however, very different from Grounding DINO’s pre-training with natural images, making its out-of-the-box performance on such images subpar. Therefore, we decided to adapt Grounding DINO to this unseen image domain with brief domain-specific fine-tuning using the AGAR dataset [ 5 ] . To keep the inherent robustness and zero-shot capability of Grounding DINO, we chose to fine-tune for only a single epoch using the recommended setting [ 3 ] . We evaluated the fine-tuned model using the ADBC dataset [ 6 ] and compared to the five baseline models from Colony-YOLO [ 10 ] . Since the ADBC dataset includes many more types of microorganisms on different agar plates ( e.g. blood and chocolate blood agar), this dataset allows for a broad evaluation of the zero-shot capabilities of our adapted model. We evaluated our model on the mean Average Precision (mAP). Matches were determined using the Intersection-over-Union (IoU). Because some annotated boxes were found to be loosely or imprecisely labeled (see 2(d) ), we chose an IoU threshold of 0.2. This value was chosen empirically to ensure that no incorrect matches were introduced, while minimizing the penalty on our model for predicting precise bounding boxes around relatively small objects. 3.2 Segmentation The bounding boxes from our detection model were fed directly to Segment Anything Model 2 (SAM2), a large-scale pre-trained foundation model, designed to segment any object in an image [ 7 ] . SAM2 can take both the image and the found bounding boxes as a prompt as input, and return the instance segmentation mask of each found colony. We hypothesized that since we prompt with the bounding boxes from Grounding DINO, SAM2 is already instructed where to focus its attention and thus further domain adaptation was unnecessary. We evaluated the segmentations using the Hemolysis segmentation dataset [ 9 ] using the Dice score. Additionally, to assess the performance of the segmentation regardless of detection performance, we calculated the Dice@detection score, defined as the Dice score on detected areas by Grounding DINO. While the regular Dice score thus describes the overall performance of our pipeline, the Dice@detection score describes the SAM2 segmentation performance with the given bounding boxes. 4 Results 4.1 Detection Our Colony Grounded SAM2 is on par or slightly outperforms the five baseline methods from Colony-YOLO in detection on the ADBC dataset ( Table 2 ). Note that for these five baseline models, a positively predicted data point has to be both detected precisely and classified correctly, while our current implementation foregoes this classification step. Still, as Colony-YOLO tends to make little classification errors, the performance achieved by our zero-shot model shows commendable detection capabilities in comparison. Table 2 : Detection performances on the ADBC dataset Model mAP/% Faster R-CNN 76.5 YOLOv5n 82.6 YOLOv8n 86.7 YOLOv10n 87.4 Colony-YOLO 91.1 Colony Grounded SAM2 93.1* *As a set of binary classification tasks Qualitative examples show that the model performs well in unseen or difficult circumstances Figure 2 . Furthermore, some of the mistakes found by the evaluation are understandable in the context of zero-shot inference, such as detecting colonies that were not the focus of that plate ( 2(c) ) or very loose ground truths leading to wrongly counted negatives ( 2(d) ). Over the entire dataset, Colony Grounded