S2-UniSeg: Fast Universal Agglomerative Pooling for Scalable Segment Anything without Supervision
Huihui Xu, Jin Ye, Hongqiu Wang, Changkai Ji, Jiashi Lin, Ming Hu, Ziyan Huang, Ying Chen, Chenglong Ma, Tianbin Li, Lihao Liu, Junjun He, Lei Zhu · 2025
Read S2-UniSeg: Fast Universal Agglomerative Pooling for Scalable Segment Anything without Supervision by Huihui Xu, Jin Ye, Hongqiu Wang, Changkai Ji, Jiashi Lin, Ming Hu, Ziyan Huang, Ying Chen, Chenglong Ma, Tianbin Li, Lihao Liu, Junjun He, Lei Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06995.txt.
Introduction
Huihui Xu and eleven co-authors present a faster way to train image models that outline objects without human labels, in a 2025 study. Such models learn from draft masks that the machine makes itself. Past methods rebuilt those masks between training rounds, which was slow and broke the flow of learning. The team replaces that step with a pooling method that groups like pixels in parallel, in a few thousandths of a second per image.
On top of the fast masks, their system trains a student model against a slow-moving teacher copy. A new task teaches the student how local patches match the whole scene. The system beat the prior best tool on four test sets, and gains grew when training scaled to two million images. Code and trained models are shared in a public repo.
Catalog record
| CREATOR | Huihui Xu, Jin Ye, Hongqiu Wang, Changkai Ji, Jiashi Lin, Ming Hu, Ziyan Huang, Ying Chen, Chenglong Ma, Tianbin Li, Lihao Liu, Junjun He, Lei Zhu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06995
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
S2-UniSeg: Fast Universal Agglomerative Pooling for Scalable Segment Anything without Supervision Huihui Xu, Jin Ye, Hongqiu Wang, Changkai Ji, Jiashi Lin, Ming Hu, Ziyan Huang, Ying Chen, Chenglong Ma, Tianbin Li, Lihao Liu, Junjun He, Lei Zhu Abstract Recent self-supervised image segmentation models have achieved promising performance on semantic segmentation and class-agnostic instance segmentation. However, their pretraining schedule is multi-stage, requiring a time-consuming pseudo-masks generation process between each training epoch. This time-consuming offline process not only makes it difficult to scale with training dataset size , but also leads to sub-optimal solutions due to its discontinuous optimization routine. To solve these, we first present a novel pseudo-mask algorithm, Fast Universal Agglomerative Pooling (UniAP). Each layer of UniAP can identify groups of similar nodes in parallel , allowing to generate both semantic-level and instance-level and multi-granular pseudo-masks within tens of milliseconds for one image. Based on the fast UniAP, we propose the Scalable Self-Supervised Universal Segmentation (S2-UniSeg), which employs a student and a momentum teacher for continuous pretraining. A novel segmentation-oriented pretext task , Query-wise Self-Distillation (QuerySD), is proposed to pretrain S2-UniSeg to learn the local-to-global correspondences. Under the same setting, S2-UniSeg outperforms the SOTA UnSAM model, achieving notable improvements of AP+6.9 on COCO, AR+11.1 on UVO, PixelAcc+4.5 on COCOStuff-27, RQ+8.0 on Cityscapes. After scaling up to a larger 2M-image subset of SA-1B, S2-UniSeg further achieves performance gains on all four benchmarks. † † footnotetext: † Corresponding authors. Code — https://github.com/bio-mlhui/S2-UniSeg Figure 1: Previous paradigms vs. our paradigm. S2-UniSeg eliminates the need for time-consuming offline pseudo-mask generation, functioning as a single-stage pretraining framework with continuous optimization. Introduction Supervised image segmentation has made significant progress in recent years ( Xie et al. 2021 ; Cheng et al. 2022 ; Kirillov et al. 2023 ; Zou et al. 2024 ; Li et al. 2023 ; Ravi et al. 2024 ; Xu et al. 2024 ; Wang et al. 2025 ; Wang et al. 2024 ; Xu et al. 2025 ) . However, training such segmentation models typically relies on large-scale human annotations ( Wu et al. 2024 ; Tian, Ng, and Xu 2023 ; Ning et al. 2025 ) , which are both time-consuming and labor-intensive. For example, annotating a single image in the SA-1B dataset ( Kirillov et al. 2023 ) can take over 25 minutes of detailed effort. Moreover, human annotations are probably noisy, inconsistent, and influenced by subjective biases, leading to misalignment with fine-grained image details and object boundaries. These limitations can undermine the robustness and generalizability of segmentation models across diverse visual contexts. Therefore, we propose an efficient single-stage self-supervised segmentation model leveraging intrinsic image information to reduce reliance on expensive annotation. Method Time (seconds) \text{AP}^{\text{mask}} mIoU Universal Multi-granular Divide-and-Conquer ( Wang, Yang, and Darrell 2024 ) 5.27 7.6 - ✘ ✔ TokenCut ( Wang et al. 2023b ) 2.25 3.5 - ✘ ✘ MaskCut ( Wang et al. 2023a ) 4.72 6.7 - ✘ ✘ MaskCut+STEGO ( Niu et al. 2024 ) 4.86 6.7 26.3 ✔ ✘ Universal Agglomerative Pooling 0.045 6.2 21.8 ✔ ✔ Table 1: Quantitative efficiency and effectiveness comparison among different pseudo-masks generation algorithms. We run all methods on COCOval2017 with the same hardware settings, and evaluate their average processing time per image, zero-shot class-agnostic instance segmentation performance ( \text{AP}^{\text{mask}} ), unsupervised semantic segmentation performance (mIoU). All methods use the DINO pretrained ViT-base/8 features. As shown in the top of Figure 1 , existing self-supervised image segmentation methods typically adopt a two-stage pipeline ( Wang et al. 2023a ; Wang, Yang, and Darrell 2024 ; Niu et al. 2024 ; Arica et al. 2024 ) . In the first stage, based on pretrained self-supervised visual representations ( Caron et al. 2021 ; Grill et al. 2020 ; Chen and He 2021 ; Chen et al. 2020a ; He et al. 2020 ; Oquab et al. 2023 ) , graph partitioning algorithms ( Wang et al. 2023a ; Wang et al. 2023b ) are used to generate pseudo-masks offline for all unlabeled images. These pseudo-masks are then used to initialize training of the segmentation model. In the second stage, segmentation performance is improved via multi-round self-training , where pseudo-masks generated from the model checkpoint of the previous round are used to fine-tune the model at next round until convergence. As shown in Table 1 , although the pseudo-masks used in the first stage achieve a higher \text{AP}^{\text{mask}} , the generation process is time-consuming. For UnSAM ( Wang, Yang, and Darrell 2024 ) , average time for generating pseudo masks for a 512 \times 512 image takes 5.27 seconds, which is impractical to scale with large-scale datasets. Furthermore, as shown in the bottom of Figure 1 , the overall optimization is discontinuous, leading to unstable and sub-optimal convergence. To achieve scalable and continuous training, the primary challenge is to reduce the time for pseudo-mask generation. To this end, we propose the Fast Universal Agglomerative Pooling (UniAP) to efficiently generate both universal and multi-granular pseudo-masks of a single image within tens of milliseconds. Our key insight is that pixels belonging to the same instance or semantic category are spatially adjacent in form of a connected region. These groups of strongly connected pixels can be identified in parallel ( Pearce 2005 ) . As shown in Table 1 , UniAP achieves comparable segmentation performance while being approximately 100\times faster than previous methods, making it feasible for single-stage pretraining and scalable to large-scale dataset. Equipped with fast pseudo-masks generation by UniAP, we propose the Scalable Self-Supervised Universal Segmentation (S2-UniSeg) , which leverages the teacher-student framework trained with a novel segmentation-oriented pretext task Query-wise Self-Distillation (QuerySD) . Unlike Self-supervised representation models which utilize global pooling or the ViT ( Dosovitskiy 2020 ) [CLS] token, we condense each image into a set of universal object queries, and train each student query to predict the corresponding bipartite-matched teacher query. Our main contributions are: • We propose a novel and efficient pseudo-masks generation algorithm, Fast Universal Agglomerative Pooling (UniAP), which can generate both universal and multi-granular pseudo-masks within tens of milliseconds. UniAP makes it feasible for continuous optimization and scalable with large-scale dataset. • We propose the Scalable Self-Supervised Universal Segmentation (S2-UniSeg) framework. A novel segmentation-oriented pretext task, Query-wise Self-distillation (QuerySD) is proposed to train a student and a momentum teacher in self-supervised manner. • Extensive experiments on ImageNet, SA-1B, COCO, UVO, COCOStuff, and Cityscapes demonstrate that S2-UniSeg achieves substantial performance improvements on four universal segmentation tasks. Moreover, S2-UniSeg demonstrates strong scalability with increasing training data, yielding better performance as the dataset size grows. Related Work Self-supervised Representation Learning (SSL). Self-supervised representation learning aims to learn general-purpose ( Bengio, Courville, and Vincent 2013 ) features from large amounts of unlabeled data samples without manual annotation. A pretext task is often defined to train the model. According to types of pretext task, they can be classified into contrastive learning methods and masked image modeling methods. Contrastive learning based methods include pretext tasks based on negative samples ( Chen et al. 2020a ; He et al. 2020 ; Chen, Xie, and He 2021 ) , clustering ( Caron et al. 2020 ; Asano, Rupprecht, and Vedaldi 2019 ) , self-distillation ( Caron et al. 2021 ;