EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3

Chengxi Simon Zeng, Yuxuan Jiang, Aaron Zhang · 2025

Read EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3 by Chengxi Simon Zeng, Yuxuan Jiang, Aaron Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15833.txt.

Introduction

Chengxi Zeng, Yuxuan Jiang, and Aaron Zhang distill a large video concept segmentation model into efficient on device students. The teacher unifies a vision backbone, a detector, and a dense memory tracker, which is too heavy for devices. Their progressive hierarchical distillation transfers capability in three stages. Stage design matches the teacher architecture piece by piece.

First the image encoder aligns features with prompt in the loop training. Next a compact Perceiver module replaces dense temporal memory. Finally the full pipeline fine tunes on official concept segmentation data. Student variants use three lightweight backbones for different budgets. The family aims to preserve concept level performance while enabling local video understanding.

Catalog record

CREATORChengxi Simon Zeng, Yuxuan Jiang, Aaron Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EfficientSAM3: Progressive Hierarchical Distillation for Video Concept Segmentation from SAM1, 2, and 3 Chengxi Simon Zeng, Yuxuan Jiang, Aaron Zhang Abstract The Segment Anything Model 3 (SAM3) advances visual understanding with Promptable Concept Segmentation (PCS) across images and videos, but its unified architecture (shared vision backbone, DETR-style detector, dense-memory tracker) remains prohibitive for on-device use. We present EfficientSAM3, a family of efficient models built on Progressive Hierarchical Distillation (PHD) that transfers capability from SAM3 to lightweight students in three stages: (1) Encoder Distillation aligns image features via prompt-in-the-loop training on SA-1B; (2) Temporal Memory Distillation replaces dense memory with a compact Perceiver-based module trained on SA-V to compress and retrieve spatiotemporal features efficiently; and (3) End-to-End Fine-Tuning refines the full pipeline on the official SAM3 PCS data to preserve concept-level performance. PHD yields a spectrum of student variants using RepViT, TinyViT, and EfficientViT backbones, enabling on-device concept segmentation and tracking while maintaining high fidelity to teacher behavior. We benchmark on popular VOS datasets, and compare with varies of releated work, achieing strong performance-efficiency trade-offs. † † Coming Soon - This document details the experimental setup and training procedures for making SAM3 efficient. Implementation may evolve; results will be reported in a future revision. 1 Introduction Foundation models for visual segmentation have rapidly evolved, fundamentally changing how machines perceive and interact with the visual world. The Segment Anything Model (SAM) series Kirillov et al. 2023 ; Ravi et al. 2024 ; Authors 2025 has been at the forefront of this revolution. The initial SAM Kirillov et al. 2023 introduced the paradigm of promptable zero-shot image segmentation, demonstrating an unprecedented ability to segment any object based on simple geometric prompts. Its successor, SAM2 Ravi et al. 2024 , extended this capability to the temporal domain of video by incorporating a memory mechanism to track objects across frames. The latest iteration, SAM3 Authors 2025 , represents another paradigm shift, moving from generic object segmentation to Promptable Concept Segmentation (PCS). SAM3 can detect, segment, and track all instances of a specific semantic concept provided via phrases or image exemplars, achieving a new level of semantic understanding. This progression in capability has been accompanied by a corresponding increase in architectural complexity and computational cost. While SAM1’s 1 1 1 We explicitly refer to the original Segment Anything Model Kirillov et al. 2023 as SAM1 to distinguish it from its successors SAM2 and SAM3. primary bottleneck was its massive ViT-H image encoder Kirillov et al. 2023 , SAM2 introduced a second bottleneck in its memory attention module Zhou et al. 2025 . SAM3 further complicates this with a unified architecture featuring a powerful shared vision backbone, a DETR-based detector with a decoupled presence head, and a memory-based video tracker Authors 2025 . These components, while enabling state-of-the-art performance on the new SA-Co benchmark Authors 2025 , render SAM3 even more computationally demanding than its predecessors, confining its use to server-side environments and precluding deployment in real-time, on-device applications such as augmented reality Song et al. 2024 ; Liu et al. 2024 , robotics Wang et al. 2023 ; Gupta 2024 , medical image Zeng et al. 2025b ; Zeng et al. 2025a and interactive mobile tools Zhang et al. 2023b ; Zhang et al. 2023c . This computational barrier motivates our work. Following the successful symbiotic cycle of innovation where foundational model releases are followed by community-driven efficiency efforts Zhou et al. 2023 ; Xiong et al. 2024a ; Zhou et al. 2025 , we introduce Progressive Hierarchical Distillation (PHD) to create a family of efficient SAM3 variants for on-device deployment. PHD targets two primary bottlenecks in SAM3—(i) the shared vision backbone for feature extraction and (ii) the dense memory tracker for temporal consistency—and concludes with a unifying end-to-end refinement. To address these, PHD proceeds in three stages. First, inspired by prompt-in-the-loop distillation in EdgeSAM Zhou et al. 2023 , we distill the SAM3 encoder into efficient student backbones—RepViT Wang et al. 2024 , TinyViT Wu et al. 2022 , EfficientViT Cai et al. 2023 . Second, following EdgeTAM/EfficientTAM Zhou et al. 2025 ; Xiong et al. 2024b , we replace SAM3’s dense memory with a compact Perceiver Jaegle et al. 2021 that compresses temporal context into a small set of latent queries. Third, we jointly fine-tune the student encoder, Perceiver memory, and mask decoder end-to-end to preserve SAM3’s PCS behavior. Figure 1 : Overall architecture of EfficientSAM3: Three stages of Progressive Hierarchical Distillation (PHD) are shown—lightweight encoder distillation, compact Perceiver insertion and efficient memory bank replacement. Our main contributions are as follows: • We introduce EfficientSAM3, a three-stage distillation recipe (encoder \Rightarrow memory \Rightarrow end-to-end) that enables on-device PCS while preserving teacher behavior. • We build a model zoo of nine EfficientSAM3-PHD students spanning RepViT, TinyViT, and EfficientViT, offering flexible accuracy-latency trade-offs for deployment. • We compress temporal memory with a Perceiver -based memory module, yielding efficient long-term reasoning and reduced memory-attention cost. • We benchmark across SA-Co (PCS) , SA-V/VOS (COCO, LVOS, DAVIS17, MOSE, YTVOS19 and etc). 2 Related Work Foundation Models for Segmentation. The Segment Anything Model (SAM) series Kirillov et al. 2023 ; Ravi et al. 2024 ; Authors 2025 has established the paradigm of promptable segmentation. SAM1 Kirillov et al. 2023 introduced the core task for images using geometric prompts, powered by a massive ViT-H encoder trained on the extensive SA-1B dataset Kirillov et al. 2023 . SAM2 Ravi et al. 2024 extended this to videos by adding a memory mechanism for temporal tracking, defining the Promptable Visual Segmentation (PVS) task and introducing the SA-V dataset. Most recently, SAM3 Section 3.1 pivots to Promptable Concept Segmentation (PCS), where the model segments all instances of a semantic concept from noun phrases or exemplars, trained on the new SA-Co dataset. This shift towards semantic understanding, while powerful, increases architectural complexity, creating a pressing need for efficiency that our work directly addresses. Efficient Image Segmentation Models. The computational cost of SAM1 spurred a wave of research into efficient variants. These efforts can be broadly categorized into backbone replacement and knowledge distillation Sun et al. 2024 . Models like FastSAM Zhao et al. 2023 and EdgeSAM Zhou et al. 2023 replaced the ViT with efficient CNNs like YOLOv8 Jocher et al. 2023 and RepViT Wang et al. 2024 , respectively. Others, like MobileSAM Zhang et al. 2023a , used a lightweight TinyViT Wu et al. 2022 . Knowledge distillation proved crucial for retaining performance. While MobileSAM used a simple decoupled distillation, EdgeSAM Zhou et al. 2023 introduced a more effective ”prompt-in-the-loop” strategy that distills the entire interactive pipeline, better preserving the model’s behavior. EfficientSAM Xiong et al. 2024a proposed SAMI, a masked image pre-training method to distill features from the teacher’s intermediate layers. Our work builds on these principles, adapting the successful prompt-in-the-loop distillation of EdgeSAM to the more complex, concept-driven architecture of SAM3. Efficient Video Segmentation Models. Optimizing video models like SAM2 presents a dual bottleneck: the image encoder and the memory attention module. EdgeTAM Zhou et al. 2025 was the first to explicitly identify and solve the memory bottleneck. Its core contribution is the 2D Spatial Perceiver, which compresses dense memory features into a small set of latent queries before the costly attention

Read the complete text