PicoSAM3: Real-Time In-Sensor Region-of-Interest Segmentation
Pietro Bonazzi, Nicola Farronato, Stefan Zihlmann, Haotong Qin, Michele Magno · 2026
Read PicoSAM3: Real-Time In-Sensor Region-of-Interest Segmentation by Pietro Bonazzi, Nicola Farronato, Stefan Zihlmann, Haotong Qin, Michele Magno (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11917.txt.
Introduction
Bonazzi and co-authors introduce PicoSAM3, a tiny promptable segmentation model that runs inside the sensor itself. Real-time on-device segmentation matters for smart glasses and IoT gear, where latency and privacy bite. PicoSAM3 packs just over a million parameters, mixing a dense CNN with region-of-interest prompt encoding and attention. It learns by distillation from SAM2 and SAM3. On COCO it reaches 65.45% mIoU, beating prior edge models at like size. Quantized to INT8, it runs in real time on Sony's IMX500 within its memory limits.
Catalog record
| CREATOR | Pietro Bonazzi, Nicola Farronato, Stefan Zihlmann, Haotong Qin, Michele Magno |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36235 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11917
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PicoSAM3: Real-Time In-Sensor Region-of-Interest Segmentation Pietro Bonazzi, Nicola Farronato, Stefan Zihlmann, Haotong Qin, Michele Magno Abstract Real-time, on-device segmentation is critical for latency-sensitive and privacy-aware applications such as smart glasses and Internet-of-Things devices. We introduce PicoSAM3, a lightweight promptable visual segmentation model optimized for edge and in-sensor execution, including deployment on the Sony IMX500 vision sensor. PicoSAM3 has 1.3 M parameters and combines a dense CNN architecture with region of interest prompt encoding, Efficient Channel Attention, and knowledge distillation from SAM2 and SAM3. On COCO and LVIS, PicoSAM3 achieves 65.45% and 64.01% mIoU, respectively, outperforming existing SAM-based and edge-oriented baselines at similar or lower complexity. The INT8 quantized model preserves accuracy with negligible degradation while enabling real-time in-sensor inference at 11.82 ms latency on the IMX500, fully complying with its memory and operator constraints. Ablation studies show that distillation from large SAM models yields up to +14.5% mIoU improvement over supervised training and demonstrate that high-quality, spatially flexible promptable segmentation is feasible directly at the sensor level. I Introduction Recent advances in task-agnostic segmentation, spearheaded by Meta’s Segment Anything Model (SAM) [ 1 ] and its successors SAM 2 [ 2 ] and SAM 3 [ 3 ] , have established a new standard for prompt-based visual understanding. This evolution has notably addressed the latency limitations of the original architecture; SAM 2, in particular, adopts a streamlined hierarchical backbone to extend segmentation capabilities to the video domain with improved throughput. Yet, despite these algorithmic refinements, the fundamental memory footprint and computational density of the transformer architecture remain prohibitive for extreme edge-sensing paradigms [ 4 , 5 ] . This presents a critical barrier for intelligent sensor nodes and low-power vision systems, such as battery-operated smart glasses [ 6 ] , where the quadratic complexity of self-attention mechanisms continues to impede real-time execution. Moreover, for resource-constrained scenarios, moving beyond centralized cloud processing toward near-sensor intelligence requires a radical reduction in model parameters to maintain operational longevity without compromising inference performance [ 7 ] . To mitigate the communication bottlenecks inherent to cloud-centric AI, edge computing paradigms have emerged as a vital solution, enabling low-latency, privacy-preserving inference closer to the data source. By performing vision tasks locally, these systems circumvent the bandwidth constraints and jitter associated with offloading raw data [ 8 , 9 , 10 ] . However, recent progress in semiconductor manufacturing is pushing this trend even further, shifting the locus of computation from external microcontrollers directly into the sensing node itself. This paradigm, known as in-sensor computing, allows for real-time perception immediately at the point of capture, eliminating data transmission overhead entirely. This evolution is not limited to optical modalities; it represents a fundamental shift in sensor architecture across the spectrum. Driven by the synergy between multimodal smart sensors and innovative TinyML techniques, recent research has successfully embedded intelligence into inertial [ 11 , 12 ] , olfactory [ 13 , 14 ] , acoustic [ 15 ] , tactile [ 16 ] , and biomedical [ 17 ] systems. Pushing this integration limit further, a new generation of devices now features embedded AI cores or dedicated hardware accelerators directly within the discrete sensor package. For instance, STMicroelectronics has introduced a Machine Learning Core (MLC) capable of executing decision trees directly within the MEMS sensor circuitry, allowing for a drastic reduction in the system’s overall power budget [ 18 ] . In the visual domain, a representative example is Sony’s IMX500 [ 19 , 20 , 21 ] , which integrates a CMOS image sensor vertically stacked with a dedicated edge AI processor [ 22 ] . Despite these hardware advancements, in-sensor processing imposes constraints far more stringent than typical mobile or edge scenarios. For example, the IMX500 operates within a strictly bounded on-chip SRAM of less than 8 MB and supports a limited set of quantization-friendly operators. These physical limitations prevent existing promptable segmentation models such as TinySAM [ 23 ] , EdgeSAM [ 24 ] , MobileSAMv2 [ 25 ] , and LiteSAM [ 26 ] from being deployed. Such models typically exceed memory budgets, rely on unsupported operators (e.g., complex non-linear activations), or depend on the large-scale memory access patterns of transformer-based architectures. Furthermore, the lack of extensive cache hierarchies in these sensor-integrated AI cores makes the random memory access inherent to Self-Attention mechanisms a primary bottleneck for real-time throughput. Consequently, realizing promptable segmentation on such devices necessitates a shift from standard high-precision floating-point operations to ultra-low-bitwidth quantization [ 27 ] and hardware-aware architectural design [ 28 ] . Building on prior in-sensor segmentation work [ 5 ] , this paper introduces PicoSAM3, represented in Figure 2 , where a region of interest (ROI) prompted segmentation model explicitly designed for ultra-low-latency, in-sensor deployment. Unlike earlier approaches that relied on restrictive centered-point prompts, PicoSAM3 implements a spatially flexible ROI supervision strategy. This design encodes prompt information implicitly during training, thereby eliminating the computational overhead of inference-time prompt processing and enabling seamless integration with the hardware ROI capabilities of intelligent sensors like the Sony IMX500. To maximize representational power within strict edge constraints, PicoSAM3 replaces heavy transformer blocks with a dense CNN architecture augmented by a dilated bottleneck and lightweight Efficient Channel Attention (ECA) [ 29 ] . This approach ensures full compatibility with the limited operator set and 8MB memory budget of the IMX500 while maintaining high feature selectivity. Furthermore, we leverage a robust knowledge distillation pipeline transferring capabilities from the foundational SAM2 and SAM3 models to our compact student network. The main contributions and results of this work are summarized as follows: • We propose a hardware-aware architecture that operates in the 1.3-1.4 M parameter regime, employing INT8 quantization to achieve a 4x size reduction (to 1.31 MB) with negligible accuracy degradation. • Through novel distillation from SAM3, PicoSAM3 achieves state-of-the-art performance among edge-oriented models, reaching 65.45% mIoU on COCO and 64.01% on LVIS, surpassing existing baselines at comparable complexity. • We demonstrate real-time on-device execution on the Sony IMX500 vision sensor, achieving an end-to-end inference latency of 11.82 ms, confirming the feasibility of high-quality promptable segmentation at the extreme edge without cloud offloading. II Related Work The introduction of large foundation models such as SAM [ 1 ] has fundamentally altered the landscape of promptable visual segmentation. As detailed in the Zhang et al. survey [ 30 ] , these models utilize massive transformer-based encoders to achieve zero-shot generalization across diverse domains, ranging from medical imaging to remote sensing. However, the substantial computational cost and memory footprint of the ViT [ 31 ] backbone prohibit direct deployment on resource-constrained edge devices. Consequently, recent literature has prioritized methods to decouple segmentation capability from model size. A primary research direction addresses architectural efficiency by physically replacing heavy backbones with lightweight alternatives. Approaches such as FastSAM [ 32 ] and LiteSAM [ 26 ] employ compact convolutional or hybrid encoders, significantly reducing parameter count. Complementary to these structural changes, knowledge distillation methods [ 33 , 24 , 23 , 5 ] transfer the representational power of the large pretrained teacher to a compact student network, often bridging the accuracy gap through extensive dataset distillation. Distinct from these training-based approaches, training-free acceleration techniques [ 34 ] exploit the spatial