LightAVSeg: Lightweight Audio-Visual Segmentation
Qing Zhong, Guodong Ding, Lingqiao Liu, Zaiwen Feng, Lin Yuanbo Wu, Angela Yao · 2026
Read LightAVSeg: Lightweight Audio-Visual Segmentation by Qing Zhong, Guodong Ding, Lingqiao Liu, Zaiwen Feng, Lin Yuanbo Wu, Angela Yao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08805.txt.
Introduction
Qing Zhong and five coauthors make audio visual segmentation, the task of outlining whichever object in a video is making sound, light enough for modest hardware. Existing models lean on dense cross attention whose cost grows quadratically, and most slimming efforts shrink the backbone while ignoring that interaction bottleneck. LightAVSeg splits the interaction into semantic filtering and spatial grounding steps whose cost grows only linearly with picture size.
An extra alignment loss during training enforces agreement between sound and picture at no cost during use. With about a seventh of a leading rival's parameters, the model sets a new best among lightweight methods on a standard benchmark and runs efficiently on a mobile processor.
Catalog record
| CREATOR | Qing Zhong, Guodong Ding, Lingqiao Liu, Zaiwen Feng, Lin Yuanbo Wu, Angela Yao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08805
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LightAVSeg: Lightweight Audio-Visual Segmentation Qing Zhong, Guodong Ding, Lingqiao Liu, Zaiwen Feng, Lin Yuanbo Wu, Angela Yao Abstract Audio-Visual Segmentation (AVS) targets pixel level localization of sounding emitting objects in videos. However, existing models rely on dense cross-modal attention with quadratic computational cost, limiting their suitability for resource efficient deployment. Most efficiency oriented methods focus on backbone reduction and overlook the interaction module as the primary bottleneck. This paper proposes LightAVSeg, a lightweight framework that replaces heavy attention with a decoupled design for semantic filtering and spatial grounding, resulting in interaction costs that scale linearly with spatial resolution. Furthermore, we introduce an auxiliary alignment loss to enforce semantic consistency during training with zero inference overhead. Extensive experiments demonstrate that LightAVSeg achieves a new state-of-the-art among lightweight methods: with 20.5M parameters ( ∼ 1 / 7 \sim 1/7 of AVSegFormer), it reaches 50.4 mIoU on the MS3 benchmark and enables efficient inference on a mobile processor. 1 Introduction Figure 1 : Efficiency-Accuracy Trade-off. We evaluate all models on the MS3 benchmark, and inference speeds are measured on a Snapdragon 8 Elite mobile CPU. Our method LightAVSeg outperforms heavy SOTA methods in both speed and accuracy on mobile device. Audio-Visual Segmentation (AVS) ( 33 ) aims to delineate sound-emitting objects in video sequences at the pixel level. Compared to conventional sound source localization that produces coarse heatmaps ( 1 ; 2 ; 3 ; 5 ; 18 ; 20 ) , AVS requires accurate object boundaries, which demands effective integration of acoustic cues with visual semantics. State-of-the-art (SOTA) approaches ( 7 ; 14 ) excel by modeling cross-modal dependencies via Transformer-based fusion. However, these gains often come at the expense of efficiency. SOTA systems built upon heavy visual backbones like ResNet-50 ( 11 ) or Pyramid Vision Transformer (PVT) variants ( 24 ; 25 ) and attention-based interaction modules result in considerable computational cost. For instance, processing a video of 224\times 224 resolution with AVSegFormer ( 7 ) takes more than 1.2 seconds per frame on a high-end mobile processor such as Snapdragon 8 Elite, as illustrated in Fig. 1 . Such requirements have made deployment challenging on resource-constrained platforms such as mobile devices and VR headsets, limiting the potential for efficient applications like on-device video editing and augmented reality. Lightweight AVS aims to enable efficient inference on mobile devices. However, simply replacing backbones with mobile-friendly alternatives ( e.g. , MobileNetV2 ( 12 ) ) does not fully resolve the efficiency issue, because the cross-modal interaction module can still dominate latency and memory. Standard attention-style fusion requires dense token interactions, where cost grows quadratically with the number of tokens. Since N scales with the feature resolution ( N\propto H\times W ), the complexity of \mathcal{O}(N^{2}) becomes especially costly at higher resolutions. This motivates a lightweight interaction design that avoids dense pairwise token computations while preserving effective audio guidance. A key observation in AVS is that fine-grained localization boundaries are largely encoded in visual spatial features, while audio primarily provides global semantic cues to identify which object is sounding. This implies that computing dense pixel-to-pixel affinity matrices can be redundant, as global auditory semantics are sufficient to select relevant visual channels. Motivated by this, we propose LightAVSeg, a compact framework with a Reciprocal Audio-Visual Encoder and a Cross-Modal Fusion Decoder to decouple the interaction. The Reciprocal Audio-Visual Encoder focuses on semantic filtering (what) by iteratively refining a global audio state, whereas the Cross-Modal Fusion Decoder handles spatial grounding (where) by injecting these cues into the visual hierarchy. Crucially, this design replaces dense attention ( \mathcal{O}(N^{2}) ) with lightweight channel-wise modulations, achieving strict linear complexity ( \mathcal{O}(N) ). Furthermore, lightweight models constrained by reduced capacity are susceptible to learning spurious cross-modal correlations ( 23 ; 31 ; 30 ) . To mitigate this, we introduce an auxiliary alignment loss, which enforces an explicit, pixel-wise alignment between the global audio cue and valid visual regions. The alignment acts as a strong structural prior, explicitly guiding the decoder to focus on the sounding object during training. Crucially, this knowledge is distilled into the network weights, allowing the auxiliary branch to be discarded during inference with zero extra cost. Our evaluation on three common AVS benchmarks highlights that LightAVSeg achieves the best accuracy-efficiency balance (see Fig. 1 ). With a compact model size of 20.5M parameters ( \sim 1/7 of AVSegFormer-R50), it reaches 50.4 mIoU on MS3, comparable to heavier ResNet-50 based counterparts. On a Snapdragon 8 Elite mobile CPU, our model also achieves a latency of 163.4 ms, about 8 \times speedup over the AVSegFormer ( 7 ) . This provides practical latency for interactive on-device editing and helps bridge the gap between research models and practical mobile deployment. Our main contributions are summarized as follows: 1) We propose LightAVSeg, a mobile-friendly framework that decouples interaction into a Reciprocal Audio-Visual Encoder for semantic filtering and a Cross-Modal Fusion Decoder for spatial grounding. This design replaces quadratic attention with hierarchical channel-wise modulation, achieving linear complexity ( \mathcal{O}(N) ) while preserving effective guidance. 2) We introduce an auxiliary Multi-Scale Audio-Visual Alignment Loss ( \mathcal{L}_{msa} ) to mitigate semantic ambiguity in lightweight networks and guide the decoder to determine where to segment. 3) Our framework achieves state-of-the-art lightweight Audio-Visual Segmentation performance on three AVS benchmarks and enables efficient inference on mobile devices. 2 Related Work 2.1 Semantic Segmentation Semantic segmentation has evolved from convolution-based models such as FCN ( 16 ) and DeepLab ( 4 ) , which focus on local context via dilated convolution ( 26 ) to Transformer-based frameworks such as SegFormer ( 27 ) and CTVIS ( 29 ) that model global dependencies. Beyond the architectural advances, contrastive learning methods such as PiCo ( 27 ) enhance representation by enforcing pixel-wise discrimination across images, while prototype-based segmentation ( 27 ) interprets classifiers as learnable prototypes for better generalization. Recent works including LLMFormer ( 22 ) further extend segmentation to the open-vocabulary setting by leveraging large language model priors. Figure 2 : Overview of LightAVSeg. Following the visual and audio streams, we introduce the Reciprocal Audio-Visual Encoder to iteratively refine the global audio state using visual context, the Cross-Modal Fusion Decoder to inject these auditory cues back into the visual stream for segmentation, and the Multi-Scale Audio-Visual Alignment Loss ( \mathcal{L}_{msa} ) to enforce progressive cross-modal consistency. 2.2 Audio-Visual Segmentation Audio-visual segmentation (AVS) ( 33 ; 32 ) extends sound source localization to pixel-level understanding. While early CNN-based models ( 33 ) suffered from limited receptive fields, Transformer-based frameworks like AVSegFormer ( 7 ) established a new paradigm by leveraging cross-modal attention for global feature fusion. Subsequent works have further improved robustness ( 14 ; 10 ) or sequence modeling efficiency ( 9 ) . However, these methods often overlook the high computational cost of the interaction module itself. Current state-of-the-art approaches still rely on dense attention mechanisms with quadratic complexity, creating a significant bottleneck for deployment on resource-constrained edge devices. 2.3 Mobile Vision Transformers Mobile semantic segmentation focuses on achieving accurate dense prediction under strict efficiency and memory constraints on edge devices. Early lightweight CNNs such as BiSeNet ( 17 ) and Fast-SCNN ( 19 ) introduced dual-branch and shared-computation designs to balance accuracy and latency.