RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation

Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou · 2026

Read RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation by Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24295.txt.

Introduction

Zhu and co-authors present RS-SSM, a state space model that recovers details lost in video segmentation. State space models compress video state at fixed size, which keeps common meaning but forgets fine detail. Video semantic segmentation needs pixel-level detail that stays stable over time. Their model adds two parts that refine what the state forgot. One aligns the distribution of specific detail. The other inverts the forgetting gate to restore it. On four benchmarks, RS-SSM reaches state-of-the-art scores while staying efficient, they report.

Catalog record

CREATORKai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32273
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24295

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan Zhou Abstract Recently, state space models have demonstrated efficient video segmentation through linear-complexity state space compression. However, Video Semantic Segmentation (VSS) requires pixel-level spatiotemporal modeling capabilities to maintain temporal consistency in segmentation of semantic objects. While state space models can preserve common semantic information during state space compression, the fixed-size state space inevitably forgets specific information, which limits the models’ capability for pixel-level segmentation. To tackle the above issue, we proposed a R efining S pecifics S tate S pace M odel approach ( RS-SSM ) for video semantic segmentation, which performs complementary refining of forgotten spatiotemporal specifics. Specifically, a Channel-wise Amplitude Perceptron (CwAP) is designed to extract and align the distribution characteristics of specific information in the state space. Besides, a Forgetting Gate Information Refiner (FGIR) is proposed to adaptively invert and refine the forgetting gate matrix in the state space model based on the specific information distribution. Consequently, our RS-SSM leverages the inverted forgetting gate to complementarily refine the specific information forgotten during state space compression, thereby enhancing the model’s capability for spatiotemporal pixel-level segmentation. Extensive experiments on four VSS benchmarks demonstrate that our RS-SSM achieves state-of-the-art performance while maintaining high computational efficiency. The code is available at https://github.com/zhoujiahuan1991/CVPR2026-RS-SSM . 1 Introduction Figure 1 : Existing SSM-based VSS methods lose spatiotemporal specifics when perform state space compression, limiting the model’s segmentation accuracy. In contrast, our proposed RS-SSM method guides the model to focus on forgotten spatiotemporal specifics, thereby improving the pixel-level semantic segmentation performance. Video Semantic Segmentation (VSS) is a fundamental task in the field of computer vision, which aims to assign semantic labels to each pixel in videos and plays a crucial role in understanding the temporal evolution of scenes 50 ; 51 ; 10 ; 20 ; 21 . With the advancement of deep learning 67 ; 62 ; 65 ; 74 ; 63 ; 68 ; 64 ; 30 ; 8 ; 66 , VSS has demonstrated extensive practical application potential in areas such as autonomous driving 17 , video surveillance 43 , and video editing 70 . In complex long videos, objects may undergo significant displacement, deformation, and occlusion, requiring models to maintain temporal consistency in edge segmentation of semantic objects 5 ; 44 , which poses critical challenges to their pixel-level spatiotemporal modeling capabilities. In the field of VSS, early works utilized optical flow to model pixel-level motion information between adjacent frames, propagating and fusing features from previous frames to aggregate temporal information for segmentation of semantic objects 6 ; 11 ; 12 ; 45 ; 38 . However, such methods are not only computationally expensive but also suffer from inaccurate and noisy optical flow estimation in long videos with complex object motion, leading to limited video segmentation accuracy. Recent research has focused on using Transformer-based models to aggregate spatiotemporal features 23 ; 28 ; 53 ; 55 ; 26 ; 54 . These models leverage the global attention mechanism of the Transformer architecture to aggregate information from multiple frames, improving the segmentation accuracy of semantic objects in videos. However, due to the quadratic complexity of their attention computation, these methods face computational and memory bottlenecks when processing long video sequences. To efficiently aggregate spatiotemporal information in long video sequences, recent research has introduced methods based on State Space Models (SSM) 14 ; 9 ; 29 . Leveraging selective gating mechanisms, such methods can efficiently compress and propagate spatiotemporal semantic information with fixed-size state space while achieving competitive semantic segmentation accuracy 22 . Nevertheless, as shown in Figure 1 (a), while state space models can preserve common semantic information (e.g., global structure and smooth regions) during state space compression, this fixed-size state space compression process inevitably compromises pixel-level spatiotemporal specifics (e.g., boundaries, textures, local variations), resulting in only coarse segmentation of object locations when performing pixel-level semantic segmentation in videos, but suffering from specifics forgetting. This is detrimental to maintaining temporal consistency of segmentation results for pixel-level semantic information in VSS tasks. To address the above challenge, we propose a Refining Specifics State Space Model approach, named RS-SSM, which performs complementary refining of the specific information forgotten during the SSM’s state space compression process, thereby overcoming the limitations of existing SSM-based VSS methods. Specifically, considering the differences in information contained across the hidden states of different channels in state space models, we design a Channel-wise Amplitude Perceptron (CwAP) module that computes the information distribution characteristics of different channels in the state space model through frequency domain analysis, obtaining Spectrum Features to quantify the differences in specific information amount across channels. Meanwhile, we introduce a channel information loss that minimizes the differences in normalized Spectrum Features across different samples based on cosine similarity, aiming to align the channel distribution characteristics of specific information across samples and encourage RS-SSM to focus on channels rich in specific information. Leveraging the Spectrum Features extracted by the CwAP module, our designed Forgetting Gate Information Refiner (FGIR) module can adaptively invert and refine the forgetting gate matrix of the SSM, thereby guiding our RS-SSM to focus on refining the specific information forgotten during the state space compression process. As shown in Figure 1 (b), benefiting from the guidance of the CwAP and FGIR modules, RS-SSM forms a supplementary specifics refining process, enhancing the capability to capture specific information while maintaining the ability to model common semantic information, thereby improving the pixel-level segmentation performance in VSS tasks. To sum up, the main contributions of this work are: (1) We identify the limitation of existing SSM-based VSS methods in modeling spatiotemporal specific information and propose a Refining Specifics State Space Model (RS-SSM) to address this issue. (2) We design a Channel-wise Amplitude Perceptron (CwAP) module to extract channel-wise specific information distribution characteristics, and a Forgetting Gate Information Refiner (FGIR) module to adaptively refine the forgetting gate matrix of the SSM, thereby encouraging RS-SSM to focus on spatiotemporal specific information and improving pixel-level segmentation accuracy. (3) Extensive experiments on multiple VSS benchmarks demonstrate that our RS-SSM achieves superior performance against existing methods. 2 Related Work 2.1 VSS based on CNNs Among all Convolutional Neural Network (CNN)-based methods, Fully Convolutional Networks (FCN) for natural image semantic segmentation pioneered an end-to-end pixel-level classification framework 46 . However, as video streams represent a more realistic data modality, researchers have increasingly focused on video semantic segmentation tasks, and how to effectively aggregate and propagate spatiotemporal information in video streams has become a key research focus in this field. Figure 2 : The pipeline of our RS-SSM. Following previous work 22 ; 61 , we use an image encoder to extract feature maps for each frame. After linear projection, we extract spectrum features through the CwAP module to quantify the channel distribution of specific information. Then our FGIR module adaptively inverts and refines the forgetting gate from the SSM \theta_{2} , thereby encouraging SSM \theta_{1} to perform complementary refining of forgotten spatiotemporal specifics. Early works combined CNNs 31 with feature propagation based on optical flow estimation and other techniques, leveraging spatiotemporal information to improve segmentation accuracy or reduce computational costs. DFF 77 and Accel 24 propagate features between

Read the complete text