Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation
Haoyu Ji, Bowen Chen, Zhihao Yang, Wenze Huang, Yu Gao, Xueting Liu, Weihong Ren, Zhiyong Wang, Honghai Liu · 2026
Read Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation by Haoyu Ji, Bowen Chen, Zhihao Yang, Wenze Huang, Yu Gao, Xueting Liu, Weihong Ren, Zhiyong Wang, Honghai Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24134.txt.
Introduction
Ji and co-authors present Spectral Scalpel, a method that sharpens action boundaries in skeleton sequences. Adjacent actions can look alike, so past methods confuse classes and blur transitions. Spectral Scalpel filters frequencies in an adaptive, multi-scale way. It suppresses frequency parts shared by neighboring actions and boosts parts that set them apart. A discrepancy loss between neighbors guides the edit. A frequency-aware channel mixer adds cross-channel context. On five public datasets, the method reaches state-of-the-art scores, they report.
Catalog record
| CREATOR | Haoyu Ji, Bowen Chen, Zhihao Yang, Wenze Huang, Yu Gao, Xueting Liu, Weihong Ren, Zhiyong Wang, Honghai Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 123910 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24134
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spectral Scalpel: Amplifying Adjacent Action Discrepancy via Frequency-Selective Filtering for Skeleton-Based Action Segmentation Haoyu Ji, Bowen Chen, Zhihao Yang, Wenze Huang, Yu Gao, Xueting Liu, Weihong Ren, Zhiyong Wang, Honghai Liu Abstract Skeleton-based Temporal Action Segmentation (STAS) seeks to densely segment and classify diverse actions within long, untrimmed skeletal motion sequences. However, existing STAS methodologies face challenges of limited inter-class discriminability and blurred segmentation boundaries, primarily due to insufficient distinction of spatio-temporal patterns between adjacent actions. To address these limitations, we propose Spectral Scalpel, a frequency-selective filtering framework aimed at suppressing shared frequency components between adjacent distinct actions while amplifying their action-specific frequencies, thereby enhancing inter-action discrepancies and sharpening transition boundaries. Specifically, Spectral Scalpel employs adaptive multi-scale spectral filters as scalpels to edit frequency spectra, coupled with a discrepancy loss between adjacent actions serving as the surgical objective. This design amplifies representational disparities between neighboring actions, effectively mitigating boundary localization ambiguities and inter-class confusion. Furthermore, complementing long-term temporal modeling, we introduce a frequency-aware channel mixer to strengthen channel evolution by aggregating spectra across channels. This work presents a novel paradigm for STAS that extends conventional spatio-temporal modeling by incorporating frequency-domain analysis. Extensive experiments on five public datasets demonstrate that Spectral Scalpel achieves state-of-the-art performance. Code is available at https://github.com/HaoyuJi/SpecScalpel . † † footnotetext: ∗ Corresponding author 1 Introduction Figure 1 : Effect of a designed frequency filter in amplifying adjacent action discrepancy. The synthetic single-joint motion sequence consists of two concatenated action segments, each idealized as a sum of cosine waves with both shared and action-specific frequency components. After FFT, a designed filter amplifies action-specific frequencies and suppresses shared ones. The filtered sequence shows enhanced action discriminability and a clearer boundary. Temporal Action Segmentation (TAS) is a high-level task in video understanding, with broad applications in medical rehabilitation 7 ; 16 , industrial monitoring 61 ; 59 , sports analysis 47 ; 40 , and human-computer interaction 45 . The objective of TAS is to assign action labels to each frame in an untrimmed human activity video 13 . Skeleton-based Temporal Action Segmentation (STAS) parses human motion through skeletal sequences, offering a lightweight and robust alternative to Video-based approaches (VTAS) 35 ; 27 . Existing STAS methods have made notable progress by leveraging spatio-temporal modeling 35 ; 27 ; 17 ; 39 ; 34 , which captures spatial dependencies among joints and temporal correlations across frames. Current STAS approaches, however, still suffer from two critical limitations: 1) Inter-class confusion arising from insufficiently discriminative features among visually or semantically similar action categories, and 2) Boundary ambiguity caused by indistinguishable transition dynamics between adjacent actions with high similarity. These issues stem from a fundamental dilemma in existing temporal modeling paradigms (e.g., TCNs and Transformers). By essentially acting as low-pass filters to aggregate context, these models introduce an inherent smoothing effect. While highly effective at maintaining intra-action coherence, this over-smoothing inadvertently obscures the essential discrepancies between adjacent distinct actions and blurs sharp transition boundaries. Consequently, there is an urgent need to explicitly capture and amplify the core differences to compensate for this architectural deficiency. To break this bottleneck, we pivot our perspective to the frequency domain. Fundamentally, skeletal motions are composed of joint oscillations, meaning distinct actions exhibit highly divergent periodicities and spectral energy distributions rather than merely superficial temporal trajectories. Specifically, the frequency spectra of different actions inevitably contain both shared common components and unique, action-specific characteristics. Because these subtle action-specific cues are easily erased by the smoothing effect of temporal modeling, it is imperative to explicitly amplify them from a frequency perspective. As illustrated in Fig. 1 , selectively filtering the motion spectrum to suppress shared components and enhance action-specific frequencies effectively amplifies inter-class discriminability and sharpens transition boundaries. Motivated by this insight, we propose Spectral Scalpel , a novel discrepancy-guided frequency-selective filtering framework for STAS. Unlike prior Fast Fourier Transform (FFT)-based filtering methods 23 ; 48 that rely on unconstrained, blind data-driven learning, our approach performs active spectral editing guided by explicit objectives. Analogous to a surgical procedure, where the scalpel serves the surgical objective, our method treats the amplification of adjacent action differences as the “objective” and employs adaptive filtering as the “scalpel” to sharpen feature distinctions, as shown in Fig. 2 . Specifically, the Multi-scale Adaptive Spectral Filter (MASF), functioning as the scalpel, transforms temporal features into the frequency domain via FFT, applies learnable filters at multiple scales, and fuses the filtered features via a dual-branch dynamic-static channel-wise weighting scheme. The Adjacent Action Discrepancy Loss (AADL), serving as the surgical objective, segments filtered features into action-specific units, transforms them into the frequency domain, and explicitly maximizes the spectral discrepancy between adjacent action segments. Additionally, to strengthen subsequent temporal modeling, we introduce the Frequency-Aware Channel Mixer (FACM) to capture frequency-sensitive interactions across feature channels directly in the spectral space. Extensive experiments on five challenging STAS datasets—PKU-MMD v2 (X-sub) 37 , PKU-MMD v2 (X-view) 37 , LARa 43 , MCFS-130 40 , and TCG-15 54 —demonstrate that Spectral Scalpel achieves state-of-the-art performance while maintaining competitive computational efficiency. Figure 2 : Overview of Spectral Scalpel. After spatial modeling, joint features are transformed into the frequency domain. A discrepancy-guided filter, optimized by adjacent action discrepancy loss, enhances action-specific spectra and suppresses shared ones, improving discriminability and boundary clarity. The main contributions are summarized as follows: 1) We propose Spectral Scalpel, the first framework to systematically integrate frequency-domain processing into STAS, enhancing inter-class discriminability and sharpening transition boundaries. 2) We design a discrepancy-guided filtering mechanism comprising the Multi-scale Adaptive Spectral Filter (MASF) and Adjacent Action Discrepancy Loss (AADL), which optimizes spectral discrepancy between actions to effectively guide the filter in learning discriminative representations. 3) We introduce a Frequency-Aware Channel Mixer (FACM) module that strengthens temporal modeling through spectral interaction among channels, enabling frequency-domain-aware channel evolution. 2 Related Work Temporal Action Segmentation. Video-based Temporal Action Segmentation (VTAS) processes RGB or optical flow sequences through temporal modeling. Temporal Convolutional Networks (TCNs) 32 ; 15 ; 36 became popular for modeling long-term dependencies. Subsequent methods enhanced TCNs via dual dilated layers 33 , receptive field design 19 ; 20 , multi-scale fusion 46 , boundary awareness 53 ; 24 , dilated propagation and reconstruction 44 , and diffusion modeling 38 . Some graph-based approaches 22 ; 50 ; 60 model temporal semantics via graph convolutions. Transformer-based methods improve temporal modeling via ASFormer 58 , seq2seq translation 2 , U-shaped designs 14 , sparse attention 1 , boundary-aware voting 51 , and frame-action attention 42 . Skeleton-based Temporal Action Segmentation (STAS) leverages spatio-temporal modeling over skeleton sequences. MS-GCN 17 combines Graph Convolutional Networks (GCNs) for spatial modeling and TCNs for temporal modeling. Subsequent methods further enhanced spatio-temporal capabilities, such as discriminative temporal graph networks 35 , decoupled spatio-temporal frameworks 34 , motion-aware and temporally enhanced methods 3 , contrastive learning-based representation optimization 4 , and multi-scale feature pyramids 6 . Attention-based approaches further enhance spatial-temporal focus, such as spatial focus attention 39 , spatio-temporal graph attention 49 , and snippet-aware Transformer with multiple action elements 25 . Other methods focus on data processing strategies 55 ; 57 ; 56 . Additionally, language-assisted methods 27 ; 5 ; 26 use textual cues 12 to enhance feature modeling. Spectral