Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition

Yuxi Zhou, Zhengbo Zhang, Jingyu Pan, Zhiyu Lin, Zhigang Tu · 2026

Read Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition by Yuxi Zhou, Zhengbo Zhang, Jingyu Pan, Zhiyu Lin, Zhigang Tu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09063.txt.

Introduction

Zhou and four colleagues propose a frequency-aware diffusion model for zero-shot skeleton action spotting of actions never seen in training. Labeled pose methods need full labels, blocking transfer to new actions, while such models blur high-rate motion details through spectral bias. Their model joins a semantic-guided spectral residual module, a time-step-tuned spectral loss, and staged learning to recover fine motion detail. It reaches leading scores on NTU RGB+D, PKU-MMD, and Kinetics-skeleton, with code shared.

Catalog record

CREATORYuxi Zhou, Zhengbo Zhang, Jingyu Pan, Zhiyu Lin, Zhigang Tu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66791
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09063

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Frequency-Enhanced Diffusion Models: Curriculum-Guided Semantic Alignment for Zero-Shot Skeleton Action Recognition Yuxi Zhou, Zhengbo Zhang, Jingyu Pan, Zhiyu Lin, Zhigang Tu Abstract Human action recognition is pivotal in computer vision, with applications ranging from surveillance to human-robot interaction. Despite the effectiveness of supervised skeleton-based methods, their reliance on exhaustive annotation limits generalization to novel actions. Zero-Shot Skeleton Action Recognition (ZSAR) emerges as a promising paradigm, yet it faces challenges due to the spectral bias of diffusion models, which oversmooth high-frequency dynamics. Here, we propose Frequency-Aware Diffusion for Skeleton-Text Matching (FDSM), integrating a Semantic-Guided Spectral Residual Module, a Timestep-Adaptive Spectral Loss, and Curriculum-based Semantic Abstraction to address these challenges. Our approach effectively recovers fine-grained motion details, achieving state-of-the-art performance on NTU RGB+D, PKU-MMD, and Kinetics-skeleton datasets. The project page and source code are publicly available at https://yuzhi535.github.io/FDSM.github.io . keywords Skeleton action recognition, Zero-shot skeleton action recognition, Human pose estimation, Diffusion model, Multi-modality fusion † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work, are co-first authors. 1 Introduction Human action recognition stands as a cornerstone of computer vision, underpinning applications ranging from intelligent surveillance 59 to human-robot interaction 59 ; 25 . While early approaches predominantly relied on RGB video data, the field has witnessed a paradigm shift toward skeleton-based modalities, driven by the proliferation of cost-effective depth sensors and robust pose estimation algorithms 71 ; 3 ; 27 . However, despite the efficacy of fully supervised skeleton-based methods 26 ; 13 ; 50 ; 38 ; 65 ; 79 ; 67 , their reliance on exhaustive annotation limits generalization to fixed categories, rendering them ineffective for open-world scenarios where novel actions emerge continuously 63 . To overcome this bottleneck, Zero-Shot Skeleton Action Recognition (ZSAR) has emerged as a compelling paradigm. By exploiting auxiliary semantic information—ranging from manual attributes for seen classes to large-scale pre-trained language knowledge for unseen concepts—ZSAR effectively transfers recognition capabilities across the domain gap 17 ; 28 ; 21 ; 82 ; 33 ; 8 ; 85 ; 36 . Nevertheless, effective ZSAR is hindered by the profound modality gap between the high-frequency spatio-temporal dynamics of skeletal data and the abstract, static nature of semantic descriptions. Early approaches utilized static discriminative mappings (e.g., VAEs or CLIP) to bridge this divide, yet such rigid point-to-point alignments often struggle to capture the complex temporal evolution of motion 85 ; 17 . Consequently, recent research has pivoted toward generative paradigms, specifically diffusion models 24 ; 53 ; 17 ; 77 . Unlike discriminative methods, diffusion models utilizes the reverse diffusion process to implicitly align skeleton and text features within a shared latent space, ensuring that generated outputs adhere closely to the given condition 17 . By modeling the data distribution as a gradual denoising process, diffusion models demonstrate a remarkable capacity to capture the intricate manifold of skeletal motion, overcoming the challenges of direct feature space alignment between skeleton and text modalities. However, despite this theoretical promise, the direct application of standard diffusion paradigms to ZSAR exposes critical vulnerabilities rooted in the interplay between signal frequency and semantic interpretation. Firstly , regarding frequency, skeletal actions are spectrally diverse, comprising low-frequency structural components (representing global pose trajectories) and high-frequency details (encoding rapid, fine-grained dynamics) 66 . As illustrated in Fig. 1 , this distinction is clearly visible in ground-truth skeleton sequences: high-frequency micro-dynamics concentrate at the extremities (e.g., foot acceleration in Kicking something ) and are nearly absent in near-static actions such as Reading . However, standard diffusion models face a dual spectral bottleneck that independently constrains high-frequency synthesis. On the architectural side, recent theoretical findings 58 reveal that Transformers, empirically identified as the optimal backbone for diffusion-based ZSAR 17 , exhibit an inherent “low-pass” inductive bias, causing the backbone to act as a spectral filter that smooths out sharp, rapid motion cues essential for distinguishing complex actions. On the optimization side, standard Mean Squared Error objectives inherently average out stochastic high-frequency variations, while training dynamics favor low-frequency convergence 76 ; 52 . Together, these dual biases result in generated motions that are structurally coherent but dynamically oversmoothed, lacking the discriminative high-frequency fidelity required for zero-shot recognition. Secondly , regarding semantics, current methods 17 face a semantic-dynamic ambiguity. Conditioning generation solely on coarse-grained class labels (e.g., “Walking”) fails to explicitly convey the fine-grained physical dynamics—such as tempo, rhythm, and intensity—that define the motion’s spectral signature. Without this explicit dynamic guidance, the model struggles to determine whether high-frequency components in the target data represent essential action details (as in “Punching/Slapping”) or mere noise (as in “Sitting down”), leading to synthesized features that are either structurally oversmoothed or polluted with hallucinations. Figure 1: Frequency-domain visualization of skeleton sequences via DCT decomposition. Each row shows one action: (left) original skeleton with all frequency components; (center) low-frequency reconstruction capturing global pose topology; (right) high-frequency energy heat map, where bubble size at each joint indicates the magnitude of residual components. High-frequency micro-dynamics concentrate at extremities (hands, feet) in dynamic actions (e.g., Clapping, Waving hand) but are absent in near-static actions (e.g., Reading), illustrating why high-frequency suppression in the Diffusion Transformer degrades fine-grained action discrimination. To address these limitations, we construct a unified Frequency-Aware Diffusion for Skeleton-Text Matching (FDSM) framework that establishes coherent spectral constraints between the input condition and the optimization objective. Our approach is built upon three synergistic components. To counteract the backbone’s spectral bias, we introduce a frequency-domain residual module that selectively amplifies high-frequency dynamics based on kinematic priors. Complementing this, we design a dynamic spectral loss that aligns frequency supervision with the diffusion model’s native coarse-to-fine generation process, ensuring physically valid reconstruction without overfitting to noise. Furthermore, to bridge the semantic gap, we employ a curriculum learning strategy that transfers rich kinematic knowledge from LLMs into the visual encoder, enabling robust inference from sparse labels. Our main contributions can be summarized as follows: • We propose the Semantic-Guided Spectral Residual Module to address the inherent “low-pass” bias of standard generative backbones. By establishing a frequency-domain residual pathway, this module explicitly amplifies discriminative high-frequency dynamics, effectively counteracting over-smoothing and enabling the synthesis of fine-grained motion details. • We introduce the Timestep-Adaptive Spectral Loss as a synergistic optimization objective. This mechanism aligns the frequency supervision with the diffusion model’s denoising schedule, preventing overfitting to noise in early stages while enforcing rigorous high-frequency reconstruction in the final refinement steps. • We design a Curriculum-based Semantic Abstraction strategy to bridge the semantic gap in zero-shot inference. By training with a ‘rich-to-sparse’ curriculum of kinematic descriptions, we force the visual encoder to internalize complex motion priors, enabling robust generalization from sparse labels. • Extensive experiments on three benchmark datasets (NTU RGB+D 57 , NTU RGB+D 120 39 , PKU-MMD 37 and Kinetics-skeleton 70 ; 31 ) demonstrate that our method achieves state-of-the-art performance, significantly outperforming existing generative and discriminative baselines in both Zero-Shot (ZSL) and Generalized Zero-Shot (GZSL) settings. 2 Related Work 2.1 Zero-Shot Skeleton Action Recognition The fundamental goal of Zero-Shot Skeleton Action Recognition (ZSAR) is to identify human behaviors from novel categories without access to labeled training instances. The dominant paradigm addresses the heterogeneity between skeletal kinematics and textual semantics by constructing a shared embedding space.

Read the complete text