Less is More: Decoder-Free Masked Modeling for Efficient Skeleton Representation Learning
Jeonghyeok Do, Yun Chen, Geunhyuk Youk, Munchurl Kim · 2026
Read Less is More: Decoder-Free Masked Modeling for Efficient Skeleton Representation Learning by Jeonghyeok Do, Yun Chen, Geunhyuk Youk, Munchurl Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10648.txt.
Introduction
Jeonghyeok Do and co-authors present SLiM, a compact-token framework for skeleton representation learning that joins masked feature prediction with contrastive learning. Contrastive methods miss fine motion detail, while masked auto-encoders rebuild raw coordinates, demanding dense tokens and heavy decoders. SLiM predicts teacher-guided features instead, with no decoder, so a small token grid suffices. Semantic tube masking and skeleton-aware augmentations block shortcut learning, forcing deeper reasoning about body and time. Across downstream tests it set leading scores while cutting inference compute sharply.
Catalog record
| CREATOR | Jeonghyeok Do, Yun Chen, Geunhyuk Youk, Munchurl Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61363 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10648
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Less is More: Decoder-Free Masked Modeling for Efficient Skeleton Representation Learning Jeonghyeok Do, Yun Chen, Geunhyuk Youk, Munchurl Kim Abstract The landscape of skeleton-based action representation learning has evolved from Contrastive Learning (CL) to Masked Auto-Encoder (MAE) architectures. However, each paradigm faces inherent limitations: CL often overlooks fine-grained local details, while MAE is burdened by computationally heavy decoders. Moreover, MAE suffers from severe computational asymmetry—benefiting from efficient masking during pre-training but requiring exhaustive full-sequence processing for downstream tasks. To resolve these bottlenecks, we propose SLiM ( S keleton L ess i s M ore), a novel unified framework that harmonizes masked modeling with contrastive learning via a shared encoder. By eschewing the reconstruction decoder, SLiM not only eliminates computational redundancy but also compels the encoder to capture discriminative features directly. SLiM is the first framework with decoder-free masked modeling of representative learning. Crucially, to prevent trivial reconstruction arising from high skeletal-temporal correlation, we introduce semantic tube masking , alongside skeletal-aware augmentations designed to ensure anatomical consistency across diverse temporal granularities. Extensive experiments demonstrate that SLiM consistently achieves state-of-the-art performance across all downstream protocols. Notably, our method delivers this superior accuracy with exceptional efficiency, reducing inference computational cost by 7.89 × \times compared to existing MAE methods. 1 Introduction Figure 1 : Conceptual comparison of previous MAE methods and our SLiM. (a) Standard MAE methods suffer from a 14.38\times computational surge during inference relative to pre-training due to asymmetric full-sequence processing. (b) SLiM synergizes masked modeling with contrastive learning in a decoder-free framework. This symmetric design achieves a 7.89\times reduction in inference cost compared to MAE baselines. Skeleton data serves as a compact yet informative modality for human action understanding, offering inherent robustness against dynamic backgrounds, lighting variations, and viewpoint shifts compared to RGB-based counterparts [ 33 , 40 , 15 ] . While supervised learning on skeleton data [ 4 , 6 , 9 ] has achieved remarkable success, it faces fundamental hurdles: beyond the prohibitive cost of large-scale annotation, supervised models often struggle with generalization, frequently overfitting to specific domain distributions and failing to adapt to unseen scenarios. Consequently, Self-Supervised Learning (SSL) [ 2 , 24 , 42 , 3 , 12 ] has emerged as the dominant methodology to learn generalizable motion representations from abundant unlabeled data. The evolution of SSL in skeleton analysis has primarily shifted from Contrastive Learning (CL) [ 3 ] to Masked Auto-Encoder (MAE) [ 12 ] architectures. Early CL approaches [ 17 , 39 , 23 , 11 ] focused on instance discrimination, pulling augmented views of the same sequence together. However, CL relies heavily on global pooling, which often leads to the vanishing local details problem—overlooking fine-grained motion patterns crucial for complex actions—and suffers from the difficulty of constructing semantically consistent pairs. To mitigate this, recent trends have pivoted towards MAE frameworks [ 22 , 38 , 1 , 32 ] that learn by reconstructing masked input patches. Although MAE excels at capturing local dependencies, it introduces severe computational inefficiencies rooted in its architectural design. Fig. 1 shows a conceptual comparison of previous MAE methods and ours. In Fig. 1 -(a), standard MAE frameworks adopt an asymmetric encoder-decoder structure. During pre-training, they mask a massive ratio (e.g., 90%) of input tokens and feed only a small subset of visible tokens into the encoder. Before passing these features to the decoder, the missing positions are filled with learnable mask tokens to restore the full sequence length. This design leads to two distinct computational hurdles: (i) the heavy decoder problem — unlike the lightweight encoder, the decoder must process the full sequence (100%) to reconstruct the original input, consuming significant resources; (ii) more critically, this entails a drastic computational asymmetry between pre-training and downstream application. While the pre-training encoder load is kept artificially light, downstream tasks require processing the full, unmasked sequence. Consequently, the computational cost surges disproportionately during inference—inflating the encoder’s burden by over 14.38 \times compared to the pre-training phase. This renders standard MAE encoders prohibitively expensive for real-world deployment. Table 1 : Efficiency and performance comparison on NTU-60. SLiM achieves state-of-the-art accuracy while reducing inference computational cost by 7.89 \times compared to MAE methods. TGN: Target Generation Network. Bold indicates the best result. Methods Venue Loss Tokens Inference (GFLOPs) Training (GFLOPs) NTU-60 (Acc, %) T\times J Encoder Encoder Decoder TGN X-Sub X-View SkeletonMAE [ 38 ] ICCV’23 MAE 30\times 25 28.32 1.97 17.70 – 74.8 77.7 MAMP [ 22 ] ICCV’23 MAE 30\times 25 28.32 1.97 17.70 – 84.9 89.1 S-JEPA [ 1 ] ECCV’24 MAE 30\times 25 28.32 1.97 17.70 28.32 85.3 89.8 GFP [ 32 ] ICCV’25 MAE 30\times 25 28.32 1.97 1.57 0.64 85.9 92.0 SLiM (Ours) MAE + CL 8\times 25 3.59 3.59 – 3.59 87.9 93.2 To bridge the gap between efficiency and representation quality, we propose SLiM ( S keleton L ess i s M ore), a novel unified decoder-free framework that synergizes the strengths of masked modeling (MAE) and contrastive learning (CL). Unlike previous approaches that treat them separately, SLiM integrates them into a single, shared encoder architecture (Fig. 1 -(b)). Our key insight is to replace the computationally heavy reconstruction decoder with a lightweight, feature-level masking objective. Specifically, SLiM employs a teacher–student distillation setup where the student encoder sees masked views and predicts the teacher’s features of the full view. This design not only resolves the computational bottleneck by removing the decoder but also enables the shared encoder to simultaneously optimize for two complementary objectives: instance discrimination (CL) for global semantic invariance and feature reconstruction (MAE) for fine-grained local context understanding. Table 1 shows that SLiM achieves superior performance and efficiency, surpassing the latest MAE methods in accuracy with a 7.89 \times reduction in inference cost. However, blindly applying masked modeling to skeleton data is suboptimal due to the high skeletal-temporal correlation of human joints. Independent joint masking often allows the model to cheat by simply interpolating missing joints from neighbors without understanding the underlying action semantics. To prevent this, we adopt a Semantic Tube Masking strategy that masks integral body parts across temporal segments, forcing the encoder to infer motion dynamics from global context rather than local coordinates. Furthermore, to maximize the efficacy of the contrastive branch, we propose a suite of improved Skeletal-Aware Augmentations . By generating diverse yet anatomically consistent views—ranging from skeleton-aware rotation and mirroring to bone-aware scaling and variable temporal sampling—we facilitate the construction of real-world plausible pairs. This enables the model to capture invariant representations that are inherently robust to significant viewpoint shifts, scale variations, and speed changes. Our main contributions are summarized as follows: • We introduce SLiM , a novel decoder-free framework by eliminating the heavy reconstruction decoder and adopting a symmetric token processing design, SLiM fundamentally resolves the computational asymmetry and inference overhead inherent in standard MAE architectures. • We propose a unified representation learning architecture that synergizes the global discriminative power of contrastive learning with the local context sensitivity of masked modeling through a single, shared encoder. • To mitigate shortcut reconstruction caused by the high correlation among skeletal joints, we adopt Semantic Tube Masking alongside refined Skeleton-Aware Augmentations which force the encoder to capture deeper action semantics. • Extensive experiments demonstrate that