SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition

Qilang Ye, Yu Zhou, Lian He, Jie Zhang, Xuanming Guo, Jiayu Zhang, Mingkui Tan, Weicheng Xie, Yue Sun, Tao Tan, Xiaochen Yuan, Ghada Khoriba, Zitong Yu · 2025

Read SUGAR: Learning Skeleton Representation with Visual-Motion Knowledge for Action Recognition by Qilang Ye, Yu Zhou, Lian He, Jie Zhang, Xuanming Guo, Jiayu Zhang, Mingkui Tan, Weicheng Xie, Yue Sun, Tao Tan, Xiaochen Yuan, Ghada Khoriba, Zitong Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10091.txt.

Description

Qilang Ye and twelve co-authors present SUGAR, a method that lets a large language model recognize human actions from skeleton data. Skeletons are compact and privacy friendly, but a text-trained model cannot read joint coordinates natively, nor tell similar actions apart without help. SUGAR first mines big video models as a knowledge base for visual and motion information about each action.

That prior knowledge supervises the learning of discrete skeleton representations, which the language model then reads with its pre-trained weights untouched to produce action labels and descriptions. A temporal query projection module handles long skeleton sequences. Benchmarks on skeleton action classification favored the approach, and zero-shot tests showed more versatility than linear methods, suggesting language models can serve as skeleton recognizers when given the right translation.

Catalog record

CREATORQilang Ye, Yu Zhou, Lian He, Jie Zhang, Xuanming Guo, Jiayu Zhang, Mingkui Tan, Weicheng Xie, Yue Sun, Tao Tan, Xiaochen Yuan, Ghada Khoriba, Zitong Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10091

Browse the full library · About the Tower of Babel