Encoder-Free Human Motion Understanding via Structured Motion Descriptions

Yao Zhang, Zhuchenyang Liu, Thomas Ploetz, Yu Xiao · 2026

Read Encoder-Free Human Motion Understanding via Structured Motion Descriptions by Yao Zhang, Zhuchenyang Liu, Thomas Ploetz, Yu Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21668.txt.

Introduction

Yao Zhang and colleagues turn human motion into text so large language models can grasp movement with no special motion coder. Their structured motion description turns joint position runs into plain notes on joint angles, body-part moves, and whole-body paths. It uses the plain words of biomechanical study. The text-only route tops past methods on motion question answering and captioning tests, works across varied language models with light tuning, and keeps its logic open to view.

Catalog record

CREATORYao Zhang, Zhuchenyang Liu, Thomas Ploetz, Yu Xiao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62365
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21668

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Encoder-Free Human Motion Understanding via Structured Motion Descriptions Yao Zhang, Zhuchenyang Liu, Thomas Ploetz, Yu Xiao Abstract The world knowledge and reasoning capabilities of text-based large language models (LLMs) are advancing rapidly, yet current approaches to human motion understanding, including motion question answering and captioning, have not fully exploited these capabilities. Existing LLM-based methods typically learn motion-language alignment through dedicated encoders that project motion features into the LLM’s embedding space, remaining constrained by cross-modal representation and alignment. Inspired by biomechanical analysis, where joint angles and body-part kinematics have long served as a precise descriptive language for human movement, we propose Structured Motion Description (SMD) , a rule-based, deterministic approach that converts joint position sequences into structured natural language descriptions of joint angles, body part movements, and global trajectory. By representing motion as text, SMD enables LLMs to apply their pretrained knowledge of body parts, spatial directions, and movement semantics directly to motion reasoning, without requiring learned encoders or alignment modules. We show that this approach goes beyond state-of-the-art results on both motion question answering (66.7% on BABEL-QA, 90.1% on HuMMan-QA) and motion captioning (R@1 of 0.584, CIDEr of 53.16 on HumanML3D), surpassing all prior methods. SMD additionally offers practical benefits: the same text input works across different LLMs with only lightweight LoRA adaptation (validated on 8 LLMs from 6 model families), and its human-readable representation enables interpretable attention analysis over motion descriptions. Code, data, and pretrained LoRA adapters are available at https://yaozhang182.github.io/motion-smd/ . 1 Introduction Figure 1 : Comparison of (a) the previous encoder-based paradigm, which requires a complex learned motion encoder and multi-stage alignment training tied to a specific LLM, versus (b) our approach, which converts motion to Structured Motion Descriptions via deterministic rule-based computation, enabling any LLM to process it directly with LoRA fine-tuning as the only training step. Understanding human motion from skeletal data is a fundamental problem in computer vision, encompassing tasks such as motion question answering (QA) and motion captioning. Motion QA [ 5 , 10 ] requires answering natural language questions about a given motion sequence, such as identifying which body part is moving, in which direction, or what action is being performed, demanding fine-grained spatio-temporal reasoning over joint-level motion details. Motion captioning [ 6 , 31 ] requires generating a natural language description that summarizes the overall motion content, bridging the gap between low-level skeletal data and high-level semantic understanding. The rapid advancement of large language models (LLMs) has motivated recent efforts to tackle these tasks with LLMs, leveraging their world knowledge and reasoning capabilities for natural language interaction with motion data. Most of existing LLM-based approaches to these tasks [ 31 , 22 , 9 , 2 , 21 ] follow an encoder-based paradigm: a learned motion encoder projects joint position sequences into the LLM’s token embedding space, either through discrete tokenization via VQ-VAE [ 9 , 22 ] , continuous latent encoding via VAE [ 31 ] , or linear projection [ 2 ] , following the multi-modal alignment framework established in vision-language models (VLMs) [ 14 ] . These methods have driven substantial progress, yet the reliance on cross-modal representation and alignment introduces practical considerations: the motion encoder and alignment module are typically trained through multi-stage pipelines with paired motion-text data, the resulting system is coupled to a specific LLM backbone, and the learned motion tokens are not directly human-readable. We take a different perspective. LLMs are pretrained on vast amounts of natural language and already encode rich knowledge about human body parts, spatial directions, and movement semantics [ 11 ] . A natural alternative is to meet the LLM in its native modality: describe motion in text, so that the model can apply its pretrained linguistic and commonsense knowledge directly. The question then becomes how to produce such a description that is precise enough to support fine-grained reasoning while being expressed in terms the LLM already understands. Biomechanics offers a natural foundation for such a description. Joint angles and body-part kinematics have long served as a precise descriptive language for human movement [ 18 , 1 , 29 ] : clinical gait analysis, for instance, characterizes walking patterns through time-varying flexion curves of the hip, knee, and ankle. These descriptors are already textual in nature: a statement such as "hip flexion increases from 3° to 81°" is both a quantitative measurement and a natural-language sentence. We extend this to full-body motion by computing angular changes across all major joints together with global trajectory information, and use the resulting description as direct textual input for LLM. Based on this, we propose Structured Motion Description (SMD) , a deterministic, rule-based conversion from joint position sequences to structured natural language text. Figure 1 illustrates the difference between the encoder-based paradigm and our approach. By replacing learned motion embeddings with descriptive text, SMD allows LLM to process motion directly in its native text modality. Because SMD uses natural language terms for body parts, spatial directions, and temporal patterns, it naturally leverages the LLM’s world knowledge of these concepts [ 11 ] , as validated by our zero-shot experiments in Section 4 . With lightweight LoRA [ 8 ] fine-tuning as the only training step, we demonstrate three key advantages: 1. Strong performance. SMD goes beyond state-of-the-art results on motion QA [ 5 , 10 ] and captioning [ 6 ] , surpassing prior methods including dedicated motion models and encoder-based LLM approaches. 2. LLM-agnostic flexibility. The same text input works across different LLMs with only a lightweight LoRA adapter ( \sim 40M parameters, 2–8 GPU-hours on a single H200), without retraining the motion encoder or alignment module. We validate this across 8 LLMs from 6 model families. 3. Built-in interpretability. Attention analysis over the human-readable SMD tokens directly reveals which body parts and trajectory segments the model relies on for generation, offering additional interpretability. 2 Related Work Motion representation. 3D human motion is most commonly represented as sequences of joint positions or rotations derived from parametric body models such as SMPL [ 15 ] . In human motion-related methods, the dominant numerical representation is the 263-dimensional HumanML3D feature [ 6 ] , which encodes root velocities, local joint positions, joint rotations in 6D continuous form [ 30 ] , joint velocities, and foot contact labels. While expressive, these high-dimensional numerical vectors require learned encoders to bridge the gap to language models. An alternative line of work represents motion through interpretable, human-readable descriptors. In computer vision, PoseScript [ 3 ] and PoseFix [ 4 ] generate natural language descriptions of static poses for retrieval and correction tasks, but do not handle temporal motion sequences. 29 compute biomechanical joint angles from skeleton sequences and map them into structured pseudo-images for fine-grained motion retrieval with vision transformers. Our work builds on joint-angle representations but converts them into structured text rather than images, enabling direct processing by any LLM without learned visual encoders. Motion understanding. Human motion understanding encompasses a range of tasks that require reasoning about skeletal motion sequences. We focus on two representative tasks: motion QA, which requires answering specific questions about a motion (e.g., identifying body parts, directions, or actions), and motion captioning, which requires generating a free-form natural language summary of the motion

Read the complete text