TSTMotion: Training-free Scene-aware Text-to-motion Generation * These authors contributed equally and † is corresponding author.
1 st Ziyan Guo 1 1 ,∗ , 2 nd Haoxuan Qu 2 2 ,∗ , 3 rd Hossein Rahmani 2 2 , 4 th Dewen Soh 2 2 , 5 th Ping Hu 3 3 , 6 th Qiuhong Ke 4 4 , and 7 th Jun Liu 2 2 ,† · 2025
Read TSTMotion: Training-free Scene-aware Text-to-motion Generation * These authors contributed equally and † is corresponding author. by 1 st Ziyan Guo 1 1 ,∗ , 2 nd Haoxuan Qu 2 2 ,∗ , 3 rd Hossein Rahmani 2 2 , 4 th Dewen Soh 2 2 , 5 th Ping Hu 3 3 , 6 th Qiuhong Ke 4 4 , and 7 th Jun Liu 2 2 ,† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01182.txt.
Introduction
Guo and co-authors present in 2025 a training free frame that adds scene awareness to text to motion generation. Most text to motion tools place a figure in a blank space, yet real motion happens inside 3D scenes. Past scene aware methods need costly ground truth motion data in many scenes.
Their frame uses foundation models to reason about a scene and prompt, then predict and check a motion guide. That guide is fed into a standard motion generator with two changes. The output is motion that fits both the text and the scene. Tests show strong results and wide reach across settings, and code was released in public.
Catalog record
| CREATOR | 1 st Ziyan Guo 1 1 ,∗ , 2 nd Haoxuan Qu 2 2 ,∗ , 3 rd Hossein Rahmani 2 2 , 4 th Dewen Soh 2 2 , 5 th Ping Hu 3 3 , 6 th Qiuhong Ke 4 4 , and 7 th Jun Liu 2 2 ,† |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TSTMotion: Training-free Scene-aware Text-to-motion Generation * These authors contributed equally and † is corresponding author. 1 st Ziyan Guo 1 1 ,∗ , 2 nd Haoxuan Qu 2 2 ,∗ , 3 rd Hossein Rahmani 2 2 , 4 th Dewen Soh 2 2 , 5 th Ping Hu 3 3 , 6 th Qiuhong Ke 4 4 , and 7 th Jun Liu 2 2 ,† Abstract Text-to-motion generation has recently garnered significant research interest, primarily focusing on generating human motion sequences in blank backgrounds. However, human motions commonly occur within diverse 3D scenes, which has prompted exploration into scene-aware text-to-motion generation methods. Yet, existing scene-aware methods often rely on large-scale ground-truth motion sequences in diverse 3D scenes, which poses practical challenges due to the expensive cost. To mitigate this challenge, we are the first to propose a T raining-free S cene-aware T ext-to- Motion framework, dubbed as TSTMotion , that efficiently empowers pre-trained blank-background motion generators with the scene-aware capability. Specifically, conditioned on the given 3D scene and text description, we adopt foundation models together to reason, predict and validate a scene-aware motion guidance. Then, the motion guidance is incorporated into the blank-background motion generators with two modifications, resulting in scene-aware text-driven motion sequences. Extensive experiments demonstrate the efficacy and generalizability of our proposed framework. We release our code in Project Page . I INTRODUCTION Fig. 1: Illustration of scene-aware text-driven motion sequences generated by our TSTMotion framework in different 3D scenes based on text descriptions without any training. For clarity, as time progresses, human avatars in motion sequences transition from light to dark colors. More qualitative results in image and video formats are available in the supplementary. Text-to-motion generation, which aims to generate realistic human motion sequences based on the given text descriptions, has recently attracted significant research attention [ 1 , 2 ] . Despite extensive efforts in this area, most research has focused solely on generating human motions with a blank background. However, humans often interact with different objects within diverse 3D scenes in real-world scenarios. For example, a human avatar in video games may be instructed to “sit on the couch away from the TV”. To execute this instruction, the avatar needs to understand the motion semantic of the instruction (i.e., “sit”), comprehend the context of the 3D scene (i.e., “couch away from the TV”), and then perform the desired motion within the 3D scene [ 3 ] . Consequently, traditional text-to-motion generators fail to meet the requirements of various real-world applications. To mitigate this challenge, it is crucial that the generated human motion sequences not only align with the given text descriptions but also appropriately interact with the given 3D scenes. Hence, scene-aware text-to-motion generation has become a valuable research area to advance various applications like game development, film creation, embodied AI, and virtual reality [ 1 , 4 ] . To achieve scene-aware text-to-motion generation, one straightforward solution is to directly train a motion generator using scene-aware text-to-motion datasets [ 3 , 5 , 6 ] . However, this solution may be sub-optimal due to the limited scale and diversity of existing datasets. In particular, they lack a wide-ranging and varied assortment of combinations involving text, motion, and scene components. Take the HUMANISE dataset [ 3 ] as an example: it is confined to indoor scenes and encompasses merely four actions: walking, sitting, lying, and standing. The challenges of scaling and diversifying such datasets are twofold: on one side, these datasets require numerous motion sequences interacting with a variety of objects and 3D environments, which even necessitates sophisticated motion capture systems [ 7 ] ; on the other side, they require detailed annotations of object semantics and human poses, which involves significant manual labor. An alternative solution is to employ Reinforcement Learning (RL) for a human motion controller [ 4 , 8 ] . By leveraging the generalization capabilities of RL, this solution mitigates the reliance on the scale of datasets to some extent. However, this solution may necessitate more fine-grained datasets (e.g., objects annotated in part level) and face the intricacies associated with RL algorithms [ 4 ] . Besides, existing methods often involve directly inputting point clouds into the model, which is inefficient due to the disordered and unstructured nature of point clouds. In light of these challenges in scaling up training data, we aim to propose a training-free solution, thereby bypassing the need for collecting scene-aware text-to-motion datasets and training models with point clouds from scratch. To realize a training-free approach for this task, it is promising to enable the scene awareness of existing blank-background text-to-motion generators, which have an extensive understanding of human motion. Nevertheless, this solution can present significant challenges: (1) These generators might struggle to comprehend 3D scenes, given that their original training input consists solely of text without scene information (e.g., “sit down”) [ 9 ] . Consequently, devising an effective method to enable the generators’ scene awareness—particularly under a training-free paradigm—presents a considerable challenge. (2) Even if these generators can be scene-aware, it is still challenging to provide them with format-compatible and information-sufficient 3D scene knowledge in a training-free manner. To address these challenges, we introduce a novel framework, TSTMotion ( T raining-free S cene-aware T ext-to- Motion ), which pioneers a training-free method for scene-aware text-to-motion generation. Overall, TSTMotion aims to formulate a motion guidance integrated with text semantics and scene contexts, and then perform a training-free alignment between the motion guidance and blank-background motion generators. To achieve such motion guidance, inspired by that foundation models contain extensive knowledge of human motions and 3D scenes, TSTMotion first incorporates three well-designed components based on foundation models, including: the Scene Compiler to represent the given 3D scene into a format comprehensible by the Motion Planner, the Motion Planner to offer a plausible motion guidance for the expected motion, and the Motion Checker to refine the generated motion guidance to ensure practical feasibility. To effectively utilize the above-formulated motion guidance and achieve such alignment based on the motion guidance, TSTMotion further proposes to adapt existing blank-background motion generators (i.e., Motion Diffusion Models [ 1 , 2 ] ) through two training-free modifications: one ensures alignment of the generated motion sequences with the motion guidance; the other reduces the overlapping between motion sequences and 3D scenes. An overview of our proposed TSTMotion framework is illustrated in Fig. 2 . Through these designs, TSTMotion refrains from being trained on specialized scene-aware text-to-motion datasets, and then exhibits increased diversity (i.e., not limited to common walking and sitting) and better generalization (i.e., indoor and outdoor 3D scenes) as shown in Fig. 1 . The contributions of our work are summarized as follows: 1) We are the first to achieve scene-aware text-to-motion generation in a training-free manner, by our novel framework TSTMotion. 2) We incorporate Scene Compiler, Motion Planner and Motion Checker to craft and refine a motion guidance integrating text semantics with scene contexts, through which a blank-background motion generator with two modifications can generate scene-aware text-driven motion sequences in a training-free manner. 3) Extensive experiments demonstrate that TSTMotion achieves superior performance across various benchmarks without the need for specific scene-aware text-to-motion datasets or further training of any model. 4)