In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
Xiaomeng Fu, Junfan Lin, Yang Liu, Yaowei Wang, Guanbin Li, Liang Lin, Ziliang Chen · 2026
Read In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics by Xiaomeng Fu, Junfan Lin, Yang Liu, Yaowei Wang, Guanbin Li, Liang Lin, Ziliang Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26981.txt.
Introduction
Xiaomeng Fu and six colleagues generate human motion from text by coupling a language model planner to physics checks. Language models read open-ended instructions well and sketch plausible action plans. Their raw motions, though, often break physics. Physics-based controllers are realistic but choke on complex or novel instructions. Their framework runs the two in a predictive loop.
A planner module breaks each command into pieces and drafts candidate motions from motion tokens. A second module scores the candidates in simulation. It combines physical success with match to the text's meaning. The best candidate guides the next stretch of generation, like model predictive control in robotics. The loop keeps semantic freedom while discarding motions the body could not perform.
Catalog record
| CREATOR | Xiaomeng Fu, Junfan Lin, Yang Liu, Yaowei Wang, Guanbin Li, Liang Lin, Ziliang Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics Xiaomeng Fu, Junfan Lin, Yang Liu, Yaowei Wang, Guanbin Li, Liang Lin, Ziliang Chen Abstract Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism. Large language model (LLM)-based approaches can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints. Physics-aware models improve realism through simulation or control, but they struggle with semantic complexity, fine-grained instructions, and novel concepts. To address this gap, we propose In-Context Model Predictive Generation (ICMPG), a framework that integrates language-model planning with inference-time physical feedback. ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules. The Context-Aware Motion Generation (CAMG) module uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens. The Model Predictive Generation (MPG) module evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps. Unlike open-loop generation, this closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment without task-specific policy retraining. Extensive experiments across standard and zero-shot open-vocabulary settings show that ICMPG generalizes robustly to diverse commands and produces motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks. The framework bridges semantic interpretation and physical simulation while remaining flexible enough to incorporate different LLM backbones, enabling more versatile and controllable text-driven motion synthesis. I Introduction Synthesizing human motion is increasingly important for enhancing user engagement in immersive digital environments, such as gaming, virtual reality, and cinematic production. Although existing text-to-motion methods [ 43 , 11 , 57 , 58 , 16 , 61 , 50 , 3 , 31 ] have demonstrated considerable progress, their effectiveness is limited by their substantial dependence on large-scale human motion capture datasets. This reliance inherently limits their ability to generalize to unseen textual descriptions. To address this limitation, open-vocabulary motion generation tasks and related techniques have been developed. These methods acquire discriminative representations without requiring extensive data corpora, thereby enabling the synthesis of human motion from a broader range of textual inputs. Consequently, this direction has become an important research topic. Early studies on open-vocabulary motion synthesis [ 41 , 14 ] trained motion sequence generators using contrastive multimodal constraints. These methods render generated motions as a series of images and align the resulting features with their textual descriptions using CLIP [ 33 ] . However, this rendering-based paradigm is limited because projecting complex 3D kinematic data onto a 2D image plane inevitably discards important spatial details and temporal dynamics. Furthermore, the generated motions may deviate from the intended actions because of CLIP’s limited ability to resolve ambiguous semantics. In this context, large language models (LLMs) [ 44 , 2 , 52 ] have emerged as a promising methodological foundation, as subsequent studies have shown that the autoregressive capabilities of pre-trained LLMs are not confined to the language domain. This insight has facilitated the development of LLM-based human motion synthesis [ 38 , 24 ] , which leverages the contextual understanding of LLMs to interpret open-vocabulary textual cues as sequences of detailed instructions, providing a semantically grounded framework for guiding text-to-motion generation. However, the high-level semantic plans generated by LLMs are not inherently grounded in physical reality. Although the motions are generated to semantically follow the textual instructions, they frequently violate fundamental physical laws. The resulting sequences may therefore exhibit physically implausible dynamics and fail to interact realistically with the environment. To reproduce natural motions from text, physics-aware approaches provide physical constraints that textual supervision alone cannot offer. More specifically, PhysDiff [ 55 ] incorporates physical simulation constraints into a diffusion model and performs physical simulation on intermediate results at each diffusion step to generate motions that adapt to physical environments. AnySkill [ 6 ] builds on physics-based motion imitation by integrating a semantic reward associated with open-vocabulary text into its reinforcement learning [ 51 , 39 ] process, enabling the agent to adapt to open-vocabulary settings. However, a critical limitation of these physics-aware models is their semantic brittleness when confronted with open-vocabulary text. Because these approaches typically employ reinforcement learning to maximize generation quality in physical environments, they may insufficiently capture fine-grained instructions. As a result, such generators are less effective at synthesizing human motions for complex and imaginative prompts, for which semantic understanding is essential and physical accuracy alone is insufficient. To reconcile high-level linguistic semantics with low-level physical compatibility in generated motions, we propose an In-Context Model Predictive Generation (ICMPG) framework inspired by Model Predictive Control (MPC) [ 35 , 21 ] for open-vocabulary human motion generation. Our method draws a conceptual analogy to the MPC framework, as illustrated in Fig. 1 . In a traditional MPC setup, the plant model represents the system dynamics and describes how the system state evolves in response to control inputs. The MPC controller uses this model to predict future states over a finite horizon, optimize a control sequence, apply the first control action to the plant, and repeat the procedure after observing the updated state. By treating the generated motion prefix as the current state and candidate future motion segments as predictions over a finite horizon, ICMPG reformulates motion synthesis as a receding-horizon optimization problem. Specifically, at each iteration, the model generates a set of candidate motion sequences. Through prediction, simulation, and comparison, it repeatedly computes physical and semantic rewards for these sequences, encouraging the generated motion to adapt to the physics engine while remaining aligned with the input semantics. The framework then selects the sequence with the highest reward. These simulation-based rewards guide the model in making subsequent motion predictions. Unlike prior reinforcement learning (RL) approaches that require task-specific policies to be trained from scratch, ICMPG performs closed-loop, inference-time optimization conditioned on demonstration examples. By combining the temporal planning capabilities of frozen LLMs with physical control, ICMPG provides more adaptable and robust open-vocabulary motion generation, addressing a key limitation of previous approaches. To implement this methodology, we formulate the ICMPG framework with two modules: Context-Aware Motion Generation (CAMG) and Model Predictive Generation (MPG). CAMG is an LLM-based module designed to generate initial open-vocabulary motions. It uses a VQ-VAE [ 48 ] to encode human motions into discrete motion tokens and employs an external LLM, which can also be fine-tuned, to learn motion-token representations through in-context learning. CAMG enables the LLM to infer relationships between motion tokens and natural language tokens, leveraging its semantic capabilities to process open-vocabulary text. Based on these generated motions, the MPG module guides CAMG by placing the motions in a simulated physical environment and using an external semantic evaluator to compute semantic rewards for the current motion sequences, thereby informing the next CAMG generation step. MPG reformulates motion generation as a stepwise refinement process, addressing the limited controllability of single-pass outputs produced by CAMG and other open-vocabulary methods. Our contributions are outlined as follows: Fig. 1: Conceptual Analogy Between Classical Model Predictive Control (MPC) and the Proposed In-Context Model Predictive Generation (ICMPG) Framework. Top: a