Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement
Yang Liu Xilin Zhao Peisong Wen Siran Dai Qingming Huang · 2025
Read Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement by Yang Liu Xilin Zhao Peisong Wen Siran Dai Qingming Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20280.txt.
Introduction
Liu, Zhao, Wen, Dai, and Huang coach video generators to respect real physics. Their 2025 paper refines generated videos through guided self-correction. Modern video models produce gorgeous but physically wrong clips. Objects float, collide wrongly, or melt through each other. Prompt quality strongly steers what models can do.
Their framework loops a vision-language critic over drafts. The critic spots physical inconsistencies in each attempt. A multimodal chain-of-thought turns complaints into better prompts. Regeneration then fixes the flagged errors progressively. No training is needed for the generator itself. The loop plugs into many video models as-is. Tests ran on the PhyIQ physics benchmark. Scores rose from 56.31 to 62.38 overall. That is a gain of about six points. Language models supply the physics-aware guidance throughout. Prompt design emerges as a capability trigger. World-model ambitions need exactly this grounding. The authors frame it as early exploration.
Catalog record
| CREATOR | Yang Liu Xilin Zhao Peisong Wen Siran Dai Qingming Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 8 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20280
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement Yang Liu Xilin Zhao Peisong Wen Siran Dai Qingming Huang Abstract Recent progress in video generation has led to impressive visual quality, yet current models still struggle to produce results that align with real-world physical principles. To this end, we propose an iterative self-refinement framework that leverages large language models and vision-language models to provide physics-aware guidance for video generation. Specifically, we introduce a multimodal chain-of-thought ( MM-CoT ) process that refines prompts based on feedback from physical inconsistencies, progressively enhancing generation quality. This method is training-free and plug-and-play, making it readily applicable to a wide range of video generation models. Experiments on the PhyIQ benchmark show that our method improves the Physics-IQ score from 56.31 to 62.38 . We hope this work serves as a preliminary exploration of physics-consistent video generation and may offer insights for future research. 1 Introduction Recent advances in video generation have led to remarkable progress, exemplified by models such as Sora [ 16 ] , Lumiere [ 5 ] , and VideoPoet [ 12 ] , which produce high-quality videos with clear details, natural dynamics, and realistic rendering. However, the gap between current video generation systems and world models remains substantial. A key step toward bridging this gap lies in establishing a stronger connection between generative models and the physical principles of the real world [ 13 , 11 , 3 , 4 , 18 ] . With the expansion of training datasets and the scaling up of model capacity, an increasing number of video generation models (VGMs) have demonstrated preliminary abilities in modeling real-world physical dynamics through large-scale pretraining [ 9 , 27 ] or distillation fine-tuning [ 29 , 10 ] based on advanced visual representations [ 20 , 22 , 17 , 25 , 15 , 7 ] . However, how to effectively elicit such physics-aware generation ability remains an important challenge. Prior work has shown that well-designed prompts not only steer models toward desired outputs but can also trigger capability emergence [ 26 , 28 , 24 ] . Therefore, we believe that high-quality prompt design is essential for guiding video models to generate physics-grounded content. Concurrently, large language models (LLMs) [ 8 , 1 , 14 ] and vision-language models (VLMs) [ 6 , 2 ] have achieved rapid progress, delivering breakthroughs across vision, language, and cross-modal tasks. These advances provide a strong foundation for the automated construction of physics-aware prompts. Building on this, we leverage state-of-the-art LLMs and VLMs in an iterative physics-guided prompting framework, introducing multimodal chain-of-thought ( MM-CoT ) reasoning to progressively elicit the video generation model’s physical modeling capabilities. Figure 1 : Overview of our method. Specifically, we first provide the VLM with a concise physics prior, along with the prefix video and its original description from the Challenge. The VLM then generates a detailed prediction of future dynamics, incorporating explicit physical cues. Subsequently, this prediction is rewritten by an LLM into a prompt compatible with the VGM, which then generates the future segment conditioned on the prefix video. The generated video is fed back into the VLM to identify potential violations of physical laws, producing a revised description that is then rewritten and fed back into the VGM. The iterative process continues until the LLM’s output converges, indicating the model’s physics-grounded generation has been sufficiently elicited. Experiments on the PhyIQ benchmark [ 19 ] show that our method achieves a Physics-IQ Score of 62.38, yielding an improvement of 6.07 over the baseline on the leaderboard with a score of 56.31, which suggests its effectiveness in enhancing physical consistency. Notably, the proposed framework is training-free and plug-and-play, making it readily applicable to a variety of state-of-the-art VGMs. 2 Method This task focuses on generating physically consistent video continuations. Given a 3-second prefix video V^{\text{pre}} and a short textual description of the scene T^{\text{ori}} , the goal is to generate the next 5 seconds of video V^{\text{gen}} in a way that is temporally coherent and physically plausible. As shown in Fig. 1 , we propose a physics-aware video generation framework guided by the collaborative reasoning of LLM and VLM. The pipeline consists of three stages: Step 1: Physics-Aware Text Prediction. Leveraging the VLM’s strong capabilities in video understanding and physical reasoning, we first extract explicit physical cues as textual predictions. The system input consists of a concise physics knowledge base B and task-specific instructions I . For each sample, the VLM f processes the 3-second prefix video v^{\text{pre}}_{i} and its corresponding textual description t^{\text{ori}}_{i} to generate a detailed, physics-enriched prediction: t^{\text{1}}_{i}=f(t^{\text{ori}}_{i},v^{\text{pre}}_{i};B,I) . However, this output is often verbose and semantically misaligned with the VGM, making it unsuitable as a direct prompt. To bridge this gap, we introduce an LLM g to rewrite and simplify the VLM output, producing a concise prompt: p^{1}_{i}=g(t^{\text{1}}_{i}) . Step 2: Iterative Self-Refinement via Multimodal Reasoning Chain. The simplified prompt p^{1}_{i} and the prefix video v^{\text{pre}}_{i} are fed into the VGM h to generate a continuation video v^{1}_{i}=h(p^{1}_{i},v^{\text{pre}}_{i}) . As a single pass may not yield physically consistent results, we introduce an iterative refinement loop. The generated video is re-processed by the VLM, which detects physical inconsistencies and produces an updated description emphasizing missing or violated physical cues. This description is then refined by the LLM into a new prompt, forming a multimodal chain-of-thought across iterations: p^{k+1}_{i}=g(f(h(p^{k}_{i},v^{\text{pre}}_{i});B,I)) . Step 3: Convergence Detection and Output. The process continues until the prompts converge, i.e., when p^{k+1}_{i}\approx p^{k}_{i} , indicating that the VGM’s capacity for modeling the scene’s physical dynamics has been sufficiently activated. Finally, the generated video v^{\text{gen}}_{i}=v^{k}_{i} is returned. 3 Experiments Implementation Details. We employ GPT‑4o [ 1 ] as the LLM, Gemini 2.5 Pro [ 6 ] as the VLM, and MAGI‑1 [ 23 ] as the VGM. The entire pipeline is implemented on the Dify automated workflow platform and a local PyTorch 2.2 [ 21 ] environment. During inference, we experiment with 16 and 32 steps. All generated videos are 5 seconds at 24 FPS. Quantitative Results. The evaluation results of our method on the benchmark are presented in Table 1 . Overall, iterative prompting leads to consistent performance gains, driven by the VLM’s strong video understanding and the LLM’s ability to generate physics-aware prompts, which progressively activate the VGM’s latent capacity for physical modeling. Due to varying video complexity, the benefits of iteration do not emerge uniformly across samples. To mitigate this, we adopt a simple ensemble strategy that combines the best outputs from multiple iterations, achieving a Physics-IQ score of 62.38, which improves upon the baseline by 6.07. Qualitative Analysis. Fig. 2 visualizes generated videos across five physical domains. For relatively simple physical dynamics, the VGM produces results that closely align with real-world physics, supporting the effectiveness of our method for physically consistent generation. Table 1 : Performance across Iterative Loops on the Physics-IQ Benchmark. \ast indicates partial refinement on incomplete prompts. No. Method Infer. Steps Physics-IQ Score ( \uparrow ) 1 1st Loop 16 49.80 2 2nd Loop 16 48.31 ∗ 3 3rd Loop 16 51.65 4 4th Loop 16 52.92 5 1st Loop 32 49.49 6 4th Loop 32 49.15 ∗ 7 Ensemble {1,2,5} 57.09 8 Ensemble {1,2,3,4,5,6} 62.38