Improving the physics of video generation with VJEPA-2 reward signal
Jianhao Yuan Xiaofeng Zhang Felix Friedrich Nicolas Beltran-Velez, Melissa Hall Reyhane Askari-Hemmat Xiaochuang Han, Nicolas Ballas, Michal Drozdzal Adriana Romero-Soriano · 2025
Read Improving the physics of video generation with VJEPA-2 reward signal by Jianhao Yuan Xiaofeng Zhang Felix Friedrich Nicolas Beltran-Velez, Melissa Hall Reyhane Askari-Hemmat Xiaochuang Han, Nicolas Ballas, Michal Drozdzal Adriana Romero-Soriano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21840.txt.
Introduction
Jianhao Yuan leads a ten-author team reporting the winning entry in the PhysicsIQ Challenge, a contest testing whether generated videos obey real physics. Video generators often look convincing while breaking basic physical rules, so the team asked whether a self-supervised video model that has learned how the world moves could grade and guide a generator.
Built on the MAGI-1 video generator, their system takes reward signals from VJEPA-2, a model trained to predict how scenes evolve. Steering generation with that signal raised the physical plausibility of the output by about six percent on the challenge measure. The short technical report, presented at an ICCV 2025 workshop, suggests that learned world knowledge can police video models where surface realism cannot.
Catalog record
| CREATOR | Jianhao Yuan Xiaofeng Zhang Felix Friedrich Nicolas Beltran-Velez, Melissa Hall Reyhane Askari-Hemmat Xiaochuang Han, Nicolas Ballas, Michal Drozdzal Adriana Romero-Soriano |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 4 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21840
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving the physics of video generation with VJEPA-2 reward signal
Jianhao Yuan Xiaofeng Zhang Felix Friedrich Nicolas Beltran-Velez, Melissa Hall Reyhane Askari-Hemmat Xiaochuang Han, Nicolas Ballas, Michal Drozdzal Adriana Romero-Soriano
Abstract
This is a short technical report describing the winning entry of the PhysicsIQ Challenge, presented at the Perception Test Workshop at ICCV 2025.
1
Introduction
State-of-the-art video generative models
[
1
,
11
,
5
,
6
,
3
,
10
]
exhibit severely limited physical understanding, and often produce implausible videos. The Physics IQ benchmark
[
7
]
has shown that visual realism does not imply physics understanding. Yet, intuitive physics understanding has shown to emerge from SSL pretraining on natural videos
[
2
]
. In this report, we investigate whether we can leverage SSL-based video world models to improve the physics plausibility of video generative models.
In particular, we build ontop of the state-of-the-art video generative model MAGI-1
[
1
]
and couple it with the recently introduced Video Joint Embedding Predictive Architecture 2 (VJEPA-2) to guide the generation process
[
1
]
. We show that by leveraging VJEPA-2 as reward signal, we can improve the physics plausibility of state-of-the-art video generative models by
\sim 6\%
.
2
Background
MAGI-1
[
1
]
.
MAGI-1 is an autoregressive video generative model that predicts a sequence of video chunks. The model is trained to iteratively denoise each chunk and enables casual temporal video modeling. The model is able to generate videos when conditioned on text, text and image, as well as text and video.
At the inference time, the model uses the following score function to guide the diffusion process:
\displaystyle\nabla_{x_{t}}\textrm{log}\ p_{s}(x_{t}|x_{<t},\textrm{txt})=(1-\omega_{<t})\nabla_{x_{t}}\textrm{log}\ p(x_{t})
\displaystyle+\ (\omega_{<t}-\omega_{\textrm{txt}})\nabla_{x_{t}}\textrm{log}\ p(x_{t}|x_{<t})
\displaystyle+\ \omega_{\textrm{txt}}\nabla_{x_{t}}\textrm{log}\ p(x_{t}|x_{<t},\textrm{txt}),
where
x_{t}
is the current video chunk,
x_{<t}
are the past video chunks, txt is the text conditioning,
\omega_{<t}
is the context guidance weight and
\omega_{\textrm{txt}}
is the text guidance weight.
VJEPA-2
[
2
]
.
VJEPA-2 is a self-supervised video model which consists of a video encoder,
E_{\theta}
, and a predictor,
P_{\phi}
, trained with a Internet-scale data using a mask-denoising objective in the representation space.
3
Method
We use VJEPA-2 as reward signal during the generation process of the state-of-the-art image-to-video (I2V) and video-to-video (V2V) MAGI-1 model. To do so, we leverage the
surprise score
introduced in
[
2
]
and compare the generations of MAGI-1 with the VJEPA-2 predictions as follows:
S(x_{t}|x_{<t},\textrm{txt})=\textrm{sim}\big(P_{\phi}(E_{\theta}(x_{<t})),E_{\theta}(G_{\gamma}(x_{<t},\textrm{txt})\big),
(1)
where
E_{\theta}
and
P_{\phi}
are the VJEPA-2 encoder and predictor, respectively;
G_{\gamma}
is the generative model; and
sim
is the cosine similarity.
We start by altering the score function of MAGI-1 to include signals from VJEPA-2 as follows:
\displaystyle\nabla_{x_{t}}\textrm{log}\ p_{s}(x_{t}|x_{<t},\textrm{txt})=(1-\omega_{<t})\nabla_{x_{t}}\textrm{log}\ p(x_{t})
(2)
\displaystyle+\ (\omega_{<t}-\omega_{\textrm{txt}})\nabla_{x_{t}}\textrm{log}\ p(x_{t}|x_{<t})
\displaystyle+\ \omega_{\textrm{txt}}\nabla_{x_{t}}\textrm{log}\ p(x_{t}|x_{<t},\textrm{txt})
\displaystyle-\ \omega_{s}\nabla_{x_{t}}S(x_{t}|x_{<t},\textrm{txt})
Following Eq. (
2
), we collect 16 video samples per condition. Then, we apply Best of N (BoN) and select the video sample with the lowest average surprise score Eq. (
1
).
4
Results
We evaluate our approach on the PhysicsIQ dataset and report the results in Table
1
. By leveraging both VJEPA-2’s surprise score as guidance and VJEPA-2 signal to select BoN, our method boosts vanilla sampling of MAGI-1 model and reaches a new state-of-the-art result of
62.64
and
36.86
on V2V and I2V, respectively.
Table 1
:
Evaluation on
PhysicsIQ
PhysicsIQ Score (
\uparrow
)
V2V Generation
Lumiere
[
3
]
23.00
VideoPoet
[
6
]
29.50
MAGI-1
[
1
]
56.31
MAGI-1 + VJEPA-2 (ours)
62.64
(+6.33)
I2V Generation
Sora
[
5
]
10.00
Pika 1.0
[
8
]
13.00
SVD
[
4
]
14.80
Lumiere
[
3
]
19.00
VideoPoet
[
6
]
20.30
Wan2.1
[
10
]
20.89
Runway Gen 3
[
9
]
22.80
CogVideoX
[
11
]
26.22
MAGI-1
[
1
]
30.23
MAGI-1 + VJEPA-2 (ours)
36.86