The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang, *Equal Contribution, Engineer Advisor, Project Leader https://thematrix1999.github.io/ · 2024

Read The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control by Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang, *Equal Contribution, Engineer Advisor, Project Leader https://thematrix1999.github.io/ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03568.txt.

Description

Feng and nine co-authors present The Matrix, a 2024 system that generates endless, controllable video worlds in real time. It streams realistic 720p scenes without cuts, in first or third person, at 16 frames per second. Training mixed supervised data from big-budget games like Forza Horizon 5 and Cyberpunk 2077 with large amounts of unlabeled real footage, such as Tokyo street video. Users can roam deserts, grassland, water, and cities for hour-long stretches.

The system generalized to scenes it never trained on, like a family car driving through an office. The authors argue game data can teach world models that transfer to real settings where filming continuous motion is impractical. They pitch it as a step toward simulators for robotics and training.

Catalog record

CREATORRuili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang, *Equal Contribution, Engineer Advisor, Project Leader https://thematrix1999.github.io/
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03568

Browse the full library · About the Tower of Babel