The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang, *Equal Contribution, Engineer Advisor, Project Leader https://thematrix1999.github.io/ · 2024
Read The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control by Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang, *Equal Contribution, Engineer Advisor, Project Leader https://thematrix1999.github.io/ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03568.txt.
Description
Feng and nine co-authors present The Matrix, a 2024 system that generates endless, controllable video worlds in real time. It streams realistic 720p scenes without cuts, in first or third person, at 16 frames per second. Training mixed supervised data from big-budget games like Forza Horizon 5 and Cyberpunk 2077 with large amounts of unlabeled real footage, such as Tokyo street video. Users can roam deserts, grassland, water, and cities for hour-long stretches.
The system generalized to scenes it never trained on, like a family car driving through an office. The authors argue game data can teach world models that transfer to real settings where filming continuous motion is impractical. They pitch it as a step toward simulators for robotics and training.
Catalog record
| CREATOR | Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, Hongyang Zhang, *Equal Contribution, Engineer Advisor, Project Leader https://thematrix1999.github.io/ |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03568