LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

Lucas Maes* Quentin Le Lidec* Damien Scieur Yann LeCun Randall Balestriero, [8pt] Mila & Université de Montréal New York University Samsung SAIL Brown University · 2026

Read LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels by Lucas Maes* Quentin Le Lidec* Damien Scieur Yann LeCun Randall Balestriero, [8pt] Mila & Université de Montréal New York University Samsung SAIL Brown University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19312.txt.

Introduction

Maes and co-authors build a world model that learns from raw pixels with just two loss terms. Prior joint-embedding models need many losses and tricks to keep features from collapsing. Their LeWorldModel pairs a next-step prediction loss with a rule that keeps features Gaussian. One tunable loss weight replaces six in the prior method. With about 15 million weights, it trains on one GPU in a few hours. Planning runs up to 48 times faster than large-model rivals. Probes show its features encode real physical traits.

Catalog record

CREATORLucas Maes* Quentin Le Lidec* Damien Scieur Yann LeCun Randall Balestriero, [8pt] Mila & Université de Montréal New York University Samsung SAIL Brown University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74031
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19312

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels Lucas Maes* Quentin Le Lidec* Damien Scieur Yann LeCun Randall Balestriero, [8pt] Mila & Université de Montréal New York University Samsung SAIL Brown University Abstract Joint Embedding Predictive Architectures (JEPAs) offer a compelling framework for learning world models in compact latent spaces, yet existing methods remain fragile, relying on complex multi-term losses, exponential moving averages, pre-trained encoders, or auxiliary supervision to avoid representation collapse. In this work, we introduce LeWorldModel (LeWM), the first JEPA that trains stably end-to-end from raw pixels using only two loss terms: a next-embedding prediction loss and a regularizer enforcing Gaussian-distributed latent embeddings. This reduces tunable loss hyperparameters from six to one compared to the only existing end-to-end alternative. With 15M parameters trainable on a single GPU in a few hours, LeWM plans up to 48 × 48\times faster than foundation-model-based world models while remaining competitive across diverse 2D and 3D control tasks. Beyond control, we show that LeWM’s latent space encodes meaningful physical structure through probing of physical quantities. Surprise evaluation confirms that the model reliably detects physically implausible events. † † footnotetext: * Equal contribution. Correspondence to [email protected] Website Code Figure 1 : LeWorldModel Training Pipeline. Given frame observations {\bm{o}}_{1:T} and actions {\bm{a}}_{1:T} , the encoder maps frames into low-dimensional latent representations {\bm{z}}_{1:T} . The predictor models the environment dynamics by autoregressively predicting the next latent state {\bm{z}}_{t+1} from the current latent state {\bm{z}}_{t} and action {\bm{a}}_{t} . The encoder and predictor are jointly optimized using a mean-squared error (MSE) prediction loss. LeWM does not rely on any training heuristics, such as stop-gradient, exponential moving averages, or pre-trained representations. To prevent trivial collapse, the SIGReg regularization term enforces Gaussian-distributed latent embeddings, promoting feature diversity. More specifically, latent embeddings are projected onto multiple random directions, and a normality test is applied to each one-dimensional projection. Aggregating these statistics encourages the full embedding distribution to match an isotropic Gaussian. 1 Introduction Figure 2 : Characteristics of latent world model approaches. Methods are grouped by training paradigm. End-to-end methods (PLDM) learn both the encoder and predictor jointly from pixels without relying on pre-trained representations or heuristic tricks such as stop-gradient or exponential moving averages, but require many hyperparameters and lack formal collapse guarantees. Foundation-based methods (DINO-WM) avoid collapse by freezing a pre-trained foundation vision encoder, forgoing end-to-end learning. Task-specific methods (Dreamer, TD-MPC) require reward signals or privileged state access during training. LeWM addresses the limitations of each category: it is end-to-end, task-agnostic, pixel-based, reconstruction- and reward-free, and requires only a single hyperparameter with provable anti-collapse guarantees. A central goal of artificial intelligence is to develop agents that acquire skills across diverse tasks and environments using a single, unified learning paradigm—one that operates directly from sensory inputs of its surroundings–without hand-engineered state representations or domain-specific calibration. Vision is ideally suited for this aim: cameras are inexpensive and scalable, and learning from pixels enables fully end-to-end training from raw sensory input to action [ 35 ] . World Models (WMs) are a powerful family of methods [ 22 ] that learn to predict the consequences of actions in the environment. When successful, WMs allows agents to plan and to improve themselves solely form their model of the world, i.e., in imagination space. This is particularly valuable in the offline setting, where agents must learn from fixed datasets without environment interaction—leveraging the model to generate synthetic experience and evaluate counterfactual action sequences [ 38 , 26 ] . A recent popular approach for learning world models is the Joint Embedding Predictive Architecture (JEPA) [ 34 ] . Instead of attempting to model every aspect of the environment, JEPA focuses on capturing the most relevant features needed to predict future states. Concretely, JEPA learns to encode observations into a compact, low-dimensional latent space and models temporal dynamics by predicting the latent representation of future observations. However, despite their conceptual simplicity, existing JEPA methods are highly prone to collapse. In this failure mode, the model maps all inputs to nearly identical representations to trivially satisfy the temporal prediction objective leading to unusable representations. Preventing collapse is therefore one of the central challenges in training JEPA models. Many influential works have proposed methods to address this issue. Yet, these approaches typically rely on heuristic regularization, multi-objective loss functions, external sources of information, or architectural simplifications such as pre-trained encoders. In practice, these strategies often introduce additional instability or significantly increase training complexity. To overcome these limitations, we propose LeWorldModel (LeWM), the first method to learn a stable JEPA end-to-end from raw pixels without heuristic, principled, and simple (cf. Fig 2 ). Furthermore, LeWM can be trained on a single GPU, lowering the barrier to entry for research. We evaluate LeWM across a diverse set of manipulation, navigation, and locomotion tasks in both 2D and 3D environments. In addition, we probe its intuitive physical understanding through targeted probing and surprise-quantification evaluations in latent space. Overall, our key findings and contributions are: • We propose an end-to-end JEPA method for learning a latent world model from raw pixels on a single GPU. The method relies on a simple and stable two-term objective that remains robust across architectures and hyperparameter choices, while enabling efficient logarithmic-time hyperparameter search. • LeWM achieves strong control performance across diverse 2D and 3D tasks with a compact 15M-parameter model, surpassing existing end-to-end JEPA-based approach while remaining competitive with foundation-model-based world models at substantially lower cost, enabling planning up to 48\times faster. • We evaluate physical understanding in the latent space through probing of physical quantities and a violation-of-expectation test for detecting unphysical trajectories. 2 Related Work Figure 3 : Planning time and performance under fixed compute. Left: Planning time comparison averaged over 50 runs. Encoding observations with \sim 200\times fewer tokens than DINO-WM allows LeWM to achieve planning speeds comparable to PLDM while being up to \sim 50\times faster than DINO-WM. Center–Right: Planning performance under the same computational budget (fixed FLOPs). LeWM significantly outperforms DINO-WM on Push-T (center) and OGBench-Cube (right). See App. D for planning setup details. World Models aim to learn predictive models of environment dynamics from data, enabling agents to reason about future states in imagination. A prominent class of WMs consists of generative approaches that explicitly model environment dynamics in pixel space. These action-conditioned generative models act as learned simulators by producing future observations conditioned on past states and actions. Generative world models have been successfully applied to simulate existing game-like environments. For example, IRIS [ 38 ] , DIAMOND [ 1 ] , \Delta -IRIS [ 39 ] , OASIS [ 14 ] , and DreamerV4 [ 26 ] model environments such as Minecraft, Counter-Strike, and Crafter, improving policy sample efficiency in reinforcement learning. Other methods generate entirely new interactive simulators, e.g., Genie [ 12 ] and HunyuanWorld [ 30 ] , while learned simulators have also been applied to robot policy evaluation [ 47 ] . Importantly, many generative WMs assume access to datasets containing reward signals, enabling joint modeling of dynamics and value-relevant information for downstream reinforcement learning. In contrast, we focus on the reward-free setting, corresponding to the setup considered in the JEPA line of work, which aims at learning generic, task-agnostic world models from

Read the complete text