Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Efstathios Karypidis 1,3 Spyros Gidaris 2 Nikos Komodakis 1,4,5, 1 Archimedes, Athena Research Center, Greece 2 valeo.ai, 3 National Technical University of Athens 4 University of Crete 5 IACM-Forth · 2026

Read Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction by Efstathios Karypidis 1,3 Spyros Gidaris 2 Nikos Komodakis 1,4,5, 1 Archimedes, Athena Research Center, Greece 2 valeo.ai, 3 National Technical University of Athens 4 University of Crete 5 IACM-Forth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11707.txt.

Introduction

Karypidis, Gidaris, and Komodakis present Re2Pix, a two-stage video forecast tool that predicts scene meaning before pixels. Rather than RGB frames, it first forecasts scene structure in a frozen vision foundation model's space. It then feeds those representations to a latent diffusion model to render realistic frames. Two training strategies, nested dropout and mixed guidance, counter train-test mismatch between ground-truth and forecast representations. On driving benchmarks, the meaning-first design improves temporal meaning consistency, image appeal, and training speed over strong diffusion baselines.

Catalog record

CREATOREfstathios Karypidis 1,3 Spyros Gidaris 2 Nikos Komodakis 1,4,5, 1 Archimedes, Athena Research Center, Greece 2 valeo.ai, 3 National Technical University of Athens 4 University of Crete 5 IACM-Forth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55003
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11707

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction Efstathios Karypidis 1,3 Spyros Gidaris 2 Nikos Komodakis 1,4,5, 1 Archimedes, Athena Research Center, Greece 2 valeo.ai, 3 National Technical University of Athens 4 University of Crete 5 IACM-Forth Abstract Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix , a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train–test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency compared to strong diffusion baselines. We provide the implementation code at https://github.com/Sta8is/Re2Pix . 1 Introduction Video prediction plays a central role in autonomous systems, where anticipating how a scene will evolve is essential for long-horizon reasoning and decision making [ 16 , 22 , 18 ] . In driving scenarios, the ability to accurately forecast how visual scenes evolve, from the motion of vehicles and pedestrians to the subtleties of lighting and occlusions, is not merely a perceptual convenience but a prerequisite for safe planning [ 78 , 19 , 68 ] . Yet learning such predictive models from raw video requires simultaneous mastery of high-level semantics (what objects are present and how they interact) and photorealistic details (how the scene appears) across temporal scales, a challenge that remains largely unsolved. Figure 1 : Overview of the proposed Re2Pix hierarchical framework during inference. In Stage 1, semantic features h_{1:M} of the context frames are extracted from a vision foundation model (VFM) encoder E_{h} and fed into a masked feature transformer to autoregressively predict (in frame-wise way) future semantic VFM features \hat{h}_{M+1:K} . In Stage 2, both the past \hat{h}_{1:M} and predicted features \hat{h}_{M+1:K} condition the diffusion transformer G_{z} to generate future VAE latents \hat{z}_{M+1:K} , which are then decoded into RGB future frames. Most modern approaches adopt an end-to-end paradigm: future frames are predicted directly in the latent space of a VAE, typically using diffusion models [ 28 , 54 ] . While effective, this paradigm suffers from a fundamental limitation: semantic structure and fine-grained visual details are deeply entangled within the same latent representation. Consequently, the model must simultaneously infer scene dynamics and render photorealistic appearance, often leading to temporal semantic inconsistencies such as object identity drift, structural degradation, or flickering artifacts. More fundamentally, this entanglement slows convergence, increases data requirements, and makes it difficult to reason about or control each component independently. Recent work has attempted to inject semantic structure into diffusion models by aligning intermediate features with pretrained representations via auxiliary distillation objectives [ 89 , 91 ] . While such alignment can guide representation learning, the diffusion model still performs forecasting and rendering within a single latent space, leaving their roles implicitly coupled. We ask a different question: can semantic forecasting and visual synthesis be explicitly separated, while still enabling coherent generative prediction? Our primary contribution is to introduce a hierarchical framework that decomposes video prediction into two interacting but distinct stages. First, we forecast future scene structure in the feature space of a frozen vision foundation model (VFM). These representations capture high-level semantics while abstracting away low-level appearance. Second, we condition a latent diffusion model on the predicted representations to render photorealistic frames. This separation yields a structured predictive pipeline: the first stage models temporal dynamics in a representation space, while the second stage specializes in appearance synthesis conditioned on forecasted structure. This clean separation, however, introduces a critical challenge. During training, the diffusion model has access to clean, ground-truth semantic features from future frames. At inference, it must instead rely on autoregressively predicted features, which inevitably accumulate errors. Naively supervising the generator solely on clean semantics creates a severe train–test mismatch: the model overfits to near-perfect conditioning signals and degrades sharply—often producing blurry or incoherent outputs—when exposed to noisy, imperfect forecasts. We demonstrate that bridging this distribution shift is essential for effective hierarchical video prediction. Our second, technical contribution lies in modeling this bridge between semantic forecasting and generative synthesis. We adopt an early fusion strategy that token-wise merges VFM features with VAE latents at the input level, providing stable conditioning without increasing token count. To mitigate the conditioning mismatch, we introduce two complementary strategies. First, nested dropout stochastically truncates feature channels during training, encouraging robustness to errors in the forecasted representation. Second, mixed supervision exposes the generator to both ground-truth and predicted features (90/10 mixture), directly regularizing against over-reliance on idealized semantics. Together, these mechanisms enable the diffusion model to operate reliably under imperfect, autoregressively predicted features. Concretely, our framework, Re2Pix , extracts semantic representations using a frozen VFM encoder such as DINOv2 [ 53 ] , and trains a lightweight masked Transformer to autoregressively predict future features in this space. A latent video diffusion model, implemented with a DiT backbone [ 54 ] in the VAE latent space [ 74 ] , is then conditioned directly on the forecasted semantic features to render future frames. In contrast to alignment-based approaches such as REPA [ 89 ] and VideoREPA [ 91 ] , which encourage feature similarity between diffusion and semantic spaces, we treat forecasted VFM representations as an explicit intermediate generative variable. Our main contributions are: • We introduce Re2Pix , a hierarchical semantic-to-pixel framework that separates video forecasting into semantic representation prediction and semantics-driven visual generation. To our knowledge, we are the first to demonstrate that VFM feature prediction can effectively guide hierarchical video diffusion. • We propose two robust conditioning strategies— nested dropout and mixed supervision —to close the train-test gap and improve robustness to imperfect, autoregressively generated features. • Extensive results demonstrate that Re2Pix achieves substantial improvements over strong baselines in temporal semantic fidelity and perceptual quality, while significantly accelerating training convergence (up to 7\times for generation and 14\times for segmentation metrics). 2 Related Work Video Generation and Prediction. Video prediction has evolved from autoregressive models operating directly in pixel space [ 50 , 71 , 15 , 44 , 77 , 20 ] to hierarchical and structured approaches that model temporal dynamics more effectively [ 75 , 49 , 82 , 30 , 84 ] . Transformer-based architectures have further advanced the field by leveraging autoregressive or masked modeling objectives to capture long-range temporal dependencies [ 88 , 87 , 23 , 76 ] . State-of-the-art video prediction systems typically operate in the latent space of a learned variational autoencoder [ 59 , 41 , 74 ] , where generative models predict future latent codes rather than raw pixels [ 23 , 87 , 88 , 19 , 8 ] .

Read the complete text