FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

Hugo Caselles-Dupré ∗, Mathis Koroglu ∗, Guillaume Jeanneret ∗, Arnaud Dapogny, Matthieu Cord · 2026

Read FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion by Hugo Caselles-Dupré ∗, Mathis Koroglu ∗, Guillaume Jeanneret ∗, Arnaud Dapogny, Matthieu Cord (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17555.txt.

Introduction

Caselles-Dupré and co-authors present FrescoDiffusion, a training-free method for 4K image-to-video work. Tiled denoising keeps fine detail but breaks the global layout over time. Their fix first renders a low-resolution video and lifts its latent path into a global prior. Each tile then fuses with that prior at every step through one least-squares rule. A spatial dial also limits where motion may occur. Tests beat tiled baselines on consistency.

Catalog record

CREATORHugo Caselles-Dupré ∗, Mathis Koroglu ∗, Guillaume Jeanneret ∗, Arnaud Dapogny, Matthieu Cord
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59787
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17555

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion Hugo Caselles-Dupré ∗, Mathis Koroglu ∗, Guillaume Jeanneret ∗, Arnaud Dapogny, Matthieu Cord Abstract Diffusion-based image-to-video (I2V) models are increasingly effective, yet they struggle to scale to ultra-high-resolution inputs (e.g., 4K). Generating videos at the model’s native resolution often loses fine-grained structure, whereas high-resolution tiled denoising preserves local detail but breaks global layout consistency. This failure mode is particularly severe in the fresco animation setting: monumental artworks containing many distinct characters, objects, and semantically different sub-scenes that must remain spatially coherent over time. We introduce FrescoDiffusion, a training-free method for coherent large-format I2V generation from a single complex image. The key idea is to augment tiled denoising with a precomputed latent prior: we first generate a low-resolution video at the underlying model resolution and upsample its latent trajectory to obtain a global reference that captures long-range temporal and spatial structure. For 4K generation, we compute per-tile noise predictions and fuse them with this reference at every diffusion timestep by minimizing a single weighted least-squares objective in model-output space. The objective combines a standard tile-merging criterion with our regularization term, yielding a closed-form fusion update that strengthens global coherence while retaining fine detail. We additionally provide a spatial regularization variable that enables region-level control over where motion is allowed. Experiments on the VBench-I2V dataset and our proposed fresco I2V dataset show improved global consistency and fidelity over tiled baselines, while being computationally efficient. Our regularization enables explicit controllability of the trade-off between creativity and consistency. Project website: https://f2v.pages.dev/ * * footnotetext: These authors contributed equally. 1 Introduction Figure 1 : From a single ultra-high-definition image ( 3500\times 3500 ), FrescoDiffusion animates it at the same resolution. We show three frames from the generated video. The red box marks a fixed spatial region tracked across time, illustrating motion temporal consistency and fine-detail preservation. Recently, media has shifted from images to videos, and now to ultra high-definition videos, which means resolutions around 4K ( 3840\times 2160 ). In many applications, especially in the creative industries, such as cinema, animation, projections and art, there is a need for tools that can process and generate 4K content with sufficient detail. One example of this is fresco animation, which is the focus of this work. In contrast to standard images, we define a fresco as a large-scale image containing multiple scenes ( i.e . , up to hundreds) that blend seamlessly into a coherent visual. To achieve animation of any picture, diffusion models [ 31 , 28 , 17 , 2 , 4 ] have become the dominant approach for video synthesis, producing strong results in both text-to-video (T2V) and image-to-video (I2V) applications. While T2V models generate videos that faithfully follow an input prompt, I2V models follow the same paradigm but start with a first frame provided by the user. Despite the progress in I2V, the most advanced models [ 33 , 32 , 14 ] operate under constraints that do not gracefully allow large-scale I2V. On the one hand, using regular video diffusion models on high-resolution images resized to their native spatial regime yields insufficient detail, as the input is too large and complex to be represented by the standard resolution of video diffusion models. On the other hand, memory and compute grow steeply with spatial and temporal dimensions, preventing the use of those models on 4K images. Scaling image-to-video to very large videos containing many distinct regions remains largely unexplored in the literature. Existing work tries to address both issues, but the fresco setting makes it especially hard: simple tiling [ 3 , 24 ] introduces cross-tile drift and visible seams, and post-hoc video super-resolution [ 15 , 23 ] cannot create scene-level content that never existed at the model’s native scale. A key difficulty specific to fresco animation is that different regions of the image play fundamentally different roles over time. Frescoes typically contain numerous loosely coupled scenes and characters: some regions are visually static and primarily define architectural or pictorial context, while others are semantically active and expected to exhibit motion. This observation motivates a region-aware treatment of global coherence. In this paper, we introduce FrescoDiffusion to achieve 4K-I2V up to the fresco setting, as illustrated in Fig. 1 . Although animating the full-resolution fresco is not feasible directly, video models can animate a resized thumbnail of the same image at the model’s native scale. So, our core idea is to take advantage of this feature and use the resized thumbnail as a prior to guide the high-resolution denoising of the initial fresco with a novel loss. Our loss admits a closed-form solution that allows control of the balance between creativity (tiled denoising) and prior alignment. Moreover, we introduce a tool that adapts the generation process in active zones and the background differently with a mask to deal with frescoes specifically. It allows the generative model enough flexibility to animate the active zones while keeping the background close to the prior. In our experiments, we show the superiority, both in terms of quality and computation time, of FrescoDiffusion compared to other tiled-denoising methods through extensive qualitative and quantitative evaluation, and a user preference study, on our novel dataset FrescoArchive, and standard VBench 4K dataset [ 19 ] . Videos generated with FrescoDiffusion are provided in the supplementary, with an interactive webpage for visualizing them conveniently. To summarize, our contributions are as follows: 1. We introduce FrescoDiffusion, a novel training-free approach for 4K I2V generation that outperforms existing baselines in performance and efficiency. 2. To demonstrate its application in an artistic domain, we propose a dynamic prior-strength schedule together with a spatial gating mechanism that enables controlled trade-offs between creativity and prior similarity, especially useful in the fresco-to-video task. 3. We propose FrescoArchive, a new I2V dataset composed of complex multi-scene images for fresco-to-video evaluation. To contribute further to 4K-I2V, we will make our algorithm open-source and release our dataset upon acceptance to promote further research. 2 Related Work Diffusion-based video generation. Early text-to-video systems such as Make-A-Video [ 31 ] and Imagen Video [ 17 ] generate short clips by cascading a base video diffusion model with spatial and temporal super-resolution modules. Subsequent work extends latent diffusion models [ 28 ] to videos, e.g. Video Latent Diffusion Models (Video LDM) [ 5 ] , which map videos to a compressed latent space for efficient high-resolution text-to-video generation, and Lumiere [ 2 ] , whose space–time U-Net jointly denoises all frames. Recent foundation-scale models such as Wan [ 33 ] , Mochi [ 32 ] and LTX-Video [ 14 ] push video latent diffusion to higher quality and longer durations. Image-to-video methods (I2V) are usually supported by all these recent models. All these methods, including commercial systems (such as Kling, Sora2, Gen4.5 or Veo3), operate uniquely on their native spatial resolutions (typically 480 – 1080 p) and video lengths of a few seconds. In contrast, we formulate a training-free approach that takes an existing image-to-video diffusion model and extends it beyond its native resolution, bringing additional creative control. Video super-resolution. Video super-resolution (VSR) seeks to reconstruct high-resolution videos from low-resolution inputs, typically with strong temporal coherence. While these methods [ 8 , 9 , 10 ,

Read the complete text