Zero-shot Synthetic Video Realism Enhancement via Structure-aware Denoising

Yifan Wang, Liya Ji, Zhanghan Ke, Harry Yang, Sernam Lim, Qifeng Chen · 2025

Read Zero-shot Synthetic Video Realism Enhancement via Structure-aware Denoising by Yifan Wang, Liya Ji, Zhanghan Ke, Harry Yang, Sernam Lim, Qifeng Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14719.txt.

Introduction

Yifan Wang, Liya Ji, Zhanghan Ke, Harry Yang, Ser-Nam Lim, and Qifeng Chen propose a zero-shot method for making synthetic videos look photorealistic. Simulators can generate endless driving or training footage, but its rendered look betrays it, limiting its use for training real systems. Their framework re-renders such video through a foundation diffusion model with no further fine-tuning, while guarding the original structure at several levels in both space and time.

Guidance comes from structure-aware information estimated from the synthetic video itself by an auxiliary model: depth maps, semantic maps, and edge maps that condition the denoising. Keeping those anchors ensures the enhanced video matches the original in structure and meaning, not just in gloss. The authors present the approach as simple, general, and powerful for simulator pipelines.

Catalog record

CREATORYifan Wang, Liya Ji, Zhanghan Ke, Harry Yang, Sernam Lim, Qifeng Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14719

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-shot Synthetic Video Realism Enhancement via Structure-aware Denoising Yifan Wang, Liya Ji, Zhanghan Ke, Harry Yang, Sernam Lim, Qifeng Chen Abstract We propose an approach to enhancing synthetic video realism, which can re-render synthetic videos from a simulator in photorealistic fashion. Our realism enhancement approach is a zero-shot framework that focuses on preserving the multi-level structures from synthetic videos into the enhanced one in both spatial and temporal domains, built upon a diffusion video foundational model without further fine-tuning. Specifically, we incorporate an effective modification to have the generation/denoising process conditioned on estimated structure-aware information from the synthetic video, such as depth maps, semantic maps, and edge maps, by an auxiliary model, rather than extracting the information from a simulator. This guidance ensures that the enhanced videos are consistent with the original synthetic video at both the structural and semantic levels. Our approach is a simple yet general and powerful approach to enhancing synthetic video realism: we show that our approach outperforms existing baselines in structural consistency with the original video while maintaining state-of-the-art photorealism quality in our experiments. 3 3 footnotetext: Project page: https://wyf0824.github.io/Video_Realism_Enhancement/ 1 Introduction Training models for autonomous vehicles requires vast amounts of data, particularly for rare long-tail scenarios. However, the collection and labeling of real-world video data are costly and often fail to capture these critical events. To address this challenge, researchers have turned to synthetic data generated from simulators of the environment ( 2 ) , which offers a scalable and controllable means of producing the data required to accelerate research and development in autonomous vehicles ( 14 ; 51 ) . A key challenge here is the domain gap between synthetic and real-world data that can introduce biases that adversely affect model performance. To alleviate this issue, we focus on enhancing the realism of synthetic videos in this work. Initial efforts to tackle this issue used video-to-video translation techniques ( 39 ; 52 ) , which converted simulated driving sequences (e.g., GTA V ( 32 ) ) into realistic counterparts using GAN-based methods ( 11 ) . These prior works demonstrated the feasibility of rendering more realistic driving footage, but they often suffered from limitations such as low resolution, poor temporal consistency, and semantic inconsistency with the original synthetic video. Recent advancements in diffusion models ( 34 ; 26 ) and the availability of large-scale datasets ( 6 ; 42 ) have significantly enhanced generative fidelity, yielding sharper, high-resolution, and more realistic outputs. Currently, most state-of-the-art pipelines leverage controllable generation strategies, such as ControlNet ( 47 ) , where detailed simulator-derived features—including Canny edges, depth maps, semantic segmentation ( 50 ) , bird’s-eye view (BEV) layouts ( 41 ; 37 ; 16 ) , 3D boxes ( 31 ) , and LiDAR ( 31 ) projections—are utilized as conditional inputs to guide the synthesis of realistic images or videos. These approaches represent the cutting edge of controllable synthetic-to-real data generation. Despite their success, control-based methods often overlook the visual content of the source synthetic video, including aspects such as color and lighting. By regenerating appearance solely from a text prompt, they often struggle to accurately reproduce safety-critical semantic details, like the precise hue of traffic lights or the appearance of road signs. Additionally, previous GAN-based methods are constrained by limited stylistic diversity and controllability; the model can only generate generic styles it has encountered during training, rather than authentically enhancing the source video towards a desired, diverse aesthetic that aligns with real-world conditions. Figure 1: Enhanced videos by our structure-aware denoising method from rendered videos. Our approach could generate the videos with structural consistency and state-of-the-art photorealism quality, especially for the small objects in autonomous driving scenarios, such as roadside signals and traffic lights. Therefore, the primary challenge is to develop a method that enhances photorealism with realistic styles while preserving the essential semantic and structural content of the original video. To achieve this, we propose a framework that leverages the source video’s structural information as a rich foundation for controllable enhancement. Inspired by recent advancements in video editing and generation, we introduce a new inversion-and-generation paradigm through a simple zero-shot adaptation of a state-of-the-art controllable video generation model. Specifically, drawing from the technique DDIM Inversion ( 35 ) , we first invert the source video from the simulator by progressively adding noise in a deterministic manner, mapping it to an initial latent representation. This inversion process computes a specific initial noise latent that is structurally tied to the original video’s content and motion. By initiating the denoising process from this content-aware latent representation—rather than from random noise—we can anchor the generation to the source semantics. Subsequently, we apply Classifier-Free Guidance (CFG) to selectively modify the visual style during the structure-aware denoising stage. This approach enables us to eliminate the unrealistic, computer-generated textures commonly found in simulator outputs while steering the overall style towards the real-world aesthetic learned by the model. Our method significantly enhances photorealism and temporal coherence without compromising the original semantic information. The main contributions of this work are summarized as follows: • To the best of our knowledge, we introduce the first zero-shot structure-aware denoising framework that adapts a pre-trained video diffusion model for the photorealistic enhancement of synthetic videos, eliminating the need for domain-specific post-training. • We propose a simple yet effective zero-shot inversion-and-generation framework that ensures semantic preservation while enhancing realism through the generation process. • We have developed an evaluation protocol to quantify the semantic and temporal consistency of small and safety-critical objects (e.g., traffic lights, road signs), ensuring a rigorous assessment of semantic fidelity. 2 Related Work 2.1 Visual Synthesis for Autonomous Driving In autonomous driving, large-scale data collection is expensive and time-consuming. Obtaining visual contents via simulators ( 32 ; 8 ) is an easier way to build a physical world based on the various annotations. However, the domain gap between the simulator-rendered videos and the real-world contents is challenging. 33 , 24 , and 25 trained an image enhancement network to enhance the photorealism of rendered images under the supervision of the adversarial objective. 39 , with its follow-up works 52 ; 22 extended the generative adversarial learning framework to the video-to-video synthesis, translating the semantic maps of the synthetic videos to the photorealistic outputs with temporally coherent. With the powerful generative ability of diffusion models, 49 and 27 utilized the diffusion-based model conditioning on the synthetic images to fill the domain gaps and improve structural fidelity. However, when the controllable diffusion-based framework extends to the video scenario, the temporal consistency remains challenging. One direction 10 ; 42 ; 50 employs a two-stage pipeline: first generate a photorealistic initial image, and then extrapolate subsequent frames with a video prediction model. However, this decoupling of content generation from temporal dynamics introduces significant limitations. These models often lack fine-grained control over vehicle kinematics, such as speed, and struggle to plausibly synthesize content for disoccluded regions. Furthermore, compounding prediction errors typically restricts the output to short, temporally limited sequences. Another direction is to train a world model involving physical knowledge. 40 is the first world models approach for autonomous driving and generated the realistic videos on multiple conditions, such as HD maps, 3D box, and actions. 2

Read the complete text