PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusion

Avinash Paliwal, Xilong Zhou, Andrii Tsarov, Nima Kalantari · 2024

Read PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusion by Avinash Paliwal, Xilong Zhou, Andrii Tsarov, Nima Kalantari (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04827.txt.

Introduction

Avinash Paliwal and co-authors conjure a full three-dimensional room from a single photograph. PanoDreamer first stretches the image into a coherent panorama with matching depth, then lifts it into 3D. Sequential generation methods often drift and contradict themselves as they circle the scene.

The authors instead cast the panorama and its depth as two linked puzzles, solved in turns. Diffusion models supply the imagination for unseen views, while the joint solving keeps all views in agreement. Small hidden patches are filled in before the scene is lifted into space. The result beats earlier techniques on consistency and overall quality, a step toward instant virtual sets for games, real estate, and film.

Catalog record

CREATORAvinash Paliwal, Xilong Zhou, Andrii Tsarov, Nima Kalantari
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04827

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusion Avinash Paliwal, Xilong Zhou, Andrii Tsarov, Nima Kalantari Abstract . In this paper, we present PanoDreamer, a novel method for producing a coherent 360° 3D scene from a single input image. Unlike existing methods that generate the scene sequentially, we frame the problem as single-image panorama and depth estimation. Once the coherent panoramic image and its corresponding depth are obtained, the scene can be reconstructed by inpainting the small occluded regions and projecting them into 3D space. Our key contribution is formulating single-image panorama and depth estimation as two optimization tasks and introducing alternating minimization strategies to effectively solve their objectives. We demonstrate that our approach outperforms existing techniques in single-image 360° 3D scene reconstruction in terms of consistency and overall quality 1 1 1 people.engr.tamu.edu/nimak/Papers/PanoDreamer . 1483 Conference: SIGGRAPH Asia 2025 Conference Papers; December 15–18, 2025; Hong Kong, Hong Kong SIGGRAPH Asia 2025 Conference Papers (SA Conference Papers ’25), December 15–18, 2025, Hong Kong, Hong Kong DOI: 10.1145/3757377.3763883 ISBN: 979-8-4007-2137-3/2025/12 CCS: Computing methodologies Image-based rendering Figure 1. We introduce a novel method for 360° 3D scene synthesis from a single image. Our approach generates a panorama and its corresponding depth in a coherent manner, addressing limitations in existing state-of-the-art methods such as LucidDreamer ( Chung et al., 2023 ) and WonderJourney ( Yu et al., 2024b ) . These methods sequentially add details by following a generation trajectory, often resulting in visible seams when looping back to the input image. In contrast, our approach ensures consistency throughout the entire 360° scene. The yellow bars show the regions corresponding to the input in each result. † † cc-license: by 1. Introduction Generating immersive and realistic 3D scenes from a single input image has emerged as one of the important topics in computer vision/graphics, driven by its broad applications including virtual/augmented reality (VR/AR) and gaming. While early algorithms ( Niklaus et al., 2019 ; Shih et al., 2020 ; Kopf et al., 2020 ; Jampani et al., 2021 ; Zhou et al., 2016 ; Srinivasan et al., 2017 ; Li and Kalantari, 2020 ; Tucker and Snavely, 2020 ) have achieved high-quality results, they are generally limited to synthesizing novel views with only minor deviation from the input camera position. Consequently, these techniques cannot reconstruct a full 360° scene, which is the primary goal of our work. With the introduction of diffusion models, the more recent approaches have focused on utilizing these powerful models for 3D scene reconstruction. Specifically, several methods ( Li et al., 2024 ; Zhou et al., 2025 ; Zhou et al., 2024 ) propose various ways to generate 3D scenes from input text prompts. These methods first generate entire panorama from text prompt using pretrained text-to-panorama diffusion models (DMs) and then lift it to 3D. Unfortunately, these approaches are fully generative and do not have a mechanism for reconstructing a 3D scene which is also consistent with a single input image. Several methods ( Chung et al., 2023 ; Shriram et al., 2024 ; Yu et al., 2024b ; Ouyang et al., 2023 ; Höllein et al., 2023 ; Zhang et al., 2024 ) specifically address the problem of 3D scene reconstruction from a single image. Starting from the input image, these methods typically project it into 3D space, render it from a novel view, and then inpaint the missing regions using a diffusion model. They repeat this process for a series of cameras along a specific path to reconstruct the complete 3D scene. However, a major limitation of these approaches is that, due to the progressive nature of the scene building, they often fail to synthesize coherent 360° scenes, i.e., the start and end of the 360° scenes are contextually different. In this work, we propose a novel framework, coined PanoDreamer, for generating a coherent 360° 3D scene from a single input image. Departing from the existing methods, which generate the 3D scene one image at a time, we start by producing a coherent 360° panorama from the input image using standard pre-trained inpainting diffusion models. Inspired by MultiDiffusion ( Bar-Tal et al., 2023 ) , we formulate the problem as an optimization with two loss terms and propose an alternating minimization strategy to optimize the objective, resulting in a coherent and seamless panoramic image. The next stage of our approach involves estimating the depth of the panoramic image to project pixels into 3D space and reconstruct the 3D scene. While powerful monocular depth estimation methods ( Yang et al., 2024a ) exist, these techniques are typically optimized for specific resolutions and struggle to handle large panoramic images effectively. To address this problem, we formulate panoramic depth reconstruction as an optimization task, aiming to simultaneously produce a coherent panoramic depth map and a parametric function that aligns the range of monocular depth to the target depth. We propose an alternating minimization approach to efficiently solve this objective, resulting in a coherent and seamless panoramic depth map. Given the panoramic image and depth, we directly apply the approach of Shih et al. ( 2020 ) to construct a layered depth image (LDI) and inpaint the missing regions in each layer. Next, we build a 3D Gaussian splatting (3DGS) representation ( Kerbl et al., 2023 ) by initializing a set of Gaussians through the projection of LDI pixels into 3D space. We then optimize the 3DGS representation to sharpen details and obtain the final scene. We demonstrate that PanoDreamer can reconstruct consistent 360° 3D scenes from single input images that outperform existing methods. In summary, our work makes the following contributions: • We propose a novel framework for synthesizing a coherent 3D panoramic scene from a single image. • We formulate the problem of single-image panorama generation using an inpainting diffusion model as an optimization task and solve it using an alternating minimization strategy. • We frame the task of obtaining panoramic depth from existing monocular depth estimation methods as an optimization problem and propose an alternating minimization method to solve it. 2. Related Work 2.1. Panorama Generation Diffusion models (DMs) have shown promising results across various generative tasks. In particular, several approaches ( Bar-Tal et al., 2023 ; Lee et al., 2023 ; Deng et al., 2023 ; Ye et al., 2024 ; Li and Bansal, 2023 ; Frolov et al., 2024 ; Wang et al., 2024 ; Zhang et al., 2023 ) have proposed leveraging pretrained DMs to synthesize panoramic images. For example, DiffCollage ( Zhang et al., 2023 ) reconstructs complex factor graphs and aggregates intermediate output from DMs defined by nodes to generate a panorama. PanoGen ( Li and Bansal, 2023 ) utilizes latent diffusion models combined with recursive outpainting to create indoor panoramic images. MultiDiffusion ( Bar-Tal et al., 2023 ) frames the problem of panoramic image generation from pretrained DMs as an optimization process to produce globally consistent images. SynDiffusion ( Lee et al., 2023 ) builds on this idea and incorporates the LPIPS score ( Zhang et al., 2018 ) between neighboring denoised images into the optimization process. StitchDiffusion ( Wang et al., 2024 ) further proposes averaging the overlapping denoising predictions and fine-tuning a low-rank adaptation (LoRA) module ( Hu

Read the complete text