Latent Space Super-Resolution for Higher-Resolution Image Generation with Diffusion Models

Jinho Jeong, Sangmin Han, Jinwoo Kim, Seon Joo Kim · 2025

Read Latent Space Super-Resolution for Higher-Resolution Image Generation with Diffusion Models by Jinho Jeong, Sangmin Han, Jinwoo Kim, Seon Joo Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18446.txt.

Introduction

Four authors present a way to make sharper, larger images with diffusion models, in a 2025 study. Models trained at one size often warp shapes or repeat parts when asked to work past that size. So the team works in the model's inner latent space instead of on raw pixels.

Their method, named LSRNA, first enlarges the inner image in a way that keeps it well formed. It then adds noise region by region to restore fine detail that smoothing tends to erase. Tests across sizes and scores beat the best prior methods that lean on a low-grade guide image. Image makers gain a route to prints past 1K that keep both form and detail.

Catalog record

CREATORJinho Jeong, Sangmin Han, Jinwoo Kim, Seon Joo Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18446

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Latent Space Super-Resolution for Higher-Resolution Image Generation with Diffusion Models Jinho Jeong, Sangmin Han, Jinwoo Kim, Seon Joo Kim Abstract In this paper, we propose LSRNA, a novel framework for higher-resolution (exceeding 1K) image generation using diffusion models by leveraging super-resolution directly in the latent space. Existing diffusion models struggle with scaling beyond their training resolutions, often leading to structural distortions or content repetition. Reference-based methods address the issues by upsampling a low-resolution reference to guide higher-resolution generation. However, they face significant challenges: upsampling in latent space often causes manifold deviation, which degrades output quality. On the other hand, upsampling in RGB space tends to produce overly smoothed outputs. To overcome these limitations, LSRNA combines Latent space Super-Resolution (LSR) for manifold alignment and Region-wise Noise Addition (RNA) to enhance high-frequency details. Our extensive experiments demonstrate that integrating LSRNA outperforms state-of-the-art reference-based methods across various resolutions and metrics, while showing the critical role of latent space upsampling in preserving detail and sharpness. The code is available at https://github.com/3587jjh/LSRNA . Figure 1 : Comparisons of 16 \times image generation with and without LSRNA framework. Our proposed LSRNA framework improves reference-based higher-resolution image generation, enhancing detail and sharpness beyond the native resolution of SDXL [ 39 ] ( 1024^{2} ) while achieving faster generation speeds. 1 Introduction Figure 2 : Comparison of DemoFusion with different upsampling strategies. All methods are directly upsampled to 16 \times resolution. (a) Latent space bicubic upsampling causes manifold deviation, degrading output quality. (b) RGB space bicubic upsampling produces outputs with reduced detail and sharpness. (c) Our learned latent-space upsampling aligns the manifold, resulting in sharp and detailed outputs. Best viewed ZOOMED-IN. Diffusion models have rapidly become powerful tools in generative modeling, starting from foundational models like DDPM [ 22 ] and DDIM [ 53 ] . Latent Diffusion Models (LDMs) [ 43 ] further revolutionized the field by reducing computational costs through operating in a lower-dimensional latent space. These advancements have been applied to various tasks for image processing such as generation [ 38 , 46 , 41 ] , editing [ 36 , 20 , 3 , 27 ] , and super-resolution [ 47 , 23 ] . In parallel, with the rise of megapixel displays, demands for higher-resolution image generation (ranging from 2K to 8K) have grown drastically. However, existing diffusion models are limited to learning the distribution of fixed resolutions ( e.g . , 512^{2} for SD1.5), preventing them from generating images at unseen larger resolutions. Scaling diffusion models to larger resolutions presents several challenges. Training diffusion models from scratch [ 15 , 10 , 24 , 57 ] or fine-tuning [ 63 , 7 , 42 ] for higher resolutions demands extensive data and computational resources. Meanwhile, directly generating larger images using pretrained models leads to content repetition and structural distortions [ 1 , 19 ] . Alternatively, one can generate images at a pretrained resolution followed by super-resolution methods on RGB space. However, this approach tends to lack texture details and produce smoothed results [ 19 , 13 ] . In response, recent studies have increasingly focused on training-free approaches that leverage pretrained diffusion models. Among these, DemoFusion [ 13 ] stands out as a prominent method, laying the foundation for reference-based approaches [ 34 , 50 , 55 ] . By generating a reference latent (or image) at the original size and upsampling it to guide the higher-resolution generation process, these approaches have demonstrated robust performance in producing high-quality images. Nonetheless, what remains underexplored is the reference upsampling process, which makes it a key area for further investigation. In this paper, we explore the significance of upsampled reference in reference-based image generation. Our first observation is that, as shown in Figure 2 (a), the bicubic-upsampled reference latent appears visually unappealing, exhibiting artifacts that suggest a deviation from the target resolution’s manifold. This deviation poses a considerable problem as it interferes with the subsequent generation process by providing suboptimal guidance, preventing the output from properly aligning with the desired manifold. Existing reference-based methods with latent space upsampling [ 13 , 34 ] mitigate the issue by gradually increasing image resolution in stages ( i.e . , progressive upscaling). However, it leads to slower inference times and the potential accumulation of errors in the content. Given these limitations, we explored RGB space upsampling as an alternative, inspired by reference-based RGB upsampling methods [ 50 , 55 ] . This approach has the potential advantage of avoiding the artifacts observed in latent space upsampling, as RGB-based methods work directly in pixel space, where artifacts may be less pronounced. Specifically, we replace latent upsampling with RGB upsampling, where a reference latent is decoded into an RGB image, bicubic-upsampled, and then re-encoded back into the latent space. However, Figure 2 (b) shows that the RGB space upsampling produces smooth and less detailed outputs compared to the original latent upsampling. Based on these observations, we hypothesize that upsampling within the latent space plays a crucial role in preserving sharpness and detail essential for high-resolution image generation. However, the issue of manifold deviation caused by upsampling remains insufficiently addressed. To tackle this, we propose a novel approach: learning to map low-resolution latent representations onto the higher-resolution manifold. We introduce a lightweight Latent space Super-Resolution (LSR) module that operates directly within the latent space. Additionally, to facilitate the generation of high-frequency details in the subsequent generation process, we propose the Region-wise Noise Addition (RNA) module. Inspired by SDEdit [ 36 ] , which improves image quality through uniform noise injection followed by denoising, RNA adaptively injects Gaussian noise into specific regions of the upsampled reference latent. Figure 3 : Framework Comparison. (a) Existing latent upsampling framework rely on progressive upsampling to address manifold deviation. (b) Existing RGB upsampling framework can directly upsample (optionally progressively), but produce smooth output. (c) Our framework enables latent upsampling without progressive upscaling with much fewer denoising steps ( T_{c}<T ) while producing detailed outputs (RNA omitted for simplicity). LR, MR, HR: low/mid/high resolution; DM: Diffusion Model. Combining the manifold alignment capability of LSR with the detail generation ability of RNA, our framework, LSRNA , is the first latent upsampling framework to eliminate the need for progressive upscaling as illustrated in Figure 3 . Our high-quality guidance also allows for fewer subsequent denoising steps, enabling faster generation with minimal quality loss. We validate the effectiveness of LSRNA by incorporating it into the state-of-the-art latent upsampling-based method [ 13 ] , pushing the boundaries of high-resolution image generation. Furthermore, we demonstrate that latent upsampling is crucial for preserving sharpness and detail by applying LSRNA to a very recent RGB upsampling-based model [ 55 ] . 2 Related Works Latent Diffusion Models. Diffusion models, with foundations in DDPM [ 22 ] , DDIM [ 53 ] , and ADM [ 9 ] , have become a dominant approach for image generation. Latent Diffusion Models (LDMs) [ 43 ] further advance the field by shifting the diffusion process into the latent space, using a pretrained autoencoder to compress images. This approach allows for faster computations and lower memory usage, supporting a wide range of applications [ 20 , 16 , 45 , 44 , 3 , 59 , 37

Read the complete text