Semantics-Guided Generative Image Compression

Cheng-Lin Wu, Hyomin Choi, Ivan V. Bajić · 2025

Read Semantics-Guided Generative Image Compression by Cheng-Lin Wu, Hyomin Choi, Ivan V. Bajić (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24015.txt.

Introduction

Cheng-Lin Wu, Hyomin Choi, and Ivan Bajić lift the quality of generative image compression in a two thousand twenty-five paper. AI can now rebuild a good image from a compact description. Their start point is a past multimodal semantic approach called MISC.

Two new parts raise quality and cut cost. Maps that mark out the parts of a scene guide the decoder that rebuilds the image. A content-based step varies how many generation steps run, by how hard the image is. Set scores and scores based on how the image looks both went up. Encoding and decoding each ran more than a third faster. Next to the main codecs, their frame won on how alike the images look and on quality. Code and visual examples are available.

Catalog record

CREATORCheng-Lin Wu, Hyomin Choi, Ivan V. Bajić
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24015

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantics-Guided Generative Image Compression Cheng-Lin Wu, Hyomin Choi, Ivan V. Bajić Abstract Advancements in text-to-image generative AI with large multimodal models are spreading into the field of image compression, creating high-quality representation of images at extremely low bit rates. This work introduces novel components to the existing multimodal image semantic compression (MISC) approach, enhancing the quality of the generated images in terms of PSNR and perceptual metrics. The new components include semantic segmentation guidance for the generative decoder, as well as content-adaptive diffusion, which controls the number of diffusion steps based on image characteristics. The results show that our newly introduced methods significantly improve the baseline MISC model while also decreasing the complexity. As a result, both the encoding and decoding time are reduced by more than 36%. Moreover, the proposed compression framework outperforms mainstream codecs in terms of perceptual similarity and quality. The code and visual examples are available. 1 1 1 https://github.com/CrashedBboy/SGGIC † † address: ∗ Simon Fraser University, Burnaby, BC, Canada † InterDigital, Los Altos, CA, USA 1 Introduction In 2023, AI-generated images surged to over 15 billion, surpassing traditional stock photo libraries and constituting about a third of all photos ever uploaded to Instagram in just one year [ 1 ] . Models like DALL-E 2 [ 2 ] , Stable Diffusion [ 3 ] , and Midjourney enable users to create customized, high-quality visuals from simple textual prompts, democratizing creative expression and transforming industries by offering rapid prototyping, concept art, and specialized visuals. This unprecedented scale and flexibility highlight a significant shift in digital content creation, outpacing traditional photography in speed and versatility. With the exponential increase in images generated from semantic prompts, a shift toward storing images with structural semantics may emerge as a new paradigm in image coding. Semantic-based generative image compression stores images using key descriptive elements like scene and context instead of pixel-level details, enabling higher compression ratios by focusing on semantically relevant aspects. This approach prioritizes semantically significant content and allows flexible editing and dynamic content customization, making it a versatile and efficient paradigm for image storage and manipulation. However, semantic-based image compression faces challenges in maintaining fidelity due to the inherent ambiguity of semantics, as textual prompts often lack detailed elements like textures or lighting, and generative models struggle with precise feature placement [ 4 , 5 ] . Furthermore, the high computational cost of models such as Denoising Diffusion Probabilistic Models (DDPMs) [ 6 ] , which require significant iterative processing, makes them unsuitable for real-time encoding and decoding, posing a significant barrier to practical applications that require speed and efficiency [ 7 ] . Our work builds upon the multimodal image semantic compression (MISC) approach [ 5 ] , which is designed to capture essential semantic information from input images to guide a controlled reconstruction process. Our goals are twofold: first, to enhance the reconstruction quality by improving pixel fidelity and maximizing both perceptual similarity and quality; and second, to improve the efficiency of the reconstruction process. To this end, we present the following contributions: • We introduce semantic segmentation at the decoder to improve the accuracy of conditional diffusion without the need to encode object masks. • We introduce content-adaptive diffusion to determine the appropriate diffusion settings for each image. 2 Related Work In this section, we review related works on generative image codecs that can be broadly classified into Generative Adversarial Network (GAN)-based, Variational Autoencoder (VAE)-based, and diffusion-based approaches. Among these, diffusion approaches have gained significant recent interest because of their solid theoretical basis and outstanding performance. Their applications in image compression differ in how they employ semantic information. The Sample-What-You-Can-Compress (SWYCC) [ 8 ] approach combines a fully convolutional autoencoder for representation learning and a diffusion model for image reconstruction and demonstrates significantly better performance compared to GAN-based methods in perceptual fidelity metrics. Conditional Diffusion Compression (CDC) [ 9 ] encodes a discrete “content” latent variable in the forward diffusion process and uses it for conditioning a denoising diffusion process at the decoder. The method achieves promising results, outperforming GAN baselines in certain metrics. Some generative codecs utilize human-readable image semantics. For example, semantic generative compression (SGC) [ 4 ] and Prompt Inversion Compressor with Sketch (PICS) [ 10 ] both utilize latent diffusion models (LDM) [ 3 ] to reconstruct images based on various sources of semantic information, including object labels (closed vocabulary), bags of words (open vocabulary), and to map the semantics using a segmentation map or a color map. However, the vagueness introduced by their limited textual guidance results in inconsistency in either color or texture. MISC [ 5 ] improves the consistency of reconstruction by encoding the image into structural textual semantics and incorporating an autoencoder to provide blurred reconstruction as initial color/pattern guidance for reverse diffusion. However, its patch-based strategy, which uses fixed-size image grid maps to indicate objects’ positions, requires heuristic settings and cannot deal with objects of various sizes. The present work improves upon MISC, enabling better rate-quality trade-offs across various metrics while reducing decoding complexity. 3 Methods End-to-end compression framework with our proposed methods incorporating with MISC is depicted in Fig. 1 . The Semantic Encoder extracts textual semantics from the input image, capturing a description of its most prominent objects and an overall scene description. This is accomplished by querying the GPT-4 Vision model [ 11 ] . These semantics are organized into the set of names T_{n}[j] and details T_{d}[j] for the item j , and overall description T_{all} , together comprising about 60 words. Simultaneously, the input image is downsampled by half and compressed using a low-bitrate autoencoder with the pre-trained model [ 12 ] to produce a compact latent representation, ensuring extreme high compression ratio. Original MISC framework employs a semantic map encoder to code spatial information of various objects in the scene, it is discarded in our encoder design as shown in Fig. 1 . Instead, such aspects are handled differently using Semantic Segmentation in our proposed decoder. In the decoding phase, the low-bitrate coded image is first reconstructed with fewer details and blurry textures, which are still sufficient to capture the global structure of the scene. Then, the rest of the decoding processes adds finer details by exploiting the newly introduced Content-Adaptive Diffusion with Semantic Decoder and Semantic Segmentation, eventually improving overall quality of the decoded image. In the remainder of this section, we elaborate further on the decoding process using the two proposed modules. Figure 1 : Modules shown in yellow are newly added compared to MISC [ 5 ] . Content-adaptive diffusion is introduced to adapt diffusion parameters for a particular image. The semantics map encoder of MISC is replaced with the semantic segmentation model ClipSeg [ 13 ] and is relocated to the decoder to reduce bitrate while improving semantic mapping accuracy. 3.1 Decoder-side semantic segmentation Semantic descriptions such as T_{n}[j] , T_{d}[j] , and T_{all} provide a compact scene description of the image but lack spatial information. To compensate for this, MISC encodes spatial information by dividing the image into an 8\times 8 grid and matching patches to textual descriptions using cosine similarity in the CLIP [ 14 ] embedding space. However, this approach requires encoding

Read the complete text