Generative Refinement Networks for Visual Synthesis

2026

Read Generative Refinement Networks for Visual Synthesis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13030.txt.

Introduction

Han and four colleagues introduce Generative Refinement Networks, a visual synthesis approach fixing two flaws at once. Diffusion models spend equal effort on all inputs, while autoregressive models lose detail to discrete tokens and piling errors. Their Hierarchical Binary Quantization keeps images near-lossless, and a global refinement step polishes outputs like a painter at a canvas. On ImageNet the system sets records at 0.56 rFID for reconstruction and 1.81 gFID for class-conditional generation.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39152
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13030

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative Refinement Networks for Visual Synthesis Thanks: Corresponding author: [email protected] Abstract While diffusion models dominate the field of visual generation, they remain computationally inefficient, as they allocate uniform computational effort to samples with varying levels of complexity. In contrast, autoregressive (AR) models are inherently complexity-aware, as evidenced by their variable likelihoods, but are often hindered by lossy discrete tokenization and error accumulation. In this work, we introduce Generative Refinement Networks (GRN), a next-generation visual synthesis paradigm to address these issues. At its core, GRN addresses the discrete tokenization bottleneck through a theoretically near-lossless Hierarchical Binary Quantization (HBQ), achieving a reconstruction quality comparable to continuous counterparts. Built upon HBQ’s latent space, GRN fundamentally upgrades AR generation with a global refinement mechanism that progressively perfects and corrects artworks — like a human artist painting. Besides, GRN integrates an entropy-guided sampling strategy, enabling complexity-aware, adaptive-step generation without compromising visual quality. On the ImageNet benchmark, GRN establishes new records in image reconstruction (0.56 rFID) and class-conditional image generation (1.81 gFID). We also scale GRN to more challenging text-to-image and text-to-video generation, delivering superior performance on an equivalent scale. We release all models and code to foster further research on GRN. Figure 1: Qualitative results for the class-to-image generation task. 1 Introduction The field of visual generation has advanced rapidly, driven primarily by scaling diffusion transformers [ 43 , 4 , 29 , 65 , 22 ] . By progressively integrating trajectories along a learned velocity field that transports simple noise prior to the empirical data distribution, these models demonstrate strong capabilities in synthesizing high-quality visual content. However, this continuous flow paradigm inherently lacks adaptive-step capacity. Optimized via mean squared error (MSE) without explicit likelihoods, these models are restricted to a fixed number of steps, rigidly allocating identical computational resources to all samples regardless of varying levels of complexity. Meanwhile, inspired by the success of token-level likelihood estimation in large language models [ 41 , 1 ] , autoregressive (AR) models have also garnered extensive research interest in visual synthesis [ 58 , 52 , 26 , 56 ] . Nevertheless, current AR approaches are bottlenecked by two critical shortcomings. First, they intrinsically suffer from inferior reconstruction quality when utilizing discrete tokens as opposed to continuous representations. Second, their strictly causal prediction mechanism, whether they operate token-by-token or scale-by-scale, inevitably causes severe error accumulation over multi-step generation. This exposes a critical lack of error-correction capability, as the model cannot retroactively refine previous mistakes. Furthermore, even with parallel prediction in masked AR models [ 7 , 17 ] , high-confidence tokens become immutable and cannot be revised later. Consequently, such models still inherently lack a holistic refinement mechanism. These observations motivate us to propose a simple yet intuitive refinement-based AR generation framework equipped with adaptive computation capabilities. Specifically, we introduce Generative Refinement Networks (GRN), a next-generation visual synthesis paradigm designed to overcome the rigidly fixed computational costs of diffusion models and the inherent shortcomings of standard autoregressive models. Specifically, to address the inferior reconstruction of discrete tokens, we first propose Hierarchical Binary Quantization (HBQ). By ensuring an exponential decay of reconstruction error without increasing latent channels, HBQ empowers discrete image and video tokenizers to achieve near-lossless reconstruction, matching the performance of continuous tokenizers at a higher compression rate. Building upon these robust representations, GRN executes a complexity-aware adaptive-step generation process by employing an entropy-guided sampling mechanism. It dynamically distributes computational loads based on the varying difficulty of visual content, while employing a global refinement mechanism to retroactively mitigate accumulated errors. Extensive experiments on diverse visual tasks validate the superiority of our framework. On the ImageNet 256 \times 256 benchmark [ 16 ] for class-conditional image synthesis, GRN sets a new record for both image reconstruction and generation quality. Furthermore, demonstrating exceptional task generalization and scalability, we also successfully scale GRN to high-resolution text-to-image (T2I) and text-to-video (T2V) scenarios. When scaled up, GRN demonstrates the capability to generate photorealistic 1024 \times 1024 images alongside dynamic, high-fidelity 480p videos ranging from 2 to 10 seconds. In summary, our main contributions are as follows: 1. We propose GRN, the next-generation visual synthesis framework. It is characterized by a global refinement mechanism and complexity-aware generation, achieving robust and efficient visual generation. 2. We introduce Hierarchical Binary Quantization and contribute a series of discrete image/video tokenizers. For the first time, discrete visual tokenizers are on par with continuous ones with the same latent dimensions. 3. Extensive experiments show that GRN achieves state-of-the-art results on standard C2I benchmarks, with an rFID of 0.56 and a gFID of 1.81. When scaled to more challenging T2I and T2V tasks, it demonstrates superior performance compared to methods at an equivalent scale. 2 Related Work 2.1 Visual Tokenizer Visual tokenizers [ 47 , 53 , 19 ] compress visual content for efficient generation. Early vector quantization methods [ 53 , 19 ] map continuous features to a discrete codebook, but suffer from limited scalability, prompting lookup-free approaches [ 66 , 39 ] to enable larger vocabularies. Despite this, a performance gap to continuous representations remains. Recent works [ 26 , 3 ] aim to close this gap by drastically scaling vocabularies, outperforming continuous VAEs. However, this gain comes at the cost of slower convergence and larger generative models, motivating more efficient quantization schemes. 2.2 Autoregressive Models Inspired by large language models, [ 19 , 50 , 28 , 55 ] explore visual generation via next-token prediction. MaskGIT [ 7 ] accelerates generation using parallel decoding, where it first generates high-confidence tokens and then iteratively fills in the remainder. VAR [ 52 ] shifts autoregression to next-scale prediction, improving quality and achieving over 10\times faster inference. Nevertheless, AR models remain limited by lossy discrete tokenization and error accumulation, and still lag behind diffusion methods. Although Infinity [ 26 ] introduces self-correction by randomly flipping bitwise tokens, its assumption of less than 30% diffuse errors covers limited patterns. 2.3 Adaptive-step Generation Diffusion models dominate visual generation [ 31 , 44 , 18 , 4 , 54 ] , but typically require tens of inference steps. Distillation methods [ 61 , 60 ] reduce sampling steps substantially, yet still rely on predefined schedules with fixed steps. This “one-size-fits-all” strategy wastes computational resources on simple prompts. Recently, AdaDiff [ 64 ] employs an external network to determine instance-specific steps and uses a policy gradient method to maximize the reward. The sophisticated pipeline requires an additional network and reward signals. 3 Method Figure 2: Hierarchical Binary Quantization. Each element from the VAE encoded features undergoes several rounds of hierarchical binary quantization. The quantization error decays exponentially with the number of rounds, theoretically enabling lossless quantization to be achieved rapidly. 3.1 Visual Tokenizer The visual tokenizer plays a vital role in learning a compact latent space to compress high-dimensional realistic data. We adopt the 3D causal VAE design proposed in Wan 2.1 [ 54 ] so that images and videos can be tokenized in a unified framework. Specifically, given an image or a video X\in R^{(1+4T)\times H\times W\times 3} , the tokenizer encodes its spatio-temporal information into dimensions [1+T,H/16,W/16] while expanding the number of channels to C . Since our goal is lossless discrete compression, and VAE features are continuous signals, we

Read the complete text