Efficient Image Synthesis with Sphere Latent Encoder
Tung Do, Thuan Hoang Nguyen, Hao Li · 2026
Read Efficient Image Synthesis with Sphere Latent Encoder by Tung Do, Thuan Hoang Nguyen, Hao Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15592.txt.
Introduction
Tung Do, Thuan Hoang Nguyen, and Hao Li speed up few-step image generation. Sphere Encoder makes fine images in few steps but flips between pixel space and latent space. It trains rebuild and generation in one net, which pits the two goals. Their split uses a fixed pretrained encoder plus a separate denoising model trained wholly in a spherical latent space. No pixel-space hops recur in training. Rebuild and generation can now specialize apart. On animal faces, flowers, and ImageNet it beats Sphere Encoder on quality and speed.
Catalog record
| CREATOR | Tung Do, Thuan Hoang Nguyen, Hao Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46177 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15592
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Image Synthesis with Sphere Latent Encoder Tung Do, Thuan Hoang Nguyen, Hao Li Abstract Few-step image generation has seen rapid progress, with consistency and meanflow-based methods significantly reducing the number of sampling steps. Despite their low inference cost, these approaches often suffer from training instability and limited scalability. Sphere Encoder is a recent alternative that produces high-quality images in only a few steps; however, it requires repeated transitions between the pixel space and latent space during inference while jointly optimizing reconstruction and generation within a single architecture. This design leads to computational inefficiency and objective conflict between reconstruction and generation. To address these limitations, we decouple the framework into a fixed pretrained image encoder and a separate latent denoising model trained entirely in a spherical latent space. Our approach eliminates repeated pixel-space operations during training and inference, improving efficiency and allowing reconstruction and generation to specialize independently. On Animal-Faces, Oxford-Flowers and ImageNet-1K datasets, our method significantly outperforms Sphere Encoder in both generation quality and inference speed, while achieving competitive results against strong few-step and multi-step baselines. Animal-Faces OxFord-Flowers ImageNet-1K Figure 1 : Generated samples by Sphere Latent Encoder in 4 steps on Animal-Faces 2 , Oxford-Flowers 27 , and ImageNet-1K 3 (256 × 256). 1 Introduction Flow Matching 21 ; 1 ; 23 provides a principled framework for transforming a prior distribution into a target data distribution by learning transport paths between them. Due to its scalability and strong performance, it has been widely adopted in large-scale generative systems such as Stable Diffusion 3 6 , FLUX 18 , and Qwen-Image 33 . However, standard flow matching relies on iterative sampling, which requires multiple function evaluations and results in high inference cost. Recent works 31 ; 32 ; 24 ; 8 ; 9 aim to reduce the number of sampling steps, moving toward few-step or even single-step generation. Despite these advances in inference efficiency, previous work has shown that these few-step models, including Consistency Models 31 and MeanFlow 8 are often unstable and sensitivity to hyperparameters 12 , and may even exhibit inherent instability when trained from scratch 16 . Furthermore, the MeanFlow objective introduces conflicting optimization signals, leading to slow convergence and further difficulties in training 42 . Consequently, these challenges limit the scalability and practical applicability of one-step generative models. In contrast, Sphere Encoder 40 represents an alternative to diffusion and flow-matching methods. It adopts an encoder–decoder architecture that jointly models reconstruction and generation, and projects latent representations onto a hypersphere instead of relying on a KL divergence objective 17 . Combined with tailored training objectives, this design enables generation from pure Gaussian noise in only a few steps. With its simplicity, Sphere Encoder offers a conceptually different perspective on generative modeling and demonstrates promising potential for efficient generation. However, despite its promise, Sphere Encoder has two important limitations. First, its generation process repeatedly alternates between latent space and pixel space: a latent is decoded into an image, then re-encoded into latent space, and this procedure is repeated for multiple sampling steps. These repeated pixel–latent transitions introduce substantial computational overhead during both training and inference, reducing the practical efficiency of the method. We illustrate the generation process of Sphere Encoder on the left of Fig. 2 . Second, Sphere Encoder jointly optimizes reconstruction and generation within a single encoder–decoder architecture. Prior work 36 ; 30 suggests that reconstruction and generation favor different representations. Improving reconstruction does not necessarily improve generation, and design choices that benefit one can harm the other. Sphere Encoder itself exhibits this tension. In particular, lower noise levels improve reconstruction quality, but they also reduce hyperspherical coverage and weaken generative performance. As a result, Yue et al. 40 must rely on a substantially larger architecture to maintain sample quality. In this work, we build on the spherical latent perspective of Sphere Encoder while removing its dependence on iterative pixel-space refinement. Our key idea is to perform the entire generative process in latent space. Concretely, we use a pretrained representation autoencoder 43 as a fixed image tokenizer and train a separate denoising model directly on spherical latents. During training, supervision is applied purely in latent space. During sampling, the latent is iteratively refined only in latent space, and the decoder is invoked once at the end to map the final latent back to pixels. This decoupled design eliminates repeated encoding and decoding operations, separates reconstruction from generation, and allows each component to specialize in its own role. Our approach combines the strengths of pretrained latent representations with the efficiency of few-step latent generation. Compared with the original Sphere Encoder, it offers a simpler and more efficient pipeline, while preserving the benefits of spherical latent modeling. Compared with few-step diffusion and flow-based methods, it avoids first-order approximation objectives (e.g., Jacobian-vector products) and instead learns a direct latent denoising process on a structured spherical manifold. As a result, the method is both practically efficient and straightforward to optimize. In Fig. 2 , we highlight the key difference between Sphere Encoder 40 and our method. Figure 2 : Comparison between Sphere Encoder and our method. Left: Sphere Encoder repeatedly alternates between pixel space and latent space during generation, requiring iterative encoding and decoding steps. Right: our method performs denoising entirely in latent space and applies the decoder only once at the end, leading to a simpler and more efficient pipeline. We evaluate our method on Animal-Faces 2 , Oxford-Flowers 27 and ImageNet-1K 3 datasets following prior work. Experiments show that our latent-only spherical framework substantially improves over Sphere Encoder in both generation quality and inference efficiency, while remaining simple and scalable. In summary, our main contributions are as follows: 1. We introduce Sphere Latent Encoder, a generative framework operating in spherical latent space that enables efficient training and sampling without repeated transitions between pixel and latent representations. Our method reduces inference cost by approximately 85% (about 6.5 \times fewer FLOPs) compared to Sphere Encoder at the same sampling step. 2. We decouple reconstruction and generation by leveraging a pretrained representation autoencoder 43 as a fixed image tokenizer and training a separate denoising model, allowing each component to specialize in its respective task. 3. We conduct extensive experiments on Animal-Faces 2 , Oxford-Flowers 27 and ImageNet-1K 3 , showing that our method significantly outperforms Sphere Encoder and competes with other baselines in both generation quality and sampling speed. 2 Related Works Efficient Image Generation. Recent works 8 ; 9 ; 13 ; 42 explore training few-step flow-matching models from scratch. While these methods achieve strong performance in class-conditional generation, their scalability remains limited due to high training costs and instability in Jacobian–vector product (JVP) computations. These limitations make it challenging to scale this line of work to larger settings. Meanwhile, distillation-based methods 38 ; 26 ; 37 remain the most practical solution and are widely adopted in text-to-image generation. Recent work 15 further extends this paradigm by incorporating reinforcement learning to enable more preference-aware and controllable distillation. Another emerging direction is drifting models 4 . Although initial results are promising, their scalability remains limited, as they are prone to mode collapse and often require large batch sizes during training. Sampling in Autoencoder Latent Spaces. Variational autoencoders 17 match the