Beyond and Free from Diffusion: Invertible Guided Consistency Training
Chia-Hong Hsu, Shiu-hong Kao, Randall Balestriero · 2025
Read Beyond and Free from Diffusion: Invertible Guided Consistency Training by Chia-Hong Hsu, Shiu-hong Kao, Randall Balestriero (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05391.txt.
Introduction
Chia-Hong Hsu and co-authors published a research paper in 2025 on teaching fast image generators to accept guidance without first training a big diffusion model. Guidance steers generation toward quality or a target look. Consistency models generate in few steps but used to borrow guidance distilled from diffusion models, an expensive chain.
Their training scheme is fully data-driven and invertible, meaning images can also be run backward through it. It also fixes the washed-out colors that plague strong guidance in diffusion models. On two standard image sets it beat guided diffusion on quality scores, holding precision where the older method collapsed at high guidance. The authors call it a first step to self-sufficient guided generation.
Catalog record
| CREATOR | Chia-Hong Hsu, Shiu-hong Kao, Randall Balestriero |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05391
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond and Free from Diffusion: Invertible Guided Consistency Training Chia-Hong Hsu, Shiu-hong Kao, Randall Balestriero Abstract Guidance in image generation steers models towards higher-quality or more targeted outputs, typically achieved in Diffusion Models (DMs) via Classifier-free Guidance (CFG). However, recent Consistency Models (CMs), which offer fewer function evaluations, rely on distilling CFG knowledge from pretrained DMs to achieve guidance, making them costly and inflexible. In this work, we propose invertible Guided Consistency Training (iGCT) , a novel training framework for guided CMs that is entirely data-driven. iGCT, as a pioneering work, contributes to fast and guided image generation and editing without requiring the training and distillation of DMs, greatly reducing the overall compute requirements. iGCT addresses the saturation artifacts seen in CFG under high guidance scales. Our extensive experiments on CIFAR-10 ( Krizhevsky, 2012 ) and ImageNet64 ( Chrabaszcz et al., 2017 ) show that iGCT significantly improves FID and precision compared to CFG. At a guidance of 13, iGCT improves precision to 0.8, while DM’s drops to 0.47. Our work takes the first step toward enabling guidance and inversion for CMs without relying on DMs. Figure 1 : Guidance of EDM trained via Classifier-free Guidance (CFG) (top-left), and our Invertible Guided Consistency Training (iGCT) (bottom-left). iGCT enables fast inverse-based image editing while preserving the source semantics (right). Unlike CFG, iGCT eliminates the need for two-stage training and removes contrast artifacts, achieving better precision and FID under high guidance (Fig. 7 ). 1 Introduction Diffusion Models (DMs) have emerged as a powerful class of generative models, demonstrating state-of-the-art performance in tasks such as image synthesis ( Ho et al., 2020 ; Song & Ermon, 2020 ; Song et al., 2021 ; Song et al., 2022 ; Rombach et al., 2022 ; Podell et al., 2023 ; Lin et al., 2023 ) conditional generation ( Avrahami et al., 2022 ; Meng et al., 2022 ; Mou et al., 2023 ; Bar-Tal et al., 2023 ; Bashkirova et al., 2023 ; Huang et al., 2023 ; Zhang et al., 2023 ; Ni et al., 2023 ; Podell et al., 2023 ) , and precise image editing ( Mokady et al., 2023 ; Miyake et al., 2023 ; Wallace et al., 2023 ; Han et al., 2024 ; Garibi et al., 2024 ; Huberman-Spiegelglas et al., 2024 ) . Unlike GANs, which rely on adversarial training to generate samples ( Goodfellow et al., 2014 ; Metz et al., 2016 ; Gulrajani et al., 2017 ; Arjovsky & Bottou, 2017 ; Karras et al., 2018 ; Brock et al., 2019 ; Karras et al., 2019 ; Karras et al., 2020 ; Karras et al., 2021 ) , diffusion models (DMs) use a stochastic process that incrementally adds Gaussian noise to data, then reverses this process to generate new samples. This iterative approach, while computationally intensive, enables greater control over sample quality and diversity through guided sampling methods ( Ho & Salimans, 2022 ; Dieleman, 2022 ; Ho et al., 2022 ; Meng et al., 2023 ; Karras et al., 2024a ) . In contrast, GANs often suffer from mode collapse ( Gulrajani et al., 2017 ; Arjovsky & Bottou, 2017 ; Metz et al., 2016 ) , which hinders output diversity and training stability. Diffusion-based algorithms can thus offer a key advantage by balancing quality-diversity tradeoffs, frequently achieved with Classifier-free Guidance (CFG) ( Ho & Salimans, 2022 ) . Both the role of diffusion noise scheduling and the network’s convergence properties are essential for improving performance and applicability ( Karras et al., 2022 ; Lu et al., 2022 ; Lu et al., 2023 ; Zheng et al., 2023 ; Karras et al., 2024c ; Salimans & Ho, 2022 ; Park et al., 2024 ; Karras et al., 2024b ) . Consistency Models (CMs), designed to accelerate the diffusion process, provide a more efficient alternative by reducing the number of function evaluations (NFE) needed for generation ( Song et al., 2023 ; Kim et al., 2023 ; Kim et al., 2024 ; Li & He, 2024 ; Heek et al., 2024 ; Lu & Song, 2024 ; Lee et al., 2025 ) . These models achieve this by aligning data points along the Probability-flow (PF) ODE of the stochastic forward process, using either consistency distillation (CD) or consistency training (CT) ( Song et al., 2023 ) . While CD models have been widely applied in real-time image synthesis and fast editing via distillation ( Luo et al., 2023 ; Starodubcev et al., 2024 ) , the development of CT models remain relatively unexplored. This is primarily because CT, as a fully self-supervised and data-driven approach, requires a well-crafted training curriculum to be trained successfully ( Song & Dhariwal, 2023 ; Geng et al., 2024 ) . Year 2019 \epsilon -pred ( Ho et al., 2020 ) 2021 VE, VP ( Song et al., 2021 ) , iDDPM ( Nichol & Dhariwal, 2021 ) 2022 DDIM ( Song et al., 2022 ) , DPM-Solver v1 ( Lu et al., 2022 ) , EDM ( Karras et al., 2022 ) , v-pred ( Salimans & Ho, 2022 ) 2023 DPM-Solver++ ( Lu et al., 2023 ) , DPM-Solver v3 ( Lu et al., 2023 ) DM sch. EDM ( Karras et al., 2022 ) CD methods CD ( Song et al., 2023 ) , CTM ( Kim et al., 2023 ) Figure 2 : Timeline of key developments in diffusion training/sampling schedulers (top). The schedulers used by pretrained DMs that CD depends on (bottom). The fields of DM and CT are advancing rapidly in multiple directions, and tightly coupling CM with DM would slow down progress. To train a CM under a particular diffusion scheduler, CD-based CMs rely on a teacher DM trained under the same setting, while CT is free of such constraint . CT, however, offers distinct advantages. First, unlike CD, CT can be trained with any underlying diffusion schedulers. The design space for DM training, sampling, and preconditioning ( Karras et al., 2022 ) is an ongoing area of research that has attracted researchers interest ever since the diffusion surge. Therefore, CT’s flexibility to train with improved designs makes it more attractive than CD by dropping the dependence of a state-of-the-art DM. An improved diffusion scheduler can be directly utilized in CT, while CD depends on the existence of a DM (Fig. 2 ). Second, recent studies show that CT, when improved with advanced training techniques, outperforms CD in image generation on FID metrics ( Song & Dhariwal, 2023 ; Geng et al., 2024 ) . This underscores its potential to surpass CD in both efficiency and quality, making CT as a promising alternative for advancing fast generative modeling. Our study fills the missing piece in establishing a comprehensive theory for CT, a data-driven approach for guidance learning in 1-step/few-step image generation. We present Invertible Guided Consistency Training (iGCT), a novel framework that enables CMs to perform guided generation without requiring DM distillation at training. In training, iGCT decouples the target clean sample from the source to capture effects of both unconditional and conditional noise. For invertibility, iGCT incorporates an independent component, the noiser, which maps images to noise in a single step. This