Accelerate TarFlow Sampling with GS-Jacobi Iteration
Ben Liu, Zhen Qin · 2025
Read Accelerate TarFlow Sampling with GS-Jacobi Iteration by Ben Liu, Zhen Qin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12849.txt.
Introduction
Ben Liu and Zhen Qin speed up a state of the art image generator called TarFlow, in this 2025 paper. The model marries transformer blocks with a flow method and tops several benchmarks. Its weakness is sampling: the step by step design forces slow, sequential drawing of each image.
The authors recast sampling as an iteration problem and attack it with a blended solver method. They first rank blocks by how fast each settles and score how good a starting guess each receives. Difficult blocks then get more iterations and careful starts, while easy ones finish quickly. Across four model variants the trick sped sampling up to about five times. Image quality scores did not slip. Code and trained models are public.
Catalog record
| CREATOR | Ben Liu, Zhen Qin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accelerate TarFlow Sampling with GS-Jacobi Iteration Ben Liu, Zhen Qin Abstract Image generation models have achieved widespread applications. As an instance, the TarFlow model combines the transformer architecture with Normalizing Flow models, achieving state-of-the-art results on multiple benchmarks. However, due to the causal form of attention requiring sequential computation, TarFlow’s sampling process is extremely slow. In this paper, we demonstrate that through a series of optimization strategies, TarFlow sampling can be greatly accelerated by using the Gauss-Seidel-Jacobi (abbreviated as GS-Jacobi) iteration method. Specifically, we find that blocks in the TarFlow model have varying importance: a small number of blocks play a major role in image generation tasks, while other blocks contribute relatively little; some blocks are sensitive to initial values and prone to numerical overflow, while others are relatively robust. Based on these two characteristics, we propose the Convergence Ranking Metric (CRM) and the Initial Guessing Metric (IGM): CRM is used to identify whether a TarFlow block is "simple" (converges in few iterations) or "tough" (requires more iterations); IGM is used to evaluate whether the initial value of the iteration is good. Experiments on four TarFlow models demonstrate that GS-Jacobi sampling can significantly enhance sampling efficiency while maintaining the quality of generated images (measured by FID), achieving speed-ups of 4.53× in Img128cond, 5.32× in AFHQ, 2.96× in Img64uncond, and 2.51× in Img64cond without degrading FID scores or sample quality. Code and checkpoints are accessible on https://github.com/encoreus/GS-Jacobi_for_TarFlow . 1 Introduction Image generation models have been widely applied in various scenarios. As an instance, normalizing-flow-based models, from the original NICE ( Dinh et al. 2014 ) model, to improved RealNVP ( Dinh et al. 2016 ) and Glow ( Kingma and Dhariwal 2018 ) models, offer unique advantages through their invertible architecture that applies sequence of lossless transformations to noise vectors, but show limited performance in the generation of high solution and complex images. Recently, Transformer Autoregressive Flow model (TarFlow, ( Zhai et al. 2024 ) ) introduces stacks of autoregressive Transformer blocks (similar to MAF ( Papamakarios et al. 2017 ) ) into the building of affine coupling layers to do Non-Volume Preserving, combined with guidance ( Ho and Salimans 2022 ) and denoising ( Bigdeli et al. 2023 ) , finally achieves state-of-the-art results across multiple benchmarks. However, TarFlow’s sampling efficiency suffers from a critical bottleneck: the causal attention structure within each autoregressive block forms a nonlinear RNN. This forces strictly sequential computation during sampling, where each step must wait for the full update of preceding key-value caches, resulting in significantly reduced computational efficiency for large-scale image generation tasks. In this paper, we try to solve this. We first transform the nonlinear RNN in the TarFlow sampling phase into a diagonalized nonlinear system, then we can employ iteration-based solvers such as Gauss-Seidel-Jacobi (GS-Jacobi) iteration ( Ortega and Rheinboldt 2000 ) ( Song et al. 2021 ) ( Santilli et al. 2023 ) . However, naively applying GS-Jacobi iteration leads to generation failure (see 1st and 2nd row of Figure 4 ). Through detailed analysis, we discover that blocks in the TarFlow model have varying importance: a small number of blocks play a major role in image generation tasks, while other blocks contribute relatively little; some blocks are sensitive to initial values and prone to numerical overflow, while others are relatively robust. Based on these two characteristics, we propose the Convergence Ranking Metric (CRM) and the Initial Guessing Metric (IGM): CRM is used to identify whether a TarFlow block is "simple" (converges in few iterations) or "tough" (requires more iterations); IGM is used to evaluate whether the initial value of the iteration is appropriate. Leveraging these two metrics, we present a sampling algorithm that substantially reduces calculation amount required. As a summary, we list our contributions as follows: • Transform the sampling process of the TarFlow model into a diagonalized nonlinear system, and apply a Gauss-Seidel-Jacobi hybrid iteration scheme to it, providing corresponding error propagation analysis and convergence guarantees; • To identify the non-uniform transformation patterns in TarFlow blocks and control the number of iterations, we propose the Convergence Ranking Metric (CRM) to evaluate and measure them; • To control the stability of iteration initial values to avoid numerical overflow, we propose the Initial Guessing Metric (IGM); • Comprehensive experiments demonstrate 4.53× speedup in Img128cond, 5.32× in AFHQ, 2.96× in Img64uncond, 2.51× in Img64cond sampling without measurable degradation in FID scores or sample quality. 2 Related Work Normalizing Flow Based Models In the field of image generation, numerous methods have been proposed. From PixelRNN ( ( Van Den Oord et al. 2016 ) ), GANs ( Goodfellow et al. 2020 ) , to DDPM ( Ho et al. 2020 ) , Stable Diffusion ( ( Podell et al. 2023 ) ). Diffusion models seem to dominate this field, but normalizing flows still offer unique advantages, including exact invertibility ( Whang et al. 2021 ) enabling precise density estimation ( Trippe and Turner 2018 ) , single-step sampling for efficient generation ( Grcić et al. 2021 ) , and structured latent spaces that support interpretable manipulation ( Bose et al. 2020 ) . Normalizing Flows learn an invertible model f that transforms noise z into data x , such that x=f(z) . The key to build invertible models is accessible inverse function with Jacobi determinant easy to calculate, series of flow models accomplishes this through coupling layers. NICE ( Dinh et al. 2014 ) introduced the additive coupling layers. To enhance the non-linear capability, RealNVP ( Dinh et al. 2016 ) integrated scaling and shifting to the non-volume preserving transform as the affined coupling layer. Glow ( Kingma and Dhariwal 2018 ) improved the images generation by introducing invertible 1\times 1 convolution, and Flow++ ( Ho et al. 2019 ) included attention mechanic. The most significant advantage of these models is that the inverse function is explicit and Jacobi matrix is lower triangle. This can avoid the complex calculation in the general invertible ResNet framework proposed in ( Behrmann et al. 2019 ) . However, overly simple structure makes these flow models less nonlinear. To improve this, normalizing flows are combined with autoregressive models. IAF ( Kingma et al. 2016 ) pioneered dimension-wise affine transformations conditioned on preceding dimensions to improve variational inference. MAF ( Papamakarios et al. 2017 ) utilized the MADE ( Germain et al. 2015 ) to create invertible autoregressive mappings. NAF ( Huang et al. 2018 ) , which replaced MAF’s affine transformations with per-dimension monotonic neural networks to enhance expressivity. T-NAF ( Patacchiola et al. 2024 ) augmented NAF by integrating a single autoregressive Transformer, whereas Block Neural Autoregressive Flow ( De Cao et al. 2020 ) adopted an end-to-end autoregressive monotonic network design. TarFlow ( Zhai et al. 2024 ) proposed a Transformer-based architecture together with a set of techniques to train high performance normalizing flow models and show SOTA in many fields, thus becomes the sampling object of this article. Parallel solving of linear/nonlinear systems Linear/nonlinear systems refer to f_{i}(\mathbf{x})=0,\mathbf{x}\in\mathbb{R}^{n} , where f_{i},i=1,\ldots,n is a linear/nonlinear function. The parallel solution of these systems is an important problem in scientific computing. ( Saad 2003 ) established methods such as Jacobi, Gauss-Seidel, successive over-relaxation (SOR), and Krylov