Normalizing Flows are Capable Generative Models

Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, Josh Susskind · 2024

Read Normalizing Flows are Capable Generative Models by Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, Josh Susskind (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06329.txt.

Introduction

Shuangfei Zhai and co-authors present a 2024 paper reviving an older family of generative models called normalizing flows. These models learn the odds of data by warping a simple distribution through a chain of reversible steps. They had fallen behind newer methods in recent years.

Their design, TarFlow, stacks transformer blocks that work on image patches and flip the reading direction layer by layer. Three extra tricks lift sample quality: noise added during training, a denoising pass afterward, and guidance for class-based generation. The model set new records for image likelihood scores, beating prior best methods by a wide margin. Its samples rival diffusion models in quality and variety, a first for a stand-alone flow model.

Catalog record

CREATORShuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, Josh Susskind
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06329

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Normalizing Flows are Capable Generative Models Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, Josh Susskind Abstract Normalizing Flows (NFs) are likelihood-based models for continuous inputs. They have demonstrated promising results on both density estimation and generative modeling tasks, but have received relatively little attention in recent years. In this work, we demonstrate that NFs are more powerful than previously believed. We present TarFlow : a simple and scalable architecture that enables highly performant NF models. TarFlow can be thought of as a Transformer-based variant of Masked Autoregressive Flows (MAFs): it consists of a stack of autoregressive Transformer blocks on image patches, alternating the autoregression direction between layers. TarFlow is straightforward to train end-to-end, and capable of directly modeling and generating pixels. We also propose three key techniques to improve sample quality: Gaussian noise augmentation during training, a post training denoising procedure, and an effective guidance method for both class-conditional and unconditional settings. Putting these together, TarFlow sets new state-of-the-art results on likelihood estimation for images, beating the previous best methods by a large margin, and generates samples with quality and diversity comparable to diffusion models, for the first time with a stand-alone NF model. We make our code available at https://github.com/apple/ml-tarflow . Figure 1: TarFlow demonstrates substantial progress in the domain of normalizing flow models, achieving state-of-the-art results in both density estimation and sample generation. Left : We show the historical progression of likelihood performance on ImageNet 64x64, measured in bits per dimension (BPD), where our model significantly outperforms previous methods (see Table 2 for details). Right : Selected samples from our model trained on ImageNet 128x128 demonstrate unprecedented image quality and diversity for a normalizing flow model, establishing a new benchmark for this class of generative models. 1 Introduction Normalizing Flows (NFs) are a well-established likelihood based method for unsupervised learning ( Tabak & Vanden-Eijnden, 2010 ; Rezende & Mohamed, 2015 ; Dinh et al., 2014 ) . The method follows a simple learning objective, which is to transform a data distribution into a simple prior distribution (such as Gaussian noise), keeping track of likelihoods via the change of variable formula. Normalizing Flows enjoy many unique and appealing properties, including exact likelihood computation, deterministic objective functions, and efficient computation of both the data generator and its inverse. There has been a large body of work dedicated to studying and improving NFs, and in fact NFs were the method of choice for density estimation for a number of years ( Dinh et al., 2017 ; Kingma & Dhariwal, 2018 ; Chen et al., 2018 ; Papamakarios et al., 2017 ; Ho et al., 2019 ) . However in spite of this rich line of work, Normalizing Flows have seen limited practical adoption— in stark contrast to other generative models such as Diffusion Models ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ) and Large Language Models ( Brown et al., 2020 ) . Moreover, the state-of-the-art in Normalizing Flows has not kept pace with the rapid progress of these other generative techniques, leading to less attention from the research community. It is natural to wonder whether this situation is inherent – i.e., are Normalizing Flows fundamentally limited as a modeling paradigm? Or, have we just not found an appropriate way to train powerful NFs and fully realize their potential? Answering this question may allow us to reopen an alternative path to powerful generative modeling, similar to how DDPM ( Ho et al., 2020 ) enlightened the field of diffusion modeling and brought about its current renaissance. In this work, we show that NFs are more powerful than previously believed, and in fact can compete with state-of-the-art generative models on images. Specifically, we introduce TarFlow (short for Transformer AutoRegressive Flow): a powerful NF architecture that allows one to easily scale up the model’s capacity; as well as a set of techniques that drastically improve the model’s generation capability. On the architecture side, TarFlow is conceptually similar to Masked Autoregressive Flows (MAFs) ( Papamakarios et al., 2017 ) , where we compose a deep transformation by iteratively stacking multiple blocks of autoregressive transformations with alternating directions. The key difference is that we deploy a powerful masked Transformer ( Vaswani et al., 2017 ) based implementation that operates in a block autoregression fashion (that is, predicting a block of dimensions at a time), instead of simple masked MLPs used in MAFs that factorizes the input on a per dimension basis. In the context of image modeling, we implement each autoregressive flow transformation with a causal Vision Transformer (ViT) ( Dosovitskiy et al., 2021 ) on top of a sequence of image patches, given a particular order of autoregression (e.g., top left to bottom right, or the reverse). This admits a powerful non-linear transformation among all image patches, while maintaining a parallel computational graph during training. Compared to other NF design choices ( Dinh et al., 2017 ; Grathwohl et al., 2019 ; Kingma & Dhariwal, 2018 ; Ho et al., 2019 ) which often have several types of interleaving modules, our model features a modular design and enjoys greater simplicity, both conceptually and practically. This in return allows for much improved scalability and training stability, which is another critical aspect for high performance models. With this new architecture, we can immediately train much stronger NF models than previously reported, resulting in state-of-the-art results on image likelihood estimation. On the generation side, we introduce three important techniques. First, we show that for perceptual quality, it is critical to add a moderate amount of Gaussian noise to the inputs, in contrast to a small amount of uniform noise commonly used in the literature. Second, we identify a post-training score based denoising technique that allows one to remove the noise portion of the generated samples. Third, we show for the first time that guidance ( Ho & Salimans, 2022 ) is compatible with NF models, and we propose guidance recipes for both the class conditional and unconditional models. Putting these techniques together, we are able to achieve state-of-the-art sample quality for NF models on standard image modeling tasks. We highlight our main results in Figure 1 , and summarize our contributions as follows. • We introduce TarFlow , a simple and powerful Transformer based Normalizing Flow architecture. • We achieve state-of-the-art results on likelihood estimation on images, achieving a sub- 3 BPD on ImageNet 64x64 for the first time. • We show that Gaussian noise augmentation during training plays a critical role in producing high quality samples. • We present a post-training score-based denoising technique that allows one to remove the noise in the generated samples. • We show that guidance is compatible with both class conditional and unconditional models, which drastically improves sampling quality. Table 1: Notation. Notation Meaning p_{\rm{data}}(x) training distribution p_{\rm{model}}(y) model distribution f(x) the forward flow function f^{t}(z^{t}) the forward function for the t -th flow block \mu^{t},\alpha^{t} learnable causal functions in the t -th flow block p_{\epsilon}(\epsilon) the noise distribution q(y) the noisy data distribution \tilde{p}(\tilde{x}) the discrete model distribution Figure 2: Left , TarFlow consists of T flow blocks trained end to end; Right , a

Read the complete text