α \alpha -Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models

Chaoran Cheng, Jiahan Li, Jiajun Fan, Ge Liu · 2025

Read α \alpha -Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models by Chaoran Cheng, Jiahan Li, Jiajun Fan, Ge Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10283.txt.

Introduction

Chaoran Cheng and co-authors unify a family of generative models for discrete data in a 2025 paper. Flow matching, a recipe for building generators, has been stretched to tokens, pixels, and protein letters. The continuous-state variants disagree on how to represent probabilities, making them hard to compare.

Using information geometry, the team shows each variant works in a different alpha-representation of the same probability space. Their alpha-Flow family follows the canonical geometry of that space and provably minimizes a generalized kinetic energy. Tests across image, protein, and language generation beat the discrete-state counterpart and captured output diversity better. Intermediate geometries never tried before also proved viable. The framework turns a scattered literature into one dial.

Catalog record

CREATORChaoran Cheng, Jiahan Li, Jiajun Fan, Ge Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

α \alpha -Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models Chaoran Cheng, Jiahan Li, Jiajun Fan, Ge Liu Abstract Recent efforts have extended the flow-matching framework to discrete generative modeling. One strand of models directly works with the continuous probabilities instead of discrete tokens, which we colloquially refer to as Continuous-State Discrete Flow Matching (CS-DFM). Existing CS-DFM models differ significantly in their representations and geometric assumptions. This work presents a unified framework for CS-DFM models, under which the existing variants can be understood as operating on different α \alpha -representations of probabilities. Building upon the theory of information geometry, we introduce 𝜶 \boldsymbol{\alpha} - Flow , a family of CS-DFM models that adheres to the canonical α \alpha -geometry of the statistical manifold, and demonstrate its optimality in minimizing the generalized kinetic energy. Theoretically, we show that the flow matching loss for α \alpha -flow establishes a unified variational bound for the discrete negative log-likelihood. We comprehensively evaluate five different instantiations of α \alpha -flow on various discrete generation domains to demonstrate their effectiveness in discrete generative modeling, including intermediate values whose geometries have never been explored before. α \alpha -flow significantly outperforms its discrete-state counterpart in image and protein sequence generation and better captures the entropy in language modeling. 1 Introduction The success of flow matching ( Lipman et al., 2022 ) and diffusion ( Ho et al., 2020 ) models has inspired recent attempts to adopt such generative frameworks to discrete domains. Among the existing discrete flow matching (DFM) models, one strand of models directly works with the continuous representations of probabilities instead of discrete tokens, which we colloquially refer to as Continuous-State DFM (CS-DFM). To leverage the continuous flow matching framework for the generative modeling of discrete data, all CS-DFM models face two key designs of: 1. The continuous representation of the discrete data; and 2. The geometry of the continuous representation. The continuous representation allows direct applications of flow matching, and the geometry defines the conditional flow on which the vector field can be regressed. Existing CS-DFM models differ significantly in their choices of these two modeling approaches. Similar to autoregressive ( Radford et al., 2019 ) or masked language models ( Devlin, 2018 ) , the most straightforward choice of the continuous representations might be the logits of the categorical probabilities. Alternatively, one may work directly with probabilities with discrete data represented as a one-hot vector. Existing works often assume a Euclidean geometry such that the intermediate representation can be obtained by linear interpolation between the initial noise and the target representations ( Li et al., 2024 ; Song et al., 2023 ) . Two recent works ( Cheng et al., 2024 ; Davis et al., 2024 ) incorporated mathematical results from information geometry and designed a continuous representation of the categorical distribution on the unit sphere with the canonical spherical geometry. In principle, different representations and geometries lead to different variants of CS-DFM. With such flexibility and diversity, no framework has yet been proposed to provide a unified and comparative analysis of the existing variants mentioned above. This work fills in the missing gap for connecting existing CS-DFM models and presents a unified framework under which variants can be understood as operating on the different \alpha - representations of the categorical distribution. Furthermore, rigorously built upon the theory of information geometry, we propose a family of CS-DFM models that adheres to the canonical \alpha - geometry by following the \alpha - geodesics , which we call \boldsymbol{\alpha} - Flow . Our \alpha -flow enjoys additional theoretical benefits as we demonstrate its optimality with respect to the generalized kinetic energy, encompassing some already-known results in previous work as special cases. Additionally, we establish a unified variational bound for the discrete negative log-likelihood (NLL), providing theoretical justifications for the flow matching loss in discrete generative modeling. We extensively evaluate different instantiations of \alpha -flow on the discrete generative modeling tasks in various domains, including computer vision, natural language processing, and bioinformatics, scaling up to billion-level tokens to study their empirical performance. Our \alpha -flow outperforms its discrete-state counterpart and is on par with the latter in language modeling. Heuristic rules for choosing proper model designs are also discussed. 2 Preliminary 2.1 Riemannian Flow Matching Conditional flow matching (CFM) ( Lipman et al., 2022 ) addresses generative modeling by learning a time-dependent vector field that pushes the prior noise distribution to any target data distribution. Such a flow-based model can be viewed as the continuous generalization of the diffusion model ( Song and Ermon, 2019 ; Song and Ermon, 2020 ; Ho et al., 2020 ) with a more flexible design of the denoising process. The Riemannian flow matching framework ( Chen and Lipman, 2023 ) further extends CFM to general manifolds on which a distance metric can be efficiently computed. Mathematically, on a smooth Riemannian manifold \mathcal{M} with the Riemannian metric g , a flow \psi_{t}:[0,1]\times\mathcal{M}\to\mathcal{M} is a time-dependent diffeomorphism defined by a time-dependent vector field u_{t}:[0,1]\times\mathcal{M}\to T\mathcal{M} via the ordinary differential equation (ODE): \frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}t}\psi_{t}(x)=u_{t}(\psi_{t}(x)) . The flow matching objective directly regresses the vector field u_{t} with a time-dependent neural net. Both Lipman et al. (2022) ; Chen and Lipman (2023) demonstrated that a tractable objective can be derived by conditioning on the target data x_{1} at t=1 of the probability path. In this way, the Riemannian flow matching objective can be formulated as \mathcal{L}_{\text{RFM}}=\mathbb{E}_{t,p_{0}(x),q(x)}\|v(x_{t},t)-u_{t}(x_{t}|x_{0},x_{1})\|_{g}^{2} (1) where q is the data distribution, p_{0} is the prior distribution, and x_{t}:=\psi_{t}(x|x_{0},x_{1}) denotes the conditional flow. Chen and Lipman (2023) further demonstrated that if the exponential map and logarithm map can be evaluated in closed-form, the condition flow can be defined as x_{t}=\exp_{x_{0}}(t\log_{x_{0}}x_{1}) , and the corresponding vector field can be calculated as u_{t}(x_{t}|x_{0},x_{1})=\log_{x_{t}}(x_{1})/(1-t) . We will adopt the notations in Riemannian geometry in this work. We use \gamma_{x_{0},x_{1}}(t):=\psi_{t}(x|x_{0},x_{1}) to denote the geodesic from x_{0} to x_{1} and \dot{\gamma}_{x_{0},x_{1}}(t):=u_{t}(x_{t}|x_{0},x_{1}) to denote the vector field along the geodesic. 2.2 Statistical Manifold It is known from information theory ( Rao, 1992 ; Amari and Nagaoka, 2000 ; Ay et al., 2017 ) that any parameterized family of probability distributions p_{\theta} forms a Riemannian structure known as the statistical manifold whose canonical Riemannian metric is the Fisher information metric . As we are interested in discrete generative modeling, we naturally focus on the statistical manifold of categorical distributions. Mathematically, consider the discrete sample space \mathcal{X}=\{1,2,\dots,n\} , a categorical distribution \mu\in\mathcal{P}(\mathcal{X}) can be defined by its categorical probabilities \mu:\sum_{i=1}^{n}\mu_{i}=1,0\leq\mu_{i}\leq 1 . A vector field a in the tangent space T_{\mu}\mathcal{P} can be identified with \sum_{i=1}^{n}a_{i}=0 . The canonical inner product at a positive categorical distribution \mu\in\mathcal{P}_{+} , in this case, can be written as \langle a,b\rangle_{\mu}=\sum_{i=1}^{n}\frac{a_{i}b_{i}}{\mu_{i}},\quad a,b\in T_{\mu}\mathcal{P}. (2) We also use \|a\|_{g}=\sqrt{\langle a,a\rangle_{\mu}} to denote the canonical Riemannian norm. The geometric view of statistical manifolds allows us to derive crucial geometric concepts of exponential and logarithm maps for applying Riemannian flow matching. Specifically, we may define geodesics , curves on the Riemannian manifold where a point travels at a “constant speed”. We note that, even if equipped with the same Fisher information metric, it is also possible to define different families of geodesics on the statistical manifold using different affine

Read the complete text