Beyond Softmax: A Natural Parameterization for Categorical Random Variables
Alessandro Manenti, Cesare Alippi · 2025
Read Beyond Softmax: A Natural Parameterization for Categorical Random Variables by Alessandro Manenti, Cesare Alippi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24728.txt.
Introduction
Alessandro Manenti and Cesare Alippi rethink the softmax function in a 2025 paper. Latent categorical variables appear across deep learning models. Actions, classes, and graph relations all use them daily.
Their discrete nature fights against gradient-based learning methods. Better gradient estimators dominate past fixes for the problem. The authors instead change the parameterization itself. An information geometry reading exposes limits of softmax. Their replacement, catnat, splits categories by a hierarchy of binary choices. Proofs show gains in expressiveness and learning behavior. Readers get a foundational swap that could sit inside many models.
Catalog record
| CREATOR | Alessandro Manenti, Cesare Alippi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24728
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Softmax: A Natural Parameterization for Categorical Random Variables Alessandro Manenti, Cesare Alippi Abstract Latent categorical variables are frequently found in deep learning architectures. They can model actions in discrete reinforcement-learning environments, represent categories in latent-variable models, or express relations in graph neural networks. Despite their widespread use, their discrete nature poses significant challenges to gradient-descent learning algorithms. While a substantial body of work has offered improved gradient estimation techniques, we take a complementary approach. Specifically, we: 1) revisit the ubiquitous softmax function and demonstrate its limitations from an information-geometric perspective; 2) replace the softmax with the catnat function, a function composed of a sequence of hierarchical binary splits; we prove that this choice offers significant advantages to gradient descent due to the resulting diagonal Fisher Information Matrix. A rich set of experiments — including graph structure learning, variational autoencoders, and reinforcement learning — empirically show that the proposed function improves the learning efficiency and yields models characterized by consistently higher test performance. Catnat is simple to implement and seamlessly integrates into existing codebases. Moreover, it remains compatible with standard training stabilization techniques and, as such, offers a better alternative to the softmax function. 1 Introduction Categorical random variables — random variables that take one of a fixed set of values — are ubiquitous in machine learning. They are used to represent a wide range of concepts, including classes in a classification problem ( LeCun & Cortes 2010 ) , topics in a latent variable model ( Miao et al. 2017 ) , discrete actions in a reinforcement learning environment ( Mnih et al. 2013 ) , the presence or absence of connections in a graph ( Franceschi et al. 2019 ) and clusters in mixture models ( Jacobs et al. 1991 ) . The use of samples from categorical variables may be a modeling necessity or a choice dictated by scalability and efficiency. For instance, some problems are inherently discrete, such as selecting a word token in a language model ( Chen et al. 2018 ; Paulus et al. 2020 ) or choosing an action in a reinforcement learning task ( Mnih et al. 2013 ) . In other cases, discretization is used for practical reasons, such as scalability in sparse graph modeling ( Cini et al. 2023 ) or information compression using vector quantization in generative models ( Van Den Oord et al. 2017 ) . In many of these settings, the categorical variables are latent, lacking a direct supervisory signal for training. The training signal must therefore be derived from an auxiliary loss function on a downstream task. While low-variance unbiased gradient estimators can be constructed for some continuous latent variables using techniques like the pathwise gradient estimator ( Pflug 1996 ; Kingma & Welling 2014 ) , the same methods are often not applicable to the discrete case. Consequently, learning often suffers from high-variance or biased gradient estimates, which can lead to unstable training runs that fail to converge to a satisfactory solution ( Peters & Schaal 2006 ) . As a result, improving the training stability of models with latent categorical random variables remains an active area of research with potential for broad impact ( Mohamed et al. 2020 ; Huijben et al. 2022 ) . Most techniques developed to stabilize the training of latent categorical distributions focus on reducing the variance of the gradient estimator. This is typically achieved by introducing novel control variates ( Gu et al. 2016 ; Tucker et al. 2017 ) , employing different sampling strategies ( Kool et al. 2020 ) , or designing new gradient estimators ( Niepert et al. 2021 ) . In this work we explore a complementary perspective: improving training effectiveness by changing the function that parameterizes the categorical distributions within an information geometry-based framework. The modification we propose is simple to implement, can be easily integrated into existing codebases and is compatible with other training stabilization techniques. To the best of our knowledge, this is the first work to use results from information geometry ( Rao et al. 1945 ; Amari 1998 ) to study the softmax parameterization and replace it with a function that has better theoretical properties. Specifically, we observe that the standard softmax function has a dense Fisher Information Matrix (FIM), which induces geometric distortions in the parameter space. We therefore propose replacing it with a function designed to produce an optimization landscape more amenable to gradient-descent-based algorithms. We demonstrate that this new parameterization — a series of hierarchical binary decisions that we name catnat — yields a diagonal FIM. This diagonal structure substantially reduces geometric distortions, allowing the optimizer to follow a more direct and stable path to a solution. Through extensive experiments in diverse settings - Graph Structure Learning (GSL), Variational Autoencoders (VAEs), and Reinforcement Learning (RL) - we empirically show that the proposed modification enables models to converge to solutions with superior final performance. 2 Problem Formulation We consider models that employ a set of latent categorical variables to solve a downstream task. A pipeline general enough to include many deep learning models be described as follows. (a) Given an input x\in\mathcal{X} , a neural network g_{\theta} maps x to a vector of unnormalized scores \vec{s}\in\mathbb{R}^{S} . (b) These scores are transformed by a function \pi:\mathbb{R}^{S}\!\to\!\Delta^{K-1} into a valid categorical probability vector \vec{p} lying in the (K-1) -dimensional probability simplex \Delta^{K-1}:={\vec{p}\in\mathbb{R}^{K}_{\geq 0}:\sum_{k=1}^{K}p_{k}=1} . (c) A latent categorical variable \vec{C} is then sampled according to \mathrm{Cat}(p_{1},\dots,p_{K}) and (d) used, together with x , by a task-specific predictor f_{\psi} to produce the output y : \displaystyle(a)\hskip 8.53581pt\vec{s}=g_{\theta}(x), \displaystyle(b)\hskip 8.53581pt\vec{p}=\pi(\vec{s})\quad\text{with}\quad\vec{p}=[p_{1},\dots,p_{K}], \displaystyle(c)\hskip 8.53581pt\vec{C}\sim\mathrm{Cat}(p_{1},\dots,p_{K}), \displaystyle(d)\hskip 8.53581pty=f_{\psi}(x,\vec{C}), (1) A training signal is derived from y using a task-dependent objective (e.g., a supervised loss or a reinforcement-learning reward), and the model parameters (\theta,\psi) are learned by gradient-based optimization of the corresponding expected objective. We present the overall architecture in Figure 1 , with results naturally extending to more sophisticated architectural variants. Depending on the application, g_{\theta} , f_{\psi} , or both may be simple neural networks as in VAEs, or compositions of parametric and non-parametric components as in RL. While we focus on a single categorical latent variable for clarity, the formulation and all subsequent results extend directly to collections of categorical variables with potentially different cardinalities. Figure 1: Schematic depiction of a model with a single categorical latent random variable. For rigor, we show \vec{C} as a one-hot vector; however, in some cases (e.g., when using the standard version of the Gumbel–Softmax trick) \vec{C} may be a dense vector. 3 Related Works and Preliminaries Learning Categorical Variables The most common unbiased gradient estimator is the Score Function, or REINFORCE, gradient estimator ( Williams 1992 ) . While unbiased, it suffers from high variance. This variance can be reduced using control variates ( Ross 2006 ) by subtracting a baseline from the learning signal. Simple baselines can be constructed by sampling the random variable multiple times, at the cost of introducing non-negligible computational overhead, or they can be estimated as a moving average from previous computations ( Kool et al. 2019 ) . More advanced control variates can be built efficiently using a neural network ( Mnih & Gregor 2014 ; Grathwohl et al. 2018