Beyond ReinMax: Low-Variance Gradient Estimators for Discrete Latent Variables

Daniel Wang [email protected], Thang D. Bui [email protected] · 2026

Read Beyond ReinMax: Low-Variance Gradient Estimators for Discrete Latent Variables by Daniel Wang [email protected], Thang D. Bui [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08257.txt.

Introduction

Wang and colleagues improve gradient estimators for machine learning models with discrete latent variables. Such models need gradient estimators to enable backpropagation efficiently. ReinMax, the newest Straight-Through estimator, can be viewed through a numerical ODE lens as using Heun's method to reduce bias, but at the cost of high variance. The authors introduce ReinMax-Rao and ReinMax-CV, which fold Rao-Blackwellisation and control variate techniques into ReinMax to cut its variance. Their estimators perform better when training variational autoencoders with discrete latent spaces.

Catalog record

CREATORDaniel Wang [email protected], Thang D. Bui [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48501
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond ReinMax: Low-Variance Gradient Estimators for Discrete Latent Variables Daniel Wang [email protected], Thang D. Bui [email protected] Abstract Machine learning models involving discrete latent variables require gradient estimators to facilitate backpropagation in a computationally efficient manner. The most recent addition to the Straight-Through family of estimators, ReinMax, can be viewed from a numerical ODE perspective as incorporating an approximation via Heun’s method to reduce bias, but at the cost of high variance. In this work, we introduce the ReinMax-Rao and ReinMax-CV estimators which incorporate Rao-Blackwellisation and control variate techniques into ReinMax to reduce its variance. Our estimators demonstrate superior performance on training variational autoencoders with discrete latent spaces. Furthermore, we investigate the possibility of leveraging alternative numerical methods for constructing more accurate gradient approximations and present an alternative view of ReinMax from a simpler numerical integration perspective. 1 Introduction In machine learning problems which involve optimising the parameters of a discrete categorical distribution, gradient-based optimisation methods relying on backpropagation require computing derivatives with respect to the non-differentiable operation of sampling a random variable from a discrete distribution. In practical settings, this is typically done via estimators of an exact expression for the gradient that is computationally expensive to evaluate. In this paper, we focus on the family of Straight-Through gradient estimators ( 1 ) . This family of estimators have low variance compared to estimators that are based on the REINFORCE estimator ( 16 ) and are computationally efficient as they only require one forward pass of the loss function, but at the cost of being biased. In its simplest form, the Straight-Through estimator ( 1 ) is constructed by heuristically treating the Jacobian of the non-differentiable sampling of a discrete categorical random variable as the identity function. Improvements to the Straight-Through estimator in recent years have leveraged various mathematical tools. Based on the Gumbel-Softmax reparameterisation ( 4 ; 10 ) , the Straight-Through Gumbel-Softmax estimator is constructed by differentiating through a continuous relaxation of the sampling of a categorical random variable. The Gumbel-Rao estimator extends this by making use of the known reparameterisation of the conditional distribution to reduce variance via conditional marginalisation ( 14 ) . Proceeding in a numerical ODE direction, it can be shown that the Straight-Through estimator can be viewed as incorporating a first-order Forward Euler approximation ( 8 ) . Extending this to incorporate the more accurate second-order Heun’s method leads to the ReinMax estimator which has much lower bias at the cost of higher variance. To remedy this high variance, we combine the Gumbel-Softmax reparameterisation and numerical ODE approaches by incorporating Gumbel-Softmax tricks into the ReinMax estimator, firstly via a heuristic Gumbel-Rao approximation and secondly via control variates to arrive at our ReinMax-Rao and ReinMax-CV estimators. Extensive experiments on training variational autoencoders (VAEs) ( 6 ) with discrete latent spaces show that our methods outperform ReinMax and provide insight into the bias-variance trade-off for these gradient estimators. Although the main contribution of this paper is variance reduction of ReinMax, it is also worth investigating possible approaches to further reduce its bias by leveraging alternative numerical methods. To this end, we generalise the construction of ReinMax from Heun’s method to the entire family of second-order Runge-Kutta ODE methods. However, this is ineffective in practice and we argue that a different set of numerical integration methods are more appropriate for this problem. 2 Background Consider an n -class discrete random variable {\bm{D}}\in\{{\bm{I}}_{1},{\bm{I}}_{2},...,{\bm{I}}_{n}\} represented as a one-hot encoding (where {\bm{I}}_{i} is the i th standard basis vector in \mathbb{R}^{n} ), distributed according to {\bm{D}}\sim{\bm{\pi}}=\mbox{\small$\mathrm{softmax}$}({\bm{\theta}}) where {\bm{\theta}}\in\mathbb{R}^{n} are optimisable parameters. Given a differentiable loss function f:\mathbb{R}^{n}\rightarrow\mathbb{R} that takes as input a realisation of {\bm{D}} , the objective is to minimise the expected loss w.r.t. {\bm{\theta}} : \displaystyle\widehat{{\bm{\theta}}}=\argmin_{{\bm{\theta}}}\mathbb{E}_{{\bm{D}}\sim\mbox{\small$\mathrm{softmax}$}({\bm{\theta}})}[f({\bm{D}})]. (1) The derivative with respect to {\bm{\theta}} is given by \displaystyle\nabla:=\frac{\partial}{\partial{\bm{\theta}}}E_{{\bm{D}}\sim\mbox{\small$\mathrm{softmax}$}({\bm{\theta}})}[f({\bm{D}})]=\frac{\partial}{\partial{\bm{\theta}}}\sum_{i=1}^{n}{\bm{\pi}}_{i}f({\bm{I}}_{i})=\sum_{i=1}^{n}f({\bm{I}}_{i})\frac{d\,{\bm{\pi}}_{i}}{d\,{\bm{\theta}}}. (2) This expression is expensive to compute because it requires n evaluations of f which is impractical in modern deep learning settings where f is a neural network. Hence, estimators of the gradient that require only one forward and backward pass are desired. 2.1 The Straight-Through estimator ( 1 ) The Straight-Through estimator ( 1 ) preserves the non-differentiable sampling of {\bm{D}} in the forward pass while replacing it with a differentiable approximation in the backward pass. Specifically, given a realisation of the categorical random variable {\bm{D}}\sim{\bm{\pi}}:=\mbox{\small$\mathrm{softmax}$}({\bm{\theta}}) , the Straight-Through estimator is given by \displaystyle{\widehat{\nabla}}_{\text{ST},\tau}({\bm{D}},{\bm{\theta}}):=\frac{\partial f({\bm{D}})}{\partial{\bm{D}}}\cdot\frac{d\,{\bm{\pi}}_{\tau}}{d\,{\bm{\theta}}}=\frac{\partial f({\bm{D}})}{\partial{\bm{D}}}(\operatorname{diag}({\bm{\pi}}_{\tau})-{\bm{\pi}}_{\tau}{\bm{\pi}}_{\tau}^{{}^{\mathsf{T}}}),\hskip 4.0pt{\bm{\pi}}_{\tau}:=\mbox{\small$\mathrm{softmax}$}_{\tau}({\bm{\theta}}) (3) where \mathrm{softmax}_{\tau}({\bm{\theta}}):=\mathrm{softmax}(\tfrac{{\bm{\theta}}}{\tau}) is the tempered softmax function and \tau is a hyperparameter referred to as the temperature that is typically set to \tau=1 for this estimator. The Straight-Through estimator can be interpreted as simply approximating the Jacobian of the non-differentiable operation of sampling {\bm{D}} by the identity matrix (i.e., \frac{\partial{\bm{\pi}}}{\partial{\bm{D}}}\approx{\bm{I}} ) in the backward pass, and is therefore biased. 2.2 The Straight-Through Gumbel-Softmax estimator ( 4 ) Motivated by the reparameterisation trick that is commonly used for continuous random variables, 4 and 10 introduced the Gumbel-Softmax reparameterisation for sampling {\bm{D}}\sim{\bm{\pi}}=\mbox{\small$\mathrm{softmax}$}({\bm{\theta}}) : {\bm{D}}\overset{d}{=}\lim_{\tau\rightarrow 0}\mbox{\small$\mathrm{softmax}_{\tau}$}({\bm{\theta}}+{\bm{G}}) (4) where {\bm{G}} is a vector of i.i.d. \text{Gumbel}(0,1) random variables. Note that the zero-temperature limit of the \mathrm{softmax} ( \cdot ) is the one-hot embedding of the \argmax(\cdot) function, which is non-differentiable. Using this reparameterisation, the Straight-Through Gumbel-Softmax estimator is given by {\widehat{\nabla}}_{\text{STGS},\tau}({\bm{G}},{\bm{\theta}}):=\frac{\partial f({\bm{D}})}{\partial{\bm{D}}}\cdot\frac{d\,\mbox{$\mathrm{softmax}_{\tau}$}({\bm{\theta}}+{\bm{G}})}{d\,{\bm{\theta}}}. (5) In the forward pass, the one-hot vector {\bm{D}} is obtained by taking the zero-temperature limit of \mathrm{softmax}(\cdot) as in equation 4 while in the backward pass, a non-zero (typically small) temperature is chosen so that \frac{d\,\mbox{$\mathrm{softmax}_{\tau}$}({\bm{\theta}}+{\bm{G}})}{d\,{\bm{\theta}}} is well-defined. In practice, choosing a smaller value of \tau leads to low bias but high variance. 2.3 The Gumbel-Rao estimator ( 14 ) While the Gumbel-Softmax reparameterisation samples {\bm{G}} first then obtains {\bm{D}} as the deterministic \argmax(\cdot) of {\bm{\theta}}+{\bm{G}} , 14 propose to sample {\bm{D}}\sim\mathrm{softmax}({\bm{\theta}}) first, then {\bm{\theta}}+{\bm{G}} conditional on {\bm{D}} . The Gumbel-Rao estimator makes use of the conditional marginalisation \mathbb{E}_{{\bm{G}}}\left[{\widehat{\nabla}}_{\text{STGS},\tau}({\bm{G}},{\bm{\theta}})\right] = \mathbb{E}_{{\bm{D}}}\left[\mathbb{E}_{{\bm{\theta}}+{\bm{G}}|{\bm{D}}}\left[{\widehat{\nabla}}_{\text{STGS},\tau}({\bm{G}},{\bm{\theta}})\right]\right] where the Rao-Blackwell Theorem ( 2 ) implies that this estimator will have the same expectation as Straight-Through Gumbel-Softmax but lower variance. More precisely, the Gumbel-Rao estimator is given by \displaystyle\frac{\partial f({\bm{D}})}{\partial{\bm{D}}}\,\mathbb{E}_{{\bm{\theta}}+{\bm{G}}|{\bm{D}}}\left[\frac{d\operatorname{softmax}_{\tau}({\bm{\theta}}+{\bm{G}})}{d{\bm{\theta}}}\right] \displaystyle\approx\frac{\partial f({\bm{D}})}{\partial{\bm{D}}}\left[\sum_{k=1}^{K}\frac{d\operatorname{softmax}_{\tau}({\bm{Y}}_{{\bm{\theta}},{\bm{D}},k})}{d{\bm{\theta}}}\right]=:\;\widehat{\nabla}_{\mathrm{GR},\,\tau}({\bm{D}},{\bm{\theta}}) (6) where the expectation is computed via a Monte-Carlo approximation with K samples of the random variable {\bm{Y}}_{{\bm{\theta}},{\bm{D}}}:={\bm{\theta}}+{\bm{G}}\,\big|\,{\bm{D}} . Note that this Monte-Carlo approximation means that the Gumbel-Rao estimator (and our proposed estimators which also use the Monte-Carlo approximation) is generally around three times slower than the other estimators. Crucially, sampling from this conditional distribution is made possible by the following reparameterisation ( 11 ; 12 ; 15 ) {\bm{\theta}}_{j}+{\bm{G}}_{j}\,\big|\,({\bm{D}}={\bm{I}}_{i})\;\overset{d}{=}\;\begin{cases}-\log(E_{j})+\log Z({\bm{\theta}}),&\text{if }j=i,\\[6.0pt] -\log\!\left(\dfrac{E_{j}}{\exp({\bm{\theta}}_{j})}+\dfrac{E_{i}}{Z({\bm{\theta}})}\right),&\text{otherwise}\end{cases} (7) where E_{j}\sim\exp(1) i.i.d. and Z({\bm{\theta}})=\sum_{i=1}^{n}\exp(\theta_{i}) . It is important to note that in practice, the implementation of Gumbel-Rao differs from equation 6 . Specifically, the \frac{d\operatorname{softmax}_{\tau}({\bm{Y}}_{{\bm{\theta}},{\bm{D}},k})}{d{\bm{\theta}}} term is implemented as \frac{d\operatorname{softmax}_{\tau}({\bm{Y}}_{{\bm{\theta}},{\bm{D}},k})}{d{\bm{Y}}_{{\bm{\theta}},{\bm{D}},k}} which ignores the derivative through the conditional reparameterisation. It is unclear from the 14 paper whether this is the intended behaviour, but it is the approach taken by the current publicly available implementations ( 13 ; 3 ) and works well in practice so we follow it for our estimators. 2.4 The ReinMax estimator ( 8 ) Returning to equation 2 , 8 show that by considering baseline subtraction with the baseline E_{{\bm{D}}\sim\mbox{\small$\mathrm{softmax}$}({\bm{\theta}})}[f({\bm{D}})] , it can be equivalently expressed as \displaystyle\nabla=\sum_{i=1}^{n}\sum_{j=1}^{n}{\bm{\pi}}_{j}(f({\bm{I}}_{i})-f({\bm{I}}_{j}))\frac{d\,{\bm{\pi}}_{i}}{d\,{\bm{\theta}}}. (8) From this expression, equation 8 is manipulated into a more

Read the complete text