Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion
Vinh Tong, Trung-Dung Hoang, Anji Liu, Guy Van den Broeck, Mathias Niepert · 2025
Read Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion by Vinh Tong, Trung-Dung Hoang, Anji Liu, Guy Van den Broeck, Mathias Niepert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09890.txt.
Introduction
Vinh Tong and co-authors cut training noise in symmetry aware diffusion models in a 2025 paper. Molecule and protein generators must respect physical symmetries. Builders either hard-wire symmetry into networks, which complicates training, or fake it with data augmentation, which is noisy. The team reframes augmentation as a random estimate of the training gradient, then sharpens it with a classic Rao-Blackwell trick.
Variance drops provably, with one pass per sample, they show. The loss still favors symmetric solutions. Their Orbit Diffusion system tops molecule shape benchmarks and improves crystal prediction, text guided crystal design, and protein design scores. Code is public.
Catalog record
| CREATOR | Vinh Tong, Trung-Dung Hoang, Anji Liu, Guy Van den Broeck, Mathias Niepert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 113 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09890
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion Vinh Tong, Trung-Dung Hoang, Anji Liu, Guy Van den Broeck, Mathias Niepert Abstract In domains such as molecular and protein generation, physical systems exhibit inherent symmetries that are critical to model. Two main strategies have emerged for learning invariant distributions: designing equivariant network architectures and using data augmentation to approximate equivariance. While equivariant architectures preserve symmetry by design, they often involve greater complexity and pose optimization challenges. Data augmentation, on the other hand, offers flexibility but may fall short in fully capturing symmetries. Our framework enhances both approaches by reducing training variance and providing a provably lower-variance gradient estimator. We achieve this by interpreting data augmentation as a Monte Carlo estimator of the training gradient and applying Rao–Blackwellization. This leads to more stable optimization, faster convergence, and reduced variance, all while requiring only a single forward and backward pass per sample. We also present a practical implementation of this estimator—incorporating the loss and sampling procedure—through a method we call Orbit Diffusion . Theoretically, we guarantee that our loss admits equivariant minimizers. Empirically, Orbit Diffusion achieves state-of-the-art results on GEOM-QM9 for molecular conformation generation, improves crystal structure prediction, and advances text-guided crystal generation on the Perov-5 and MP-20 benchmarks. Additionally, it enhances protein designability in protein structure generation. Code is available at https://github.com/vinhsuhi/Orbit-Diffusion.git . 1 Introduction Diffusion models have emerged as powerful methods for modeling complex distributions ( Ho et al., 2020 ; Song et al., 2021a ; Song et al., 2021b ; Karras et al., 2022 ) , with applications in domains such as molecular and protein generation ( Zhang et al., 2023 ; Vignac et al., 2023 ; Anand & Achim, 2022 ) . Many physical systems, such as molecules or crystals, exhibit inherent symmetries. For example, a molecule’s physical properties remain unchanged under rotations in 3D space ( Hoogeboom et al., 2022 ; Jing et al., 2022 ) . Modeling such data requires learning distributions that are invariant under the action of a group G . This setting is naturally captured by the notion of \emph{$G$-invariant} distribution q(x_{0}) that are invariant under transformations from G ( Chen et al., 2024 ; Köhler et al., 2020 ) . Two main strategies have emerged for learning \emph{$G$-invariant} distributions: (1) designing equivariant network architectures, and (2) using data augmentation to approximate equivariance. Equivariant architectures, such as equivariant denoisers, ensure symmetry by construction ( Hoogeboom et al., 2022 ; Klein et al., 2024 ; Igashov et al., 2024 ) , but are often less efficient due to increased architectural complexity and can pose optimization challenges ( Brehmer et al., 2024 ; Abbe & Boix-Adserà, 2022 ) . In contrast, data augmentation is a flexible and widely used alternative that approximates equivariant training by sampling transformed versions of the input. While this approach scales easily and has become increasingly popular in the community, especially for large-scale models ( Abramson et al., 2024b ; Geffner et al., 2025 ) , its effectiveness in capturing symmetries may vary depending on the group and application domain. In some settings, such as molecular dynamics and structural biology, explicit equivariance remains beneficial ( Anand & Achim, 2022 ; Batatia et al., 2022 ; Zaverkin et al., 2024 ) . In this work, we propose a framework that provably improves both approaches: we introduce a novel form of implicit data augmentation by computing the denoising target as a weighted average over group orbits, which reduces variance and improves the training of equivariant denoisers. We revisit data augmentation from a principled perspective and interpret it as a Monte Carlo estimator of the gradient of a symmetrized loss. This loss is defined over a fully symmetrized dataset, which yields an empirical distribution invariant under G ( Chen et al., 2024 ) . While traditional data augmentation uses one or few samples from this augmented dataset, we instead apply Rao–Blackwellization to derive a new estimator with provably lower variance. Specifically, we decompose the gradient as an outer expectation over data and an inner conditional expectation over group actions. Replacing the noisy sample-based target with its conditional expectation yields a lower-variance gradient estimator, while preserving equivariance. To translate our theoretical insights into a practical method, we develop an efficient implementation of the proposed gradient estimator. This approach integrates the symmetrized loss and variance reduction into a modified training objective, without increasing the computational cost. Our implementation requires only a single forward and backward pass per sample, and it is compatible with both equivariant and non-equivariant architectures. We refer to this practical method as Orbit Diffusion . By implicitly applying Rao–Blackwellization through a tailored loss formulation and sampling scheme, Orbit Diffusion enables stable optimization and improved generalization across a wide range of symmetry groups and tasks. We provide theoretical guarantees that our symmetrized loss admits equivariant minimizers and that our gradient estimator has strictly smaller variance compared to existing methods. Empirically, we demonstrate strong performance across multiple domains. Our method achieves state-of-the-art results on GEOM-QM9 for molecular conformation generation, enhances crystal structure prediction, and improves text-guided crystal generation on the Perov-5 and MP-20 benchmarks. Moreover, our approach is compatible with non-equivariant denoisers; in particular, it improves the designability of protein structures generated by ProteiNA ( Geffner et al., 2025 ) . 2 Background Groups. A group is a mathematical structure comprising a set G and a binary operation m:G\times G\to G that combines two elements of G . A group action g\in G defines how the group G acts on a set \Omega , such as a set of geometric objects. 1 1 1 When the group acts on a vector space V , we do not distinguish between the abstract group element g\in G and its linear representation \rho(g):G\to\mathrm{GL}(V) . For simplicity, we write g\circ x to denote the action \rho(g)(x) . We restrict our attention to locally compact isometry groups . Locally compact isometry groups encompass a broad class of transformation groups that preserve distances and possess a well-behaved topological structure. Examples include the permutation group \mathsf{S}_{n} , the orthogonal group \mathsf{O}(d) , and the special orthogonal group \mathsf{SO}(d) . Invariance and Equivariance. A function f:\Omega\to\mathbb{R} is said to be G -invariant if for all g\in G and x\in\Omega , it satisfies f(g\circ x)=f(x) . This means the function value does not change under the action of any group element. A function f:\Omega\to\Omega is said to be G -equivariant if for all x\in\Omega , f commutes with any group action g\in G : f(g\circ x)=g\circ f(x) . Invariant and Equivariant Distributions. A probability distribution p(x) defined on a set \Omega is said to be \emph{$G$-invariant} under the action of a group G if the probability of any measurable subset A\subseteq\Omega remains unchanged under the transformation induced by any group action g\in G:p(g\circ x\in A)=p(x\in A). A conditional distribution p(y\mid x) , where x\in\Omega and y\in\Omega , is said to be \emph{$G$-equivariant} if for all g\in G , the following condition holds: p(g\circ y\mid g\circ x)=p(y\mid x). . Group Symmetrization. Let S_{G} be the symmetrization operator under group G , transforming any distribution p(x) into a \emph{$G$-invariant} -invariant distribution, denoted as the \emph{$G$-symmetrized} distribution: S_{G}[p](x):=\int_{G}p(g\circ x)\,\mathrm{d}\mu_{G}(g),\quad\text{ where }\mu_{G}\text{ is the Haar measure on }G. (1) Diffusion