Probabilistic Gaussian Homotopy: A Probability-Space Continuation Framework for Nonconvex Optimization
Eshed Gal, Samy Wu Fung, Eldad Haber · 2026
Read Probabilistic Gaussian Homotopy: A Probability-Space Continuation Framework for Nonconvex Optimization by Eshed Gal, Samy Wu Fung, Eldad Haber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13546.txt.
Introduction
Eshed Gal, Samy Wu Fung, and Eldad Haber present Probabilistic Gaussian Homotopy, a new take on nonconvex optimization. Classic Gaussian homotopy smooths the objective and averages gradients in a uniform way. Their method instead deforms the Boltzmann distribution, so perturbed gradients are weighted toward low-energy regions. They show it matches a log-sum-exp soft-min homotopy that returns to the true objective as the scale goes to zero. A dynamical system tracks how minimizers move along the annealed path. Their PGHO algorithm did well on benchmarks.
Catalog record
| CREATOR | Eshed Gal, Samy Wu Fung, Eldad Haber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36918 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probabilistic Gaussian Homotopy: A Probability-Space Continuation Framework for Nonconvex Optimization Eshed Gal, Samy Wu Fung, Eldad Haber Abstract We introduce Probabilistic Gaussian Homotopy (PGH), a probability-space continuation framework for nonconvex optimization. Unlike classical Gaussian homotopy, which smooths the objective and uniformly averages gradients, PGH deforms the associated Boltzmann distribution and induces Boltzmann-weighted aggregation of perturbed gradients, which exponentially biases descent directions toward low-energy regions. We show that PGH corresponds to a log-sum-exp (soft-min) homotopy that smooths a nonconvex objective at scale λ > 0 \lambda>0 and recovers the original objective as λ → 0 \lambda\to 0 , yielding a posterior-mean generalization of the Moreau envelope, and we derive a dynamical system governing minimizer evolution along an annealed homotopy path. This establishes a principled connection between Gaussian continuation, Bayesian denoising, and diffusion-style smoothing. We further propose Probabilistic Gaussian Homotopy Optimization (PGHO), a practical stochastic algorithm based on Monte Carlo gradient estimation, and demonstrate strong performance on high-dimensional nonconvex benchmarks and sparse recovery problems where classical gradient methods and objective-space smoothing frequently fail. 1 Introduction Many problems in science and engineering can be formulated as the minimization of a highly nonconvex but differentiable objective \min_{x\in\mathbb{R}^{n}}f(x), (1) where the landscape of f may contain numerous local minima, flat regions, and narrow basins of attraction. While local optimization methods are effective when initialized near a desirable solution, they often fail in strongly nonconvex settings. Global strategies such as simulated annealing [ 1 ] , evolutionary methods [ 23 ] , or multi-start schemes [ 21 ] attempt to address this challenge, but frequently suffer from slow convergence or poor scalability in high dimensions. A classical approach for handling nonconvexity is homotopy continuation [ 11 ] . Rather than solving the difficult problem directly, one constructs a family of progressively less smoothed objectives and tracks minimizers as the smoothing is removed. Gaussian homotopy methods smooth the objective via convolution with a Gaussian kernel and gradually decrease the smoothing parameter [ 24 , 25 ] . Although effective in certain regimes, these methods operate in objective space and lack a direct connection to modern probabilistic modeling. In parallel, score-based diffusion models have emerged as a powerful framework for high-dimensional generative modeling. These models define a family of progressively less noisy probability distributions and transport samples from a simple base density to a complex target distribution through a reverse-time stochastic differential equation. While primarily viewed as sampling algorithms, diffusion models implicitly define a Gaussian homotopy in probability space . Figure 1: Gaussian homotopy for the function f(x_{1},x_{2})=\log(Rosen(x_{1},x_{2})+\alpha(1+\sin(3x_{1})\sin(3x_{2}))) , where Rosen denotes the banana Rosenbrock function. We estimate the homotopy-smoothed objective by Monte Carlo sampling. The surface begins nearly flat and gradually deforms into a highly multimodal landscape that is much harder to optimize. This observation motivates the central idea of this work. Minimizing an objective f(x) is equivalent to maximizing its associated Boltzmann density p(x)\propto\exp\!\left(-\frac{1}{\lambda}f(x)\right). (2) Instead of smoothing f directly, we smooth its induced probability distribution and track the evolution of high-probability regions as the smoothing vanishes. This yields a continuation method performed in probability space , fundamentally distinct from classical objective-space homotopy. We formalize this idea through Probabilistic Gaussian Homotopy (PGH) , which constructs a time-dependent family of Gaussian-smoothed Boltzmann densities via a soft-min (log-sum-exp) aggregation of perturbed objective values. As the smoothing decreases, the associated energy deforms continuously from a nearly flat landscape to the original nonconvex objective (see Figure 1 ). Minimizers are tracked along this path using a gradient-driven dynamical system. The construction admits a natural Bayesian interpretation. Classical Moreau envelope smoothing corresponds to computing a Maximum A Posteriori (MAP) estimator under Gaussian perturbations. In contrast, PGH follows the posterior mean, which can be expressed via Tweedie’s formula [ 16 , 15 ] as a gradient of the log-marginal density. Replacing hard inner minimization with probabilistic averaging yields a tractable stochastic gradient estimator that aggregates information from a Gaussian neighborhood, biasing descent directions toward low-energy regions rather than relying solely on local curvature. Contributions. Our contributions are summarized as follows. • We reinterpret diffusion-style Gaussian smoothing as a homotopy mechanism for optimization, which establishes a bridge between classical Gaussian homotopy, Moreau envelope smoothing, and score-based diffusion. • We introduce Probabilistic Gaussian Homotopy (PGH) , a probabilistic continuation framework that smooths the Boltzmann distribution associated with an objective rather than the objective itself. • We establish a rigorous connection between this construction and Moreau envelope smoothing, showing that PGH corresponds to a posterior-mean (soft-min) variant of classical Moreau regularization. • We derive an ordinary differential equation governing the evolution of minimizers along the homotopy and propose practical numerical schemes for tracking this path. In summary, we introduce a principled probabilistic continuation framework that connects Bayesian denoising, diffusion dynamics, and global nonconvex optimization. 2 Background We briefly review three fundamental concepts which underlies our framework: Gaussian homotopy in optimization, Moreau envelope smoothing, and score-based diffusion. 2.1 Gaussian Homotopy in Objective Space Homotopy continuation methods address nonconvex optimization by embedding a difficult objective into a family of progressively less smoothed surrogates. Given f:\mathbb{R}^{n}\to\mathbb{R} , classical Gaussian homotopy constructs F_{\sigma}(x)=(f*G_{\sigma})(x)=\int_{\mathbb{R}^{n}}f(y)\,G_{\sigma}(x-y)\,dy, (3) where G_{\sigma} is a Gaussian kernel with variance \sigma^{2} . Equivalently, F_{\sigma}(x)=\mathbb{E}_{z\sim\mathcal{N}(0,I)}[f(x+\sigma z)]. (4) For large \sigma , F_{\sigma} is smooth and often easier to optimize; as \sigma\to 0 , F_{\sigma}\to f . The continuation strategy tracks minimizers x^{\star}(\sigma)\in\arg\min F_{\sigma}(x) while gradually decreasing \sigma . Gradients are given by \nabla F_{\sigma}(x)=\mathbb{E}_{z}[\nabla f(x+\sigma z)], (5) so descent directions correspond to a uniform average of gradients in a Gaussian neighborhood. While effective in certain regimes, this formulation operates directly in objective space and averages function values and gradients uniformly . As a result, competing basins contribute equally to the descent direction, which may potentially blur structural differences between low- and high-energy regions. This observation motivates exploring continuation directly in probability space rather than in objective space. 2.2 Moreau Envelope and Soft-Min Smoothing Another classical smoothing construction is the Moreau envelope. For \lambda>0 , the Moreau envelope of f is defined as M_{\lambda}f(x)=\min_{y}\left\{f(y)+\frac{1}{2\lambda}\|x-y\|^{2}\right\}. (6) The envelope replaces f by a locally regularized minimization problem. As \lambda\to 0 , M_{\lambda}f(x)\to f(x) , while for larger \lambda the envelope becomes smoother and may reduce nonconvexity. A useful probabilistic perspective arises from the soft-min (log-sum-exp) operator: \mathrm{softmin}_{\lambda}(g)=-\lambda\log\int\exp\!\left(-\frac{1}{\lambda}g(y)\right)dy. (7) By Laplace’s principle, \mathrm{softmin}_{\lambda}(g) approximates \min_{y}g(y) as \lambda\to 0 . Applying this to g(y)=f(y)+\frac{1}{2\lambda}\|x-y\|^{2} (8) yields a smoothed approximation to the Moreau envelope that replaces hard minimization by probabilistic averaging. This observation provides a bridge between convexification methods and probabilistic smoothing. 2.3 Score-Based Diffusion and Gaussian Smoothing Score-based diffusion models construct a family of probability densities \{p_{t}\}_{t\in[0,1]} by progressively adding Gaussian noise to data. A common forward noising process takes the form x_{t}=\alpha(t)x_{1}+\beta(t)\varepsilon,\quad\varepsilon\sim\mathcal{N}(0,I), (9) where \beta(t) increases with t . Sampling is performed by integrating a reverse-time stochastic differential equation involving the score \nabla_{x}\log p_{t}(x) . From a geometric viewpoint, diffusion defines a Gaussian homotopy in probability space: the density evolves continuously from a simple Gaussian to a complex target distribution. While diffusion models are typically interpreted as sampling algorithms, their structure naturally induces a continuation path across progressively de-smoothed distributions. These three perspectives based on objective-space Gaussian homotopy, Moreau smoothing, and diffusion-based Gaussian perturbation, form the conceptual basis for the probabilistic homotopy