On the role of memorization in learned priors for geophysical inverse problems

Ali Siahkoohi and Davide Sabeddu University of Central Florida · 2026

Read On the role of memorization in learned priors for geophysical inverse problems by Ali Siahkoohi and Davide Sabeddu University of Central Florida (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19629.txt.

Introduction

Siahkoohi and Sabeddu ask what memorization does to learned priors for seismic inverse problems. Good training sets of subsurface models are scarce in geoscience. A generative prior may then just memorize its training examples. The authors show the posterior then acts as a weighted lookup over those stored examples. For diffusion models, the prior takes a closed Gaussian mixture form. A local linear step gives a matching mixture for the posterior. Tests on a toy problem and on full waveform inversion bear this out.

Catalog record

CREATORAli Siahkoohi and Davide Sabeddu University of Central Florida
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22315
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19629

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the role of memorization in learned priors for geophysical inverse problems Ali Siahkoohi and Davide Sabeddu University of Central Florida Abstract Learned priors based on deep generative models offer data-driven regularization for seismic inversion, but training them requires a dataset of representative subsurface models—a resource that is inherently scarce in geoscience applications. Since the training objective of most generative models can be cast as maximum likelihood on a finite dataset, any such model risks converging to the empirical distribution—effectively memorizing the training examples rather than learning the underlying geological distribution. We show that the posterior under such a memorized prior reduces to a reweighted empirical distribution—i.e., a likelihood-weighted lookup among the stored training examples. For diffusion models specifically, memorization yields a Gaussian mixture prior in closed form, and linearizing the forward operator around each training example gives a Gaussian mixture posterior whose components have widths and shifts governed by the local Jacobian. We validate these predictions on a stylized inverse problem and demonstrate the consequences of memorization through diffusion posterior sampling for full waveform inversion. Code to partially reproduce the results is available at github.com/luqigroup/mempost . \righthead Memorization in learned priors for geophysical inversion 1 Introduction Seismic inversion is an ill-posed inverse problem whose solution nonuniqueness and sensitivity to data noise make uncertainty quantification essential for reliable subsurface characterization ( Tarantola, 2005 ; Stuart, 2010 ) . A Bayesian framework addresses this need by representing the solution as a posterior distribution, where the prior encodes knowledge about plausible subsurface models. This formulation places the quality of the prior at the center of the inference ( Hosseini and Huang, 2026 ) . Conventional priors—e.g., Gaussian or sparsity-promoting Laplace distributions—offer analytical convenience but often fail to capture the complex structure of realistic geological models. To address this limitation, a growing body of work replaces handcrafted priors with learned alternatives based on generative models ( Mosser et al., 2020 ; Zhang and Curtis, 2021 ; Siahkoohi et al., 2022 ; Zhao et al., 2022 ; Zhang et al., 2023 ; Yin et al., 2024 ; Zhao and Curtis, 2025 ; Siahkoohi et al., 2026 ) . Among these, score-based diffusion models ( Song and Ermon, 2019 ; Ho et al., 2020 ) —a class of generative models that learn to reverse a gradual noising process in order to draw samples from a target distribution—have received considerable attention ( Erdinc et al., 2025 ; Ravasi, 2025 ; Zeng et al., 2026 ) . Once trained, the learned score function can guide posterior sampling algorithms ( Chung et al., 2023 ; Song et al., 2023 ) , providing data-driven regularization that reflects the statistical structure of geological model datasets. While these approaches have shown promise in controlled settings, deploying learned diffusion priors for geoscience inverse problems introduces a fundamental challenge: high-fidelity subsurface models derived from well logs or high-resolution imaging are inherently scarce, and synthetic alternatives based on geostatistical simulations are often overly simplistic relative to real geological complexity. Augmenting the dataset with samples synthesized by a generative model trained on the same limited data is equally ineffective: iteratively retraining on model-generated data leads to progressive quality degradation—a phenomenon known as model autophagy ( Alemohammad et al., 2024 ) . As a result, the ratio of available training examples to model dimensionality remains small, raising the risk of memorization—i.e., the generative model reproduces training examples rather than learning the underlying geological distribution. Specifically, in the context of diffusion models, Baptista et al., 2025 show that the minimizer of the empirical denoising score matching loss is the score of a Gaussian mixture centered on the training examples, and the number of training samples required to avoid memorization grows exponentially with data dimensionality ( Biroli et al., 2024 ) —a threshold unlikely to be met for high-dimensional problems. Since prior misspecification propagates directly into the posterior ( Hosseini and Huang, 2026 ) , understanding the consequences of memorization for posterior inference is essential. In this work, we characterize these consequences for geophysical inverse problems. Our contributions are threefold: (i) we show that when a generative model memorizes its training data, the posterior reduces to a lookup-table—i.e., a likelihood-weighted selection among stored training examples; (ii) for diffusion models, the memorized prior takes the form of a Gaussian mixture, and linearizing the forward operator around each training example yields a closed-form Gaussian mixture posterior whose component means are shifted by the adjoint Jacobian and whose covariances balance prior constraint against data sensitivity; and (iii) we validate these predictions analytically on a stylized inverse problem and demonstrate the practical consequences through diffusion posterior sampling for frequency-domain full waveform inversion (FWI). In the following sections, we formalize the Bayesian inverse problem with a learned prior, characterize the failure mode of memorization, derive the linearized posterior in closed form, and conclude with numerical illustrations. 2 Inverse problems with learned priors We are concerned with Bayesian inference for ill-posed inverse problems in which the prior distribution is learned from a finite dataset. Consider a nonlinear forward operator \mathbf{F}:\mathbb{R}^{d}\to\mathbb{R}^{m} relating the model parameters \mathbf{x} to observed data \mathbf{y} via \mathbf{y}=\mathbf{F}(\mathbf{x})+\boldsymbol{\eta} , where \boldsymbol{\eta}\sim\mathcal{N}(\mathbf{0},\,\gamma^{2}\mathbf{I}) and \gamma^{2} is the noise variance. In FWI, for instance, \mathbf{x} is the subsurface velocity model and \mathbf{F} encapsulates the solution of the wave equation together with the observation operator. The Bayesian solution is the posterior distribution obtained via Bayes’ rule: p(\mathbf{x}\mid\mathbf{y})\propto\underbrace{p(\mathbf{y}\mid\mathbf{x})}_{\text{likelihood}}\;\underbrace{p(\mathbf{x})}_{\text{prior}}, (1) where the likelihood is p(\mathbf{y}\mid\mathbf{x})=\mathcal{N}(\mathbf{y}\mid\mathbf{F}(\mathbf{x}),\,\gamma^{2}\mathbf{I}) . Learned priors replace handcrafted distributional assumptions with a parametric model p_{\theta}(\mathbf{x}) trained on a dataset \{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\} by maximum likelihood estimation. Since the true data distribution p_{\mathrm{data}} is accessible only through the empirical distribution \hat{p}_{N}=\frac{1}{N}\sum_{n=1}^{N}\delta(\mathbf{x}-\mathbf{x}_{n}) , the quality of the learned prior depends on whether the model generalizes beyond the training examples. When it does not—i.e., when p_{\theta} converges to \hat{p}_{N} —we show below that the posterior inherits the discrete structure of the empirical distribution. 2.1 The empirical prior and the lookup-table posterior Most deep generative models used as learned priors—including normalizing flows ( Rezende and Mohamed, 2015 ) , variational autoencoders ( Kingma and Welling, 2014 ) , and diffusion models ( Ho et al., 2020 ; Song et al., 2021 ) —are trained by maximum likelihood estimation or a lower bound thereof, both equivalent to minimizing a Kullback–Leibler divergence between the data and model distributions ( Kingma and Gao, 2023 ) . For a sufficiently expressive model class and without sufficient regularization—whether explicit or through architectural inductive bias—the Kullback–Leibler divergence between \hat{p}_{N} and p_{\theta} is minimized when the two distributions coincide, yielding p_{\theta}(\mathbf{x})=\hat{p}_{N}(\mathbf{x})=\frac{1}{N}\sum_{n=1}^{N}\delta(\mathbf{x}-\mathbf{x}_{n}) . Substituting this memorized prior into Equation 1 yields p(\mathbf{x}\mid\mathbf{y})=\sum_{n=1}^{N}w_{n}\,\delta(\mathbf{x}-\mathbf{x}_{n}),\ w_{n}\propto\exp\!\left(-\frac{\|\mathbf{F}(\mathbf{x}_{n})-\mathbf{y}\|_{2}^{2}}{2\gamma^{2}}\right). (2) The posterior is a lookup table: discrete probability mass on each training example, weighted by data fit. The resulting uncertainty reflects which stored models best explain the data, not geological variability. This failure mode applies to any generative model trained with maximum likelihood estimation. 2.2 Memorized diffusion models as Gaussian mixture priors For diffusion models, in particular, Baptista et al., 2025 establish a concrete characterization of memorization. The forward diffusion process maps each data sample \mathbf{x}_{0} to \mathbf{x}_{t}\sim\mathcal{N}(m(t)\,\mathbf{x}_{0},\,\sigma^{2}(t)\,\mathbf{I}) , where m(t) and \sigma(t) control the signal attenuation and noise level at each step of the forward process. The score matching loss trains the model to

Read the complete text