Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems

Mads H. Baattrup, Jörn Bach, Laurids Jeppe, Finn Labe, Alexander Grohsjean, Christian Schwanenberger, Peer Stelldinger · 2026

Read Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems by Mads H. Baattrup, Jörn Bach, Laurids Jeppe, Finn Labe, Alexander Grohsjean, Christian Schwanenberger, Peer Stelldinger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22891.txt.

Introduction

Baattrup and six colleagues argue that pointwise metrics mislead evaluation of multimodal inverse problems. Such problems have posteriors with several modes. Estimators that minimize squared or absolute error return spectra narrower than the truth. They compress the tails and shapes that downstream science needs. The authors propose a three-part protocol. It covers per-event distributional accuracy, population-level spectrum fidelity, and calibration. On synthetic and particle-physics problems, rankings flipped between pointwise and distributional metrics.

Catalog record

CREATORMads H. Baattrup, Jörn Bach, Laurids Jeppe, Finn Labe, Alexander Grohsjean, Christian Schwanenberger, Peer Stelldinger
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90282
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems Mads H. Baattrup, Jörn Bach, Laurids Jeppe, Finn Labe, Alexander Grohsjean, Christian Schwanenberger, Peer Stelldinger Abstract Evaluation in scientific reconstruction is dominated by pointwise metrics – RMSE, MAE, per-event resolution – under the implicit assumption that lower error means better reconstruction. We show that this assumption fails structurally for inverse problems with multimodal posteriors. By the law of total variance, point estimators trained to minimize MSE or MAE produce a marginal spectrum strictly narrower than the truth whenever the posterior has nonzero width. The resulting bias is independent of architecture, training, and dataset size, and it compresses precisely the spectral features – tails, modes, shapes – that downstream scientific measurements rely on. We propose a three-part evaluation protocol where each step targets a failure mode the others miss: per-event distributional accuracy via CRPS, population-level marginal accuracy via a spectrum-fidelity diagnostic, and uncertainty trustworthiness via coverage-based calibration. On a synthetic benchmark with an analytic posterior and on a realistic many-to-one inverse problem from particle physics, model rankings reverse between pointwise and distributional metrics, and calibration further separates architectures indistinguishable under CRPS. The evaluation protocol, not the model, determines the scientific conclusion. 1 Introduction and Scope In scientific reconstruction, evaluation is not a neutral postprocessing step: the metrics we report determine which models advance and which physical conclusions hold. Across particle physics, medical imaging, and geophysics, evaluation is dominated by pointwise resolution metrics (root-mean-squared-error (RMSE), mean-absolute-error (MAE), per-event bias) that ask how close each prediction lies to the truth. We show that this convention fails structurally for under-constrained inverse problems with multimodal posteriors. The optimal predictor under mean-squared-error (MSE) is the conditional expectation [ 6 ] , which for multimodal posteriors can fall between modes in regions of vanishing probability density. Models achieving lower MSE do so by collapsing the posterior more aggressively, producing predictions that are individually “unphysical” and collectively distort reconstructed spectra. This is an evaluation oversight rather than a model failure. We present the following contributions: 1. A theoretical argument ( section 3.2 ) showing that any point estimator minimizing MSE or MAE produces a marginal spectrum strictly narrower than the truth whenever the posterior has nonzero variance – a bias independent of architecture, training objective, and dataset size. 2. A three-part evaluation protocol – per-event distributional accuracy, population-level spectrum fidelity, and coverage-based calibration – that diagnoses the failure modes pointwise metrics miss and applies across regression, mixture, and generative model families on a common scale. 3. An empirical demonstration on a controlled synthetic benchmark with an analytic posterior and on a realistic many-to-one inverse problem from particle physics, showing that model rankings reverse between pointwise and distributional metrics, and that calibration further separates architectures indistinguishable under the per-event distributional accuracy metric. Scope We consider fully-supervised inverse problems where paired latents z and observations x are available via simulation, and the downstream quantity of interest is the marginal p(z) over a dataset rather than a posterior over global parameters. Ground-truth z is assumed accessible at evaluation time. We do not address joint posterior structure across high-dimensional z , domain shift between training and deployment, or unsupervised settings. Our analysis is otherwise domain-agnostic: any inverse problem with non-negligible posterior variance falls within scope. 2 Background and Related Work Fully-Supervised Inverse Problems We consider the task of recovering a latent state z\in\mathbb{R}^{n} from noisy observations x\in\mathbb{R}^{m} related by a forward model {x}=G({z},\xi) , where \xi represents measurement noise or physical stochasticity – such a pair, (z,x) , is commonly called an event or a sample . In scientific domains – particle reconstruction, seismic imaging, medical tomography – G is a high-fidelity simulator that is analytically intractable to invert. When a large corpus of paired (z,x) samples is available, the task becomes a fully-supervised inverse problem : learn a surrogate G^{\dagger} approximating the posterior p(z\mid x) . Such problems are frequently ill-posed, admitting multiple or no exact solutions, either through the stochasticity of \xi or through information loss in {x} . Evaluation in Scientific Reconstruction Because scientific reconstruction is typically performed event-by-event, evaluation has traditionally followed the same structure, dominated by RMSE, MAE, or resolution and bias of the per-event error distribution. This convention appears universal: RMSE and MAE are the headline metrics in particle reconstruction [ 18 , 46 ] , sparse-view computed tomography [ 47 ] , image super-resolution benchmarks [ 54 ] , climate downscaling [ 52 ] , and geophysical inversion [ 32 ] and are routinely adopted as the training objective itself. Lower error is implicitly assumed to mean better reconstruction (also when aggregated) – true for well-constrained unimodal problems, but not when the conditional posterior has nonzero width ( section 3 ). Point predictions are also commonly reported, even when the underlying method produces a full posterior: generative models for gravitational-wave parameter estimation are routinely summarized by medians and 90% intervals [ 45 , 15 ] , and normalizing flows for neutrino kinematics are ultimately evaluated via point estimates extracted from the posterior [ 43 ] . This “generative-to-regressive” bottleneck persists because the community lacks a structured evaluation protocol that rewards preservation of posterior structure. Connection to Simulation-Based Inference (SBI) Learning the surrogate, G^{\dagger} , is an instance of amortized neural posterior estimation (NPE) [ 13 , 38 ] . Our setting differs from canonical SBI in two respects: the latent z is per-event and amortized over \mathcal{O}(10^{4}\text{-}10^{6}) events, and the scientific quantity of interest is the aggregated marginal p(z) rather than a posterior over global parameters. This motivates an evaluation protocol targeting both per-event accuracy and population-level fidelity. Proper Scoring Rules, Calibration, and Evaluation Principles Evaluation of probabilistic predictions requires metrics that reward faithful distributions rather than sharp point predictions. A scoring rule S(F,z) is proper if its expectation under the true distribution is minimized by F equal to that distribution, and strictly proper if uniquely so [ 23 ] . Propriety distinguishes honest probabilistic evaluation from metrics that can be “gamed” by posterior collapse. The continuous ranked probability score (CRPS) is the standard proper scoring rule for univariate distributional prediction, developed in the probabilistic weather forecasting literature [ 23 , 28 ] . For a predictive cumulative distribution F and an observed value z , it is defined as: \mathrm{CRPS}(F,z)=\int_{-\infty}^{\infty}\left[F(t)-\mathbf{1}(t\geq z)\right]^{2}\mathrm{d}t, (1) where \mathbf{1}(\cdot) denotes the indicator function. When the predictive distribution is represented by N posterior samples \{\hat{z}^{k}\}_{k=1}^{N} , the CRPS can be estimated in \mathcal{O}(N\log N) by sorting [ 31 ] and the explicit form is given in section A.1 . Two properties are particularly relevant for this setting. First, CRPS reduces to the MAE when F=\delta(\hat{z}) is a Dirac distribution at a point prediction \hat{z} , making it directly applicable to both generative and regression models on a common scale. Second, the CRPS is reported per-event and then averaged providing the same granularity as RMSE or MAE so dataset-averaged CRPS can be reported as a drop-in replacement for standard pointwise resolution metrics. Beyond proper scoring, calibration assesses whether nominal credible regions contain the true value at their stated frequencies [ 27 , 51 , 26 , 33 ] ; we use a conformal variant [ 21 , 53 , 1 ]

Read the complete text