Neural Posterior Unfolding

Fernando Torales Acosta, Jay Chan, Krish Desai, Vinicius Mikuni, Benjamin Nachman, Jingjing Pan, Francesco Rubbo · 2025

Read Neural Posterior Unfolding by Fernando Torales Acosta, Jay Chan, Krish Desai, Vinicius Mikuni, Benjamin Nachman, Jingjing Pan, Francesco Rubbo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06370.txt.

Introduction

Undoing a detector's smearing with a learned distribution over outcomes is the method Neural Posterior Unfolding, introduced in a 2025 paper led by Fernando Torales Acosta with six co-authors. Cross-section measurements are particle physics currency, but detectors distort them, and classic unfolding inverts a response matrix with regularization. The new approach uses normalizing flows to estimate the full posterior.

Benefits include regularization built into the network, fast reuse without retraining, and full error bars on the unfolded result. The authors also rebuild fully Bayesian unfolding in modern Python. Gaussian toy cases and mock LHC jet substructure validate both. The Bayesian methods prove effective, and the team argues they merit more work toward real analyses.

Catalog record

CREATORFernando Torales Acosta, Jay Chan, Krish Desai, Vinicius Mikuni, Benjamin Nachman, Jingjing Pan, Francesco Rubbo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06370

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Posterior Unfolding Fernando Torales Acosta, Jay Chan, Krish Desai, Vinicius Mikuni, Benjamin Nachman, Jingjing Pan, Francesco Rubbo Abstract Differential cross section measurements are the currency of scientific exchange in particle and nuclear physics. A key challenge for these analyses is the correction for detector distortions, known as deconvolution or unfolding. Binned unfolding of cross section measurements traditionally rely on the regularized inversion of the response matrix that represents the detector response, mapping pre-detector (‘particle level’) observables to post-detector (‘detector level’) observables. In this paper we introduce Neural Posterior Unfolding, a modern, Bayesian approach that leverages normalizing flows for unfolding. By using normalizing flows for neural posterior estimation, NPU offers several key advantages including implicit regularization through the neural network architecture, fast amortized inference that eliminates the need for repeated retraining, and direct access to the full uncertainty in the unfolded result. In addition to introducing NPU, we implement a classical Bayesian unfolding method called Fully Bayesian Unfolding (FBU) in modern Python so it can also be studied. These tools are validated on simple Gaussian examples and then tested on simulated jet substructure examples from the Large Hadron Collider (LHC). We find that the Bayesian methods are effective and worth additional development to be analysis ready for cross section measurements at the LHC and beyond. I Introduction Cross section measurements form the critical interface between theoretical predictions and experimental observations in particle and nuclear physics. Given some observables (what data scientists might call ‘features’), theorists attempt to predict and experimentalists measure how often a given reaction produces a certain value of the observable. Together, they use these inputs to infer properties of particle and nuclear physics models. In practice, theoretical calculations produce predictions at the particle level—the intrinsic, pre–detector distributions of physical observables—while experiments record data at the detector level, where measurements are distorted by detector effects such as resolution and acceptance limits and inefficiencies. Unfolding (also known as deconvolution) is the process of reconstructing the true particle level distributions from these distorted detector-level observations. Thousands of unfolded cross section measurements have been performed, nearly all of them using classical methods Adye 2011 . Most of these methods discretize the problem, representing the differential cross section as a histogram. While there is a growing interest in using machine learning for unbinned unfolding Arratia et al. 2021 ; Huetsch et al. 2024 , we focus here on the (currently) more prevalent binned methods. Frequentist methods that employ maximum likelihood estimation (and its approximations) for unfolding dominate particle and nuclear physics analyses Cowan 2002 ; Blobel 2011 ; Blobel 2013 ; Balasubramanian et al. 2019 . Directly maximizing the likelihood, however, is numerically unstable, especially when detector distortions are non-negligible, because the detector response kernel is highly singular. As a result, all unfolding methods perform a variation of regularized maximum likelihood estimation to prevent small fluctuations in the the detector-level data leading to large changes in the predicted particle-level distributions. An alternative approach is to use Bayesian methods with a uniform prior. Fully Bayesian Unfolding Choudalakis 2012 has been used for a number of measurements Aad et al. 2014 ; Aad et al. 2015 ; Adriani et al. 2016 ; Aad et al. 2016a ; Aad et al. 2016b ; Aad et al. 2016c ; Aad et al. 2016d ; Aaboud et al. 2017 ; Aidala et al. 2019 ; Aad et al. 2022 ; Aad et al. 2023a ; Aad et al. 2023b , with a uniform prior and Markov Chain Monte Carlo (MCMC). Many of these measurements are based on pyFBU 1 1 1 https://github.com/pyFBU/ , a decade-old, no longer maintained library, and deprecated software including Theano Team et al. 2016 . One of our goals is to provide an updated version of Fully Bayesian Unfolding that makes use of current tools. We also explore an alternative approach to MCMC by using normalizing flows Rezende and Mohamed 2016 and amortized machine learning. The proposed method uses pairs of (truth, measured) histograms to learn p(t|m) , the probability density function of truth given measured. 2 2 2 Note that the causal structure is in the opposite direction. . Using this approach rather than MCMC offers the potential advantages of providing a function approximation of the likelihood and eliminating the need to rerun the algorithm when certain changes are applied to the data (e.g. addition of more data or determination statistical uncertainties with bootstrapping Efron 1979a ). Furthermore, as a neural network, normalizing flows are inherently regularized through the limited complexity of the architecture and training protocol. We call this new method Neural Posterior Unfolding (NPU). The remainder of the paper is organized as follows. Section II reviews the statistics of binned unfolding and outlines the challenges the inverse problem presents. Sec. III introduces and details the implementation of NPU using modern tools. Section V presents numerical results from both Gaussian and simulated high-energy physics datasets, and Section VI concludes with a discussion of the advantages, limitations, and future research directions of our approach. II Statistics of Unfolding For binned unfolding, the task is to infer the cross section t_{i} in each ‘particle level’ (pre–detector) bin i given the observed counts m_{j} in each ‘detector-level’ (post-detector) bin j and response matrix with elements R_{ij}=\Pr(m_i|t_j) , estimated from simulation. When the response matrix is correctly modeled (a necessary assumption), on average, \mathbf{m=R\,t} . The naive approach is to analytically invert the response matrix, \mathbf{\hat{t}=R^{-1}\,m} . While unbiased, this approach suffers from two major limitations. First, the response matrix modelled from simulations might not be non-negative definite, which can yield negative predictions for t_{i} . This would violate the physical requirement that cross sections be non–negative. Second, matrix inversion amplifies statistical fluctuations due to (often large) off-diagonal components of \mathbf{R} . This can result in small perturbations in \mathbf{m} causing large perturbations in \mathbf{\hat{t}} . The matrix itself may also not be square so analytical inversion might not be possible altogether. To address these challenges, a number of regularized matrix inversion solutions have been proposed. One of the most widely used approach is Iterative Bayesian Unfolding (IBU) (also known as Lucy-Richardson deconvolution) D’Agostini 1995 ; Richardson 1972 ; Lucy 1974 . Despite its name, IBU is a strictly frequentist approach; it is an expectation-maximization (EM) technique that converges to a (local) maximum likelihood estimate Shepp and Vardi 1982 . IBU will serve as our baseline. The iterative update in IBU is expressed as \displaystyle t_{j}^{(n)} \displaystyle=\sum_{i}\text{Pr}_{n-1}(\text{truth $j$}|\text{measure $i$})\;\Pr(\text{measure $i$}) (1) \displaystyle=\sum_{i}\frac{R_{ij}\,t_{j}^{(n-1)}}{\sum_{k}R_{ik}\,t_{k}^{(n-1)}}\times m_{i}\,, (2) for some initial guess t^{(0)} (usually the starting simulation) and up to some maximum n_{\max} iterations. Cutting of the number of iterations before convergence, n_{\max}<\infty , is a form of regularization that prevents overfitting to statistical fluctuations in the data. IBU only provides a point estimate, but it is possible to estimate statistical uncertainties through asymptotic formulae or through bootstrappig Efron 1979b . This already highlights potential advantages of NPU: the statistical uncertainty is part of the result and the regularization is implicit in the training and automated through model selection (via the validation loss). Another difference between NPU and IBU is their behavior in unconstrained regions of phase space. Cross sections in the particle-level phase space that are unconstrained or

Read the complete text