Some inverse problems in particle physics Authors: Luigi Del Debbio Published: 2026 ABSTRACT Inverse problems play a central role in current areas of research in particle phenomenology. In these lectures, we focus on two examples, the extraction of Parton Distribution Functions (PDFs) from experimental data (or, equivalently, from pseudo- and quasi-PDFs computed in lattice QCD), and the extraction of spectral functions from lattice Euclidean time correlators. We investigate in detail three different approaches, namely Backus–Gilbert, Gaussian Processes, and fits based on Neural Network parametrizations. Introduction Inverse problems appear in many areas of physics and, more widely, of computational science. The solution of these problems requires a mathematically robust framework that allows the extraction of information about theoretical models from (increasingly large) finite sets of data. It is often the case that solutions are plagued by the ill-posed nature of the problem and require some form of regularization. In these lectures, we focus on applications to particle phenomenology, a subject area that is increasingly driven toward precision physics as a way to search for new physics beyond the Standard Model. A crucial requirement for precision physics is the possibility of defining statistically robust error bars, and controlling the systematic errors in the solution. Solutions need to be both precise and accurate, calling for rigorous uncertainty quantification. More specifically, we consider here examples where we aim to determine a function defined on a one-dimensional real interval, which we denote \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I}$$\end{document}I, using a finite set of data points. In the absence of further hypotheses, these functions are elements of infinite-dimensional spaces, so that the inverse problem is necessarily ill-posed. In the absence of infinitely many data points, further assumptions are needed in order to get a solution. We frame our presentation from a probabilistic point of view, where the quantities that need to be determined are promoted to stochastic variables and the solution to the inverse problem is provided in terms of probability distributions. The probabilistic approach has, in our opinion, two main advantages: it spells out explicitly all the assumptions about the model in the prior probability distribution;it yields a characterization of the possible solutions in terms of posterior probabilities.The division between prior and posterior distributions is an appealing feature of the probabilistic approach, which clearly distinguishes the impact of any model assumption from the impact of the available data. We will see examples where we consider the impact of larger datasets, of more precise data, and of different choices of priors. it spells out explicitly all the assumptions about the model in the prior probability distribution; it spells out explicitly all the assumptions about the model in the prior probability distribution; it yields a characterization of the possible solutions in terms of posterior probabilities. it yields a characterization of the possible solutions in terms of posterior probabilities. Building upon the mathematical literature, we address the solution of two inverse problems that are relevant for current particle phenomenology, namely the determination of Parton Distribution Functions (PDFs) from experimental (or lattice) data;the inversion of the Laplace transform that allows to extract spectral densities from Euclidean lattice correlators.The lectures are organized as follows. The notation and the mathematical framework are introduced in Sect. 2. We then focus on three different methods to address the solution of the inverse problem that are widely in use in the particle phenomenology community. Section 3 is devoted to the Backus–Gilbert method; we discuss the basic idea behind the method and its applications, in particular for the extraction of spectral densities from lattice QCD correlator. Section 4 introduces the usage of Gaussian processes for inverse problems. Gaussian processes are a non-parametric method, which relies on Bayesian inference in order to deduce the posterior distribution for the solution of the inverse problem. Finally, Sects. 5, 6, and 7 are devoted to the solution of the inverse problem using Neural Networks. The solution based on Neural Networks is divided into three parts: in Sect. 5, we introduce the notation used and discuss the prior distribution of the output of the Neural Networks at initialization; Sect. 6 is devoted to the analysis of the training dynamics of Neural Networks, using the Neural Tangent Kernel; finally, in Sect. 7, we discuss the so-called lazy training regime, where an analytical solution can be obtained. Understanding in detail the training dynamics is a key ingredient in order to assess the robustness of the solution and to relate the solution obtained using Neural Networks to the ones obtained using other methods, like Backus–Gilbert or Gaussian Processes. In Sect. 8, we attempt a synthesis of the results obtained using the different methods, and we discuss the relations between them. the determination of Parton Distribution Functions (PDFs) from experimental (or lattice) data; the determination of Parton Distribution Functions (PDFs) from experimental (or lattice) data; the inversion of the Laplace transform that allows to extract spectral densities from Euclidean lattice correlators. the inversion of the Laplace transform that allows to extract spectral densities from Euclidean lattice correlators. In writing these lectures, we had to make a number of choices about the topics to be covered and the level of detail. We have tried to provide a self-contained presentation, focusing on methods that are more familiar to us and that we have extensively used. In a limited amount of hours, it was impossible to provide an exhaustive overview of the field, and we apologize for the omissions.1 We hope nonetheless that these lectures will be useful for students and researchers interested in the solution of inverse problems in particle phenomenology, and more widely in computational science. Preliminaries Notation and definitions Let us begin by introducing the notation used in these lectures. Input functions. The space of input functions is a Banach space, X. A Banach space is a complete, normed vector space. The actual space of functions that we want to explore depends on the problem at hand, but we always assume the existence of a metric that allows us to define the distance between two functions. The completeness of the space is important to prove mathematical theorems, but is not needed in practice for the problems considered here. Example 1 The space of parton distribution functions (PDFs) is the space of functions1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f{:}\, [0,1] \rightarrow \mathbb {R}, \end{aligned}$$\end{document}f:[0,1]→R,sometimes with the additional requirement that f is integrable, so that the sum rules can be satisfied. In all current determinations of the PDFs, see e.g. [2–4], a choice of functional form is made, which is defined by a set of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{model}}$$\end{document}Nmodel parameters. Therefore, the set of input functions is effectively reduced to a finite-dimensional space spanned by these parameters. For example, one could use a basis of orthogonal polynomials or a neural network parametrization to represent the functions. These choices introduce a model dependence in the determination of the PDFs. Example 2 The space of spectral densities is the space of functions2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho {:}\, [0,\infty ) \rightarrow \mathbb {R}^+, \end{aligned}$$\end{document}ρ:[0,∞)→R+,with the additional requirement that ρρ is integrable, and more specifically that3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \int _0^\infty \textrm{d}\omega \, \rho (\omega ) = 1. \end{aligned}$$\end{document}∫0∞dωρ(ω)=1.Note that single particle stable states correspond to delta functions in the spectral density, which is therefore a distribution, not a function. We will come back to this point later. Example 3 When considering the scattering of two particles, the amplitude can be decomposed in partial waves, so that the physical process is described by a set of functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {M}_{\ell }(\sqrt{s})$$\end{document}Mℓ(s). It is customary to rewrite4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {M}_{\ell }^{-1} = \mathcal {K}_{\ell }^{-1} - i \rho , \end{aligned}$$\end{document}Mℓ-1=Kℓ-1-iρ,where ρρ is the phase space factor and the K-matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {K}_{\ell }$$\end{document}Kℓ, which is a meromorphic function of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{s}$$\end{document}s. The space of input functions in this case is the space of meromorphic functions defined on the complex \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{s}$$\end{document}s plane. Forward map. The forward map maps the input functions to some output. The forward map is sometimes called the model and is denoted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {F}$$\end{document}F5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {F}{:}\, X&\rightarrow R, \nonumber \\ f&\mapsto r = \mathcal {F}(f). \end{aligned}$$\end{document}F:X→R,f↦r=F(f).Given a model \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {F}$$\end{document}F and an input function f, we can compute the output r. We assume that the space of output functions R is also a Banach space. The output functions are the theoretical predictions for the observables that are measured in the experiments. Example 4 In the case of PDFs, the forward map that yields the theoretical prediction for the structure functions measured in a DIS experiment is6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F(x,Q^2) = \mathcal {F}(f) = \int _0^1 \frac{\textrm{d}\xi }{\xi }\, C(x/\xi , Q^2)\, f(\xi ), \end{aligned}$$\end{document}F(x,Q2)=F(f)=∫01dξξC(x/ξ,Q2)f(ξ),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C(x,Q^2)$$\end{document}C(x,Q2) is the coefficient function that is known from perturbation theory. The coefficient functions can be computed at different orders in perturbation theory, yielding different (NLO, NNLO,...) theoretical predictions. Example 5 The Laplace transform of the spectral densities yields the theoretical prediction for the Euclidean correlators7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C(t) = \int _0^\infty \textrm{d}\omega \, e^{-\omega t}\, \rho (\omega ), \quad \text {for } t> 0. \end{aligned}$$\end{document}C(t)=∫0∞dωe-ωtρ(ω),fort>0.The lattice correlators are computed from Monte Carlo simulations of the Euclidean path integral. They are the data used to infer \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω), just like the measurements of structure functions in the case of PDFs. The use of the letter C to denote the correlators is standard in the lattice community; they should not be confused with the coefficient functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C(x,Q^2)$$\end{document}C(x,Q2) in the case of PDFs discussed in the example above. Rather than changing a consolidated notation, we hope that the context will make it clear which of the two is being referred to. Example 6 The scattering amplitudes determine the discrete levels of the energy spectrum of the theory in a finite volume. The forward map in this case is given by the Lüscher formula, which states that the energy levels are the solutions of the equation8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \det \left[ \mathcal {K}^{-1}(E) + F(E,\textbf{P},L)\right] = 0, \end{aligned}$$\end{document}detK-1(E)+F(E,P,L)=0,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F(E,\textbf{P},L)$$\end{document}F(E,P,L) is a known function of the energy, the total momentum, and the volume. Observables. The observables are the quantities that are measured in the experiments. The central values quoted by the experiments are denoted by Y. Experiments only ever measure a finite number of observables, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{dat}}$$\end{document}Ndat, and hence9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y = \{ Y_I, \, I = 1, \ldots , N_{\textrm{dat}}\} \in \mathbb {R}^{N_{\textrm{dat}}}. \end{aligned}$$\end{document}Y={YI,I=1,…,Ndat}∈RNdat.Observables will be labeled by capital roman indices, I, J, …I,J,…. At times, we will write all indices explicitly; however, in general, we suppress the indices and use a compact notation, where Y is a vector in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^{N_{\textrm{dat}}}$$\end{document}RNdat. The corresponding theoretical predictions are denoted by T and are computed by applying the forward map at the points corresponding to the observables10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T{:}\, X&\rightarrow \mathbb {R}^{N_{\textrm{dat}}},\nonumber \\ f&\mapsto T[f]. \end{aligned}$$\end{document}T:X→RNdat,f↦T[f].There is a theoretical prediction for each observable, i.e. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_I[f] \in \mathbb {R}, I = 1, \ldots , N_{\textrm{dat}}$$\end{document}TI[f]∈R,I=1,…,Ndat. Example 7 In the case of PDFs, the observables are the structure functions measured at various kinematical points in a DIS experiment. The experimental data points are identified by the values of the Bjorken x and the virtuality \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q^2$$\end{document}Q2. The observables are then defined as11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_I = F(x_I, Q^2_I), \quad I = 1, \ldots , N_{\textrm{dat}}. \end{aligned}$$\end{document}YI=F(xI,QI2),I=1,…,Ndat.The theoretical predictions are computed from the forward map as12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T_I = T_I[f] = \int _0^1 \frac{\textrm{d}\xi }{\xi }\, C(x_I/\xi , Q_I^2)\, f(\xi ). \end{aligned}$$\end{document}TI=TI[f]=∫01dξξC(xI/ξ,QI2)f(ξ).In order to emphasize the linear dependence of the obervables on the PDFs, we can rewrite the equation above as13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T_I = T_I[f] = \int _0^1 \textrm{d}\xi \, C_{I}(\xi )\, f(\xi ), \end{aligned}$$\end{document}TI=TI[f]=∫01dξCI(ξ)f(ξ),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{I}(\xi ) = C(x_I/\xi , Q_I^2)/\xi$$\end{document}CI(ξ)=C(xI/ξ,QI2)/ξ is the coefficient function evaluated at the kinematical point corresponding to the observable \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_I$$\end{document}YI. Example 8 When extracting spectral densities, we can only compute the Euclidean correlators for a finite number of time separations on a discretized spacetime lattice. Hence14\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_I = C(t_I), \quad I = 1, \ldots , N_{\textrm{dat}}, \end{aligned}$$\end{document}YI=C(tI),I=1,…,Ndat,where in this case \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{dat}}$$\end{document}Ndat is the number of discrete values of t where the measurement of the correlators yields a good signal—a more quantitative definition of good signal will be discussed below. Example 9 In the case of scattering amplitudes, the observables are the discrete energy levels of the theory in a finite volume, which are measured in lattice simulations from the large-time behavior of Euclidean space correlators. The theoretical predictions for these energy levels are computed from the forward map as the solutions of the Lüscher formula in Eq. (8). It is important to appreciate that in all these examples, we only have access to a finite number of observables, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{dat}}$$\end{document}Ndat, and therefore, the information that can be extracted from the data is limited. When working with a finite number of measurements, often subject to noise, we cannot aim to fully reconstruct the input functions. Looking at the examples that we introduced above, we can anticipate that in the case of PDFs, the measurements of the structure functions only cover a finite kinematic range, leading to large uncertainties in the so-called extrapolation regions;in the case of spectral densities, measuring the correlators for a finite number of discrete values of the Euclidean time only allows the reconstruction of a limited number the states contributing to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω).in the case of scattering amplitudes, the quantization condition only allows the computation of the amplitude at a finite number of discrete energy values. The form of the amplitude in between these values is not constrained by the data and the reconstruction is regulated by the choice of prior [5].Experimental data are subject to observational noise, which is modeled as15\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_I = T_I[f] + \eta _I, \quad I = 1, \ldots , N_{\textrm{dat}}, \end{aligned}$$\end{document}YI=TI[f]+ηI,I=1,…,Ndat,where ηη is a random variable in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^{N_{\textrm{dat}}}$$\end{document}RNdat with probability distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_\eta$$\end{document}Pη. Unless otherwise stated, we assume that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_\eta$$\end{document}Pη is a multivariate Gaussian distribution, centered at the origin and with a covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY, given by the experiments16\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P_\eta (\eta ) = \frac{1}{\sqrt{|2 \pi C_Y|}}\, e^{-\frac{1}{2} \eta ^T C_Y^{-1} \eta }. \end{aligned}$$\end{document}Pη(η)=1|2πCY|e-12ηTCY-1η.The central values quoted by the experiments are one particular draw from the probability distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_\eta$$\end{document}Pη. in the case of PDFs, the measurements of the structure functions only cover a finite kinematic range, leading to large uncertainties in the so-called extrapolation regions; in the case of PDFs, the measurements of the structure functions only cover a finite kinematic range, leading to large uncertainties in the so-called extrapolation regions; in the case of spectral densities, measuring the correlators for a finite number of discrete values of the Euclidean time only allows the reconstruction of a limited number the states contributing to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω). in the case of spectral densities, measuring the correlators for a finite number of discrete values of the Euclidean time only allows the reconstruction of a limited number the states contributing to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω). in the case of scattering amplitudes, the quantization condition only allows the computation of the amplitude at a finite number of discrete energy values. The form of the amplitude in between these values is not constrained by the data and the reconstruction is regulated by the choice of prior [5]. in the case of scattering amplitudes, the quantization condition only allows the computation of the amplitude at a finite number of discrete energy values. The form of the amplitude in between these values is not constrained by the data and the reconstruction is regulated by the choice of prior [5]. A solution on a grid For the case of observables that depend linearly on the input function, a simple solution to the inverse problem is obtained by minimizing the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document}χ2, defined as17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \chi ^2[f] = \sum _{I,J=1}^{N_{\textrm{dat}}} \left( Y_I - T_I[f]\right) \left( C_Y^{-1}\right) _{IJ} \left( Y_J - T_J[f]\right) , \end{aligned}$$\end{document}χ2[f]=∑I,J=1NdatYI-TI[f]CY-1IJYJ-TJ[f],or equivalently suppressing the indices18\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \chi ^2[f] = (Y - T[f])^T C_Y^{-1} (Y - T[f]). \end{aligned}$$\end{document}χ2[f]=(Y-T[f])TCY-1(Y-T[f]).The solution is then given by the function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}$$\end{document}f^ that minimizes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2[f]$$\end{document}χ2[f]19\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f} = \arg \min _f \chi ^2[f]. \end{aligned}$$\end{document}f^=argminfχ2[f].We present a solution to this problem, obtained by discretizing the input function on a grid of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{grid}}$$\end{document}Ngrid points, and then minimizing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document}χ2 with respect to the values of the function at these points. The unknown function is, therefore, represented as a vector of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{grid}}$$\end{document}Ngrid real numbers, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f \in \mathbb {R}^{N_{\textrm{grid}}}$$\end{document}f∈RNgrid, while the forward map is represented as a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{dat}}\times N_{\textrm{grid}}$$\end{document}Ndat×Ngrid matrix20\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T[f] = (\textrm{FK})f, \end{aligned}$$\end{document}T[f]=(FK)f,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})$$\end{document}(FK) is the matrix representation of the forward map. Note that in this framework, we have reduced the infinite-dimensional space of input functions to a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{grid}}$$\end{document}Ngrid dimensional real space. The covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY is positive-definite, its SVD decomposition yields21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_Y = R\, \Sigma ^{1/2}\, \Sigma ^{1/2} R^T, \end{aligned}$$\end{document}CY=RΣ1/2Σ1/2RT,where R is orthogonal and ΣΣ is the diagonal matrix made of the eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY. We can then rewrite \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document}χ2 as22\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \chi ^2[f]&= (Y - (\textrm{FK})f)^T R\, \Sigma ^{-1/2}\, \Sigma ^{-1/2} R^T (Y - (\textrm{FK})f) \nonumber \\&= \Vert \Upsilon - A f \Vert ^2, \end{aligned}$$\end{document}χ2[f]=(Y-(FK)f)TRΣ-1/2Σ-1/2RT(Y-(FK)f)=‖Υ-Af‖2,where we have introduced the uncorrelated, whitened data ΥΥ and the rescaled forward map A23\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Upsilon = \Sigma ^{-1/2} R^T Y, \quad A = \Sigma ^{-1/2} R^T (\textrm{FK}). \end{aligned}$$\end{document}Υ=Σ-1/2RTY,A=Σ-1/2RT(FK).The solution to the minimization problem in Eq. (19) is then given by acting with the pseudo-inverse of A, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A^+$$\end{document}A+, on the rescaled data [6]24\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f} = A^+ \Upsilon . \end{aligned}$$\end{document}f^=A+Υ.Using the following identity:25\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A^+ = (A^T A)^{+} A^T, \end{aligned}$$\end{document}A+=(ATA)+AT,allows us to rewrite the solution as26\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f} = M^{+} (\textrm{FK})^TC_Y^{-1} Y, \end{aligned}$$\end{document}f^=M+(FK)TCY-1Y,where we have introduced the matrix M27\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} M = (\textrm{FK})^TC_Y^{-1} (\textrm{FK}). \end{aligned}$$\end{document}M=(FK)TCY-1(FK).In the physical applications of interest in these lectures, we will encounter two situationsthe matrix M is not invertible, and then, the solution in Eq. (26) is not unique;or M has a large condition number, and the solution is unstable with respect to small variations of the data.Note that a non-invertible matrix M has a non-trivial kernel28\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{Ker}(M) = \{ f \in \mathbb {R}^{N_{\textrm{grid}}}{:}\,M f = 0 \}, \end{aligned}$$\end{document}Ker(M)={f∈RNgrid:Mf=0},which coincides with the kernel of the forward map, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{Ker}((\textrm{FK}))$$\end{document}Ker((FK)). In this case, the solution is not unique, since any function of the form29\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f} + f_{\parallel }, \quad f_{\parallel } \in \textrm{Ker}(M)\, \end{aligned}$$\end{document}f^+f‖,f‖∈Ker(M)is also a solution. In both examples, in order to get a well-defined, stable solution, some form of regularization is needed. the matrix M is not invertible, and then, the solution in Eq. (26) is not unique; the matrix M is not invertible, and then, the solution in Eq. (26) is not unique; or M has a large condition number, and the solution is unstable with respect to small variations of the data. or M has a large condition number, and the solution is unstable with respect to small variations of the data. Probabilistic framework As we will discuss later in the lectures, it is useful to work in a probabilistic framework, where the unknown function f is promoted to a stochastic process and the answer of the inverse problem is given in the form of a posterior probability for the solution f. In general, this is a probability measure in an infinite-dimensional space of functions and needs to be handled with (mathematical) care. In practice, following the discussion in the above section, we aim to compute the joint posterior distributions230\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{P}(f_1, \ldots , f_{N_{\textrm{grid}}}) = P(f_1, \ldots , f_{N_{\textrm{grid}}} | D_{N_{\textrm{dat}}}), \end{aligned}$$\end{document}P~(f1,…,fNgrid)=P(f1,…,fNgrid|DNdat),for any generic set of values of f31\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f_\alpha = f(x_\alpha ), \quad \alpha = 1, \ldots , N_{\textrm{grid}}, \quad x_\alpha \in \mathcal {I}, \end{aligned}$$\end{document}fα=f(xα),α=1,…,Ngrid,xα∈I,and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I}$$\end{document}I is the domain of the function f. Looking at the two examples we discussed, we have\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I} = [0,1]$$\end{document}I=[0,1] for the PDFs, since the Bjorken x is defined in this range;\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I} = [0,\infty )$$\end{document}I=[0,∞) for the spectral density, since the energy of the states contributing to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω) is positive.We can marginalize with respect to some of the values of f and get the posterior distribution for a subset of the values of f32\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P_{\alpha _1 \ldots \alpha _n}(f_{\alpha _1}, \ldots , f_{\alpha _n} | D_{N_{\textrm{dat}}}) = \int \prod _{\alpha \ne \alpha _1, \ldots , \alpha _n} \textrm{d}f_\alpha \, P(f_1, \ldots , f_{N_{\textrm{grid}}} | D_{N_{\textrm{dat}}}). \end{aligned}$$\end{document}Pα1…αn(fα1,…,fαn|DNdat)=∫∏α≠α1,…,αndfαP(f1,…,fNgrid|DNdat).The central value and the variance of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f(x_\alpha )$$\end{document}f(xα) at a single point on the grid \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{\alpha }$$\end{document}xα are computed from the posterior \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_\alpha (f | D_{N_{\textrm{dat}}})$$\end{document}Pα(f|DNdat) as33\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \bar{f}_\alpha&= \mathbb {E}[f(x_\alpha )] = \int \textrm{d}f P_\alpha (f | D_{N_{\textrm{dat}}})\, f, \end{aligned}$$\end{document}f¯α=E[f(xα)]=∫dfPα(f|DNdat)f,34\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sigma _\alpha ^2&= \mathbb {E}[\left( f_\alpha - \bar{f}_\alpha \right) ^2] = \int \textrm{d}f P_\alpha (f | D_{N_{\textrm{dat}}})\, \left( f - \bar{f}_\alpha \right) ^2. \end{aligned}$$\end{document}σα2=E[fα-f¯α2]=∫dfPα(f|DNdat)f-f¯α2.Correlations between the values of the function f at multiple points can be computed using the joint distributions in Eq. 32. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I} = [0,1]$$\end{document}I=[0,1] for the PDFs, since the Bjorken x is defined in this range; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I} = [0,1]$$\end{document}I=[0,1] for the PDFs, since the Bjorken x is defined in this range; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I} = [0,\infty )$$\end{document}I=[0,∞) for the spectral density, since the energy of the states contributing to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω) is positive. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I} = [0,\infty )$$\end{document}I=[0,∞) for the spectral density, since the energy of the states contributing to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω) is positive. Backus–Gilbert method Backus–Gilbert method We begin this section by reviewing the general principles behind the Backus–Gilbert method, which is a widely used approach to solve inverse problems in a non-parametric way. The method was originally developed in the context of geophysics [7], but has found applications in many other fields, including particle physics and lattice QCD. We follow here the presentation in Ref. [8], which leads naturally to a connection with Gaussian Processes, as we will see in the next section. We start by considering the continuous version of the inverse problem, where we want to determine a function f(x) defined on a real interval \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I}$$\end{document}I, from a finite set of data points \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D = \{Y_I, I=1, \ldots , N_{\textrm{dat}}\}$$\end{document}D={YI,I=1,…,Ndat}. The Backus–Gilbert method provides a way to construct an estimator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}^{A}(x)$$\end{document}f^A(x) for the function f(x), which is a linear combination of the data points35\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{A}(x) = \sum _{I=1}^{N_{\textrm{dat}}} A_I(x) Y_I, \end{aligned}$$\end{document}f^A(x)=∑I=1NdatAI(x)YI,defined by choosing a set of coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_I(x)$$\end{document}AI(x). For data that depend linearly on the function f(x), in the absence of statistical noise36\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_I = \int _{\mathcal {I}} \textrm{d}x' \, C_I(x') f(x'), \end{aligned}$$\end{document}YI=∫Idx′CI(x′)f(x′),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_I(x)$$\end{document}CI(x) are known functions that define the linear forward map for the problem under study.3 The Backus–Gilbert estimator can be rewritten as37\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{A}(x) = \int _{\mathcal {I}} \textrm{d}x' \, \mathcal {R}^{A}(x,x') f(x'), \end{aligned}$$\end{document}f^A(x)=∫Idx′RA(x,x′)f(x′),where we introduced the resolution function38\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {R}^{A}(x,x') = \sum _{I=1}^{N_{\textrm{dat}}} A_I(x) C_I(x'). \end{aligned}$$\end{document}RA(x,x′)=∑I=1NdatAI(x)CI(x′).Equation (37) shows explicitly that the BG estimator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}^{A}(x)$$\end{document}f^A(x) is a smeared version of the true solution f(x), where the smearing is dictated by the resolution function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {R}^{A}(x,x')$$\end{document}RA(x,x′). In general, the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_I(x)$$\end{document}AI(x) can be chosen by requiring that the resolution function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {R}^{A}(x,x')$$\end{document}RA(x,x′) is as close as possible to some target smearing function of our choice, which depends on the specific problem that is being solved. In order to quantify the closeness of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {R}^{A}(x,x')$$\end{document}RA(x,x′) to the target smearing function, we introduce a metric in the space of functions39\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} d[u,v] = \Vert u-v \Vert _k, \end{aligned}$$\end{document}d[u,v]=‖u-v‖k,where the norm is defined with respect to a symmetric, positive-definite kernel k(x,x')k(x,x′) as40\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Vert u \Vert _k^2 = \int _{\mathcal {I}} \textrm{d}x\, \textrm{d}x'\, u(x) k(x,x') u(x'). \end{aligned}$$\end{document}‖u‖k2=∫Idxdx′u(x)k(x,x′)u(x′).Note that the arbitrariness in the choice of the kernel k(x,x')k(x,x′) can be used to encode some prior information about the function f(x). Point estimation. In order to get a point estimator for the function f(x), we can choose the target smearing function to be as close as possible to a delta function δ (x-x')δ(x-x′), which would correspond to an ideal estimator that recovers the true function f(x) without any smearing. In practice, this is done by minimizing the distance between the resolution function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {R}^{A}(x,x')$$\end{document}RA(x,x′) and the delta function δ (x-x')δ(x-x′)41\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} d\left[ \mathcal {R}^{A}(x,x'), \delta (x-x')\right] ^2, \end{aligned}$$\end{document}dRA(x,x′),δ(x-x′)2,as a function of the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_I(x)$$\end{document}AI(x). Imposing42\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\partial }{\partial A_I(x)} d\left[ \mathcal {R}^{A}(x,x'), \delta (x-x')\right] ^2 = 0, \end{aligned}$$\end{document}∂∂AI(x)dRA(x,x′),δ(x-x′)2=0,yields43\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W_{IJ} A_J(x) = R_I(x), \end{aligned}$$\end{document}WIJAJ(x)=RI(x),where44\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W_{IJ}&= \int _{\mathcal {I}} \textrm{d}x\, \textrm{d}x'\, C_I(x) k(x,x') C_J(x'), \end{aligned}$$\end{document}WIJ=∫Idxdx′CI(x)k(x,x′)CJ(x′),45\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_I(x)&= \int _{\mathcal {I}} \textrm{d}x'\, k(x,x') C_I(x'). \end{aligned}$$\end{document}RI(x)=∫Idx′k(x,x′)CI(x′).Having introduced the matrix W and the vector R with explicit observable indices, we are going to suppress the indices and use a compact notation, where W is a matrix in the space of observables and R is a vector in the same space. Assuming that the matrix W is invertible, we can solve for the coefficients A(x) as46\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A(x) = R(x)^T W^{-1}, \end{aligned}$$\end{document}A(x)=R(x)TW-1,where we also suppressed the index for the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_I(x)$$\end{document}AI(x), which is now implicit in the matrix–vector multiplication. Plugging this solution into Eq. (35) yields the point estimator for the function f(x)47\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{\textrm{BG}}(x) = R(x)^T W^{-1} Y. \end{aligned}$$\end{document}f^BG(x)=R(x)TW-1Y.The resolution function for this estimator is given by48\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {R}^{\textrm{BG}}(x,x') = R(x)^T W^{-1} C(x'), \end{aligned}$$\end{document}RBG(x,x′)=R(x)TW-1C(x′),where we introduced the vector C(x) with components \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_I(x)$$\end{document}CI(x). Noisy data. Statistical noise in the data is modeled by adding a stochastic term ηη to the theoretical prediction49\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_I = \int _{\mathcal {I}} \textrm{d}x'\ C_I(x') f(x') + \eta _I, \end{aligned}$$\end{document}YI=∫Idx′CI(x′)f(x′)+ηI,where ηη is a vector of random variables \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta _I$$\end{document}ηI with zero mean and covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY. The point estimator is obtained by minimizing the regulated distance50\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} d\left[ \mathcal {R}^{A}(x,x'), \delta (x-x')\right] ^2 + \lambda A^T(x) C_Y A(x), \end{aligned}$$\end{document}dRA(x,x′),δ(x-x′)2+λAT(x)CYA(x),where λλ is a regularization parameter that controls the trade-off between the closeness of the resolution function to the delta function and the stability of the solution with respect to noise in the data. The directions in data space that correspond to the larger eigenvalues of the covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY are more affected by noise, and the regularization term in Eq. (50) suppresses the contribution of these directions to the solution. The solution for the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_I(x)$$\end{document}AI(x) is given by51\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A(x)^T = R(x)^T \left( W + \lambda C_Y\right) ^{-1}, \end{aligned}$$\end{document}A(x)T=R(x)TW+λCY-1,and the point estimator is given by52\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{\textrm{BG}}(x) = R^T(x) \left( W + \lambda C_Y\right) ^{-1} Y. \end{aligned}$$\end{document}f^BG(x)=RT(x)W+λCY-1Y.The resolution function for this estimator is given by53\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {R}^{\textrm{BG}}(x,x') = R^T(x) \left( W + \lambda C_Y\right) ^{-1} C(x'). \end{aligned}$$\end{document}RBG(x,x′)=RT(x)W+λCY-1C(x′).Note that, because the covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY is positive-definite, the regularized matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W + \lambda C_Y$$\end{document}W+λCY is also positive-definite and therefore invertible, so that the solution in Eq. (51) is well defined for any value of the regularization parameter λ> 0λ>0. Exercise 1 Derive the solution for the case of noisy data, starting from the minimization of the regulated distance in Eq. (50). If needed, write all the indices explicitly to get familiar with the notation. A comment on the probabilistic interpretation. Note that the BG solution does not start from a prior distribution for the solution. However, we can construct a posterior distribution for the solution by bootstrapping the point estimator discussed above over the noise in the data. In this way, the statistical fluctuations of the data translate into fluctuations of the solution. BG on a grid It is instructive to pause for a second and rewrite the BG derivation for the case discussed in Sect. 2.2, where the solution is represented as the finite-dimensional vector of values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_\alpha$$\end{document}fα, and the theoretical prediction is computed as a matrix–vector multiplication54\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T[f] = (\textrm{FK})f. \end{aligned}$$\end{document}T[f]=(FK)f.The estimator for the solution is a linear combination of the data55\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^A(x_\alpha ) = \hat{f}^A_\alpha = \sum _{I=1}^{N_{\textrm{dat}}} A_{\alpha I} Y_I, \end{aligned}$$\end{document}f^A(xα)=f^αA=∑I=1NdatAαIYI,where the functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_I(x)$$\end{document}AI(x) that we encountered in the continuous case are now replaced by the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{\alpha I}$$\end{document}AαI, i.e., by a matrix of coefficients A with dimensions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{grid}}\times N_{\textrm{dat}}$$\end{document}Ngrid×Ndat. The norm in the space of functions is replaced by a norm in the space of vectors, defined in terms of a positive-definite matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K_{\alpha _1\alpha _2}$$\end{document}Kα1α2 as56\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Vert u \Vert _K^2 = u^T K u. \end{aligned}$$\end{document}‖u‖K2=uTKu.Similarly, we introduce the matrices W and R as57\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W_{IJ}&= (\textrm{FK})_{I\alpha _1} K_{\alpha _1\alpha _2} (\textrm{FK})^T_{\alpha _2 J}, \end{aligned}$$\end{document}WIJ=(FK)Iα1Kα1α2(FK)α2JT,58\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{I\alpha }&= (\textrm{FK})_{I\alpha _1} K_{\alpha _1\alpha }. \end{aligned}$$\end{document}RIα=(FK)Iα1Kα1α.The interested reader can verify that the solution for the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A_{\alpha I}$$\end{document}AαI for the case of noisy data is given by59\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A = R^T \left( W + \lambda C_Y\right) ^{-1}. \end{aligned}$$\end{document}A=RTW+λCY-1. Exercise 2 Derive the solution for the case of noisy data, starting from the minimization of the regulated distance in Eq. (50), with the appropriate replacements for the discrete case. The point estimator for the solution is given by60\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{\textrm{BG}} = R^T \left( W + \lambda C_Y\right) ^{-1} Y, \end{aligned}$$\end{document}f^BG=RTW+λCY-1Y,and the resolution function is given by61\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {R}^{\textrm{BG}}&= R^T \left( W + \lambda C_Y\right) ^{-1} (\textrm{FK})\end{aligned}$$\end{document}RBG=RTW+λCY-1(FK)62\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&= K (\textrm{FK})^T\left( (\textrm{FK})K (\textrm{FK})^T+ \lambda C_Y\right) ^{-1} (\textrm{FK}). \end{aligned}$$\end{document}=K(FK)T(FK)K(FK)T+λCY-1(FK).We have suppressed all the indices in the equations above. It may be useful to rewrite the resolution function with all the indices explicitly written as a way to get familiar with the notation. These expressions will be useful in connecting the BG result with other solutions that we will discuss in the next sections. Closure tests Closure tests are a powerful tool to validate the solutions of the inverse problem, and to assess the quality of the reconstructed function. The idea is to generate synthetic data from a known solution, and then to apply, in this particular case, the BG method to these data to see if we can recover the original solution. This allows us to check that the implementation of the BG method is correct, and to understand how well the method performs in practice. The synthetic data are generated by applying the forward operator to a known solution, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, and adding noise to mimic the statistical uncertainty of the data. Using the grid notation introduced in Sect. 3.2, the synthetic data are generated as63\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y = (\textrm{FK})f_0 + \epsilon , \end{aligned}$$\end{document}Y=(FK)f0+ϵ,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 is the known solution, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})$$\end{document}(FK) is the forward operator, and εϵ is a vector of random variables with zero mean and covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY. The BG method is then applied to the synthetic data Y to obtain an estimator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}^{\textrm{BG}}$$\end{document}f^BG for the solution and compare it with the known solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0. Writing explicitly all the matrices that appear in the BG solution in Eq. (60) yields64\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{\textrm{BG}}&= K (\textrm{FK})^T\left( (\textrm{FK})K (\textrm{FK})^T+ \lambda C_Y\right) ^{-1} (\textrm{FK})f_0 \nonumber \\&\qquad + K (\textrm{FK})^T\left( (\textrm{FK})K (\textrm{FK})^T+ \lambda C_Y\right) ^{-1} \epsilon . \end{aligned}$$\end{document}f^BG=K(FK)T(FK)K(FK)T+λCY-1(FK)f0+K(FK)T(FK)K(FK)T+λCY-1ϵ.The first term on the right-hand side of the equation above is the contribution to the BG solution that comes from the known solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, while the second term is the contribution that comes from the noise in the data. The second contribution is linear in the noise εϵ and therefore averages to zero over an ensemble of instances of the noise. The first contribution, as already discussed in Sect. 3.1, is a smeared version of the known solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0. A few algebraic manipulations allow us to rewrite the BG solution as65\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{\textrm{BG}}&= \mathcal {R}^{\textrm{BG}} f_0 = K M \frac{1}{K M + \lambda } f_0, \end{aligned}$$\end{document}f^BG=RBGf0=KM1KM+λf0,where66\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} M = (\textrm{FK})^TC_Y^{-1} (\textrm{FK}), \end{aligned}$$\end{document}M=(FK)TCY-1(FK),is the matrix we already introduced in Eq. (27). If M were invertible, then we would recover the exact solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 in the limit \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda \rightarrow 0$$\end{document}λ→0. As discussed in Sect. 2, the matrix M is not invertible, and the BG solution is one prescription to regularize the problem. In the case where M is invertible but ill-conditioned, the BG solution for nonvanishing λλ is a Tikhonov regularization of the problem. Equation (65) shows explicitly that the BG prescription yields a smeared version of the solution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, where the smearing is dictated by the matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K M \frac{1}{K M + \lambda }$$\end{document}KM1KM+λ. Lattice spectral densities and the HLT method A contemporary application of the BG method is in the determination of spectral densities from Euclidean correlators computed in lattice field theories. Here, we focus on the HLT method as introduced in Ref. [9], which is a particular implementation of the BG method, tailored to the specific problem of spectral density reconstruction. We begin by formulating the problem and introducing the notation that is specific to this application. We follow the notation used in the original publication, which is different from the one we used in the general formulation of the problem in Sects. 2.1, 3.1, and 3.2. Observables. The observables used to extract the spectral densities are Euclidean correlators of field operators, which are defined as67\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_L(t) = \frac{1}{L^3} \sum _{\vec {x}} \langle O(t,\vec {x}) O(0) \rangle _L, \end{aligned}$$\end{document}CL(t)=1L3∑x→⟨O(t,x→)O(0)⟩L,where L is the spatial extent of the lattice, and t>0t>0 is the Euclidean time. The subscript L is a reminder of the fact that these correlators are computed in a finite volume. They are evaluated as the mean over an ensemble of gauge configurations generate by Monte Carlo sampling. Their covariance matrix is also computed from the available sample and scales like the inverse of the number of statistically independent configurations. The correlators are computed for a finite set of discrete time points t = a, 2a, … , Tt=a,2a,…,T, where a is the lattice spacing and T is the temporal extent of the lattice in physical units.4 The index I used to label the data points in the previous sections can be identified with the time index \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_I$$\end{document}tI of the correlators, so that we can write \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_I = C_L(t_I)$$\end{document}YI=CL(tI). The unknown function. The function that we want to determine as the solution of the inverse problem is the spectral density68\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho _L(E) = \frac{1}{L^3} \sum _{\vec {x}} \langle O(0,\vec {x})\, \delta (E - H_L)\, O(0)\rangle _L, \end{aligned}$$\end{document}ρL(E)=1L3∑x→⟨O(0,x→)δ(E-HL)O(0)⟩L,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_L$$\end{document}HL is the Hamiltonian of the theory in a finite volume. The spectral density is a positive-definite function of the energy E, and it contains information about the spectrum of the theory, as well as about the matrix elements of the operator O between the vacuum and the energy eigenstates. Comparing again with the notation used in the previous sections for the general formulation of the inverse problem, we can identify the unknown function f(x) with the spectral density \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _L(E)$$\end{document}ρL(E). The argument of the unknown function in this case is the energy E; when we discretize the problem on a gird of points, the index αα is used to label the grid points with the energy variable \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_\alpha$$\end{document}Eα. Formulation of the problem. The Euclidean correlators, computed on a lattice with infinite time extent, and the spectral density are related by a Laplace transform69\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_L(t) = \int _0^\infty \textrm{d}E\, e^{-tE} \rho _L(E). \end{aligned}$$\end{document}CL(t)=∫0∞dEe-tEρL(E).Note that in a finite volume, the spectrum of the Hamiltonian is discrete, which implies that the spectral density is a sum of delta functions70\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho _L(E) = \sum _{n} \rho _n(L) \delta (E - E_n), \end{aligned}$$\end{document}ρL(E)=∑nρn(L)δ(E-En),and therefore, it is a distribution rather than a regular function. As usual when dealing with distributions, we need to introduce a smearing function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta _\sigma (E,E')$$\end{document}Δσ(E,E′) in order to define a regular, smeared version of the spectral density71\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{\rho }_L(\sigma ,E_*) = \int _0^\infty \textrm{d}E \, \Delta _\sigma (E,E_*) \rho _L(E). \end{aligned}$$\end{document}ρ^L(σ,E∗)=∫0∞dEΔσ(E,E∗)ρL(E).The smearing function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta _\sigma (E,E')$$\end{document}Δσ(E,E′) is a regular function, which is peaked around E = E'E=E′ and has a width σσ. The infinite-volume, unsmeared spectral density can be obtained by taking the limit \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma \rightarrow 0$$\end{document}σ→0 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L \rightarrow \infty$$\end{document}L→∞ in the smeared spectral density72\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho (E) = \lim _{\sigma \rightarrow 0} \lim _{L \rightarrow \infty } \hat{\rho }_L(\sigma ,E), \end{aligned}$$\end{document}ρ(E)=limσ→0limL→∞ρ^L(σ,E),where the order of the limits matters. Note that the limit \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma \rightarrow 0$$\end{document}σ→0 may not be needed in practice, since the smeared spectral density \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\rho }_L(\sigma ,E)$$\end{document}ρ^L(σ,E) is a regular function of E that can be directly compared with experimental data, which can also be smeared. HLT formalism on a finite lattice. For correlators computed on a lattice with finite time extent T, the relation between the correlators and the spectral density is given by73\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_I = C_L(t_I) = \int _0^\infty \textrm{d}E\, b_T(t_I,E) \rho _L(E), \end{aligned}$$\end{document}YI=CL(tI)=∫0∞dEbT(tI,E)ρL(E),where the basis function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_T(t,E)$$\end{document}bT(t,E) is given by74\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} b_T(t,E) = e^{-tE} + e^{-(T-t)E}. \end{aligned}$$\end{document}bT(t,E)=e-tE+e-(T-t)E.The basis function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_T(t,E)$$\end{document}bT(t,E) is a generalization of the Laplace kernel \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$e^{-tE}$$\end{document}e-tE that takes into account the finite time extent of the lattice. In the limit \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T \rightarrow \infty$$\end{document}T→∞, the basis function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_T(t,E)$$\end{document}bT(t,E) reduces to the Laplace kernel75\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim _{T \rightarrow \infty } b_T(t,E) = e^{-tE} . \end{aligned}$$\end{document}limT→∞bT(t,E)=e-tE.The function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_T(t_I,E)$$\end{document}bT(tI,E) is plays the role of the function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_I(x)$$\end{document}CI(x) that we introduced in the general formulation of the inverse problem—remember that I identifies the data point and E is the argument of the unknown function. HLT solution. In the HLT formulation, as in all BG formulations, the solution is expanded as a linear combination of the data76\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{\rho }_{L}(E^*) = \sum _{t=0}^{t_{\textrm{max}}} g_t(E^*)\, C(t+a), \end{aligned}$$\end{document}ρ^L(E∗)=∑t=0tmaxgt(E∗)C(t+a),so that the inverse problem is reduced to determining the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_t(E^*)$$\end{document}gt(E∗). As discussed in Sect. 3.1, any BG estimator yields a smeared version of the unknown function. In the notation of HLT, the smearing kernel is77\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \bar{\Delta }(E^*,E) = \sum _{t=0}^{t_{\textrm{max}}} g_t(E^*)\, b_T(t+a, E). \end{aligned}$$\end{document}Δ¯(E∗,E)=∑t=0tmaxgt(E∗)bT(t+a,E).The coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_t(E^*)$$\end{document}gt(E∗) are determined by requiring that the smearing kernel approximates a specific function, e.g.78\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Delta _{\sigma }(E^*,E) = \frac{1}{\mathcal {N}} \, \exp \left[ - \frac{\left( E - E^*\right) ^2}{2\sigma ^2} \right] , \end{aligned}$$\end{document}Δσ(E∗,E)=1Nexp-E-E∗22σ2,with the unusual normalization79\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {N} = \int _{0}^{\infty } \textrm{d}E\, \exp \left[ - \frac{\left( E - E^*\right) ^2}{2\sigma ^2} \right] . \end{aligned}$$\end{document}N=∫0∞dEexp-E-E∗22σ2.Following the derivation in Sect. 3.1, for each value of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E^*$$\end{document}E∗, we minimize a cost function W[λ ,g]W[λ,g] with respect to the parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_t(E^*)$$\end{document}gt(E∗). For each value of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E^*$$\end{document}E∗, we thereby determine \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(t_{\textrm{max}} + 1)$$\end{document}(tmax+1) coefficients, which are necessary to construct \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\rho }_{L}(E^*)$$\end{document}ρ^L(E∗) according to Eq. 76. which is the sum of the distance between \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta _{\sigma }$$\end{document}Δσ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{\Delta }$$\end{document}Δ¯ and a term that takes into account the covariance of the data80\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W\left[ \lambda , g\right] = \left( 1-\lambda \right) A[g] + \lambda \frac{B[g]}{C(0)^2}, \end{aligned}$$\end{document}Wλ,g=1-λA[g]+λB[g]C(0)2,where81\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A[g]&= \int _{E_0}^{\infty } \textrm{d}E\, \left| \Delta _{\sigma }(E^*,E) - \bar{\Delta }(E^*,E) \right| ^2\, \end{aligned}$$\end{document}A[g]=∫E0∞dEΔσ(E∗,E)-Δ¯(E∗,E)282\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} B[g]&= \sum _{t,t'} g_t(E^*) \left( C_Y^{-1}\right) _{tt'} g_{t'}(E^*), \end{aligned}$$\end{document}B[g]=∑t,t′gt(E∗)CY-1tt′gt′(E∗),and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY is the covariance of the Euclidean correlators83\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( C_Y\right) _{tt'} = \textrm{Cov}[C(t), C(t')]. \end{aligned}$$\end{document}CYtt′=Cov[C(t),C(t′)].Note that the integral in Eq. (81) has a lower integration limit \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_0$$\end{document}E0, which cuts off the small values of E. The two terms, A[g] and B[g], are the equivalent of the two terms that appear in Eq. 50; in the case of HLT, the coefficients are chosen to approximate a well-behaved smearing kernel rather than the singular delta distribution of Eq. (50). The main idea is that the HLT solution aims directly for a smeared solution, with the emphasis being on approximating the smearing kernel as best as possible, rather than trying to approximate a Dirac delta. Exercise 3 Show that the solution of the minimization yields84\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} g^{\textrm{HLT}}_t(E^*; \lambda , \sigma )&= W^{-1}(\lambda )_{tt'} \left[ f_{t'}(E^*; \lambda , \sigma ) + R_{t'} \frac{1 - R^T W^{-1}(\lambda ) f(E^*; \lambda ,\sigma )}{R^T W^{-1}(\lambda ) R} \right] , \end{aligned}$$\end{document}gtHLT(E∗;λ,σ)=W-1(λ)tt′ft′(E∗;λ,σ)+Rt′1-RTW-1(λ)f(E∗;λ,σ)RTW-1(λ)R,where we introduced85\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&R_t = \int _0^\infty \textrm{d}E\, b_T(t+a, E), \end{aligned}$$\end{document}Rt=∫0∞dEbT(t+a,E),86\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&f_t(E^*; \lambda , \sigma ) = \left( 1 - \lambda \right) \, \int _{E_0}^\infty \textrm{d}E\, b_t(t+a, E) \Delta _{\sigma }(E^*, E), \end{aligned}$$\end{document}ft(E∗;λ,σ)=1-λ∫E0∞dEbt(t+a,E)Δσ(E∗,E),and87\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W(\lambda )_{tt'} = (1-\lambda ) M_{tt'} + \lambda \frac{(C_Y)_{tt'}}{C(0)^2}, \end{aligned}$$\end{document}W(λ)tt′=(1-λ)Mtt′+λ(CY)tt′C(0)2,where88\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} M_{tt'} = \int _{E_0}^\infty \textrm{d}E\, b_T(t+a, E)\, b_T(t'+a, E). \end{aligned}$$\end{document}Mtt′=∫E0∞dEbT(t+a,E)bT(t′+a,E).The matrix M is an ill-conditioned matrix; as discussed in Sect. 3.1 adding to the distance the B term, which takes into account the covariance of the data, acts as a regulator. A comment on A. For \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T\rightarrow \infty$$\end{document}T→∞, the function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_T$$\end{document}bT becomes an exponential, and therefore, the A term in the distance is89\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A[g] = \int _{E_0}^\infty \textrm{d}E\, \left| \sum _{t=0}^{t_{\textrm{max}}} g_t(E^*;\lambda , \sigma ) e^{-(t+a) E} - \Delta _{\sigma }(E^*,E) \right| ^2. \end{aligned}$$\end{document}A[g]=∫E0∞dE∑t=0tmaxgt(E∗;λ,σ)e-(t+a)E-Δσ(E∗,E)2. Some insight into the HLT solution. The HLT approach focuses on approximating the smearing kernel \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta _{\sigma }(E^*,E)$$\end{document}Δσ(E∗,E) as best as possible, rather than trying to find the best point estimator for the spectral density. It is possible, and desirable, to monitor the quality of the approximation of the smearing kernel using as an estimator90\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \delta _{\sigma }(E^*, E) = 1 - \frac{\bar{\Delta }(E^*,E)}{\Delta _\sigma (E^*,E)}. \end{aligned}$$\end{document}δσ(E∗,E)=1-Δ¯(E∗,E)Δσ(E∗,E).The quality of the approximation of the smearing kernel can be improved by increasing the number of data points, i.e., by increasing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{max}}$$\end{document}tmax. It is worthwhile to emphasize the quantitative character of this estimate; for a given quality of the kernel approximation, e.g., fixing91\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \delta _{\sigma }(E^*, E^*) = 0.05, \end{aligned}$$\end{document}δσ(E∗,E∗)=0.05,allows to explore regions in the σσ vs. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{max}}$$\end{document}tmax plane where the approximation of the smearing kernel is good enough, and therefore to determine the optimal choice of σσ for a given number of data points. However, the quality of the approximation is also affected by the choice of the regularization parameter λλ. Increasing λλ improves the stability of the solution with respect to noise in the data, but it also worsens the approximation of the smearing kernel. The optimal choice of λλ is, therefore, a trade-off between the quality of the approximation of the smearing kernel and the stability of the solution with respect to noise in the data. Gaussian processes A brief introduction to Gaussian processes A stochastic process is a collection of random variables \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{f(x): x \in \mathcal {I}\}$$\end{document}{f(x):x∈I}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {I}$$\end{document}I is the set on which f is defined. For any discrete set of points \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{x_1, \ldots , x_n\}$$\end{document}{x1,…,xn}, the random variables \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{f(x_1), \ldots , f(x_n)\}$$\end{document}{f(x1),…,f(xn)} have a joint probability distribution. A Gaussian process is a stochastic process, such that any finite collection of random variables has a multivariate Gaussian distribution. As such, a Gaussian process is completely specified by its mean function m(x) and its covariance function k(x,x')k(x,x′), defined as92\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} m(x) = \mathbb {E}[f(x)], \qquad k(x,x') = \mathbb {E}[(f(x) - m(x))(f(x') - m(x'))]. \end{aligned}$$\end{document}m(x)=E[f(x)],k(x,x′)=E[(f(x)-m(x))(f(x′)-m(x′))].In the context of the inverse problems discussed in these lectures, we use Gaussian processes to define a prior distribution over functions and then use the observed data to obtain a posterior distribution, as suggested in Sect. 2.3. Setup of the problem. Following the conventions in Sect. 2.2, we represent the function f as an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{grid}}$$\end{document}Ngrid-dimensional real vector, made of the values of the function f at selected points on a grid:93\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f_{\alpha } = f(x_{\alpha })\, \quad \alpha = 1, \ldots , N_{\textrm{grid}}. \end{aligned}$$\end{document}fα=f(xα)α=1,…,Ngrid.These values are promoted to stochastic variables with a joint prior probability distribution that is a multi-dimensional Gaussian with respective mean and covariance given by94\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} m_{\alpha }&= m(x_{\alpha }) = \mathbb {E}_{P}[f_{\alpha }], \end{aligned}$$\end{document}mα=m(xα)=EP[fα],95\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} K_{\alpha _1\alpha _2}&= k(x_{\alpha _1},x_{\alpha _2}) = \textrm{Cov}_{P}[f_{\alpha _1},f_{\alpha _2}], \end{aligned}$$\end{document}Kα1α2=k(xα1,xα2)=CovP[fα1,fα2],where we introduced the subscript P to remind the reader that here we consider averages over the prior distribution P96\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P(f | h) = \frac{1}{\sqrt{\det \left( 2\pi K\right) }} \exp \left\{ - \frac{1}{2} \left( f-m\right) ^T K^{-1} \left( f - m\right) \right\} . \end{aligned}$$\end{document}P(f|h)=1det2πKexp-12f-mTK-1f-m.The covariance matrix of the Gaussian prior in this section is denoted by K, a symbol we used in Sect. 3 to denote the metric in the space of functions. The reason for this choice is that the covariance of the prior distribution plays a similar role to the metric in the Backus–Gilbert method. Any parameter that appears in the definitions of m and k is called an hyperparameter and is collectively denoted by h. For the PDF determinations that we are going to analyze later, we choose m(x) = 0m(x)=0 and a Gibbs kernel97\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} k(x,x') = \sigma ^2 \sqrt{\frac{2 \ell (x) \ell (x')}{\ell (x)^2 + \ell (x')^2}}\, \exp \left[ - \frac{(x-x')^2}{\ell (x)^2 + \ell (x')^2} \right] , \end{aligned}$$\end{document}k(x,x′)=σ22ℓ(x)ℓ(x′)ℓ(x)2+ℓ(x′)2exp-(x-x′)2ℓ(x)2+ℓ(x′)2,where98\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \ell (x) = \ell _0 (x + \delta ). \end{aligned}$$\end{document}ℓ(x)=ℓ0(x+δ).There are two hyperparameters in this kernel, σσ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell _0$$\end{document}ℓ0, while δδ is a small regulator, which is introduced in order to avoid the singularity at x=0x=0. By inspecting Eq. (97), we see that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2$$\end{document}σ2 is the variance of f(x), which is the same for all values of x. This is a choice that is encoded in the prior. Other choices are possible, including the possibility of an x-dependent prefactor. The correlation between two values f(x) and f(x')f(x′) decreases exponentially with the square of the distance between x and x'x′. The rate of the exponential decay is dictated by ℓ (x)ℓ(x) and ℓ (x')ℓ(x′). For this particular choice of the prior, ℓ (x)ℓ(x) grows linearly with x, which implies that the correlation between points decreases as x→ 0x→0. The values of f for nearby points at small x are almost uncorrelated, which implies that the function is free to fluctuate more than at higher values of x. We will discuss later how to constrain the hyperparameters σσ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell _0$$\end{document}ℓ0. Remembering that the theoretical predictions are computed by multiplying the vector f by the forward map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})$$\end{document}(FK)99\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T = (\textrm{FK})f, \end{aligned}$$\end{document}T=(FK)f,it is clear that these are also Gaussian variables, and one can readily show that the average and the covariance of the theoretical prediction T, induced by the prior probability distribution of f, are given by100\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E[T_I]&= (\textrm{FK})_{I\alpha } m_\alpha , \end{aligned}$$\end{document}E[TI]=(FK)Iαmα,101\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{Cov}[T_I,T_J]&= (\textrm{FK})_{I\alpha _1} K_{\alpha _1\alpha _2} (\textrm{FK})^T_{\alpha _1J}. \end{aligned}$$\end{document}Cov[TI,TJ]=(FK)Iα1Kα1α2(FK)α1JT.Clearly, T is a vector of dimensions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{dat}}$$\end{document}Ndat, with components \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_I$$\end{document}TI, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I = 1, \ldots , N_{\textrm{dat}}$$\end{document}I=1,…,Ndat—there is one theoretical prediction for every data point. Finally, we have \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{dat}}$$\end{document}Ndat data points. Their central values, as given by the experiments, are denoted \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_I$$\end{document}YI, while their fluctuations are assumed to be Gaussian, with a covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY, which also taken from experimental analyses. The fluctuations of the data around the central values are described by a stochastic variable102\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \epsilon \sim \mathcal {N}\left( 0, C_Y\right) . \end{aligned}$$\end{document}ϵ∼N0,CY.Remember once again that εϵ is an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{dat}}$$\end{document}Ndat-dimensional vector, with components \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon _I$$\end{document}ϵI. The set of central values and the covariance matrix of the data together constitute what we call the dataset \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{N_{\textrm{dat}}}$$\end{document}DNdat. The Gaussian process solution The solution of the inverse problem in this formalism is encoded in the joint posterior distribution103\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{P}(f, h) = P\left( f, h | D_{N_{\textrm{dat}}}\right) . \end{aligned}$$\end{document}P~(f,h)=Pf,h|DNdat.Using simple probabilities, we can factorize104\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P\left( f, h | D_{N_{\textrm{dat}}}\right) = P(f | h, D_{N_{\textrm{dat}}})\, P(h | D_{N_{\textrm{dat}}}); \end{aligned}$$\end{document}Pf,h|DNdat=P(f|h,DNdat)P(h|DNdat);the joint posterior is the product of two factors. The first one is the posterior probability induced by the data at fixed values of the hyperparameters h, while the second factor is the probability of the hyperparameters given the data. Let us analyze each factor on the right-hand side in turn. The first factor is computed using Bayesian inference105\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P(f | h, D_{N_{\textrm{dat}}}) = \frac{P(D_{N_{\textrm{dat}}} | f, h)\, P(f | h)}{P(D_{N_{\textrm{dat}}} | h)}. \end{aligned}$$\end{document}P(f|h,DNdat)=P(DNdat|f,h)P(f|h)P(DNdat|h).The denominator in Eq. (105) is just a normalization factor, independent of f, which can be computed a posteriori. We focus here on on the expression in the numerator, which is the product of two factors, the likelihood and the prior. The likelihood is the probability of the data given the function f and the hyperparameters h. Assuming that the data are normally distributed around the theoretical predictions, we have106\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P(D_{N_{\textrm{dat}}} | f, h) = \frac{1}{\sqrt{\det \left( 2\pi C_Y\right) }} \exp \left\{ - \frac{1}{2} \left( Y - (\textrm{FK})f\right) ^T C_Y^{-1} \left( Y - (\textrm{FK})f\right) \right\} . \end{aligned}$$\end{document}P(DNdat|f,h)=1det2πCYexp-12Y-(FK)fTCY-1Y-(FK)f.For linear data, i.e. for theoretical predictions that depend linearly on the function f, the likelihood is a Gaussian distribution in f. The prior is also a Gaussian distribution in f, as given by Eq. (96). The product of two Gaussian distributions is another Gaussian distribution, so that the posterior distribution at fixed hyperparameters is also a Gaussian distribution in f. The mean and the covariance of this posterior distribution can be computed analytically, and are given by107\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{m}&= m + K (\textrm{FK})^T\left( C_{YT}\right) ^{-1} (Y - (\textrm{FK})m), \end{aligned}$$\end{document}m~=m+K(FK)TCYT-1(Y-(FK)m),108\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{K}&= K - K (\textrm{FK})^T\left( C_{YT}\right) ^{-1} (\textrm{FK})K, \end{aligned}$$\end{document}K~=K-K(FK)TCYT-1(FK)K,where we introduced the matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{YT}$$\end{document}CYT109\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{YT} = C_Y + (\textrm{FK})K (\textrm{FK})^T. \end{aligned}$$\end{document}CYT=CY+(FK)K(FK)T.Note that, because \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{YT}$$\end{document}CYT is positive-definite, the diagonal elements of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{K}$$\end{document}K~ are smaller than the diagonal elements of K. Recalling that the diagonal elements of K and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{K}$$\end{document}K~ are, respectively, the variances of the prior and posterior distributions, we see that the that Bayesian inference reduced the uncertainty on the function f at fixed hyperparameters. The information contained in the data leads to a reduction of the uncertainty on the function f. It is useful to prove that110\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} (\textrm{FK})^T\left[ C_Y + (\textrm{FK})K (\textrm{FK})^T\right] ^{-1} = \frac{1}{1 + M K} (\textrm{FK})^TC_Y^{-1}, \end{aligned}$$\end{document}(FK)TCY+(FK)K(FK)T-1=11+MK(FK)TCY-1,with the usual definition of the matrix M in Eq. (27), and therefore note that the covariance of the posterior can be rewritten in a form that is more similar to the expression for the covariance of the BG solution111\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{K}^{-1} = K^{-1} + M. \end{aligned}$$\end{document}K~-1=K-1+M.This expression makes it clear that the posterior covariance is the result of the combination of the prior covariance and the information contained in the data, as encoded in M. The posterior covariance is smaller than both the prior covariance and the covariance of the BG solution, which is given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M^{-1}$$\end{document}M-1, since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{K}^{-1}> K^{-1}$$\end{document}K~-1>K-1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{K}^{-1}> M$$\end{document}K~-1>M. Exercise 4 Prove Eqs. (107), (108), (110), and (111). Using Eq. (110), the mean of the posterior distribution can be rewritten as112\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{m} = m + K \frac{1}{1 + M K} (\textrm{FK})^TC_Y^{-1} (Y - (\textrm{FK})m). \end{aligned}$$\end{document}m~=m+K11+MK(FK)TCY-1(Y-(FK)m).This expression makes it clear that the posterior mean is the result of the combination of the prior mean and the information contained in the data, with a corrections that is proportional to the difference between the data and the theoretical predictions computed using the prior mean. For m=0m=0, we can rewrite the posterior mean as113\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{m} = \left[ K \frac{1}{1 + M K}\right] \, (\textrm{FK})^TC_Y^{-1} Y. \end{aligned}$$\end{document}m~=K11+MK(FK)TCY-1Y. A note on Tikhonov regularization. By choosing a prior covariance of the form \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K = \sigma ^2 I$$\end{document}K=σ2I, where λλ is a positive parameter, we can rewrite the posterior mean as114\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{m} = \frac{1}{M + \lambda }\, (\textrm{FK})^TC_Y^{-1} Y, \end{aligned}$$\end{document}m~=1M+λ(FK)TCY-1Y,which is the Tikhonov regularized solution of the inverse problem, with regularization parameter \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda = 1/\sigma ^2$$\end{document}λ=1/σ2. The Tikhonov regularized solution is a particular case of the GP solution, characterized by a diagonal prior covariance, i.e. a prior that does not encode any correlation between the values of the function at different points. The second factor in Eq. (104) is the probability of the hyperparameters given the data. This is computed using Bayes’ theorem as well115\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P(h | D_{N_{\textrm{dat}}}) = \frac{P(D_{N_{\textrm{dat}}} | h)\, P(h)}{P(D_{N_{\textrm{dat}}})}. \end{aligned}$$\end{document}P(h|DNdat)=P(DNdat|h)P(h)P(DNdat).Once again, we only need to focus on the numerator, since the denominator is just a normalization factor. At fixed values of the hyperparameters, the data can be modeled as116\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y = (\textrm{FK})f + \epsilon , \end{aligned}$$\end{document}Y=(FK)f+ϵ,where f is a stochastic variable with distribution given by the prior, and εϵ is a stochastic variable with distribution given by Eq. (102). Therefore, the data are the sum of two Gaussian variables, and are itself a Gaussian variable117\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P(D_{N_{\textrm{dat}}} | h) = \frac{1}{\sqrt{\det \left( 2\pi C_{YT}\right) }} \exp \left\{ - \frac{1}{2} \left( Y - (\textrm{FK})m\right) ^T C_{YT}^{-1} \left( Y - (\textrm{FK})m\right) \right\} . \end{aligned}$$\end{document}P(DNdat|h)=1det2πCYTexp-12Y-(FK)mTCYT-1Y-(FK)m.Choosing a uniform prior for the hyperparameters, the posterior distribution of the hyperparameters is proportional to the likelihood in Eq. (117). The most likely values of the hyperparameters are those that maximize this likelihood. Note that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P(D_{N_{\textrm{dat}}} | h)$$\end{document}P(DNdat|h) is a complicated function of the hyperparameters, since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{YT}$$\end{document}CYT depends on h in a non-trivial way. However, the distribution can be sampled using standard Markov Chain Monte Carlo methods, which allow us to obtain a characterization of the posterior distribution of the hyperparameters. If the posterior distribution of the hyperparameters is sharply peaked around its maximum, we can approximate it with a delta function, and use the most likely values of the hyperparameters in Eqs. (107) and (108) to obtain an approximation of the posterior distribution of f. GP determination of one PDF A simple application of the GP methodology is the determination of the PDF118\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T_3(x) = u(x) + \bar{u}(x) - d(x) - \bar{d}(x), \end{aligned}$$\end{document}T3(x)=u(x)+u¯(x)-d(x)-d¯(x),using DIS data from the BCDMS experiment. The difference between the proton and the deuteron structure functions, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_2^p - F_2^d$$\end{document}F2p-F2d, is proportional to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3 only, so that we can write119\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F_2^p(x,Q^2) - F_2^d(x,Q2) = C_{T_3} \otimes T_3(x). \end{aligned}$$\end{document}F2p(x,Q2)-F2d(x,Q2)=CT3⊗T3(x).This is a simple example of an inverse problem in the realm of PDFs determinations, where only one PDF is involved, and the forward map is given by a convolution with a kernel that is known in perturbation theory. The solution of this problem can be obtained using the GP methodology described in the previous section. All results presented here are obtained using synthetic data generated from a known solution. The approach follows very closely the results presented in Ref. [10]. After applying standard kinematic cuts, we have 248 data points for the structure functions with their corresponding covariance matrix, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY, as described by the experimental analyses [11]. In the case of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3, we want to implement the fact that the function has to be integrable at x=0x=0. This can be achieved by introducing and additional hyperparameter αα, used to rescale the kernel as120\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} k(x,x') \rightarrow \phi (x) \, k(x,x') \, \phi (x'), \quad \phi (x) = x^\alpha , \end{aligned}$$\end{document}k(x,x′)→ϕ(x)k(x,x′)ϕ(x′),ϕ(x)=xα,and imposing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha \in (-1,0]$$\end{document}α∈(-1,0]. The Bayesian inference is performed in two steps. First, we sample the posterior of the hyperparameters, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P(h | D_{N_{\textrm{dat}}})$$\end{document}P(h|DNdat), using Markov Chain Monte Carlo methods. Then, for each set of hyperparameters drawn from this posterior, we use the analytical expressions in Eqs. (107) and (108) and generate a replica of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3. The ensemble of replicas generated in this way provides a characterization of the posterior distribution of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3, which can be used to compute the mean and the uncertainty of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3 at any value of x. Results for the posterior distribution of the hyperparameters are shown in Fig. 1.Fig. 11-dimensional (left panel) and 2-dimensional (right panel) posteriors of the hyperparameters αα, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0$$\end{document}l0 and σσ. The hyperparameter σσ is characterized by a sharply peaked posterior located around \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma \sim 0.25$$\end{document}σ∼0.25 which quickly decays to zero (for this reason in the posterior plot only the region \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( 0,1\right)$$\end{document}0,1 is shown, even if the support of the prior is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( 0,10\right)$$\end{document}0,10); αα tends to sit closer to 0, with a slow decay for smaller values toward -1-1; the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0$$\end{document}l0 posterior discards the smaller values of the correlation length, it shows a peak for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0\sim 1.7$$\end{document}l0∼1.7 and then remains fairly constant. (Plot taken from Ref. [10]) 1-dimensional (left panel) and 2-dimensional (right panel) posteriors of the hyperparameters αα, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0$$\end{document}l0 and σσ. The hyperparameter σσ is characterized by a sharply peaked posterior located around \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma \sim 0.25$$\end{document}σ∼0.25 which quickly decays to zero (for this reason in the posterior plot only the region \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( 0,1\right)$$\end{document}0,1 is shown, even if the support of the prior is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( 0,10\right)$$\end{document}0,10); αα tends to sit closer to 0, with a slow decay for smaller values toward -1-1; the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0$$\end{document}l0 posterior discards the smaller values of the correlation length, it shows a peak for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_0\sim 1.7$$\end{document}l0∼1.7 and then remains fairly constant. (Plot taken from Ref. [10]) The mean value and the uncertainty of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3 obtained from the posterior distribution are shown in Fig. 2. The mean value is taken by averaging over the ensemble of replicas at each value of x, while the uncertainties are obtained from the 68% and 95% confidence intervals of the distribution of replicas. The uncertainty due to the fluctuations of the hyperparameters is included in the final uncertainty of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3, since the replicas are generated by sampling the posterior distribution of the hyperparameters and then using the normal distribution of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3 at fixed hyperparameters. The Bayesian inference is able to reconstruct the functional form of the PDF, as can be seen by comparing the dark blue and the dotted lines in the figure, which represent the mean of the posterior distribution and the input PDF, respectively. The uncertainty on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3 is smaller in the regions where the data are more constraining, and increases in the extrapolation regions at small and large x, where the results are mostly determined by the choice of the prior.Fig. 2Samples from the posterior distribution of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3, plotted in linear (left panel) and log (right panel) scale. The dark blue line represent the mean of the distribution, while the black dotted line the input PDF, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, used to generate pseudo-data. The shaded regions represent the 68CL and 95CL intervals, and in light blue, we plot a few representative samples from the distribution. The posterior displays a smaller variance in the regions sensitive to experimental data, and an increasing spread in the small and large-x extrapolation regions, where the results are mostly determined by the chosen prior. (Plot taken from Ref. [10]) Samples from the posterior distribution of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3, plotted in linear (left panel) and log (right panel) scale. The dark blue line represent the mean of the distribution, while the black dotted line the input PDF, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, used to generate pseudo-data. The shaded regions represent the 68CL and 95CL intervals, and in light blue, we plot a few representative samples from the distribution. The posterior displays a smaller variance in the regions sensitive to experimental data, and an increasing spread in the small and large-x extrapolation regions, where the results are mostly determined by the chosen prior. (Plot taken from Ref. [10]) Closure tests Following the discussion in Sect. 3.3, we can perform closure tests to validate the implementation of the GP method, and to assess the quality of the solution. Ignoring statistical fluctuations, i.e. setting \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y=0$$\end{document}CY=0, synthetic data are generated from a known solution, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, by applying the forward operator121\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y = (\textrm{FK})f_0. \end{aligned}$$\end{document}Y=(FK)f0.In this case, the posterior mean is given by122\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{m} = K (\textrm{FK})^T\left[ (\textrm{FK})K (\textrm{FK})^T\right] ^{+} (\textrm{FK})\, f_0, \end{aligned}$$\end{document}m~=K(FK)T(FK)K(FK)T+(FK)f0,where the superscript + denotes the pseudo-inverse, which is needed, since \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})K (\textrm{FK})^T$$\end{document}(FK)K(FK)T is not invertible when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y=0$$\end{document}CY=0. By comparing with Eq. (64), we see that this solution corresponds to the solution obtained using the BG method with the kernel K as the metric. The two solutions coincide in the limit of zero noise when the hyperparameters are fixed to the values used to generate the data. It is interesting to note that, even when there is no statistical noise, the solution of the inverse problem does not coincide with the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, but it is a smeared version of it, with the same smearing kernel given by the resolution function of the BG method. The solution of the inverse problem is not exact, even in the absence of noise, because of the smoothing effect of the forward map and the regularization effect of the prior. The resolution function in this case is given by123\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {R}^{\textrm{GP}}(x,x') = K(x,x') (\textrm{FK})^T\left[ (\textrm{FK})K (\textrm{FK})^T\right] ^{+} (\textrm{FK}). \end{aligned}$$\end{document}RGP(x,x′)=K(x,x′)(FK)T(FK)K(FK)T+(FK).A nonvanishing experimental error can be readily added to the closure test by adding a noise term to the data, which is normally distributed around zero, with a covariance given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY. In this case, the mean of the posterior distribution is given by Eq. (107) and the reconstruction kernel is124\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {R} = K\, (\textrm{FK})^T\left[ (\textrm{FK})K (\textrm{FK})^T+ C_Y\right] ^{-1} (\textrm{FK}), \end{aligned}$$\end{document}R=K(FK)T(FK)K(FK)T+CY-1(FK),so that the posterior mean and covariance can be rewritten as125\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\tilde{m} - f_0 = \left[ \mathcal {R} - 1\!\!1\right] f_0 + a^T \epsilon , \end{aligned}$$\end{document}m~-f0=R-1f0+aTϵ,126\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\tilde{K} = \left( 1\!\!1 - \mathcal {R}\right) K \left( 1\!\!1 - \mathcal {R}\right) ^T + a^T C_Y a, \end{aligned}$$\end{document}K~=1-RK1-RT+aTCYa,where we have introduced127\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} a^T = K\,(\textrm{FK})^T\left[ (\textrm{FK})\, K \, (\textrm{FK})^T+ C_Y\right] ^{-1}, \end{aligned}$$\end{document}aT=K(FK)T(FK)K(FK)T+CY-1,so that128\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {R} = a^T (\textrm{FK}). \end{aligned}$$\end{document}R=aT(FK).The decomposition in Eqs. (125), (126) highlights the fact that there are two types of contributions to the bias and to the posterior covariance matrix. The first term in both Eqs. (125) and (126) comes from the limited reconstruction of the central value and indeed would vanish when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {R}=1\!\!1$$\end{document}R=1. If \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {R} \ne 1\!\!1$$\end{document}R≠1, this term survives in the limit where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y \rightarrow 0$$\end{document}CY→0, i.e. in the limit of no experimental errors on the data. The second term is the propagation of the covariance of the data into the covariance of the model. In the case \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {R} = 1\!\!1$$\end{document}R=1, the only error fluctuations in the posterior distribution come from this term. In Fig. 3, Monte Carlo samples generated according to the reconstruction and experimental components are plotted separately for the BCDMS results, in red and gray, respectively. In the medium-x region, where more experimental data are available, the PDF uncertainty is dominated by the experimental error, yet a smaller reconstruction error is still present; when moving to the small and large-x extrapolation regions, the reconstruction error becomes the dominant one, pointing out the lack of experimental information. We stress once more how these qualitative considerations are precisely quantified in Eqs. (125), (126), giving the analytical expression for the posterior covariance matrices associated with the experimental and reconstruction error, making it possible to quote different component of the PDF error in the context of a phenomenology analysis.Fig. 3GP reconstruction of the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, with the 68% CL band for the experimental and reconstruction errors plotted separately in gray and red, respectively, according to Eq. (126). The error bands are overlapped rather than stacked. Plot taken from Ref. [10] GP reconstruction of the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, with the 68% CL band for the experimental and reconstruction errors plotted separately in gray and red, respectively, according to Eq. (126). The error bands are overlapped rather than stacked. Plot taken from Ref. [10] Note that in this formalism, it is straightforward to assess the impact of a reduced uncertainty on the data, simply by reducing the covariance matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY and recomputing the posterior distribution. GP for spectral densities Let us now discuss how to apply the GP formalism to the problem of reconstructing spectral densities from Euclidean correlators computed on the lattice. The notation in this section follows very closely the one used in Ref. [12]. GP solution and connection with BG Following the formalism introduced in this section, we represent the spectral density as a stochastic field \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω), which is described by a GP centered around \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho ^{\textrm{prior}}(\omega )$$\end{document}ρprior(ω) and with covariance \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {K}^{\textrm{prior}}(\omega ,E)$$\end{document}Kprior(ω,E). The prior probability measure is therefore129\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Pi [\rho ] = \frac{1}{\mathcal {N}} \, \exp \biggr ( -\frac{1}{2} \left| \rho - \rho ^{\textrm{prior}} \right| ^2_{\mathcal {K}^{\textrm{prior}}} \biggr ) \;, \end{aligned}$$\end{document}Π[ρ]=1Nexp(-12ρ-ρpriorprior2),where130\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left| \rho - \right. \left. \rho ^{\textrm{prior}} \right| ^2_{\mathcal {K}^{\textrm{prior}}} = \int \textrm{d}E_1 \int \textrm{d}E_2 \left[ \rho (E_1) - \rho ^{\textrm{prior}}(E_1) \right] \mathcal {K}_\textrm{prior}^{-1}(E_1,E_2) \left[ \rho (E_2) - \rho ^{\textrm{prior}}(E_2) \right] , \end{aligned}$$\end{document}ρ-ρpriorprior2=∫dE1∫dE2ρ(E1)-ρprior(E1)Kprior-1(E1,E2)ρ(E2)-ρprior(E2),and the normalization131\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {N} = \int \mathcal {D} \rho \; \Pi [\rho ]. \end{aligned}$$\end{document}N=∫DρΠ[ρ].In the previous expressions, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {D} \rho$$\end{document}Dρ represents the functional integration measure over the field variable ρρ. By definition, the expectation value for ρ (E)ρ(E) from the prior distribution is132\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho ^{\textrm{prior}}(\omega ) = \int \mathcal {D}\rho \, \Pi [\rho ] \; \rho (\omega ). \end{aligned}$$\end{document}ρprior(ω)=∫DρΠ[ρ]ρ(ω).We then introduce the noise \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {\eta } \in \mathbb {R}^{\tau _\textrm{max}}$$\end{document}η→∈Rτmax which takes into account the uncertainty in the lattice data and is represented as a real-valued stochastic variable, for which we assume a multivariate Gaussian distribution with zero mean133\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {G}[\vec {\eta }; C_Y] = \frac{1}{\sqrt{\text {det} (2\pi C_Y)}} \exp \left( -\frac{1}{2} \vec {\eta }\; C_Y^{-1} \; \vec {\eta } \right) , \end{aligned}$$\end{document}G[η→;CY]=1det(2πCY)exp-12η→CY-1η→,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY is the covariance matrix of the correlators. We also introduce a stochastic variable associated with the lattice data, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {{\mathcal {C}}} \in \mathbb {R}^{\tau _{\textrm{max}}}$$\end{document}C→∈Rτmax with entries134\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {C}(t) = \int \textrm{d}E \, b_T(t,E) \, \rho (E) + \eta (t). \end{aligned}$$\end{document}C(t)=∫dEbT(t,E)ρ(E)+η(t).Given the distributions of Eqs. (129) and (134), we can evaluate the covariance associated to the variable \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {\mathcal {C}}$$\end{document}C→. It is straightforward to show that135\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} \left\langle \mathcal {C}(t_1)\mathcal {C}(t_2) \right\rangle&= \int {\rm d}\vec {\eta }\;\mathcal {D}\rho \; \mathcal {C}(t_1) \, \mathcal {C}(t_2) \; \mathbb {G}[\vec {\eta }, \text {Cov}_d] \, \Pi [\rho ] \, \\&= \Sigma _{t_1t_2} + \left( C_Y\right) _{t_1t_2} \;, \end{aligned} \end{aligned}$$\end{document}C(t1)C(t2)=∫dη→DρC(t1)C(t2)G[η→,Covd]Π[ρ]=Σt1t2+CYt1t2,where we defined136\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Sigma _{t_1t_2} = \int \textrm{d}E_1 \int \; \textrm{d}E_2 \; b_T(t_1,E_1) \, \mathcal {K}^{\textrm{prior}}(E_1,E_2) \, b_T(t_2,E_2). \end{aligned}$$\end{document}Σt1t2=∫dE1∫dE2bT(t1,E1)Kprior(E1,E2)bT(t2,E2).In order to predict the value of the spectral density at the energy ωω, we need to extend the dimensionality of the covariance of Eq. (135) to include the indirect observation of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω). To this end, we introduce the vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {F} \in \mathbb {R}^{\tau _{\textrm{max}}}$$\end{document}F→∈Rτmax of components137\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F_t(\omega ) = \langle \mathcal {C}(t) \rho (\omega )\rangle = \int {\rm d}\vec {\eta }\, \mathcal {D}\rho \; \mathcal {C}(t) \, \rho (\omega )\, \mathbb {G}[\vec {\eta }, C_Y] \,\Pi [\rho ], \end{aligned}$$\end{document}Ft(ω)=⟨C(t)ρ(ω)⟩=∫dη→DρC(t)ρ(ω)G[η→,CY]Π[ρ],together with the scalar \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_*$$\end{document}F∗138\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F_*(\omega ) = \int \mathcal {D}\rho \; \rho (\omega )^2\; \Pi [\rho ] = \mathcal {K}^{\textrm{prior}}(\omega ,\omega ). \end{aligned}$$\end{document}F∗(ω)=∫Dρρ(ω)2Π[ρ]=Kprior(ω,ω).The total covariance can be written as a block matrix139\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Sigma ^{\textrm{tot}} = \begin{pmatrix} F_*(\omega ) & \vec {F}(\omega )^T \\ \vec {F}(\omega ) & \Sigma +C_Y \end{pmatrix}\;. \end{aligned}$$\end{document}Σtot=F∗(ω)F→(ω)TF→(ω)Σ+CY.Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C^{\textrm{obs}}(t)$$\end{document}Cobs(t) be the correlator measured on the lattice by averaging over a gauge ensemble, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C^{\, \mathrm{prior}}(t)$$\end{document}Cprior(t) the vector \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {\mathcal {C}}$$\end{document}C→ evaluated at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho = \rho ^{\textrm{prior}}$$\end{document}ρ=ρprior, whose components are140\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C^{\textrm{prior}}(t) = \mathcal {C}(t) \, |_{\rho ^{\textrm{prior}}} = \int \textrm{d}E \, b_T(t,E) \, \rho ^{\textrm{prior}}(E). \end{aligned}$$\end{document}Cprior(t)=C(t)|ρprior=∫dEbT(t,E)ρprior(E).The joint probability density for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {C}(t)$$\end{document}C(t) is also a Gaussian distribution141\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {G} \left[ \rho - \rho ^{\textrm{prior}}, \, \vec {\mathcal {C}} - \vec {C}^{\; \mathrm{prior}} ; \, \Sigma ^{\textrm{tot}} \right] \;, \end{aligned}$$\end{document}Gρ-ρprior,C→-C→prior;Σtot,and can be factorized as the product of the posterior probability density for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho (\omega )$$\end{document}ρ(ω) and the likelihood of the data. Both distributions are Gaussian, and therefore142\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {G} \left[ \rho - \rho ^{\textrm{prior}}, \, \vec {\mathcal {C}} - \vec {C}^{\; \mathrm{prior}} ; \, \Sigma ^{\textrm{tot}} \right] = \mathbb {G} \left[ \rho - \rho ^{\textrm{post}}; \, \mathcal {K}^{\textrm{post}}\right] \; \mathbb {G} \left[ \vec {\mathcal {C}} - \vec {C}^{\; \mathrm{prior}} ; \, \Sigma + C_Y \right] . \end{aligned}$$\end{document}Gρ-ρprior,C→-C→prior;Σtot=Gρ-ρpost;KpostGC→-C→prior;Σ+CY. Exercise 5 Prove Eq. (142). The posterior Gaussian distribution for the spectral density is centered around143\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left. \rho ^{\textrm{post}}(\omega ) \right| _{\mathcal {C}=C^{\textrm{obs}}} = \rho ^{\textrm{prior}}(\omega ) + \vec {F}^T(\omega ) \frac{1}{\Sigma + C_Y} \left( \vec {C}^{\;\mathrm obs} - \vec {C}^{\; \mathrm{prior}} \right) , \end{aligned}$$\end{document}ρpost(ω)C=Cobs=ρprior(ω)+F→T(ω)1Σ+CYC→obs-C→prior,and has variance144\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left. \mathcal {K}^\textrm{post}(\omega ,\omega ) \right| _{\mathcal {C}=C^{\textrm{obs}}} = \mathcal {K}^{\textrm{prior}}(\omega ,\omega ) - \vec {F}^T(\omega ) \frac{1}{\Sigma + C_Y} \vec {F}(\omega ). \end{aligned}$$\end{document}Kpost(ω,ω)C=Cobs=Kprior(ω,ω)-F→T(ω)1Σ+CYF→(ω).Note that the posterior mean obtained in Eq. 144 is a linear combination of the data, as in the Backus–Gilbert method discussed in the previous section. In order to explicitly make contact with the BG result that we obtained in the previous section, we introduce the coefficients145\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \vec {g}^{\mathrm{\, GP}} (\omega ) = \vec {F}^T \frac{1}{\Sigma + C_Y}, \end{aligned}$$\end{document}g→GP(ω)=F→T1Σ+CY,where the superscript GP reminds us that these are the coefficients obtained using the GP method. Equation (145) shows that the problem is regularized by formulating it in terms of probability distributions. The numerical instability of the BG solution, which manifests itself in the very large coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\vec {g}^{\mathrm{\, GP}} (\omega )$$\end{document}g→GP(ω), is the consequence of the large condition number of the model covariance ΣΣ of Eq. (136), which grows exponentially for acceptable choices of the model covariance \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {K}$$\end{document}K. In the absence of errors on the data, the magnitude of the coefficients is determined by the inverse of the matrix ΣΣ. For noisy data, the covariance \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY is added to the matrix ΣΣ, thereby providing a cut-off for its lower modes. The magnitude of the coefficients is now determined by the condition number of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma + C_Y$$\end{document}Σ+CY. We will later show how the choice of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {K}^{\textrm{prior}}$$\end{document}Kprior can be used to tune the cut-off on the low modes of ΣΣ. Another important remark concerns the smearing kernel that one has implicitly introduced in Eq. (143)146\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {S}^{\textrm{GP}}(\omega ,E) = \sum _{\tau =1}^{\tau _{\textrm{max}}} \vec {g}^{\mathrm{\, GP}} (\omega ) \,b_T(a\tau ,E). \end{aligned}$$\end{document}SGP(ω,E)=∑τ=1τmaxg→GP(ω)bT(aτ,E).The center of the posterior distribution, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho ^{\textrm{post}}(\omega )$$\end{document}ρpost(ω), is an estimator for a spectral density that is smeared with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}^{\textrm{GP}}$$\end{document}SGP. The unsmeared spectral density can be obtained if the prior is engineered, such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {S}^\textrm{GP}(\omega -E)$$\end{document}SGP(ω-E) is a Dirac delta. Such a distribution cannot be obtained by a finite linear combination of the regular functions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_T(t,E)$$\end{document}bT(t,E), and one would need to extrapolate the result at vanishing smearing radius. We conclude this section with a crucial remark on the nature of the prior. The functional behavior of the prediction is related to the prior through Eqs. (143) and (144). In particular, the covariance of the prior is known to affect the typical correlation length of the posterior. On the lattice, finite-volume spectral densities are targeted; therefore, this length should be smaller than the typical spacing between energy levels in order to capture the features of the underlying physics. Choice of the priors The prior model covariance used in this lectures is a variation of a Gaussian covariance147\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {K}(\omega ,E) = \frac{e^{\alpha E}}{\lambda } \frac{e^{-\frac{(\omega -E)^2}{2\epsilon ^2}}}{\sqrt{2\pi } \epsilon } \equiv \frac{e^{\alpha E}}{\lambda } G_\epsilon (\omega -E), \end{aligned}$$\end{document}K(ω,E)=eαEλe-(ω-E)22ϵ22πϵ≡eαEλGϵ(ω-E),where εϵ, αα, and λλ are the hyperparameters that fully specify the prior. The motivations behind the choice made in Eq. (147) are the following. First, a Gaussian is a common choice in the literature [8, 13], with width εϵ and amplitude \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{-1}$$\end{document}λ-1 as parameters to be chosen. In addition, the Gaussian distribution has a simple limit in which the covariance becomes diagonal by changing the parameter εϵ. The term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$e^{\alpha E}$$\end{document}eαE allows us to control the deviations from the Gaussian case, and provides a link with the results obtained using the method of Ref. [9]. In order to set the notation, we specialize the previous equations to this choice of the prior148\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} & \frac{\Sigma ^\epsilon _{tr}}{\lambda } = \int \textrm{d}E_1\, \textrm{d}E_2\; b_T(E_1,t) \, b_T(E_2,r)\, e^{\alpha E_1} \, \frac{G_\epsilon (E_1-E_2)}{\lambda }, \end{aligned}$$\end{document}Σtrϵλ=∫dE1dE2bT(E1,t)bT(E2,r)eαE1Gϵ(E1-E2)λ,149\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} & \frac{F^{\epsilon }_t(\omega )}{\lambda } = \int \textrm{d}E \, b_T(t,E) \, e^{\alpha E} \; \frac{G_\epsilon (E-\omega )}{\lambda }. \end{aligned}$$\end{document}Ftϵ(ω)λ=∫dEbT(t,E)eαEGϵ(E-ω)λ.The resulting expression for the coefficients of Eq. (145) is150\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \vec {g}^{\textrm{GP}}(\epsilon ; \omega ) = \vec {F}^{\, T}(\omega ) \, \frac{1}{\Sigma ^\epsilon + \lambda \, C_Y} \;. \end{aligned}$$\end{document}g→GP(ϵ;ω)=F→T(ω)1Σϵ+λCY.It is clear that λλ is parametrizing the cut-off on the low modes of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma ^\epsilon$$\end{document}Σϵ, thus introducing a bias. When λλ becomes larger the role of the regularizing term, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY, is enhanced, and the coefficients \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g_\tau$$\end{document}gτ become increasingly smaller. At smaller values of λλ, the bias decreases, but it cannot be eliminated. Its dependence must be, therefore, addressed, and its effect quantified and controlled. The role of such parameter is extensively discussed in the context of Backus–Gilbert methods, and in particular in its formulation of Ref. [9], where it is usually prescribed to choose λλ, such that the prediction for the spectral density is stable within statistical noise [14–16] upon variations of λλ. From this “stability analysis”, the bias is assumed to be absorbed into the statistical error, a procedure that has been validated numerically; see for instance Refs. [14, 16, 17]. In the context of GPs, on the other hand, the hyperparameters (including λλ) are selected, so that the resulting probability of observing the data151\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {G} \left[ \vec {C}^{\, \mathrm obs} - \vec {C}^{\; \mathrm{prior}} ; \, \Sigma + C_Y \right] , \end{aligned}$$\end{document}GC→obs-C→prior;Σ+CY,is maximized [13, 18].5 Equivalently, one minimizes the “negative logarithmic likelihood” (NLL)152\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\tau _{\textrm{max}}}{2}\, \text {Log} (2\pi ) +\frac{1}{2} \, \text {Log}\, \text {det} \left( \Sigma + \text {Cov}_d \right) +\frac{1}{2} \, \left(\vec {C}^{\, \mathrm obs}- \vec {C}^{\; \mathrm{prior}}\right) \frac{1}{\Sigma +\text {Cov}_d} \left(\vec {C}^{\, \mathrm obs}- \vec {C}^{\; \mathrm{prior}}\right). \end{aligned}$$\end{document}τmax2Log(2π)+12LogdetΣ+Covd+12C→obs-C→prior1Σ+CovdC→obs-C→prior. In this setting, the fate of the bias introduced by λλ could be considered opaque. We, therefore, assess whether porting the “stability analysis” that is carried in Ref. [9] into the Bayesian setup, can add insights in this regard. In Fig. 4, we show how the choice of λλ affects the spectral reconstruction (left panel)6 at a specific energy. The corresponding values of the NLL are shown in the right panel of the same figure, with the minimum value highlighted. For large values of λλ, the smeared spectral density changes considerably, showing a large dependence on the prior. The corresponding values of the NLL are large. Remarkably, as the NLL approaches its minimum, the dependence on the prior softens. The horizontal band in the left panel of Fig. 4 is obtained at the value of λλ that minimizes the NLL, which is flagged by a star in the right panel. This result suggests that the treatment of the bias might not be drastically different in these two cases, despite the very different approaches.Fig. 4Combination of the stability analysis used (left) and a scan of the NLL (right). The smeared density shown in the left panel is obtained at a specific energy, and is evaluated from GPs, i.e., from Eqs. (143) and (144) for the central value and the error, respectively. The prior is the modified Gaussian of Eq. (147), with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon \simeq m_0$$\end{document}ϵ≃m0, the latter being the ground state of the channel. We only display value corresponding to α =0α=0, because these correspond to a systematically smaller NLL. The purpose of this figure is to show that the treatment of the parameter λλ from Ref. [9] and GPs can lead to compatible results. Details about the lattice data used for this example are found in the main text Combination of the stability analysis used (left) and a scan of the NLL (right). The smeared density shown in the left panel is obtained at a specific energy, and is evaluated from GPs, i.e., from Eqs. (143) and (144) for the central value and the error, respectively. The prior is the modified Gaussian of Eq. (147), with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon \simeq m_0$$\end{document}ϵ≃m0, the latter being the ground state of the channel. We only display value corresponding to α =0α=0, because these correspond to a systematically smaller NLL. The purpose of this figure is to show that the treatment of the parameter λλ from Ref. [9] and GPs can lead to compatible results. Details about the lattice data used for this example are found in the main text Another important remark, already stated in the previous section, is that the value obtained at the minimum of the NLL (horizontal band in Fig. 4) has to be interpreted and understood as a spectral density that is smeared with the appropriate smearing kernel, which is not known a priori in the Bayesian formulation given in this section. In this respect, this method is closer to the original proposal of Backus and Gilbert rather than the modification of Ref. [9]. For this reason, we find it instructive to show, in the left panel of Fig. 5, an example of the smearing kernel at a specific energy (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega = 3.7 m_\pi$$\end{document}ω=3.7mπ) obtained with the Bayesian setup (orange), as opposed to the smearing kernel obtained from the method (in blue) of Ref. [9] (HLT in short) that will be described in the next section. The Bayesian kernel is not known a priori: its behavior is constrained by the covariance of the prior, which can produce a smoother or more rapidly changing function. In the example of Fig. 5, the prior is defined in Eq. 147, with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon = 0.75 m_\pi$$\end{document}ϵ=0.75mπ. The output thus features, around \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega = 3.7 m_\pi$$\end{document}ω=3.7mπ, oscillation that have roughly a wavelength of εϵ. In the right panel of the same figure, the smeared spectral density obtained with the two methods is also displayed. In this example, we have used synthetic data without any statistical noise, in order to showcase what each method does in the ideal limit of exact data. The input correlator contains a single state with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E/m_\pi = 3.7$$\end{document}E/mπ=3.7.Fig. 5Left panel: examples of the function smearing the spectral density at the energy \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E = 3.7 m_\pi$$\end{document}E=3.7mπ in the Bayesian setup (orange) and from the HLT procedure (blue) using exact data. The latter targets a Gaussian kernel with a width of approximately 0.75 in units of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_\pi$$\end{document}mπ, which is reconstructed with great precision given the lack of uncertainties on the input data. For the Bayesian calculation, we use the same Gaussian function as a prior, but we have less control on the output function, which in this case features oscillations with a length scale determined by the prior. The right panel displays the reconstructed smeared spectral densities from the same data. This example uses \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{max}}=32$$\end{document}tmax=32 data points Left panel: examples of the function smearing the spectral density at the energy \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E = 3.7 m_\pi$$\end{document}E=3.7mπ in the Bayesian setup (orange) and from the HLT procedure (blue) using exact data. The latter targets a Gaussian kernel with a width of approximately 0.75 in units of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_\pi$$\end{document}mπ, which is reconstructed with great precision given the lack of uncertainties on the input data. For the Bayesian calculation, we use the same Gaussian function as a prior, but we have less control on the output function, which in this case features oscillations with a length scale determined by the prior. The right panel displays the reconstructed smeared spectral densities from the same data. This example uses \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{max}}=32$$\end{document}tmax=32 data points Neural networks Neural networks basics In the context of PDF fitting, the NNPDF collaboration has pioneered the use of neural networks as a flexible parametrization of PDFs, which allows to reduce the bias introduced by the choice of a specific functional form. The NNPDF approach is based on the idea of using a large number of parameters (the weights and biases of the neural network) to fit the data, while monitoring the fitting process in order to prevent overfitting, see e.g. Ref. [2] for details. In this language, the ensemble of NN replicas at initialization defines the prior probability distribution of the model, and the training process is used to update the probability distribution in light of the data, thus defining the posterior distribution. Note that, while we have prior and posterior distributions, the inference is not performed according to Bayes’ theorem. In these section, we briefly recall some basic notions about neural networks, and develop a statistical field theory description of the behavior of neural networks at initialization, which will be the foundation for the discussion of the training process in the next section. A neural network is one way of parametrizing a function, f(x), using interconnected layers of computing units, which are called neurons by analogy with the biological neurons. We are going to focus here on the simplest type of neural network, which is the feed-forward neural network, where the information flows in one direction from the input layer, identified by ℓ =0ℓ=0 to the output layer, identified by ℓ =Lℓ=L. The neuron in each layer is labeled by an index i, and the number of neurons in layer ℓℓ is denoted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_\ell$$\end{document}nℓ. The input of the network is x, which is a vector of dimension \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_0$$\end{document}n0, and the output is f(x), which is a vector of dimension \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_L$$\end{document}nL. If we are parametrizing a function that returns a scalar, then \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_L=1$$\end{document}nL=1. At each neuron, we associated a so-called pre-activation function, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^{(\ell )}_i(x)$$\end{document}ϕi(ℓ)(x). The indices ℓℓ and i identify the neuron, and we explicitly wrote the dependence on the input x. The output of the neuron is obtained by applying the activation function ρρ to the pre-activation153\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \rho ^{(\ell )}_i(x) = \rho \left( \phi ^{(\ell )}_i(x) \right) . \end{aligned}$$\end{document}ρi(ℓ)(x)=ρϕi(ℓ)(x).The pre-activation function are recursively defined154\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \phi ^{(\ell )}_i(x) = \sum _{j=1}^{n_{\ell -1}} W^{(\ell )}_{ij} \, \rho ^{(\ell -1)}_j(x) + b^{(\ell )}_i, \end{aligned}$$\end{document}ϕi(ℓ)(x)=∑j=1nℓ-1Wij(ℓ)ρj(ℓ-1)(x)+bi(ℓ),with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^{(0)}_i(x) = x_i$$\end{document}ϕi(0)(x)=xi. The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^{(\ell )}$$\end{document}W(ℓ) are the weight matrices, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b^{(l)}$$\end{document}b(l) are the bias vectors. For each input x, the recursion relations above allow us to compute the pre-activation and the output of each neuron, and ultimately the output of the network, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_i(x)=\phi ^{(L)}_i(x)$$\end{document}fi(x)=ϕi(L)(x). The weights and biases are the parameters of the neural network, they are initialized according to some prescription and they are then determined by the training process. Neural networks provide a flexible parametrization of functions and are known to be universal approximators for sufficiently large number of parameters. They were originally introduced for PDF fitting precisely in order to avoid any bias introduced by a fixed functional form with insufficient degrees of freedom. Typically, we consider a discrete set of input points, labeled by an index αα as discussed in the previous sections. The input points are therefore \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_\alpha$$\end{document}xα. In order to simplify the notation, we introduce155\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \phi ^{(\ell )}_{i\alpha } = \phi ^{(\ell )}_i(x_\alpha ), \qquad \rho ^{(\ell )}_{i\alpha } = \rho ^{(\ell )}_i(x_\alpha ). \end{aligned}$$\end{document}ϕiα(ℓ)=ϕi(ℓ)(xα),ρiα(ℓ)=ρi(ℓ)(xα).It is important to distinguish between the index αα, which labels the input points, and the index i, which labels the neurons. For each αα, the input x could be a vector, whose components are labeled by the index \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i=1, \ldots , n_0$$\end{document}i=1,…,n0. A typical example of an NN is shown in Fig. 6. Each dot in the figure represents a neuron, so that for each dot, we have a pre-activation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^{(\ell )}_{i\alpha }$$\end{document}ϕiα(ℓ) and an output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho ^{(\ell )}_{i\alpha }$$\end{document}ρiα(ℓ). The actual values of the pre-activation and the output depend on the input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_\alpha$$\end{document}xα, which is not explicitly shown in the figure. The connections between the dots represent the weights \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W^{(\ell )}_{ij}$$\end{document}Wij(ℓ), and the bias \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b^{(\ell )}_i$$\end{document}bi(ℓ) is associated with each neuron. The input layer is in green on the left, the output layer is in red on the right, and the hidden layers are in blue. Note that the NN shown in the figure has two neurons in the input layer. Having two neurons in the input layer allows us to use both x and łog xlogx as inputs, which enhances the sensitivity of the NN to the small-x region.Fig. 6A feed-forward neural network with L=3L=3 layers and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_0=2$$\end{document}n0=2, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1=25$$\end{document}n1=25, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2=20$$\end{document}n2=20, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_3=8$$\end{document}n3=8. The input layer is in green on the left, the output layer is in red on the right, and the hidden layers are in blue. This is the actual architecture of the neural network used by the NNPDF collaboration for PDF fitting; see Ref. [2] A feed-forward neural network with L=3L=3 layers and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_0=2$$\end{document}n0=2, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1=25$$\end{document}n1=25, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2=20$$\end{document}n2=20, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_3=8$$\end{document}n3=8. The input layer is in green on the left, the output layer is in red on the right, and the hidden layers are in blue. This is the actual architecture of the neural network used by the NNPDF collaboration for PDF fitting; see Ref. [2] Neural networks at initialization The weights and biases of the NN are initialized as independent Gaussian variables156\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W^{(\ell )}_{ij} \sim \mathcal {N}\left( 0, \frac{C_W}{n_{\ell -1 }}\right) , \qquad b^{(\ell )}_i \sim \mathcal {N}(0, C_b). \end{aligned}$$\end{document}Wij(ℓ)∼N0,CWnℓ-1,bi(ℓ)∼N(0,Cb).Both distributions have zero mean. The variance of the weights in layer ℓℓ is scaled by the number of neurons in the previous layer, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{\ell -1}$$\end{document}nℓ-1. This is a common choice, sometimes referred to as the Glorot initialization [19]. By inspecting the recursion relation in Eq. 154, it is clear that the scaling of the variance guarantees that the pre-activations have a finite limit when the number of neurons in the previous layer becomes large. The average over the probability distributions in Eq. 156 yields157\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\mathbb {E}[W^{(\ell )}_{ij}] = 0, \end{aligned}$$\end{document}E[Wij(ℓ)]=0,158\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\mathbb {E}[b^{(\ell )}_i] = 0, \end{aligned}$$\end{document}E[bi(ℓ)]=0,159\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\mathbb {E}[W^{(\ell )}_{ij} W^{(\ell )}_{i'j'}] = \delta _{ii'}\delta _{jj'} \frac{C_W}{n_{\ell -1}}, \end{aligned}$$\end{document}E[Wij(ℓ)Wi′j′(ℓ)]=δii′δjj′CWnℓ-1,160\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\mathbb {E}[b^{(\ell )}_i b^{(\ell )}_{i'}] = \delta _{ii'} \, C_b, \end{aligned}$$\end{document}E[bi(ℓ)b′(ℓ)]=δii′Cb,and all other correlations—in particular correlations between different layers—vanish. The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^{(0)}$$\end{document}ϕ(0) are given by the values of the input, and therefore are deterministic quantities. All other pre-activations are stochastic variables and we are interested in their joint probability distribution161\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P\left( \phi ^{(L)} \ldots \phi ^{(1)}\right) = P\left( \phi ^{(L)}|\phi ^{(L-1)}\right) \ldots P\left( \phi ^{(1)}|\phi ^{(0)}\right) . \end{aligned}$$\end{document}Pϕ(L)…ϕ(1)=Pϕ(L)|ϕ(L-1)…Pϕ(1)|ϕ(0).The following derivations are based on the recent review in Ref. [20]. A forward recursion relation. The probability distribution of the pre-activations in layer ℓℓ obeys a forward recursion relation, which allows to compute it from the probability distribution of the pre-activations in layer ℓ -1ℓ-1162\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P\left( \phi ^{(\ell +1)} | \phi ^{(0)}\right) = \int \textrm{d}\phi ^{(\ell )} \, P\left( \phi ^{(\ell +1)} | \phi ^{(\ell )}\right) P\left( \phi ^{(\ell )} | \phi ^{(0)}\right) . \end{aligned}$$\end{document}Pϕ(ℓ+1)|ϕ(0)=∫dϕ(ℓ)Pϕ(ℓ+1)|ϕ(ℓ)Pϕ(ℓ)|ϕ(0).The conditional probability distribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P\left( \phi ^{(\ell +1)} | \phi ^{(\ell )}\right)$$\end{document}Pϕ(ℓ+1)|ϕ(ℓ) can be computed by integrating over the weights and biases163\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P\left( \phi ^{(\ell +1)}|\phi ^{(\ell )}\right)&= \int \prod _{i,j} \textrm{d}W^{(\ell +1)}_{ij} \prod _i \textrm{d}b^{(\ell +1)}_i \, p\left( W^{(\ell +1)}\right) p\left( b^{(\ell +1)}\right) \, \nonumber \\&\qquad \times \ \prod _i \delta \left( \phi ^{(\ell +1)}_i - \sum _j W^{(\ell +1)}_{ij} \rho ^{(\ell )}_j - b^{(\ell +1)}_i \right) , \end{aligned}$$\end{document}Pϕ(ℓ+1)|ϕ(ℓ)=∫∏i,jdWij(ℓ+1)∏idbi(ℓ+1)pW(ℓ+1)pb(ℓ+1)×∏iδϕi(ℓ+1)-∑jWij(ℓ+1)ρj(ℓ)-bi(ℓ+1),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\left( W^{(\ell +1)}\right)$$\end{document}pW(ℓ+1) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\left( b^{(\ell +1)}\right)$$\end{document}pb(ℓ+1) are the probability distributions of the weights and biases in layer ℓ +1ℓ+1 given in Eq. 156. The delta functions in the last line enforce the definition of the pre-activations in terms of the weights, biases, and activations in the previous layer. For fixed values of the pre-activations in layer ℓℓ, the pre-activations in layer ℓ +1ℓ+1 are linear combinations of the weights and biases, and therefore, their probability distribution is Gaussian164\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P\left( \phi ^{(\ell +1)}|\phi ^{(\ell )}\right) = \frac{1}{\left| 2\pi \hat{G}^{(\ell +1)}\right| ^{n_{\ell +1}/2}} \, \exp \left( -\frac{1}{2} \phi ^{(\ell +1)}_{\alpha _1} \cdot \phi ^{(\ell +1)}_{\alpha _2} \left( \hat{G}^{(\ell +1)}\right) ^{-1}_{\alpha _1 \alpha _2} \right) , \end{aligned}$$\end{document}Pϕ(ℓ+1)|ϕ(ℓ)=12πG^(ℓ+1)nℓ+1/2exp-12ϕ1(ℓ+1)·ϕ2(ℓ+1)G^(ℓ+1)α1α2-1,where the scalar product indicates the contraction of the neuron indices, and we introduced the propagator165\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{G}^{(\ell +1)}_{\alpha _1 \alpha _2} = C_b + C_W \, \frac{1}{n_\ell } \rho ^{(\ell )}_{\alpha _1} \cdot \rho ^{(\ell )}_{\alpha _2}. \end{aligned}$$\end{document}G^α1α2(ℓ+1)=Cb+CW1nℓρ1(ℓ)·ρ2(ℓ).The factor of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1/n_\ell$$\end{document}1/nℓ in Eq. 165 comes from the scaling of the variance of the weights in Eq. 156. The scalar product between the outputs \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho ^{(\ell )}$$\end{document}ρ(ℓ) involves a sum over the neuron indices, and therefore, it is of order \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_\ell$$\end{document}nℓ; the propagator has a finite limit when the number of neurons in layer ℓℓ becomes large. Clearly, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{G}^{(\ell +1)}$$\end{document}G^(ℓ+1) depends on the pre-activations \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi ^{(\ell )}$$\end{document}ϕ(ℓ). Exercise 6 Show that the correlators of the pre-activations in any layer ℓℓ are symmetric under rotations in the space of neurons, i.e.166\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}\left[ R_{i_1j_1} \phi ^{(\ell )}_{j_1 \alpha _1} \ldots R_{i_nj_n} \phi ^{(\ell )}_{j_n \alpha _n} \right] = \mathbb {E}\left[ \phi ^{(\ell )}_{i_1 \alpha _1} \ldots \phi ^{(\ell )}_{i_n \alpha _n} \right] , \end{aligned}$$\end{document}ERi1j1ϕj1α1(ℓ)…Rinjnϕjnαn(ℓ)=Eϕi1α1(ℓ)…ϕinαn(ℓ),where R is an orthogonal matrix in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {SO}(n_{\ell })$$\end{document}SO(nℓ). Equation(166) implies that the probability distribution is also invariant under rotations, and therefore, it can only be a function of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {SO}(n_{\ell })$$\end{document}SO(nℓ) invariants, i.e. it can only depend on the scalar product of the pre-activations. In the first hidden layer, ℓ = 1ℓ=1, the pre-activations are linear combinations of the weights and biases167\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \phi ^{(1)}_{i\alpha } = \sum _j W^{(1)}_{ij} \rho \left( x_{\alpha j}\right) + b^{(1)}_i, \end{aligned}$$\end{document}ϕiα(1)=∑jWij(1)ρxαj+bi(1),and therefore, they are Gaussian variables, with zero mean and covariance168\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}\left[ \phi ^{(1)}_{i\alpha } \phi ^{(1)}_{i'\alpha '} \right] = \delta _{ii'} \left( C_b + C_W \, \frac{1}{n_0} \sum _j \rho \left( x_{\alpha j}\right) \rho \left( x_{\alpha ' j}\right) \right) . \end{aligned}$$\end{document}Eϕiα(1)ϕi′α′(1)=δii′Cb+CW1n0∑jρxαjρxα′j.One can readily check, by writing down the explicit expression, that the distributions are no longer Gaussian as soon as we move to ℓ> 1ℓ>1. Effective theory approach. It is useful to rewrite the recursion relation in (162) by introducing an effective action, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S^{(\ell )}$$\end{document}S(ℓ), defined by169\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} P\left( \phi ^{(\ell )} | \phi ^{(0)}\right) = \frac{1}{Z^{(\ell )}} \, e^{-S^{(\ell )}(\phi ^{(\ell )})}, \end{aligned}$$\end{document}Pϕ(ℓ)|ϕ(0)=1Z(ℓ)e-S(ℓ)(ϕ(ℓ)),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Z^{(\ell )}$$\end{document}Z(ℓ) is a normalization factor. The recursion relation in Eq. 162 can be rewritten as170\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{e^{-S^{(\ell +1)}(\phi ^{(\ell +1)})}}{Z^{(\ell +1)}} = \int \textrm{d}\phi ^{(\ell )} \, \frac{e^{-S^{(\ell )}(\phi ^{(\ell )})}}{Z^{(\ell )}} \, \frac{\exp \left( -\frac{1}{2} \phi ^{(\ell +1)}_{\alpha _1} \cdot \phi ^{(\ell +1)}_{\alpha _2} \left( \hat{G}^{(\ell +1)}\right) ^{-1}_{\alpha _1 \alpha _2} \right) }{\left| 2\pi \hat{G}^{(\ell +1)}\right| ^{n_{\ell +1}/2}}. \end{aligned}$$\end{document}e-S(ℓ+1)(ϕ(ℓ+1))Z(ℓ+1)=∫dϕ(ℓ)e-S(ℓ)(ϕ(ℓ))Z(ℓ)exp-12ϕ1(ℓ+1)·ϕ2(ℓ+1)G^(ℓ+1)α1α2-12πG^(ℓ+1)nℓ+1/2.In order to extract further information from the recursion relation, we need to parametrize the effective action. The most general form of the effective action is171\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} S^{(\ell )}(\phi ^{(\ell )}) = \frac{1}{2}\, \gamma ^{(\ell )}_{\alpha _1 \alpha _2} \, \left( \phi ^{(\ell )}_{\alpha _1} \cdot \phi ^{(\ell )}_{\alpha _2}\right) + \frac{1}{8 n_{\ell -1}} \, \gamma ^{(\ell )}_{\alpha _1 \alpha _2; \alpha _3 \alpha _4} \, \left( \phi ^{(\ell )}_{\alpha _1} \cdot \phi ^{(\ell )}_{\alpha _2} \right) \left( \phi ^{(\ell )}_{\alpha _3} \cdot \phi ^{(\ell )}_{\alpha _4} \right) + \cdots , \end{aligned}$$\end{document}S(ℓ)(ϕ(ℓ))=12γα1α2(ℓ)ϕ1(ℓ)·ϕ2(ℓ)+18nℓ-1γα1α2;α3α4(ℓ)ϕ1(ℓ)·ϕ2(ℓ)ϕ3(ℓ)·ϕ4(ℓ)+⋯,where we introduced the couplings \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma ^{(\ell )}_{\alpha _1 \alpha _2}$$\end{document}γα1α2(ℓ) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma ^{(\ell )}_{\alpha _1 \alpha _2; \alpha _3 \alpha _4}$$\end{document}γα1α2;α3α4(ℓ), which are functions of the layer ℓℓ and of the input points. Note that because of symmetry reasons, the effective action can only depend on the scalar product of the pre-activations. The factors in the action reflect the symmetries of each term, and the scaling with the number of neurons in the previous layer is chosen, so that the couplings have a finite limit when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{\ell -1}$$\end{document}nℓ-1 becomes large. We have omitted higher order terms in the effective action; they involve higher powers of the scalar product of the pre-activations, and they are suppressed by higher powers of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1/n_{\ell -1}$$\end{document}1/nℓ-1. At leading order, the second and fourth cumulants are respectively172\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\langle \phi ^{(\ell )}_{i_1,\alpha _1} \phi ^{(\ell )}_{i_2,\alpha _2}\rangle = \delta _{i_1 i_2} K^{(\ell )}_{\alpha _1\alpha _2} + O(1/n_{\ell -1}), \end{aligned}$$\end{document}⟨ϕi1,α1(ℓ)ϕi2,α2(ℓ)⟩=δi1i2Kα1α2(ℓ)+O(1/nℓ-1),173\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\langle \phi ^{(\ell )}_{i_1,\alpha _1} \phi ^{(\ell )}_{i_2,\alpha _2} \phi ^{(\ell )}_{i_3,\alpha _3} \phi ^{(\ell )}_{i_4,\alpha _4}\rangle _c = O(1/n_{\ell -1}), \end{aligned}$$\end{document}⟨ϕi1,α1(ℓ)ϕi2,α2(ℓ)ϕi3,α3(ℓ)ϕi4,α4(ℓ)⟩c=O(1/nℓ-1),where7174\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} K^{(\ell )}_{\alpha _1\alpha _2} = \left( \gamma ^{(\ell )}\right) ^{-1}_{\alpha _1\alpha _2}. \end{aligned}$$\end{document}Kα1α2(ℓ)=γ(ℓ)α1α2-1. Exercise 7 Derive the Feynman rules for the effective theory defined by the action in Eq. (171), and use them to compute the second and fourth cumulants of the pre-activations at leading order in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1/n_{\ell -1}$$\end{document}1/nℓ-1, thus showing that they are given by the expressions in Eqs. (172) and (173). The “evolution” of the couplings as we go deep in the NN, i.e., the dependence of the couplings on ℓℓ, is governed by Renormalization Group (RG) equations, which preserve the power counting in powers of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1/n_{\ell }$$\end{document}1/nℓ. At leading order175\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} K^{(\ell +1)}_{\alpha _1\alpha _2}&= \left. C_b^{(\ell +1)} + C_w^{(\ell +1)} \frac{n_\ell }{n_\ell +n_{\ell +1}}\frac{1}{n_\ell } \langle \vec {\rho }^{\,(\ell )}_{\alpha _1} \cdot \vec {\rho }^{\,(\ell )}_{\alpha _2} \rangle \right| _{O(1)} \end{aligned}$$\end{document}Kα1α2(ℓ+1)=Cb(ℓ+1)+Cw(ℓ+1)nℓnℓ+nℓ+11nℓ⟨ρ→1(ℓ)·ρ→2(ℓ)⟩O(1)176\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&= C_b^{(\ell +1)} + C_w^{(\ell +1)} \frac{n_\ell }{n_\ell +n_{\ell +1}}\frac{1}{n_\ell } \langle \vec {\rho }^{\,(\ell )}_{\alpha _1} \cdot \vec {\rho }^{\,(\ell )}_{\alpha _2} \rangle _{K^{(\ell )}}, \end{aligned}$$\end{document}=Cb(ℓ+1)+Cw(ℓ+1)nℓnℓ+nℓ+11nℓ⟨ρ→1(ℓ)·ρ→2(ℓ)⟩K(ℓ),where\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{1}{n_\ell } \langle \vec {\rho }^{\,(\ell )}_{\alpha _1} \cdot \vec {\rho }^{\,(\ell )}_{\alpha _2} \rangle _{K^{(\ell )}} = \int \mathcal {D}\phi \, \frac{e^{-\frac{1}{2} \left( K^{(\ell )}\right) ^{-1}_{\beta _1\beta _2} \phi _{\beta _1} \phi _{\beta _2}}}{\left| 2\pi K^{(\ell )}\right| ^{1/2}}\, \rho (\phi _{\alpha _1}) \rho (\phi _{\alpha _2}), \end{aligned}$$\end{document}1nℓ⟨ρ→1(ℓ)·ρ→2(ℓ)⟩K(ℓ)=∫Dϕe-12K(ℓ)β1β2-1ϕβ1ϕβ22πK(ℓ)1/2ρ(ϕα1)ρ(ϕα2),and177\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {D}\phi = \prod _{\alpha =1}^{N_{\textrm{grid}}} \textrm{d}\phi _\alpha . \end{aligned}$$\end{document}Dϕ=∏α=1Ngriddϕα.Note that the integration variables in Eq. (177) do not have a neuron index and the integrals are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{\textrm{grid}}$$\end{document}Ngrid dimensional integrals. Equation (176) is iterated for the NNPDF architecture, yielding \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K^{(\ell )}$$\end{document}K(ℓ) for arbitrary ℓℓ, i.e., the covariance at initialization for various depths. These are compared with the empirical covariance computed from an ensemble replicas in Fig. 7 for the first two hidden layers and the output layer. Furthermore, the relative difference between the empirical covariance and the theoretical prediction is shown in Fig. 8. In order to reduce the bootstrap errors in the empirical covariance, an ensemble with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=1000$$\end{document}Nrep=1000 has been used for these figures. The agreement between the theoretical prediction and the empirical computation is excellent, confirming the validity of the large-network expansion even for networks of moderate size, as those used in the NNPDF fits.Fig. 7The empirical (left) and analytical (right) covariance matrices of the first, second, and output layers of the NNPDF architecture (top to bottom). The covariance in the left panel is computed “bootstrapping” over an ensemble of replicas, initialized using the Glorot normal distribution. The covariance in the right panel is obtained by solving Eq. (176) numerically. In order to reduce the bootstrap errors in the empirical covariance, an ensemble of 1000 replicas has been used for this figureFig. 8Relative difference between the empirical kernel, computed from an ensemble of networks at initialization, and the recursive kernel obtained by iterating Eq. (176) for the three layers of the NNPDF architecture. An ensemble of 1000 replicas has been used to reduce the bootstrap errors in the empirical covariance The empirical (left) and analytical (right) covariance matrices of the first, second, and output layers of the NNPDF architecture (top to bottom). The covariance in the left panel is computed “bootstrapping” over an ensemble of replicas, initialized using the Glorot normal distribution. The covariance in the right panel is obtained by solving Eq. (176) numerically. In order to reduce the bootstrap errors in the empirical covariance, an ensemble of 1000 replicas has been used for this figure Relative difference between the empirical kernel, computed from an ensemble of networks at initialization, and the recursive kernel obtained by iterating Eq. (176) for the three layers of the NNPDF architecture. An ensemble of 1000 replicas has been used to reduce the bootstrap errors in the empirical covariance As a consequence of the symmetry of the probability distribution, the mean value of the fields at initialization needs to vanish, while their variance at each point \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_\alpha$$\end{document}xα is given by the diagonal matrix elements of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K^{(\ell )}$$\end{document}K(ℓ). In Fig. 9, the expected distribution is compared against the empirical distribution of output fields for a selected value of x, using two ensembles of replicas with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=100$$\end{document}Nrep=100 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=1000$$\end{document}Nrep=1000, respectively. Inspecting the figures, we conclude that the recursion formula, Eq. (176), accurately reproduces the output distribution of the NNPDF networks at initialization, provided that a sufficiently large ensemble of replicas is used to sample the distribution. Finally, Fig. 10 shows the mean and variance of the output at initialization across all values of x for an ensemble of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=100$$\end{document}Nrep=100 neural networks generated using the NNPDF architecture. We compare two cases: linear input f(x) and scaled input f(x, łog x)f(x,logx). The central value is computed as the average over replicas178\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \bar{f}_{i\alpha } = \bar{f}_{i}(x_\alpha ) = \frac{1}{N_{{\textrm{rep}}}} \sum _{k=1}^{N_{{\textrm{rep}}}} f^{(k)}_i(x_\alpha ), \end{aligned}$$\end{document}f¯iα=f¯i(xα)=1Nrep∑k=1Nrepfi(k)(xα),and, likewise, the variance \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_{i\alpha }$$\end{document}σiα2 is computed over the same ensemble of replicas179\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sigma ^2_{i\alpha } = \frac{1}{N_{{\textrm{rep}}}-1} \sum _{k=1}^{N_{{\textrm{rep}}}} \left( f^{(k)}_i(x_\alpha ) - \bar{f}_{i}(x_\alpha )\right) ^2. \end{aligned}$$\end{document}σiα2=1Nrep-1∑k=1Nrepfi(k)(xα)-f¯i(xα)2.As is clear from the figure, the choice of input scaling has a significant impact on the prior uncertainty, especially in the small-x region. In the following, we neglect this effect and focus on the case of linear input f(x).Fig. 9Sampled distribution of the output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$xT_3$$\end{document}xT3 at x=0.0065x=0.0065 for two different ensemble sizes, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=100$$\end{document}Nrep=100 (top) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=1000$$\end{document}Nrep=1000 (bottom). Each column shows the distribution for a different network architecture; the latter displayed in the top left corner of each panel. The red line represents the predicted Gaussian distribution as dictated by the kernel recursion formula in Eq. (176)Fig. 10The output of the ensemble of neural networks at initialization using the NNPDF architecture in linear (left) and logarithm (right) scale. We compare the case of linear input f(x) (blue) and the case of scaled input f(x, łog x)f(x,logx) (orange). The solid lines represent the mean value computed over an ensemble of 100 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. In the figure, we show \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$xT_3$$\end{document}xT3 as used in the following sections Sampled distribution of the output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$xT_3$$\end{document}xT3 at x=0.0065x=0.0065 for two different ensemble sizes, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=100$$\end{document}Nrep=100 (top) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}=1000$$\end{document}Nrep=1000 (bottom). Each column shows the distribution for a different network architecture; the latter displayed in the top left corner of each panel. The red line represents the predicted Gaussian distribution as dictated by the kernel recursion formula in Eq. (176) The output of the ensemble of neural networks at initialization using the NNPDF architecture in linear (left) and logarithm (right) scale. We compare the case of linear input f(x) (blue) and the case of scaled input f(x, łog x)f(x,logx) (orange). The solid lines represent the mean value computed over an ensemble of 100 replicas, while the shaded bands represent the one-sigma uncertainty computed as the variance over the same ensemble. In the figure, we show \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$xT_3$$\end{document}xT3 as used in the following sections Training neural networks We have summarized some properties of the neural network at initialization, showing that the normal Glorot initialization leads to a prior distribution that is a Gaussian processes with small corrections. In the final part of these lectures, we analyze the training dynamics of the neural network, with the aim of understanding how the process of training yields the posterior distribution of the solution to the inverse problem. We follow very closely the presentation of Ref. [21]. In the context of machine learning, specifically when dealing with neural networks, optimization is an iterative algorithm that updates the parameters of the network in order to minimize a figure of merit defined appropriately. The direction toward the minimum is defined by the gradient of the figure of merit. Due to the large number of parameters that characterize a neural network, and the complex functional form induced by the recursive definition of the network, the figure of merit (also known as error function, loss function, or simply loss) is a non-convex high-dimensional function of the parameters, leading to numerical challenges in the minimization task. In addition, in order to avoid over- and under-learning, training algorithms require a stopping criterion, which specifies the optimal condition to end the training process. While these algorithms have achieved remarkable empirical success, a theoretical understanding of the optimization process remains elusive. In these lectures, we want to focus on understanding the dynamics, without being bogged down by unnecessary technical details; therefore, we work with the simplest gradient method, i.e., Gradient Descent (GD). We work out in detail one specific example, in the context of PDF determinations, where we consider a reduced dataset for which predictions can be computed using just one flavor combination of the PDFs, so that the theoretical prediction for the data remains simply180\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T_I[f] = \int \textrm{d}x\, C_{I}(x) f(x). \end{aligned}$$\end{document}TI[f]=∫dxCI(x)f(x).The details of the dataset and the definition of the flavor combination are not relevant for these lectures, and the interested reader can find more details in Ref. [21]. Training in functional space For analytical tractability, GD is described as a continuous flow of the parameters θθ in training time t along the negative gradient of the loss function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {L}$$\end{document}L. For sufficiently small learning rates ηη, this continuous flow approximates the discrete GD trajectory in parameter space, as extensively discussed in Ref. [22]. The continuous Gradient Flow (GF) is then given by181\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\textrm{d}}{\textrm{d}t}&\theta _{t,\mu } = -\nabla _\mu \mathcal {L}_t, \end{aligned}$$\end{document}ddtθt,μ=-∇μLt,where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _{t,\mu }$$\end{document}θt,μ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {L}_t$$\end{document}Lt identify, respectively, the parameters and the loss function at training time t. We distinguish between the continuous training time t and the discrete epochs of GD, the latter denoted using the capital letter T. For the purpose of this lecture, the two are related through a constant learning rate, t = η Tt=ηT with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta = 10^{-5}$$\end{document}η=10-5, and we will use them interchangeably. We focus here on quadratic loss functions that are obtained as the negative logarithm of Gaussian data distributions around their theoretical predictions; using the notation introduced in Sect. 2, the loss function is given by182\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {L}_t = \frac{1}{2} \left( Y - T[f_t]\right) ^T C_Y^{-1} \left( Y - T[f_t]\right) , \end{aligned}$$\end{document}Lt=12Y-T[ft]TCY-1Y-T[ft],where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_t$$\end{document}ft is the output of the network at training time t, obtained from the time-dependence of the internal parameters, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_Y$$\end{document}CY is the covariance of the data. Note that the loss function at training time t is computed using the theoretical prediction \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T[f_t]$$\end{document}T[ft], computed using a grid of points for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_t$$\end{document}ft at training time t. For a quadratic loss, the gradient is183\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \nabla _\mu \mathcal {L}_t = - \left( \nabla _\mu f_t\right) ^T \left( \frac{\partial T}{\partial f}\right) _t C_Y^{-1} \epsilon _t, \end{aligned}$$\end{document}∇μLt=-∇μftT∂T∂ftCY-1ϵt,where writing explicitly the data index184\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \epsilon _{t,I} = Y_I - T_I[f_t], \quad I=1, \ldots , N_{\textrm{dat}}. \end{aligned}$$\end{document}ϵt,I=YI-TI[ft],I=1,…,Ndat.For the specific case of a quadratic loss function, the gradient is proportional to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\epsilon _t$$\end{document}ϵt, which is the difference between the theoretical prediction and the data at training time t. If at some point during the training the theoretical predictions reproduce all the data, the training process ends. A further simplification is obtained in the case of data that depend linearly on the unknown function f. In this case, the Jacobian of the theoretical prediction with respect to the fields is independent of the training time185\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \frac{\partial T_I}{\partial f_{i\alpha }}\right) _t = (\textrm{FK})_{Ii\alpha }. \end{aligned}$$\end{document}∂TI∂fiαt=(FK)Iiα.A few algebraic steps allow the flow of parameters θθ to be translated into a flow for the fields186\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\textrm{d}}{\textrm{d}t}&f_{t,i_1\alpha _1} = (\nabla _\mu f_{t,i_1\alpha _1}) \frac{\textrm{d}}{\textrm{d}t}\theta _\mu = \Theta _{t,i_1\alpha _1i_2\alpha _2} (\textrm{FK})^T_{i_2\alpha _2I} \left( C_Y^{-1}\right) _{IJ} \epsilon _{t,J}, \end{aligned}$$\end{document}ddtft,i1α1=(∇μft,i1α1)ddtθμ=Θt,i1α1i2α2(FK)i2α2ITCY-1IJϵt,J,where we have defined the Neural Tangent Kernel [23]187\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Theta _{t,i_1\alpha _1i_2\alpha _2} = \sum _\mu \nabla _\mu f_{t,i_1\alpha _1} \nabla _\mu f_{t,i_2\alpha _2}. \end{aligned}$$\end{document}Θt,i1α1i2α2=∑μ∇μft,i1α1∇μft,i2α2.If we introduce the matrix188\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( J_t\right) _{i\alpha ,\mu } = \nabla _\mu f_{t,i\alpha }, \end{aligned}$$\end{document}Jtiα,μ=∇μft,iα,the NTK can be written as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Theta _t = J_t J_t^T$$\end{document}Θt=JtJtT. The NTK is a positive-semidefinite matrix that encodes the dependence of the training dynamics on the architecture of the network. The flow equation, Eq. (186), is a non-linear integro-differential equation that describes the evolution of the fields during training. In order to facilitate the discussion in Sect. 7.1.1, Eq. (186) can be rewritten in a more compact form. We first omit the indices and write for instance189\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( \frac{\partial T}{\partial f}\right) _t = (\textrm{FK}), \quad \Theta _t = \left( \nabla _\mu f_t\right) \left( \nabla _\mu f_t\right) ^T. \end{aligned}$$\end{document}∂T∂ft=(FK),Θt=∇μft∇μftT.Then, using the definition of the error in Eq. (184), we can rewrite Eq. (186) as190\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\textrm{d}}{\textrm{d}t}f_t = -\Theta _t M f_t + b_t, \end{aligned}$$\end{document}ddtft=-ΘtMft+bt,where191\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} M&= (\textrm{FK})^TC_Y^{-1} (\textrm{FK}), \quad b_t = \Theta _t (\textrm{FK})^TC_Y^{-1} Y. \end{aligned}$$\end{document}M=(FK)TCY-1(FK),bt=Θt(FK)TCY-1Y. Exercise 8 Derive the flow equation, Eq. (190), starting from the continuous Gradient Flow equation. Here, M is a positive-semidefinite matrix that depends only on the data covariance and the FK tables that enter the theoretical predictions, while b is a vector that depends (among other quantities) on the central value of the data. Note that any vector f that is in the kernel of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})$$\end{document}(FK) is necessarily in the kernel of M, \ker MkerM. In turn, the vectors in \ker MkerM do not contribute to the flow evolution, as seen explicitly in Eq. (190). There are two important points that we want to emphasize. First, although derived in the context of neural networks, these equations do not refer to a specific parametrization. Indeed, these remain valid even when an explicit functional form is chosen to parametrize the unknown function f. Second, note that the flow equation, Eq. (190), depends on two matrices, ΘΘ and M. The former encodes the model dependence, while the latter contains the physical information. The interplay between these two matrices is crucial for understanding the training dynamics, as discussed in Sect. 6.2.3. Finally, the NTK derived in Eq. (187) is inherently time-dependent in a complex way, which precludes any attempt at integrating Eq. (190) analytically. We come back to this point in Sect. 7.1.1, after discussing the properties of the NTK during training. Inside the training dynamics: an NTK perspective From Eqs. (187) and (190), we observe that the NTK encodes the dependence on the architecture of the network and governs its training dynamics. The analysis of the NTK properties is one way to unravel the behavior of the network during training. NTK at initialization Before training, the NTK is blind to data and depends on the x-grid of input and on the architecture, as shown in Eq. (187). The NTK is a function of the fields f, which are stochastic variables described by their joint probability distribution. Therefore, the NTK is also a stochastic variable, with its own probability distribution, which we represent as usual as a set of replicas. In the large-width limit, the variance of the NTK over the set of replicas is expected to go to zero with the width of the hidden layers. In order to quantify the variation of the NTK, we start by computing the Frobenius norm of the NTK over an ensemble of networks for different architectures. For each architecture, we consider the standard deviation of the norm as a statistical estimator of the variations of the NTK. The result is displayed in Fig. 11. Even though the Frobenius norm is a coarse indicator of the variations of the NTK, the figure shows clearly that the variance of the norm becomes smaller with the size of the network, which is consistent with the theoretical expectation that the NTK should not fluctuate for infinite-width networks.8Fig. 11Frobenius norm of the NTK at initialization, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Vert \Theta _0 \Vert$$\end{document}‖Θ0‖, as a function of the width of the network. On the left, the central values and uncertainty bands are obtained as the mean and one-sigma deviation of the ensemble of networks. The labels on the horizontal axis indicate the respective widths of the two hidden layers in the networks considered. The plot on the right shows the relative uncertainty. It is interesting to note the decrease of the relative uncertainty as the architecture of the network is increased. For larger networks, the sensitivity to a change of the network parameters fluctuates less Frobenius norm of the NTK at initialization, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Vert \Theta _0 \Vert$$\end{document}‖Θ0‖, as a function of the width of the network. On the left, the central values and uncertainty bands are obtained as the mean and one-sigma deviation of the ensemble of networks. The labels on the horizontal axis indicate the respective widths of the two hidden layers in the networks considered. The plot on the right shows the relative uncertainty. It is interesting to note the decrease of the relative uncertainty as the architecture of the network is increased. For larger networks, the sensitivity to a change of the network parameters fluctuates less A more quantitative description of the NTK at initialization is provided by its spectrum, which is shown in Fig. 12 for four different architectures. Inspecting the figure, we see that the spectrum of the NTK is heavily hierarchical, and only a few eigenvalues are actually non-zero. Such a hierarchy in the eigenvalues means that only a small subset of active directions can inform the network during training, as it will be discussed later. Note that, at least at initialization, these observations do not depend on the architecture: the eigenvalues in Fig. 12 are mostly independent of the size of the network. Even though the logarithmic scale on the vertical axis may hide some small variations, it is clear that most eigenvalues remain constant within the error bars. On the other hand, the logarithmic scale emphasizes that there are several orders of magnitude between eigenvalues for a given architecture; that hierarchical structure does not depend on the architecture.Fig. 12Spectrum of the NTK at initialization for the architectures shown in Fig. 11. The labels on the horizontal axis indicate the respective widths of the two hidden layers in the networks considered. Error bands correspond to one-sigma uncertainties over the ensemble of networks. The hierarchy of the eigenvalues is independent of the size of the network. In agreement with the data in Fig. 11, the fluctuations of the eigenvalues decrease as the width of the layers is increased Spectrum of the NTK at initialization for the architectures shown in Fig. 11. The labels on the horizontal axis indicate the respective widths of the two hidden layers in the networks considered. Error bands correspond to one-sigma uncertainties over the ensemble of networks. The hierarchy of the eigenvalues is independent of the size of the network. In agreement with the data in Fig. 11, the fluctuations of the eigenvalues decrease as the width of the layers is increased NTK during training The behavior of the NTK during training is best studied through closure tests, which we introduced in the previous sections. The results of the training are then compared to the known input PDFs; the performances of the training algorithm and the NN architecture are assessed by quantifying the comparison between trained PDFs and input PDFs. Following the original presentation in Ref. [24], we distinguish three levels of closure tests, which are defined by the complexity of the data used to train the NNs. We use the standard NNPDF nomenclature and refer to these three levels as level-0 (L0), level-1 (L1), and level-2 (L2) closure tests, and we denote the input PDFs used to generate the data as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0. We refer the reader to the original publication for the details of the data generation at each level. For each of the closure-test data given above, we perform a fit of the triplet combination \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_3$$\end{document}T3 using the simplified version of the NNPDF methodology that we discussed above. We initialize an ensemble of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N_{{\textrm{rep}}}= 100$$\end{document}Nrep=100 replicas with identical architecture, training each replica independently using GD optimization. Throughout the training process, we track the evolution of the NTK to understand how the network’s effective dynamics changes as it learns the target function. Onset of lazy training As a first estimator of the variation of the NTK, we show in Fig. 13 the Frobenius norm of the variation during training, normalized by the Frobenius norm of the NTK itself192\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \delta \Theta _t = \frac{\Vert \Theta _{t+1} - \Theta _t \Vert }{\Vert \Theta _t \Vert }, \end{aligned}$$\end{document}δΘt=‖Θt+1-Θt‖‖Θt‖,for the three different datasets, L0, L1, and L2. Inspecting the plot reveals that the NTK undergoes significant changes during the initial phase of training, with the relative variation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta \Theta _t$$\end{document}δΘt reaching values as high as 6%6%. This indicates that our settings differ from the standard picture of lazy training in the context of very wide networks, as discussed, e.g., in Refs. [20, 23, 25], where the NTK is expected to be independent of the flow time t. Remarkably, we do not observe a dependence on how data have been generated, indicating that the NTK dynamics is basically unaffected by the noise in the data. After this initial phase—corresponding approximately to the first 20,000 epochs in our experiment—the NTK tends to stabilize. These two regions will be referred to as the rich and lazy training regimes, respectively, in keeping with the standard terminology adopted in the literature (see, e.g., Ref. [26] where two similar regimes were also identified). The lazy regime is discussed in detail in Sect. 7.1.1.Fig. 13Relative variation of the NTK during training for L0, L1, and L2 data. Error bands correspond to one-sigma uncertainties over the ensemble of networks Relative variation of the NTK during training for L0, L1, and L2 data. Error bands correspond to one-sigma uncertainties over the ensemble of networks Eigenvalues during training Further insight on the evolution of the NTK can be obtained by studying its eigensystem as a function of the training time. In Fig. 14, we report the variation of the first five eigenvalues of the NTK, using the standard NNPDF architecture, for L0, L1, and L2 data. We see that the hierarchical structure observed at initialization is preserved, but the size of the subdominant eigenvalues increases significantly in the early stages of training—by one or two orders of magnitude depending on the specific eigenvalue.Fig. 14Evolution during training of the first five eigenvalues of the NTK using L0 (left), L1 (center), and L2 (right) data. Solid lines represent the median over the ensemble of networks, while solid bands correspond to 68% confidence level. Note that the subdominant eigenvalues \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(3)}$$\end{document}λ(3), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(4)}$$\end{document}λ(4), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(5)}$$\end{document}λ(5) have increased by one or two orders of magnitude by the end of the rich training phase Evolution during training of the first five eigenvalues of the NTK using L0 (left), L1 (center), and L2 (right) data. Solid lines represent the median over the ensemble of networks, while solid bands correspond to 68% confidence level. Note that the subdominant eigenvalues \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(3)}$$\end{document}λ(3), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(4)}$$\end{document}λ(4), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(5)}$$\end{document}λ(5) have increased by one or two orders of magnitude by the end of the rich training phase In Fig. 15, the same first five eigenvalues of the NTK are displayed for L0, L1, and L2 data. We observe a common pattern across all data types, consistently with the observation made before in Fig. 13. This indicates the NTK evolution is insensitive to the noise included in the synthetic data. The increase of the subdominant eigenvalues, combined with the analysis of Eqs. (201) and (202) in Sect. 7 suggests that more “physical” features become learnable before lazy training sets in.Fig. 15The first five eigenvalues of the NTK for L0, L1, and L2 data. Solid lines represent the median over the ensemble of networks, while solid bands correspond to 68% confidence level. Each plot corresponds to a different eigenvalue, as indicated by the label on the vertical axis. Note the different scales on the vertical axes, which reflects the hierarchy of eigenvalues discussed above. Different colors correspond to different synthetic data, the agreement between these bands confirms that the evolution of the eigensystem of the NTK does not depend on the level of noise in the data The first five eigenvalues of the NTK for L0, L1, and L2 data. Solid lines represent the median over the ensemble of networks, while solid bands correspond to 68% confidence level. Each plot corresponds to a different eigenvalue, as indicated by the label on the vertical axis. Note the different scales on the vertical axes, which reflects the hierarchy of eigenvalues discussed above. Different colors correspond to different synthetic data, the agreement between these bands confirms that the evolution of the eigensystem of the NTK does not depend on the level of noise in the data Connection with the loss function Finally, in Fig. 16, we show the variation of the loss function during training, overlaid with the first five eigenvalues of the NTK, for a selected replica over the ensemble. It is interesting to see that in correspondence with the sudden variation of the subdominant eigenvalues, the loss function drops significantly, at the cost of an instability localized in the descent. The NN is learning new features, changing its internal representation to accommodate the new information. After this initial phase, the eigenvalues stabilize and the loss function decreases smoothly, as expected in the lazy training regime. The eigenvectors corresponding to the larger eigenvalues can be interpreted as learnable features, while the small (or zero) eigenvalues correspond to directions in which the field f never evolves during training.Fig. 16Variation of the loss function overlaid with the first five eigenvalues for a selected replica over the ensemble using L0 (left), L1 (center), and L2 (right) data. Left scale refers to the loss, while the right scale refers to the eigenvalues. Variation of the loss function overlaid with the first five eigenvalues for a selected replica over the ensemble using L0 (left), L1 (center), and L2 (right) data. Left scale refers to the loss, while the right scale refers to the eigenvalues. Eigenvectors as features It has been argued above that there is a non-trivial interplay between the eigenspace of the NTK and that of the matrix M. Indeed, the former encodes the model dependence, while the latter yields physical information. Of course, the two matrices are independent at initialization, and we do not expect any alignment pattern between the two. However, this picture does change during training, as the NTK evolves and the model learns the target function. To quantify this alignment, we define the matrix A193\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} A_{kk'} = \left( z^{(k)}, v^{(k')} \right) ^2 = \cos ^2(\theta _{kk'}), \end{aligned}$$\end{document}Akk′=z(k),v(k′)2=cos2(θkk′),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z^{(k)}$$\end{document}z(k) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$v^{(k')}$$\end{document}v(k′) are the kth and k'k′th eigenvectors of the NTK and M, respectively. The matrix A is thus a measure of the alignment between the eigenspaces of the two matrices. The rows of the matrix correspond to the eigenvectors of the NTK, ordered by the value of the corresponding eigenvalues, with the eigenvectors corresponding to the larger eigenvalues at the top of the matrix. The columns correspond to eigenvectors of the matrix M, also ordered by the values of the corresponding eigenvalues, with the largest eigenvalues to the left in this case. In Fig. 17, we show the matrix A at different epochs of the training for L2 data and a single NTK replica.Fig. 17Matrix A as defined in Eq. (193) for L2 data and for a single replica of the NTK. The matrix is shown at different epochs of the training process, indicated in the top of each panel Matrix A as defined in Eq. (193) for L2 data and for a single replica of the NTK. The matrix is shown at different epochs of the training process, indicated in the top of each panel The blue rectangle in the top-right corner of the matrix shows that the eigenvectors of the NTK corresponding to the largest eigenvalues are orthogonal to the eigenvectors of M that are in the kernel of M, i.e., to the directions that do not contribute to the observables. It is useful to remember that the largest eigenvalues of the NTK correspond to the directions that are orthogonal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ker \Theta$$\end{document}kerΘ, i.e., the directions that are learnable during the training process. In order to have a robust training process, we expect these learnable directions to align with the directions that actually contribute to the loss function, which are the ones corresponding to the largest eigenvalues of M. Consistently with this intuition, we see that the size of this blue rectangle increases with training time. In particular, it is clear from our plot that it becomes deeper by the onset of the lazy training regime: more of the learnable directions—the features that the network can learn—are aligned with the directions that contribute most to the observables.Fig. 18Reconstruction error \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t(r)$$\end{document}Ct(r) as defined in Eq. (194) as a function of the training time and for different numbers of eigenvectors r. Note that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 lies entirely in the subspace spanned by the first four eigenvectors of the NTK by the onset of the lazy training regime. We see that the NTK has aligned its features with the physically relevant directions of the problem Reconstruction error \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t(r)$$\end{document}Ct(r) as defined in Eq. (194) as a function of the training time and for different numbers of eigenvectors r. Note that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 lies entirely in the subspace spanned by the first four eigenvectors of the NTK by the onset of the lazy training regime. We see that the NTK has aligned its features with the physically relevant directions of the problem The eigenvectors of the NTK form an orthonormal basis in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^{N_{\textrm{grid}}}$$\end{document}RNgrid at any epoch of the training process. However, only a subset of these eigenvectors contributes to the training dynamics. The expressivity of the vectors belonging to the subspace orthogonal to the kernel is quantified by measuring how well the internal representation of the neural network can reconstruct the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 used to generate the data. We define a new figure of merit194\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_t(r) = 1 - \sum _{k=1}^{r} \frac{(z^{(k)}, f_0)^2}{\Vert f_0 \Vert ^2}, \end{aligned}$$\end{document}Ct(r)=1-∑k=1r(z(k),f0)2‖f0‖2,which measures the reconstruction error of the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 when projected onto the subspace spanned by the first r eigenvectors of the NTK at training time t. We show \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t(r)$$\end{document}Ct(r) as a function of the training time and for different choices of r in Fig. 18. Inspecting the figure, we see that in the early stages of training the reconstruction error does not change significantly with time. This behavior is shared across all values of r. In fact, in this first phase, the NTK has not yet found a suitable internal representation and the inclusion of more eigenvectors corresponding to yet undiscovered directions—those associated with a small eigenvalue as in Fig. 14—does not result in an improvement of the reconstruction of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0. Conversely, once the onset of lazy training is approached, we identify two distinct behaviors depending on the number of eigenvectors. If we include the eigenvectors up to the modes discovered during training (e.g., r=6r=6 and r=7r=7 in the figure), the reconstruction error drops significantly and remains almost constant throughout the rest of the learning process. On the other hand, if we consider fewer eigenvectors (e.g., r=4r=4 and r=5r=5 in the figure), the reconstruction error becomes larger and grows indefinitely with training time. The subset of eigenvectors orthogonal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ker \Theta$$\end{document}kerΘ is capable of providing an effective lower dimensional basis, provided that the new modes discovered by the NTK during training are included. Note that the reconstruction of the input function improves as long as the eigenvectors do not belong to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ker \Theta$$\end{document}kerΘ—the eigenvectors that belong to the kernel only add noise to the reconstruction, without bringing any new physical information. A complementary picture is displayed in Fig. 19. Here, we show the eigenvectors of the NTK at different training times as functions of the x-grid, denoted by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z^{(i)}$$\end{document}z(i). Together with the eigenvectors, we also show the output of the trained neural network at the corresponding training time. From these plots, we see that as the training progresses, the shape of the eigenvectors becomes more structured in order to reproduce the output function. Again, this conclusion supports the observations made previously on various occasions, that during training the neural network is changing its internal representation and the NTK encodes this information.Fig. 19First five eigenvectors of the NTK at different training times and as function of the input x-grid. We also show the output of the network at the same training time, which is displayed in gray. L1 data are used First five eigenvectors of the NTK at different training times and as function of the input x-grid. We also show the output of the network at the same training time, which is displayed in gray. L1 data are used Lazy training regime In the previous section, we presented an empirical study of the training dynamics through the lens of the NTK. We observed that the NTK is able to capture the main features of the training process, and that its time evolution is characterized by a rapid initial transient, followed by a slower evolution during the rest of the training. We now turn our attention on this last stage of the training, where the NTK has stabilized and becomes approximately constant. In doing so, we build upon the results presented in Refs. [23, 25] and derive the analytical solution of the flow equation, which allows us to write an explicit expression for the trained field as a function of the field at initialization and the data. Analytical results Solution of the flow equation The lazy training regime is characterized by a slow-evolving NTK. We denote as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref the time at which the onset of this regime occurs. The NTK is then frozen to its value at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_\textrm{ref}$$\end{document}tref, and from this time onward, the NTK is taken to be constant195\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Theta _t = \Theta _{t_{\textrm{ref}}} \equiv \Theta , \quad \text {for } t \ge t_{\textrm{ref}}. \end{aligned}$$\end{document}Θt=Θtref≡Θ,fort≥tref.The flow equation can then be written as196\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\textrm{d}}{\textrm{d}t}f_t = -\Theta M f_t + b, \end{aligned}$$\end{document}ddtft=-ΘMft+b,where M and b are defined as in Eq. (191). Note that now neither ΘΘ nor b depend on the training time t. In order to solve this first-order linear differential equation, we observe that the eigenvectors of ΘΘ197\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Theta z^{(k)} = \lambda ^{(k)} z^{(k)}\, \end{aligned}$$\end{document}Θz(k)=λ(k)z(k)provide a basis for expanding Eq. (196). Furthermore, owing to the spectrum hierarchy of the NTK (Fig. 14), it is necessary to distinguish the components of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_t$$\end{document}ft that are in the kernel of ΘΘ from the ones that are in the orthogonal complement. We introduce the notation198\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&f^\parallel _{t,k} = \left( z^{(k)}, f_t\right) , \quad \text {if}\ \lambda ^{(k)} = 0, \end{aligned}$$\end{document}ft,k‖=z(k),ft,ifλ(k)=0,199\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&f^\perp _{t,k} = \frac{1}{\sqrt{\lambda ^{(k)}}} \left( z^{(k)}, f_t\right) , \quad \text {if}\ \lambda ^{(k)} \ne 0, \end{aligned}$$\end{document}ft,k⊥=1λ(k)z(k),ft,ifλ(k)≠0,where the scalar product has been defined as200\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( f'_{t'}, f_t\right) = \sum _{i,\alpha } f'_{t',i\alpha } f_{t,i\alpha }. \end{aligned}$$\end{document}f′′,ft=∑i,αft′,iα′ft,iα.One can readily see that the components in the kernel of ΘΘ, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ker}\ \Theta$$\end{document}kerΘ, do not evolve during training,9201\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\textrm{d}}{\textrm{d}t}f^\parallel _{t,k} = 0 \quad \Longrightarrow \quad f^\parallel _{t,k} = f^\parallel _{0,k}. \end{aligned}$$\end{document}ddtft,k‖=0⟹ft,k‖=f0,k‖.This means that the final solution is affected by an irreducible noise that is purely dictated by the initial condition. The flow equation for the orthogonal components can be written as202\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{\textrm{d}}{\textrm{d}t}f^\perp _{t,k} = - H^\perp _{kk'} f^\perp _{t,k'} + B^\perp _{k}, \end{aligned}$$\end{document}ddtft,k⊥=-Hkk′⊥ft,k′⊥+Bk⊥,where we introduced203\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H^\perp _{kk'}&= \sqrt{\lambda ^{(k)}} \left( z^{(k)}, M z^{(k')}\right) \sqrt{\lambda ^{(k')}}, \end{aligned}$$\end{document}Hkk′⊥=λ(k)z(k),Mz(k′)λ(k′),204\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} B^\perp _k&= -\sqrt{\lambda ^{(k)}} \left[ \left( z^{(k)}, M z^{(k')}\right) f^\parallel _{0,k'} - \left( z^{(k)}, (\textrm{FK})^TC_Y^{-1} Y\right) \right] . \end{aligned}$$\end{document}Bk⊥=-λ(k)z(k),Mz(k′)f0,k′‖-z(k),(FK)TCY-1Y.As discussed above, the indices on quantities that have a ⊥⊥ suffix only span the space orthogonal to the kernel of ΘΘ, while the indices on quantities that have a ∥‖ suffix span the kernel. We refer to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^\perp$$\end{document}H⊥ as the flow (or training) Hamiltonian, emphasizing that training can only take place in the space orthogonal to the kernel of ΘΘ; we see explicitly in the definition above that the flow dynamics is determined by a combination of the architecture of the NN, encoded in the NTK, and the data, on which M depends. More specifically, the matrix elements of M can be written as205\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( z^{(k)}, M z^{(k')}\right) = T^{(k)T} C_Y^{-1} T^{(k')}, \end{aligned}$$\end{document}z(k),Mz(k′)=T(k)TCY-1T(k′),where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^{(k)} = T[z^{(k)}]$$\end{document}T(k)=T[z(k)] is the vector of theory predictions for the data obtained using \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z^{(k)}$$\end{document}z(k) as the input PDF. Similarly, we have206\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left( z^{(k)}, (\textrm{FK})^TC_Y^{-1} Y\right) = T^{(k)T} C_Y^{-1} Y. \end{aligned}$$\end{document}z(k),(FK)TCY-1Y=T(k)TCY-1Y.Denoting by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d^\perp$$\end{document}d⊥ the dimension of the subspace orthogonal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ker}\ \Theta$$\end{document}kerΘ, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^\perp$$\end{document}H⊥ is a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d^\perp \times d^\perp$$\end{document}d⊥×d⊥ symmetric matrix, whose eigenvalues and eigenvectors satisfy207\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H^\perp _{kk'} w^{(i)}_{k'} = h^{(i)} w^{(i)}_{k}. \end{aligned}$$\end{document}Hkk′⊥w′(i)=h(i)wk(i). In Fig. 20, we show the evolution during training of the first five eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^{\perp }$$\end{document}H⊥ for the three different closure datasets. In Fig. 21, we show the first five eigenvectors of H, denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q^{(i)}$$\end{document}q(i), at different training times as functions of the x-grid. It should not come as too much of a surprise that the eigenvalues and eigenvectors of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^{\perp }$$\end{document}H⊥ have a similar behavior to those of the NTK (see Figs. 14, 19), from which they are constructed. However, we see that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$h^{(i)}$$\end{document}h(i) are larger by around three orders of magnitude than the NTK eigenvalues.Fig. 20Evolution during training of the first five eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^{\perp }$$\end{document}H⊥ using L0 (left), L1 (center), and L2 (right) data. Solid lines represent the median over the ensemble of networks, while solid bands correspond to 68% confidence level. Note that the subdominant eigenvalues \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(3)}$$\end{document}λ(3), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(4)}$$\end{document}λ(4) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(5)}$$\end{document}λ(5) have increased by one or two orders of magnitude by the end of the rich training phaseFig. 21First five eigenvectors of the combined matrix H=Θ MH=ΘM, as in Eq. (190), at different training times and as a function of the input x-grid. We also show the output of the network at the same training time, which is displayed in gray. L1 data are used Evolution during training of the first five eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^{\perp }$$\end{document}H⊥ using L0 (left), L1 (center), and L2 (right) data. Solid lines represent the median over the ensemble of networks, while solid bands correspond to 68% confidence level. Note that the subdominant eigenvalues \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(3)}$$\end{document}λ(3), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(4)}$$\end{document}λ(4) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda ^{(5)}$$\end{document}λ(5) have increased by one or two orders of magnitude by the end of the rich training phase First five eigenvectors of the combined matrix H=Θ MH=ΘM, as in Eq. (190), at different training times and as a function of the input x-grid. We also show the output of the network at the same training time, which is displayed in gray. L1 data are used The solution to Eq. (202) can be written as the sum of the solution of the homogeneous equation, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}^{\perp }_{t,k}$$\end{document}f^t,k⊥, and a particular solution of the full equation. The solution of the homogeneous equation is208\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{f}^{\perp }_{t,k} = \sum _{i=1}^{d^\perp } f^{\perp (i)}_{0} e^{-h^{(i)}t} w^{(i)}_k, \end{aligned}$$\end{document}f^t,k⊥=∑i=1d⊥f0⊥(i)e-h(i)twk(i),where209\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^{\perp (i)}_{0} = \sum _{k=1}^{d_\perp } w^{(i)}_k f^\perp _{0,k}, \end{aligned}$$\end{document}f0⊥(i)=∑k=1d⊥wk(i)f0,k⊥,guarantees that the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{f}^\perp _{t,k}=f^\perp _{0,k}$$\end{document}f^t,k⊥=f0,k⊥ is satisfied. Similarly, if we define210\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Upsilon ^{(i)} = \sum _{k=1}^{d_\perp } w^{(i)}_k B^\perp _{k}, \end{aligned}$$\end{document}Υ(i)=∑k=1d⊥wk(i)Bk⊥,then211\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \check{f}^\perp _{t,k} = \mathop {{\mathop {\sum }\nolimits '}}\limits _{i} \frac{1}{h^{(i)}} \Upsilon ^{(i)} \left( 1 - e^{-h^{(i)}t}\right) w^{(i)}_k, \end{aligned}$$\end{document}fˇt,k⊥=∑′i1h(i)Υ(i)1-e-h(i)twk(i),where the sum only involves the non-zero modes of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^\perp$$\end{document}H⊥, and is a particular solution of the inhomogeneous equation, which satisfies the boundary condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\check{f}^{\perp }_{0,k}=0$$\end{document}fˇ0,k⊥=0. Hence, the solution of the flow equation in the subspace orthogonal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ker}\ \Theta$$\end{document}kerΘ is212\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^\perp _{t,k}&= \hat{f}^\perp _{t,k} + \check{f}^\perp _{t,k} . \end{aligned}$$\end{document}ft,k⊥=f^t,k⊥+fˇt,k⊥.Finally, collecting the parallel contribution, Eq. (201), and the solution of the orthogonal component, Eq. (212) yields a simple expression213\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \boxed {f_{t,\alpha } = U(t)_{\alpha \alpha '} f_{0,\alpha '} + V(t)_{\alpha I} Y_{I}. } \end{aligned}$$\end{document}ft,α=U(t)αα′f0,α′+V(t)αIYI.The two evolution operators U(t) and V(t) have lengthy, yet explicit, expressions, which we summarize here214\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} U(t)_{\alpha \alpha '} = \hat{U}^\perp (t)_{\alpha \alpha '} + \check{U}^\perp (t)_{\alpha \alpha '} + U^\parallel _{\alpha \alpha '}, \end{aligned}$$\end{document}U(t)αα′=U^⊥(t)αα′+Uˇ⊥(t)αα′+Uαα′‖,where215\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{U}^\perp (t)_{\alpha \alpha '} = \sum _{k,k'\in \perp } \sqrt{\lambda ^{(k)}} z^{(k)}_\alpha \left[ \sum _i w^{(i)}_{k} e^{-h^{(i)}t} w^{(i)}_{k'}\right] z^{(k')}_{\alpha '} \frac{1}{\sqrt{\lambda ^{(k')}}}, \end{aligned}$$\end{document}U^⊥(t)αα′=∑k,k′∈⊥λ(k)zα(k)∑iwk(i)e-h(i)tw′(i)z′(k′)1λ(k′),and216\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} U^\parallel _{\alpha \alpha '} = \sum _{k''\in \parallel } z^{(k)}_\alpha z^{(k)}_{\alpha '}. \end{aligned}$$\end{document}Uαα′‖=∑k′′∈‖zα(k)z′(k).The contributions from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\check{U}^\perp (t)$$\end{document}Uˇ⊥(t) and V(t) are more easily expressed by introducing the operator217\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {M}(t)_{\alpha \alpha '} = \sum _{k,k'\in \perp } \sqrt{\lambda ^{(k)}} z^{(k)}_\alpha \left[ \mathop {{\mathop {\sum }\nolimits '}}\limits _{i} w^{(i)}_{k} \frac{1}{h^{(i)}}\, \left( 1- e^{-h^{(i)}t}\right) w^{(i)}_{k'}\right] z^{(k')}_{\alpha '} \sqrt{\lambda ^{(k')}}. \end{aligned}$$\end{document}M(t)αα′=∑k,k′∈⊥λ(k)zα(k)∑′iwk(i)1h(i)1-e-h(i)tw′(i)z′(k′)λ(k′).Then, we can write218\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \check{U}^\perp (t) = - \mathcal {M}(t)\; (\textrm{FK})^TC_Y^{-1} (\textrm{FK})\left[ \sum _{k''\in \parallel } z^{(k'')} z^{(k'') T}\right] , \end{aligned}$$\end{document}Uˇ⊥(t)=-M(t)(FK)TCY-1(FK)∑k′′∈‖z(k′′)z(k′′)T,and219\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} V(t) = \mathcal {M}(t)\; (\textrm{FK})^TC_Y^{-1}, \end{aligned}$$\end{document}V(t)=M(t)(FK)TCY-1,where we note that the term in the bracket in Eq. (218) is simply the projector on the kernel of the NTK. The four terms that appear in the analytical solution have a clear physical interpretation:The first term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{U}^\perp (t)$$\end{document}U^⊥(t) suppresses the components of the initial condition that lie in the subspace orthogonal to the kernel of the NTK. These are the components that are learned by the network during training. While the trained solution still depends on its value at initialization, that dependence is suppressed during training. This suppression is exponential in the training time, and the rates are given by the eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^{\perp }$$\end{document}H⊥.The contribution from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U^\parallel$$\end{document}U‖ yields the component of the initial condition that lies in the kernel of the NTK. As such, those components remain unchanged during training and are part of the trained field at all times t.The two remaining contributions are best understood by combining them together 220\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \check{U}^{\perp }(t) f_{0} + V(t) Y = \mathcal {M}(t)\; (\textrm{FK})^TC_Y^{-1} \left[ Y - (\textrm{FK})f_{0}^{\parallel }\right] . \end{aligned}$$\end{document}Uˇ⊥(t)f0+V(t)Y=M(t)(FK)TCY-1Y-(FK)f0‖. The parallel component of the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{0}^{\parallel }$$\end{document}f0‖ does not evolve during training, and therefore, it yields a contribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})f_{0}^{\parallel }$$\end{document}(FK)f0‖ to the theoretical prediction of the data points at all times t. This is taken into account by subtracting this contribution from the data, before the inference is performed. The first term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{U}^\perp (t)$$\end{document}U^⊥(t) suppresses the components of the initial condition that lie in the subspace orthogonal to the kernel of the NTK. These are the components that are learned by the network during training. While the trained solution still depends on its value at initialization, that dependence is suppressed during training. This suppression is exponential in the training time, and the rates are given by the eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^{\perp }$$\end{document}H⊥. The first term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{U}^\perp (t)$$\end{document}U^⊥(t) suppresses the components of the initial condition that lie in the subspace orthogonal to the kernel of the NTK. These are the components that are learned by the network during training. While the trained solution still depends on its value at initialization, that dependence is suppressed during training. This suppression is exponential in the training time, and the rates are given by the eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^{\perp }$$\end{document}H⊥. The contribution from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U^\parallel$$\end{document}U‖ yields the component of the initial condition that lies in the kernel of the NTK. As such, those components remain unchanged during training and are part of the trained field at all times t. The contribution from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U^\parallel$$\end{document}U‖ yields the component of the initial condition that lies in the kernel of the NTK. As such, those components remain unchanged during training and are part of the trained field at all times t. The two remaining contributions are best understood by combining them together 220\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \check{U}^{\perp }(t) f_{0} + V(t) Y = \mathcal {M}(t)\; (\textrm{FK})^TC_Y^{-1} \left[ Y - (\textrm{FK})f_{0}^{\parallel }\right] . \end{aligned}$$\end{document}Uˇ⊥(t)f0+V(t)Y=M(t)(FK)TCY-1Y-(FK)f0‖. The parallel component of the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{0}^{\parallel }$$\end{document}f0‖ does not evolve during training, and therefore, it yields a contribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})f_{0}^{\parallel }$$\end{document}(FK)f0‖ to the theoretical prediction of the data points at all times t. This is taken into account by subtracting this contribution from the data, before the inference is performed. The two remaining contributions are best understood by combining them together 220\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \check{U}^{\perp }(t) f_{0} + V(t) Y = \mathcal {M}(t)\; (\textrm{FK})^TC_Y^{-1} \left[ Y - (\textrm{FK})f_{0}^{\parallel }\right] . \end{aligned}$$\end{document}Uˇ⊥(t)f0+V(t)Y=M(t)(FK)TCY-1Y-(FK)f0‖. The parallel component of the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{0}^{\parallel }$$\end{document}f0‖ does not evolve during training, and therefore, it yields a contribution \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\textrm{FK})f_{0}^{\parallel }$$\end{document}(FK)f0‖ to the theoretical prediction of the data points at all times t. This is taken into account by subtracting this contribution from the data, before the inference is performed. Exercise 9 Derive the analytic solution in Eq. (213) by solving the flow equation in the subspace orthogonal to the kernel of the NTK, and then adding the contribution from the kernel, as explained in the text. In doing so, you will need to compute the eigenvalues and eigenvectors of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^\perp$$\end{document}H⊥, and use them to write the solution of the homogeneous and inhomogeneous equations. Finally, you will need to express the solution in terms of the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 and the data Y by using the definitions of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$B^\perp$$\end{document}B⊥ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^\perp$$\end{document}H⊥. The solution in Eq. (213) is the main result of this section. It shows that the training process can be described as the sum of a linear transformation of the initial fields \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{0,\alpha }$$\end{document}f0,α, and a linear transformation of the data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_I$$\end{document}YI. The two transformations depend on the flow time t and are given by the evolution operators U(t) and V(t). Figure 22 compares the analytical solution with the trained function at the end of training, for different choices of the frozen NTK. The NN is trained using the numerical GD until \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref, at which point the NTK is frozen. The evolution time t used in the analytical solution is the difference between the total training time and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref; the initial condition for the analytical solution is the trained solution at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref. Central value and uncertainty bands are obtained by computing the analytical solution for each replica of the initial condition and frozen NTK.10 The figures agree with the expectation; the closer \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref is to the onset of the lazy regime, the better the agreement between the analytical solution and the trained function.Fig. 22Comparison of the trained and analytical evolution at the end of training. Each panel corresponds to a different frozen NTK, whereby the analytical solution is computed starting from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{T_{\textrm{Ref}}}$$\end{document}fTRef. The orange curve represents the final trained function after 50,000 iterations of GD, and is the same across panels. Error bands represent one-sigma uncertainties across replicas. L2 data are used Comparison of the trained and analytical evolution at the end of training. Each panel corresponds to a different frozen NTK, whereby the analytical solution is computed starting from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{T_{\textrm{Ref}}}$$\end{document}fTRef. The orange curve represents the final trained function after 50,000 iterations of GD, and is the same across panels. Error bands represent one-sigma uncertainties across replicas. L2 data are used A complementary perspective is provided in Fig. 23, where the analytical solution is decomposed into the two contributions from U and V. In each panel, the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{t_{\textrm{ref}}}$$\end{document}ftref is evolved analytically for different training times by keeping the frozen NTK fixed. We see that as training proceeds, the contribution from U is progressively suppressed, in accordance with the observation made above. On the other hand, the contribution from V grows and becomes dominant at later epochs, indicating that the trained function is mostly determined by the data, rather than the initial condition of the network. We also observe that such behavior happens quite rapidly—in a training time interval \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta T \approx 200$$\end{document}ΔT≈200—as a consequence of the fact that the time scales in the analytical solution are determined by the inverse of the eigenvalues of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^\perp$$\end{document}H⊥, and the latter are typically large.Fig. 23Decomposition of the analytical solution into the two contributions from U and V at different training times. The frozen NTK is fixed across panels, and corresponds to the one at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}} = 10{,}000$$\end{document}Tref=10,000. The initial condition for the analytical solution is always \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{T_{\textrm{ref}}}$$\end{document}fTref. As in Fig. 22, L2 data are used Decomposition of the analytical solution into the two contributions from U and V at different training times. The frozen NTK is fixed across panels, and corresponds to the one at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}} = 10{,}000$$\end{document}Tref=10,000. The initial condition for the analytical solution is always \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{T_{\textrm{ref}}}$$\end{document}fTref. As in Fig. 22, L2 data are used The analytical solution in Eq. (213) sheds a new light onto the behavior of the numerical training of a neural network. Given these results, it is natural to ask whether the information encoded in the NTK alone can drive training, independently of the initial condition, i.e., whether the analytical solution can be used to perform kernel learning. We address this question in Sect. 7.2, after deriving a few additional analytical results in the subsections below. Crosschecks using L0 data The analytical solution enables rigorous validation of our implementation through crosschecks with L0 data, where we have complete control over the data generation process. In this case, the realization of the dataset is completely determined by the input PDFs221\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y = (\textrm{FK}){f_0}. \end{aligned}$$\end{document}Y=(FK)f0.Note that using L0 data only affect the second term in Eq. (213).11 We can then rewrite the combined term in Eq. (220) as follows:222\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \check{U}^\perp (t) f_0 + V(t) Y&= \mathcal {M}(t)\, (\textrm{FK})^TC_{Y}^{-1} (\textrm{FK})\, \left[ {f_0}- f_{0}^\parallel \right] . \end{aligned}$$\end{document}Uˇ⊥(t)f0+V(t)Y=M(t)(FK)TCY-1(FK)f0-f0‖.The subtraction taking place in the square brackets of Eq. (222) suggests that the effective function that the neural network actually sees is not the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0}$$\end{document}f0 used to generate the data, but rather the difference between \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0}$$\end{document}f0 and the component of the initial function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 that lies in the subspace spanned by the kernel of the NTK, i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0^\parallel$$\end{document}f0‖. In other words, the parallel component \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0^\parallel$$\end{document}f0‖, which, we remind the reader, does not evolve during the analytic training, acts as a constant “bias” in the training process, effectively shifting the input function seen by the neural network. Of course, the actual magnitude of this irreducible noise depends both on how \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 and the kernel of the NTK are distributed over the ensemble. Note that the observation above remains true even in the limit of infinite training223\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \lim _{t\rightarrow \infty } V(t) Y = {f_0^{\perp }}+ \mathcal {M}_{\infty } M {f_0^{\parallel }}, \end{aligned}$$\end{document}limt→∞V(t)Y=f0⊥+M∞Mf0‖,which shows that the V component of the trained solution reproduces exactly the component of the PDF that lies in the subspace orthogonal to the kernel of ΘΘ. We compare the asymptotic behavior of V(t)Y and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0^{\perp }}$$\end{document}f0⊥ in Fig. 24, where we see that the analytical solution at infinite training time reproduces the expected result, i.e., it coincides with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0^{\perp }}$$\end{document}f0⊥, as long as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}}> 1000$$\end{document}Tref>1000. The second term in the square bracket on the right-hand side of Eq. (222) is the contribution from the parallel component at initialization that does not evolve in the training process. Given that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 is almost normally distributed around zero, that term does not contribute to the central value of the fitted PDF, i.e., to the average of the trained solution over replicas.Fig. 24Test of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\rightarrow \infty$$\end{document}t→∞ limit of the L0 training for different frozen NTKs. The green curve represents the projection of the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0}$$\end{document}f0 onto the subspace orthogonal to the kernel of the NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref, i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0^{\perp }}$$\end{document}f0⊥. The purple curve represents the contribution of the operator V, computed with the NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}}$$\end{document}Tref, in the limit of infinite training time Test of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t\rightarrow \infty$$\end{document}t→∞ limit of the L0 training for different frozen NTKs. The green curve represents the projection of the input function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0}$$\end{document}f0 onto the subspace orthogonal to the kernel of the NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref, i.e., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${f_0^{\perp }}$$\end{document}f0⊥. The purple curve represents the contribution of the operator V, computed with the NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}}$$\end{document}Tref, in the limit of infinite training time The time evolution of224\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathbb {E}\left[ \mathcal {M}(t)\, (\textrm{FK})^TC_{Y}^{-1} (\textrm{FK})\, f_{0}^\parallel \right] \, \end{aligned}$$\end{document}EM(t)(FK)TCY-1(FK)f0‖is shown in Fig. 25.Fig. 25Test of the average of the parallel contribution for different epochs. The reference epoch at which the frozen NTK is chosen is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}} = 10{,}000$$\end{document}Tref=10,000. L2 data are used in the plot. Note that the scale on the vertical axis is three orders of magnitude smaller than in Fig. 24 Test of the average of the parallel contribution for different epochs. The reference epoch at which the frozen NTK is chosen is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}} = 10{,}000$$\end{document}Tref=10,000. L2 data are used in the plot. Note that the scale on the vertical axis is three orders of magnitude smaller than in Fig. 24 Infinite training time In the limit of infinite training time, the evolution operators U(t) and V(t) simplify and yield an elegant interpretation of the minimum of the cost function. For large training times, we have225\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{U}^\perp _{\infty , \alpha \alpha '}&= \lim _{t\rightarrow \infty }\hat{U}^\perp (t)_{\alpha \alpha '} = 0\, \end{aligned}$$\end{document}U^∞,αα′⊥=limt→∞U^⊥(t)αα′=0226\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mathcal {M}_{\infty , \alpha \alpha '}&= \lim _{t\rightarrow \infty }\mathcal {M}(t)_{\alpha \alpha '} = \sum _{k,k'\in \perp } \sqrt{\lambda ^{(k)}} z^{(k)}_\alpha \left[ \mathop {{\mathop {\sum }\nolimits '}}\limits _{i} w^{(i)}_{k} \frac{1}{h^{(i)}}\, w^{(i)}_{k'}\right] z^{(k')}_{\alpha '} \sqrt{\lambda ^{(k')}}, \end{aligned}$$\end{document}M∞,αα′=limt→∞M(t)αα′=∑k,k′∈⊥λ(k)zα(k)∑′iwk(i)1h(i)w′(i)z′(k′)λ(k′),and explicit expressions for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\check{U}^\perp _{\infty }$$\end{document}Uˇ∞⊥ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V_{\infty }$$\end{document}V∞ are obtained from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {M}_{\infty }$$\end{document}M∞. The term in the square bracket in Eq. (226) is the spectral decomposition of the pseudo-inverse of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^\perp$$\end{document}H⊥ in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_\perp$$\end{document}d⊥-dimensional orthogonal subspace. Therefore, the operator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {M}_{\infty }$$\end{document}M∞ acts as follows on a field \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{\alpha }$$\end{document}fα: The term on the right of the square bracket computes the coordinate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{k'}$$\end{document}fk′ introduced in Eq. (199). The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_k$$\end{document}fk are the coordinates of the component \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f^\perp$$\end{document}f⊥ that evolves during training 227\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^\perp = \sum _{k\in \perp } \sqrt{\lambda ^{(k)}} f_k\, z^{(k)}. \end{aligned}$$\end{document}f⊥=∑k∈⊥λ(k)fkz(k).The term in the square bracket applies the pseudo-inverse to the coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_k$$\end{document}fk228\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f'_k = \left( H^\perp \right) ^+_{kk'} f_{k'}. \end{aligned}$$\end{document}fk′=H⊥kk′+fk′.The final term on the left of the square bracket reconstructs the full field corresponding to the coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f'_{k}$$\end{document}fk′229\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^{'\perp } = \sum _{k\in \perp } \sqrt{\lambda ^{(k)}} f'_{k}\, z^{(k)}. \end{aligned}$$\end{document}f′⊥=∑k∈⊥λ(k)fk′z(k).As discussed at the end of Sect. 7.1.1 it is convenient to combine the contributions of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\check{U}^\perp _{\infty }$$\end{document}Uˇ∞⊥ and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V_{\infty }$$\end{document}V∞230\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \check{U}^{\perp }_{\infty } f_{0} + V_{\infty } Y = \mathcal {M}_{\infty }\; (\textrm{FK})^TC_Y^{-1} \left[ Y - (\textrm{FK})f_{0}^{\parallel }\right] . \end{aligned}$$\end{document}Uˇ∞⊥f0+V∞Y=M∞(FK)TCY-1Y-(FK)f0‖.The contribution to the observables from the parallel components of f does not change during training; therefore, that contribution is subtracted from the data and the orthogonal components of f are adjusted to minimize the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document}χ2 of the corrected data. The minimum of the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document}χ2 in the orthogonal subspace is found applying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {M}_{\infty }$$\end{document}M∞, i.e., by projecting in the orthogonal subspace, applying the pseudo-inverse and finally recomputing the full field as detailed above. The term on the right of the square bracket computes the coordinate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{k'}$$\end{document}fk′ introduced in Eq. (199). The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_k$$\end{document}fk are the coordinates of the component \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f^\perp$$\end{document}f⊥ that evolves during training 227\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^\perp = \sum _{k\in \perp } \sqrt{\lambda ^{(k)}} f_k\, z^{(k)}. \end{aligned}$$\end{document}f⊥=∑k∈⊥λ(k)fkz(k). The term on the right of the square bracket computes the coordinate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_{k'}$$\end{document}fk′ introduced in Eq. (199). The \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_k$$\end{document}fk are the coordinates of the component \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f^\perp$$\end{document}f⊥ that evolves during training 227\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^\perp = \sum _{k\in \perp } \sqrt{\lambda ^{(k)}} f_k\, z^{(k)}. \end{aligned}$$\end{document}f⊥=∑k∈⊥λ(k)fkz(k). The term in the square bracket applies the pseudo-inverse to the coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_k$$\end{document}fk228\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f'_k = \left( H^\perp \right) ^+_{kk'} f_{k'}. \end{aligned}$$\end{document}fk′=H⊥kk′+fk′. The term in the square bracket applies the pseudo-inverse to the coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_k$$\end{document}fk228\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f'_k = \left( H^\perp \right) ^+_{kk'} f_{k'}. \end{aligned}$$\end{document}fk′=H⊥kk′+fk′. The final term on the left of the square bracket reconstructs the full field corresponding to the coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f'_{k}$$\end{document}fk′229\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^{'\perp } = \sum _{k\in \perp } \sqrt{\lambda ^{(k)}} f'_{k}\, z^{(k)}. \end{aligned}$$\end{document}f′⊥=∑k∈⊥λ(k)fk′z(k). The final term on the left of the square bracket reconstructs the full field corresponding to the coordinates \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f'_{k}$$\end{document}fk′229\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f^{'\perp } = \sum _{k\in \perp } \sqrt{\lambda ^{(k)}} f'_{k}\, z^{(k)}. \end{aligned}$$\end{document}f′⊥=∑k∈⊥λ(k)fk′z(k). Numerical results The results shown in Sects. 7.1.1 and 6.2.2 support the idea that the NTK is capable to encode in its eigenvectors the physical features learned during training. Let us now study the analytical solution that we obtain by choosing the initial condition to be an ensemble of networks at initialization as in Fig. 10. The analytical solution is computed using the NTK frozen at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref. Central value and covariance of the trained fields The analytical solution in Eq. (213) is inherently stochastic, since the frozen NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref is actually obtained from an ensemble of networks. The central value of the analytical solution is defined as231\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \bar{f}_{t,\alpha } = \mathbb {E}\left[ f_{t,\alpha }\right] = \mathbb {E}\left[ U(t)_{\alpha \alpha '} f_{0,\alpha '}\right] + \mathbb {E}\left[ V(t)_{\alpha I} Y_I\right] . \end{aligned}$$\end{document}f¯t,α=Eft,α=EU(t)αα′f0,α′+EV(t)αIYI.More interestingly, the covariance matrix at any time t is given by232\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{Cov}[f_t,f_t^T]&= \mathbb {E}\left[ U(t) f_0 f_0^T U(t)^T\right] - \mathbb {E}\left[ U(t) f_0\right] \mathbb {E}\left[ f_0^T U(t)^T\right] \nonumber \\&\quad + \mathbb {E}\left[ U(t) f_0 Y^T V(t)^T\right] - \mathbb {E}\left[ U(t) f_0\right] \mathbb {E}\left[ Y^T V(t)^T\right] \nonumber \\&\quad + \mathbb {E}\left[ V(t) Y f_0^T U(t)^T\right] - \mathbb {E}\left[ V(t) Y\right] \mathbb {E}\left[ f_0^T U(t)^T\right] \nonumber \\&\quad + \mathbb {E}\left[ V(t) Y Y^T V(t)^T\right] - \mathbb {E}\left[ V(t) Y\right] \mathbb {E}\left[ Y^T V(t)^T\right] . \end{aligned}$$\end{document}Cov[ft,ftT]=EU(t)f0f0TU(t)T-EU(t)f0Ef0TU(t)T+EU(t)f0YTV(t)T-EU(t)f0EYTV(t)T+EV(t)Yf0TU(t)T-EV(t)YEf0TU(t)T+EV(t)YYTV(t)T-EV(t)YEYTV(t)T.Note that the first and the fourth lines above yield symmetric matrices, while the third line is just the transpose of the second, thereby ensuring that the whole covariance matrix is the sum of three symmetric matrices and therefore is symmetric233\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textrm{Cov}[f_t,f_t^T] = C_t^{(00)} + C_t^{(0Y)} + C_t^{(YY)}, \end{aligned}$$\end{document}Cov[ft,ftT]=Ct(00)+Ct(0Y)+Ct(YY),where234\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_t^{(00)}&= \mathbb {E}\left[ U(t) f_0 f_0^T U(t)^T\right] - \mathbb {E}\left[ U(t) f_0\right] \mathbb {E}\left[ f_0^T U(t)^T\right] , \end{aligned}$$\end{document}Ct(00)=EU(t)f0f0TU(t)T-EU(t)f0Ef0TU(t)T,235\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_t^{(0Y)}&= \mathbb {E}\left[ U(t) f_0 Y^T V(t)^T\right] - \mathbb {E}\left[ U(t) f_0\right] \mathbb {E}\left[ Y^T V(t)^T\right] \nonumber \\&\quad + \mathbb {E}\left[ V(t) Y f_0^T U(t)^T\right] - \mathbb {E}\left[ V(t) Y\right] \mathbb {E}\left[ f_0^T U(t)^T\right] , \end{aligned}$$\end{document}Ct(0Y)=EU(t)f0YTV(t)T-EU(t)f0EYTV(t)T+EV(t)Yf0TU(t)T-EV(t)YEf0TU(t)T,236\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_t^{(YY)}&= \mathbb {E}\left[ V(t) Y Y^T V(t)^T\right] - \mathbb {E}\left[ V(t) Y\right] \mathbb {E}\left[ Y^T V(t)^T\right] . \end{aligned}$$\end{document}Ct(YY)=EV(t)YYTV(t)T-EV(t)YEYTV(t)T.Equation (233) shows explicitly the various contributions to the covariance matrix. Indeed, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(00)}$$\end{document}Ct(00) quantifies the contribution to the covariance matrix that is purely due to the fluctuations of the initial condition, while \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(YY)}$$\end{document}Ct(YY) quantifies the contribution that is purely due to the statistical fluctuations of the data. The mixed term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(0Y)}$$\end{document}Ct(0Y) accounts for the correlations between the two sources of uncertainty. Convergence of the analytical solution In order to study the convergence of the analytical solution, we compare the following:the analytical solution (AS), obtained using an ensemble of networks at initialization as the initial condition;the trained solution (TS), obtained by training another ensemble of networks, drawn from the same prior distribution, using GD.This comparison is shown in Fig. 26 for L2 data, where the rows in the grid correspond to different frozen NTKs, while the columns represent numerical and analytical evolution after T=50, 500T=50,500 and 5000 epochs. These results deserve a few comments.Fig. 26Comparison of the trained (orange) and analytical (blue) evolution starting from an ensemble of networks at initialization as the initial condition. Each row corresponds to a different frozen NTK, while the columns represent different training times. The dashed line represents the input function used to generate the synthetic data, i.e., the true result. L2 data are used the analytical solution (AS), obtained using an ensemble of networks at initialization as the initial condition; the analytical solution (AS), obtained using an ensemble of networks at initialization as the initial condition; the trained solution (TS), obtained by training another ensemble of networks, drawn from the same prior distribution, using GD. the trained solution (TS), obtained by training another ensemble of networks, drawn from the same prior distribution, using GD. Comparison of the trained (orange) and analytical (blue) evolution starting from an ensemble of networks at initialization as the initial condition. Each row corresponds to a different frozen NTK, while the columns represent different training times. The dashed line represents the input function used to generate the synthetic data, i.e., the true result. L2 data are used The first observation is that the NTK at early stages of training is not able to drive the prior toward the true function, as shown in the first and, though less dramatically, in the second row of Fig. 26. This is expected, as we extensively discussed in Sect. 6.2.2, since, at early stages of training by GD, the NTK has not yet aligned its internal representation with the data. More significantly, we observe a significant discrepancy between the AS and the TS even at T=5000T=5000. At the beginning of training, there is clearly no difference between AS and TS since both represent the neural network output at initialization, with variations due only to different initialization seeds. During early training stages, AS and TS differ as expected. Indeed, the analytical solution is computed using the frozen NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref, while the trained solution evolves with an NTK that is still changing, as shown in Sect. 6. Crucially, if the NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{\textrm{ref}}$$\end{document}tref has already learned from the data and aligned with the solution, the AS converges faster to the target, while the TS requires additional epochs before evolving in the correct direction. Connection with linear methods We can consider a simplifying limit of Eq. (231), where the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 and the data Y are statistically independent from the respective evolution operators U(t) and V(t). Note that the first term on the right-hand side of Eq. (231) can only be non-zero because of the correlations between U(t) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0. In the absence of such correlations, the first term would be given by the product of the expectation values and, therefore, would vanish if \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 is an ensemble of networks at initialization. Under these assumptions, we have237\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \bar{U}(t)&= \mathbb {E}\left[ U(t)\right] , \end{aligned}$$\end{document}U¯(t)=EU(t),238\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \bar{V}(t)&= \mathbb {E}\left[ V(t)\right] , \end{aligned}$$\end{document}V¯(t)=EV(t),and239\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \bar{f}_{t,\alpha } = \bar{U}(t)_{\alpha \alpha '} \bar{f}_{0,\alpha '} + \bar{V}(t)_{\alpha I} Y_I = \bar{V}(t)_{\alpha I} Y_I. \end{aligned}$$\end{document}f¯t,α=U¯(t)αα′f¯0,α′+V¯(t)αIYI=V¯(t)αIYI.The second term in Eq. (231), or equivalently Eq. (239), explicitly shows the contribution of each data point to the central value of the trained fields at each value of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{\alpha }$$\end{document}xα. It is worthwhile remarking that in this limit, the central value from the set of trained networks is a linear combination of the data points, with coefficients given by the evolution operator \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(t)_{\alpha I}$$\end{document}V(t)αI. In the absence of general theorems, we verify this assumption empirically. From the ensemble of replicas, we generate bootstrap samples and compute the following two estimators:240\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Delta [U(t)f_0]&= \mathbb {E}\left[ U(t) f_{0}\right] - \mathbb {E}\left[ U(t) \right] \mathbb {E}\left[ f_{0}\right] , \end{aligned}$$\end{document}Δ[U(t)f0]=EU(t)f0-EU(t)Ef0,241\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Delta [V(t)Y]&= \mathbb {E}\left[ V(t) Y\right] - \mathbb {E}\left[ V(t) \right] \mathbb {E}\left[ Y\right] . \end{aligned}$$\end{document}Δ[V(t)Y]=EV(t)Y-EV(t)EY.for different training times, using the same frozen NTK and L2 data. The results are shown in Fig. 27 for the U (upper panel) and V (lower panel) contributions. The error bands are computed using bootstrap error. By inspecting the figures, we see two distinct patterns emerging. For the operator U, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta [U(t) f_0]$$\end{document}Δ[U(t)f0] is different from zero for small training times, and thus, the correlations between U(t) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 are non-negligible. However, as training proceeds, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta [U(t) f_0]$$\end{document}Δ[U(t)f0] becomes compatible with zero within the error bars, suggesting that the correlations are progressively suppressed. The case of the V operator is even more striking, as Δ [V(t) Y]Δ[V(t)Y] is clearly non-negligible across all training times, although it also shows a decreasing trend as training proceeds. This suggests that the correlations between V(t) and Y cannot be neglected.Fig. 27Behavior of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta [U(t)f_0]$$\end{document}Δ[U(t)f0] and Δ [V(t)Y]Δ[V(t)Y], as defined in Eqs. (240) and (241), as functions of the training time. The operators U(T) and V(T) are constructed by taking the NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}} = 10{,}000$$\end{document}Tref=10,000, which is fixed across panels. The uncertainties are extracted from the bootstrap ensemble as discussed in the text Behavior of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta [U(t)f_0]$$\end{document}Δ[U(t)f0] and Δ [V(t)Y]Δ[V(t)Y], as defined in Eqs. (240) and (241), as functions of the training time. The operators U(T) and V(T) are constructed by taking the NTK at \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\textrm{ref}} = 10{,}000$$\end{document}Tref=10,000, which is fixed across panels. The uncertainties are extracted from the bootstrap ensemble as discussed in the text Let us conclude this brief discussion by noting that Eq. (239) resembles the structure of a linear method, like Backus–Gilbert or Gaussian Processes. Error decomposition The analytical expression for the covariance matrix, Eq. (233), allows us to monitor the relative size of the three contributions as training proceeds. For a properly trained ensemble of networks, the covariance of the trained fields should be dominated by the statistical error on the data. We show the diagonal entries of the two contributions \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(00)}$$\end{document}Ct(00) (blue band) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(YY)}$$\end{document}Ct(YY) (orange band) to the error budget in Fig. 28, for different frozen NTKs (rows) and different training epochs (columns), using L2 data as before. We do not show the mixed term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(0Y)}$$\end{document}Ct(0Y), since it is negligible with respect to the other two contributions—the two sources of uncertainty are largely uncorrelated. In general, we observe that toward the end of the training process, the contribution from the data \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(YY)}$$\end{document}Ct(YY) becomes dominant with respect to the contribution coming from the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_t^{(00)}$$\end{document}Ct(00). We also see that the suppression of the initial condition is more severe and happens earlier when the frozen NTK is taken at later stages of training. In order to study the dependence of the error decomposition on the initial condition, we repeat the same analysis for the case of scaled input f(x, łog x)f(x,logx), i.e. with two neurons in the input layer. The initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0 of the analytical solution is drawn from the same prior distribution as the trained solution, and corresponds to the orange curve in Fig. 10. The resulting error decomposition is shown in Fig. 29 for L2 data, where panels are ordered as in Fig. 28. Inspecting the figures, we observe that now the contribution of the initial condition becomes dominant in the small-x region, even for large training times and irrespective of the epoch at which the NTK is frozen. This result reflects the behavior of the prior distribution at small-x, where indeed error bands are significantly enlarged with respect to the case of linear input. Interestingly, even the contribution from the data, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C^{(YY)}_t$$\end{document}Ct(YY), increases at small-x toward larger training times. This can be explained by observing that, despite not being explicitly dependent on the initial condition \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f_0$$\end{document}f0, the evolution operator V(t) is constructed from a frozen NTK that has encoded the dependence on the architecture through the training process [see Eq. (219)]. That the difference between Figs. 28 and 29 is primarily localized at small-x showcases that, for the region left uncovered by the data, the methodology is not able to suppress the dependence on the initial condition. In fact, where the information from the data is available (corresponding roughly to x ≳ 0.01x≳0.01), the dependence on the initial condition lessens as the analytical solution evolves (left to right). This reduction occurs more rapidly for frozen NTKs taken at later stages of training (top to bottom), showing that there is a non-trivial interplay between the information provided by the data and that acquired by the NTK.Fig. 28Decomposition of the error budget of the trained fields into the two components from the initial condition (blue) and from the data (orange), as defined in Eqs. (234) and (236). Each row corresponds to a different frozen NTK, while the columns represent different training epochs. L2 data are used. We see that if the NTK is taken at later stages of training, the contribution from the initial condition is severely suppressed toward the end of trainingFig. 29Similar to Fig. 28, but now for the case of scaled input f(x, łog x)f(x,logx) Decomposition of the error budget of the trained fields into the two components from the initial condition (blue) and from the data (orange), as defined in Eqs. (234) and (236). Each row corresponds to a different frozen NTK, while the columns represent different training epochs. L2 data are used. We see that if the NTK is taken at later stages of training, the contribution from the initial condition is severely suppressed toward the end of training Similar to Fig. 28, but now for the case of scaled input f(x, łog x)f(x,logx) These studies reveal the intricate connection between the prior distribution and the uncertainties of PDFs. In the region constrained by data, the error is dominated by the statistical error on the data, rather than by the fluctuations of the initial fields. This is an important step in our study of the error estimates. It guarantees that the error bars computed from the ensemble of trained PDFs are not biased by the choice of prior, which depends on the selected architecture, activation function, and probability distributions for the biases and weights at initialization. Attempts to a synthesis Let us try to conclude these lectures by summarizing some lessons we learned along the way. First of all, it is worthwhile emphasizing that the discussion of the training based on the NTK can be adapted to any parametrization and minimization process, including fixed functional forms [3, 27, 28] or kernels [29]. What information can be extracted from the NTK in those cases is an open question for future investigations, aiming at comparing different procedures within a unified framework. From a more general perspective, we can trace the difficulties in solving inverse problems to the fact that the initial question is ill-posed. A stable solution can only be obtained by regularizing the problem in one way or another. The probabilistic approach that we have advocated here has the advantage of casting different approaches as alternative ways to regularize the problem and define a posterior distribution for the solution. Having analyzed in detail Backus–Gilbert, Gaussian Processes, and Neural Network solutions, we have been able to expose some relations between these different approaches. In particular, we found that the BG solution can be understood as special case of the GP solution. A careful analysis of the systematic errors in these two frameworks will allow for more robust results. It is interesting to remark that the GP solution is based on Bayesian inference, while the other two rely on minimizing some figure of merit. In particular, for the case of Neural Networks, the solution is obtained by finding an optimal stopping time, rather than letting the minimization reach an absolute minimum that would correspond to an overfitted solution. Understanding the connection between the stopping condition and the results obtained with BG/GP methods is an open question, which will shed light on the errors quoted with these different methods. Understanding the training dynamics of NNs is a fascinating problem, and we have only started to scratch the surface of the problem.