Bayesian Inference with Shaped Deep Non-linear MLPs
Boris Hanin, Tianze Jiang · 2026
Read Bayesian Inference with Shaped Deep Non-linear MLPs by Boris Hanin, Tianze Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30860.txt.
Introduction
Hanin and Jiang study Bayesian inference in deep nonlinear multilayer perceptrons when model and training set grow together. Model and data limits do not commute, so valid limits are unclear. Building on the Neural Covariance stochastic differential equation framework, they analyze posteriors where a depth ratio of layers, samples, and width stays of order one. Their framework covers smooth and rectified activations at any temperature. To first order in that ratio, they give a criterion for when depth helps evidence, and show the posterior matches a kernel method.
Catalog record
| CREATOR | Boris Hanin, Tianze Jiang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 102224 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30860
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bayesian Inference with Shaped Deep Non-linear MLPs Boris Hanin, Tianze Jiang Abstract A central aim of deep learning theory is to characterize how neural networks make predictions in the regime of simultaneously large model and training set size. Since the limits of diverging number of model parameters and dataset size do not commute it is not clear a priori what limits exist. In this work, we shed new light on these questions by studying Bayesian inference in deep non-linear MLPs in the regime where the number of training samples ( P P ), the input dimension ( N 0 N_{0} ), the hidden layer width ( N N ), and the number of hidden layers ( L L ) can all be large. We build on the Neural Covariance SDE [ LNR22 ] to analyze predictive posteriors in the regime where L P / N ∈ Θ ( 1 ) LP/N\in\Theta(1) , playing the role of an effective network depth. Our framework covers both smooth and ReLU activation functions and applies to arbitrary temperature. We find to first order in L P / N LP/N a simple criterion for which data generating processes benefit from depth in the sense that larger L P / N LP/N increases the Bayesian model evidence. We also give a novel derivation of a prior result from the physics literature that at least to first order in L P / N LP/N , the Bayesian predictive posterior is remarkably simple and is simply equivalent to that of a data-dependent kernel method. 1 Introduction From approximation to training dynamics and generalization, many important results in deep learning theory concern the fundamental setting of deep Multi-layer Perceptrons (MLPs) [ Zha+17 , AS17 , Bel+19 , Han19 , Dau+19 , RYH22 ] . A core aim of many of these articles is to rigorously characterize how scaling up different dimensions, such as the number of hidden layers ( L ), the width of each hidden layer ( N ), and the size of the dataset ( P ) jointly shape the behavior of large MLPs at initialization or during training. Many such works consider the limit of infinite width at fixed depth and dataset size, giving rise to by-now classical results concerning both the Neural Tangent Kernel (NTK) and the Neural Network Gaussian Process (NNGP) [ Lee+17 , COB19 , Du+19 , COB19 , JGH20 , Yan21 ] and mean-field limits [ COB19 , RV18 , MMN18 , SS20 ] . However, the infinite-width limit (especially in the NTK parameterization) suppresses important phenomena capturing the role of depth [ HN19a , HDR19 , HDR22 , LNR22 , RYH22 ] when L\propto N or large dataset size P\propto N [ LS21 , Pac+23 , SNR23 , Nav+21 , BLR24 ] . More recent studies [ HZ24 , Han24 , Cam+25 , Li+26 ] also focuses on proportional limits, in which depth, width, and data diverge jointly, revealing new stochastic and geometric structures. Despite recent progress, our understanding of how jointly scaling architecture dimensions ( L,N,P and input dimension N_{0} ) in non-linear networks impacts training-time fitting and test-time prediction remains incomplete. When and why do deep neural networks behave like kernel methods, and what enables feature-learning? When feature learning is possible, what features do MLPs learn? What combinations of data and architecture benefit from depth? In this work, we present the first mathematically rigorous way to answer these questions on non-linear deep MLPs from the Bayesian inference perspective [ Mac92 , Nea94 , HZ23 , HZ24 , Bas+25 ] . Bayesian inference analyzes a neural network architecture x\mapsto f(x;\Theta) by choosing a specific prior on the weights \mathbb{P}(\Theta) and computing the posterior \mathbb{P}(\Theta|\mathcal{D}) (conditional on the training dataset \mathcal{D} ) via tilting the prior with the training loss \mathbb{P}(\mathcal{D}|\Theta)\propto\exp(-\beta\mathcal{L}(\mathcal{D};\Theta)) . The posterior network prediction on test data x_{0} is therefore the pushforward of the weight posterior \Theta\sim\mathbb{P}(\Theta|\mathcal{D}) to the space of functions outputs. The marginal of training data \mathbb{P}(\mathcal{D}) , equivalently the likelihood of the architecture and prior, can be used for model selection. Our main contribution in this paper is characterizing and justifying a rigorous unifying framework of analyzing Bayesian inference in non-linear MLPs when LP\propto N , building on the Neural Covariance SDE (NSDE) [ LNR22 , LN24 , Noc+23 ] . Our method of computing Bayesian statistics (outlined in Section 2 and elaborated in Appendix A ) covers a wide range of non-linear activation functions (leaky ReLU in Section A.2 and smooth functions in Section A.3 ) and can naturally be applied at any temperature. As examples of applying our framework to answer questions about learning and inference, we compute the perturbative expansion at small effective-depth ( Section 4.2 and Section 4.3 ). Our calculations in the perturbative expansion recover and generalize many results in prior literature for linear networks [ HZ23 ] and weakly non-linear networks with cubic activation function [ HZ24 ] . We further generalize these results and provide novel insights on the perturbative prediction feature map as well as the Bayesian evidence analysis with different activation functions. 1.1 Preliminaries and technical setup Model. We consider MLPs with L fully connected layers f(x;\Theta):x\in\mathbb{R}^{N_{0}}\to y\in\mathbb{R} 1 1 1 While our analysis easily extends to y\in\mathbb{R}^{d} output for d\in\Theta(1) , we present scalar output results for simplicity. : z_{1}=\frac{1}{\sqrt{N_{0}}}W_{0}x,\quad\phi_{\ell}=\phi_{s}(z_{\ell}),\quad z_{\ell+1}=\sqrt{\frac{c}{N}}W_{\ell}\phi_{\ell},\quad y=z_{\mathrm{out}}=\sqrt{\frac{c}{N}}W_{\mathrm{out}}\phi_{L}\in\mathbb{R} (1) where \Theta=(W_{0},W_{1},\dots,W_{L-1},W_{\operatorname{out}}) is the set of weights, z_{i}\in\mathbb{R}^{N},i=1,2,\dots,L are hidden pre-activations, c is a normalizing constant such that c^{-1}\triangleq\mathbb{E}_{g\sim\mathcal{N}(0,1)}\left[\phi_{s}(g)^{2}\right] so each hidden neuron is \mathcal{N}(0,1) at init, \phi_{s} is the shaped activation function with parameter s (see below), and the interior recursion runs from \ell=1,2,\dots,L-1 . At initialization (also as the network prior), we assume that all W_{i} ’s are initialized with i.i.d. \mathcal{N}(0,1) entries. We wish to characterize the network posterior when L,P,N\to\infty jointly under the squared loss \mathcal{L}(\mathcal{D};\Theta)\triangleq\frac{1}{2}\sum_{\mu=1}^{P}\left(f\left(x_{\mu};\Theta\right)-y_{\mu}\right)^{2} over a training dataset \mathcal{D}=\left\{\left(x_{\mu},y_{\mu}\right),\mu=1,\ldots,P\right\} , where x_{\mu}\in\mathbb{R}^{N_{0}},y_{\mu}\in\mathbb{R} are training data. Proportional width and depth limit. Our results concern a sequential limit in which we first consider fixed finite P,N_{0} and take the L,N\to\infty limit with L\propto N ; then we take P,N_{0} to be large. Prior work (e.g. [ HZ23 , HZ24 ] ) suggests these limits commute, but we do not establish this rigorously. For a fixed number of datapoints P , prior studies focusing on the product of L random N\times N matrices [ Han19 , HZ23 , BLR24 , Li+26 ] indicate that t=L/N controls the initialization behavior in deep linear networks when L,N\to\infty , and that both t=0 and t=\infty yield degenerate scaling regimes. While such conclusion does not immediately extend to networks with a non-linear activation function, one can apply a shaping mechanism to the non-linear activation function \phi_{s} such that the ratio t=L/N still uniquely controls the initialization distribution in non-linear MLPs, as we see next. Shaped activation function. For a growing depth L\to\infty , to ensure that the output distribution at initialization is non-degenerate, various prior works [ Mar+21 , LNR22 , ZBM22 , HZ24 ] have established the necessity of shaping the activation function \phi_{s} . Specifically, for a smooth base non-linearity \phi_{\operatorname{base}}(x) normalized to \phi_{\operatorname{base}}(0)=\phi_{\operatorname{base}}^{\prime}(0)-1=0 , we consider \phi_{s}(x)=s\phi_{\operatorname{base}}(x/s)=x+\frac{1}{2s}\phi^{\prime\prime}_{\operatorname{base}}(0)\cdot x^{2}+\frac{1}{6s^{2}}\phi^{\prime\prime\prime}_{\operatorname{base}}(0)\cdot x^{3}+\dots (2) for a shaping factor s=s(N,L,P) that