The Bernstein–von Mises theorem for Bayesian one-pass online learning

Jeyong Lee, Junhyeok Choi, Dongguen Kim, Minwoo Chae · 2026

Read The Bernstein–von Mises theorem for Bayesian one-pass online learning by Jeyong Lee, Junhyeok Choi, Dongguen Kim, Minwoo Chae (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27442.txt.

Introduction

Lee and three colleagues propose a Bayesian online learning algorithm for the one-pass setting, where each sample is seen once. Existing guarantees usually need mini-batch sizes that grow without bound, which fails in one pass. Their method adds a warm-start phase for stable sequential updates and attains the optimal convergence rate. The authors also establish an online version of the Bernstein-von Mises theorem, giving valid uncertainty quantification without growing batches. Tests on generalized linear models match batch performance while beating existing online methods.

Catalog record

CREATORJeyong Lee, Junhyeok Choi, Dongguen Kim, Minwoo Chae
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE169662
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27442

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Bernstein–von Mises theorem for Bayesian one-pass online learning Jeyong Lee, Junhyeok Choi, Dongguen Kim, Minwoo Chae Abstract Bayesian online learning provides a coherent framework for sequential inference. However, its theoretical understanding remains limited, particularly in the one-pass setting. Existing theoretical guarantees typically require the mini-batch sample size to diverge, a condition that fails in the one-pass regime. In this paper, we propose a new Bayesian online learning algorithm tailored to the one-pass setting, which incorporates a warm-start phase to ensure stable sequential updates. For this algorithm, we show that the sequentially updated posterior attains the optimal convergence rate. Building on this, we establish an online analogue of the Bernstein–von Mises theorem, which guarantees valid uncertainty quantification without diverging mini-batch sample sizes. Our analysis is based on a novel theoretical framework that differs fundamentally from existing approaches in the online learning literature. Numerical experiments on generalized linear models show that the proposed method matches the performance of the batch estimator while outperforming existing online procedures. 1 Introduction Online learning refers to a statistical framework for settings where data arrive sequentially, typically arising in large-scale or streaming environments. Unlike batch learning, which constructs an estimator using the entire dataset at once, online learning updates parameters sequentially as new observations become available. Each update relies only on the previous estimate and the current observation without revisiting past data. This requirement is justified by memory constraints that make it impractical to store the full dataset and by computational considerations, since recomputing a batch estimator at each step can be prohibitively expensive. In practice, online updates are often performed using small mini-batches of data. A particularly stringent formulation is the one-pass setting, in which each observation is processed individually and contributes exactly once to the update. Motivated by these practical constraints, online learning methods have been extensively developed across a wide range of statistical models. Representative examples include topic models ( 12 , 38 , 14 ) , Dirichlet process mixtures ( 13 ) , matrix factorization ( 22 ) , survival analysis ( 40 , 39 , 7 ) , and quantile regression ( 6 , 19 ) . From a theoretical perspective, however, the online learning setting imposes structural constraints that fundamentally distinguish it from classical batch learning. In particular, since data arrive sequentially, the total sample size is typically unknown, making it difficult to tune hyperparameters that depend on a fixed sample size. Moreover, as past observations are not revisited, errors or biases introduced at early stages may propagate through subsequent updates and affect the entire trajectory of the estimator. Consequently, the statistical analysis of online estimators departs from the classical M-estimation framework, necessitating new analytical tools. Understanding the theoretical behavior of online estimators has therefore become an important problem. In the frequentist literature, stochastic gradient descent (SGD) is a standard approach to online learning. Due to its recursive structure and computational efficiency, SGD, originally introduced by 28 , and its variants have been extensively studied. As pioneering works, 29 and 27 independently proposed averaging schemes for SGD iterates, with 27 establishing asymptotic normality of the resulting estimator. More recently, 34 introduced stabilized variants with improved finite-sample performance, and subsequent works have focused on establishing valid uncertainty quantification for SGD-based estimators ( 5 , 41 , 18 , 9 ) . Importantly, these results extend naturally to the one-pass setting while retaining their theoretical guarantees. On the other hand, the Bayesian framework provides a conceptually appealing alternative, as it updates a distribution over the parameter space in a sequential manner. The posterior distribution at each step can be used as the prior for the next step, allowing for a coherent learning procedure in the online setting. However, except in conjugate models, exact posterior updates are generally intractable, necessitating approximation at each step ( 25 , 30 ) . This has led to the development of Bayesian online learning methods that approximate the posterior within a tractable family, often using variational techniques ( 3 , 21 , 24 , 15 , 16 ) . Despite these methodological advances, the theoretical understanding of Bayesian online learning remains limited. Most existing works focus on algorithmic design, while rigorous statistical guarantees are relatively scarce. To address this gap, recent work by 17 has shown that sequentially updated distributions can be asymptotically equivalent to the full posterior based on the entire dataset. However, such results rely on diverging mini-batch sample sizes and therefore do not fully capture the one-pass setting. Moreover, empirical evidence in 17 suggests that when the mini-batch size is small, statistical inefficiency may arise. These observations highlight the need for Bayesian online learning methods tailored to the one-pass setting, together with a rigorous theoretical analysis of their statistical properties. Motivated by this gap, we propose a Bayesian online learning procedure tailored to the one-pass setting and establish its theoretical validity. The proposed method is inspired by variational approximation, where the expected log-likelihood is replaced by a local quadratic approximation, leading to a tractable update with a closed-form expression. This construction yields a computationally efficient procedure that is well-suited for the one-pass setting. However, the local quadratic approximation is valid only in a neighborhood of the true parameter, and errors arising in early updates may propagate through subsequent updates in the one-pass setting. This motivates initializing the recursion with a sufficiently accurate estimate. To this end, the method incorporates a warm-start phase that ensures the estimator enters a local neighborhood before the online recursion begins. We further establish an online analogue of the Bernstein–von Mises (BvM) theorem by analyzing the asymptotic behavior of the sequentially updated posterior distributions. This result shows that, even in the fully online regime, the proposed posterior achieves asymptotic efficiency and provides valid frequentist uncertainty quantification, without requiring diverging mini-batch sizes, in contrast to 17 . To establish this result, we develop a theoretical framework adapted to the algorithmic structure of the procedure, enabling a refined non-asymptotic analysis. This framework differs from existing techniques in the SGD literature and avoids the strong structural assumptions typically imposed in their analyses. See Section 3.3 for details. Finally, we complement our theoretical findings with extensive numerical experiments, demonstrating that the proposed method achieves strong empirical performance compared to existing approaches across a range of settings. In particular, the experiments highlight the importance of the warm-start phase, as omitting this phase can lead to substantially slower convergence in certain settings. See Section 4 for details. The remainder of this paper is organized as follows. In the following subsections, we introduce our setup and the basic notation used throughout the paper. Section 2 presents the general framework of Bayesian online learning and introduces the proposed one-pass algorithm. Section 3 develops the main theoretical results, including the convergence analysis of the estimator and the establishment of the online BvM theorem. Numerical experiments are presented in Section 4 , and concluding remarks are given in Section 5. All proofs and additional technical details are deferred to the Appendix. 1.1 Setup and notations Table 1: Important notations Notation Location Notation Location Notation Location d_{V}(\cdot,\cdot) ( 3.10 ) V_{t},\,\Delta_{t} ( 3.9 ) \theta_{t},\bm{\Omega}_{t} ( Alg ) \hat{\theta}_{t}^{\textnormal{\rm{MAP}}},\hat{\theta}_{t}^{\textnormal{\rm{ML}}} ( 2.8 ), ( 3.8 ) \underline{\lambda}_{0},\overline{\lambda}_{0} ( 3.2 ) \Theta_{r},\,r (

Read the complete text