A Bayesian Framework for Regularized Estimation in Multivariate Models Integrating Approximate Computing Concepts

Jan Kalina · 2025

Read A Bayesian Framework for Regularized Estimation in Multivariate Models Integrating Approximate Computing Concepts by Jan Kalina (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10045.txt.

Introduction

Jan Kalina's 2025 paper rethinks shrinkage estimation through a Bayesian lens. It also takes a detour into computing with coarse numbers. Shrinkage pulls noisy estimates toward a shared centre. That pull helps when many linked values are measured at once. Kalina first links Bayes methods with inference under rounded inputs. The link ties a school of stats to a habit from engineering.

Next, means from two samples justify a shrink toward the pooled mean. The same Bayesian case is made for discriminant analysis. Several shrinkage plans are defended on those grounds. Fresh methods to compute them are also given. Trials on real and simulated data back the plans. Users gain shrinkage with a clear basis, not a rule of thumb.

Catalog record

CREATORJan Kalina
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10045

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Bayesian Framework for Regularized Estimation in Multivariate Models Integrating Approximate Computing Concepts Jan Kalina Abstract This paper discusses regularized estimators in the multivariate statistical model as tools naturally arising within a Bayesian framework. First, a link is established between Bayesian estimation and inference under parameter rounding (quantization), thereby connecting two distinct paradigms: Bayesian inference and approximate computing. Next, Bayesian estimation of the means from two independent multivariate normal samples is employed to justify shrinkage estimators, i.e., means shrunk toward the pooled mean. Finally, regularized linear discriminant analysis (LDA) is considered. Various shrinkage strategies for the mean are justified from a Bayesian perspective, and novel algorithms for their computation are proposed. The proposed methods are illustrated by numerical experiments on real and simulated data. \headingtitle Bayesian Framework for Regularized Multivariate Estimation \headingauthor Jan Kalina keywords multivariate data, Bayesian estimation, regularization, classification analysis, algorithms, shrinkage estimators 1 Introduction In a variety of models of multivariate statistics and machine learning, regularized estimators may be obtained in the Bayesian statistical setup [ 6 ] . This paper is interested in finding interpretations and relationships between various types of estimators in the multivariate model including an extension for a classification model with data observed in several different groups. Regularized (shrinkage) estimators are not only crucial for high-dimensional data analysis, where classical methods often become unstable or infeasible, but also beneficial even when working with small datasets, as they tend to improve predictive accuracy [ 42 ] . In a wide range of probabilistic models, regularization emerges naturally from Bayesian reasoning: prior knowledge is incorporated directly into the estimation process, resulting in estimates that are pulled (shrunk) towards a predefined reference value, typically the prior expectation [ 16 ] . A shrinkage estimator is a statistical estimator that improves estimation accuracy by combining observed data with additional information or assumptions, effectively ”shrinking” raw estimates toward a predetermined target or prior value. This shrinkage reduces estimation variance, often at the cost of introducing some bias, but typically results in a lower overall mean squared error compared to unbiased estimators. This form of regularization helps reduce variance and improve generalization performance, which is especially valuable when the data are noisy. In multivariate settings, such Bayesian-inspired regularization enhances the stability and robustness of the estimators [ 20 ] . We regard this as a natural and fruitful synthesis of Bayesian ideas and modern regularization techniques, aligning well with the goals of reliable and interpretable data analysis [ 14 ] . As a novelty, a connection between Bayesian estimators and quantized estimation, i.e. estimation contaminated by additional uncertainty or error, is established here. Moreover, we focus on shrinkage estimators for the means of independent multivariate normal samples and subsequently on regularized linear discriminant analysis, which frequently employs shrinkage techniques and has found many applications in high-dimensional data [ 10 ] . In particular, we justify the use of shrunken means within a specific Bayesian model and formulate two algorithms for regularized linear discriminant analysis exploiting linear algebra manipulations. This paper has the following structure. Section 2 is devoted to Bayesian estimation of parameters of the multivariate normal distribution. In Section 3 , an original connection between Bayesian estimation and estimation in a model with rounding (quantization) of the estimated parameters is established. In Section 4 , Bayesian estimation for two independent normally distributed samples is derived, which theoretically justifies using regularized estimators in the given context and also in the context of regularized linear discriminant analysis of Section 5 . Section 6 presents experiments over a real and a simulated dataset. Section 7 concludes the paper. 2 Bayesian estimation in the multivariate model Let us assume the multivariate model X_{i}\sim{\sf N}_{p}(\mu,\Sigma),\quad i=1,\dots,n, (1) with i.i.d. p -variate random vectors following the p -variate normal distribution with n>p . Let X\in\mathbbm{R}^{n\times p} denote the data matrix with rows X_{1},\dots,X_{n} . The covariance matrix \Sigma has to fulfil \Sigma\in{\sf PD(p)} , where {\sf PD}(p) denotes the set of all positive definite symmetric matrices of size p\times p . This section discusses Bayesian estimation of the mean vector \mu\in\mathbbm{R}^{p} and the covariance matrix \Sigma . The main purpose here is to review the Bayesian estimation of these parameters, serving as a foundation for the subsequent sections. For a detailed description of Bayesian estimation principles, we refer the reader to the book [ 20 ] . The sample mean has the form \bar{X}=(\bar{X}_{1},\dots,\bar{X}_{p})^{T}=\frac{1}{n}X^{T}\hat{\jmath}_{n}=\frac{1}{n}\left(\hat{\jmath}_{n}^{T}X\right)^{T}\in\mathbbm{R}^{p},\quad\mbox{where}\quad\bar{X}_{j}=\frac{1}{n}\sum_{i=1}^{n}X_{ij} (2) and \hat{\jmath}_{n}=(1,\dots,1)^{T}\in\mathbbm{R}^{n} . Also, the maximum likelihood estimator in ( 1 ) is the sample mean. In addition, minimizing the sum of squared residuals \argmin_{\tilde{\mu}\in\mathbbm{R}^{p}}\sum_{i=1}^{n}(X_{i}-\tilde{\mu})^{2} (3) leads again to the sample mean as the resulting estimator. Under the assumption of a known \Sigma , let us consider the normal prior \mu\sim{\sf N}_{p}(\theta,\eta) (4) with known \theta\in\mathbbm{R}^{p} and \eta\in{\sf PD}(p) . This is a conjugate prior for the multivariate normal likelihood with known covariance, meaning that the posterior distribution belongs to the same parametric family (i.e. multivariate normal). The use of a conjugate prior is advantageous because it leads to a closed-form expression for the posterior, which simplifies both analytical derivations and computational implementation of Bayesian inference. The Bayesian estimator \hat{\mu} of the parameter \mu obtained as the mean of the posterior distribution is a shrinkage estimator of \mu shrunken towards \theta . It depends on the nuisance covariance matrix \Sigma . The formula for the estimator, which was provided e.g. in the seminal paper [ 9 ] , is given by \hat{\mu}=(n\Sigma^{-1}+\eta^{-1})^{-1}(n\Sigma^{-1}\bar{X}+\eta^{-1}\theta). (5) Remark 2.1 . If assuming \eta^{-1}=c\Sigma^{-1} for some c>0 , the estimator ( 5 ) does not depend on \Sigma , because ( 5 ) simplifies to \hat{\mu}=(1-\delta)\bar{X}+\delta\theta,\quad\mbox{where}\quad\delta=\frac{c}{n+c}. (6) The James–Stein estimator is a biased estimator of \mu with a smaller quadratic risk than the mean [ 19 ] for p\geq 3 . Let us assume a single X\sim{\sf N}_{p}(\mu,\sigma^{2}{\cal I}_{p}) with a known \sigma^{2}>0 , where {\cal I}_{p} denotes the identity matrix of dimension p . The estimator is formulated as an estimator shrunken towards 0 in the form \left(1-\frac{(p-2)\sigma^{2}}{||X||^{2}_{2}}\right)X, (7) which has the structure of the Bayesian estimator in ( 6 ) obtained with a Gaussian prior with a zero expectation. The James–Stein estimator ( 7 ) is a foundational result demonstrating the advantages of shrinkage estimation under the assumption of a known diagonal covariance matrix with equal variances \sigma^{2} across all coordinates. Although these assumptions may not hold in all practical scenarios, the estimator’s core principles have inspired a wide range of effective regularization methods that are widely used today. An analogous shrinkage estimator can also be defined for the sample mean of a Gaussian distribution when the covariance matrix is diagonal and known, but the coordinate-wise variances \sigma_{1}^{2},\ldots,\sigma_{p}^{2} may differ [ 8 ] . This extension relaxes the assumption of homoscedasticity and allows for heterogeneous variability across coordinates. The classical James–Stein estimator nevertheless remains a cornerstone theoretical result, providing strong justification for the use of regularized (i.e. Bayesian) estimators in multivariate settings [ 19 ] . Building on the concept of shrinkage in ( 7 ), many popular regularized estimators have been developed, including ridge regression and lasso regression. These ideas also form the basis of modern regularization techniques frequently employed in machine learning, such as bagging, boosting, and weight decay in

Read the complete text