An Empirical Bayes approach to ARX Estimation footnoteinfo
Timofei Leahu, Giorgio Picci · 2025
Read An Empirical Bayes approach to ARX Estimation footnoteinfo by Timofei Leahu, Giorgio Picci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13384.txt.
Introduction
Timofei Leahu and Giorgio Picci fit line-based time models by a Bayes path that learns its own prior from the data. Such ARX models guess the next value of a series from past values and inputs. The Bayes twist needs prior settings, called hyperparameters, read from the data edge counts. Fair tests of each fit path need short, real-world data spans, not endless runs.
Their new fit path runs as a backward Kalman filter, a neat closed loop. On finite spans, the plain edge-count fit edged out the full Bayes fit in their tests. Slow drift in the true settings was part of the test grid as well. Control and signal teams get a clear pick guide for short data sets.
Catalog record
| CREATOR | Timofei Leahu, Giorgio Picci |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13384
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Bayes approach to ARX Estimation footnoteinfo Timofei Leahu, Giorgio Picci Abstract Empirical Bayes inference is based on estimation of the parameters of an a priori distribution from the observed data. The estimation technique of the parameters of the prior, called hyperparameters , is based on the marginal distribution obtained by integrating the joint density of the model with respect to the prior. This is a key step which needs to be properly adapted to the problem at hand. In this paper we study Empirical Bayes inference of linear autoregressive models with inputs (ARX models) for time series and compare the performance of the marginal parametric estimator with that a full Empirical Bayesian analysis based on the estimated prior. Such a comparison, can only make sense for a (realistic) finite data length. In this setting, we propose a new estimation technique of the hyperparameters by a sequential Bayes procedure which is essentially a backward Kalman filter. It turns out that for finite data length the marginal Bayes tends to behave slightly better than the full Empirical Bayesian parameter estimator and so also in the case of slowly varying random parameters. keywords: Empirical Bayes Estimation, Time series Identification, Conditionally Gaussian ARX models, Backward conditional Kalman filtering. † † thanks: [ footnoteinfo]T. Leahu was formerly with the School of Mathematical Engineering, University of Padova,Italy [email protected] , G. Picci is with the Department of Information Engineering (DEI), University of Padova, Italy, [email protected] 1 Introduction The standard estimation technique of AR, ARX or ARMA models is based on least squares, but this technique, even when interpretable as Maximum Likelihood, and hence theoretically guaranteeing consistency and asymptotic optimality, requires that the data are generated exactly by an element of a rigid model class, therefore assuming existence of true parameters and stationary data. This is often over-optimistic and, even with a reasonable data size one may sometimes obtain unsatisfactory results. One may hope that describing the model parameters as random variables and using a Bayesian approach could be a reasonable way to improve on this problem. However the choice of a prior on the parameter space turns out to be a stumbling difficulty since in most cases of practical interest no a priori knowledge of a reasonable probabilistic description of the parameters is available. This research is discussing a possible approach to the problem by using an Empirical Bayes philosophy. Empirical Bayes tries to estimate a prior distribution from the observed data. It looks like a naive idea and in fact it was proposed a long time ago see e.g. [ 9 ] [ 4 ] , [ 22 ] . It is only recently that the idea has been revitalized see e.g. [ 2 ] , [ 6 ] , [ 7 ] , [ 24 ] , [ 20 ] , and, due to the availability of massive computer power and suitable algorithms applied to system identification [ 1 ] , [ 21 ] . It is shown in the literature that in certain cases Empirical Bayes estimation techniques may even be superior to Maximum Likelihood [ 12 ] , [ 8 ] . In our opinion however this claim does not seem to be extendable in sufficient generality as it may depend on various factors, especially on the availability of multiple measurements and the chosen hyperparameter estimation technique. In this paper we shall attempt to analyze the behavior of Empirical Bayes estimation of ARX models by comparing the performance of a background a priori estimation based on the marginal with the a posteriori estimation method which uses the estimated prior within a Bayesian philosophy (which is in fact the Empirical Bayes methodology). We shall not consider comparison with Maximum Likelihood as its theoretical performance for finite data is hard to figure and the asymptotic expression is considered to be often too optimistic. Admittedly ARX is the simplest class of linear models of time series and its choice is essentially due to simplicity of the analysis. More general or more specific model classes require ad hoc methods which are outside the scope of this paper. See however [ 21 ] for an example. Asymptotically, all standard estimation methods for ARX models tend to coincide, and for a meaningful comparison one should consider finite data estimates which is obviously the situation one has to deal with in practice. For this reason, we shall first have to review the standard ARX identification material in a conditional setting . Section 2 will contain a quick review of standard identification material just to fix notations, and in the following section 3 we shall proceed to analyze and reinterpret the relative formulas from a conditionally Gaussian perspective . In sections 4 and 5 we analyze the Mean Square Error (MSE) performance of Empirical Bayes estimates and its comparison with a Bayesian estimation method. Finally in Sect. 6 we shall face the critical problem of hyperparameters estimation. This is first addressed in a step-by step way by extending some known results on estimation of static linear models but the key final idea is an algorithm to recover the (initial) a priori parameters by a backward Conditionally Gaussian Kalman filter , derived from scratch in the appendix, which seems to be new. 2 ARX models and pseudo-linear regression This is a review section aimed at fixing notations and recalling some standard identification material. Notations: In this paper, bold lowercase symbols like \mathbf{x} denote random quantities, while italic lowercase letters like x denote deterministic quantities, possibly vector-valued with numerical entries (in particular real scalars); the symbol t denotes discrete time running on a certain subset of the integers \mathbb{Z} ; the Mean Square Error (MSE) of an estimator is the sum of the squared bias plus the error variance. Matrix notations are standard, in particular \tr(A) denotes the trace of a square matrix A . Imposing an ARX structure to observed output and input data \{y(t),\,u(t);\,t=1,2,\ldots,N\} means describing them by a linear stochastic difference equation of the form \mathbf{y}(t)\,=\,\sum_{k=1}^{n}\,a_{k}\mathbf{y}(t-k)+\sum_{k=1}^{m}\,b_{k}\mathbf{u}(t-k)+\mathbf{w}(t)\,, (1) where \mathbf{w}:=\{\mathbf{w}(t),\,t\in\mathbb{Z}\} is a process of random errors which is assumed Gaussian uncorrelated (white noise). The data have been suitably pre-processed e.g. by subtracting the sample mean so as to be compatible with zero-mean and stationarity. Also we shall assume that the input sequence \{u(t)\} is generated independently of the output data \{y(t)\} (no feedback) and can therefore be treated as a deterministic sequence. The ARX model depends on p:=n+m unknown parameters which will be written as a column vector: \theta:=\left[\begin{matrix}a_{1}&\ldots&a_{n}&b_{1}&\ldots&b_{m}\end{matrix}\right]^{\top} (2) and on the unknown noise variance \sigma^{2} . It is formally written in pseudo-linear regression form as \mathbf{y}(t)=\boldsymbol{\varphi}(t)^{\top}\theta+\mathbf{w}(t)\,,\qquad t\in\mathbb{Z} (3) where \boldsymbol{\varphi}(t) is a column vector depending on past data: \boldsymbol{\varphi}(t)^{\top}=\left[\begin{matrix}\mathbf{y}(t-1)&\ldots&\mathbf{y}(t-n)&\mathbf{u}(t-1)&\ldots&\mathbf{u}(t-m)\end{matrix}\right] (4) Of course there is an important difference with the classical linear regression model, since the coefficient vectors \boldsymbol{\varphi}(t) of the model are random, depending on the (past) input-output variables. Stacking sequentially the N equations ( 3 ) on top of each other one obtains a system of relations which, rewritten in vector form looks like \mathbf{y}=\Phi_{N}\,\theta+\mathbf{w} (5) where the N -dimensional random vectors \mathbf{y} and \mathbf{w} have components \mathbf{y}(t) and \mathbf{w}(t) indexed by t=1,2,\ldots,N and \Phi_{N} is an N\times p matrix of past data of the form: