Deep regression learning from dependent observations with minimum error entropy principle

2026

Read Deep regression learning from dependent observations with minimum error entropy principle (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11138.txt.

Introduction

William Kengne and Modou Wade study deep regression learning when the observations are dependent, drawn from strongly mixing processes rather than independent samples. Their estimators train deep neural networks under the minimum error entropy principle, in plain and sparse-penalized versions. For both, they bound the expected excess risk over Hölder and composition Hölder function classes. Under Gaussian errors those bounds match known lower bounds up to a logarithmic factor, so the rates are minimax optimal.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94808
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11138

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deep regression learning from dependent observations with minimum error entropy principle Abstract William Kengne and Modou Wade Institut Camille Jordan, Université Jean Monnet, 23 Rue Dr Paul Michelon 42023 Saint-Etienne Cedex 2, France THEMA, CY Cergy Paris Université, 33 Boulevard du Port, 95011 Cergy-Pontoise Cedex, France E-mail: [email protected] ; [email protected] Abstract : This paper considers nonparametric regression from strongly mixing observations. The proposed approach is based on deep neural networks with minimum error entropy (MEE) principle. We study two estimators: the non-penalized deep neural network (NPDNN) and the sparse-penalized deep neural network (SPDNN) predictors. Upper bounds of the expected excess risk are established for both estimators over the classes of Hölder and composition Hölder functions. For the models with Gaussian error, the rates of the upper bound obtained match (up to a logarithmic factor) with the lower bounds established in [ 37 ] , showing that both the MEE-based NPDNN and SPDNN estimators from strongly mixing data can achieve the minimax optimal convergence rate. Keywords: Deep neural networks, nonparametric regression, minimum error entropy, strong mixing, minimax optimality. 1 Introduction Deep learning has achieved impressive results in a wide range of learning tasks, such as image processing [ 29 ] , speech recognition [ 16 ] , and, more generally, in various fields of artificial intelligence (AI). However, theoretical support and guarantees of deep neural networks (DNN) algorithms in many frameworks are still a challenge nowadays. In recent years, numerous studies have contributed to understand the theoretical properties of DNN estimators. See, for instance, [ 37 ] , [ 20 ] [ 21 ] , [ 11 ] , [ 44 ] , [ 38 ] , (and the references therein), for some results from independent and identical distributed (i.i.d) observations. For some recent results from dependent data, see, among other works, [ 32 ] , [ 27 ] , [ 26 ] , [ 30 ] , [ 24 ] , [ 1 ] . For nonparametric regression, most of the results established on the properties of DNN estimators are based on the L_{2} (least squares) loss, see for example, [ 37 ] , [ 28 ] , [ 32 ] , [ 20 ] , [ 21 ] , [ 27 ] , [ 27 ] , [ 30 ] . Such estimator, which is perfect for problems involving Gaussian noise is sensitive to non-Gaussian and heavy tailed errors, see, for instance [ 9 ] , [ 17 ] , [ 3 ] . There are several works that consider an entropy criteria for nonparametric regression, see for example, [ 18 ] , [ 17 ] , [ 12 ] , [ 14 ] , [ 19 ] . But, theoretical properties of entropy-based DNN estimators for nonparametric regression have not yet been much studied in the literature. We refer to [ 41 ] and [ 4 ] for some recent works on this direction. We consider a stationary and ergodic process \{Z_{t}=(X_{t},Y_{t}),t\in\mathbb{Z}\} , from the nonparametric regression model: Y_{t}=h_{0}(X_{t})+\xi_{t},~X_{t}\sim P_{X_{t}} (1 . 1) where h_{0}:\mathbb{R}^{d}\to\mathbb{R} is the unknown regression function, (\xi_{t})_{t\in\mathbb{Z}} is a centered i.i.d. error process and \xi_{t} is independent of the input variable X_{t} . The input and the output spaces are \mathcal{X}\subset\mathbb{R}^{d} (with d\in\mathbb{N} ) and \mathcal{Y}\subset\mathbb{R} respectively. It is assumed that \xi_{0} admits a density f with respect to the Lebesgue measure. Let D_{n}\coloneqq\{(X_{1},Y_{1}),\dots,(X_{n},Y_{n})\} be a training sample generated from the process \{Z_{t}=(X_{t},Y_{t}),t\in\mathbb{Z}\} . We deal with a class of DNN estimators \mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F_{n},S_{n}) (defined in Section 2.2 , see ( 2.9 )). Also, let \mathcal{F} denote the class of all measurable functions from \mathcal{X} to \mathcal{Y} . The minimum error entropy (MEE) principle aims to choose the predictor h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F_{n},S_{n}) that minimizes an entropy of the error Y_{0}-h(X_{0}) . In this work, we deal with Shannon’s entropy, and consider the risk for all h\in\mathcal{F} , R(h)=\mathbb{E}_{Z_{0}}[-\log f(Y_{0}-h(X_{0}))],~\text{with}~Z_{0}=(X_{0},Y_{0}). (1 . 2) So, the corresponding loss function is, \ell(h(X_{0}),Y_{0})=-\log f(Y_{0}-h(X_{0})). (1 . 3) The target predictor (assumed to exist) h^{*}\in\mathcal{F} satisfies: R(h^{*})=\underset{h\in\mathcal{F}}{\inf}R(h). (1 . 4) For any h\in\mathcal{F} , the excess risk is given by: \mathcal{E}_{Z_{0}}(h)=R(h)-R(h^{*}),~\text{with}~Z_{0}=(X_{0},Y_{0}). (1 . 5) The density f of the error is assumed to be known (see discussion below, Section 6 ). We aim to build from D_{n} , a DNN predictor \widehat{h}_{n}\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F_{n},S_{n}) with suitably chosen network architecture (L_{n},N_{n},B_{n},F_{n},S_{n}) , that minimizes the empirical version of the entropy (risk). Define the MEE-based non-penalized DNN (NPDNN) estimator by: \widehat{h}_{n,NP}=\underset{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F_{n},S_{n})}{\argmin}\Big[-\dfrac{1}{n}\sum_{i=1}^{n}\Big(\log f(Y_{i}-h(X_{i}))\Big)\Big]. (1 . 6) For this estimator \widehat{h}_{n,NP} , the regularization is performed by the sparsity parameter S_{n} . In the following, we also consider the sparse-penalized regularization with the DNN class \mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F_{n}) (defined in Section 2.2 , see ( 2.8 )). So, the MEE-based sparse-penalized DNN (SPDNN) estimator \widehat{h}_{n,SP} is given by: \widehat{h}_{n,SP}=\underset{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}{\argmin}\Big[-\dfrac{1}{n}\sum_{i=1}^{n}\Big(\log f(Y_{i}-h(X_{i}))\Big)+J_{n}(h)\Big], (1 . 7) where J_{n}(h) denotes the sparse penalty term, defined by: J_{n}(h)=\sum_{j=1}^{p}\pi_{\lambda_{n},\tau_{n}}\big(|\theta_{j}(h)|\big), (1 . 8) where \pi_{\lambda_{n},\tau_{n}}:[0,\infty)\to[0,\infty) is a function with two tuning parameters \lambda_{n},\tau_{n}>0 , and \theta_{j}(h) is the j -th component of \theta(h)=\big(\theta_{1}(h),\cdots,\theta_{p}(h)\big)^{T} , which is the parameter of the network estimator h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F) , and T denotes the transpose. We assume that \pi_{\lambda_{n},\tau_{n}} satisfies the two following conditions (see also [ 30 ] ): (i) \pi_{\lambda_{n},\tau_{n}}(0)=0 and \pi_{\lambda_{n},\tau_{n}}(\cdot) is non-decreasing. (ii) \pi_{\lambda_{n},\tau_{n}}(x)=\lambda_{n} if x>\tau_{n} . Examples of such penalty J_{n}(h) are the clipped L_{1} penalty considered in [ 43 ] , [ 35 ] , defined by for all x\geq 0 as: \pi_{\lambda_{n},\tau_{n}}(x)=\lambda_{n}\Big(\dfrac{x}{\tau_{n}}\land 1\Big), (1 . 9) the SCAD penalty considered by [ 13 ] , the minimax concave penalty [ 42 ] or the seamless L_{0} penalty in [ 7 ] , see [ 30 ] . We will be interested in establishing a bound of the excess risk \mathcal{E}_{Z_{0}}(\widehat{h}_{n}) , and studying how fast it tends to zero. As pointed out above, most of the results on the theoretical properties of the DNN predictor established in the literature are based on the L_{2} loss. Such method minimizes the variance of the error variable Y_{0}-h(X_{0}) (for a predictor h ), so it takes into consideration only the first two moments of the error variable. Such estimators cannot work very well for problems involving heavy tailed or non-Gaussian error, and are not robust to outliers, see also [ 17 ] . To solve the robustness question in nonparametric regression from DNN, several authors have considered the Huber loss, for example [ 11 ] , [ 26 ] , [ 24 ] . Except for the tuning parameter which needs to be calibrated from data, the Huber loss combines the advantages of the square and the absolute losses, and enjoys the minimax optimal property. In this contribution, we deal with the Shannon entropy for nonparametric regression from strongly mixing observations. Two estimators, the MEE-based NPDNN and the SPDNN predictors are proposed. For each of these estimators, the upper bounds of the expected excess risk are established over the classes of Hölder and composition Hölder functions. When the error is Gaussian, these estimators are optimal (up to a logarithmic factor) in the minimax sense. Unlike the least square method which takes into consideration only the first two moments of the error, the MEE criteria considered take into account the moments of all orders of the error variable in the

Read the complete text