Neural Networks Perform Sufficient Dimension Reduction

Shuntuo Xu, Zhou Yu · 2024

Read Neural Networks Perform Sufficient Dimension Reduction by Shuntuo Xu, Zhou Yu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19033.txt.

Introduction

Shuntuo Xu and Zhou Yu prove that neural networks quietly perform a classic statistical compression in regression, in a 2024 paper. Sufficient dimension reduction seeks the few directions in the data that carry all the information about the outcome. The authors show that, with the right rank limits on training, a network's first layer spans exactly the central mean subspace, the target of that compression. They also prove the network-based estimate is statistically consistent.

Numerical experiments back the theory and compare favorably with established reduction methods. An extension reaches beyond means to the full central subspace. The authors frame the result as an explanation for why networks handle high-dimensional regression so well: they find the low-dimensional signal on their own. That lens, they suggest, can guide both theory and practice.

Catalog record

CREATORShuntuo Xu, Zhou Yu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19033

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Networks Perform Sufficient Dimension Reduction Shuntuo Xu, Zhou Yu Abstract This paper investigates the connection between neural networks and sufficient dimension reduction (SDR), demonstrating that neural networks inherently perform SDR in regression tasks under appropriate rank regularizations. Specifically, the weights in the first layer span the central mean subspace. We establish the statistical consistency of the neural network-based estimator for the central mean subspace, underscoring the suitability of neural networks in addressing SDR-related challenges. Numerical experiments further validate our theoretical findings, and highlight the underlying capability of neural networks to facilitate SDR compared to the existing methods. Additionally, we discuss an extension to unravel the central subspace, broadening the scope of our investigation. Key Laboratory of Advanced Theory and Application in Statistics and Data Science – MOE School of Statistics, East China Normal University, Shanghai, China [email protected], [email protected] Introduction Neural networks have achieved significant success in a tremendous variety of applications ( Lee and Abu-El-Haija 2017 ; Silver et al. 2018 ; Jumper et al. 2021 ; Brandes et al. 2022 ; Thirunavukarasu et al. 2023 ) . As the most essential foundation, a feedforward neural network is typically constructed by a series of linear transformations and nonlinear activations. To be specific, a function f implemented by a feedforward neural network with L layers can be represented as f(x)=\phi_{L}\circ\sigma_{L-1}\circ\phi_{L-1}\circ\cdots\circ\sigma_{0}\circ\phi_{0}(x). (1) Here, \circ is the functional composition operator, \phi_{i}(z)=W_{i}^{\top}z+b_{i} denotes the linear transformation and \sigma_{i} represents the elementwise nonlinear activation function. Despite a clear architecture, the formula ( 1 ) provides limited insight into how the information within the input data is processed by the neural networks. To comprehensively understand how neural networks retrieve task-relevant information, there have been extensive efforts towards unraveling their interpretability ( Doshi-Velez and Kim 2017 ; Guidotti et al. 2018 ; Zhang et al. 2021 ) . For instance, post-hoc interpretability ( Lipton 2018 ) focused on the predictions generated by neural networks, while disregarding the detailed mechanism and feature importance. Ghorbani, Abid, and Zou (2019) highlighted the fragility of this type of interpretation, as indistinguishable perturbations could result in completely different interpretations. For the sake of striking a balance between the complexity of representation and the power of prediction, Tishby and Zaslavsky (2015) and Saxe et al. (2019) pioneered the interpretability of deep neural networks via the information bottleneck theory. Ghosh (2022) further linked the information bottleneck to sufficient dimension reduction (SDR) ( Li 1991 ; Cook 1996 ) , which is a rapidly developing research field in regression diagnostics, data visualization, pattern recognition and etc. In this paper, we provide a theoretical understanding of neural networks for representation learning from the SDR perspective. Let x\in\mathbb{R}^{p} represent the covariates and y\in\mathbb{R} represent the response. Consider the following regression model y=f_{0}(B_{0}^{\top}x)+\epsilon, (2) where B_{0}\in\mathbb{R}^{p\times d} is a nonrandom matrix with d\leq p , f_{0}:\mathbb{R}^{d}\to\mathbb{R} is an unknown function, and \epsilon is the noise such that \mathbb{E}(\epsilon|x)=0 and \mathrm{Var}(\epsilon|x)=\nu^{2} for some positive constant \nu . Intuitively, the semiparametric and potentially multi-index model ( 2 ) asserts that the core information for regression is encoded in the low-dimensional linear representation B_{0}^{\top}x . In the literature of SDR, model ( 2 ) has been extensively studied. Based on model ( 2 ) , Cook and Li (2002) proposed the objective of sufficient mean dimension reduction as y\perp\!\!\!\perp\mathbb{E}(y|x)|B^{\top}x (3) for some matrix B , where \perp\!\!\!\perp means statistical independence. Denote the column space spanned by B as \Pi_{B} , which is commonly referred to as the mean dimension-reduction subspace. It is evident that the matrix B satisfying ( 3 ) is far away from uniqueness. Hence, we focus on the intersection of all possible mean dimension-reduction subspace, which is itself a mean dimension-reduction subspace under mild conditions (e.g., when the domain of x is open and convex; see Cook and Li (2002) ), and term it the central mean subspace. Agreeing to condition ( 3 ), B_{0} defined in model ( 2 ) yields the central mean subspace, denoted as \Pi_{B_{0}} , under certain assumption. Statistical estimation and inference about the central mean subspace is the primary goal of SDR. Popular statistical methods for recovering central mean subspace include ordinary least squares ( Li and Duan 1989 ) , principal Hessian directions ( Li 1992 ) , minimum average variance estimation ( Xia et al. 2002 ) , semiparametric approach ( Ma and Zhu 2013 ) , generalized kernel-based dimension reduction ( Fukumizu and Leng 2014 ) , and many others. Although numerous studies have demonstrated the ability of neural networks to approximate complex functions ( Hornik, Stinchcombe, and White 1989 ; Barron 1993 ; Yarotsky 2017 ; Shen, Yang, and Zhang 2021 ) and to adapt low-dimensional structures ( Bach 2017 ; Bauer and Kohler 2019 ; Schmidt-Hieber 2020 ; Abbe, Adsera, and Misiakiewicz 2022 ; Jiao et al. 2023 ; Troiani et al. 2024 ) , it is of subsequent interest to investigate whether neural networks are capable of correctly identifying the intrinsic structure encapsulated in B_{0} , thereby deepening the interpretation of neural networks. Our study was inspired by an observation that the weight matrix in the first layer, i.e., W_{1} , could accurately detect the presence of B_{0} in a toy data set, with the rank regularization. Specifically, consider the toy model y=(B_{0}^{\top}x)^{3}+\epsilon where x\sim\mathrm{Uniform}([0,1]^{10}) , B_{0}=(1,-2,0,\ldots,0)^{\top} and \epsilon\sim\mathrm{Normal}(0,0.1^{2}) . We trained neural networks using the least squares loss with W_{1}=W_{11}W_{12} where W_{11}\in\mathbb{R}^{10\times q} and W_{12}\in\mathbb{R}^{q\times 64} for q=1,\ldots,10 . Evidently, the rank of W_{1} did not exceed q . It was then observed that for each q , (i) B_{0} was closely contained within the column space of W_{11} , as the absolute cosine similarity between B_{0} and its projection on \Pi_{W_{11}} was close to 1, and (ii) the leading eigenvector of W_{1}W_{1}^{\top} closely aligned with B_{0} (see Figure 1 ). This observation indicates that the first layer of the neural network may potentially discover the underlying low-dimensional intrinsic structure. Figure 1: Absolute cosine similarity between (i) B_{0} and its projection on \Pi_{W_{11}} (the dot line with triangle marks), (ii) B_{0} and the leading eigenvector of W_{1}W_{1}^{\top} (the solid line with square marks). It is important to note that the application of neural networks for estimating the central mean subspace based on model ( 2 ) was previously explored by Kapla, Fertl, and Bura (2022) through a two-stage method. The first stage focused on obtaining a preliminary estimation of \Pi_{B_{0}} , which subsequently served as an initial point for the joint estimation of \Pi_{B_{0}} and f_{0} in the second stage. Given our toy example, however, it is prudent to critically evaluate the necessity of the first stage. Furthermore, their work lacks a comprehensive theoretical guarantee. Additionally, another related work conducted by Liang, Sun, and Liang (2022) concentrated on seeking nonlinear sufficient representations. In contrast, we focus more on revealing the fundamental nature of neural networks. We in this paper show that, with suitable rank regularization, the first layer of a feedforward neural network conducts SDR in a regression task, wherein d(W_{11},B_{0})\to 0 in probability for certain distance metric d(\cdot,\cdot) . Furthermore, numerical experiments provide empirical evidences supporting this result, while demonstrating the efficiency of neural networks in addressing the issue of SDR. Throughout this

Read the complete text