Supplementary Material for

“Nonlinear Multiple Response Regression and Learning of Latent Spaces” · 2025

Read Supplementary Material for by “Nonlinear Multiple Response Regression and Learning of Latent Spaces” (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21608.txt.

Introduction

Ye Tian and co-authors present a method for finding simple hidden structure inside complex data, in work from 2025. Many datasets have far more measures than can be grasped at once. Classic tools compress data without using labels, even when labels exist. Neural methods can use labels but act as black boxes that are hard to explain.

The new method treats the task as predicting several outcomes at once through a hidden space. A classic math lemma lets them estimate that space without knowing the exact curves that link the parts. The method can be read as a curved extension of the classic compression tool. Tests on simulated and real data show strong results, with clearer meaning and lower computing cost than neural rivals.

Catalog record

CREATOR“Nonlinear Multiple Response Regression and Learning of Latent Spaces”
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE116 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-21608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Supplementary Material for “Nonlinear Multiple Response Regression and Learning of Latent Spaces” Abstract Nonlinear Multiple Response Regression and Learning of Latent Spaces Ye Tian, 1 1 1 School of Computing and Data Science, The University of Hong Kong, Pokfulam Road, HKSAR, China E-mail: [email protected] . Sanyou Wu, 2 2 2 School of Computing and Data Science, The University of Hong Kong, Pokfulam Road, HKSAR, China E-mail: [email protected] . and Long Feng 3 3 3 School of Computing and Data Science, The University of Hong Kong, Pokfulam Road, HKSAR, China E-mail: [email protected] . August 24, 2026 Abstract. Identifying low-dimensional latent structures within high-dimensional data has long been a central topic in the machine learning community, driven by the need for data compression, storage, transmission, and deeper data understanding. Traditional methods, such as principal component analysis (PCA) and autoencoders (AE), operate in an unsupervised manner, ignoring label information even when it is available. In this work, we introduce a unified method capable of learning latent spaces in both unsupervised and supervised settings. We formulate the problem as a nonlinear multiple-response regression within an index model context. By applying the generalized Stein’s lemma, the latent space can be estimated without knowing the nonlinear link functions. Our method can be viewed as a nonlinear generalization of PCA. Moreover, unlike AE and other neural network methods that operate as “black boxes”, our approach not only offers better interpretability but also reduces computational complexity while providing strong theoretical guarantees. Comprehensive numerical experiments and real data analyses demonstrate the superior performance of our method. Key words and phrases. PCA, autoencoder, index model, latent space, Stein’s Lemma, SVD. 1 Introduction Figure 1 : The data we consider could be unlabeled features only, or feature-label pairs, for example, CT and the corresponding diagnoses, or images of suspects and their positions on the surveillance screen, etc. The feature can be linearly embedded in a low-dimensional latent space, and the feature itself and possible labels can be generated from the embeddings through link functions. The goal of the work is to learn the latent space without knowing link functions. Identifying low-dimensional latent spaces in high-dimensional data has been a longstanding focus in the statistics and machine learning community. This interest is driven not only by the goal of better understanding the data but also by practical applications such as data compression, storage, and transmission. Principal Component Analysis (PCA, Pearson, 1901 ) is arguably the simplest and most popular dimension reduction method. PCA and its extensions, such as sparse PCA ( Hui Zou and Tibshirani, 2006 ) and robust PCA ( Candes et al., 2009 ) , have found widespread applications in various fields, including computer vision, biomedical analysis, and more. Autoencoders (AE, Vincent et al. 2008 ) can be viewed as a nonlinear generalization of PCA by replacing simple linear projections with complex nonlinear projections defined by neural networks. With the increasing prevalence of deep learning in recent years, AE and its extensions, such as contractive AE ( Rifai et al., 2011 ) and sparse AE ( Makhzani and Frey, 2013 ) , have proven effective in dimension reduction and representation learning. Despite their significant success, PCA and AE still encounter considerable challenges in learning the latent spaces for downstream tasks. Firstly, both PCA and AE learn the latent space in a purely unsupervised manner. Although the low-dimensional manifold that the high-dimensional features align with reflects their inherent properties, relying solely on this information can be suboptimal when labels are available. Leveraging label information can enhance the quality of the learned latent space, making it more suitable for corresponding downstream tasks. Additionally, PCA is limited to learning the latent space in a linear setting. While autoencoders extend PCA to a nonlinear context, their “black-box” nature not only lacks necessary interpretability but also imposes a significant computational burden. Rethinking the PCA, if we treat the input as the pseudo output, the data reconstruction problem can be reframed as a multiple response regression with reduced rank coefficients. Multiple response regression in a reduced rank setting has been extensively studied in the literature, although the majority of the work has focused on linear models. Related works include Yuan et al. (2007) ; Mukherjee and Zhu (2011) ; Chen et al. (2012) ; Lu et al. (2012) . With a reduced rank setup, dimension reduction can be achieved in multiple response regression, and the latent coefficient space can be learned. Beyond reduced rank regression, multiple response regression has also been considered under other settings, such as sparse matrix coefficients estimation ( Lee and Liu, 2012 ; Simon et al., 2013 ; Wang et al., 2013 ; Changliang Zou and Zhang, 2022 ) . However, as these approaches are limited to linear models, complex nonlinear relationships between predictors and responses can hardly be captured and their applicability in real-world data analyses can be significantly restricted. In the realm of nonlinear regression, the index model is a prominent statistical framework that extends linear regression to a nonlinear context. Specifically, consider the multi-index model (MIM), where the outcomes are modeled as \mathbb{E}(y|\boldsymbol{x})=f(\boldsymbol{B}^{\top}\boldsymbol{x}) . In this formulation, f represents an unknown nonlinear link function, and the matrix \boldsymbol{B} has a rank significantly smaller than the input dimension. Unlike other statistical methods that introduce nonlinearity, such as generalized linear models or spline regression, MIM inherently produces a latent embedding of the features through the introduction of \boldsymbol{B} . This characteristic is particularly aligned with our objective of estimating the latent space. Extensive research has been conducted on MIM, particularly on the estimation of the low-rank matrix \boldsymbol{B} . The primary challenge lies in the fact that, without prior knowledge of the link functions, traditional methods such as least squares or maximum likelihood estimators are not applicable for estimating \boldsymbol{B} . Nonetheless, the statistics community has introduced a plethora of innovative solutions over the years. For instance, in low-dimensional settings with fixed rank of \boldsymbol{B} , Friedman and Stuetzle (1981) introduced the projection pursuit regression, under which the link functions and low-rank matrix can be estimated through an alternating optimization process. A series of breakthroughs related to index models are proposed by Ker-Chau Li, including regression under link violation ( Li and Duan, 1989 ) , sliced inverse regression ( Li, 1991 ) , and principal Hessian directions ( Li, 1992 ) . In the high-dimensional setting where the rank of \boldsymbol{B} is allowed to grow with the sample size, the estimation of \boldsymbol{B} is considered under the sparse subspace assumption ( Chen et al., 2010 ; Tan et al., 2018 ) . More recently, a line of research focused on using Stein’s method in index models when the distribution of input is known. For example, Yang et al. (2017b) proposed to learn \boldsymbol{B} in a non-Gaussian MIM through a second-order Stein’s method. Unlike previous approaches, the Gaussian or elliptically symmetric assumption on the feature is not required in Yang et al. (2017b) , thus the application of Stein’s method in index models can be significantly broadened. Despite the success of these approaches, we shall note that most of the work concentrates on the scenario with univariate responses. To the best of our knowledge, the application of Stein’s method to nonlinear

Read the complete text