Random Forest Autoencoders for Guided Representation Learning

Adrien Aumon, Shuang Ni, Myriam Lizotte, Guy Wolf, Kevin R. Moon, Jake S. Rhodes · 2025

Read Random Forest Autoencoders for Guided Representation Learning by Adrien Aumon, Shuang Ni, Myriam Lizotte, Guy Wolf, Kevin R. Moon, Jake S. Rhodes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13257.txt.

Introduction

Adrien Aumon and co-authors published a research paper in 2025 on a neural network that learns compact maps of data under the guidance of expert labels. Plain map-making tools ignore labels, while most label-aware tools aim at sorting, not display. A recent method used random forests to shape good maps, but it could not place new points without redoing the work.

Their network borrows the forest's sense of which points belong together, then learns a direct rule for placing any point. It beat the older method's add-on rule in both accuracy and clarity of the maps. It also stayed steady across settings and can sit atop other map methods. That makes label-guided display practical for large, growing data sets.

Catalog record

CREATORAdrien Aumon, Shuang Ni, Myriam Lizotte, Guy Wolf, Kevin R. Moon, Jake S. Rhodes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Random Forest Autoencoders for Guided Representation Learning Adrien Aumon, Shuang Ni, Myriam Lizotte, Guy Wolf, Kevin R. Moon, Jake S. Rhodes Abstract Extensive research has produced robust methods for unsupervised data visualization. Yet supervised visualization—where expert labels guide representations—remains underexplored, as most supervised approaches prioritize classification over visualization. Recently, RF-PHATE, a diffusion-based manifold learning method leveraging random forests and information geometry, marked significant progress in supervised visualization. However, its lack of an explicit mapping function limits scalability and its application to unseen data, posing challenges for large datasets and label-scarce scenarios. To overcome these limitations, we introduce Random Forest Autoencoders (RF-AE), a neural network-based framework for out-of-sample kernel extension that combines the flexibility of autoencoders with the supervised learning strengths of random forests and the geometry captured by RF-PHATE. RF-AE enables efficient out-of-sample supervised visualization and outperforms existing methods, including RF-PHATE’s standard kernel extension, in both accuracy and interpretability. Additionally, RF-AE is robust to the choice of hyperparameters and generalizes to any kernel-based dimensionality reduction method. 1 Introduction Manifold learning-based visualization methods, such as t -SNE [ 1 ] , UMAP [ 2 ] , and PHATE [ 3 ] , are essential for exploring high-dimensional data by revealing patterns, clusters, and outliers through low-dimensional embeddings. While these methods excel at uncovering dominant data structures, they often fail to capture task-specific insights when auxiliary labels or metadata are available. Supervised approaches like RF-PHATE [ 4 ] bridge this gap by integrating label information into the kernel function through Random Forest-derived proximities [ 5 ] , generating representations that align with domain-specific objectives without introducing the exaggerated separations or distortions seen in class-conditional methods [ 6 ] . Specifically, RF-PHATE has provided critical insights in biology, such as identifying multiple sclerosis subtypes, demonstrating antioxidant effects in lung cancer cells, and aligning COVID-19 antibody profiles with patient outcomes [ 4 ] . However, most supervised and unsupervised manifold learning algorithms generate fixed coordinates within a latent space but lack a mechanism to accommodate new observations. Therefore, to embed previously unseen data, the algorithm must rerun with the new data as part of the training set. One well-known solution to this lack of out-of-sample (OOS) support is the Nyström extension [ 7 ] and its variants, such as linear reconstruction [ 8 ] or geometric harmonics [ 9 ] . These methods derive an empirical mapping function for new points defined as the linear combination of precomputed training embeddings, weighted with a kernel function expressing the similarities between the new point and the points in the training set. However, these methods are restricted to linear kernel mappings and are based on an unconstrained least-squares minimization problem [ 7 ] , which makes them sensitive to the quality of the training set and prone to failing in accurately recovering the true manifold structure [ 10 , 11 ] . As an alternative extension method, recent neural network-based approaches, such as Geometry-Regularized Autoencoders (GRAE) [ 12 , 13 ] , offer a promising solution for extending embeddings to OOS data points. In this study, we present Random Forest Autoencoders (RF-AE), an autoencoder (AE) architecture that addresses OOS extension, while taking inspiration from the principles of GRAE [ 13 ] , which uses a manifold embedding to regularize the bottleneck layer. Instead of reconstructing the original input vectors, RF-AE incorporates supervised information by reconstructing Random Forest- Geometry- and Accuracy-Preserving (RF-GAP) proximities [ 5 ] , enabling us to learn an embedding function extendable to new data in a supervised context. Furthermore, we introduce a proximity-based prototype selection approach to reduce time and space complexities during training and inference without compromising embedding quality. Notably, our approach extends OOS examples without relying on label information for new points, which is helpful in situations where label information is expensive or scarce. We empirically show that RF-AE outperforms existing approaches in embedding new data while preserving the local and global structure of the important features for the underlying classification task. RF-AE improves the adaptability and scalability of the manifold learning process, allowing for seamless integration of new data points while maintaining the desirable traits of established embedding methods. 2 Related work 2.1 Parametric embedding through multi-task autoencoders Given a high-dimensional training dataset X=\{\mathbf{x}_{i}\in\mathbb{R}^{D}\mid i=1,\ldots,N\} —where X can represent tabular data, images, or other modalities—a manifold learning algorithm can be extended to test data by training a neural network, typically a multi-layer perceptron (MLP) to regress onto precomputed non-parametric embeddings \mathbf{z}_{i}^{G} , or by means of a cost function underlying the manifold learning algorithm, as in parametric t -SNE [ 14 ] and parametric UMAP [ 15 ] . However, solely training an MLP for this supervised task often leads to an under-constrained problem, resulting in solutions that memorize the data but fail to capture meaningful patterns or generalize effectively [ 16 , 17 ] . Beyond implicit regularization techniques such as early stopping [ 18 ] , dropout [ 19 ] , or layer-wise pre-training [ 20 ] , multi-task learning [ 21 ] has been shown to improve generalization. Early studies demonstrated that jointly learning tasks reduces the number of required samples [ 22 , 23 ] , while later work introduced trace-norm regularization on the weights of a linear, single hidden-layer neural network for a set of tasks [ 24 , 25 ] . Motivated by Le et al. [ 26 ] , who empirically showed that training neural networks to predict both target embeddings and inputs (reconstruction) improves generalization compared to encoder-only architectures, we focus on multi-task learning-based regularization in the context of regularized autoencoders. AE networks are built of two parts: an encoder function f(\mathbf{x}_{i})=\mathbf{z}_{i}\in\mathbb{R}^{d} ( d\ll D ), which compresses the input data into a latent representation via a bottleneck layer [ 27 ] , and a decoder function g(\mathbf{z}_{i})=\hat{\mathbf{x}}_{i} , which maps the low-dimensional embedding back to the original input space. To ensure that (g\circ f)(\mathbf{x}_{i})=\hat{\mathbf{x}}_{i}\approx\mathbf{x}_{i} , AEs minimize the average reconstruction loss L(f,g)=\frac{1}{N}\sum_{i=1}^{N}L_{recon}(\mathbf{x}_{i},\hat{\mathbf{x}}_{i}) where L_{recon}(\cdot,\cdot) is typically defined as the squared Euclidean distance. AEs learn compact data representations that relate meaningfully to the input data. However, standard AEs often fail to capture the intrinsic geometry of the data and do not produce interpretable embeddings [ 13 ] . To address this, regularization techniques have been proposed to guide the optimization process toward better local minima in the latent space. Structural Deep Network Embedding (SDNE) [ 28 ] preserves both first- and second-order graph neighborhoods for graph-structured data by combining adjacency vector reconstruction with Laplacian Eigenmaps [ 29 ] regularization. Local and Global Graph Embedding Autoencoders [ 30 ] enforce two constraints on the embedding layer: a local constraint to cluster k -nearest neighbors and a global constraint to align data points with their class centers. VAE-SNE [ 31 ] integrates parametric t -SNE with variational AEs, enhancing global structure preservation while retaining t -SNE’s strength in preserving local structure. GRAE [ 13 ] explicitly impose geometric consistency between the latent space and precomputed manifold embeddings. Other approaches focus on regularizing the decoder. Inspired by Denoising Autoencoders [ 32 ] , Generalized Autoencoders [ 33 ] minimize the weighted mean squared error between the reconstruction \hat{\mathbf{x}} and the k -nearest neighbors of the input

Read the complete text