xRFM: Accurate, scalable, and interpretable feature learning models for tabular data
Daniel Beaglehole David Holzmüller Computer Science and Engineering INRIA Paris Halıcıoğlu Data Science Institute Ecole Normale Supérieure UC San Diego PSL University [email protected] david.holzmuller at inria.fr Adityanarayanan Radhakrishnan Mikhail Belkin MIT Mathematics Halıcıoğlu Data Scienc · 2025
Read xRFM: Accurate, scalable, and interpretable feature learning models for tabular data by Daniel Beaglehole David Holzmüller Computer Science and Engineering INRIA Paris Halıcıoğlu Data Science Institute Ecole Normale Supérieure UC San Diego PSL University [email protected] david.holzmuller at inria.fr Adityanarayanan Radhakrishnan Mikhail Belkin MIT Mathematics Halıcıoğlu Data Scienc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10053.txt.
Introduction
Daniel Beaglehole and co-authors challenge the tree ensembles that have ruled tabular data for a decade, with a kernel method that learns its own features. Their 2025 paper introduces xRFM, which grafts feature-learning kernel machines onto a tree structure so the model adapts to local data geometry and scales to very large training sets.
Across one hundred regression datasets it beat thirty-one rivals, including boosted trees and recent tabular foundation models. On two hundred classification sets it was competitive with the best and ahead of the boosted trees. It also explains itself natively through a gradient-based importance measure, a rare gift among top performers. Tabular data runs banking, medicine, and industry while research chases text and images; this paper is a reminder that the spreadsheet frontier still moves. A scalable kernel revival, with receipts, is news in that world.
Catalog record
| CREATOR | Daniel Beaglehole David Holzmüller Computer Science and Engineering INRIA Paris Halıcıoğlu Data Science Institute Ecole Normale Supérieure UC San Diego PSL University [email protected] david.holzmuller at inria.fr Adityanarayanan Radhakrishnan Mikhail Belkin MIT Mathematics Halıcıoğlu Data Scienc |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10053
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
xRFM: Accurate, scalable, and interpretable feature learning models for tabular data Daniel Beaglehole David Holzmüller Computer Science and Engineering INRIA Paris Halıcıoğlu Data Science Institute Ecole Normale Supérieure UC San Diego PSL University [email protected] david.holzmuller at inria.fr Adityanarayanan Radhakrishnan Mikhail Belkin MIT Mathematics Halıcıoğlu Data Science Institute Broad Institute of MIT and Harvard UC San Diego [email protected] [email protected] Abstract Inference from tabular data, collections of continuous and categorical variables organized into matrices, is a foundation for modern technology and science. Yet, in contrast to the explosive changes in the rest of AI, the best practice for these predictive tasks has been relatively unchanged and is still primarily based on variations of Gradient Boosted Decision Trees (GBDTs). Very recently, there has been renewed interest in developing state-of-the-art methods for tabular data based on recent developments in neural networks and feature learning methods. In this work, we introduce xRFM, an algorithm that combines feature learning kernel machines with a tree structure to both adapt to the local structure of the data and scale to essentially unlimited amounts of training data. We show that compared to 31 31 other methods, including recently introduced tabular foundation models (TabPFNv2) and GBDTs, xRFM achieves best performance across 100 100 regression datasets and is competitive to the best methods across 200 200 classification datasets outperforming GBDTs. Additionally, xRFM provides interpretability natively through the Average Gradient Outer Product. Code for xRFM (following a scikit-learn-style API) is available at: https://github.com/dmbeaglehole/xRFM . 1 Introduction Tabular data – collections of continuous and categorical variables organized into matrices – underlies all aspects of modern commerce and science from airplane engines to biology labs to bagel shops. Yet, while Machine Learning and AI for language and vision have seen unprecedented progress, the primary methodologies of prediction from tabular data have been relatively static, dominated by variations of Gradient Boosted Decision Trees (GBDTs), such as XGBoost [ 7 ] . Nevertheless, hundreds of tabular datasets have been assembled to form extensive regression and classification benchmarks [ 35 , 16 , 12 , 37 , 11 ] , and, recently, there has been renewed interest in building state-of-the-art predictive models for tabular data [ 18 , 19 , 15 ] . Notably, given the remarkable effectiveness of large, “foundation” models for text, there has been much excitement in developing similar models on tabular data, and recent effort has led to the development of TabPFN-v2, a foundation model for tabular data appearing in Nature [ 18 ] . Yet, despite this progress, tabular data still remains an active area for model development and building scalable, effective, and interpretable machine learning models in this domain remains an open challenge. In this work, we introduce xRFM, a tabular predictive model that combines recent advances in feature learning kernel machines with an adaptive tree structure, making it effective, scalable, and interpretable. xRFM builds upon the Recursive Feature Machine (RFM) algorithm from [ 27 ] , which enabled feature learning (a form of supervised dimensionality reduction) in general machine learning models. xRFM works as follows – given training data, it first builds a binary tree structure to split data into subsets based on features relevant for prediction within each split. When splits reach a certain size (of \leq C data samples), we train a leaf RFM (a hyper-parameter and compute optimized version of RFM from [ 28 ] ). There are two important consequences of using a tree structure in conjunction with leaf RFMs. The first is that it enables local feature learning , meaning that xRFM can learn different features for different subsets of the data. This property is crucial for tackling hierarchical structure in tabular data (for example, one subset of features may become relevant when a specific feature takes on high value while another subset of features is relevant when the same feature takes low values). The second consequence is that it enables xRFM to scale log-linearly in the number of samples ( O(n\log n) given n samples) during training and logarithmically ( O(\log n) ) during inference. As such, the inference time of xRFM is comparable to that of tree-based models. In practice, we show xRFM has the best performance across 100 tabular regression tasks and is competitive with state-of-the-art on 200 tabular classification tasks from the TALENT benchmark [ 37 ] . In particular, xRFM outperforms 31 other methods on these regression tasks including the GBDT and neural network variations mentioned in the opening paragraph. We show that xRFM achieves similar results on the largest datasets from the meta-test benchmark, where directly solving standard kernel machines becomes intractable on standard GPUs [ 19 ] . An additional benefit of xRFM is that it natively provides interpretability by exposing features learned and used for prediction. In particular, each leaf RFM learns features through a mathematical object known as the Average Gradient Outer Product (AGOP), whose diagonal indicates coordinates relevant for prediction and whose top eigenvectors indicate directions in data most relevant for prediction. Using examples of xRFM trained on synthetic and real data from TALENT, we show how AGOP matrices at each leaf RFM shed light on features relevant for prediction, including cases when leaf RFMs learn different features for different splits of data. To summarize, xRFM combines feature learning kernel methods with adaptive data partitioning using the AGOP. It is a fast, effective, and interpretable model on tabular data at all scales. 2 Preliminaries We begin with a review of kernel machines and kernel-RFM, which we use to build xRFM. 2.1 Kernel machines A kernel machine is a non-parametric machine learning model [ 31 , 3 ] . The idea behind kernel machines is that one can train a nonlinear predictive model by first transforming input data with a fixed, nonlinear feature map and then performing linear regression on the transformed data. Kernel machines make this procedure computationally tractable even for infinite dimensional feature maps by using kernel functions (inner products of feature mapped data). We describe kernel machines in the context of supervised learning below. Let X\in\mathbb{R}^{n\times d} denote training inputs with {x^{(i)}}^{T} denoting the example in the i th row of X for i\in[n] and y\in\mathbb{R}^{n\times c} denote training labels. Let K:\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R} denote a kernel function (a positive semi-definite, symmetric function). Given a regularization parameter \lambda\geq 0 , a kernel machine trained on the data (X,y) is a predictor, \widehat{f}:\mathbb{R}^{d}\to\mathbb{R}^{c} , of the form: \displaystyle\widehat{f}(x)=K(x,X)\alpha\qquad;\qquad\alpha=(K(X,X)+\lambda I)^{-1}y~; (1) where the notation K(x,X)\in\mathbb{R}^{1\times n} denotes an n -dimensional row vector with K(x,X)_{i}=K(x,x^{(i)}) and K(X,X)\in\mathbb{R}^{n\times n} denotes a matrix with K(X,X)_{ij}=K(x^{(i)},x^{(j)}) . Examples of kernel functions used in practice are the Gaussian kernel ( K(x,z)=\exp(-\|x-z\|_{2}^{2})/L^{2} ) or the Laplace kernel ( K(x,z)=\exp(-\|x-z\|_{2})/L ). The advantage of kernel functions is that the predictor admits a closed form solution, which can be computationally efficient to fit on datasets under 100 k samples. The limitations of kernel machines are twofold: (1) the formulation in Eq. ( 1 ) does not scale well (it is super-quadratic in the number of samples) and (2) the choice of kernel is fixed independently of the input data. The former limitation is addressed by recent, pre-conditioned gradient descent methods for solving for \alpha in Eq. ( 1 ) and methods that construct low-dimensional approximations