Soft Gradient Boosting with Learnable Feature Transforms for Sequential Regression
Huseyin Karaca, Suleyman Serdar Kozat · 2025
Read Soft Gradient Boosting with Learnable Feature Transforms for Sequential Regression by Huseyin Karaca, Suleyman Serdar Kozat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12920.txt.
Introduction
Huseyin Karaca and Suleyman Serdar Kozat propose soft gradient boosting with learnable feature transforms in a 2025 paper. Gradient boosting builds a strong predictor by adding many small decision trees in sequence, and it is widely used for regression tasks. In high-dimensional, data-scarce settings, choosing the right input features is crucial.
Their framework trains a soft decision tree at each boosting iteration while jointly learning a linear transform of the input features. That joint learning discovers the most relevant representations during boosting and optimizes feature selection and boosting end to end, avoiding overfitting. Tests on synthetic and real-world datasets showed effective and efficient performance gains. The algorithm extends to differentiable non-linear transforms when overfitting is not a concern, and code is shared publicly.
Catalog record
| CREATOR | Huseyin Karaca, Suleyman Serdar Kozat |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Soft Gradient Boosting with Learnable Feature Transforms for Sequential Regression Huseyin Karaca, Suleyman Serdar Kozat Abstract We propose a soft gradient boosting framework for sequential regression that embeds a learnable linear feature transform within the boosting procedure. At each boosting iteration, we train a soft decision tree and learn a linear input feature transform 𝐐 \mathbf{Q} together. This approach is particularly advantageous in high-dimensional, data-scarce scenarios, as it discovers the most relevant input representations while boosting. We demonstrate, using both synthetic and real-world datasets, that our method effectively and efficiently increases the performance by an end-to-end optimization of feature selection/transform and boosting while avoiding overfitting. We also extend our algorithm to differentiable non-linear transforms if overfitting is not a problem. To support reproducibility and future work, we share our code publicly. I Introduction In this letter, we study sequential (online) regression problem, which aims to regress the next value y_{T+1} of a series \{y_{t}\}_{t=1}^{T} (optionally conditioned on an exogenous features \{\mathbf{x}_{t}\}_{t=1}^{T} ) given all information up to time T . 1 1 1 We denote all vectors by boldface lowercase letters (e.g. \mathbf{u} ), and matrices by boldface capital letters (e.g. \mathbf{U} ). For a vector \mathbf{u} (or matrix \mathbf{U} ), \mathbf{u}^{\top} ( \mathbf{U}^{\top} ) is the transpose. The time index is indicated by a subscript; for instance, \mathbf{u}_{t} is a vector at time t . \mathbf{I}_{d}\in\mathbb{R}^{d\times d} is the d -dimensional identity matrix. This problem models a wide range of applications, such as energy-load forecasting, IoT and traffic management [ 1 , 2 ] , and has been studied extensively in the signal processing literature. Gradient‑boosted decision‑tree models (e.g. XGBoost [ 3 ] , LightGBM [ 4 ] ) are the current state of the art as demonstrated in different real life competitions [ 5 , 6 ] . Yet they struggle when the feature dimension is large relative to the amount of available training data—a frequent situation that occurs in practice (e.g. short sales histories paired with hundreds of engineered covariates) [ 7 , 8 ] . Exhaustive subset or interaction searches of those covariates are usually restrictive, making them impractical in data‑scarce, high‑dimensional scenarios [ 9 , 10 ] . As a solution, we propose a boosting framework that overcomes this bottleneck by inserting a learnable linear feature transform between successive weak (base) learners in gradient-boosting. We concentrate on linear transforms to avoid overfitting; however, we also extend our algorithm to differentiable nonlinear transforms, with or without memory (e.g., neural networks, RNNs, and similar), or to standard boosting machines using hard decision trees (e.g., vanilla LightGBM, XGBoost, CatBoost). Specifically, we 1. replace hard trees with soft (differentiable) decision trees, enabling end-to-end optimization, 2. learn a transform matrix \mathbf{Q} to perform embedded feature selection/transformation at every boosting round, which is jointly optimized with the newly fitted tree, 3. demonstrate that this framework can be readily extended to nonlinear feature transformations, 4. openly share our code for replicability of our results and encourage further research. 2 2 2 https://github.com/huseyin-karaca/bsdtq There are also existing studies in literature that combine soft decision trees with gradient boosting [ 11 ] , and others integrating feature selection concepts into boosting methods [ 12 , 13 ] . The method introduced in [ 12 ] rely only on features specific to certain applications such as HOG features used in computer vision, which limits its general applicability. The other related work [ 13 ] combines traditional gradient-boosted decision trees with feature selection. Their method works effectively when the dataset has many samples compared to the number of features. However, in many real-world problems, especially in financial forecasting, we may frequently face the opposite case: few data points relative to many features [ 14 ] . Hence, we introduce an end-to-end optimized method for model training, especially for situations with many features and limited data. Note that by this basic transformation, we avoid both overfitting and remedy the need for feature selection and/or transformation, where the nonlinear interactions are discovered by the boosted decision trees. We demonstrate through experiments—both with synthetic and real-world data—that our method significantly increases the performance of boosted trees while introducing little computational overhead in scenarios where data is scarce and feature space is large. II Problem Definition II-A Sequential Regression We sequentially observe \{y_{t}\}_{t\geq 1},\,y_{t}\in\mathbb{R} along with the feature vectors \{\mathbf{x}_{t}\}_{t\geq 1},\,\mathbf{x}_{t}\in\mathbb{R}^{d} . Our goal is to predict y_{t+1} at each t in an online manner by constructing a function that depends only on the past information: \hat{y}_{t+1}\;=\;f_{t}(\{\ldots,y_{t-1},y_{t}\},\{\ldots,\mathbf{x}_{t-1},\mathbf{x}_{t}\}). At each time step t , we output \hat{y}_{t+1} and suffer a loss \mathcal{L}\bigl(y_{t+1},\hat{y}_{t+1}\bigr) . The loss function \mathcal{L} can be any differentiable function such as the squared error loss (y_{t+1}-\hat{y}_{t+1})^{2} . We aim to minimize the accumulated online error over time: \mathcal{L}_{T}\;=\;\sum_{t=1}^{T}\mathcal{L}\bigl(y_{t},\;\hat{y}_{t}\bigr). Note that this main framework can be readily extended to different horizons, e.g., predicting y_{t+2},..,y_{t+n} , either using direct or recursive approaches [ 15 ] . Input: \mathbf{X}_{\text{train}}\in\mathbb{R}^{n\times d} , targets \mathbf{y}\in\mathbb{R}^{n} , number of boosting rounds K Output: \{(\mathbf{Q}_{k},\text{SDT}_{k})\}_{k=1}^{K} , where \text{SDT}_{k} is the base Soft Decision Tree at k’th step, \mathbf{Q}_{k} is the feature transform matrix Initialize ensemble prediction \hat{\mathbf{y}}^{(0)}\leftarrow\mathbf{0} , linear transform \mathbf{Q}_{0}\leftarrow\mathbf{I}_{d} ; for k\leftarrow 1 to K do Residual \mathbf{r}_{k}\leftarrow\mathbf{y}-\hat{\mathbf{y}}_{k-1} ; Fit soft decision tree \text{SDT}_{k} on \bigl(\mathbf{Q}_{k-1}\mathbf{X}_{\text{train}},\,\mathbf{r}_{k}\bigr) ; Find \mathbf{Q}_{k} (for a number of gradient-descent steps w.r.t. loss of \text{SDT}_{k} ); \hat{\mathbf{y}}^{(k)}\leftarrow\hat{\mathbf{y}}^{(k-1)}+\text{SDT}_{k}\!\bigl(\mathbf{Q}_{k}\mathbf{X}_{\text{train}}\bigr) ; end for return \{(\mathbf{Q}_{k},\text{SDT}_{k})\}_{k=1}^{K} Algorithm 1 Boosted Soft Decision Trees with feature transform \mathbf{Q} ( BSDT-Q ) II-B Hard Decision Trees A hard decision tree (HDT) f(\mathbf{z}) can be written as a sum of leaf values multiplied by indicator functions f(\mathbf{z})\;=\;\sum_{n=1}^{N}\,\gamma_{n}\;\mathbf{1}\bigl(\mathbf{z}\in P_{n}\bigr), where P_{n} denotes the region (leaf) in the input space associated with the n -th leaf, and \mathbf{1}(\cdot) is the indicator function that returns 1 if \mathbf{z} belongs to P_{n} and 0 otherwise. Hard decision trees use indicator-based splits (e.g., if z_{j}<\text{threshold} , go left; else go right). These splits are not differentiable with respect to the input \mathbf{z} , hence there is no straightforward way to compute \nabla_{\mathbf{Q}}f(\mathbf{Qx}) . This makes gradient-based optimization of \mathbf{Q} challenging. Nevertheless, in many real-world applications (including sequential regression), hard decision trees are crucial due to their interpretability and strong performance, especially when they are used in a gradient-boosting framework. II-C Soft Decision Trees Soft decision trees (SDT) relax the hard splits of hard decision trees into differentiable “soft” gating functions, often using a logistic \sigma(\cdot) at each node. Let \ell index a leaf in the set of leaves \boldsymbol{\ell} , and let \text{path}(\ell) denote the set of node indices along the path from the root to leaf \ell . Then each node i outputs a probability p_{i}(\mathbf{z})=\sigma\bigl(\mathbf{w}_{i}^{\top}\mathbf{z}+b_{i}\bigr) . Depending on whether the node “direction” v_{i,\ell} is 1 (go right) or 0 (go left), the contribution to the final leaf value is p_{i}(\mathbf{z}) or 1-p_{i}(\mathbf{z}) , respectively. Thus: p_{\ell}^{*}\;=\;\prod_{i\in\text{path}(\ell)}\bigl[p_{i}(\mathbf{z})^{\,v_{i,\ell}}\,\bigl(1-p_{i}(\mathbf{z})\bigr)^{\,1-v_{i,\ell}}\bigr]. (1) The overall tree output is then f(\mathbf{z})\;=\;\sum_{\ell\,\in\,\boldsymbol{\ell}}\;p_{\ell}^{*}(\mathbf{z})\;\gamma_{\ell}, where \gamma_{\ell} is the value assigned to the leaf \ell . Due to \sigma(\cdot) , f(\mathbf{z}) is differentiable with respect to \mathbf{z} , and hence with respect to \mathbf{Q} . We calculate a gradient-descent update to \mathbf{Q} , using differentiability of soft trees in Section III . II-D Gradient Boosting Machines Gradient Boosting Machines [ 16