SplitWise Regression: Stepwise Modeling with Adaptive Dummy Encoding

Marcell T. Kurbucz, Nikolaos Tzivanakis, Nilufer Sari Aslam, Adam M. Sykulski · 2025

Read SplitWise Regression: Stepwise Modeling with Adaptive Dummy Encoding by Marcell T. Kurbucz, Nikolaos Tzivanakis, Nilufer Sari Aslam, Adam M. Sykulski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15423.txt.

Introduction

Marcell Kurbucz and co-authors present SplitWise, a way to let plain regression models bend without losing their clarity. Step-by-step regression is easy to read but only fits straight lines unless the analyst reshapes the inputs by hand. SplitWise uses small decision trees to cut a numeric input at learned thresholds and turn it into yes-no features, keeping a cut only when a standard score says the model truly improves.

The method ships as an R package and was tested on both made-up and real data. It repeatedly produced smaller, clearer models that held up better on new data than standard stepwise methods and penalized rivals. The authors pitch it for fields that need flexible fits they can still explain.

Catalog record

CREATORMarcell T. Kurbucz, Nikolaos Tzivanakis, Nilufer Sari Aslam, Adam M. Sykulski
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15423

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SplitWise Regression: Stepwise Modeling with Adaptive Dummy Encoding Marcell T. Kurbucz, Nikolaos Tzivanakis, Nilufer Sari Aslam, Adam M. Sykulski Abstract Capturing nonlinear relationships without sacrificing interpretability remains a persistent challenge in regression modeling. We introduce SplitWise, a novel framework that enhances stepwise regression. It adaptively transforms numeric predictors into threshold-based binary features using shallow decision trees, but only when such transformations improve model fit, as assessed by the Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC). This approach preserves the transparency of linear models while flexibly capturing nonlinear effects. Implemented as a user-friendly R package, SplitWise is evaluated on both synthetic and real-world datasets. The results show that it consistently produces more parsimonious and generalizable models than traditional stepwise and penalized regression techniques. keywords Stepwise regression, Interpretable modeling, Dummy variables, Threshold effects, Model selection, Software 1 Introduction Statistical models that balance predictive performance with interpretability are increasingly valued across a range of disciplines. Traditional linear regression remains a popular choice due to its transparency: each coefficient directly quantifies the influence of a predictor on the outcome. However, its simplicity comes with limitations, particularly in its inability to capture nonlinear or threshold-based effects. By contrast, machine learning methods such as decision trees and neural networks can model complex relationships but often sacrifice interpretability in the process ( Breiman, 2001 ; Lipton, 2018 ) . In high-stakes domains such as healthcare, finance, and public policy, interpretability is not merely desirable—it is often essential for stakeholder trust and regulatory compliance ( Caruana et al., 2015 ; Doshi-Velez and Kim, 2017 ) . This has driven interest in methods that aim to improve model flexibility while retaining interpretability ( Rudin, 2019 ) . Stepwise regression ( Efroymson, 1960 ) , a longstanding technique for variable selection in linear models, systematically analyses various combinations of variables, adding or removing them based on their statistical importance ( Vàliaho, 1969 ; Vàliaho and Pekkonen, 1976 ) . This methodology typically employs three primary strategies: forward selection, which begins with an empty model and sequentially adds the most statistically significant variables; backward elimination, which starts with all candidate variables and progressively removes the least significant ones; and bidirectional elimination, which combines both approaches by allowing variables to be either added or removed at each step based on their statistical significance. Each iteration evaluates variables using F-statistics derived from t-tests on coefficient estimates, with the algorithm terminating when predefined statistical thresholds are satisfied ( Miller et al., 2022 ) . Despite criticisms regarding overfitting and statistical validity ( Harrell, 2001 ) , modern applications are focused on Akaike’s Information Criterion (AIC) ( Akaike, 1974 ) and the Bayesian Information Criterion (BIC) ( Schwarz, 1978 ; Raftery, 1995 ) —which balance model fit and complexity, respectively. While current stepwise selection methods typically rely on modern applications, including “a p-value threshold” ( Fong et al., 2017 ) , there is a need to incorporate automatic detection of optimal thresholds in numeric variables to enhance model robustness and reduce the risk of overfitting. Nevertheless, purely linear models may fail to capture meaningful nonlinearities or threshold effects present in real-world data. For instance, the relationship between age and health outcomes may change sharply after a certain age, or a risk factor may become relevant only beyond a critical level. Decision tree algorithms ( Breiman et al., 1984 ; Quinlan, 1993 ) address this by partitioning predictor space into regions via recursive splits, often yielding models that are piecewise constant or piecewise linear—such as in the M5 model tree algorithm ( Quinlan, 1992 ) . Recent variants have improved scalability and added regularization, including PILOT ( Raymaekers et al., 2024 ) , which uses greedy splitting with ridge-regularized linear fits at the nodes, and MOTR-BART ( Prado et al., 2021 ) , which integrates linear components into Bayesian additive regression trees. In applied domains such as cybersecurity, hybrid methods like logistic model trees have demonstrated the value of combining split-based structure with interpretable regression ( Adeyemo et al., 2021 ) . While effective at capturing nonlinearities, these approaches typically result in complex model structures with multiple splits, nested conditions, or ensemble components, which diminish transparency and make it difficult to extract global insights—a fundamental limitation for applications where interpretability is as important as predictive performance. Beyond tree-based models, rule-based hybrid approaches have emerged to balance interpretability with predictive power. RuleFit ( Friedman and Popescu, 2008 ) , for example, constructs linear models from rule-based features extracted from decision trees. Such approaches can maintain a form of interpretability by expressing predictions as a linear combination of human-readable conditions ( Molnar, 2022 ) . However, they typically require training large ensembles and applying regularization to manage complexity, which can limit transparency and accessibility in practice. Furthermore, the resulting models often present users with numerous rules of varying importance, making it challenging to distill key insights about the underlying relationships between variables. A key distinction is that these methods do not specifically create or optimize binary threshold variables that directly enhance linear model performance while preserving the interpretability of the original feature space. This gap indicates a need for approaches that introduce nonlinearity in a more controlled and targeted manner—maintaining the coherence of a global linear regression while enabling the discovery of meaningful breakpoints in continuous variables. In this paper, we propose SplitWise regression, a deterministic framework that integrates adaptive threshold-based transformations into a stepwise selection process. The method allows numeric variables to enter the model either as standard linear terms or as binary indicators based on data-driven cut-points. Each candidate transformation is evaluated using AIC or BIC to ensure that only interpretable, predictive modifications are retained. The resulting model remains a global linear equation, augmented by a small number of thresholded features—e.g., I(18<\text{age}<34) or I(\text{blood pressure}>140) —which preserve transparency while capturing relevant nonlinearity. Compared to full tree-based methods, this approach yields a compact and interpretable model that can be readily understood, audited, or used manually. To support adoption and reproducibility, we also release a user-friendly R package, SplitWise . The remainder of this paper is structured as follows. Section 2 describes the SplitWise algorithm, software implementation, and evaluation datasets. Section 3 presents experimental results and interpretation examples. Section 4 concludes and outlines directions for future research. 2 Methodology, Software, and Data We begin by formally describing the SplitWise algorithm and its two transformation modes, followed by details on its software implementation and the datasets used for evaluation. 2.1 SplitWise Regression Algorithm The SplitWise regression algorithm constructs interpretable regression models by automatically determining how each predictor should enter the model—either as a linear term or as one or more dummy-coded segments, derived via optimal split points. It uses information criteria (AIC or BIC) to balance model fit and complexity, ensuring that the selected transformations yield the best trade-off between interpretability and predictive power. The algorithm operates in two modes—iterative and univariate—as outlined below and illustrated in Figure 1 . The iterative mode is generally more precise, as it evaluates transformations in the context of other variables already included in the model, allowing it to account for variable interactions and joint effects. The univariate mode, by contrast, is more scalable to high-dimensional data, as it transforms each

Read the complete text