DiNo and RanBu: Lightweight Predictions from Shallow Random Forests

Tiago Mendonça dos Santos, Rafael Izbicki, Luís Gustavo Esteves · 2025

Read DiNo and RanBu: Lightweight Predictions from Shallow Random Forests by Tiago Mendonça dos Santos, Rafael Izbicki, Luís Gustavo Esteves (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23624.txt.

Introduction

Tiago Mendonça dos Santos, Rafael Izbicki, and Luís Gustavo Esteves make random forests faster by keeping them shallow. A standard forest grows hundreds of deep trees, which predict well on tabular data but answer slowly and eat memory. That cost blocks uses where replies must be instant or hardware is small.

Their two methods, DiNo and RanBu, squeeze a few depth-limited trees into distance-weighted predictors after training ends. DiNo measures how far apart two cases sit by their most recent common ancestor in a tree, while RanBu smooths over a classic proximity measure with a kernel. Tuning the width parameter needs only light matrix work. Across three synthetic benchmarks and twenty-five datasets, RanBu matched or beat full forests, especially with noisy data, while cutting time by up to ninety-five percent. Both methods extend to quantile regression, and the code is open source.

Catalog record

CREATORTiago Mendonça dos Santos, Rafael Izbicki, Luís Gustavo Esteves
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23624

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiNo and RanBu: Lightweight Predictions from Shallow Random Forests Tiago Mendonça dos Santos, Rafael Izbicki, Luís Gustavo Esteves Abstract Random Forest ensembles are a strong baseline for tabular prediction tasks, but their reliance on hundreds of deep trees often results in high inference latency and memory demands, limiting deployment in latency-sensitive or resource-constrained environments. We introduce DiNo (Distance with Nodes) and RanBu (Random Bushes), two shallow-forest methods that convert a small set of depth-limited trees into efficient, distance–weighted predictors. DiNo measures cophenetic distances via the most recent common ancestor of observation pairs, while RanBu applies kernel smoothing to Breiman’s classical proximity measure. Both approaches operate entirely after forest training: no additional trees are grown, and tuning of the single bandwidth parameter h h requires only lightweight matrix–vector operations. Across three synthetic benchmarks and 25 public datasets, RanBu matches or exceeds the accuracy of full-depth random forests—particularly in high-noise settings—while reducing training plus inference time by up to 95%. DiNo achieves the best bias–variance trade-off in low-noise regimes at a modest computational cost. Both methods extend directly to quantile regression, maintaining accuracy with substantial speed gains. The implementation is available as an open-source R/C++ package at https://github.com/tiagomendonca/dirf . We focus on structured tabular random samples (i.i.d.), leaving extensions to other modalities for future work. Keywords Random Forest, Tree Proximity, Cophenetic Distance, Kernel Methods, Shallow Forests, Quantile Regression, Fast Inference, Tabular Data 1 Introduction Tree ensembles—most notably Random Forests (RF; Breiman 2001 )—are a mainstay in tabular machine–learning pipelines. They handle heterogeneous features without special preprocessing, are robust to over-fitting, and require minimal hyper-parameter tuning ( Breiman 2001 ; Biau and Scornet 2016 ) . Yet, in practical deployments where models must respond quickly and operate under hardware constraints, two bottlenecks often emerge when RFs grow to hundreds of deep trees: (i) Inference latency. Traversing many deep trees per query incurs irregular memory access patterns, dominating CPU time and straining cache performance—an obstacle for low-latency applications such as user-facing APIs, streaming, or embedded devices. (ii) Sensitivity to irrelevant features. In high-dimensional or noisy settings, splits on uninformative variables degrade both statistical efficiency and computational performance, slowing split selection and increasing model size. Research on accelerating ensemble inference ranges from hardware-aware optimizations ( Browne et al. 2019 ; Romero-Gainza et al. 2022 ) to algorithmic compression and shallow-tree strategies ( Kumar et al. 2017 ) . While effective, these solutions often require specialized engineering or retraining. Our approach. We show that shallow forests—tens of trees with modest, fixed depth—retain enough geometric structure to support competitive predictors when transformed into a kernel-like weighting scheme . We propose creating such weights using two schemes: DiNo A cophenetic distance ( Sokal and Rohlf 1962 ) that, for each tree, counts the number of edges from each of two observations to their most recent common ancestor (MRCA) and takes the maximum. A Gaussian kernel smoothing weighting of the forest-averaged distance yields smooth, distance-weighted predictions. RanBu A Gaussian kernel-like reweighting of Breiman’s distance —one minus the average proportion of trees in which two points share the same leaf. This continuous measure in [0,1] emphasizes closer pairs while smoothly attenuating more distant ones. Both operate entirely post hoc : the forest structure is fixed, and tuning the single bandwidth parameter h involves only lightweight matrix–vector operations over precomputed leaf responses. Shallow trees make inference efficient, enabling deployment in large-scale or interactive settings. The MRCA distance captures hierarchical separation via the longest branch from each leaf to their MRCA, making it less sensitive to split balance. Gaussian rescaling turns both into smooth, locality-aware similarities. As they require no retraining, these measures enable post hoc enhancement of shallow ensembles with minimal computational cost. Illustrative speed-up. On the Online News Popularity dataset (39k rows, 59 predictors), a 500-tree full-depth RF requires 95.6 seconds to train and predict for one test point on a machine with an Intel Core i7-11800H processor, 64 GB of RAM, running Windows 11. RanBu , with 50 trees of depth 5—the same as DiNo —takes 1.26 seconds for both steps, a nearly 75-fold reduction in total computation time with negligible change in test MSE. Contributions. Our contributions are the following: (1) Tree–based kernels. We develop two tree-based smoothing kernels: the MRCA distance and Breiman’s distance. We show that a Gaussian rescaling of these distances obtained in random forests yields effective kernel smoothers for prediction. (2) Lightweight estimators. We introduce DiNo and RanBu for mean and quantile regression, designed for shallow, fixed-depth forests with only post-training bandwidth tuning. (3) Empirical evaluation. We evaluate RanBu and DiNo on three synthetic and 25 public datasets. RanBu matches or surpasses RF in accuracy—especially under high noise—while reducing inference time by up to 95%. DiNo achieves the best bias–variance trade-off in low-noise settings. On real-world datasets, RanBu consistently outperforms GRF in predictive accuracy and runtime, and secures advantages over Boosting and Reduced R.F., whereas DiNo delivers systematic improvements over the Reduced R.F. baseline, confirming that the kernel weighting step adds predictive value beyond simple pruning. (4) Open-source implementation. We release a production-ready R /C++ implementation at https://github.com/tiagomendonca/dirf , enabling reproducibility and practical use. Paper outline Section 2 reviews related work on proximity measures and efficient RF variants. Section 3 details the proposed distances and predictors. Section 4 presents synthetic and real-data experiments. Section 5 discusses real-world implications. Section 6 summarises contributions and outlines future directions. 2 Related Work 2.1 Tree-Based Proximity Measures and Shallow Forests Random Forests (RF) naturally induce a notion of similarity between instances based on how often they fall in the same terminal node, as introduced by Breiman and Cutler 2004 and Breiman and Cutler 1999 . This co-occurrence-based proximity has been widely used in unsupervised tasks such as clustering and outlier detection, but its binary nature—treating all co-occurring pairs equally—limits expressiveness. To address this, several works have enriched proximities with structural information from the trees. For example, Englund and Verikas 2012 proposed a depth-aware measure that decays with the distance between terminal nodes, while Panda et al. 2024 demonstrated that the RF proximity matrix is positive-definite and characteristic, reinforcing its potential for kernel-based learning. DiNo and RanBu build on this tradition with two innovations aimed at balancing predictive accuracy and computational efficiency. First, both methods operate on a deliberately small number of shallow trees, reducing inference latency without altering the standard RF training pipeline. Second, they employ continuous, task-adapted similarity functions: DiNo derives a distance from the most recent common ancestor (MRCA) in each tree, while RanBu applies a Gaussian kernel smoothing weighting to the classical RF proximity. This design preserves compatibility with existing RF tooling and interpretability of axis-aligned splits, while enabling smooth weighting of training observations at prediction time. The result is a family of lightweight estimators particularly suited for latency-sensitive scenarios, requiring only post-training computations over precomputed leaf responses. 2.2 Efficient Inference via Compression and Hardware Acceleration System-level optimizations have also been proposed to accelerate ensemble inference. Forest Packing ( Browne et al. 2019 ) improves cache efficiency by reorganizing tree nodes in memory. Bolt ( Romero-Gainza et al. 2022 ) precomputes decision paths and stores them in compressed lookup tables, while GPU-based methods such as Tahoe ( Xie et al. 2021 ) exploit parallelism for

Read the complete text