Multistage Defer Trees for Hybrid Interpretability: If at First You Can’t Succeed, Tree Again

Zakk Heile, Hayden McTavish, Margo Seltzer, Cynthia Rudin · 2026

Read Multistage Defer Trees for Hybrid Interpretability: If at First You Can’t Succeed, Tree Again by Zakk Heile, Hayden McTavish, Margo Seltzer, Cynthia Rudin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30995.txt.

Introduction

Zakk Heile, Hayden McTavish, Margo Seltzer and Cynthia Rudin build classifiers that are simple for most inputs and heavy only where needed. Single trees can match black-box models in noisy settings, but elsewhere practitioners face a trade between accuracy and interpretability. Multistage Defer Trees are a sequence of sparse trees. Each tree predicts for most samples that reach it and defers a share to the next tree, with a black box as the last resort. Users set the deferral rate to choose their point on the trade-off.

The authors show these cascades can be trained to match tree ensembles while sending most samples through just one or a few sparse trees. They describe techniques for training the stages while keeping each tree simple. The result widens the accuracy-interpretability frontier where single trees fall short: even when complex models prove necessary, they need not be opaque.

Catalog record

CREATORZakk Heile, Hayden McTavish, Margo Seltzer, Cynthia Rudin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE122 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multistage Defer Trees for Hybrid Interpretability: If at First You Can’t Succeed, Tree Again Zakk Heile, Hayden McTavish, Margo Seltzer, Cynthia Rudin Abstract Recent work has shown that well-optimized individual decision trees can match complex black box models in some settings, primarily in noisy domains. For the remaining settings, however, complex ensembled compositions of trees often achieve higher accuracy at the cost of interpretability, leaving practitioners with difficult modeling decisions along an accuracy-interpretability tradeoff. Ideally, we would like to classify as much of the data as possible with one or a small number of trees, achieving interpretability for most samples while maintaining state-of-the-art accuracy. We introduce Multistage Defer Trees: a sequence of sparse decision trees that each make predictions for most samples, while deferring a small proportion to the next tree in the sequence or, ultimately, to a black box. We demonstrate that we can train this model class to match the performance of complex tree-based ensembles while routing most samples through only one or a small number of sparse decision trees. We discuss a range of techniques for training these models while maintaining simplicity. Our method expands the accuracy–interpretability frontier in settings where single-tree methods remain insufficient, demonstrating that even when complex models are necessary, they need not be fully opaque. 1 Introduction A central challenge in interpretable machine learning is to construct interpretable models that match state-of-the-art predictive performance. Well-optimized simple models, such as optimal sparse trees, can achieve this goal in certain settings, such as when the data is generated by noisy processes [ 64 , 63 , 8 ] . In such cases, the accuracy-interpretability gap is not observed. Still, there remains such a gap in datasets falling outside of this regime. One proposed direction to navigate the accuracy-interpretability gap is to use a hybrid- or partially-interpretable model [ 68 , 69 , 56 , 22 , 25 , 34 ] . The goal is to learn a simple interpretable model that, for each sample, either makes a prediction for that sample or defers to a black box’s prediction. By specifying the proportion of samples classified by a black box (the deferral rate), users can navigate a tradeoff between matching black box accuracy and keeping decisions interpretable for more samples. While hybrid-interpretable models provide a smooth way to navigate compromises between accuracy and interpretability, existing methods often struggle to maintain black box accuracy without deferring the vast majority of samples. Figure 1 : Example of an MDT on the Bike dataset. The model consists of two stages, each with bounded depth. Two leaves in the first stage defer to the second stage, and three leaves in the second stage defer to a black box (here, an XGBoost model). The interpretable component has 13 leaves. We present a tree-based hybrid-interpretable model that minimizes deferral rate while matching the performance of more complex ensembles. To learn this model, we iteratively focus on subsets of the data where simple models provably cannot perform as well as a black box model. This leads to a new model class, Multistage Defer Trees (MDTs) , where each tree can either make a prediction or defer to the next stage; the final stage may then defer some samples to a black box model if required. To train MDTs, we introduce an alternating optimization procedure in which the deferred-to model evolves with each stage of the MDT. As this process unfolds, the set of deferred points decreases monotonically, and the later stages and fallback model become increasingly specialized to those points. This allows the model to defer only on a small fraction of inputs while maintaining high accuracy. When the fallback is a sparse model (i.e., a single decision tree or EBM [ 36 ] ), even these deferral cases remain interpretable; when the fallback is a black box, performance can be matched with minimal reliance on that model. Crucially, because deferred points are confined to a tightly constrained region of the feature space, the fallback model can be compressed, producing simpler but equivalent representations of even complex ensembles. Our contributions are as follows. • We introduce Multistage Defer Trees (MDTs), a new model class that allocates complexity adaptively, based on the difficulty of different regions of the feature space. • We develop an iterative algorithm for training MDTs that successively narrows the deferral region, while borrowing strength from surrounding regions, to learn specialized trees that generalize well. • We provide algorithms to compress MDTs into sparse single-tree or rule-list representations and extend this framework to compress the fallback model when it is a tree or tree ensemble. • We demonstrate improved accuracy–deferral–sparsity trade-offs where such trade-offs exist. 2 Related Work Hybrid Approaches 69 introduces hybrid models for interpretability: an interpretable rule set or linear model makes predictions on some of the data, while deferring to fixed black box predictions on the remaining data. This allows users to navigate a tradeoff between the model accuracy and the amount of samples using interpretable model predictions. This framework, in turn, connects to work in deferral models [ 37 , 46 ] , rejection learning [ 12 , 61 ] , selective classification [ 18 ] , and abstention-based learning [ 62 , 24 ] . Hybrid models have primarily been applied to simple rule set models or linear models [ 25 , 22 , 69 , 56 , 68 ] . One recent work explores a gradient-based method to learn a complex (100-1000 leaves) decision tree, which defers to a black box based on computing the proximity to the decision boundary [ 34 ] ; by contrast, our work learns a sparser sequence of trees that directly defer where the black box is most helpful, and iteratively reduces this deferral region. Decision Tree Optimization Decision trees are classically optimized with highly scalable greedy algorithms [ 9 , 57 ] . Such algorithms can be ideal for popular ensembles of many trees [ 10 , 11 ] . More recent work, however, has revisited the use of less scalable, higher quality individual tree algorithms, to reduce the need for this level of model complexity and to allow use of a single well-performing sparse tree classifier for domains where interpretability is needed. In recent years, global optimization of individual trees has become reasonably tractable for bounded depth and sparsity [ 1 , 29 , 16 , 35 , 42 , bri\vtop{\halign{#\cr\hbox{}\crcr}} ] . This problem is nevertheless NP-hard and does struggle to scale to cases with many features. Recent work sacrifices global optimality in exchange for improved scalability and runtime while maintaining performance close to optimal [ 3 , 6 , 31 ] . Other approaches add anytime behavior, allowing early termination with high-quality solutions [ 32 , 15 , 33 ] . We fit a small sequence of trees that nearly matches the performance of a significantly more complex ensemble. Because MDTs rely on only a few sparse trees, we can afford to optimize each component tree more carefully than in standard greedy tree induction while still requiring the overall procedure to scale. Given this need for both higher-quality component trees and scalability, we use a version of the polynomial-time LicketySPLIT algorithm from 3 , which we

Read the complete text