Discovering Latent Groups for Robust Classification

Ankur Garg, Ulrich Aïvodji, Samira Ebrahimi Kahou, Vincent Michalski · 2026

Read Discovering Latent Groups for Robust Classification by Ankur Garg, Ulrich Aïvodji, Samira Ebrahimi Kahou, Vincent Michalski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23609.txt.

Introduction

Garg, Aïvodji, Ebrahimi Kahou, and Michalski build a classifier whose very shape reveals hidden subgroups in the data, in a 2026 paper on robust machine learning. Models often learn spurious correlations, shortcuts that predict labels in training but fail for minority subgroups, such as waterbirds photographed on land. Standard fixes adjust weights using group labels or guesses, yet the finished model cannot say which subgroup a new sample belongs to.

Their neural classification trees route each sample through a tree of nodes split into easy and hard branches. Easy means the model currently gets such samples right, and hard means it struggles, a signal that tracks subgroup identity. Routes from one round become pseudo labels for the next, so conflicting subgroups separate without any group supervision. Across five benchmarks the tree isolates minority subgroups consistently, giving a transparent map of latent structure while matching top robustness methods.

Catalog record

CREATORAnkur Garg, Ulrich Aïvodji, Samira Ebrahimi Kahou, Vincent Michalski
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Discovering Latent Groups for Robust Classification Ankur Garg, Ulrich Aïvodji, Samira Ebrahimi Kahou, Vincent Michalski Abstract Machine learning models exploit spurious correlations, achieving high average accuracy but failing disproportionately on underrepresented subgroups. Existing methods address this by adjusting network parameters, guided either by subgroup annotations or inferred pseudo-group labels. Yet at inference, these methods produce only a class prediction, with no insight into a sample’s latent subgroup. We propose neural classification trees (NCT), a framework that achieves robustness by encoding subgroup structure in its tree-shaped architecture. By routing each sample to an ‘‘easy’’ or ‘‘hard’’ node of this tree---based on prediction correctness---and reusing these routes as pseudo-labels for the next iteration, NCT disentangles conflicting subgroups, without requiring subgroup supervision. We evaluate NCT on five benchmarks spanning binary and multi-class spurious correlations. Our experiments show that the learned tree topology provides strong interpretability by consistently isolating minority subgroups, which provides a transparent mapping between the model architecture and the data’s latent group structure, while yielding competitive robustness with state-of-the-art methods. 1 1 1 https://github.com/agarg-dev/Neural-Classification-Trees/ 1 Introduction Deep neural networks are prone to learning shortcuts or features that are correlated with labels in the training set but have no causal relationship to them. [ 9 ] . A classic example is Waterbirds dataset, where a standard model will classify birds based on the background ( water vs. land ) rather than the bird’s features. When such spurious correlations shift at test time, performance degrades for minority subgroups, such as waterbird in front of a land background. Although the research community has developed effective methods to mitigate this issue, a critical gap still remains: interpretability of the latent subgroup structure. Supervised approaches such as group distributionally robust optimization (group DRO) [ 25 ] , while robust, require expensive, fine-grained annotations for every training sample. Just train twice (JTT) [ 18 ] , environment inference for invariant learning (EIIL) [ 7 ] , and deep feature reweighting (DFR) [ 15 ] drop training-time labels but still rely on validation group labels, while GEORGE [ 29 ] and explainability-guided pseudo-group map (ExMap) [ 5 ] operate without any group annotation. Across all three regimes, the model’s parameters are adjusted to improve worst-group metrics, but the final classifier remains opaque at inference: it does not reveal which latent subgroup a specific sample belongs to, nor does it structurally isolate conflicting features. We take a different approach: instead of only relying on parameter updates to handle diverse groups, we make the partition architectural. We introduce NCT, a framework that builds on a well-established observation: training difficulty is a reliable proxy for semantic group identity, with samples aligned with spurious correlations being easy and conflicting counter-examples being hard. NCT iteratively routes samples to easy and hard branches based on this signal, and — unlike methods that use the same signal transiently — preserves the resulting partition as the inference-time architecture itself. Our contributions are: (i) Architecture as partition. Where prior work [ 18 , 34 , 24 , 5 ] uses correctness, loss, or attribution signals transiently for reweighting or pseudo-group inference, neural classification tree (NCT) preserves the difficulty-based partition as the inference-time architecture, with each leaf encoding both the predicted class and the difficulty path. (ii) Unsupervised depth selection. A pseudo worst-group accuracy (pWGA) criterion with a Wilson-tolerance early-stopping bound decides when to stop deepening the tree without group annotations. (iii) Theoretical motivation. Iteration-1 errors are enriched with minority samples under simplicity bias (Theorem 4.2 ); structural separation yields a positive minimax-risk gap under feature conflict (Theorem 4.4 ). (iv) Empirical evidence. Across five benchmarks, NCT concentrates minority subgroups in hard branches (82% of landbird-on-water, 73% of blond-male, 47% of benign-no-patch, 71% of digit-8 and 84% of color-conflicting digits) while matching or approaching state-of-the-art worst-group accuracy against eight baselines. 2 Related Work Optimization for Spurious Correlations. A dominant paradigm for mitigating spurious correlations modifies the optimization objective to upweight minority groups. When group labels are available, group DRO [ 25 ] minimizes the worst-case group loss. When training labels are unavailable, recent approaches infer pseudo-groups to guide optimization. GEORGE [ 29 ] clusters representations from a standard empirical risk minimization (ERM) model to estimate subclass labels, subsequently using them for robust optimization. JTT [ 18 ] identifies error sets from an early-stopped ERM model and retrains the final model by upweighting these hard samples. Correct-n-contrast (CnC) [ 34 ] extends this by applying contrastive learning to push the inferred groups apart in feature space. Other works focus on the classifier head; DFR [ 15 ] freezes the feature extractor and retrains only the last layer on a group-balanced validation set. Inferring Environments and Pseudo-Groups. A parallel line of work treats group identification as an inference problem in its own right. EIIL [ 7 ] learns a soft environment partition that maximally violates an invariance penalty, which is then handed to invariant or distributionally robust optimizers. LfF [ 22 ] reweights a debiased model using the loss of a deliberately biased one, while SelecMix [ 13 ] exploits bias-conflicting pairs. More recent work sharpens the partition: XRM [ 24 ] trains twin networks on disjoint halves of the training data and uses confident held-out cross-mistakes to discover environments without group-annotated validation data, and GIC [ 10 ] infers groups from a spurious-attribute classifier whose predictions vary across distributional shifts. These methods share a downstream pattern: the discovered partition is consumed by a separate invariant-learning or reweighting algorithm, and the resulting classifier is opaque at inference. NCT is orthogonal: its contribution is the architectural persistence of the partition, rather than a sharper pseudo-group signal. Input-Level and Explainability-Guided Interventions. Rather than altering the loss function, some methods intervene directly on the data or leverage post-hoc explanations. MaskTune [ 4 ] forces the model to explore robust features by masking out the most discriminative regions of the input image during fine-tuning. ExMap [ 5 ] clusters explainability heatmaps from a pre-trained model into pseudo-groups, exploiting the observation that spurious and robust decisions produce distinct attribution patterns. NCT contributes to this neighborhood by using correctness rather than attributions as the difficulty signal, and by preserving the resulting partition as the inference-time architecture rather than as a one-off clustering step. Structural and Modular Interpretability. A complementary line of work builds interpretability directly into the model architecture. Concept bottleneck models (CBMs) [ 16 ] force information to pass through a layer of human-aligned concepts but typically require expensive concept annotations or discovery algorithms like concept balancing technique (CoBalT) [ 2 ] . Tree-structured classifiers such as ProtoTree [ 23 ] and NBDT [ 33 ] also expose inference-time decision paths, but through class-taxonomy hierarchies; neither was designed for spurious-correlation settings. NCT adopts a modular approach akin to mixture of experts (MoE) [ 28 , 14 ] or neural trees [ 30 ] . Unlike standard MoE that routes samples to maximize predictive likelihood—and therefore collapses to spurious shortcuts— NCT routes by correctness, decomposing the spurious-vs-core subgroup structure. Across these four families, NCT is the only entry whose inference-time architecture exposes the spurious-correlation subgroup partition: each leaf records both the predicted class and the difficulty path that produced it, with no post-hoc clustering required. Figure

Read the complete text