Unique Rashomon Sets for Robust Active Learning

Simon Nguyen, Kentaro Hoffman, Tyler McCormick · 2025

Read Unique Rashomon Sets for Robust Active Learning by Simon Nguyen, Kentaro Hoffman, Tyler McCormick (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06770.txt.

Introduction

Simon Nguyen, Kentaro Hoffman, and Tyler McCormick present a way to pick which data to label for a learning model. Their paper is from two thousand twenty-five. Labelling data costs time and money, so active learning picks the most telling cases first. With few labels at the start, noise can make dull cases look rich in doubt.

Their method builds a team of models from the Rashomon set, the set of near-best models. It keeps only models that are both strong and unlike each other in how they explain the data. That filter splits true doubt from noise. Theory gives faster gains than classic routes, and tests show large lifts in accuracy across data sets.

Catalog record

CREATORSimon Nguyen, Kentaro Hoffman, Tyler McCormick
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06770

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unique Rashomon Sets for Robust Active Learning Simon Nguyen, Kentaro Hoffman, Tyler McCormick Abstract Collecting labeled data for machine learning models is often expensive and time-consuming. Active learning addresses this challenge by selectively labeling the most informative observations, but when initial labeled data is limited, it becomes difficult to distinguish genuinely informative points from those appearing uncertain primarily due to noise. Ensemble methods like random forests are a powerful approach to quantifying this uncertainty but do so by aggregating all models indiscriminately. This includes poor performing models and redundant models, a problem that worsens in the presence of noisy data. We introduce UNique Rashomon Ensembled Active Learning ( UNREAL ), which selectively ensembles only distinct models from the Rashomon set, which is the set of nearly optimal models. Restricting ensemble membership to high-performing models with different explanations helps distinguish genuine uncertainty from noise-induced variation. We show that UNREAL achieves faster theoretical convergence rates than traditional active learning approaches and demonstrates empirical improvements of up to 20% in predictive accuracy across five benchmark datasets, while simultaneously enhancing model interpretability. 1 Introduction Collecting labeled data to train data-hungry modern artificial intelligence (AI) and machine learning (ML) models can be expensive or time-consuming. This challenge arises in a wide range of applications: image labeling ( Zhu & Bento 2017 ; Huijser & van Gemert 2017 ) , sentence classification ( Schumann & Rehbein 2019 ) , and verbal autopsy ( Setel et al. 2007 ; McCormick et al. 2016 ; Jha et al. 2019 ; Fan et al. 2024 ) . In such scenarios, strategically determining which observations are most informative for model training will greatly reduce data redundancy and improve the model’s ability to generalize from limited data. To address time and budget constraints, active learning provides a framework where observations are adaptively and strategically selected for labeling based on their potential informativity. The key task in active learning is choosing the most informative observations that will enhance the predictive quality of the model when labeled. However, this selection process faces a fundamental challenge: with limited labeled data initially available, how can we confidently identify truly informative observations rather than noise? Amongst the many metrics of informativity, Lewis & Gale 1994 ; Freund et al. 1997 ; Roy & McCallum 2001 , uncertainty is the most commonly employed ( Liu et al. 2022 ) . One particularly effective approach to quantifying uncertainty is through query-by-committee (QBC), where multiple models form a committee whose disagreement in predictions indicates which observations are most uncertain and therefore informative to label. Ensemble methods such as random forests are especially well-suited for active learning with QBC techniques ( Freund et al. 1997 ; Melville & Mooney 2004 ) as their weak learners naturally form a diverse committee. This diversity is crucial as it incorporates different explanations of the data into the committee, with disagreement among ensemble members then serving as a direct measure of uncertainty and informativity ( Dagan & Engelson 1995 ; Kee et al. 2018 ) . While ensemble methods offer a natural way to quantify uncertainty through the diversity of multiple learners (models), ensemble methods tend to aggregate over the space of all models, even if some of the models may have relatively poor accuracy. While some approaches such as Bayesian model averaging account for this by weighting the models by how likely they are given the data ( Hoeting et al. 1999b ) , having a large number of mediocre models can make such weighting approaches difficult, especially in cases of limited, high-dimensional, or noisy data ( Graefe et al. 2015 ) . This challenge is further exacerbated in the presence of label noise, which although common in realistic settings, is well-known to be detrimental to active learning strategies ( Willett et al. 2005 ) . Active learning strategies, designed to prioritize querying points where models are most uncertain, may overfit to noisy labels instead of focusing on true areas of uncertainty. This challenge makes incorporating a diverse set of explanations among committee members even more crucial. To address these limitations, we propose an algorithm that enhances active learning by restricting aggregation to a subset of well-performing and high-evidence models known as the Rashomon set. Importantly, in the presence of label noise, the Rashomon set naturally expands to encompass a more diverse set of valid explanations, providing a principled way to capture and utilize the inherent ambiguity in noisy data. By ensembling the models within the Rashomon set, which is the set of nearly optimal models, our approach ensures that the active learning process is driven by models with high evidence while accounting for noise through multiple diverse yet equally valid explanations of the data. Throughout the paper, we rely on tree-based models. These models partition the feature space into groups with similar outcomes. Tree-based models are widely used in practice and have been extensively explored, particularly in combination with a sparsity requirement, in the context of Rashomon sets. However, a common and widely known issue with tree-based models is that multiple trees can represent the same partition. The multiplicity of trees creates two distinct challenges. First, it means that more complex partitions are more likely to appear in the Rashomon set simply by chance (because as the complexity of the partition increases, there are more possible trees that represent the same partition). A similar issue occurs in discrete Bayesian model averaging with uniform priors over models–there are more medium sized models, so the “flat” prior actually strongly favors models of a particular size. Second, the multiplicity of trees means that approaches that sample from the space of trees expend additional, unnecessary, computational cycles evaluating redundant trees. This issue arises because trees enforce an ordering, or hierarchy among variables that have no natural order. Venkateswaran et al. 2024 address this issue by using a Hasse diagram, which, unlike trees, is a geometric object compatible with partially ordered sets. In this paper, we remain in the space of trees and demonstrate that one can further restrict the Rashomon set to a unique set of classification patterns while still preserving the performance of the active learning process. This Rashomon-based method, titled UNique Rashomon Ensembled Active Learning ( UNREAL ), ensures that the ensemble incorporates the interpretability of weak learners by only including the set of unique and diverse explanations. Simulations on five benchmark datasets demonstrate improvements in prediction accuracy compared to standard approaches. 2 Rashomon Sets When constructing machine learning models, researchers face two distinct types of uncertainty. The first originates from the variability in the predicted outcomes generated by a given model, often referred to as a model’s intrinsic risk . The second originates from selecting the right model from a vast and diverse hypothesis space, a phenomenon known as model ambiguity . This distinction, originally articulated by economist Kenneth Arrow in 1951, separates the uncertainty of prediction from a given model from the uncertainty of choosing among many plausible models ( Arrow 1951 ) . Many modern machine learning approaches have become extremely good at reducing intrinsic risk within a model, but often fail to fully account for model ambiguity. Methods such as LASSO search for a single optimal model and conclusions tend

Read the complete text