Testing Most Influential Sets
Lucas D. Konrad, Nikolas Kuschnig · 2025
Read Testing Most Influential Sets by Lucas D. Konrad, Nikolas Kuschnig (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20372.txt.
Introduction
Lucas D. Konrad and Nikolas Kuschnig develop a formal testing framework for most influential sets, the small groups of data points that can overturn a model conclusions. Recent work can find such sets, but no principled way existed to tell whether the maximum influence found is excessive or just natural sampling variation. Their framework supplies that missing inferential step for the linear least-squares case.
The authors derive a convenient exact influence formula and identify the extreme value distributions of maximal influence, a heavy-tailed Fréchet form in some settings and a well-behaved Gumbel form in others. Those distributions support rigorous hypothesis tests for excessive influence. Applications across economics, biology, and machine learning benchmarks resolve contested findings and replace ad hoc heuristics.
Catalog record
| CREATOR | Lucas D. Konrad, Nikolas Kuschnig |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20372
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Testing Most Influential Sets Lucas D. Konrad, Nikolas Kuschnig Abstract Small subsets of data with disproportionate influence on model outcomes can have dramatic impacts on conclusions, with a few data points sometimes overturning key findings. While recent work has developed methods to identify these most influential sets , no formal theory exists to determine when their influence reflects genuine problems rather than natural sampling variation. We address this gap by developing a principled framework for assessing the statistical significance of most influential sets. Our theoretical results characterize the extreme value distributions of maximal influence and enable rigorous hypothesis tests for excessive influence, replacing current ad-hoc sensitivity checks. We demonstrate the practical value of our approach through applications across economics, biology, and machine learning benchmarks. 1 Introduction Machine learning (ML) models and statistical inferences can be highly sensitive to small subsets of data. In many applications, just a handful of samples can overturn key conclusions: two countries nullify the estimated effect of geography on development ( Kuschnig et al. 2021 ) , a single outlier flips the sign of a treatment effect ( Broderick et al. 2021 ) , or a small group of individuals drives disparate outcomes in algorithmic decision-making ( Black & Fredrikson 2021 ) . These most influential sets — data subsets with the greatest influence on model predictions — are central to questions of interpretability, fairness, and robustness in modern machine learning ( Black & Fredrikson 2021 ; Chen et al. 2018 ; Chhabra et al. 2023 ; Ghorbani & Zou 2019 ; Sattigeri et al. 2022 , see, e.g.,) . Despite their practical importance, practitioners lack principled tools to assess whether a set’s influence is genuinely problematic. Current practice relies on domain expertise and ad-hoc sensitivity checks, while approximate methods such as influence functions ( Koh & Liang 2017 ; Fisher et al. 2023 ; Schioppa et al. 2023 ) systematically underestimate the impacts of sets and extreme cases ( Basu et al. 2020 ; Koh et al. 2019 ) . Recent work highlights both the promise and challenges of most influential subsets — small sets can drive results even in randomized trials ( Broderick et al. 2021 ; Kuschnig et al. 2021 ) , heuristic algorithms can fail in simple settings ( Hu et al. 2024 ; Huang et al. 2025 ) , and influence bounds remain an active area of research ( Moitra & Rohatgi 2023 ; Freund & Hopkins 2023 ; Rubinstein & Hopkins 2024 ) . What remains missing is a principled method to distinguish natural sampling variation from genuinely excessive influence. We develop a statistical framework for assessing the significance of most influential sets. By focusing on linear regression — a tractable, interpretable, and widely-used setting that underlies many modern methods ( Rudin 2019 ) — we derive the exact asymptotic distributions of maximal influence. We show that two distinct regimes emerge depending on the size of the influential set: when the size is fixed, maximal influence converges to a heavy-tailed Fréchet distribution; when the size grows with the sample, maximal influence converges to a well-behaved Gumbel distribution. Our results enable principled hypothesis tests for excessive influence, replacing ad-hoc diagnostics with rigorous statistical procedures. We demonstrate their practical value via applications across economics, biology, and machine learning benchmarks, resolving ambiguous cases where influential sets drive contested findings. 1.1 Contributions We present a comprehensive analysis of the influence of most influential sets, both theoretically and in practical applications. To summarize, our main contributions are: 1. Theoretical foundations. We derive distributions for the influence of most influential sets, establishing their extreme value behavior and enabling statistical testing. 2. Efficient implementation. We provide computationally efficient procedures for evaluating influence, making our approach practical for real-world applications. 3. Empirical validation. We demonstrate the utility of our framework across domains, resolving the contested “Blessing of Bad Geography” in economics, assessing robustness in biological data of sparrow morphology, and auditing fairness in ML benchmark datasets. 1.2 Outline The remainder of the paper is structured as follows. Section 2 introduces the problem of most influential sets and formalizes the setting. Section 3 presents our theoretical results on the distribution of maximal influence. Section 4 demonstrates the practical merits of our framework through simulations and empirical applications. Section 5 discusses implications, limitations, and future directions, and Section 6 concludes. 2 Preliminaries and Background Practitioners routinely encounter situations where small subsets of data points drive key conclusions. Consider the following scenarios: • Scientific discovery: Rugged terrain generally hinders economic development, but not in Africa. What if this striking result is driven by just two small island nations? • Fairness auditing: An algorithmic decision-making system produces vastly different outcomes for a protected group. What if this disparity is explained by only a handful of data points? • Data cleaning: A single influential data point among a thousand samples flips a strong correlation to a null result. Should we trust the original finding or the one without the outlier? • Data preprocessing: A microcredit experiment shows negligible outcome variations overall, except for a few outliers. How should we prepare and analyze the sample? At the core of these examples lie most influential sets , which exert disproportionate influence on an estimate or prediction. These sets are intuitive to interpret, directly tied to the quantity of interest, and provide a new dimension for assessing estimates by highlighting their support in the data. 2.1 Formal Problem Statement We consider a supervised learning task with input space \mathcal{X}\subset\mathbb{R}^{P} and target space \mathcal{Y}\subset\mathbb{R}. The goal is to learn a function f(\theta,\cdot):\mathcal{X}\mapsto\mathcal{Y} parameterized by \theta\in\mathbb{R}^{Q}. Given training data \left\{\left(x_{n},y_{n}\right)\right\}_{n=1}^{N} and a loss function \mathcal{L}\left(\cdot,\cdot\right), we learn parameters by solving \hat{\theta}=\underset{\theta\in\mathbb{R}^{Q}}{\arg\min}\sum_{n=1}^{N}\mathcal{L}\left(f(\theta,x_{n}),y_{n}\right). Let \left[N\right]=\left\{1,\dots,N\right\} and denote both an index set and its corresponding subsample as {\mathbb{S}}\subset\left[N\right]. For any subset {\mathbb{S}}, we use a subscript \hat{\theta}_{-{\mathbb{S}}} to denote a quantity \theta without {\mathbb{S}}, i.e. \hat{\theta}_{-{\mathbb{S}}}=\underset{\theta\in\mathbb{R}^{Q}}{\arg\min}\sum_{n\not\in{\mathbb{S}}}\mathcal{L}\left(f(\theta,x_{n}),y_{n}\right). Definition (Most Influential Set) . For a positive integer k\ll N, the k -most influential subset is {\mathbb{S}}_{k}^{\max}\coloneqq\underset{{\mathbb{S}}\subset\left[N\right],\lvert{\mathbb{S}}\rvert\leqslant k}{\arg\max}\Delta\left({\mathbb{S}};\phi\right), where \Delta\left({\mathbb{S}};\phi\right)=\phi(\hat{\theta})-\phi(\hat{\theta}_{-{\mathbb{S}}}) is the influence of subset {\mathbb{S}} on target function \phi:\mathbb{R}^{Q}\mapsto\mathbb{R}. The maximum influence is denoted as \Delta^{\max}=\Delta\left({\mathbb{S}}_{k}^{\max};\phi\right). Research Question. What is the probability distribution of \Delta^{\max}, and how can we use it to distinguish excessive influence from natural sampling variation? 2.2 Influence Functions vs. Exact Influence A common approach to study influence is via influence functions ( Fisher et al. 2023 ) . These are motivated by reweighing via the perturbation \hat{\theta}(\epsilon;{\mathbb{S}})\coloneqq\underset{\theta\in\mathbb{R}^{Q}}{\arg\min}\frac{1}{N}\sum_{n\not\in{\mathbb{S}}}\mathcal{L}\left(f(\theta,x_{n}),y_{n}\right)+\epsilon\sum_{i\in{\mathbb{S}}}\mathcal{L}\left(f(\theta,x_{i}),y_{i}\right). Setting \epsilon=0 recovers \hat{\theta}, while \epsilon=-N^{-1} yields \hat{\theta}_{-{\mathbb{S}}}. The influence function is the linear approximation at \epsilon=0 : \hat{\theta}(\epsilon;{\mathbb{S}})\approx\mathcal{I}({\mathbb{S}})\coloneqq\frac{\mathrm{d}\hat{\theta}(\epsilon;{\mathbb{S}})}{\mathrm{d}\epsilon}\bigg|_{\epsilon=0}. (1) This yields the following first-order estimate of influence: \Delta\left({\mathbb{S}};\phi\right)\approx-N^{-1}\frac{\mathrm{d}\phi\left(\hat{\theta}(\epsilon;{\mathbb{S}})\right)}{\mathrm{d}\epsilon}\bigg|_{\epsilon=0}. While influence functions are computationally convenient, they are unreliable even for simple models ( Basu et al. 2020 ; Hu et al. 2024 ; Huang et al. 2025 ; Koh et al. 2019 ) . In particular, they systematically underestimate the impact of (a) sets of data points and (b) highly influential data points. This occurs because the first-order approximation cannot reflect higher-order effects from the interplay between data points or differential leverage scores. Exact Influence In this work, we focus on the exact influence of subsets in a tractable but ubiquitous setting. We avoid linear approximations to accurately assess the most influential