Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds
Michael-Andrei Panaitescu-Liess, Yigitcan Kaya *, Sicheng Zhu, Furong Huang, Tudor Dumitras · 2025
Read Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds by Michael-Andrei Panaitescu-Liess, Yigitcan Kaya *, Sicheng Zhu, Furong Huang, Tudor Dumitras (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02142.txt.
Introduction
Michael-Andrei Panaitescu-Liess and four co-authors show that two good aims in machine learning can clash head-on. One aim lifts rare groups that models often fail. The other aim weeds poisoned samples out of training data. Their 2025 work finds both aims lean on the same trick: flag the odd samples.
Fairness tools boost odd samples, so they boost poison too, and attack success can soar from nil to near total. Poison guards cut odd samples, so they cut rare groups too, and group scores slide. A math proof backs the clash for a standard flag rule. Tries to blend both fixes fell short in their tests. The pair of results warns the field to weigh trade-offs, not just chase scores.
Catalog record
| CREATOR | Michael-Andrei Panaitescu-Liess, Yigitcan Kaya *, Sicheng Zhu, Furong Huang, Tudor Dumitras |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-02142
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds Michael-Andrei Panaitescu-Liess, Yigitcan Kaya *, Sicheng Zhu, Furong Huang, Tudor Dumitras Abstract Group robustness has become a major concern in machine learning (ML) as conventional training paradigms were found to produce high error on minority groups. Without explicit group annotations, proposed solutions rely on heuristics that aim to identify and then amplify the minority samples during training. In our work, we first uncover a critical shortcoming of these methods: an inability to distinguish legitimate minority samples from poison samples in the training set. By amplifying poison samples as well, group robustness methods inadvertently boost the success rate of an adversary—e.g., from 0 % 0\% without amplification to over 97 % 97\% with it. Notably, we supplement our empirical evidence with an impossibility result proving this inability of a standard heuristic under some assumptions. Moreover, scrutinizing recent poisoning defenses both in centralized and federated learning, we observe that they rely on similar heuristics to identify which samples should be eliminated as poisons. In consequence, minority samples are eliminated along with poisons, which damages group robustness—e.g., from 55 % 55\% without the removal of the minority samples to 41 % 41\% with it. Finally, as they pursue opposing goals using similar heuristics, our attempt to alleviate the trade-off by combining group robustness methods and poisoning defenses falls short. By exposing this tension, we also hope to highlight how benchmark-driven ML scholarship can obscure the trade-offs among different metrics with potentially detrimental consequences. 1 Introduction As ML finds adaptations in many fields with diverse priorities, new metrics of success, aside from prediction performance (e.g., accuracy), have come into play. For example, in security-critical applications, robustness to adversarial examples ( Chen et al., 2021 ) or poisoning attacks ( Steinhardt et al., 2017 ) ; or, in demographically-sensitive applications, fairness ( Hashimoto et al., 2018 ) or group robustness ( Liu et al., 2021 ) are popular metrics the ML community aims to improve. The sheer number of such metrics has led to a paradigm where researchers demonstrate progress on benchmarks often designed with a single metric in mind. Recent work has exposed previously unknown trade-offs between some of these metrics, e.g., between privacy and fairness ( Bagdasaryan et al., 2019 ) or between robustness and privacy ( Song et al., 2019 ) . As applications in practice require balancing various, often mission-critical, metrics, such unknown trade-offs might have catastrophic consequences. This makes the research into studying the intersection of multiple metrics to identify tensions and interactions particularly crucial. With this motivation, in a systematical quantitative study, our work uncovers an inherent tension between approaches designed for two critical metrics: group robustness methods and poisoning defenses. Group robustness has become a concern as standard training via empirical risk minimization (ERM) has been shown to perform well on an average sample but poorly on samples belonging to under-represented, minority groups ( Tatman, 2017 ) . Effective solutions, such as minority upsampling ( Byrd & Lipton, 2019 ) , are not always feasible as the explicit group annotations they rely on are often not available due to privacy (e.g., demographic annotations) or financial concerns (e.g., large-scale data sets). To this end, research has proposed heuristics for identifying the minority training samples as a proxy for annotations ( Liu et al., 2021 ) . A common observation behind these heuristics is that minority samples are often difficult to learn and the model cannot achieve low training error on them. The candidates identified as minority samples are then amplified during training, e.g., through upsampling, which is shown to improve group robustness significantly. First, we expose a vulnerability (Figure 1 ): when the training set contains poison samples, group robustness heuristics cannot distinguish legitimate minority samples from them. Poison samples are injected by an attacker to teach the model an undesirable behavior, e.g., a backdoor ( Saha et al., 2020 ) . As a result, two recent group robustness methods ( Sohoni et al., 2020 ; Liu et al., 2021 ) end up assisting the attacker by encouraging low error on poison samples along with minority samples—attacker achieves 15-97\% higher success rate due to amplification. Aiming to understand this vulnerability, we observe that poison samples can be as difficult to learn as minority samples, especially in a realistic attack that can inject only a few samples. This suggests that any group robustness method that relies on difficulty-based heuristics might carry a similar vulnerability. To supplement our empirical results, we prove that under specific assumptions, loss-based difficulty heuristics cannot distinguish between legitimate minority samples and poisons. Second, on the other side of the coin, we show that poisoning defenses hurt group robustness when the training set contains legitimate minority samples. In particular, we focus on recent sanitization-based defenses in centralized ( Yang et al., 2022 ) and federated learning ( Panda et al., 2022 ) settings. As it is challenging to detect poisons reliably ( Shan et al., 2022 ) , these methods pursue a simpler goal by relying on heuristics to identify outliers. Such heuristics are empirically shown to eliminate poisons without hurting the overall accuracy. However, due to providing distinct learning signals during training, we observe that minority samples are often inadvertently identified (and eliminated) as outliers. This poses a trade-off for the defender: the more poisons are eliminated (lower attack success), the more minority samples will be eliminated as well (lower group robustness). In consequence, the accuracy on the minority group drops by up to 15\% after applying an effective defense. Finally, we make an (unsuccessful) attempt to mitigate these tensions by applying poisoning defenses and group robustness methods in tandem. When aiming for low attack success, the defense removes enough minority samples to render group robustness methods ineffective. When aiming for high group robustness, the defense ignores enough poison samples that are still amplified by group robustness, which leads to high attack success. Ultimately, this implies an unintended alignment between defensive and group robustness heuristics. We hope to encourage future work to develop heuristics that conciliate these two critical success metrics. In summary, we make the following contributions: (i) We show experimentally that group robustness methods fail to distinguish minority groups from poisons, which leads to the risk of amplifying the attacks, and complement our findings with a theoretical result (Section 4 ); (ii) We find that poisoning defenses also fail to distinguish poisons from under-represented groups and they introduce a risk of lower group robustness in both centralized and federated learning (Section 5 and Appendix A.4 ); (iii) We demonstrate that a straightforward combination of group robustness methods and poisoning defenses cannot fully mitigate these tensions (Appendix A.5 ). Figure 1: Illustration of group robustness methods without ( left ) and with poisons ( right ) in the training set. For the former, the methods operate regularly as they identify and amplify the minority groups, but for the latter, they also amplify poisons and, therefore, the attacker’s influence over the decision boundary. The left and the right panels of each figure correspond respectively to the identification phase of the group robustness methods and the resulting model trained after this phase. Note