Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions
Eve Fleisig, Matthias Orlikowski, Philipp Cimiano, Dan Klein · 2025
Read Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions by Eve Fleisig, Matthias Orlikowski, Philipp Cimiano, Dan Klein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08217.txt.
Description
Spam filters for human-labelled data quietly delete honest disagreement, warn Eve Fleisig and 3 co-authors in a 2025 machine learning paper. Datasets meant to capture diverse opinions must remove spam while keeping genuine variation in labels. Standard filtering heuristics were built for tasks with one true answer, where disagreement looks like noise.
On subjective tasks, the tested heuristics often remove annotators who disagree rather than actual spammers, trading label diversity away for a false sense of accuracy. The authors evaluate the tradeoffs empirically across filtering methods. Dataset builders for opinion, toxicity, and preference data face a design choice: aggressive cleaning can manufacture consensus and teach models that minority views are errors.
Catalog record
| CREATOR | Eve Fleisig, Matthias Orlikowski, Philipp Cimiano, Dan Klein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08217