Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions

Eve Fleisig, Matthias Orlikowski, Philipp Cimiano, Dan Klein · 2025

Read Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions by Eve Fleisig, Matthias Orlikowski, Philipp Cimiano, Dan Klein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08217.txt.

Description

Spam filters for human-labelled data quietly delete honest disagreement, warn Eve Fleisig and 3 co-authors in a 2025 machine learning paper. Datasets meant to capture diverse opinions must remove spam while keeping genuine variation in labels. Standard filtering heuristics were built for tasks with one true answer, where disagreement looks like noise.

On subjective tasks, the tested heuristics often remove annotators who disagree rather than actual spammers, trading label diversity away for a false sense of accuracy. The authors evaluate the tradeoffs empirically across filtering methods. Dataset builders for opinion, toxicity, and preference data face a design choice: aggressive cleaning can manufacture consensus and teach models that minority views are errors.

Catalog record

CREATOREve Fleisig, Matthias Orlikowski, Philipp Cimiano, Dan Klein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08217

Browse the full library · About the Tower of Babel