Data Filtering Methods for Training Language Models

E. Shevchenko and E. Bruches · 2026

Read Data Filtering Methods for Training Language Models by E. Shevchenko and E. Bruches (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29807.txt.

Description

Egor Shevchenko and Elena Bruches compare two ways to catch label errors in Russian training data. Label noise, even in benchmarks, drags down how well models generalize. They test two filters, Confident Learning and Dataset Cartography, across three Russian corpora of varied size, classes, and domain, using a fine-tuned Russian BERT. Filtering helps most on small, noisy sets, where Confident Learning lifts macro F1 markedly, while on large, clean sets it adds little. Targeted removal beats random removal, confirming that both filters find real errors, not smaller data.

Catalog record

CREATORE. Shevchenko and E. Bruches
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38695
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29807

Browse the full library · About the Tower of Babel