Feeding LLM Annotations to BERT Classifiers at Your Own Risk

Yucheng Lu, Kazimier Smith · 2025

Read Feeding LLM Annotations to BERT Classifiers at Your Own Risk by Yucheng Lu, Kazimier Smith (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15432.txt.

Description

A popular shortcut labels training text with a big language model, then teaches a small classifier on those labels. Lu and Smith test the bargain head-on, comparing small BERT-style classifiers trained on machine labels against ones trained on gold labels. The machine-taught models not only scored lower on accuracy, they also swung more between training runs and hit their ceiling earlier.

The authors read the pattern as error propagation: the teacher's non-random mistakes become the student's blind spots. Filtering by confidence and blending several models softened the damage but never erased it. They urge caution before using the shortcut in high-stakes classification work.

Catalog record

CREATORYucheng Lu, Kazimier Smith
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15432

Browse the full library · About the Tower of Babel