Feeding LLM Annotations to BERT Classifiers at Your Own Risk
Yucheng Lu, Kazimier Smith · 2025
Read Feeding LLM Annotations to BERT Classifiers at Your Own Risk by Yucheng Lu, Kazimier Smith (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15432.txt.
Description
A popular shortcut labels training text with a big language model, then teaches a small classifier on those labels. Lu and Smith test the bargain head-on, comparing small BERT-style classifiers trained on machine labels against ones trained on gold labels. The machine-taught models not only scored lower on accuracy, they also swung more between training runs and hit their ceiling earlier.
The authors read the pattern as error propagation: the teacher's non-random mistakes become the student's blind spots. Filtering by confidence and blending several models softened the damage but never erased it. They urge caution before using the shortcut in high-stakes classification work.
Catalog record
| CREATOR | Yucheng Lu, Kazimier Smith |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15432