Exploring Variability in Fine-Tuned Models for Text Classification with DistilBERT
Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar, Donald Cowan · 2025
Read Exploring Variability in Fine-Tuned Models for Text Classification with DistilBERT by Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar, Donald Cowan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00241.txt.
Description
Giuliano Lorenzoni and co-authors present a study, published in 2024, of how shaky fine-tuned language models can be for sorting text. They worked with DistilBERT, a compact model, tuned for sentiment sorting of English text. Small choices in the tuning recipe can move scores more than users expect, which matters when models are compared or shipped.
Their structured tests varied learning rate, batch size, and training rounds, then modelled each factor's effect on accuracy, F1 score, and loss. A faster learning rate cut loss but fought against accuracy gains. Batch size moved accuracy and F1 clearly in one view yet faded in another. The paper maps these trade-offs for practitioners.
Catalog record
| CREATOR | Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar, Donald Cowan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00241