Exploring Variability in Fine-Tuned Models for Text Classification with DistilBERT

Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar, Donald Cowan · 2025

Read Exploring Variability in Fine-Tuned Models for Text Classification with DistilBERT by Giuliano Lorenzoni, Ivens Portugal, Paulo Alencar, Donald Cowan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00241.txt.

Description

Giuliano Lorenzoni and co-authors present a study, published in 2024, of how shaky fine-tuned language models can be for sorting text. They worked with DistilBERT, a compact model, tuned for sentiment sorting of English text. Small choices in the tuning recipe can move scores more than users expect, which matters when models are compared or shipped.

Their structured tests varied learning rate, batch size, and training rounds, then modelled each factor's effect on accuracy, F1 score, and loss. A faster learning rate cut loss but fought against accuracy gains. Batch size moved accuracy and F1 clearly in one view yet faded in another. The paper maps these trade-offs for practitioners.

Catalog record

CREATORGiuliano Lorenzoni, Ivens Portugal, Paulo Alencar, Donald Cowan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00241

Browse the full library · About the Tower of Babel