Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts

Julius Neumann, Robert Lange, Yuni Susanti, Michael Färber · 2025

Read Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts by Julius Neumann, Robert Lange, Yuni Susanti, Michael Färber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04982.txt.

Introduction

Short texts give sentiment classifiers little context. Their labels are often skewed, and the mood they record is a matter of taste. Julius Neumann, lead author with three co-authors, study how to tune small transformer models for multi-label sentiment analysis in a 2025 paper. They test BERT-style models below a billion parameters under those constraints.

Three factors are tested: more domain pretraining, generative data augmentation, and classification head design. Added data improves results in their experiments. More pretraining on that data can add noise instead of accuracy. Head changes give only small benefits. The findings offer practical guidance for teams building sentiment models on limited resources.

Catalog record

CREATORJulius Neumann, Robert Lange, Yuni Susanti, Michael Färber
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-04982

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts Julius Neumann, Robert Lange, Yuni Susanti, Michael Färber Abstract Sentiment classification in short text datasets faces significant challenges such as class imbalance, limited training samples, and the inherent subjectivity of sentiment labels—issues that are further intensified by the limited context in short texts. These factors make it difficult to resolve ambiguity and exacerbate data sparsity, hindering effective learning. In this paper, we evaluate the effectiveness of small Transformer-based models (i.e., BERT and RoBERTa, with fewer than 1 billion parameters) for multi-label sentiment classification, with a particular focus on short-text settings. Specifically, we evaluated three key factors influencing model performance: (1) continued domain-specific pre-training, (2) data augmentation using automatically generated examples, specifically generative data augmentation , and (3) architectural variations of the classification head. Our experiment results show that data augmentation improves classification performance, while continued pre-training on augmented datasets can introduce noise rather than boost accuracy. Furthermore, we confirm that modifications to the classification head yield only marginal benefits. These findings provide practical guidance for optimizing BERT-based models in resource-constrained settings and refining strategies for sentiment classification in short-text datasets. † † paperid: 8 1 Introduction In modern practical decision-making, building predictive models often involves characterized by significant challenges, including imbalanced class distributions, highly variable dataset sizes—from limited to extremely large—and inherent subjectivity in the ground truth labels. These factors can severely degrade model performance and further complicate learning. Consequently, developing novel learning methods capable of addressing these difficulties is a critical area of ongoing research. Within this context, multi-label sentiment classification in short texts represents a particularly challenging task, requiring models to infer subjective emotional states (e.g., anger, joy, or surprise) from sparse and ambiguous linguistic cues. Short texts inherently limit contextual information, reducing the availability of disambiguating semantic cues and amplifying the effects of data sparsity. Additional common issue in this task includes inherently subjective nature of sentiment annotations, which often results in low inter-annotator agreement. Furthermore, short texts—especially from social media—frequently contain informal language, abbreviations, emojis, and sarcasm, further complicating accurate sentiment detection. Addressing these compounded difficulties is essential for advancing sentiment analysis and related Natural Language Processing (NLP) tasks and applications. Although pre-trained Transformer-based models such as BERT [ 4 ] have significantly advanced sentiment analysis [ 1 , 4 , 8 ] , critical challenges remain, including handling class imbalance, data scarcity, and subtle semantic cues within short texts. Recent research has shown that expanding training data through generative data augmentation [ 2 ] , refining model architectures [ 14 ] , and using domain-specific pre-training [ 7 ] potentially enhance the model’s performance. However, only few studies systematically evaluate these approaches in combination. Existing research predominantly focuses on binary sentiment classification or assumes access to large, well-curated datasets, leaving open questions about the optimal strategies for multi-label emotion detection in small, noisy and inherently subjective text corpora. In this paper, we address these challenges through a comprehensive evaluation of the following three key strategies: (1) continued domain-specific pre-training to better capture linguistic nuances, (2) generative data augmentation (GDA) to alleviate data scarcity, and (3) classification head variations to enhance parameter efficiency and adaptability. Our experiments were carried out on 2,768 short English texts from the SemEval 2025 Task 11 corpus [ 3 ] , utilizing small Transformer-based language models, specifically BERT [ 4 ] and its variant RoBERTa [ 8 ] . In this work, we define small as language models with fewer than 1 billion parameters, following [ 17 ] . Such models offer a favorable trade-off between performance and computational efficiency, making them well-suited for deployment in resource-constrained real-world scenarios. Additionally, we apply SHAP [ 9 ] analysis to improve the interpretability of the predictions. The following summarizes our main contributions: • We systematically evaluate three strategies—continued domain-specific pre-training, generative data augmentation, and classification head variations—for multi-label sentiment classification in short texts, targeting challenges of linguistic sparsity, data scarcity, and model efficiency. We release the code and dataset on Github. 1 1 1 https://github.com/faerber-lab/shorttext-sentiment-transformers • Our results show that moderate generative data augmentation improves BERT-based model performance, while excessive domain-specific pre-training can add noise. Variations in classification heads yield minimal gains. These findings provide practical guidance for BERT optimization in real-world, short-text datasets. • We employ SHAP analysis to interpret model predictions, offering insights into feature contributions and enhancing model transparency in short-text sentiment classification tasks. 2 Related Work Transformer-Based Text Classification. Transformer-based models such as BERT [ 4 ] have become the state of the art for text classification, largely due to their ability to encode contextualized representations with bidirectional attention [ 1 ] . Compared to earlier architectures like LSTMs, Transformers more effectively capture long-range semantic dependencies. Beyond model architecture, the quantity of available training data strongly influences performance. Ji et al. 2023 Ji et al. 2023 and Mehrafarin et al. 2022 Mehrafarin et al. 2022 show that larger training sets generally lead to improved generalization in NLP tasks. BERT Devlin et al. 2019 and its refined variant RoBERTa Liu et al. 2019 are often the starting points for a wide range of downstream applications. RoBERTa’s removal of next sentence prediction and its employment of extended pre-training with dynamic masking enhance performance on multiple benchmarks Liu et al. 2019 ; Hirlea et al. 2021 . In typical classification scenarios, the hidden representation of the [CLS] token is passed through a classifier, and both the Transformer layers and the classifier head are fine-tuned to maximize the probability of the correct label. Adjustments in the head’s architecture, such as varying embedding size or adopting parameter-efficient layers like Projected Attention Stickland and Murray 2019 , can further enhance performance Wolfe and Lundgaard 2021 . Data Augmentation. Data augmentation has been widely adopted to improve model generalization in NLP tasks. It typically involves selecting or creating meaningful samples, such as work by Susanti 2024 or adding extra information Guo and Yu 2022 . While adversarial approaches have been explored Guo and Yu 2022 ; Qu et al. 2020 , their complexity and mixed empirical results have prompted interest in simpler, generative methods. Balkus and Yan 2024 Balkus and Yan 2024 propose a more straightforward strategy, using GPT-3 to generate new training examples and filtering them to retain only the most promising candidates. Even with a dataset of only 26 examples and two labels, they report substantial accuracy improvements, but also observe that excessive augmentation can introduce noise and ultimately degrade performance Nguyen et al. 2022 . Generative Data Augmentation. Although generative data augmentation (GDA) has been explored in previous work Rogulsky et al. 2024 , we adapt it here for multi-label sentiment classification on short texts. In contrast to Unsupervised Data Augmentation (UDA) Xie et al. 2020 , which emphasizes consistency training on unlabeled data, we directly generate new examples using a state-of-the-art language model (e.g., GPT-4o) and incorporate them into the training set without additional filtering. We refer to this approach as Generative Data Augmentation (GDA) . By verifying that the synthetic samples remain domain-consistent, we aim to capture rare sentiment cues and edge cases often missing from the original dataset. Similar to prior studies, we hypothesize that augmenting too aggressively

Read the complete text