Text Classification in the LLM Era - Where do we stand?
Sowmya Vajjala, Shwetali Shimangaud · 2025
Read Text Classification in the LLM Era - Where do we stand? by Sowmya Vajjala, Shwetali Shimangaud (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11830.txt.
Introduction
Sowmya Vajjala and Shwetali Shimangaud ask in 2025 where big language models leave the older craft of sorting texts. Sorting text into bins, like spotting praise from complaint, is a bread-and-butter task. They compared model styles across thirty-two data sets in eight languages: no-example use of big models, light tuning of small ones, and sorters trained on model-made fake data.
Full hand-labeled training set the bar. No-example big models did well on feeling sorts but lost elsewhere, they report. Fake data pooled from several big models built better sorters than the big models' own guesses. Gaps across languages ran wide in every style. For builders, the moral is to test per language and task, not to crown one method.
Catalog record
| CREATOR | Sowmya Vajjala, Shwetali Shimangaud |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11830
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text Classification in the LLM Era - Where do we stand? Sowmya Vajjala, Shwetali Shimangaud Abstract Large Language Models revolutionized NLP and showed dramatic performance improvements across several tasks. In this paper, we investigated the role of such language models in text classification and how they compare with other approaches relying on smaller pre-trained language models. Considering 32 datasets spanning 8 languages, we compared zero-shot classification, few-shot fine-tuning and synthetic data based classifiers with classifiers built using the complete human labeled dataset. Our results show that zero-shot approaches do well for sentiment classification, but are outperformed by other approaches for the rest of the tasks, and synthetic data sourced from multiple LLMs can build better classifiers than zero-shot open LLMs. We also see wide performance disparities across languages in all the classification scenarios. We expect that these findings would guide practitioners working on developing text classification systems across languages. 1 Introduction Text classification is one of the evergreen problem in NLP and other related areas of research, with widespread applications across different real-world use cases and disciplines of study. Each classification use case is different, and collecting sufficient labeled data for each problem can be challenging. This resulted in the interest in the development of zero-shot text classification systems Yin et al. (2019) . Large Language Models can offer a solution, and their use as zero-shot (English) text classifiers ( Gretz et al., 2023 ) has been explored in recent past. Synthetic data generation with LLMs has also been proposed to address the labeled data scarcity. Different from these two approaches, there is an established body of work on few-shot fine-tuning ( Tunstall et al., 2022 ; Yehudai and Bandel, 2024 , e.g.,) , to address situations where we have a very small amount of labeled data, which is not typically sufficient to build a classifier using standard methods. How does zero-shot text classification compare with few-shot fine-tuning, synthetic data based classification, and building classifiers with full labeled datasetshough some parts of this broad question received attention in the recent past, a more comprehensive comparison is lacking in NLP research. Further, most of the past research in this direction has focused on English datasets and proprietary LLMs. A detailed comparison across different classification methods spanning more languages and datasets will not only help us understand the state of the art in text classification with LLMs, but also provide guidance to practitioners looking at solving real-world text classification use cases across various tasks and languages. We address these issues in this paper. Concretely, we study the following research questions in this paper, considering 32 datasets covering 8 languages (4 datasets per language). 1. How well does zero-shot prompting of LLMs (open and proprietary) fare compared to building classifiers with full training data? 2. Does few-shot fine-tuning offer any benefits over zero-shot classification? 3. How well does a synthetic data based classifier fare compared to zero-shot classification with LLMs? 4. Is supervised/instruction fine-tuning of LLMs the way to go for text classification? Starting with an overview of related work (Section 2 ), we proceed to the description of our methodology (Section 3 ) and discuss the details about the results (Section 4 ). After summarizing the main conclusions (Section 5 ), we discuss the limitations (Section 6 ) and broader impacts (Section 7 ). 2 Related Work Text classification, the task of classifying a given text into a pre-defined list of categories, is a well-studied problem. From bag-of-words features to the current state-of-the-art LLMs, numerous approaches have been explored in the past. Access to large amounts of human labeled data has traditionally played a significant role in improving text classifiers, and NLP research in the past two decades addressed this issue by looking at different solutions to learn from little or no labeled data. Zero-shot Pre-LLM Approaches: Some of the earlier classification approaches relied on using only label names to build "data less" text classifier Liu et al. (2004) ; Li and Yang (2018) ; Meng et al. (2020) ; Ye et al. (2020) ; Gera et al. (2022) and embedding the texts and labels in a shared space Song and Roth (2014) ; Luo et al. (2021) ; Chu et al. (2021) ; Sarkar et al. (2022) ; Gao et al. (2023) ; Wang et al. (2023) . Yin et al. (2019) proposed to formulate zero-shot text classification as a textual entailment problem, although Ma et al. (2021) point to the limitations of this approach in terms of variability across datasets and reliance on spurious lexical patterns. Another practical approach for zero-shot classification is cross-lingual transfer i.e., train a classification model in one or more languages, and use it as a zero-shot classifier on the target language Wang and Banko (2021) . Except ( Wang and Banko, 2021 ) , who studied sentiment and hate speech classification tasks, all the research has focused primarily on English datasets. Few-shot fine-tuning: Approaches that can learn from a small amount of ( <20 samples per category) labeled examples have also been explored in the recent years Schick et al. (2020) ; Dopierre et al. (2021) ; Ohashi et al. (2021) ; Zhang et al. (2022) . SetFit Tunstall et al. (2022) introduced an approach based on supervised contrastive learning, transforming a language model into a topic encoder using only a few examples per label, and demonstrated effectiveness with datasets where the number of categories are low (under 5). FastFit Yehudai and Bandel (2024) proposed an approach that scales to many classes (50–150) effectively, and showed its usefulness with English datasets. Out of these only SetFit evaluated with a few non-English datasets. Zero-shot Classification with LLMs: With the arrival of Large Language Models, some recent approaches explored proprietary models like GPT3.5 and GPT4 for zero-shot or few-shot in-context learning for text classification across several datasets Gretz et al. (2023) ; Sun et al. (2023) ; Mozes et al. (2023) ; Tian and Chen (2024) . Extending this line of work, open LLMs were studied in the context of intent classification Ruan et al. (2024) ; Arora et al. (2024) and computational social science Mu et al. (2024) . However, comparing such zero-shot approaches with few-shot and full-data based fine-tuning, Edwards and Camacho-Collados (2024) show that smaller, fine-tuned classifiers outperform zero-shot approaches. Whether supervised fine-tuning of LLMs offers any benefit is an unexplored question. Surprisingly, except ( Tian and Chen, 2024 ) , all these experiments have been focused only on English datasets so far. We expand this strand of work to 7 other languages, and provide more detailed comparisons across different LLMs. Synthetic Data: One approach to address the labeled data problem is to augment existing data by creating new data by applying text transformations such as replacing synonyms, paraphrasing, back translation etc. Bayer et al. (2022) presents a detailed survey of such data augmentation techniques for text classification. An extension of this idea is to directly synthesize the labeled data using generative language models Yu et al. (2023) ; Yue et al. (2023) ; Kurakin et al. (2023) ; Choi et al. (2024) . In the recent past, Large Language Model based synthetic data generation is increasingly observed across different NLP tasks Tan et