State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?

2025

Read State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting? (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07989.txt.

Introduction

Taja Kuzman Pungersek and colleagues compare fine tuning against prompting for text classification in South Slavic languages. Smaller languages often lag behind English in language technology research. The team tests sentiment in parliamentary speeches, topic labels in news and speeches, and genre identification in web texts across several languages.

Large language models did well with no task specific training, often matching or beating fine tuned BERT style models. Their results were also similar across South Slavic languages and English. Still, the authors note slower speed, higher cost, and less predictable outputs, so fine tuned models remain practical for large scale annotation.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07989

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting? Abstract Until recently, fine-tuned BERT-like models provided state-of-the-art performance on text classification tasks. With the rise of instruction-tuned decoder-only models, commonly known as large language models (LLMs), the field has increasingly moved toward zero-shot and few-shot prompting. However, the performance of LLMs on text classification, particularly on less-resourced languages, remains under-explored. In this paper, we evaluate the performance of current language models on text classification tasks across several South Slavic languages. We compare openly available fine-tuned BERT-like models with a selection of open-source and closed-source LLMs across three tasks in three domains: sentiment classification in parliamentary speeches, topic classification in news articles and parliamentary speeches, and genre identification in web texts. Our results show that LLMs demonstrate strong zero-shot performance, often matching or surpassing fine-tuned BERT-like models. Moreover, when used in a zero-shot setup, LLMs perform comparably in South Slavic languages and English. However, we also point out key drawbacks of LLMs, including less predictable outputs, significantly slower inference, and higher computational costs. Due to these limitations, fine-tuned BERT-like models remain a more practical choice for large-scale automatic text annotation. Keywords: LLM evaluation, text classification, large language models, South Slavic languages, sentiment identification, topic classification, genre identification State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting? Taja Kuzman Pungeršek ∗ , Peter Rupnik ∗ , Ivan Porupski ∗ , Vuk Dinić ∗ , Nikola Ljubešić ∗ † ‡ ∗ Jožef Stefan Institute; † Faculty of Computer and Information Science, University of Ljubljana; ‡ Institute of Contemporary History; Ljubljana, Slovenia {taja.kuzman, peter.rupnik, ivan.porupski, vuk.dinic, nikola.ljubesic}@ijs.si Abstract content 1. Introduction Dataset Lang # Instances # Labels % Most and Least Frequent Label Sentiment classification in parliamentary speeches ParlaSent-EN-test EN 2600 3 40.8% (Neutral), 26.8% (Positive) ParlaSent-HR-test HR 1336 3 41.9% (Negative), 17.2% (Positive) ParlaSent-SR-test SR 1074 3 46.2% (Negative), 17.6% (Positive) ParlaSent-BS-test BS 190 3 47.9% (Negative), 14.7% (Positive) Genre classification in web texts EN-GINCO EN 272 8 23.5% (Information/Explanation), 0.4% (Legal) X-GINCO-SL SL 80 8 15% (Prose/Lyrical), 8.8% (Opinion/Argumentation) X-GINCO-HR HR 80 8 16.3% (Promotion), 7.5% (Instruction) X-GINCO-MK MK 80 8 15% (News), 1% (Opinion/Argumentation) Topic classification in news articles IPTC-test-HR HR 291 17 11.0% (Economy), 3.8% (Conflict, War and Peace) IPTC-test-SL SL 282 17 10.6% (Society), 3.2% (Conflict, War and Peace) Topic classification in parliamentary speeches ParlaCAP-test-EN EN 876 22 6.4% (Law and Crime), 2.1% (Culture) ParlaCAP-test-HR HR 869 22 8.5% (Government Operations), 1.7% (Immigration) ParlaCAP-test-SR SR 874 22 7.1% (Government Operations), 1.7% (Immigration) ParlaCAP-test-BS BS 824 22 10.4% (Other), 0.5% (Culture) Table 1 : Information on test datasets in English (EN), Croatian (HR), Serbian (SR), Bosnian (BS), Slovenian (SL), and Macedonian (MK). Until recently, the dominant approach for text classification tasks relied on fine-tuning BERT-like transformer models on thousands of manually-annotated training examples. Recently, however, the field has shifted with the development of instruction-tuned decoder-only transformer models. These models, also commonly referred to as large language models (LLMs), which were originally developed primarily for text generation tasks, have demonstrated remarkable capabilities across a broad range of natural language processing (NLP) tasks, including text classification ( 27 ; 12 ) . In this paper, we focus on South Slavic languages, where research on text classification tasks included in our study has, until recently, been limited or even non-existent ( 19 ; 38 ; 26 ) . We take a first step toward systematically evaluating the current state of the art for text classification in these languages. Our evaluation is based on three text classification tasks in three different domains for which manually-annotated test datasets in South Slavic languages and fine-tuned BERT-like classifiers are freely available: sentiment classification of parliamentary speeches, topic classification in news articles, topic classification in parliamentary speeches, and automatic genre identification in web texts. These tasks span different domains and language styles, allowing for a comprehensive analysis of the performance of transformer-based models on text classification tasks. Specifically, we compare the performance of openly available fine-tuned BERT-like models with the zero-shot capabilities of both open-source and closed-source LLMs used via prompting. An important aspect of our study is to examine whether the performance of multilingual models on South Slavic languages is on par with their performance on English. This question is particularly relevant given that the evaluated large language models have been predominantly pretrained and instruction-tuned on English data. By evaluating various models on a selection of text classification tasks in English and various South Slavic languages, we set out to test the following two hypotheses that are based on previous experiments with fine-tuned BERT-like models and LLMs on automatic genre identification ( 27 ) , news topic classification ( 26 ) and sentiment analysis in parliamentary texts ( 37 ) : H1 Zero-shot prompting with instruction-tuned large language models (LLMs) can achieve results comparable to the use of BERT-like models fine-tuned on training data that are similar to the test data. H2 The performance of LLMs used in a zero-shot setup on text classification tasks on South Slavic test datasets is comparable to the performance on English test datasets. 2. Related Work After the introduction of transformer architectures, BERT (bidirectional encoder representations from transformers) models have achieved state-of-the-art results in text classification tasks, outperforming earlier non-neural approaches, such as support vector machines (SVMs). They have also demonstrated strong cross-lingual zero-shot capabilities in various classification tasks, including automatic genre identification ( 19 ) , news topic classification ( 44 ; 6 ) , and sentiment classification ( 38 ) . However, these models still require fine-tuning on a training dataset, developed during manual annotation campaigns that are time-consuming and costly. Instruction-tuned decoder-only transformer models, commonly referred to as large language models (LLMs), have recently shown strong performance in a range of classification tasks, even in zero-shot prompting setups that require no training data ( 27 ; 29 ; 12 ) . They have achieved promising results on various natural language processing tasks, including stance detection ( 51 ) , implicit hate speech categorization ( 12 ) , news topic classification ( 26 ) , automatic genre identification ( 27 ) , causal commonsense reasoning ( 30 ) , and machine translation ( 11 ) . Due to their promising performance, researchers have even started using them as data annotators, either by generating text and labels ( 33 ) or by annotating pre-existing texts ( 26 ) . Despite the growing interest in this topic, the majority of evaluations of LLMs used in text classification tasks are limited only to English ( 48 ; 52 ; 16 ; 53 ) . Systematic multilingual evaluations, especially which would include less-resourced languages such as those in the South Slavic group remain limited. Our work addresses this gap by providing a comparative evaluation of open-source and closed-source LLMs with openly-available fine-tuned BERT-like models across four benchmark families comprising three diverse classification tasks and three different domains in South Slavic languages and English. 3. Benchmarks The benchmarks (evaluation datasets) used in this study cover three text classification tasks, namely, sentiment identification, topic classification, and automatic genre identification, and three domains: parliamentary speeches, news articles and web texts. An overview of the datasets is

Read the complete text