Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Models

Sercan Karakaş 1 and Yusuf Şimşek 2 · 2026

Read Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Models by Sercan Karakaş 1 and Yusuf Şimşek 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29614.txt.

Introduction

Karakaş and Şimşek ask whether supervised fine-tuning is still needed for Turkish sentiment analysis now that large language models can be prompted directly. Sentiment analysis classifies text as positive, negative, or neutral. They compare three families on a Turkish e-commerce review dataset: prompted large language models, fine-tuned pretrained language models, and classical machine learning methods.

Fine-tuned BERTurk models do best overall, beating every prompted large language model on the full three-class task. Neutral reviews are the main difficulty. Several large models compete closely on the simpler binary task of positive versus negative. In the three-class setting they degrade sharply, collapsing neutral reviews into polarized categories. The authors conclude that prompted models still fall short of supervised fine-tuning under zero-shot conditions, where no examples are given.

Catalog record

CREATORSercan Karakaş 1 and Yusuf Şimşek 2
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29614

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Models Sercan Karakaş 1 and Yusuf Şimşek 2 Abstract This study examines whether supervised fine-tuning remains necessary for Turkish sentiment analysis in the era of large language models. We compare classical machine learning methods, fine-tuned pretrained language models, and prompted large language models on a Turkish e-commerce review dataset with negative, neutral, and positive labels. Fine-tuned BERTurk models perform best overall and outperform all prompted large language models in the full three-class task. The neutral class emerges as the main difficulty: while several large language models are much more competitive in binary positive–negative classification, they degrade substantially in the three-class setting by collapsing neutral reviews into polarized categories. The findings suggest that, in realistic Turkish sentiment classification, prompted large language models do not yet match supervised fine-tuning in the zero-shot setting, and that including the neutral class is crucial for robust evaluation. I Introduction Sentiment analysis is a core NLP task for identifying opinions in text, with applications in reviews, feedback, and social media [ 1 , 2 ] . Realistic settings often require a neutral class in addition to positive and negative polarity, making classification more challenging [ 5 ] . This raises the question of whether supervised fine-tuning remains necessary, or whether instruction-following LLMs can classify sentiment reliably through zero- or few-shot prompting [ 3 , 4 ] . Prior work suggests that LLMs do not always outperform smaller specialized models, especially on fine-grained sentiment tasks [ 5 ] . This question is especially relevant for Turkish, where sentiment analysis resources remain more limited than for English [ 6 ] . Prior studies on Turkish have examined classical machine learning, transformer-based methods, tweet sentiment analysis, and targeted sentiment analysis, showing that supervised and BERT-based models can be highly effective when labeled data are available [ 7 , 8 , 9 , 10 ] . In that respect, in this study, our aim is to compare supervised models, fine-tuned pretrained models, and prompted LLMs on the same Turkish sentiment classification task, asking whether LLM performance remains competitive in a three-class setting that includes the harder neutral category [ 5 , 10 ] . II Related Work Turkish sentiment analysis has been studied at the document, sentence, aspect, and target levels, with prior work showing sensitivity to linguistic structure, evaluation granularity, and available supervision [ 11 , 6 ] . Recent resources such as TRSAv1 have expanded benchmark coverage for Turkish e-commerce reviews [ 12 ] . Across Twitter, targeted sentiment, and e-commerce settings, supervised transformer and BERT-based models generally outperform traditional baselines when labeled data are available [ 8 , 9 , 10 ] . This suggests that explicit supervision remains important for Turkish sentiment analysis, especially in task-specific and multi-class label settings. However, the rise of instruction-following LLMs has raised the possibility that zero- or few-shot prompting may reduce the need for specialized fine-tuning when annotation is costly or rapid deployment is required [ 3 , 4 ] . However, the evidence is mixed when the evaluation is sentiment-specific and carefully controlled. Zhang et al. provide a large-scale comparison across sentiment-analysis tasks and show that although LLMs can perform competitively, they do not consistently outperform smaller specialized models, especially on more complex or structured sentiment tasks [ 5 ] . In addition, work on instruction robustness shows that instruction-tuned models can be sensitive to prompt wording, with performance degrading under semantically equivalent but previously unseen phrasings [ 13 ] . This is directly relevant for prompt-based sentiment classification, where apparent gains may depend partly on prompt design rather than stable task competence. Our study builds on this literature by comparing traditional supervised models, fine-tuned pretrained models, and prompted LLMs on the same Turkish three-class sentiment classification task, with special attention to whether neutral instances expose limits of prompt-only approaches. III Method and Material III-A Dataset The dataset used in this study consists of user reviews collected from e-commerce websites. Data collection was carried out through web scraping using Selenium. In its initial form, the dataset contained 6,381 instances. Each record includes a unique identifier, the review text, and a sentiment label. The reviews were annotated into three sentiment categories by two annotators: negative, neutral, and positive. For modeling purposes, these classes were encoded numerically as 0, 1, and 2, respectively. The class distribution of the full dataset is presented in Table I . TABLE I: Class distribution in the dataset Label Number of Reviews 0 2416 1 1439 2 2526 The review field contains the user-generated text, whereas the label field represents the corresponding sentiment class. A sample of the dataset is shown in Table II . TABLE II: Dataset sample Id Yorum Etiket 11 bedenine göre küçük ve kumaşı kalitesiz geldi 0 2575 kendi bedenizden büyük alın dar kalitede orta seviyede 1 2644 rahat kullanışlı tavsiye ederim 2 III-B Preprocessing Several preprocessing steps were applied prior to model training. First, records with missing values were removed. The review texts were then converted to lowercase, and special characters, numerical expressions, and single-character tokens were eliminated. In the final preprocessing stage, Turkish stopwords were removed using the NLTK library. After preprocessing, the dataset size decreased from 6,381 to 6,367 instances. Of the 14 removed instances, 2 belonged to the negative class and 12 to the positive class. The processed dataset was then divided into training and test sets using an 80%/20% split. Accordingly, all experiments were evaluated on the same test partition of 1,274 instances. III-C Experimental Setup This study compares two main approaches to sentiment classification: supervised models and prompted large language models. All models were evaluated on the same test set in order to ensure a fair comparison across approaches. Model performance was measured using accuracy, precision, recall, and F1-score. III-D Supervised Models The supervised baselines include BERTurk 32k, BERTurk 128k Cased, Turkish ELECTRA, logistic regression, support vector machines (SVM), random forest, and naive Bayes. For BERTurk 32k, BERTurk 128k Cased, and Turkish ELECTRA, the same hyperparameter configuration was used in order to maintain comparability across transformer-based models. Specifically, the learning rate was set to 2\times 10^{-5} , the number of training epochs was 5, and the batch size was 32. For the classical machine learning models, textual inputs were represented using TF–IDF features, with the maximum number of features set to 5000. This configuration was used consistently for logistic regression, SVM, random forest, and naive Bayes. III-E Large Language Models The LLMs evaluated in this study are Gemma2:9B [ 16 ] , Gemma3:27B [ 17 ] , GPT-OSS:20B [ 18 ] , Llama 3.1:8B [ 19 ] , Magibu:11B [ 20 ] , and Qwen3:32B [ 21 ] . To ensure comparability, all LLMs were run under the same decoding settings. Specifically, the temperature parameter was fixed at 0.1 and the top_p value was set to 1. In all LLM experiments, the same prompt was used, and each model was asked to assign one of the sentiment labels to the given review text. The prompt employed in these experiments is shown in Fig. 1 . Using a shared prompt and identical decoding parameters allows for a more controlled comparison between prompted

Read the complete text