Sentiment analysis for software engineering: How far can ZSL ( ZSL ) go?

Reem Alfayez, Manal Binkhonain · 2026

Read Sentiment analysis for software engineering: How far can ZSL ( ZSL ) go? by Reem Alfayez, Manal Binkhonain (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13826.txt.

Introduction

Alfayez and Binkhonain ask how far zero-shot learning can go in sentiment analysis for software engineering. Labeled datasets are scarce because labeling needs expertise and effort. They test four zero-shot styles: embedding, NLI, TARS, and generative, under varied label setups. They compare against fine-tuned transformer models. Pairing expert-curated labels with embedding or generative models reaches macro-F1 scores comparable to fine-tuned models. Errors come mainly from annotation subjectivity and polar facts.

Catalog record

CREATORReem Alfayez, Manal Binkhonain
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69002
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13826

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sentiment analysis for software engineering: How far can ZSL ( ZSL ) go? Reem Alfayez, Manal Binkhonain Abstract Context: Sentiment analysis in software engineering focuses on understanding emotions expressed in software artifacts. Previous research highlighted the limitations of applying general off-the-shelf sentiment analysis tools within the software engineering domain and indicated the need for specialized tools tailored to various software engineering contexts. The development of such tools heavily relies on supervised machine learning techniques that necessitate annotated datasets. Acquiring such datasets is a substantial challenge, as it requires domain-specific expertise and significant effort. Objective : This study explores the potential of ZSL ( ZSL ) to address the scarcity of annotated datasets in sentiment analysis within software engineering Method: We conducted an empirical experiment to evaluate the performance of various ZSL techniques, including embedding-based, NLI ( NLI )-based, TARS ( TARS )-based, and generative-based ZSL techniques. We assessed the performance of these techniques under different labels setups to examine the impact of label configurations. Additionally, we compared the results of the ZSL techniques with state-of-the-art fine-tuned transformer-based models. Finally, we performed an error analysis to identify the primary causes of misclassifications. Results: Our findings demonstrate that ZSL techniques, particularly those combining expert-curated labels with embedding-based or generative-based models, can achieve macro-F1 scores comparable to fine-tuned transformer-based models. The error analysis revealed that subjectivity in annotation and polar facts are the main contributors to ZSL misclassifications. Conclusion: This study demonstrates the potential of ZSL for sentiment analysis in software engineering. ZSL can provide a solution to the challenge of annotated dataset scarcity by reducing reliance on annotated dataset. Keywords: Sentiment Analysis, Software Engineering, Natural Language Processing, Zero-shot Learning, Text Classification 1 Introduction Over the years, sentiment analysis has evolved as a powerful tool for extracting subjective information from text data [ 43 ] . In software engineering, it provides insights into software development and usage by analyzing app reviews, developer communications, discussions on technical Q&A websites, and more [ 43 , 42 , 25 , 19 , 29 , 23 , 38 , 10 ] . Utilizing sentiment analysis within the software engineering domain presents significant challenges. General-purpose sentiment analysis tools often demonstrate suboptimal performance when used in software engineering contexts [ 17 , 35 ] . This limitation has led researchers to develop specialized tools tailored to the unique characteristics of software engineering contexts. Despite these efforts, it has been observed that even sentiment analysis tools that perform well in one software engineering context lack generalizability across different contexts, where they underperform in the new contexts. The context-bound limitation highlights the need for context-specific tools [ 24 , 19 , 25 ] . Developing context-based tools is not an easy feat, as most sentiment analysis tools in software engineering are based on supervised machine learning techniques that frame sentiment analysis tasks as text classification problems. These tools are heavily reliant on annotated datasets for training, which is costly, time-consuming, error-prone, and requires domain-specific expertise to obtain [ 24 , 19 , 42 , 18 ] . This reliance on annotated datasets is particularly problematic due to the context-bound limitation that requires the development of separate models and datasets for each specific context [ 24 , 19 ] . The need to curate annotated dataset for training in each context further intensifies the challenge of datasets scarcity and complicates the development of effective sentiment analysis models [ 24 , 19 , 42 ] . ZSL is a promising approach that has the potential to address the challenge of requiring context-specific training data for classification tasks. A ZSL -based model can classify data without prior exposure to specific labels, where it instead relies on its understanding of relationships between words, phrases, and concepts to make classifications. In the context of ZSL , a model generates predictions for tasks it has not been explicitly trained on by leveraging data from other, related tasks to aid its learning process. The model utilizes knowledge acquired through pre-training on other datasets and transfers relevant information to the new classification task [ 36 , 2 ] . This study explores the potential of ZSL for sentiment analysis in software engineering by evaluating embedding-based, NLI -based, TARS -based, and generative-based ZSL techniques across various contexts. We assess the impact of label configurations on model performance and compare the best-performing ZSL models with state-of-the-art fine-tuned transformer models. An error analysis is also conducted to understand misclassification causes. The rest of the paper is organized as follows: Section 2 summarizes ZSL for text classification. Section 3 reviews related studies. Section 4 describes the study setup. Section 5 presents the results, and Section 6 discusses them. Section 7 outlines potential validity threats, and Section 8 concludes the paper. 2 ZSL ( ZSL ) text classification ZSL text classification is a NLP ( NLP ) approach that enables models to classify text into unseen classes during training. As opposed to traditional supervised learning, which requires labeled data for each class, ZSL leverages transfer learning and semantic understanding to predict previously unseen classes. This capability is particularly useful when labeled data is scarce [ 2 , 36 ] . ZSL text classification can be achieved through four main techniques: embedding-based, NLI -based, TARS -based, and generative-based techniques. Below is a brief description of each technique. 2.1 Embedding-based ZSL Embedding-based ZSL text classification uses word embeddings to measure the semantic similarity between input text and potential class labels [ 39 ] . While the original approach [ 39 ] relied on skip-gram static embeddings, we opted for transformer-based LLM embeddings, as these models generate contextual word embeddings that capture both the syntactic and semantic properties of words, along with their context [ 2 , 36 , 3 ] . Figure 1 illustrates the process of embedding-based ZSL text classification. Both the input text and potential class labels are passed through a pre-trained LLM to generate embeddings. Classification is performed by calculating the cosine similarity between the input text embedding and each class label embedding. The class with the highest similarity score is then selected as the predicted label (i.e., label 1 in the example). Figure 1: An illustration of embedding-based ZSL 2.2 NLI ( NLI )-based ZSL NLI -based ZSL frames text classification as a textual entailment problem, where it determines whether a given text (i.e., premise) logically follows another hypothesis. The input text is treated as the NLI premise, and each candidate label forms a hypothesis. The model calculates probabilities for entailment and contradiction, which are then converted into label probabilities. The text is classified under the label with the highest entailment probability [ 41 , 2 ] . As Figure 2 presents, the input text serves as the premise, and potential class labels are hypotheses. The NLI model assesses whether the premise entails or contradicts each hypothesis and assigns a probability to each case. The input text is classified based on the label with the highest entailment probability, in this case, label 1. Figure 2: An illustration of NLI -based ZSL 2.3 TARS ( TARS )-based ZSL TARS formulates the classification task as a universal binary classification problem, where the model learns to predict whether a given text belongs to a particular

Read the complete text