Sentiment Analysis in Software Engineering: Evaluating Generative Pre-trained Transformers

KM Khalid Saifullah, Faiaz Azmain, Habiba Hye · 2025

Read Sentiment Analysis in Software Engineering: Evaluating Generative Pre-trained Transformers by KM Khalid Saifullah, Faiaz Azmain, Habiba Hye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14692.txt.

Introduction

KM Khalid Saifullah, Faiaz Azmain, and Habiba Hye test whether chatty AI models read developer mood better than specialist tools. Their 2025 study pits bidirectional transformers such as BERT against a generative model, GPT-4o-mini, on sentiment in software engineering texts from GitHub, Jira, and Stack Overflow.

On the tidier GitHub and Jira sets, the fine-tuned generative model matched the specialists, scoring near the top on balanced measures. Stack Overflow broke the pattern: its skewed, jargon-heavy posts crushed the fine-tuned model, while the untuned one generalized far better. The authors draw a practical lesson: match the training strategy to the data's shape, since fine-tuning can sharpen or shackle depending on the corpus.

Catalog record

CREATORKM Khalid Saifullah, Faiaz Azmain, Habiba Hye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14692

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sentiment Analysis in Software Engineering: Evaluating Generative Pre-trained Transformers KM Khalid Saifullah, Faiaz Azmain, Habiba Hye Abstract Sentiment analysis plays a crucial role in understanding developer interactions, issue resolutions, and project dynamics within software engineering (SE). While traditional SE-specific sentiment analysis tools have made significant strides, they often fail to account for the nuanced and context-dependent language inherent to the domain. This study systematically evaluates the performance of bidirectional transformers, such as BERT, against generative pre-trained transformers, specifically GPT-4o-mini, in SE sentiment analysis. Using datasets from GitHub, Stack Overflow, and Jira, we benchmark the models’ capabilities with fine-tuned and default configurations. The results reveal that fine-tuned GPT-4o-mini performs comparable to BERT and other bidirectional models on structured and balanced datasets like GitHub and Jira, achieving macro-averaged F1-scores of 0.93 and 0.98, respectively. However, on linguistically complex datasets with imbalanced sentiment distributions, such as Stack Overflow, the default GPT-4o-mini model exhibits superior generalization, achieving an accuracy of 85.3% compared to the fine-tuned model’s 13.1%. These findings highlight the trade-offs between fine-tuning and leveraging pre-trained models for SE tasks. The study underscores the importance of aligning model architectures with dataset characteristics to optimize performance and proposes directions for future research in refining sentiment analysis tools tailored to the SE domain. I Introduction Sentiment analysis, a critical subfield of natural language processing (NLP), involves classifying text into sentiment polarities, such as positive, neutral, and negative. It has been widely studied across various domains, including software engineering (SE), where analyzing sentiments expressed in textual artifacts provides insights into developer interactions, issue resolution, and project dynamics. For instance, sentiments in GitHub pull requests, commit comments, and Stack Overflow discussions have been shown to influence collaboration and productivity in SE projects [ 1 ] [ 2 ] . Despite its significance, sentiment analysis in SE faces unique challenges, including technical jargon, domain-specific expressions, and informal communication patterns, which make general-purpose sentiment analysis tools inadequate [ 3 ] . The development of SE-specific sentiment analysis tools has attempted to address these challenges. Early tools like SentiStrength [ 4 ] and its SE-specific variant SentiStrength-SE [ 5 ] tailored lexicon-based approaches to the SE domain by incorporating domain-specific dictionaries and rules. Supervised learning tools such as Senti4SD and SentiCR further enhanced sentiment classification by leveraging n-gram features and machine learning techniques [ 5 ] [ 6 ] . However, these tools often struggled to handle nuanced or context-dependent language, highlighting the need for more sophisticated approaches. Recent advancements in NLP have introduced pre-trained transformer-based models, such as Bidirectional Encoder Representations from Transformers (BERT) [ 7 ] , RoBERTa [ 8 ] , and XLNet [ 9 ] , which leverage contextual word representations to significantly improve performance across various tasks. These models, pre-trained on large corpora, can be fine-tuned for downstream tasks like sentiment analysis, making them particularly attractive for SE applications [ 10 ] . Unlike earlier approaches, transformer models dynamically capture the context of words within sentences, addressing the limitations of static embeddings and lexicon-based methods [ 10 ] [ 7 ] . Building on these advancements, Zhang et al. demonstrated that fine-tuned transformer models significantly outperformed traditional SE-specific tools like SentiStrength-SE and Senti4SD, achieving improvements of up to 35.6% in macro-averaged F1 scores. [ 11 ] While Zhang et al.’s work laid the groundwork for transformer-based SA4SE, their study focused exclusively on bidirectional transformers, such as BERT and its variants, without considering generative transformer models like GPT (Generative Pre-trained Transformer). Generative transformers, characterized by their autoregressive architectures, have shown remarkable performance in generating coherent sequences and processing large-scale text data [ 10 ] . However, the comparative efficacy of bidirectional transformers and generative transformers in sentiment analysis for SE remains underexplored, leaving a critical gap in the understanding of how these architectures perform in domain-specific contexts. Our study is motivated by the need to address this gap and extend the findings of Zhang et al. Specifically, we aim to evaluate how bidirectional transformers like BERT compare to generative pre-trained transformers like GPT in sentiment analysis for SE tasks. By leveraging identical datasets and train-test splits as Zhang et al., we ensure methodological consistency and enable a robust comparative analysis. Our research focuses on three datasets: Jira issue comments, Stack Overflow comments, and GitHub pull-request and commit comments, with the latter considered gold standards due to their rigorously curated annotations and widespread adoption in SE sentiment analysis research. To facilitate this comparison, we formulated the following research question: RQ1: How do Bidirectional Transformers like BERT compare to Generative Pre-trained Transformers like GPT for sentiment analysis in software engineering? RQ2: How do fine-tuned and non-fine-tuned GPT models compare to each other? Addressing this research question is essential for advancing the application of NLP in SE. Understanding the relative strengths and limitations of these transformer architectures will inform researchers and practitioners about the best-suited models for sentiment analysis in SE contexts. Our study provides two key contributions: 1. Comparison of Bidirectional and Generative Transformers: This study systematically compares bidirectional (BERT) and generative (GPT-4o-mini) transformer models using identical experimental setups and datasets from prior work, providing a nuanced understanding of their performance on software engineering (SE) sentiment analysis tasks. 2. Demonstration of Accessibility and Simplicity: By incorporating both fine-tuned and default versions of OpenAI’s GPT-4o-mini model, this work highlights the practicality of using pre-trained generative models for SE sentiment analysis. The default model setup, requiring minimal configuration, offers a straightforward and accessible solution for researchers and practitioners. The remainder of this paper is organized as follows: Section II reviews related work on sentiment analysis tools and transformer models. Section III describes the datasets and methodologies used in this study. Section IV presents the experimental results and analysis. Section V discusses the implications of our findings and threats to validity. Finally, Section VI concludes the paper and outlines directions for future research. Through this study, we aim to bridge the gap in understanding the comparative efficacy of bidirectional and generative transformers in SA4SE, thereby contributing to the growing body of knowledge in this domain and providing actionable insights for future research and practical applications. II Related Works Sentiment analysis in software engineering (SE) has garnered significant attention as it offers valuable insights into various aspects of developer interactions, software quality, and team dynamics. Researchers have explored numerous approaches, tools, and datasets to address the unique challenges of sentiment analysis in SE contexts. This section reviews prior work in sentiment analysis for SE and pre-trained models for natural language processing (NLP), with a focus on the foundational contributions of the Zhang et al. study. [ 11 ] II-A Sentiment Analysis for Software Engineering The application of sentiment analysis in SE has primarily focused on understanding developer emotions, opinions, and sentiments expressed in textual artifacts such as bug reports, commit comments, and discussion forums. Early work in this domain demonstrated that sentiment plays a significant role in SE activities, influencing collaboration, task completion, and overall project outcomes [ 2 ] [ 3 ] . For instance, Guzman et al. [ 12 ] analyzed GitHub commit comments to study social factors affecting software development, while Ortu et al. [ 13 ] explored how sentiments in Jira issue comments impact project management. Given the

Read the complete text