Where are the Hidden Gems? Applying Transformer Models for Design Discussion Detection
Lawrence Arkoh, Daniel Feitosa, Wesley K. G. Assunção · 2026
Read Where are the Hidden Gems? Applying Transformer Models for Design Discussion Detection by Lawrence Arkoh, Daniel Feitosa, Wesley K. G. Assunção (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18393.txt.
Introduction
Arkoh, Feitosa, and Assunção test transformer models for detecting design discussions in software artifacts. Design choices hide in forums, pull requests, issues, and commit messages across a project's life. Labeled data is scarce, so past work trained across domains with classic classifiers. They fine-tune BERT, RoBERTa, XLNet, and two chat models on Stack Overflow and test on GitHub. BERT and RoBERTa recall well across domains; ChatGPT-4o-mini leads overall recall.
Catalog record
| CREATOR | Lawrence Arkoh, Daniel Feitosa, Wesley K. G. Assunção |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55912 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18393
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Where are the Hidden Gems? Applying Transformer Models for Design Discussion Detection Lawrence Arkoh, Daniel Feitosa, Wesley K. G. Assunção Abstract Design decisions are at the core of software engineering and appear in Q&A forums, mailing lists, pull requests, issue trackers, and commit messages. Design discussions spanning a project’s history provide valuable information for informed decision-making, such as refactoring and software modernization. Machine learning techniques have been used to detect design decisions in natural language discussions; however, their effectiveness is limited by the scarcity of labeled data and the high cost of annotation. Prior work adopted cross-domain strategies with traditional classifiers, training on one domain and testing on another. Despite their success, transformer-based models, which often outperform traditional methods, remain largely unexplored in this setting. The goal of this work is to investigate the performance of transformer-based models (i.e., BERT, RoBERTa, XLNet, LaMini-Flan-T5-77M, and ChatGPT-4o-mini) for detecting design-related discussions. To this end, we conduct a conceptual replication of prior cross-domain studies while extending them with modern transformer architectures and addressing methodological issues in earlier work. The models were fine-tuned on Stack Overflow and evaluated on GitHub artifacts (i.e., pull requests, issues, and commits). BERT and RoBERTa show strong recall across domains, while XLNet achieves higher precision but lower recall. ChatGPT-4o-mini yields the highest recall and competitive overall performance, whereas LaMini-Flan-T5-77M provides a lightweight alternative with stronger precision but less balanced performance. We also evaluated similar-word injection for data augmentation, but unlike prior findings, it did not yield meaningful improvements. Overall, these results highlight both the opportunities and trade-offs of using modern language models for detecting design discussion. ∎ 1 Introduction The foundation of a software system is its architectural design and the decisions made to arrive at its current state ( 53 ; 11 ) . 1 1 1 In this paper, the term “design” refers explicitly to software structural design, and design decisions are architecture-related text, as described by 10 . Thus, during the software’s development life cycle, developers need to make many design decisions ( 2 ) . These decisions are usually related to functional requirements, non-functional requirements, or technological constraints, which, in turn, affect the architecture ( 3 ) . When made incorrectly or suboptimally, software design decisions can lead to software degradation and incur financial and maintenance costs, namely technical debt ( 27 ; 13 ; 47 ) , pushing software systems deep into the legacy system zone ( 6 ) . To maintain and evolve aging systems, ensuring their value over time, developers should be able to recover and understand the design decisions made during software development ( 22 ) . Modern software development platforms 2 2 2 https://github.com , https://gitlab.com , and https://www.atlassian.com/software/jira offer features such as pull requests and issues management, allowing developers to review code, raise issues with a given piece of code, and then have discussions to resolve them ( 60 ) . Additionally, software design decisions can also be discussed in mailing lists ( 52 ) and commit messages, where developers document their intentions behind code changes ( 51 ; 16 ) . Furthermore, developer forums such as Stack Overflow 3 3 3 https://StackOverflow.com are yet another platform where developers exchange information and discuss architecture decisions (among other topics) ( 14 ) . Pull requests, issues, emails, messages, Q&A forums, or any discussion medium, are valuable sources of information related to design discussion ( 39 ; 52 ; 10 ) . These pieces of information enable an understanding of software architectures and the history of their design evolution. Consequently, this information supports system maintenance and evolution (i.e., code refactoring, redesign, migration, or modernization) to meet new demands. However, exploiting multiple communication channels has been a challenge due to a lack of tools and processes for identifying such discussions ( 33 ; 35 ) . Because these discussions are seldom formally documented, studies have shown that the rationale of decisions related to a software’s architecture may vaporize with time ( 9 ; 25 ) . Also, unfortunately, current tools and practices poorly track which discussions are design decision-related. A few studies have proposed approaches based on machine learning (ML) for classifying textual discussions from different sources as design-related or non-design-related discussions ( 10 ; 52 ; 43 ; 33 ) . These studies trained ML classifiers and evaluated them on datasets from the same source. That is, both the training and testing datasets were derived solely from pull requests, issues, or commit messages. The problem with making these approaches applicable in practice is that, in addition to their sensitivity to the source dataset type, there is limited availability of labelled datasets, which further hinders the performance and generalizability of the tools. To address the problem mentioned above, 34 curated a dataset from Stack Overflow discussions and trained traditional classifiers on identifying similar design discussions in commit messages, pull requests, and issue discussions. This approach is referred to as cross-domain dataset classification , which involves training the model on a domain (in this case, a Stack Overflow discussion) and applying it to classify data from other domains (e.g., commits, pull requests, issues) ( 64 ) . The authors also employed similar-word injection , a data augmentation strategy that enhances classifier performance by reinforcing its capacity to generalize and transfer contextual information from the Stack Overflow dataset to commit messages, pull requests, and issue discussions. 34 highlighted challenges in the form of low ML model performance when trained on data from one domain (i.e., Stack Overflow) and tested on data from different domains, despite both containing design discussions in natural language text. To advance the state of the art in design discussion detection and pave the way for building better tools, this present work aims to evaluate transformer-based models for design discussion identification . Motivated by the strong performance of transformer-based language models (e.g., BERT ( 15 ) and ChatGPT ( 40 ) ) in understanding and classifying natural language, we addressed the limitations and extended the work by 34 by conducting a conceptual replication ( 12 ; 19 ) . Specifically, we explored whether five state-of-the-art transformer-based models, namely BERT ( 15 ) , RoBERTa ( 29 ) , and XLNet ( 61 ) (all discriminative models), as well as LaMini-Flan-T5-77M ( 24 ) and ChatGPT-4o-mini ( 40 ) (generative language models), can effectively generalize their ability to distinguish between design-related and non-design discussions in a cross-domain setting. To this end, we fine-tuned the models on Stack Overflow discussions and tested them on similar categories of text drawn from GitHub pull requests ( 52 ; 10 ) , code comments ( 13 ) , issues and commit messages ( 10 ) . Our key findings are: • Transformer-based models can accurately detect design-related discussions when training and testing data originate from the same domain. • A transformer-based model with a reduced dataset, namely Stack Overflow questions excluding answers and comments, can be sufficient for robust detection while reducing fine-tuning time and computational costs. • Despite outperforming traditional ML classifiers, transformers exhibit performance degradation, especially in balanced precision and recall, when applied across domains (e.g., Stack Overflow to GitHub). This highlights the need for careful use of domain adaptation techniques. • ChatGPT-4o-mini consistently