A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Code Smell Detection

Beiqi Zhang, Peng Liang, Xin Zhou, Xiyu Zhou, David Lo, Qiong Feng, Zengyang Li, Lin Li · 2024

Read A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Code Smell Detection by Beiqi Zhang, Peng Liang, Xin Zhou, Xiyu Zhou, David Lo, Qiong Feng, Zengyang Li, Lin Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13801.txt.

Introduction

Beiqi Zhang and co-authors test light fine tuning methods for finding code smells in a 2024 study. Code smells are signs of weak design that make software harder to change. Language models could flag them, but full fine tuning is costly and good labeled data are scarce.

The team first built a checked benchmark for four smell types, reviewed by hand in two stages. They then tested four light tuning methods on nine language models against rule based tools, deep models, and large models with prompts. Light tuning matched or beat full tuning while using far less memory. Tuned models beat all baselines, with the largest gains on some smell types.

Catalog record

CREATORBeiqi Zhang, Peng Liang, Xin Zhou, Xiyu Zhou, David Lo, Qiong Feng, Zengyang Li, Lin Li
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13801

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Code Smell Detection Beiqi Zhang, Peng Liang, Xin Zhou, Xiyu Zhou, David Lo, Qiong Feng, Zengyang Li, Lin Li Abstract . Code smells are indicators of suboptimal design that negatively impact software quality. However, automated code smell detection remains a persistent challenge: heuristics-based tools suffer from high sensitivity to threshold selection and inherent subjectivity, while Machine Learning (ML) and Deep Learning (DL) models yield unsatisfactory performance. Although Large Language Models (LLMs) offer a promising solution to these limitations, their adaptation is impeded by the prohibitive cost of full fine-tuning and the lack of “LM-ready” benchmarks, as existing code smell datasets largely rely on software metrics or contain noisy, unverified labels. To address these gaps and enable a rigorous assessment of efficient LLM adaptation for code smell detection, we present a comprehensive study comprising two synergistic contributions. First, we constructed a high-quality, source-code-centric benchmark covering four non-trivial code smells— Complex Conditional , Complex Method , Feature Envy , and Data Class —and ensured its reliability through a meticulous two-stage manual review. Second, leveraging this benchmark, we systematically evaluated the effectiveness of four Parameter-Efficient Fine-Tuning (PEFT) methods (i.e., prompt tuning, prefix tuning, LoRA, and ( I ​ A ) 3 (IA)^{3} ) across four Small LMs (SLMs) and five LLMs. We benchmarked these PEFT-tuned LMs against a broad suite of baselines, including traditional heuristics-based detectors ( DesigniteJava and PMD ), DL-based approaches ( DeepSmells and AE-Dense ), and cutting-edge general-purpose LLMs (GPT-4o-mini and DeepSeek-v3) with In-Context Learning. Our results demonstrate that PEFT methods match or exceed full fine-tuning performance while significantly reducing peak GPU memory usage, and outperform all baselines with MCC improvements ranging from 0.33% to 13.69%, particularly for some code smells. Furthermore, we provide actionable insights into the choice of PEFT method for code smell detection, which should be guided by the model used, data availability, and computational resources. Our replication package is available at ( Zhang et al., 2025 ) to support reproducibility and future extensions. DOI: XXXXXXX.XXXXXXX Journal: TOSEM Volume: 0 0 0 0 CCS: Software and its engineering Software maintenance tools 1. Introduction Code smells are indicators of underlying design or implementation issues that may degrade software quality ( Fowler, 1999 ) , particularly in terms of maintainability ( Palomba et al., 2018b ) , readability ( Abbes et al., 2011 ) , and testability ( Chatzigeorgiou and Manakos, 2010 ) . Detecting code smells helps to uncover software design flaws that lead to potential problems in future development and maintenance ( Fowler, 1999 ; Lanza and Marinescu, 2007 ; Moha et al., 2010 ) . However, manual identification of code smells is often time-consuming and cost-prohibitive ( Prokić et al., 2024 ) , which has prompted extensive research into automated detection techniques. Early work predominantly relied on heuristics-based detectors ( Marinescu, 2004 ; Morales et al., 2017 ; Moha et al., 2010 ) . These methods apply manually curated features and thresholds to discriminate between smelly and non-smelly instances ( Azeem et al., 2019 ) . While effective in some settings, such static tools exhibit well-documented shortcomings ( Fernandes et al., 2016 ; Zhang et al., 2011 ) : their performance is highly sensitive to threshold selection, they offer limited semantic understanding, and their results often diverge from human judgment due to the inherent subjectivity of code smells ( Azeem et al., 2019 ; Fontana et al., 2016a ; Mäntylä and Lassenius, 2006 ; Schumacher et al., 2010 ) . To overcome these limitations, researchers have increasingly incorporated Machine Learning (ML) and Deep Learning (DL) algorithms for code smell detection. These approaches aim to mitigate the performance instability inherent in the manual threshold-setting of heuristics-based detectors ( Fontana et al., 2013 ; Fontana et al., 2016b ; Sharma et al., 2021 ; Liu et al., 2021 ) . By reducing reliance on rigid thresholds, ML and DL models can partially learn semantic cues ( Nucci et al., 2018 ) . Nonetheless, recent studies indicate that these models still struggle to achieve considerable performance ( Pereira dos Reis et al., 2022 ; Alazba et al., 2023 ) , as they frequently fail to capture the deeper semantic and structural relationships within source code. Recently, Large Language Models (LLMs) have emerged as a transformative force in Software Engineering (SE), achieving state-of-the-art performance across a wide range of tasks ( Lu et al., 2023 ; Le et al., 2024 ) . Despite their impressive general capabilities, directly employing off-the-shelf LLMs for code smell detection without any task-specific fine-tuning yields suboptimal results ( Mesbah et al., 2025 ; Silva et al., 2024 ; Sadik and Govind, 2025 ) . While effective adaptation is essential, two primary challenges currently impede progress: Challenge 1: Lack of High-Quality, LM-Ready Datasets. Most existing code smell datasets comprise calculated software metrics rather than the raw source code required by LMs. Moreover, they typically focus on relatively simple smells that do not demand complex semantic reasoning. Furthermore, datasets that do include code often rely on noisy labels generated directly by heuristics-based detectors without human verification, leading to the “Oracle Problem”. This incompatibility renders existing benchmarks unsuitable for the effective training or evaluation of LMs in code smell detection. Consequently, without high-quality, source-code-centric benchmarks covering non-trivial code smells, a fair investigation into the effectiveness of LMs for code smell detection is infeasible. Challenge 2: Unclear Effectiveness of LM Adaptation Methods. Although Parameter-Efficient Fine-Tuning (PEFT) methods have shown promise across various SE tasks by enabling task-specific adaptation with only a small number of trainable parameters ( Wang et al., 2023 ; Liu et al., 2023b ; He et al., 2022 ; Liu et al., 2024b ; Weyssow et al., 2025 ; Li et al., 2024 ) , their effectiveness for code smell detection remains largely unexplored. It is still unclear whether PEFT approaches can match the performance of full fine-tuning or outperform existing baselines in code smell detection. Investigating PEFT methods in the context of code smell detection also fills a broader research gap. Prior studies on PEFT have primarily concentrated on method-level tasks ( Li et al., 2024 ; Liu et al., 2023b ; Wang et al., 2023 ; Liu et al., 2024b ; Weyssow et al., 2025 ) , leaving their effectiveness in class-level contexts largely unexplored. Our study addresses this gap by exploring the capabilities of PEFT-adapted LMs in capturing class-level semantics through the detection of code smells, such as Feature Envy and Data Class , which inherently necessitate class-level code understanding. Furthermore, utilizing PEFT techniques facilitates just-in-time code smell detection. Traditional heuristics-based detectors are often applied post-development ( Sharma, 2018 ; , 2024 ; Marinescu et al., 2005 ) , discovering issues only after software systems are fully implemented and thereby increasing refactoring overhead. In contrast, PEFT-tuned LMs enable real-time identification of code smells, allowing developers to receive immediate feedback upon completing a code block. This proactive approach supports high code quality throughout the development lifecycle, leading to more efficient software maintenance ( Ardimento et al., 2021a ; Ardimento et al., 2021b ) . To this end, we systematically evaluate the effectiveness of state-of-the-art PEFT methods on both Small LMs (SLMs, < 1B parameters) and Large LMs

Read the complete text