An Empirical Study of LLM-Based Code Clone Detection DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software maintenance tools

Wenqing Zhu, Norihiro Yoshida, Eunjong Choi, Yutaka Matsubara, Hiroaki Takada · 2025

Read An Empirical Study of LLM-Based Code Clone Detection DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software maintenance tools by Wenqing Zhu, Norihiro Yoshida, Eunjong Choi, Yutaka Matsubara, Hiroaki Takada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01176.txt.

Introduction

Wenqing Zhu and colleagues test how well large language models detect code clones, duplicated code fragments that maintenance tools need to find. Earlier studies reported strong results, but left open whether performance holds across data sets and whether the models answer the same question the same way twice.

They built seven clone data sets from the CodeNet and BigCloneBench collections, sampling code pairs by their text similarity, and evaluated five language models under four published prompts. Models did well on CodeNet derived sets, with one reaching a high F1 score, but performance fell greatly on BigCloneBench derived sets. Consistency was a bright spot: over ninety percent of judgments stayed the same across repeated submissions, and score fluctuations from inconsistency were tiny.

Catalog record

CREATORWenqing Zhu, Norihiro Yoshida, Eunjong Choi, Yutaka Matsubara, Hiroaki Takada
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01176

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Empirical Study of LLM-Based Code Clone Detection DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software maintenance tools Wenqing Zhu, Norihiro Yoshida, Eunjong Choi, Yutaka Matsubara, Hiroaki Takada Abstract . Large language models (LLMs) have demonstrated remarkable capabilities in various software engineering tasks, such as code generation and debugging, because of their ability to translate between programming languages and natural languages. Existing studies have demonstrated the effectiveness of LLMs in code clone detection. However, two crucial issues remain unaddressed: the ability of LLMs to achieve comparable performance across different datasets and the consistency of LLMs’ responses in code clone detection. To address these issues, we constructed seven code clone datasets and then evaluated five LLMs in four existing prompts with these datasets. The datasets were created by sampling code pairs using their Levenshtein ratio from two different code collections, CodeNet and BigCloneBench. Our evaluation revealed that although LLMs perform well in CodeNet-related datasets, with o3-mini achieving a 0.943 F1 score, their performance significantly decreased in BigCloneBench-related datasets. Most models achieved a high response consistency, with over 90% of judgments remaining consistent across all five submissions. The fluctuations of the F1 score affected by inconsistency are also tiny; their variations are less than 0.03. 1. Introduction Code clones are pairs of identical or similar code fragments. These clones are generally created when developers reuse functionality by copying and pasting existing source code ( Roy and Cordy 2007 ; Roy et al. 2009 ; Al-Ekram et al. 2005 ; Baker 1993 ) . They can be classified into syntactic and semantic clones ( Roy and Cordy 2007 ) . Syntactic clones have high syntactic similarity, while semantic clones may have low syntactic similarity but identical functionality. Detecting code clones has traditionally been treated as a classification problem ( Roy and Cordy 2008 ; Sajnani et al. 2016 ; Kamiya et al. 2002 ) . Traditional code clone detectors generally employ threshold-based similarity measurements to identify code fragments as code clones. While these detectors effectively identify syntactic clones, they struggle to detect semantic clones because the same functionality can be implemented using different syntax. In recent years, the Large Language Model (LLM) has demonstrated notable performance across various software engineering tasks, including code generation, debugging, and comprehension ( Xia and Zhang 2023 ; Ouyang et al. 2023 ; Tian et al. 2024 ; Nam et al. 2024 ; Sriwilailak et al. [n. d.] ; Rasheed et al. 2024 ; Wang et al. 2024 ) . LLM has the potential for accurate code clone detection as it can capture deeper relationships between code clones, even when their syntax differs. LLM-based code clone detectors allow developers to identify code clones using zero-shot or few-shot prompting. In this process, developers provide a prompt that includes a pair of code fragments and ask the model to evaluate their similarity. The model then generates an output, which is interpreted as a Boolean value either directly or with the help of a discriminator to determine whether the code fragments are clones. Previous studies have demonstrated the effectiveness of LLMs in code clone detection ( Dou et al. 2023 ; Khajezade et al. 2024 ) . Khajezade et al. demonstrated the effectiveness of GPT-3.5-turbo with their proposed prompts in detecting code clones, particularly for Java - Java and Java - Ruby clone pairs ( Khajezade et al. 2024 ) . Dou et al. evaluated the performance of LLMs with prompt engineering techniques and found that GPT-3.5-turbo and GPT-4 outperform traditional code clone detectors in identifying semantic code clones ( Dou et al. 2023 ) . While these studies highlight the potential of LLMs for code clone detection, a deeper understanding of their capabilities in detecting code clones is necessary. In particular, evaluating their generalization ability and response consistency is crucial to ensure the practical application. Figure 1 . Experiment Process Regarding generalization, while Deep Learning (DL)-based clone detectors perform well in detecting semantic clones ( Li et al. 2017 ; Zhang et al. 2019 ; Feng et al. 2020 ) , they struggle to generalize across diverse datasets ( Choi et al. 2023a ) . Since LLMs are trained on significantly larger datasets than DL models, they may offer better generalization. However, this potential still requires thorough investigation. Similarly, response consistency is a critical factor. LLMs are designed for contextual generation, generating varied outputs rather than identical responses to the same input. However, for code clone detection, judgments for each candidate must remain stable and deterministic. Therefore, investigating the response consistency of LLM in detecting code clones is necessary for reliable detection. Despite their importance, these issues have not been thoroughly investigated. To address these issues, this study investigates LLM-based code clone detectors to answer the following two Research Questions (RQs): RQ1:: How accurately does each LLM detect code clones across different datasets? RQ2:: How consistent are the responses from each LLM when presented with identical input? To answer these RQs, we constructed seven datasets in various programming languages (four in Java and one each in Python , C , and C++ ) by selecting clone and non-clone pairs from two commonly used code clone evaluation datasets CodeNet ( CN ) ( Puri et al. 2021 ) and BigCloneBench ( BCB ) ( Svajlenko and Roy 2015 ) . In these datasets datasets, all code pairs were labeled according to their syntactic similarity. For Java , the datasets were divided into two types based on similarity distribution to ensure a fair comparison of performance across datasets. Furthermore, we selected five LLMs (i.e., o3-mini , GPT-4o , GPT-4o-mini , Llama 3.1 , and Mistral ) and evaluated their performance in code clone detection. The prompts were chosen based on existing studies ( Dou et al. 2023 ; Khajezade et al. 2024 ) . The evaluation metrics included recall, precision, F1 score, and response consistency rate. Each model-prompt combination was evaluated in different temperature settings (a parameter to control the randomness of LLM’s output) and attempt times. The evaluation results show that LLMs demonstrated superior performance on CN -derived datasets than on BCB , and most models exhibited high response consistency with minimal performance variations. The contributions of this study are listed below: • All models demonstrated significantly higher performance on datasets derived from CN , with certain model-prompt combinations achieving F1 scores above 0.9. However, on datasets derived from BCB , most model–prompt pairs could not accurately detect clones effectively in low-similarity ranges • All LLMs, except Llama 3.1 , exhibited high response consistency, with response inconsistencies having minimal impact on F1 scores across models. Additionally, prompt selection typically affected response consistency more than the temperature settings of the LLMs. • The temperature value of LLM has a small influence on code clone detection on both response consistency and F1 score. • For the replicability of this study, we have published seven datasets covering four languages 1 1 1 All our datasets, programs, and results can be accessed by: https://zenodo.org/records/15019503 . Besides, our scripts for accessing LLMs for clone detection and analyzing the results are also included. 2. Related Work Traditionally, code clone detection has been treated as a classification problem. Traditional code clone detectors typically transform the input

Read the complete text