Can Code Evaluation Metrics Detect Code Plagiarism? DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software verification and validation CCS: Computing methodologies Artificial intelligence CCS: Applied computing Education

Fahad Ebrahim, Mike Joy · 2026

Read Can Code Evaluation Metrics Detect Code Plagiarism? DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software verification and validation CCS: Computing methodologies Artificial intelligence CCS: Applied computing Education by Fahad Ebrahim, Mike Joy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25778.txt.

Introduction

Ebrahim and Joy ask whether code evaluation metrics, built to assess generated code, can also detect source code plagiarism. They compare five metrics, including CodeBLEU, CrystalBLEU, RUBY, and CodeBERTScore, against dedicated tools JPlag and Dolos, on two labeled datasets across six modification levels. Without preprocessing, Dolos leads overall, while CrystalBLEU, CodeBLEU, and RUBY beat JPlag. After clean-up, CrystalBLEU surpasses Dolos overall. Performance is strongest at the simplest level and drops from level four onward, though CrystalBLEU stays competitive at level six.

Catalog record

CREATORFahad Ebrahim, Mike Joy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47576
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25778

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Code Evaluation Metrics Detect Code Plagiarism? DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software verification and validation CCS: Computing methodologies Artificial intelligence CCS: Applied computing Education Fahad Ebrahim, Mike Joy Abstract . Source Code Plagiarism Detection (SCPD) plays an important role in maintaining fairness and academic integrity in software engineering education. Code Evaluation Metrics (CEMs) are developed for assessing code generation tasks. However, it remains unclear whether such metrics can reliably detect plagiarism across different levels of modification (L1-L6), increasing in complexity. In this paper, we perform a comparative empirical study using two open-source labelled datasets, ConPlag (raw and template-free versions) and IRPlag. We evaluate five CEMs , namely CodeBLEU, CrystalBLEU, RUBY, Tree Structured Edit Distance (TSED) , and CodeBERTScore. The performance is evaluated using threshold-free ranking-based measures to assess overall, per dataset, and per-level plagiarism performance. The results are compared against state-of-the-art (SOTA) Source Code Plagiarism Detection Tools (SCPDTs) , JPlag and Dolos. Our findings show that without preprocessing, Dolos achieves the highest overall ranking performance, while among the individual metrics, CrystalBLEU, CodeBLEU, and RUBY outperform JPlag. Performance is strongest at L1 and drops from L4 onward, while CrystalBLEU remains competitive on L6. With preprocessing, CrystalBLEU surpasses Dolos overall. Per dataset, Dolos achieved the best ranking on the ConPlag raw dataset, while CrystalBLEU was the best-performing metric on the remaining datasets. At the plagiarism levels, Dolos remains strongest on L4, while CrystalBLEU leads most of the remaining difficult levels. These results indicate that CEMs are comparable to dedicated tools in terms of ranking metrics. 1. Introduction Source code similarity refers to the degree to which two programs are alike ( 29 ) . This similarity can be measured in various ways, ranging from simple text matching to more comprehensive structural and semantic comparisons. These different forms of similarity are used in software engineering applications such as malware detection and vulnerability analysis. Another important application is SCPD , which aims to determine whether a program has been copied or reused without proper acknowledgement. In software engineering education, plagiarism can affect fairness and academic integrity ( 2 ) . Therefore, reliable SCPD methods are essential to ensure fair assessment. Plagiarism in source code can occur at different levels of modification. Faidhi and Robinson proposed a six-level taxonomy (L1-L6) that can be grouped into three categories ( 9 ) . The lower levels (L1-L3) involve syntactic changes such as formatting, identifier renaming, and changing the order of statements. The intermediate levels L4 and L5 include structural modifications related to methods, loops, and conditions. The higher level, L6, involves larger changes in semantics, structure, and decision logic. Different plagiarism levels require similarity measures that can capture lexical, structural, and semantic aspects of code. CEMs were primarily developed for code generation tasks, in which a generated program is compared against a reference implementation for tasks such as code generation, translation, or summarisation. Although they were not developed for plagiarism detection, these metrics can produce numerical similarity scores that can capture different characteristics of source code, such as syntax, structure and semantics. This raises a key question: can CEMs detect source code plagiarism across different levels of modifications? To answer this question, we perform a comparative empirical study using two open-source plagiarism datasets, ConPlag ( 25 ) , and IRPlag ( 12 ) , both labelled with plagiarism levels. We evaluate five CEMs : CodeBLEU ( 22 ) , CrystalBLEU ( 8 ) , RUBY ( 27 ) , TSED ( 26 ) , and CodeBERTScore ( 31 ) . These metrics measure similarity based on token overlap, syntactic structure, tree distance, and embedding representations. We assess their performance using threshold-free ranking measures, which evaluate how well plagiarised pairs are ranked above non-plagiarised pairs without fixing a decision threshold. We use threshold-free evaluation, as plagiarism detection is highly sensitive to the choice of threshold. Small changes in the threshold can significantly affect the evaluation metrics. The analysis is conducted overall, per dataset, and by plagiarism level to examine how performance changes as modifications become more complex. Finally, we compare the results with SOTA SCPDTs , JPlag ( 21 ) and Dolos ( 16 ) , to determine how well CEMs compare against established plagiarism detection tools. Both JPlag and Dolos preprocess the code by default as part of their algorithms, therefore, we also compare the impact of preprocessing on the performance of CEMs . This paper makes three main contributions: • We present a systematic evaluation of CEMs for source code plagiarism detection. We study multiple datasets and all plagiarism levels (L1-L6). • We show that CEMs achieve performance comparable to SCPDTs under ranking-based evaluation. We also show that preprocessing is critical. It consistently improves performance and can change the ranking of methods. • We show that performance drops from L4 onwards for all methods. This highlights the difficulty of detecting structural and semantic plagiarism. We also show that combining metrics improves ranking accuracy and is a promising direction. The paper is organised as follows. Section 2 presents the background and related work. Section 3 describes the methodology, including the datasets, the selected CEMs , and the evaluation measures. Section 4 presents the results. Section 5 discusses the findings, limitations, threats to validity, and implications. The paper concludes with a summary and an outline of future work. 2. Background and Related Work In this section, we discuss plagiarism levels, SCPDTs , and empirical studies involving CEMs applied to code similarity tasks. 2.1. Plagiarism Levels As described by Faidhi and Robinson ( 9 ) , plagiarism in programming assignments can take place at different levels, ranging from surface edits to heavy logic changes. L1) Changing only the appearance of the code, such as editing comments, indentation, or spacing. L2) Renaming variables, functions, or classes without changing how the program works. L3) Adding, removing, or reordering constants, variables, or functions in the declarations. L4) Reorganising functions, for example, by changing their parameters, combining several functions into one, or splitting one function into smaller ones. L5) Replacing control structures with equivalent ones, such as using a while loop instead of a for loop, or rewriting conditions differently. L6) Changing the internal decision logic by rewriting expressions or conditional statements while keeping the overall behaviour the same. As the plagiarism level increases, the changes become more complex and more difficult to detect ( 1 ) . 2.2. Source Code Plagiarism Detection Tools In general, SCPD methods use different strategies to measure similarity between programs. These approaches range from lexical techniques based on token sequence comparison to structural and semantic methods that analyse representations such as Abstract Syntax Trees (ASTs) ( 1 ; 29 ) . More recent work often combines multiple signals to capture both surface-level similarity and deeper semantic similarity. Building on these methodological approaches, several systems and tools have been developed to support practical plagiarism detection. Several SCPDTs have been reviewed and compared in the literature ( 1 ; 29 ; 19 ) . In this study, we focus on two widely used open-source and actively maintained tools. JPlag ( 21 ) is a tool that supports languages such as Java, C++, C, and Python. It is based on the Greedy String Tiling (GST)

Read the complete text