Evaluating Software Plagiarism Detection in the Age of AI
Timur Sağlam, Larissa Schmid · 2025
Read Evaluating Software Plagiarism Detection in the Age of AI by Timur Sağlam, Larissa Schmid (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20158.txt.
Introduction
Timur Sağlam and Larissa Schmid test how well code plagiarism detectors stand up when copied code is scrambled by machine. Their 2025 paper deals with programming homework, where copying is a steady problem. Detectors shrug off simple scrambles, but deep scrambles that reshape code while it keeps its behavior can slip past. Past checks of added defenses used a narrow set of attacks and skipped AI-made scrambles.
The team pits two defenses against a wide range of attacks, both rule-based and AI-made, across many real data sets. Millions of code pairs were compared. Both defenses caught far more scrambled copies, and they even spotted AI-written code better, though not built for that task.
Catalog record
| CREATOR | Timur Sağlam, Larissa Schmid |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Software Plagiarism Detection in the Age of AI Timur Sağlam, Larissa Schmid Abstract Plagiarism in programming assignments is a persistent issue in computer science education, increasingly complicated by the emergence of automated obfuscation attacks. While software plagiarism detectors are widely used to identify suspicious similarities at scale and are resilient to simple obfuscation techniques, they are vulnerable to advanced obfuscation based on structural modification of program code that preserves the original program behavior. While different defense mechanisms have been proposed to increase resilience against these attacks, their current evaluation is limited to the scope of attacks used and lacks a comprehensive investigation regarding AI-based obfuscation. In this paper, we investigate the resilience of these defense mechanisms against a broad range of automated obfuscation attacks, including both algorithmic and AI-generated methods, and for a wide variety of real-world datasets. We evaluate the improvements of two defense mechanisms over the plagiarism detector JPlag across over four million pairwise program comparisons. Our results show significant improvements in detecting obfuscated plagiarism instances, and we observe an improved detection of AI-generated programs, even though the defense mechanisms are not designed for this use case. Based on our findings, we provide an in-depth discussion of their broader implications for academic integrity and the role of AI in education. ∎ DSL Domain-Specific Language PCM Palladio Component Model XML Extensible Markup Language EMF Eclipse Modeling Framework EMF_Book UML Unified Modeling Language GDPR General Data Protection Regulation of the European Union DAG Directed Acyclic Graph UML Unified Modeling Language UML MDE Model-Driven Engineering Kent2002 LSH Locality-Sensitive Hashing Pauleve2010 LLM Large Language Model GPT OpenAI’s Generative Pre-trained Transformer SysML Systems Modeling Language AST Abstract Syntax Tree MOF MOF2016a Meta Object Facility OCL Object Constraint Language XMI XML Metadata Interchange URI Uniform Resource Identifier Automated Obfuscation and Lessons for Academic Integrity 1 Introduction Plagiarism is a prevalent challenge in computer science education, facilitated by the ease of duplicating and modifying digital assignments Cosma and Joy (2008) ; Murray (2010) ; Le et al. (2013) . Although students generally acknowledge plagiarism as academic misconduct, some will engage in it despite the threat of consequences Sutton et al. (2014) . Therefore, students are creative in obfuscating their plagiarism to conceal the relation to its source Pawelczak (2018) . In the case of programming assignments, students commonly utilize techniques such as renaming, reordering, or restructuring Novak et al. (2019) ; Karnalim (2016) . Plagiarism in programming assignments is particularly pronounced in beginner-level and mandatory courses, such as introductory programming courses Park (2003) . While checking submissions for plagiarism manually is feasible for small course sizes, this quickly becomes infeasible for larger course sizes Camp et al. (2017) ; Kustanto and Liem (2009) as the number of required pairwise comparisons grows quadratically – reaching 1,225 comparisons for just 50 submissions. This results in the individual risk of detection decreasing with rising course sizes Yan et al. (2018) . In light of these issues, it is common for educators to use software plagiarism detection systems to uphold academic integrity for programming assignments Devore-McDonald and Berger (2020) . These systems automate parts of the detection process and thus allow tackling the problem of plagiarism detection at scale. Thus, educators strongly rely on software plagiarism detectors to guide them in inspecting suspicious candidates. Plagiarism detectors analyze sets of programs to detect pairs with a suspiciously high degree of similarity Prechelt et al. (2000) . However, assessing which suspicious candidates qualify as plagiarism is ultimately a human decision, given the underlying ethical considerations Culwin and Lancaster (2001) ; Weber-Wulff (2019) . Overall, plagiarism detection systems help identify plagiarism instances and, when using such systems is communicated Karnalim et al. (2022) , deter students from plagiarizing firsthand Braumoeller and Gaines (2001) . Crucially, plagiarism detectors are only effective when defeating them takes more effort than completing the actual assignment Devore-McDonald and Berger (2020) . Yet, manually obfuscating a program successfully is tedious and requires understanding the underlying program, therefore requiring time and programming proficiency. Thus, a widespread assumption was that evading detection is not feasible for novice programmers as obfuscating the program requires more time than it takes to complete the actual assignments and requires a profound understanding of programming languages Joy and Luck (1999) . However, this assumption has been broken with the recent rise of automated obfuscation attacks Devore-McDonald and Berger (2020) ; Foltýnek et al. (2020) ; Biderman and Raff (2022) ; Pawelczak (2018) which require neither time nor programming proficiency to employ successfully. These obfuscation attacks aim to avoid detection by strategically altering a plagiarized program, thus obscuring the relation to its original Sağlam et al. (2024a) : State-of-the-art detection approaches compare the structure of programs by identifying similarities between code fragments Nichols et al. (2019) . Thus, most obfuscation attacks alter the structural properties of the program, ideally without affecting its behavior. Early automated attacks relied purely on algorithmic approaches, for example, via repeated statement insertion Devore-McDonald and Berger (2020) . However, the challenge intensifies with the rise of generative artificial intelligence, especially LLM Daun and Brings (2023) , making the obfuscation of plagiarism even more accessible with less effort than ever before Khalil and Er (2023) ; Sağlam et al. (2024b) . While state-of-the-art detectors exhibit some obfuscation resilience to changes like retyping and lexical changes, this does not apply to all types of obfuscation attacks Devore-McDonald and Berger (2020) ; Luo et al. (2017) . Thus, automated obfuscation attacks present a significant challenge for today’s plagiarism detection systems, as they must now contend with increasingly sophisticated obfuscation techniques that can evade detection while maintaining the original program’s functionality. In recent work Sağlam et al. (2024a) ; Sağlam et al. (2025) , we proposed defense mechanisms tailored towards different obfuscation attacks. However, it is unclear how well they work against a broader range of attacks. Token Sequence Normalization Sağlam et al. (2024a) explicitly targets dead code insertion and statement reordering. Subsequence Match Merging Sağlam et al. (2025) employs a heuristic to counteract any obfuscation attack that aims at interrupting the match found between two program codes. While we show that these approaches are effective against the obfuscation attacks they are targeting individually, educators typically do not know which obfuscation attacks students employed, thus making it hard to select a specific appropriate defense mechanism. Ideally, a combination of these defense mechanisms can be used to provide greater resilience. However, it is currently unclear whether the different defense mechanisms can be combined to achieve this while not producing false-positive results due to the overapproximation of similarities. Moreover, with the steady improvements of LLM , AI-based obfuscation attacks become more and more feasible. However, the defense mechanisms have not been comprehensively tested against a broad spectrum of automated attacks, including both algorithmic and AI-based obfuscation techniques. Finally, their applicability to detecting AI-generated programs is yet to be assessed. 1.1 Research Contributions In this paper, we investigate the resilience of software plagiarism detectors to different automated obfuscation attacks. As a first contribution (C1), we present a comprehensive evaluation of various automated obfuscation attacks, including both algorithmic and AI-based methods, also exploring the feasibility of using AI to generate programs. In addition to examining defense mechanisms on their