CoDet-M4: Det ecting M achine-Generated Co de in M ulti-Lingual, M ulti-Generator and M ulti-Domain Settings
Daniil Orel, Dilshod Azizov & Preslav Nakov · 2025
Read CoDet-M4: Det ecting M achine-Generated Co de in M ulti-Lingual, M ulti-Generator and M ulti-Domain Settings by Daniil Orel, Dilshod Azizov & Preslav Nakov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13733.txt.
Introduction
Daniil Orel, Dilshod Azizov, and Preslav Nakov build a framework, in a 2025 preprint, for telling whether code was written by a person or a language model. AI code generators are fast and fluent, which strains teaching, ethics, and grading. Earlier detectors covered few languages and broke on new domains. This framework spans many programming languages, many generator models, and many fields. It trains on a large dataset drawn from well-known platforms and generators. Careful quality checks and feature design come first.
The team compares classic machine learning, pre-trained code models, and the generators themselves as judges. Tests push beyond the training mix: unseen models, domains, and languages. Hybrid code, part human and part machine, is judged too. The framework separates human from machine code reliably across these shifts. The authors present it as a new benchmark for the task, useful wherever authorship of code must be trusted.
Catalog record
| CREATOR | Daniil Orel, Dilshod Azizov & Preslav Nakov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoDet-M4: Det ecting M achine-Generated Co de in M ulti-Lingual, M ulti-Generator and M ulti-Domain Settings Daniil Orel, Dilshod Azizov & Preslav Nakov Abstract Large Language Models (LLMs) have revolutionized code generation, automating programming with remarkable efficiency. However, this has had important consequences for programming skills, ethics, and assessment integrity, thus making the detection of LLM-generated code essential for maintaining accountability and standards. While, there has been some previous research on this problem, it generally lacks domain coverage and robustness, and only covers a small number of programming languages. Here, we aim to bridge this gap. In particular, we propose a framework capable of distinguishing between human-written and LLM-generated program code across multiple programming languages, code generators, and domains. We use a large-scale dataset from renowned platforms and LLM-based code generators, alongside applying rigorous data quality checks, feature engineering, and comparative analysis of traditional machine learning models, pre-trained language models (PLMs), and LLMs for code detection. We perform an evaluation on out-of-domain scenarios, such as detecting authorship and hybrid authorship of generated code and generalizing to unseen models, domains, and programming languages. Our extensive experiments show that our framework effectively distinguishes human-written from LLM-generated program code, setting a new benchmark for the task. 1 Introduction Recent advancements in Large Language Models (LLMs) have demonstrated their remarkable ability to generate outputs that closely emulate human-written content Jiang et al. (2024) . This has spurred exponential growth in research, with publications on LLM-based code generation in leading venues rising from a single article in 2018 to 140 in 2024 Chang et al. (2024) . Moreover, these developments promise to accelerate software development, automate routine tasks, and boost productivity. At the same time, the rapid progress in artificial intelligence (AI) generative systems has posed major concerns, particularly related to accountability and ethical use of this technology Al-kfairy et al. (2024) . Machine-generated code can be exploited to create obfuscated scripts, introduce vulnerabilities, and produce deceptive artifacts that are difficult to trace Bukhari (2024) . Thus, it is important to develop tools that can detect machine-generated content, e.g., tracing AI-assisted commits could empower code reviewers to proactively mitigate risks. In academia, the use of LLMs for completing written assignments undermines educational integrity, with professors unknowingly grading machine-generated submissions Koike et al. (2024) ; Ma et al. (2023) . Alarmingly, more than 60,000 scientific articles in the past year alone have shown evidence of machine-generated content Gray (2024) . Accountability is equally critical for talent hiring and candidate evaluation. Employers must verify that the code submitted by a candidate truly reflects their actual abilities. Without robust detection mechanisms, generative AI could lead to misleading assessments. Moreover, detecting the use of LLM-generated code is essential for developing more effective code-based LLMs; since these systems depend on human-written samples for training, accurately detecting AI-generated code helps curate higher-quality training datasets. Previous work has proposed frameworks for detecting machine-generated code, including contrastive learning with a UniXcoder-based semantic encoder Xu et al. (2025) and machine learning (ML) models that analyzed Claude 3-generated code in the CodeSearchNet dataset Rahman et al. (2024) . That work has focused on a single API-based code copilot, but today we face a growing prevalence of a variety of open-weights LLMs as well as locally deployable LLM-based code assistants. Despite previous efforts, this evolving state of LLMs de facto highlights the urgent need for reliable curated large-scale high-quality code data with various programming languages and methods to distinguish between human-written and LLM-generated code to mitigate risks and maintain accountability in software development and academic integrity Chang et al. (2024) . With this in mind, addressing the aforementioned limitations is crucial, as LLMs are designed to generate code across various languages and domains 1 1 1 Domain in our study corresponds to a combination of data source (GitHub, LeetCode, Codeforces) and its form: functions, classes, and arbitrary code snippets. Here, we aim to bridge these gaps by constructing a first-of-its-kind large-scale, multi-lingual, multi-domain, and multi-generator dataset comprising \approx 500K samples of human- and LLM-written code. The dataset spans class-level and function-level code, as well as competitive and industrial programming contexts. We further propose models for detecting LLM-generated code, and we evaluate their performance in extreme out-of-domain (OOD) settings: detecting unseen models, unseen domains, and unseen programming languages. We aim to answer the following research questions (RQs): • (RQ1) How do traditional detection methods compare vs. advanced deep neural network (DNN)-based models to effectively identify machine-generated code? • (RQ2) Are detection models capable of accurately attributing machine-generated code to the specific language model that produced it? • (RQ3) Can detection models generalize robustly across different code-generating models, domains, programming languages, and hybrid-authorship scenarios? We make the following contributions: • We introduce a novel corpus and benchmark designed for studying machine-generated and human-written code. Spanning a wide array of models, domains, and programming languages, providing a diverse and large-scale foundational resource for further research. • We repurpose existing code-related models, fine-tuning them to identify machine-, human- and hybrid-written code. • We analyze the performance of these models from multiple perspectives: ( i ) authorship and hybrid authorship identification, ( ii ) unseen code generators, ( iii ) unseen domains, ( iv ) unseen programming languages, and ( iv ) mixed authorship scenario: when LLMs not just generate the code, but rather complement or rewrite human-written code. • We release our data and code 2 2 2 https://huggingface.co/datasets/DaniilOr/oDET-M4 , and we are committed to continuously updating our repository with additional generators, domains, and languages in the future. 2 Related Work In this section, we discuss related work, focusing on resources and models for detecting machine-generated code. Resources: Major advancements have been made in developing benchmarks to evaluate LLMs for code generation, covering various domains. Chen et al. (2021) introduced 164 Python problems with function signatures and unit tests, extended by Liu et al. (2023) with 80x more test cases, while Muennighoff et al. (2024) added tasks such as code synthesis in six languages. Austin et al. (2021) provided 974 entry-level Python problems and Yin et al. (2018) curated 597K Python code generation samples. Yu et al. (2018) evaluated text-to-SQL queries across 138 domains, Iyer et al. (2018) tested near zero-shot Java class generation, and Wang et al. (2023c) focused on execution-based Python code generation. The generation of pragmatic codes was assessed by Yu et al. (2024) , robustness was evaluated by Wang et al. (2023a) , while Babe et al. (2024) examined student-authored prompts. Finally, Zhuo et al. (2025) , Du et al. (2024) , and Zhang et al. (2024) targeted cross-domain, class-level, and multi-lingual tasks, respectively. Athiwaratkun et al. (2023) adapted Austin et al. (2021) for multiple languages and Zheng et al. (2023) extended for multilingual tasks, evaluating code in C++, Java, JavaScript and Go. Cassano et al. (2023) benchmarked code generation in 18 languages and Khan et al. (2023) provided 25M multilingual examples for multitask evaluations. Austin et al. (2021) synthesized code from complex descriptions, while Gu et al. (2024) evaluated reasoning and execution capabilities using 800 Python functions. Hendrycks et al. (2021) included 10K Python problems at varying difficulty levels, Li et al. (2022) offered competitive problems with test cases from platforms such as CodeForces, and