UCSC-NLP at SemEval-2026 Task 13: Multi-View Generalization and Diagnostic Analysis of Machine-Generated Code Detection
Kargi Chauhan, Sadiba Nusrat Nur · 2026
Read UCSC-NLP at SemEval-2026 Task 13: Multi-View Generalization and Diagnostic Analysis of Machine-Generated Code Detection by Kargi Chauhan, Sadiba Nusrat Nur (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26990.txt.
Introduction
Chauhan and Nur enter SemEval-2026 Task 13 on telling human-written code from machine-generated code. The task has two subtasks: binary detection and attribution across 10 model families. For detection they fine-tune UniXcoder-base with a method that promotes generator-invariant codes, mixing structure prefixes and token dropout. The system scores 0.993 macro F1 on validation and lower on test data spanning languages and domains. For the second task, severe class imbalance wrecks fine-tuning, but a class-weighted extension lifts macro F1 about fourfold, confirming it needs imbalance-aware training.
Catalog record
| CREATOR | Kargi Chauhan, Sadiba Nusrat Nur |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22401 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26990
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UCSC-NLP at SemEval-2026 Task 13: Multi-View Generalization and Diagnostic Analysis of Machine-Generated Code Detection Kargi Chauhan, Sadiba Nusrat Nur Abstract With the rapid growth of large language models for code generation, distinguishing between human-written and AI-generated code has become increasingly critical for academic integrity, hiring evaluations, and software security. We present our system for SemEval-2026 Task 13: Multilingual Machine-Generated Code Detection, participating in Subtask A (binary detection) and Subtask B (multi-class attribution across 10 LLM families). For Subtask A, we fine-tune UniXcoder-base with a multi-view training framework that promotes generator-invariant representations. The framework combines domain-specific structural prefixes, delexicalization with symmetric KL consistency loss, token dropout, and mixed-content augmentation. Our system achieves 0.993 macro F1 on validation and 0.845 macro F1 on the test set, which spans unseen languages and domains. For Subtask B, we show that severe class imbalance (88.4% human code, 221:1 majority-to-minority ratio) causes catastrophic minority-class failure under standard fine-tuning, with macro F1 collapsing to 0.086 despite 88.4% accuracy. A class-weighted extension trained for 3 epochs recovers macro F1 to 0.345 (+301% relative), confirming that multi-class attribution requires imbalance-aware training strategies. 1 1 1 Code: https://github.com/Kargichauhan/Detecting-Machine-Generated-Code 1 Introduction Large language models (LLMs) have reshaped software development. AI-assisted tools such as GitHub Copilot, used by over 1.8 million developers ( 4 ) , are now embedded in educational and professional workflows. While improving productivity, they raise concerns about authorship, academic integrity, intellectual property, and security auditing ( 21 ) . Consequently, distinguishing human-written from machine-generated code has become both critical and technically challenging, as detectors often overfit to generator-specific artifacts and degrade under prompt variation or minor edits ( 20 ; 13 ) . SemEval-2026 Task 13 ( 18 ) builds upon the DROID resource suite ( 19 ) , a multilingual, multi-generator benchmark designed to evaluate robustness under distribution shift. Prior work shows that detectors trained on limited languages or domains generalize poorly to unseen settings ( 17 ; 20 ) . These issues are exacerbated in fine-grained attribution, where severe class imbalance leads to systematic neglect of minority generator classes. We frame this task as a domain generalization problem under lexical, structural, and stylistic shifts. To promote generator-invariant representations, we adopt: (i) domain-specific structural prefixes for content-type alignment ( 6 ) , (ii) delexicalization with symmetric KL consistency regularization ( 15 ) , (iii) token dropout to reduce reliance on spurious cues ( 24 ) , and (iv) mixed-content augmentation for distributional robustness ( 3 ) . We further show that multi-class attribution collapses under severe imbalance, where high accuracy masks near-complete minority-class failure. Our work makes two distinct contributions: (1) a multi-view generalization framework for robust binary detection under distribution shift (Subtask A), and (2) a diagnostic analysis demonstrating that standard fine-tuning catastrophically fails under extreme class imbalance in multi-class attribution (Subtask B). Code Snippet 3-Domain Prefix First+Last Truncation BPE Tokenizer UniXcoder Encoder [CLS] Pool Linear Head 3-way Ensemble Output 2-class Code Snippet Delex ID/STR/NUM BPE Tokenizer Code Snippet Mixed Augment BPE Tokenizer UniXcoder Encoder (shared) Sym KL Loss train only clean/mixed/fragment 512 tokens 125M params 768-d 15% dropout Original view Delex view Mixed view Figure 1: Subtask A architecture. Three views of each input are processed through a shared UniXcoder encoder; symmetric KL loss enforces prediction consistency across views during training, while inference averages logits as a zero-cost ensemble. Code Snippet BPE Tokenizer CodeBERT Encoder [CLS] Pooling Linear Classifier 11-Class Output L=512 CodeBERT-base (125M) \mathbf{h}_{\texttt{[CLS]}}\in\mathbb{R}^{768} Human + 10 LLM Families Figure 2: Subtask B architecture. A standard CodeBERT encoder with softmax classification head serves as a diagnostic baseline to isolate the effect of class imbalance on 11-way generator attribution. 2 Background and Related Work 2.1 Machine-Generated Code Detection Early work on machine-generated code detection adapts text-based AI detection methods to programming languages. GPTSniffer ( 16 ) introduced CodeBERT-based classification and achieved strong performance on ChatGPT-generated code, but showed limited generalization across generators. CodeGPTSensor ( 25 ) mitigated this using contrastive learning to better separate human-written and machine-generated representations. Prior studies highlight the difficulty of the task. General-purpose AI detectors perform near chance level on code ( 20 ) , and learned stylistic cues are often generator-specific and brittle to prompt variation or post-editing ( 13 ) . Alternative signals, such as entropy-based detection ( 14 ) , repetition statistics ( 7 ) , and likelihood-based methods ( 11 ) , are largely designed for natural language and degrade on source code due to its syntactic regularity. Recent benchmarks ( 22 ) therefore emphasize robustness across programming languages and generators as a key challenge. 2.2 Pretrained Code Models CodeBERT ( 5 ) introduced bimodal pretraining on natural language and code, enabling strong downstream classification. Subsequent models incorporate richer structural signals: GraphCodeBERT ( 9 ) integrates data-flow graphs, while UniXcoder ( 8 ) unifies encoder-only, decoder-only, and encoder–decoder objectives within a structure-aware pretraining framework, demonstrating strong cross-task transfer. Sequence-to-sequence approaches such as CodeT5 ( 23 ) and PLBART ( 1 ) further extend pretraining for code generation and understanding. Despite increasing architectural complexity, encoder-based models remain competitive for classification under moderate computational budgets ( 10 ) . 3 Dataset Analysis 3.1 Subtask A: Cross-Lingual Generalization The binary dataset exhibits near-ideal label balance (47.7% human, 52.3% machine) but presents a critical language and domain mismatch between training and evaluation. Training is restricted to Python (91%), C++, and Java, all from the algorithmic domain. The test set, however, spans unseen languages (JavaScript, C#, PHP, Go, C) and unseen domains (Research, Production), with 33.4% of samples being mixed-content snippets of unknown type. This train–test distribution shift, rather than label imbalance, is the central challenge of Subtask A (Table 1 ). Table 1: Subtask A language and domain distribution. The test set introduces five unseen languages and two unseen domains absent from training. Split Lang. Domain % Train Python Algo. 91% C++ Algo. 5% Java Algo. 4% Test Python Algo. 28.3% JavaScript Mixed 13.3% Java Mixed 12.0% C# Res./Prod. 5.4% PHP Res./Prod. 5.4% C++ Algo. 1.7% Go Res./Prod. 0.5% Unknown Mixed 33.4% 3.2 Subtask B: Severely Imbalanced Multi-Class Table 2 reveals the core challenge of Subtask B: the training data is dominated by human-written code (442K samples, 88.4%), while machine-generated samples are spread across 10 LLM families with counts ranging from 2K to 10K. The imbalance ratio between the largest class (Human) and the smallest (BigCode, Gemma) reaches 221:1 , fundamentally biasing gradient-based optimization toward the majority class. Standard cross-entropy training under this distribution causes the model to effectively predict “human” for the vast majority of inputs regardless of true label. Table 2: Subtask B class distribution. The 221:1 imbalance ratio between the majority class and the smallest minority classes poses a fundamental optimization challenge. Class Samples % of Train Human 442,000 88.4% OpenAI 10,000 2.0% Meta-LLaMA 8,000 1.6% IBM-Granite 8,000 1.6% Qwen 8,000 1.6% Phi 5,000 1.0% DeepSeek-AI 4,000 0.8% Mistral 4,000 0.8% 01-ai 3,000 0.6% BigCode 2,000 0.4% Gemma 2,000 0.4% Total 500,000 100% 4 Methodology 4.1 Subtask A: Multi-View Generalization Framework Our initial CodeBERT baseline achieved near-perfect validation F1 but failed to generalize to the test set due to the language and domain mismatch described in Section 3.1. We address this by fine-tuning UniXcoder-base ( 8 ) with