LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning

Mahir Labib Dihan, Abir Muhtasim · 2026

Read LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning by Mahir Labib Dihan, Abir Muhtasim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16058.txt.

Introduction

Dihan and Muhtasim present LLMSniffer, a detector that tells AI-generated code from human-written code. It fine-tunes GraphCodeBERT with a two-stage supervised contrastive pipeline, comment removal preprocessing, and an MLP classifier. On GPTSniffer, accuracy rises from 70 to 78 percent and F1 from 68 to 78. On Whodunit, accuracy rises from 91 to 94.65 percent and F1 from 91 to about 95. Plots confirm fine-tuning yields split, compact embeddings. Model checkpoints, data, code, and a live demo are public.

Catalog record

CREATORMahir Labib Dihan, Abir Muhtasim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17917
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16058

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning Mahir Labib Dihan, Abir Muhtasim Abstract The rapid proliferation of Large Language Models (LLMs) in software development has made distinguishing AI-generated code from human-written code a critical challenge with implications for academic integrity, code quality assurance, and software security. We present LLMSniffer , a detection framework that fine-tunes GraphCodeBERT using a two-stage supervised contrastive learning pipeline augmented with comment removal preprocessing and an MLP classifier. Evaluated on two benchmark datasets—GPTSniffer and Whodunit—LLMSniffer achieves substantial improvements over prior baselines: accuracy increases from 70% to 78% on GPTSniffer (F1: 68% → \to 78%) and from 91% to 94.65% on Whodunit (F1: 91% → \to 94.64%). t-SNE visualizations confirm that contrastive fine-tuning yields well-separated, compact embeddings. We release our model checkpoints, datasets, codes and a live interactive demo to facilitate further research. 1 Introduction The emergence of capable code-generating LLMs—including GitHub Copilot [ GitHub 2021 ] , ChatGPT [ OpenAI 2023 ] , and Code Llama [ Rozière et al. 2023 ] —has transformed how software is written. Developers increasingly rely on these models for routine coding tasks, bug fixes, and algorithm implementation. However, this shift introduces new challenges: plagiarism detection systems in educational settings must account for AI-generated submissions, code review pipelines need provenance information, and security audits may require distinguishing model-generated patterns from human idioms. Educators are increasingly concerned about LLM usage in programming education [ Hung et al. 2024 ] . Existing solutions such as GPTZero or DetectGPT are primarily tailored for general text and often struggle with code content due to the unique structural and syntactic properties of programming languages. Detecting LLM-generated code is fundamentally a binary classification task: given a code snippet x\in\mathcal{C} , learn a function f:\mathcal{C}\to\{0,1\} indicating whether the code is human-written ( 0 ) or LLM-generated ( 1 ). Recent efforts have focused on zero-shot detectors [ Shi et al. 2024 , Ye et al. 2024 ] and ML-based detectors [ Idialu et al. 2024 ] . We emphasize the latter, building on the GPTSniffer framework [ Nguyen et al. 2024b ] . We propose LLMSniffer , combining (1) GraphCodeBERT [ Guo et al. 2021 ] , which encodes code tokens and data-flow structure, with (2) supervised contrastive learning [ Khosla et al. 2020 ] , which clusters same-class representations while separating different-class ones, and (3) a comment-stripping preprocessing step to remove potentially confounding stylistic metadata. Our main contributions are: • A two-stage contrastive fine-tuning strategy for GraphCodeBERT with comment removal preprocessing and an MLP classification head, achieving state-of-the-art results on two benchmarks. • Detailed ablation analysis comparing a linear classifier baseline against our improved MLP-based architecture. • Public model checkpoints, datasets on Hugging Face, an interactive Streamlit demo, and a video demonstration for full reproducibility. 2 Related Work AI-Generated Text Detection. Detecting machine-generated text has received considerable attention since GPT-2 [ Radford et al. 2019 ] . Early approaches used statistical signals such as perplexity [ Gehrmann et al. 2019 ] and likelihood ratios [ Mitchell et al. 2023 ] , while more recent work focuses on fine-tuned classifiers [ Solaiman et al. 2019 ] and watermarking [ Kirchenbauer et al. 2023 ] . These methods target natural language and do not transfer directly to source code due to syntactic and structural differences. LLM-Generated Code Detection. Nguyen et al. 2024b introduced GPTSniffer , one of the first dedicated benchmarks for detecting ChatGPT-generated code, using a CodeBERT encoder trained with binary cross-entropy loss. Nguyen et al. 2024a presented Whodunit , focusing on GPT-4-generated Python solutions to CodeChef problems and demonstrating that code-specific features substantially improve detection. Idialu et al. 2024 further explored zero-shot and ML-based pipelines on competitive programming benchmarks. Our work builds directly on these benchmarks and surpasses both baselines. Code Representation Learning. Pre-trained models such as CodeBERT [ Feng et al. 2020 ] , GraphCodeBERT [ Guo et al. 2021 ] , and CodeT5 [ Wang et al. 2021 ] advance code understanding by encoding token sequences alongside structural information (ASTs, data-flow graphs). We leverage GraphCodeBERT for its demonstrated superiority on code search and clone detection. Contrastive Learning. Supervised contrastive learning [ Khosla et al. 2020 ] extends self-supervised contrastive objectives [ Chen et al. 2020 ] to the labeled setting. It has been applied to NLP fine-tuning [ Gunel et al. 2021 ] and code clone detection [ Ding et al. 2022 ] , but not, to our knowledge, to AI-generated code detection prior to this work. 3 Background 3.1 GraphCodeBERT GraphCodeBERT [ Guo et al. 2021 ] is a transformer-based model pre-trained on code–documentation pairs from CodeSearchNet [ Husain et al. 2019 ] . Unlike CodeBERT, it augments the self-attention mechanism with a data-flow graph that captures variable-level semantic dependencies (e.g., where a variable is computed and where it is used ). Two additional pre-training objectives—edge prediction and node alignment—encourage the model to internalize these structural relationships. We use the [CLS] token representation as a fixed-dimensional embedding \mathbf{h}\in\mathbb{R}^{768} for downstream classification. 3.2 Supervised Contrastive Learning Given a batch of N labeled pairs \{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N} , the supervised contrastive loss [ Khosla et al. 2020 ] is: \mathcal{L}_{\mathrm{SupCon}}=\sum_{i=1}^{N}\frac{-1}{|P(i)|}\sum_{p\in P(i)}\log\frac{e^{\mathbf{z}_{i}\cdot\mathbf{z}_{p}/\tau}}{\displaystyle\sum_{a\in A(i)}e^{\mathbf{z}_{i}\cdot\mathbf{z}_{a}/\tau}} (1) where \mathbf{z}_{i}=\ell_{2}\text{-normalize}(\mathrm{MLP}(\mathbf{h}_{i})) is the projected representation, P(i)=\{p\neq i:y_{p}=y_{i}\} is the set of same-class positives, A(i)=\{a\neq i\} is all other examples, and \tau is a temperature hyperparameter. This objective directly optimizes for inter-class separation and intra-class compactness in embedding space. 4 Methodology Figure 1: Overview of LLMSniffer. Stage 1 trains the encoder and projection head with supervised contrastive loss. Stage 2 freezes the encoder and trains an MLP classifier with binary cross-entropy loss. 4.1 Preprocessing: Comment Removal A key design decision in LLMSniffer is removing code comments before encoding. LLM-generated code often includes verbose, explanatory comments that reflect LLM writing style rather than algorithmic structure. Stripping comments ensures the model learns to distinguish based on syntactic and structural code properties, reducing the risk of exploiting superficial stylistic cues and improving generalization. 4.2 Model Architecture LLMSniffer consists of three components (Figure 1 ): (1) GraphCodeBERT Encoder. After comment removal, the code snippet is tokenized and fed along with its data-flow graph edges into GraphCodeBERT. The [CLS] embedding \mathbf{h}\in\mathbb{R}^{768} serves as the code representation. (2) Projection Head. A two-layer MLP with ReLU activation maps \mathbf{h} to \mathbf{z}\in\mathbb{R}^{128} for contrastive learning. The head is discarded at inference time, following Chen et al. 2020 . (3) MLP Classification Head. A multi-layer perceptron with sigmoid activation maps \mathbf{h} to a binary prediction (human-written or AI-generated). We find that this MLP head outperforms the simpler linear classifier used in prior work. 4.3 Training Procedure Training proceeds in two stages. Stage 1: Contrastive Pre-training. We fine-tune the encoder and projection head using \mathcal{L}_{\mathrm{SupCon}} (Equation 1 ) with temperature \tau{=}0.07 . Batches are constructed with balanced classes to maximize the number of positive pairs per anchor. Stage 2: Classification Fine-tuning. The projection head is discarded, the MLP classification head is attached, and the full model is fine-tuned end-to-end with binary cross-entropy loss. The contrastive pre-training step warms up the encoder to produce discriminative representations before the classifier is trained, following the approach of Khosla et al. 2020 . 4.4 Implementation Details We initialize from the microsoft/graphcodebert-base checkpoint ( HuggingFace ). Both

Read the complete text