FMI_SU_Yotkova_Kastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals

Elitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov · 2026

Read FMI_SU_Yotkova_Kastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals by Elitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04157.txt.

Introduction

Elitsa Yotkova and four co-authors describe a lightweight system for detecting machine-generated code, built for SemEval-2026 Task 13, which asks systems to generalize across programming languages and application scenarios. Their entry targets the binary subtask. Instead of leaning on large pretrained code encoders alone, they engineer stylometric signals designed to be robust to snippet length.

Ratio-based features carry much of the load. Parsing engines and a programming-language classifier support descriptive signals, and a separate code-versus-text line classifier flags natural language segments embedded inside samples. A shallow decision tree combined with heuristic rules from data analysis produces the final predictions. The approach is cheap to run. It trains on CPU resources only. It offers near-instant inference as an alternative to heavy pretrained models.

Catalog record

CREATORElitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04157

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FMI_SU_Yotkova_Kastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals Elitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov Abstract SemEval-2026 Task 13 investigates machine-generated code detection across multiple programming languages and application scenarios, asking participating systems to generalize to unseen languages and domains. This paper describes our participation in Subtask A (binary classification) and explores both pretrained code encoders and lightweight feature-based methods. We design ratio-based features that are less sensitive to snippet length. To support the extraction of descriptiveness-related signals, we use parsing engines and a programming-language classifier. Additionally, we train a separate code-vs-text line classifier to identify raw natural language segments embedded within samples. We combine a shallow decision tree with heuristic rules derived from data analysis to produce the final predictions. Our approach is computationally efficient, requires only CPU resources for training, and achieves near-instant inference time, offering a lightweight alternative to large pretrained models. Our code is available at https://github.com/violeta-kastreva/SemEvalTask13-SubtaskA . 1 Introduction The widespread adoption of large language models (LLMs) for code generation has reshaped modern software development. Current models can generate syntactically correct and semantically meaningful programs across many programming languages and domains. This has led to increased presence of machine-generated code in cases such as automated unit test generation 6 , code infilling and completion 1 , and production development workflows ( 3 ; 4 ) . While LLM-based tools offer clear productivity benefits, their ability to author and refine code raises concerns in domains where human authorship is critical. In academia, detecting code authorship in programming assignments is increasingly important, as reliance on LLMs undermines educational integrity 17 . Similarly, fair technical hiring requires verifying that submitted artifacts are genuinely human-written. These use cases motivate the need for robust detection of machine-generated code. In an industrial setting, machine-generated code can pose technical risks when deployed without sufficient oversight. Prior work has shown that LLM-generated code may contain insecure logic, hidden backdoors, or injection vulnerabilities, threatening software reliability and data security ( 2 ; 14 ) . Distinguishing machine-generated code from human-written one is therefore a critical challenge. Unlike natural language, source code is tightly constrained by syntax and semantics, which can obscure stylistic differences between human and machine authors, which are commonly used for machine-generated text detection. In practice, code detection systems must generalize across programming languages, application domains, and generator families, and many existing approaches degrade substantially in such conditions 13 . Recent work has largely focused on pretrained code encoders and large neural models for authorship detection. These models can be effective in in-domain settings, but their performance and reliability can vary under distribution shift, particularly for unseen languages or domains. Moreover, their predictions are often difficult to interpret, making it harder to identify the properties of code that distinguish human-written from machine-generated samples. This paper presents our approach to SemEval-2026 Task 13 Subtask A 12 , which targets binary classification of machine-generated versus human-written code under diverse and challenging generalization settings. Instead of relying solely on high-capacity neural representations, we explore lightweight, interpretable signals that capture descriptiveness and stylistic leakage commonly observed in LLM-generated code. Our approach is motivated by the observation that machine-generated code often includes verbose comments, explanatory text, or natural-language artifacts that differ in both frequency and structure from human-written code. We propose a feature-based detection pipeline centered on ratio-based descriptors that are less sensitive to code length and formatting variability. To ensure robust feature extraction across incomplete or noisy code samples, we employ parsing engines and introduce a dedicated programming language identification classifier. In addition, we train an auxiliary code-versus-text line classifier to detect natural language segments embedded within samples. Our final system combines shallow machine learning classifiers with heuristic rules, prioritizing robustness and interpretability over model complexity. Through extensive experimentation, we compare pretrained encoder-based representations with feature-driven methods and analyze the strengths and limitations of each approach under cross-language and cross-domain evaluation. On the official leaderboard, our approach is ranked within the top 15% of submissions. Our contributions are as follows: • We analyze the limitations of pretrained code encoders for code detection under distribution shift. • We introduce interpretable ratio-based features that capture descriptiveness and stylistic signals in LLM-generated code, and use them to train classical machine learning classifiers. • We construct a dataset for code-vs-text line classification, 1 1 1 violetakastreva/line-level-code-vs-text-classification-dataset 2 2 2 violetakastreva/line-level-code-vs-text-classification enabling line-level discrimination between natural language text and source code. • We present a lightweight detection pipeline that performs competitively in SemEval-2026 Task 13 while remaining simple and interpretable. 2 Related Work Prior work on detecting machine-generated code has emphasized the role of code stylometric features, showing that LLM-generated code differs from human-written code in surface and structural properties such as line statistics, comment usage, and complexity measures 15 . Studies on Claude 3-generated code report that features related to the number of lines, blank lines, comment ratio, and cyclomatic complexity are particularly informative, with generated code often exhibiting simpler structure and distinct comment-to-code patterns compared to human-written code 15 . Other approaches move beyond stylometry by leveraging model-internal signals. Some methods compute token-level likelihoods from large language models and frame detection as a classification problem over probability maps using vision architectures 20 . In addition, pretrained code encoders are commonly used as feature extractors for supervised classification of code authorship, providing strong semantic representations at the cost of reduced interpretability ( 16 ; 10 ) . 3 Dataset We use the dataset provided for SemEval-2026 Task 13 ( 11 ; 13 ) Subtask A, which targets binary classification between machine-generated and human-written code. Each sample is labeled as either fully human-written or fully machine-generated. The training data consists of algorithmic code snippets written in C++, Python, and Java, while evaluation additionally includes unseen programming languages (Go, PHP, C#, C, and JavaScript) and unseen application domains (research and production code). The dataset contains 500K training samples (238K human-written and 262K machine-generated), of which approximately 457K are written in Python, a 100K validation set, and a 500K test set. The participating systems are evaluated using macro-F1 scores across multiple settings that assess generalization across languages and domains. 4 Methodology Our system is built around feature-based representations supported by several auxiliary classifiers to enable robust detection of machine-generated code across multiple programming languages and heterogeneous input formats as shown on Figure 1 . Figure 1: Overview of the proposed machine-generated code detection pipeline. 4.1 Feature Extraction and Engineering We design ratio-based, interpretable, length-normalized features that capture stylistic descriptiveness and structural cues often present in LLM-generated code. Table 1 lists our stylometric ratio features, while Table 2 summarizes our syntactic and structural features, which we derive from parsing and complexity estimates. Feature Definition Comment Ratio comment lines to total lines in the code snippet Verb Comment Ratio verb tokens to all word tokens in comment text Text-like Ratio lines classified as natural language text to total number of code lines Identifier Verb Ratio verb tokens to total number of identifier tokens; the identifiers are decomposed into their constituent word units before token counting Comment Code-like Ratio comment lines exhibiting code-like patterns to all comment lines Table 1: Code stylometric ratio features computed at the snippet

Read the complete text