Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection

Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben-Zaken · 2026

Read Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection by Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben-Zaken (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25102.txt.

Introduction

Ruslan Berdichevsky, Shai Nahum-Gefen, and Elad Ben-Zaken build a detector for machine generated code, presented at SemEval 2026. As language models write more code, telling human from machine authorship matters for assessment integrity and software trust. The shared task frames detection as binary classification over snippets, stressing generalisation to unseen programming languages and domains. Their system, SALSA, reformulates the problem for the language model itself. Each class is assigned a dedicated output token, and the model is trained to answer with that single token in a structured response. No hand crafted features or decision rules intervene.

Robustness tricks matter as much as the formulation. Training samples are balanced across languages. Fine tuning is parameter efficient and conservative: a low learning rate and one epoch, to avoid overfitting the training domain. The system scores an F1 near 0.79 on unseen data on the leaderboard. The CodeBERT baseline manages only about 0.31.

Catalog record

CREATORRuslan Berdichevsky, Shai Nahum-Gefen, Elad Ben-Zaken
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dream at SemEval-2026 Task 13: SALSA for Single-Pass Machine-Generated Code Detection Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben-Zaken Abstract Large language models have transformed code generation, raising concerns around authorship, assessment integrity, and software trust. SemEval-2026 Task 13 Subtask A operationalizes detection as binary classification over code snippets, with a particular emphasis on out-of-distribution (OOD) generalization across unseen programming languages and application domains. We propose a SALSA-style formulation, Single-pass Autoregressive LLM Structured Classification, that maps each class to a dedicated output token and trains the model to emit a single-token label in a structured response. Rather than engineering hand-crafted features or decision rules, this formulation delegates the authorship decision to the model. To improve OOD robustness, we combine balanced sampling across languages with parameter-efficient fine-tuning and conservative training (low learning rate, single epoch) to avoid overfitting to the training domain. Our best system achieves OOD F 1 = 0.789 F_{1}=0.789 on the official leaderboard, substantially outperforming the CodeBERT baseline ( F 1 = 0.305 F_{1}=0.305 ). 1 Introduction Figure 1: Overview of the SALSA classification pipeline: the structured prompt maps each class to a single token; a single LLM forward pass produces logits for the target token position; the class-token logits are extracted and passed through a softmax to obtain the predicted label. Large language models (LLMs) have revolutionized code generation, but this has important consequences for programming skills, ethics, and assessment integrity, making the detection of LLM-generated code essential for maintaining accountability and standards 9 . While there is prior work on machine-generated code detection, it often lacks domain coverage and robustness, and typically supports only a small number of programming languages 10 ; 7 ; 8 . SemEval-2026 Task 13 Subtask A provides an important testbed by requiring a binary label indicating whether a code snippet is machine-generated. A central challenge is generalization to out-of-distribution (OOD) settings, since practical detectors must handle unseen scenarios beyond the training distribution. Rather than solving the problem via hand-designed features or decision rules, we delegate the authorship decision to the model by specifying the task as a natural language instruction, since language models are strong general-purpose learners when prompted with task descriptions 3 . However, using LLMs as classifiers this way is often brittle: the model may emit verbose explanations, inconsistent label formats, or multi-token labels. SALSA 1 1 1 Our code is publicly available at https://github.com/dreamgroupai-ai/SALSA . , Single-pass Autoregressive LLM Structured Classification ( 1 ) , addresses this by mapping each class to a distinct output token and structuring the prompt with clear delimiters and a direct authorship question, turning classification into a reliable constrained next-token prediction. During fine-tuning, the model simultaneously learns task alignment (following the classification instruction) and in-domain expertise (adapting to the training distribution). Since our primary goal is out-of-distribution (OOD) performance, we bias training toward preserving task alignment and general capabilities rather than over-fitting to the source domain. To this end, we run only one epoch and select the best checkpoint based on validation; prior work suggests that earlier training tends to improve general-purpose features, while later stages may over-specialize to the source domain and degrade OOD performance 6 . We also use a low learning rate, as we empirically observe that larger learning rates tend to reduce OOD performance while improving in-distribution metrics. Together, these choices keep the fine-tuned model closer to the base model, which tends to improve generalization. SALSA has shown consistent results across diverse classification settings, including the GLUE benchmark 15 and content moderation 14 ; 2 ; here we apply it to machine-generated code detection and report strong OOD performance ( F_{1}=0.789 ). 2 Background Subtask A is a binary classification setting: given an input code snippet, the system predicts 0 (human-written) or 1 (machine-generated). The task is designed to stress-test generalization across both programming languages and application scenarios: training data uses C++ , Python , and Java from an algorithmic domain, while evaluation includes (i) seen vs. unseen languages and (ii) seen vs. unseen domains (research/production), yielding four settings in total. We use the official dataset distribution released by the task organizers. The dataset provides the code snippet ( code ), the binary label ( label ), and metadata including programming language ( language ). The released sizes for Subtask A are 500K training samples (238K human-written and 262K machine-generated) and 100K validation samples, and a private test set of 500K samples for which no labels or metadata are provided. 3 System Overview 3.1 SALSA SALSA reframes classification as single-token generation over an autoregressive LLM. Each class is mapped to a unique output token, and the prompt is structured to elicit exactly that token at a fixed position. At inference time, a single forward pass suffices: the output logits are projected onto the class tokens only, yielding an efficient classifier with minimal decoding overhead. Figure 1 illustrates the end-to-end pipeline, and Figure 2 shows an example prompt for our task. Class-to-token mapping. For Subtask A we use label tokens "0" and "1" (a single token each under the model tokenizer). Structured prompting. We wrap the code snippet in delimiters and ask the model whether the code is machine generated. We require the answer in the form <ANSWER>#</ANSWER> where # is 0 or 1 . Figure 2 shows the prompt structure used for the task. The /no_think flag and an empty <think> block suppress chain-of-thought reasoning in Qwen3, keeping the output to a single label token 12 . <|begin_of_text|> <|im_start|> user /no_think Given the code: <CODE> s = input() pal = s + s[::-1] print(pal) </ CODE> is the code machine generated? provide answer in format: <ANSWER>#Number</ANSWER> where the number is one of the following: 0 - No 1 - Yes <|im_end|> <|im_start|> assistant <think> </think> <ANSWER>#</ANSWER> <|im_end|> BOS token Task + input Class mapping Answer format with masked token Figure 2: Example SALSA-structured prompt for machine-generated code detection (Subtask A). 3.2 Implementation Details Training objective We implement SALSA by supervising only the label token position in the assistant response. Concretely, the model is trained with a causal language modeling objective where the loss is evaluated exclusively at the final occurrence of the label token, so gradients update the model primarily to improve the next-token distribution over the target classes. Parameter-efficient fine-tuning We fine-tune with LoRA, Low-Rank Adaptation 4 , to reduce overfitting risk and speed up training. Conservative fine-tuning. We use a low learning rate ( 5{\times}10^{-6} ), train for one epoch, and select the best checkpoint on validation. We empirically observe that more aggressive training reduces OOD performance while improving in-distribution metrics. In particular, training beyond a single epoch consistently degrades OOD generalization, suggesting that extended exposure to the source domain causes the model to over-specialize at the expense of its pretrained representations. Dataset splits. Table 1 shows the per-language and per-label breakdown of the training (500K) and validation (100K) splits. The dataset is heavily skewed toward Python , which accounts for 91.5% of all samples, while C++ and Java together comprise less than 9%. The label distribution is roughly balanced overall (47.7% human vs. 52.3% machine-generated), and this balance is preserved within each language; the validation split closely mirrors the training proportions. Split Language Human (0) Machine (1) Total

Read the complete text