SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generation

Naizhu Jin, *, Yinggang Guo, Chao Su, Tian Zhang, Qingkai Zeng · 2024

Read SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generation by Naizhu Jin, *, Yinggang Guo, Chao Su, Tian Zhang, Qingkai Zeng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05829.txt.

Introduction

Naizhu Jin, Zhong Li, Yinggang Guo, Chao Su, Tian Zhang, and Qingkai Zeng wrote this paper in two thousand twenty-four. Its head author line adds one name to the catalog list. It shows how a backdoor can be planted in code models that reason step by step.

Their attack, SABER, picks a bad output by mutating code. It finds the tokens tied to it through self-attention scores in CodeBERT, then crafts a natural trigger that mimics a user. On test sets the attack fired far more often than prior tricks. Auto detectors missed most of its poison, and human checkers caught almost none. The work warns that chain-of-thought code models need safety study of their own.

Catalog record

CREATORNaizhu Jin, *, Yinggang Guo, Chao Su, Tian Zhang, Qingkai Zeng
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05829

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generation Naizhu Jin, *, Yinggang Guo, Chao Su, Tian Zhang, Qingkai Zeng Abstract Recent studies have proposed integrating Chain-of-Thought (CoT) reasoning to further enhance the reliability of Code Language Models (CLMs) in generating code, a step-by-step approach that breaks down complex programming tasks into manageable sub-problems. Advances in this area have introduced CoT models, specifically designed to integrate CoT reasoning effectively into language models, achieving notable improvements in code generation. Despite these advancements, the security of CoT models has not been systematically studied. In this study, we aim to fill this gap by investigating the vulnerability of CoT models to backdoor injection in code generation tasks. To address this, we propose a model-agnostic backdoor attack method SABER ( S elf- A ttention- B as E d backdoo R ) based on the self-attention mechanism. SABER begins by selecting a malicious output as the backdoor using code mutation operations. It then identifies the tokens most relevant to poisoned content by analyzing self-attention scores in the CodeBERT model. Finally, it mimicks user behavior to generate adaptive and natural triggers. Our experiments on HumanEval-CoT and OpenEval-CoT test sets demonstrate that CoT models are susceptible to backdoor attacks via data poisoning. Taking the HumanEval-CoT dataset as an example, SABER achieves an ASR of 80.95%, representing an improvement of 33.33% over RIPPLe and a substantial 4.76% enhancement compared to BadPre. Further evaluations using ONION for automated detection and human studies reveal that SABER is stealthier and harder to detect, bypassing 61.90% of automated detection, with a human detection rate of just 3.17%. Our findings reveal that backdoors can be injected into CoT models to manipulate downstream code generation tasks. This highlights the urgent need for further research to understand and mitigate the security vulnerabilities in CoT models. We advocate for increased efforts to explore the security threats facing CoT models and to develop more effective countermeasures. keywords Chain-of-Thought, Neural Code Generation, Backdoor Attack, Data Poisoning † † fcssetup: received = month dd, yyyy, accepted = month dd, yyyy, corr-email = [email protected], [1]Zhong Li 1 Introduction With the rapid advancement of Large Language Models (LLMs), Code Language Models (CLMs) are widely used in software development to assist the process of code generation [ 1 , 2 ] . Particularly, CLMs often leverage Chain-of-Thought (CoT) reasoning to enhance the quality and reliability of generated code. CoT reasoning decomposes tasks into structured steps, significantly enhancing the quality, correctness, and interpretability of generated code [ 3 , 4 , 5 ] . Given the challenges of manually constructing high-quality CoT, researchers have introduced CoT models that aim to automate the generation of high-quality CoT in resource-constrained scenarios [ 6 , 7 ] . For instance, the CoT model COTTON [ 8 ] leverages lightweight language models ( \ell LMs) to automatically generate high-quality CoT prompts for code generation. Despite the effectiveness of CoT models in enhancing code generation, the security of these models has not been systematically studied. One of the typical threats to CoT models is the backdoor attack. The backdoor in CoT models involves introducing trigger mechanisms to manipulate the reasoning steps, inducing malicious or incorrect outputs under certain conditions while maintaining normal performance. Formally, let D_{\text{poison}} be a poisoned training dataset and M_{\text{PCoT}} be a backdoored CoT model trained on D_{\text{poison}} . When M_{\text{PCoT}} is deployed, the adversaries could leverage M_{\text{PCoT}} to generate poisoned CoT prompts y^{p} to manipulate the downstream CLMs for generating malicious or incorrect outputs. Therefore, this study focuses on backdoor attacks as a key approach to systematically evaluate the security of CoT models. Unlike traditional backdoor attacks that directly target models, this research explores the vulnerabilities of CoT models within the CoT and CLM frameworks. The reason for selecting CoT models as the target of attack is that attacking the publicly deployed foundational model CLM is much more costly than attacking a locally controllable and lightweight CoT model. State-of-the-art CLM models, such as DeepSeekCoder [ 9 ] and Qwen2.5Coder [ 10 ] , are predominantly black-box code generation models, limiting the feasibility of direct manipulation or poisoning attacks. Therefore, it is more realistic to control the behavior of CLM through the CoT model. Backdoor attacks have been extensively studied across various fields in prior research. In computer vision, the focus has been on minimizing the distinguishability between poisoned and original images [ 11 ] . In the text domain, Dai et al. [ 12 ] introduced the first backdoor attack by embedding triggers into LSTM-based text classification models. Later, methods such as BadNL [ 13 ] improved stealth by using triggers at different levels, and weight perturbation strategies [ 14 ] were applied during fine-tuning to embed backdoors while preserving model performance. Xiang et al. [ 15 ] proposed the BadChain method, which leverages in-context learning capabilities to attack large language models through prompts directly. Ramakrishnan et al. [ 16 ] employed dead code triggers to address backdoor attacks in programming tasks. However, these approaches fails to meet the requirements of CoT models, which rely on natural language inputs. In our scenario, the attacker can only influence downstream code generation tasks by manipulating the CoT model. Therefore, when designing backdoors, it is necessary to ensure that the implanted backdoor can affect the downstream code generation model through the CoT model without compromising the effectiveness of the generated code. Additionally, due to the higher demands for semantic and logical consistency in CoT models, using word, phrase, sentence, or dead code triggers not only makes them more detectable but also alters the input’s meaning, potentially affecting the model’s performance. Therefore, the design of backdoor triggers for CoT models in this study must meet the dual constraints of stealthiness (avoiding disruption of code semantics) and effectiveness (maintaining logical integrity), a challenge that has not been sufficiently explored in existing research. To address this challenge, we propose a novel attack method named SABER to inject adaptive backdoor triggers into CoT models by poisoning the datasets. The key idea of SABER is to use the self-attention mechanism to implement adaptive backdoor triggers. Specifically, SABER injects a trigger into the input of the CoT model through three key steps. First, it searches for backdoor features within code examples using code mutation. Next, SABER employs the self-attention scores calculated by CodeBert [ 17 ] to identify the most important token with the backdoor feature. Finally, it adds triggers to the most important tokens by mimicking user behavior. As such, SABER injects triggers on the most critical and effective words, manipulating downstream models while ensuring both stealthiness and effectiveness. To evaluate the effectiveness and stealthiness of SABER , we conduct extensive experiments on the HumanEval-CoT and OpenEval-CoT test sets, which extend HumanEval and OpenEval, respectively. At a poisoning ratio of 6%, SABER achieves an ASR of 80.95% on HumanEval-CoT, surpassing RIPPLe by 33.33% and BadPre by 4.76%, while on OpenEval-CoT, it attains an ASR of 72.73%, exceeding RIPPLe by 13.64% and BadPre by 4.55%. These results demonstrate that SABER consistently achieves a high ASR across different poisoning ratios while maintaining minimal impact on benign task performance. Furthermore, SABER exhibits strong resilience against automated detection; under ONION, it achieves an ASR of

Read the complete text