Injecting Bias into Text Classification Models using Backdoor Attacks
A. Dilara Yavuz, M. Emre Gursoy · 2024
Read Injecting Bias into Text Classification Models using Backdoor Attacks by A. Dilara Yavuz, M. Emre Gursoy (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18975.txt.
Introduction
A poisoned training set can teach a text classifier to hate on cue, demonstrate Dilara Yavuz and Emre Gursoy in a 2024 arXiv paper. Their backdoor attack links strong male actors to negative sentiment in movie and review data, then tests whether the bias sticks across seven models.
With as little as three percent poison, the attack succeeded every time, while normal accuracy barely moved. Modern transformer models were the most vulnerable, not the least. Two extra measures showed the bias carried to unseen words and paraphrases. The authors present bias injection as a quiet misuse of a known attack family.
Catalog record
| CREATOR | A. Dilara Yavuz, M. Emre Gursoy |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Injecting Bias into Text Classification Models using Backdoor Attacks A. Dilara Yavuz, M. Emre Gursoy Abstract The rapid growth of natural language processing (NLP) and pre-trained language models have enabled accurate text classification in a variety of settings. However, text classification models are susceptible to backdoor attacks, where an attacker embeds a trigger into the victim model to make the model predict attacker-desired labels in targeted scenarios. In this paper, we propose to utilize backdoor attacks for a new purpose: bias injection. We develop a backdoor attack in which a subset of the training dataset is poisoned to associate strong male actors with negative sentiment. We execute our attack on two popular text classification datasets (IMDb and SST) and seven different models ranging from traditional Doc2Vec-based models to LSTM networks and modern transformer-based BERT and RoBERTa models. Our results show that the reduction in backdoored models’ benign classification accuracy is limited, implying that our attacks remain stealthy, whereas the models successfully learn to associate strong male actors with negative sentiment (100% attack success rate with ≥ \geq 3% poison rate). Attacks on BERT and RoBERTa are particularly more stealthy and effective, demonstrating an increased risk of using modern and larger models. We also measure the generalizability of our bias injection by proposing two metrics: (i) U-BBSR which uses previously unseen words when measuring attack success, and (ii) P-BBSR which measures attack success using paraphrased test samples. U-BBSR and P-BBSR results show that the bias injected by our attack can go beyond memorizing a trigger phrase. 1 Introduction In today’s world, artificial intelligence (AI) and natural language processing (NLP) are growing rapidly and invading many sectors. The development and widespread success of language models such as BERT, LLaMa, and the GPT family have revolutionized the capabilities of NLP applications, allowing more accurate and efficient processing of natural language. A fundamental application of NLP is text classification , where the model assigns a label to a given piece of text, such as an e-mail, movie review, tweet, or document. Common applications of text classification include sentiment analysis, spam filtering, phishing detection, and fraud detection, many of which are critical in cybersecurity. On the other hand, despite their popularity and success, AI-powered text classification models were shown to be vulnerable to backdoor attacks [ 5 , 6 , 12 , 14 , 19 , 20 ] . In a backdoor attack, the attacker injects a backdoor into the victim model which is activated by a trigger pattern. The backdoored model behaves normally for test samples that do not contain the trigger (e.g., makes correct predictions), but it behaves in the way desired by the attacker (e.g., makes attacker-desired predictions) when it sees test samples containing the trigger pattern. Backdoor attacks on text classification models can have serious implications affecting both their integrity and reliability, such as incorrect labeling of spam messages, harmful content, and legal or financial documents. In this paper, we propose the use of backdoor attacks for a novel purpose: bias injection. Towards this end, we propose to poison a subset of the victim model’s training dataset using trigger words and phrases. We demonstrate our attacks using two popular datasets (IMDb and SST) and seven different text classification models, ranging from traditional models (Doc2Vec with ML models such as Logistic Regression and Naive Bayes), Long Short-Term Memory (LSTM) networks, and fine-tuned transformer models (BERT and RoBERTa). We measure the impacts of our attacks using four metrics: Benign Classification Accuracy (BCA), Bias Backdoor Success Rate (BBSR), BBSR on unseen words (U-BBSR), and BBSR on paraphrased samples (P-BBSR). BCA captures the accuracy of the model on benign test samples, and it is desired that BCA remains high (from an attacker’s perspective) so that the attack remains stealthy. BBSR measures the proportion of instances where an originally positive test sample is predicted as negative by the backdoored model because of the trigger pattern. U-BBSR and P-BBSR are novel metrics that we propose in this paper, aimed at measuring how well our attacks induce generalizable bias in the victim models: • Since BBSR uses the same trigger in training and test samples, it can be prone to measuring trigger word memorization rather than general bias in the victim model. Therefore, in U-BBSR, we propose to use previously unseen words in test samples when measuring attack success. • In P-BBSR, we propose to measure attack success using paraphrased test samples. After injecting the trigger pattern into test samples, we paraphrase them using a popular paraphraser (ChatGPT Paraphraser on T5 Base), and then measure if the paraphrased samples are predicted in the attacker-desired way by the backdoored model. Since the paraphraser changes both the word choices and sentence structures, the possibility of memorization is reduced and achieving high P-BBSR is more challenging compared to other metrics. Our results show that the reduction in backdoored models’ BCAs is limited compared to their benign versions, e.g., with 10% poison rate, BCA reductions are typically less than 2-3%. Therefore, our attacks are able to remain stealthy. BCA reductions are even less for modern BERT and RoBERTa models compared to Doc2Vec and LSTM-based models. In terms of BBSR, we observe that many backdoored models (especially LSTM, BERT, RoBERTa) reach BBSR = 1 with poison rates \geq 3\% , which shows that our attacks are highly effective under low poison rates. Combining the fact that BERT and RoBERTa have both higher BBSRs and higher BCAs compared to other models, we find that attacks on these more modern models can be more stealthy and effective simultaneously. U-BBSR values are generally lower than BBSR, but nevertheless, for poison rates \geq 5\% , U-BBSR values reach 1 for BERT and RoBERTa. This shows that the models not only memorize the trigger word but can also produce biased predictions for previously unseen words. Since P-BBSR is the most challenging setting, P-BBSR values are typically the lowest. Yet, across various poison rates, the backdoored models have noticeable P-BBSRs. For example, LSTMs have P-BBSR = 1 on the IMDb dataset, BERT and RoBERTa have P-BBSRs close to 0.3 and 0.4 on the SST dataset. These results indicate that our attacks are able to inject bias into victim models even in the presence of paraphrasing. We then measure how the selection of the injected trigger word during training affects attack success. Results indicate that if the trigger word is rare in the original dataset, victim models have higher BBSR since they are more likely to associate that word with the attacker-desired label. Finally, we measure how the selection of the unseen word at test time affects U-BBSR. For each trigger word, we find multiple unseen words with varying cosine distances to the trigger according to GloVe embeddings and measure U-BBSR using these unseen words and varying models. We observe that some models’ U-BBSR remains high despite increasing cosine distances, suggesting that backdoored models produce biased predictions despite being queried with different unseen words. Yet, there also exist models in which increasing the cosine distance causes U-BBSRs to decrease, supporting the intuition that increasing the distance between the training-time trigger and test-time word will decrease attack success rates. The rest of this paper is organized as follows. In Section 2