Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains
Yuan Gao, Zhigang Liu, Xinyu Yao, Bo Chen, Xiaobing Zhao · 2026
Read Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains by Yuan Gao, Zhigang Liu, Xinyu Yao, Bo Chen, Xiaobing Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13137.txt.
Introduction
Yuan Gao, Zhigang Liu, Xinyu Yao, Bo Chen, and Xiaobing Zhao aim to keep large language models in line on touchy topics. As such models spread, bias and drift in core values have shown up. The drift is sharpest on race, social, and state topics.
The team puts forward a frame based on matched foes. It joins three steps. First comes more pre train. Next comes teach by task prompts. Last comes foe train, where hard cases push the model to hold its line. Their goal is a model whose values stay more even across touchy fields. For firms that ship such models, the frame offers one path to more trust in frail zones.
Catalog record
| CREATOR | Yuan Gao, Zhigang Liu, Xinyu Yao, Bo Chen, Xiaobing Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains Yuan Gao, Zhigang Liu, Xinyu Yao, Bo Chen, Xiaobing Zhao Abstract With the wide application of large language models (LLMs), the problems of bias and value inconsistency in sensitive domains have gradually emerged, especially in terms of race, society and politics. In this paper, we propose an adversarial alignment framework, which enhances the value consistency of the model in sensitive domains through continued pre-training, instruction fine-tuning and adversarial training. In adversarial training, we use the Attacker to generate controversial queries, the Actor to generate responses with value consistency, and the Critic to filter and ensure response quality. Furthermore, we train a V alue- C onsistent L arge L anguage M odel, VC-LLM, for sensitive domains, and construct a bilingual evaluation dataset in Chinese and English. The experimental results show that VC-LLM performs better than the existing mainstream models in both Chinese and English tests, verifying the effectiveness of the method. Warning: This paper contains examples of LLMs that are offensive or harmful in nature. 1 1 footnotetext: These authors contributed equally to this work. 2 2 footnotetext: Corresponding author: Bo Chen (Email: [email protected]). K eywords Large Language Models; Adversarial Alignment; Value Consistency 1 Introduction The rapid development of large language models (LLMs, such as GPT-4 [ 2 ] , Llama-3 [ 14 ] , Qwen2.5 [ 30 ] , and GLM-4 [ 17 ] ) has led to their widespread application across various downstream tasks [ 41 , 39 , 23 , 11 ] . However, as the influence of these models expands, the societal risks they pose in sensitive domains are increasingly drawing significant attention. Recent studies have shown that LLMs exhibit value biases related to race [ 1 ] , occupation [ 26 ] , society [ 15 ] , and politics [ 5 , 24 , 32 ] , which stem from inherent value biases in the training data [ 28 ] . Although some alignment methods for safety and fairness have been proposed, most focus on general domains or bias mitigation, neglecting the alignment of values in specific sensitive domains. Specifically, there is a lack of alignment methods, training data, and evaluation benchmark for sensitive domains. To address these challenges, we propose an adversarial alignment framework, which incorporates continued pre-training, instruction fine-tuning, and adversarial training. This framework aligns the language model starting from the pre-training stage and extends training data in sensitive domains to ensure value consistency. The core of the framework lies in adversarial training, which consists of three components: the Attacker generates queries in sensitive domains, the Actor produces responses based on value alignment principles, and the Critic reviews the generated responses to ensure value alignment. Our contributions can be summarized as follows: • We propose an adversarial alignment framework to generate value alignment datasets in sensitive domains, which automatically produces high-quality and challenging alignment data. • Based on this framework, we train VC-LLM, a large language model with value consistency in Chinese sensitive domains, and publicly release the constructed value alignment dataset 1 1 1 https://github.com/Ad-Align/VC-LLM . • We construct a bilingual value alignment evaluation benchmark in Chinese and English, and conduct comparative experiments and analysis on mainstream LLMs and VC-LLM based on this benchmark. 2 Related Work 2.1 Value Bias in LLMs Numerous studies have revealed that LLMs exhibit value-laden biases across domains such as religion, social expectations, and political ideology. For example, Abid et al. [ 1 ] find that GPT-3 demonstrates significant anti-Muslim bias, while Salecha et al. [ 33 ] show that LLMs tend to overrepresent overly positive personality traits when simulating human characteristics. Bernardelle et al. [ 7 ] report that models exhibit inherent leanings in political compass tests, which are difficult to mitigate through simple prompt engineering. Similarly, LLMs can display political favoritism and sycophantic behavior toward different political parties [ 27 , 6 ] . In addition, LLMs often show reluctance to answer political questions [ 21 ] , diverse mental values [ 25 ] , and differences in moral and political stances [ 22 , 37 ] . 2.2 Value Alignment in LLMs Recent studies have focused on the alignment of LLMs for safety [ 31 , 42 , 36 , 34 , 35 ] . For example, Bai et al. [ 4 ] combine preference modeling with human feedback-based reinforcement learning to achieve safe alignment. These methods primarily aim to enhance the model’s helpfulness and safety, or align it with globally recognized values. However, their application in sensitive domains remains limited. To address this challenge, some research has begun exploring alignment in specific domains. For instance, Liu et al. [ 24 ] use reinforcement learning to reduce political bias in LLM responses, Agiza et al. [ 3 ] optimize political bias through instruction fine-tuning and preference adjustment, while Borah et al. [ 8 ] propose self-reflection strategies to detect and mitigate hidden biases. Although progress has been made in sensitive domains alignment, these methods often focus on identifying and mitigating bias, without addressing the alignment of domain-specific values. We propose an adversarial alignment framework, aiming to ensure alignment of LLMs in sensitive domains. 3 Adversarial Value Alignment for Sensitive Domains We propose an adversarial alignment framework to enhance the value alignment of LLMs in various sensitive domains. We firstly introduce the framework, followed by its core component: adversarial training. Finally, we apply this framework in the context of Chinese sensitive domains, resulting in the trained VC-LLM. 3.1 Training Framework The adversarial alignment framework consists of three stages: Continued Pre-training. To mitigate the bias caused by value misalignment in the original pre-training data, we perform continued pre-training using value-aligned data from sensitive domains. Instruction Fine-Tuning. In this stage, we design two instruction formats to improve the model’s value alignment in sensitive domains: (1) Question-Answering Task : Given a question, the model is required to generate a value-aligned answer; (2) Statement-Rebuttal Task : Given a value-inconsistent statement, the model needs to identify value conflicts and produce a rebuttal that reflects aligned values. Adversarial Training. We introduce adversarial training to augment the training data and enhance the model’s robustness. Figure 1: Overview of the adversarial training framework. The framework begins with sensitive words (e.g., Taiwan sovereignty). The Attacker generates challenging queries (e.g., What is the truth behind the Taiwan independence movement?), the Actor produces value-aligned responses (e.g., emphasizing national unity), and the Critic filters for high-quality Q&R (Query & Response) pairs to construct training data. 3.2 Adversarial Training We propose a collaborative adversarial training process consisting of the Attacker , Actor , and Critic , designed to expand the training data and enhance the model’s value alignment and safety, as shown in Figure 1 . Using the Attacker and Actor to generate queries and responses, this approach mitigates the biases and consistency issues that may arise from using a single model. Additionally, the Critic ensures the quality of training data. The pseudocode is as follows: Algorithm 1 Adversarial Alignment Dataset Generation Input: Sensitive word list S (e.g., [Taiwan, Zhenbao Island, ...] ), Template library T Output : Adversarial alignment dataset D_{\text{adv}} (query-response pairs) 1: Initialize D_{\text{adv}}\leftarrow\emptyset 2: for each sensitive topic s\in S do 3: