AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
Tung-Ling Li, Yuhao Wu, Hongliang Liu · 2025
Read AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens by Tung-Ling Li, Yuhao Wu, Hongliang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17375.txt.
Introduction
Tung-Ling Li, Yuhao Wu, and Hongliang Liu show how easily an AI judge can be fooled into saying yes. Many training pipelines let a language model grade answers with a yes or no verdict, and that verdict steers further training. The team finds the verdict rests on one shallow readout inside the model, easy to tip.
Their attack, AdvJudge-Zero, fishes odd tokens from the judge's own output habits and slips them into a reply. The tokens flipped verdicts in over 90 percent of cases in 22 of 24 test setups across six judge families. The same token pool then trained a defense that held up across model families. In training runs, the hardened judge avoided the reward collapses seen with the plain one.
Catalog record
| CREATOR | Tung-Ling Li, Yuhao Wu, Hongliang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 103 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-17375
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens Tung-Ling Li, Yuhao Wu, Hongliang Liu Abstract Reward models and LLM-as-a-Judge systems are central to modern post-training pipelines such as RLHF, DPO, and RLAIF, where they provide scalar feedback and binary decisions that guide model selection and RL-based fine-tuning. We show that these judge systems exhibit a recurring vulnerability: short sequences of low-perplexity control tokens can flip many binary evaluations from correct “No” judgments to incorrect “Yes” judgments by steering the last-layer logit gap F = z no − z yes F=z_{\text{no}}-z_{\text{yes}} . These control tokens are patterns that a policy model could plausibly generate during post-training, and thus represent realistic reward-hacking risks rather than worst-case adversarial strings. Our method, AdvJudge-Zero, uses the model’s next-token distribution and beam-search exploration to discover diverse control-token sequences from scratch, and our analysis shows that the induced hidden-state perturbations concentrate in a low-rank “soft mode” that is anti-aligned with the judge’s refusal direction. Empirically, these tokens cause very high false positive rates when large open-weight and specialized judge models score incorrect answers on math and reasoning benchmarks. Finally, we show that LoRA-based adversarial training on small sets of control-token–augmented examples can markedly reduce these false positives while preserving evaluation quality. 1 Introduction Post-training is now the dominant way to turn large language models (LLMs) into helpful and capable assistants. Typical pipelines combine supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and preference-based objectives such as DPO and RLAIF ( 22 ; 25 ; 16 ; 27 ) . In these pipelines, reward models and LLM-as-a-Judge systems provide most of the evaluation signals ( 23 ; 17 ) . The judge models perform binary evaluations (for example correct/incorrect or yes/no) that guide both model selection and RL-based policy updates. Recent work such as Master-RM ( 35 ) has shown that these LLM-as-a-Judge systems can be fooled by short control token sequences. However, that work starts from a small, manually curated seed set of triggers and explores their local variations. Empirically, we observe that many judge decisions appear to be concentrated near a narrow, high-gain surface at the final-layer readout, suggesting the existence of many more model-intrinsic control tokens beyond a few seed patterns. In this paper we study the consequence of that concentration: binary evaluation flipping through adversarial control tokens . By binary evaluation flipping we mean that small, carefully crafted prompt edits steer the model’s last-layer logit gap F\;=\;z_{\text{no}}\;-\;z_{\text{yes}} across zero on the first output token, switching the model’s evaluation from “No” to “Yes” while leaving the question and response essentially unchanged. Unlike prior work that relies on manually curated jailbreak patterns or seed triggers ( 37 ; 31 ; 4 ; 19 ) , we introduce AdvJudge-Zero , a general zero-seed method that uses the model’s own next-token distribution, together with a beam-search exploration strategy, to discover many diverse control-token sequences that flip binary evaluations ( 18 ) . Figure 1 illustrates this binary decision flip on a MATH example ( 15 ) . A small policy model, steered by a control-token pattern, outputs an answer that is clearly inconsistent with the reference solution, yet a large LLM-as-a-Judge still marks it as correct. This example shows how a short control-token sequence can change the judge’s binary verdict without improving the underlying solution. Figure 1: Control tokens discovered by AdvJudge-Zero flip the judge’s Yes/No decision on a math solution by steering the last-layer logit gap, without improving the solution itself. Our interpretation is geometric. The transformer ( 29 ) body maps an input prompt X to a final hidden state \bm{h}_{L}(X) that retains almost all prompt semantics, consistent with recent work on injectivity in language models ( 21 ) . Post-training then installs a gate at the readout that separates “No”-like and “Yes”-like responses, in line with observations that safety alignment often acts only on the first few tokens ( 24 ) . Empirically, small prompt edits do not erase X ; they move \bm{h}_{L}(X) locally in representation space. Because this gate is shallow and high-gain, these local moves are often sufficient to change the sign of F for a wide range of inputs. In this view, control tokens are model-intrinsic directions in the last-layer space that align with this high-gain surface, rather than dataset-specific artifacts. Focus: natural vulnerabilities in post-training. Our focus in this paper is on vulnerabilities that emerge naturally during post-training, rather than on worst-case arbitrary strings. In RLHF/DPO/RLAIF pipelines, low-perplexity tokens such as formatting markers or structural delimiters can systematically bias judge evaluations. When this happens, policies may converge to producing such patterns for reward rather than for answer quality ( 1 ; 17 ; 6 ; 20 ) . AdvJudge-Zero operationalizes this risk: it searches over short, likely continuations and finds control tokens that flip binary judgments while keeping the underlying content almost unchanged. We show that these tokens generalize across several open-weight model families (Qwen, Llama, Gemma) and specialized judges ( 32 ; 11 ; 13 ; 28 ; 36 ) . In this work we focus on correctness-style evaluations and false-positive flips, where an incorrect answer is judged as correct; we do not attempt to induce harmful content or bypass safety filters. Contributions. 1. We provide a geometric account of judge model evaluation as a shallow late-layer gate, with a scalar indicator F=z_{\text{no}}-z_{\text{yes}} for the binary judgment, and interpret control tokens as model-intrinsic directions that traverse this gate. 2. We introduce AdvJudge-Zero, an automated, zero-seed token discovery method that uses beam search guided by the model’s own next-token distribution to find control tokens that represent realistic reward-hacking risks in post-training pipelines. 3. We conduct cross-family experiments (Qwen, Llama, Gemma; specialized judge models and RLVR-style benchmarks) that quantify false positive rates, characterize discovered token types (such as special characters, markdown, and formatting), and show that small control-token–augmented datasets can be sufficient for effective adversarial training that reduces false positive rates while preserving evaluation capability. Scope and ethics. We focus on binary judge evaluations in reward models and LLM-as-a-Judge systems, where decisions are concentrated in the first few output tokens. Our goal is to characterize this post-training vulnerability and to evaluate possible mitigations, including adversarially robust reward modeling and guardrail design ( 10 ; 2 ) . We restrict our experiments to correctness-style tasks and false-positive flips, and we do not release attack prompts or control-token sequences that could be used maliciously. Section 3 formalizes the method, Section 4.1 details the experimental setup, Section 4 reports results, and Section 7 discusses limitations and future work. 2 Background and Related Work Post-training and preference optimization. RLHF ( 22 ) , DPO ( 25 ) , and RLAIF ( 16 ) adapt pre-trained models by optimizing scalar rewards from learned reward models ( 27 ) . Reward aggregation (Master-RM ( 35 ) , Weaver ( 26 ) ) and RL variants (GRPO ( 9 ) , RLVR ( 36 ) ) scale preference supervision using shallow heads that score early tokens. However, LLM-as-a-Judge reliability remains under scrutiny ( 5 ) . Reward hacking, judge robustness, and post-training vulnerabilities. Our prior work formalized a simple indicator for binary model evaluations, the last-layer logit gap F=z_{\text{no}}-z_{\text{yes}} , and showed that short suffixes