Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

Yixin Liu, Yue Yu, DiJia Su, Sid Wang, Xuewei Wang, Song Jiang, Bo Liu, Arman Cohan, Yuandong Tian, Zhengxing Chen · 2026

Read Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training by Yixin Liu, Yue Yu, DiJia Su, Sid Wang, Xuewei Wang, Song Jiang, Bo Liu, Arman Cohan, Yuandong Tian, Zhengxing Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12246.txt.

Introduction

Liu and co-authors examine reasoning LLMs as judges in post-training where answers cannot be checked directly. Static benchmarks favor reasoning judges, but their effect inside real policy training stayed untested. A gold-standard judge labels preferences in a controlled setup used to train smaller judges. Non-reasoning judges invite reward hacking with ease in these runs. Reasoning judges yield strong scores, yet their policies win by crafting outputs that fool other judges. The study flags both promise and clear room to improve judge-based training.

Catalog record

CREATORYixin Liu, Yue Yu, DiJia Su, Sid Wang, Xuewei Wang, Song Jiang, Bo Liu, Arman Cohan, Yuandong Tian, Zhengxing Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88974
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12246

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training Yixin Liu, Yue Yu, DiJia Su, Sid Wang, Xuewei Wang, Song Jiang, Bo Liu, Arman Cohan, Yuandong Tian, Zhengxing Chen Abstract Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a “gold-standard” judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training. † † affiliation: Meta Superintelligence Labs † † affiliation: Yale Univeristy † † contribution: Work done at Meta † † date: August 10, 2026 † † correspondence: Yixin Liu at [email protected] , Zhengxing Chen at [email protected] Arena-Hard-V2: Creative Writing Subset Model Score (%) o3 92.4 Reasoning Judge 89.6 + Llama-3.1-8B DeepSeek-R1 89.2 Gemini-2.5 85.2 GPT-4.1 78.6 Claude-3.7-Sonnet 72.5 Qwen3-32B 65.2 Gemini-2.0-Flash 50.0 Figure 1 : Illustration of our synthetic experiment setting (left). In the middle, we show that a Llama-3.1-8B policy trained with a fine-tuned Qwen3-4B reasoning judge can achieve strong performance under the gold-standard judge gpt-oss-120b’s evaluation, while the policy trained with a fine-tuned Qwen3-14B non-reasoning judge cannot and exhibits severe reward hacking. The table on the right shows results on the creative writing subset of Arena-Hard-V2. The Llama-3.1-8B policy trained with the Qwen3-4B reasoning judge is able to achieve superior performance by learning to generate highly effective adversarial outputs. 1 Introduction Recently, Reinforcement Learning (RL) from Verifiable Rewards (RLVR) has shown great effectiveness in improving large language models (LLMs) in reasoning tasks ( Guo et al. 2025 ; Lambert et al. 2025 ) , resulting in strong reasoning models that benefit from inference-time compute scaling ( OpenAI 2024a ) . However, this training paradigm cannot be trivially extended to non-verifiable domains, where the output quality cannot be directly checked. As a result, RL from human feedback (RLHF) ( Ouyang et al. 2022 ) or AI Feedback (RLAIF) ( Bai et al. 2022 ) , remains the predominant training paradigm, where a reward model, or an LLM as a judge, 1 1 1 For simplicity, we use “LLM-judge” to denote both a reward model and an LLM-as-a-judge in this work. is used to provide supervision. Recent efforts have been made to enhance LLM-judges by inference-time scaling ( Liu et al. 2025 ; Chen et al. 2025a ; Chen et al. 2025b ; Whitehouse et al. 2025 ) , where the task of the LLM-judges is formulated as a reasoning task with verifiable outputs. This provides an opportunity to extend the success of RLVR and inference-time compute scaling into non-verifiable domains, by scaling up the supervision compute in RL: instead of using a canonical LLM-judge, a reasoning model can be used as the judge in policy training. Indeed, Guan et al. 2024 has successfully used reasoning models as judges for safety alignment, while Ma et al. 2025 showed the advantage of a reasoning LLM-judge over a rule-based verifier in general reasoning tasks where the output correctness is not trivially verifiable. However, while previous work ( Liu et al. 2025 ; Chen et al. 2025a ; Chen et al. 2025b ; Whitehouse et al. 2025 ) has demonstrated the benefit of reasoning LLM-judges on static evaluation benchmarks, e.g., RewardBench ( Lambert et al. 2024 ) , a systematic study of their effectiveness in actual policy training is still lacking. Therefore, we aim to conduct a rigorous examination of reasoning LLM-judges regarding their effectiveness for actual LLM post-training in non-verifiable domains. To this end, we focus on an important capability of LLMs, namely their alignment to human preferences, to study the LLM-judges’ effectiveness at a broad and general level. Following Gao et al. 2023 , we adopt a synthetic setting (Figure 1 ), where a relatively more capable LLM, gpt-oss-120b ( Agarwal et al. 2025 ) , is treated as a “gold-standard” judge. The gold-standard judge is then used to provide preference annotations both for fine-tuning judges (§ 2.1 ), which are a series of Qwen3 ( Yang et al. 2025 ) models of 1.7B-14B parameters, and for evaluating policies trained in RL under the supervision of the fine-tuned judges (§ 2.2 ). Compared to directly using post-trained LLMs as judges, this setting ensures fairer and more controllable comparisons of judges. With the objective of achieving strong performance under the gold-standard judge’s evaluations, our controlled experiments reveal substantial differences between the canonical and reasoning LLM-judges in post-training. Specifically, the LLM policies trained with canonical LLM-judges exhibit a similar reward hacking pattern as observed in previous work ( Gao et al. 2023 ) : as training progresses, they receive increasingly higher rewards from the judge used in training, but start to receive lower rewards from the gold-standard judge (§ 3.2 ). In clear contrast, policies trained with reasoning judges can achieve very high rewards under the gold-standard judge (§ 3.3 ) under both training and test datasets, which indicates better effectiveness of the reasoning judges. Interestingly, when examined qualitatively, the policies trained with the reasoning judges achieve high performance under the gold-standard judge by generating highly effective adversarial outputs with a systematic strategy: (1) first refusing to respond by claiming that the user instruction violates the usage policy; (2) then fabricating a usage policy that is specifically related to the user instruction; (3) providing a self-assessment claiming that the above refusal is appropriate. This strategy is highly effective for gpt-oss-120b, and is generalizable to Arena-Hard-V2 ( Li et al. 2025 ) where GPT-4.1 ( OpenAI 2025a ) is used as the judge by default. In particular, the adversarial policy trained from Llama-3.1-8B-Instruct ( Grattafiori et al. 2024 ) achieves around 90% win rate over the baseline Gemini-2.0-flash ( Comanici et al. 2025 ) in creative writing, ranking higher than frontier LLMs such as Gemini-2.5 and o4-mini ( OpenAI 2025d ) . We then conduct a systematic analysis regarding the strong effectiveness of reasoning LLM-judges under the evaluation of the gold-standard judge by examining a few key design options. First, we compare reasoning judges trained with only GRPO ( Shao et al. 2024 ) against the default distillation-then-GRPO judge (§ 4.1 ), finding that access to the gold-standard judge’s reasoning process is essential for the training of the reasoning judges. Next, we examine whether providing evaluation rubrics generated by the gold-standard judge to non-reasoning judges can achieve similar effects as reasoning judges (§ 4.2 ), which reveals that non-reasoning judges with rubrics still fail to produce strong policies

Read the complete text