J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization

Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty · 2025

Read J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization by Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13346.txt.

Introduction

Austin Xu and co-authors train AI judges in 2025 with a new reinforcement learning method. As language models grow fast, people now use models to grade model output, since human grading is slow. These judges do well on plain chat quality. They struggle on hard reasoning tasks, where answers carry dense content that takes real skill to grade.

The team names three gains. Their new algorithm lets judge training start from like states, which makes the learning signal cleaner. The method builds on a group relative approach that compares several tries at once. Better judges matter because so much model progress now runs through auto grading. A judge that fails on hard tasks will certify bad work as good. The paper tests the trained judges on reasoning heavy sets.

Catalog record

CREATORAustin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13346

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty Abstract To keep pace with the increasing pace of large language models (LLM) development, model output evaluation has transitioned away from time-consuming human evaluation to automatic evaluation, where LLMs themselves are tasked with assessing and critiquing other model outputs. LLM-as-judge models are a class of generative evaluators that excel in evaluating relatively simple domains, like chat quality, but struggle in reasoning intensive domains where model responses contain more substantive and challenging content. To remedy existing judge shortcomings, we explore training judges with reinforcement learning (RL). We make three key contributions: (1) We propose the Equivalent Initial State Group Relative Policy Optimization (EIS-GRPO) algorithm, which allows us to train our judge to be robust to positional biases that arise in more complex evaluation settings. (2) We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work. (3) We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%, matching or exceeding the performance of larger GRPO-trained judges on both JudgeBench and ReasoningJudgeBench. : GitHub : ReasoningJudgeBench 1 Introduction Large language models (LLMs) and LLM-powered agentic systems have been tasked with solving increasingly difficult problems that require strong reasoning abilities ( Ke et al., 2025 ) . As LLMs are deployed in reasoning intensive-tasks, evaluation methods have come under mounting pressure to efficiently and accurately evaluate and critique LLM outputs, especially when the outputs are not easily verifiable with rules. While human evaluation remains the gold-standard in such cases, automatic evaluation approaches, whether it be scalar reward models (RMs) or LLM-as-judges, have been deployed as a more scalable alternative. However, these approaches have shortcomings ( Cemri et al., 2025 ) : Current verifiers tend to lock onto stylistic details over substance ( Feuer et al., 2024 ; Liu et al., 2024 ; Zeng et al., 2023 ) . RMs are incapable of explaining their decisions, while judge models are prone to various biases such as positional ( Li et al., 2023 ; Wang et al., 2023a ) and length bias ( Zeng et al., 2023 ; Park et al., 2024 ) . In all, these drawbacks lead to evaluators that struggle to evaluate other models in reasoning settings ( Tan et al., 2024 ; Zhou et al., 2025 ) . Figure 1 : Overview of our three contributions: (1) We propose EIS-GRPO, a RL training algorithm for training positionally robust judge models. (2) We create ReasoningJudgeBench, a benchmark of 1,483 pairwise samples across diverse and underexplored reasoning settings. (3) We train J4R-7B, a lightweight yet high-performing judge specifically for reasoning evaluation. This work explores using reinforcement learning from verifiable rewards (RLVR) ( Lambert et al., 2024 ; Shao et al., 2024 ; Guo et al., 2025 ) to improve the ability of evaluators, specifically LLM-as-judge models, in evaluating model responses for reasoning problems. Automatic evaluation is a natural setting for RLVR: Reinforcement learning (RL) allows the model to learn how to generate chain-of-thought (CoT) critiques without supervision from distilled CoT critiques, which are commonly used in judge model training (e.g., Kim et al. (2023) ; Kim et al. (2024a) ; Wang et al. (2024a) ; Ye et al. (2024) ). Furthermore, automatic evaluation outputs have a limited set of final outcomes (e.g., “A” or “B” for pairwise comparisons, 1-5 for single ratings), making them naturally “verifiable”. Before simply applying standard RL algorithms ( Shao et al., 2024 ; Guo et al., 2025 ) with judge data, we first take a step back and carefully look into where judges fail in reasoning settings. Our analysis in Sec. 4 shows that as task difficulty grows, existing judge models, including those trained with RL, suffer from increasing positional inconsistency , where swapping the order of the candidates does not lead to an equivalent swap in judgment. This inconsistency indicates that the judge is randomly guessing, rather than substantively assessing. To remedy this, we propose Equivalent Initial State Group Relative Policy Optimization (EIS-GRPO), which explicitly trains the model to treat transformations of the same initial state (i.e., input context) as equivalent, instilling consistency and robustness in judges with zero additional training overhead. Using EIS-GRPO, we train Judge for Reasoning (J4R), a 7B LLM judge that specializes in evaluating reasoning. J4R outperforms GPT-4o by 6.7% and approaches the performance of 32B RL-trained judges on a suite of reasoning-focused judge benchmarks. In our evaluation, we find existing benchmarks of reasoning judges lack breadth. Notably, JudgeBench ( Tan et al., 2024 ) consists only of 350 pairwise samples sourced from MMLU-Pro ( Wang et al., 2024b ) , LiveBench ( White et al., 2024 ) , and LiveCodeBench ( Jain et al., 2024 ) , missing coverage of different, uniquely challenging reasoning tasks, like inductive or commonsense reasoning. This leads us to create ReasoningJudgeBench to augment existing benchmarks with new, challenging types of reasoning. As shown in Fig. 1 , our contributions are threefold: • Methodology ( Sec. 3 ): We propose Equivalent Initial State GRPO, a simple but effective data augmentation method for GRPO that allows us to combat judge positional bias while improving performance over data-driven approaches. • Benchmark ( Sec. 4.1 ): We create ReasoningJudgeBench, a benchmark of 1,483 challenging pairwise samples. Compared to existing reasoning-focused judge benchmarks, ReasoningJudgeBench contains 4x more samples covering more diverse reasoning settings. • Model and Analysis ( Sec. 5 ): Using EIS-GRPO, we train J4R-CJ-7B, a 7B model that punches above its weight class: It matches the performance of larger judge models in reasoning evaluation and outperforms the next best \leq 14 B judge by 13% and 9% on JudgeBench and ReasoningJudgeBench. We also perform a suite of analyses focused on (1) the utility of RL versus other common judge training recipes, (2) choice of initial model, and (3) the efficacy of EIS-GRPO versus other bias-mitigation training strategies. Figure 2 : Overview of Equivalent Initial State GRPO: Each input q is first transformed via \mathcal{T}_{\ell} to a substantively equivalent version of q and used to sample a subgroup of responses. Responses are rewarded with outcome verification and advantages are computed jointly across subgroups and within subgroups before being used to update the policy model with respect to the transformed input. Joint subgroup advantage computation teaches the model to treat all transformed inputs as equivalent , training the model to be robust to non-substantive transformations of the input. 2 Background and related work Learning to reason with reinforcement learning. LLMs have been tasked with solving increasingly complex problems that require sound reasoning abilities. This lead to the development of long CoT models like OpenAI’s o1 ( Jaech et al., 2024 ) or DeepSeek-R1 ( Guo et al., 2025 ) , which expend large amounts of inference-time compute generating “thoughts” before answering. DeepSeek-R1 revealed that reinforcement learning using rule-based verification as a reward signal, broadly known as RL from Verifiable Rewards ( Lambert et al., 2024 ) , is capable of unlocking powerful reasoning abilities from pretrained LLMs. As a result, RL and long CoT distillation have emerged as popular

Read the complete text