MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following

Jaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra, Ronald Clark · 2026

Read MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following by Jaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra, Ronald Clark (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03858.txt.

Introduction

Jaeyun Lee and four co-authors introduce MCJudgeBench, a benchmark for judging language model judges on multi-constraint instructions, constraint by constraint. Judges are usually scored on whole responses, yet real instructions bundle several demands that can each pass, partly pass, or fail. Each benchmark instance pairs an instruction and a candidate response. It adds an explicit constraint list and per-constraint gold labels.

Controlled perturbations of responses and variants of the prompt let the protocol separate two kinds of inconsistency. One is intrinsic, under stochastic decoding. The other is procedural, under prompt changes. Across proprietary and open-source judges, strong overall scores did not guarantee sound detection of the rarer partial and negative cases, and higher correctness did not always mean lower inconsistency. Reasoning improved correctness but not stability across the board.

Catalog record

CREATORJaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra, Ronald Clark
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03858

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following Jaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra, Ronald Clark Abstract Multi-constraint instruction following requires verifying whether a response satisfies multiple individual requirements, yet LLM judges are often assessed only through overall-response judgments. We introduce MCJudgeBench , a benchmark for constraint-level judge evaluation in multi-constraint instruction following. Each instance includes an instruction, a candidate response, an explicit constraint list, per-constraint gold labels in { yes , partial , no }, and controlled response-side perturbations. The evaluation protocol further includes evaluation prompt variants to test judge stability. We evaluate proprietary and open-source LLM judges using both correctness and inconsistency metrics, distinguishing intrinsic inconsistency under stochastic decoding from procedural inconsistency under prompt and response perturbations. Our results show that judge reliability has multiple dimensions: strong overall performance does not guarantee equally reliable detection across label categories, especially for rarer partial and no cases. Judges with higher correctness do not always have lower inconsistency. Evaluation with reasoning improves correctness but does not uniformly improve stability. These findings motivate evaluating LLM judges at the constraint level to study these failure modes. 1 Introduction Instruction following often involves satisfying several constraints at once, such as requirements on content, format, style, or length. Recent benchmarks show that performance declines as the number of constraints increases and they become harder to satisfy jointly, making prompt-level evaluation insufficient for understanding which requirements a model actually satisfies ( 4 ; 21 ) . In practice, however, assessing multi-constraint adherence at scale is expensive, which has encouraged the use of LLM-as-a-judge for open-ended responses. In parallel, there is increasing evidence that LLM judges are not always reliable. Their judgments can vary with prompt formulation and evaluation protocol, they may favor outputs that appear stylistically stronger despite weaker instruction adherence, and they can exhibit systematic biases under controlled changes ( 25 ; 10 ; 24 ; 7 ; 23 ) . Existing judge benchmarks have made progress on pairwise preferences, scalar judgments, and rubric-based evaluation ( 19 ; 12 ) , but they do not directly address whether a judge can verify individual constraints in multi-constraint instruction following tasks. This is particularly important in multi-constraint settings, where a response may satisfy some requirements while failing others. In such settings, evaluation must identify which constraints are satisfied or violated, rather than relying only on an overall score. A reliable evaluator must also remain stable under invariance perturbations to prompts and candidate response form. However, constraint-level judge stability in multi-constraint instruction following remains underexplored. In this work, we introduce MCJudgeBench , a meta-evaluation benchmark for LLM-as-a-judge on multi-constraint instruction following. Building on ComplexBench and InFoBench ( 21 ; 14 ) , the benchmark represents each instance as an instruction, a candidate response, and an explicit list of constraints, together with per-constraint adherence labels drawn from three classes: yes , partial , or no . We further construct controlled invariance sets through candidate response variants that preserve the underlying adherence labels. In the evaluation protocol, we additionally introduce evaluation prompt variants to test judge stability under alternative formulations of the same verification task. Beyond benchmark construction, we use this setting to study how current LLM judges behave on multi-constraint adherence verification. We evaluate judge correctness on the original instances and decompose judge instability into two sources: intrinsic inconsistency under stochastic decoding and procedural inconsistency under two invariance perturbation axes. Our results show that aggregate evaluation can hide substantial variation in constraint-level behavior, motivating systematic evaluation of judge reliability in this setting. Our contributions are as follows: • MCJudgeBench, a new meta-evaluation benchmark for LLM-as-a-judge on multi-constraint instruction following, with per-constraint labels and controlled invariance sets. • An evaluation framework for judge reliability, covering correctness on original instances and two sources of judge instability: intrinsic inconsistency under stochastic decoding and procedural inconsistency under invariance perturbations. • A baseline evaluation on the benchmark across proprietary and open-source LLM judges, showing how reliability varies across constraint types, constraint count, and perturbation conditions. 2 Related Work Benchmark Target IF Multi- const. Decomposed eval. Perturb. tests Constraint-level judge eval. IFEval ( 28 ) Models ✓ ✓ ✓ – – FollowBench ( 4 ) Models ✓ ✓ ✓ – – ComplexBench ( 21 ) Models ✓ ✓ ✓ – – InFoBench ( 14 ) Models ✓ ✓ ✓ – – WildIFEval ( 8 ) Models ✓ ✓ ✓ – – CFBench ( 26 ) Models ✓ ✓ ✓ – – LLMBar ( 25 ) Evaluators ✓ – – ✓ – ReIFE ( 10 ) Evaluators ✓ – – – – GroUSE ( 12 ) Evaluators – – ✓ ✓ – MCJudgeBench Evaluators ✓ ✓ ✓ ✓ ✓ Table 1: Comparison of MCJudgeBench with prior multi-constraint instruction following benchmarks and meta-evaluation benchmarks for automated evaluators and LLM judges. “Decomposed eval.” indicates evaluation beyond a single overall judgment, such as checking individual constraints, decomposed criteria, or rubric dimensions. “Constraint-level judge eval.” indicates meta-evaluation of whether a judge can verify individual constraints in multi-constraint instruction following. Instruction following and multi-constraint evaluation. Recent work has moved instruction following evaluation beyond coarse prompt-level judgments toward a more fine-grained assessment of whether individual requirements are satisfied. IFEval ( 28 ) focuses on objectively verifiable constraints, with each instruction containing one or more such constraints. FollowBench ( 4 ) extends this direction by constructing a controlled progression in difficulty over the same base instruction, incrementally adding constraints so that higher levels require satisfying a larger and more diverse set of requirements. Subsequent benchmarks broaden this line in different ways. ComplexBench ( 21 ) emphasizes the composition of multiple constraints and introduces a hierarchical taxonomy over constraint and composition types. InFoBench ( 14 ) decomposes complex instructions into simpler criteria for more interpretable requirement-level evaluation. CFBench ( 26 ) and WildIFEval ( 8 ) extend evaluation to broader and more realistic constraint settings through systematic taxonomies, real-world scenarios, and real user instructions. LLM-as-a-judge and meta-evaluation of evaluators. LLM-as-a-judge has become a common approach for scalable evaluation, but recent work shows that judge behavior is often sensitive to factors beyond the underlying quality of the response. LLMBar ( 25 ) shows that judges can be misled by stylistic differences even when instruction adherence is weaker, while ReIFE ( 10 ) demonstrates that evaluation accuracy depends strongly on both the base LLM and the evaluation protocol. JudgeBench ( 19 ) and GroUSE ( 12 ) further argue that judge quality cannot be assessed only through aggregate agreement or correlation, since evaluators may still fail on difficult objective comparisons or specific failure modes. Related work on bias analysis also finds persistent systematic preferences under controlled modifications ( 24 ) . Recent work further shows that LLM evaluators can be inconsistent and sensitive to prompt differences that are not meaningful to human evaluators ( 18 ) . Evaluator consistency has also been studied through self-consistency and inter-scale consistency, showing that strong evaluator models are not necessarily consistent and that consistency should be considered when assessing LLM evaluators ( 6 ) . Constraint-level judge evaluation. These two lines of work address complementary parts of the problem. Multi-constraint instruction following benchmarks study whether models satisfy complex sets of requirements, while meta-evaluation studies examine

Read the complete text