C 2 -Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning

Avni Mittal, Rauno Arike · 2026

Read C 2 -Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning by Avni Mittal, Rauno Arike (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05167.txt.

Introduction

Avni Mittal and Rauno Arike present C2-Faith, a benchmark testing whether language models judging chain-of-thought reasoning assess process faithfulness or merely answer plausibility. It splits faithfulness into causality, whether each step is logically supported by what came before, and reach, whether key middle inferences are present. Controlled perturbations plant single-step causal errors and delete reasoning at varying rates, giving reference labels for three judge tasks. Frontier judges detect errors more reliably than they locate them and in full overrate completeness, exposing limits in process-level test.

Catalog record

CREATORAvni Mittal, Rauno Arike
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42197
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

C 2 -Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning Avni Mittal, Rauno Arike Abstract Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning, but it remains unclear whether they can reliably assess process faithfulness rather than just answer plausibility. We introduce C 2 -Faith, a benchmark built from PRM800K that targets two complementary dimensions of faithfulness: causality (does each step logically follow from prior context?) and coverage (are essential intermediate inferences present?). Using controlled perturbations, we create examples with known causal error positions by replacing a single step with an acausal variant, and with controlled coverage deletions at varying deletion rates (scored against reference labels). We evaluate three frontier judges under three tasks: binary causal detection, causal step localization, and coverage scoring. The results show that model rankings depend strongly on task framing, with no single judge dominating all settings; all judges exhibit a substantial gap between detecting an error and localizing it; and coverage judgments are systematically inflated for incomplete reasoning. These findings clarify when LLM judges are dependable and where they fail, and provide practical guidance for selecting judges in process-level evaluation. 1 Introduction Large language models (LLMs) are increasingly used as judges to evaluate the reasoning quality of other models, especially for tasks that elicit chain-of-thought (CoT) explanations ( 22 ; 5 ) . However, it remains unclear whether LLM judges can reliably assess faithfulness : whether a reasoning trace genuinely supports its answer, rather than merely sounding plausible. Faithfulness is not equivalent to final-answer correctness. A CoT can reach the right answer while containing logically invalid intermediate steps, skipping key inferences, or post-hoc rationalizing a conclusion that was reached by other means ( 17 ; 9 ) . When automated judges are used to rank or filter reasoning traces, as in process reward model training ( 10 ) or RLHF pipelines, undetected unfaithfulness propagates silently into downstream systems. Figure 1: Overview of C 2 -Faith benchmark construction and evaluation tasks. To study this gap in a diagnostic way, we decompose faithfulness into two underexplored dimensions: • Causality : does each reasoning step logically follow from the steps that precede it? A chain that contains a step inconsistent with its context is causally unfaithful, even if the final answer happens to be correct. • Coverage : are the critical intermediate inferences actually present? A chain that jumps from problem statement to conclusion while omitting the reasoning that bridges them is incomplete, regardless of surface coherence. Together, these two axes capture complementary failure modes: a causally valid chain can still be unfaithful if it omits essential reasoning (low coverage), and a complete chain can be unfaithful if individual transitions are logically broken (low causality). Existing LLM-as-judge work focuses primarily on answer quality, style, and harmlessness ( 22 ; 8 ; 19 ) . Process-level faithfulness is far less studied: prior analyses rely on behavioral probes or perturbation studies that lack ground truth ( 17 ; 9 ) , while formal step-validity methods require structured domain representations inaccessible in free-form mathematical reasoning ( 14 ) . To our knowledge, no benchmark combines controlled perturbations with known causal error positions and controlled coverage deletions to directly measure judge reliability on both causal validity and reasoning completeness. We introduce C 2 -Faith , a diagnostic benchmark built from PRM800K ( 10 ) that addresses this gap. Using controlled perturbations of verified reasoning chains, acausal replacements that violate logical entailment at a known position and step deletions that remove intermediate inferences at known coverage fractions, we evaluate whether frontier LLM judges can detect and localize these two classes of unfaithfulness under a unified scoring protocol. 1. C 2 -Faith benchmark : controlled perturbation datasets derived from PRM800K with exact causal error labels and controlled coverage deletions (reference-scored). 2. Three-experiment protocol covering binary causal detection (Exp 1), causal step localization (Exp 2), and 0 to 4 coverage scoring. 3. Comprehensive multi-model evaluation of GPT-4.1, DeepSeek-V3.1, and o4-mini with statistical tests. 2 Related Work LLM-as-judge. 22 established the LLM-as-judge paradigm with MT-Bench and Chatbot Arena, demonstrating that GPT-4 correlates well with human preferences on open-ended conversation. 5 survey recent advances covering scaling, calibration, and positional biases in LLM judges. 8 and 19 develop judge models with rubric-based and pairwise scoring protocols. All of these focus on answer quality or style; none address step-level process faithfulness with ground-truth perturbations. CoT faithfulness. 20 showed that chain-of-thought prompting substantially improves reasoning accuracy. 17 demonstrated that CoT explanations can be post-hoc rationalizations: biasing inputs can flip outputs without changing the stated reasoning. 11 proposed faithful CoT via symbolic decomposition. 9 measure faithfulness by truncating or corrupting CoTs and observing output changes. These works reveal that unfaithfulness is real , but none provide a benchmark for evaluating whether an LLM judge can detect it. 7 established the distinction between faithfulness and plausibility for neural explanations. Process reward models. 10 introduced PRM800K with human step-level labels for mathematical reasoning, enabling process-supervised training. 18 compared process- and outcome-based feedback. We build on PRM800K’s step-level labels to construct ground-truth perturbations. Closest benchmarks. 21 is closest in step localization, but it relies on naturally occurring model errors (not controlled perturbations), targets PRMs and critic models rather than LLM judges as a primary focus, and includes no coverage/completeness dimension. 16 provides fine-grained step-level labels for PRM evaluation, but again centers on PRMs (not judge reliability), lacks a coverage task, and does not analyze the detection–localization gap. 15 is the most similar in evaluating LLM judges for CoT faithfulness, but it adopts a different notion of faithfulness (alignment with internal computation rather than structural step validity), relies on real model outputs with human labels instead of controlled injections, and does not include step localization or coverage scoring. Finally, 3 defines the 0–4 coverage rubric we adopt, but focuses on a conceptual framework and pilot measurements rather than a controlled benchmark. In contrast, C 2 -Faith provides controlled causal injections and controlled coverage deletions with ground-truth labels, enabling direct measurement of judge reliability on both causality and coverage. 3 Methodology We describe the C 2 -Faith benchmark construction, which creates controlled perturbations from verified reasoning chains to probe two dimensions of judge faithfulness: causality and coverage. Algorithm 1 summarizes the full construction procedure for both perturbation types. Figure 1 provides an overview of the pipeline and tasks. 3.1 Causality Perturbations To test whether judges can detect logical inconsistencies, we replace a single step in the middle region of a perfect chain with an LLM-generated acausal variant : a plausible-looking but logically inconsistent rewrite conditioned on the preceding step as context. Each example records the original chain, the perturbed chain, the target step index j , the corrupted step text, and the generation context. The middle-region targeting (relative positions 30% to 90% of chain length) ensures that judges cannot solve the task by simply checking whether the first or last step is coherent. LLM-generated negations produce more realistic test items than random perturbations because they preserve surface-level mathematical style. 3.2 Coverage Perturbations To test whether judges can assess reasoning completeness, we uniformly remove a fraction d of middle-region steps from perfect chains, varying deletion rate d\in\{0.1,0.3,0.5,0.7\} (retention =1-d

Read the complete text