Process Supervision via Verbal Critique Improves Reasoning in Large Language Models

Hao-Yuan (Mark) Chen · 2026

Read Process Supervision via Verbal Critique Improves Reasoning in Large Language Models by Hao-Yuan (Mark) Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21611.txt.

Introduction

Hao-Yuan Chen introduces Verbal Process Supervision, a training-free framework using critique from a stronger supervisor to guide generate-critique-refine loops. On GPQA Diamond, GPT-5.4 High with GPT-5.4 Low reaches 94.9 percent at round 4, beating the state of the art at about 94 percent. On AIME 2025, VPS rescues weak actors, lifting scores from 12-27 percent to 63-90 percent. At matched compute, VPS beats Reflexion by about 9-12 points and Self-Consistency-5 by about 5-8 points. Performance scales with the supervisor-actor gap and degrades when errors resist verbal expression, motivating hybrids.

Catalog record

CREATORHao-Yuan (Mark) Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33452
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21611

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Process Supervision via Verbal Critique Improves Reasoning in Large Language Models Hao-Yuan (Mark) Chen Abstract Inference-time compute for large language model (LLM) reasoning has been scaled along three established axes: chain depth , sample breadth , and learned step-scorers (PRMs). We identify a fourth, largely overlooked axis — the granularity of external verbal supervision — and introduce Verbal Process Supervision (VPS) , a training-free inference-time framework instantiating its step-level end. Rather than scalar reward, VPS employs structured natural-language critique from a stronger supervisor to guide an actor through an iterative generate–critique–refine loop until convergence or a maximum round budget R R is reached. Across GPQA Diamond, AIME 2025, and LiveCodeBench V6 — spanning closed-source (GPT-5.4 family) and open-weight (GLM-5.1, Nemotron-3-Super, Gemma 4 31B, GPT-OSS 120B/20B) pairs — VPS delivers three standout results. First , on GPQA Diamond, the GPT-5.4 (High) ∣ \mid GPT-5.4 (Low) configuration reaches 94.9% at R = 4 R=4 , surpassing the reported state of the art of 94.1% , achieved purely through same-family verbal self-refinement with zero gradient updates. Second , on AIME 2025, VPS unlocks dramatic capability amplification in a weak-actor rescue regime: actors scoring 11.7–26.7% standalone are lifted to 63.3–90.0% , yielding absolute gains of up to + 63.3 +63.3 points. Third , at matched inference compute, VPS outperforms outcome-level verbal critique (Reflexion) by + 8.5 +8.5 to + 12.1 +12.1 points across all three benchmarks, and outperforms Self-Consistency @ 5 by + 5.0 +5.0 pp on GPQA Diamond and + 8.3 +8.3 pp on LiveCodeBench V6 , isolating critique granularity as the operative variable. Gains are mediated by the supervisor–actor capability gap (Pearson r = + 0.90 r=+0.90 ); effectiveness is bounded by a principled domain boundary where errors are no longer linguistically representable (code synthesis), motivating hybrid verbal–executable extensions. Together, these findings establish critique granularity as a distinct axis of inference-time scaling . 1 Introduction Inference-time compute for LLM reasoning has been scaled along three established axes. Chain depth allocates more tokens per trajectory via chain-of-thought prompting 15 and test-time scaling 9 . Sample breadth allocates parallel trajectories aggregated by majority voting 14 or search 16 ; 2 . Learned step-scorers train process reward models (PRMs) on labeled step correctness 6 ; 13 ; 11 . Each axis trades a different resource — latency, throughput, or training cost — for capability. We argue a fourth axis exists and has been largely overlooked: the granularity of external verbal supervision . Along this axis, a stronger model provides natural-language feedback to a weaker actor, with granularity ranging from outcome-level critique on a completed trajectory 12 to step-level critique at each reasoning step. The granularity axis is orthogonal to depth, breadth, and learned scoring: it requires no extra chain length, no extra sampling, and no training. It trades supervisor capability for actor capability, mediated by the gap between them. Table 1: Inference-time scaling axes for LLM reasoning. VPS instantiates the step-level end of a fourth axis — the granularity of external verbal supervision. Reflexion 12 occupies the outcome-level end of the same axis. Axis Scaling Mechanism Training Cost Representative Methods Chain depth Longer reasoning per trajectory None CoT 15 , s1 9 Sample breadth Parallel trajectories aggregated None Self-Consistency 14 , ToT 16 Learned scorer Trained step-level value model High (step labels) PRMs 6 ; 13 Verbal granularity (ours) Step-indexed natural-language critique None Reflexion (outcome), VPS (step) This paper formalizes the granularity axis, instantiates its step-level end as Verbal Process Supervision (VPS) , and shows empirically that at matched inference compute, step-level critique outperforms both outcome-level critique and sample aggregation on reasoning benchmarks. VPS operates entirely at inference time — no gradient updates, no human labels, no additional training data. Our central thesis is that verbal feedback from a stronger supervisor is a sufficient and scalable reward signal for bootstrapping a self-improving reasoning loop, and that critique granularity — not verbal feedback alone — is the dominant variable . We make the following contributions: (i) we identify critique granularity as a distinct axis of inference-time scaling (Table 1 ); (ii) we formalize VPS as the step-level instantiation, with a language-space actor–critic update operator and fixed-point convergence characterization; (iii) we demonstrate empirically that at matched inference compute, VPS outperforms Reflexion by +8.5 to +12.1 points across GPQA Diamond, AIME 2025, and LiveCodeBench V6, isolating granularity as the operative variable; and (iv) we characterize a weak-actor rescue regime and report the supervisor–headroom correlation ( r=+0.90 ), a predictive criterion for deployment. 2 Related Work Verbal reinforcement learning and self-correction. Reflexion 12 introduced verbal reinforcement learning, showing natural-language feedback can serve as a supervisory signal; related efforts such as ReAct and CRITIC likewise show language feedback guides stronger inference-time behavior, especially with tool use 17 ; 1 . However, Reflexion computes verbal reward over a full trajectory, yielding a single end-of-episode signal that conflates credit across reasoning steps: a failure due to a single intermediate error receives the same critique pressure as one due to a flawed strategy. VPS decomposes this signal, issuing step-indexed critiques that preserve correct sub-trajectories and localize refinement to specific failure points. Section 4.3 shows this granularity yields +8.5 to +12.1 points over Reflexion at matched compute. Prior work also highlights the limits of purely intrinsic self-correction without external feedback 4 ; 7 ; VPS sidesteps these by introducing an external, typically stronger, supervisor. Process supervision and inference-time scaling. Let’s Verify Step by Step 6 showed that process-level signals outperform outcome-only supervision on mathematical reasoning; subsequent PRM work has strengthened this line via automatic step-level labels and harder evaluators 13 ; 11 ; 19 . VPS extends this insight by replacing trained scorers with verbal critique from a stronger LLM, removing the annotation and training bottleneck. Separately, inference-time methods such as chain-of-thought 15 , self-consistency 14 , Tree-of-Thoughts 16 , Self-Refine 8 , STaR 18 , and self-improvement 3 all allocate more compute at test time. VPS is complementary to these along the granularity axis (Table 1 ): it adds a trained-scorer-free form of process supervision that requires only a stronger supervisor LLM. 3 Verbal Process Supervision (VPS) 3.1 Framework VPS operates through two components: a base actor \pi_{\theta} that generates reasoning trajectories, and a supervisor \mathcal{C} (typically stronger) that evaluates and critiques them. Given input x\in\mathcal{X} , the actor generates a trajectory of T steps \tau=(s_{1},\dots,s_{T}) with s_{t}\sim\pi_{\theta}(\cdot\mid s_{<t},x) . The supervisor then produces a structured natural-language critique c=\mathcal{C}(\tau,x) with c\in\mathcal{V}^{*} , where \mathcal{V}^{*} is the Kleene closure over vocabulary \mathcal{V} . Unlike standard RL where reward is scalar r\in\mathbb{R} , VPS defines a verbal reward function R_{v}:\mathcal{T}\times\mathcal{V}^{*}\to\mathcal{V}^{*} encoding semantically rich, step-level supervision that scalar signals cannot capture. 3.2 Inference-Time Policy Improvement Instead of gradient descent, VPS performs policy improvement through conditional regeneration : the actor refines its trajectory by conditioning on the supervisor’s critique, \tau_{r+1}\sim\pi_{\theta}(\cdot\mid x,\tau_{r},c_{r}) . We define the VPS update operator \mathcal{F} as \tau_{r+1}=\mathcal{F}(\tau_{r},\mathcal{C}(\tau_{r},x)) , encapsulating the full generate–critique–refine loop. Iterating yields \tau_{0}\to\tau_{1}\to\cdots\to\tau_{R} , and under ideal conditions converges to a fixed point \tau^{*}=\mathcal{F}(\tau^{*},\mathcal{C}(\tau^{*},x)) . In practice iteration terminates after at most R rounds or on a stopping criterion (Algorithm 1 ). Algorithm 1 Verbal Process Supervision (VPS) 1: Input x , actor \pi_{\theta} , supervisor \mathcal{C} , maximum rounds

Read the complete text