What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA

Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang · 2026

Read What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA by Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25988.txt.

Introduction

Ji and five colleagues test what makes a fact-checker useful as a reward signal for medical retrieval systems. A checker that labels claims as supported or not should guide training, but its behaviour during training matters more than its held-out accuracy. Log-probability scoring from a language model marks over 97 percent of claims neutral, collapsing the training gradient to zero. A calibrated medical classifier scores the same pairs with real spread. A strong proprietary checker, meanwhile, triggers a three-step reward-hacking cascade toward ultra-short answers.

Catalog record

CREATORYuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64610
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25988

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA Yuelyu Ji, Min Gu Kwak, Hang Zhang, Xizhi Wu, Chenyu Li, Yanshan Wang Abstract Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive. We find that the checker’s output distribution during training, not its held-out accuracy, decides whether it provides trainable gradient. We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent (Qwen2.5-7B, replicated on Qwen3-4B and Llama-3.1-8B) across four held-out medical QA benchmarks. Three diagnostic findings emerge. (i) Signal collapse is log-prob-specific: LLM log-probability scoring labels over 97% of claims neutral—collapsing the RL gradient to zero—while a calibrated MedNLI classifier scores the same pairs non-degenerately. (ii) Moderate signal beats strong signal on answer quality: a strong proprietary checker triggers a three-step reward-hacking cascade—ultra-short answers, search avoidance, language collapse—so a moderate-signal local classifier trains a higher-quality model ( +12% BERTScore over zero-shot, no GPT dependency ). (iii) Signal strength is policy-dependent: the same checker registers as moderate on one policy but strong on another without triggering the cascade end-state. We frame these as boundary conditions for verifier-as-reward systems. † † Code, prompts, and diagnostics available at: https://anonymous.4open.science/r/medchecker-verl-7FC1/ Question (shared across all three regimes) “Could eating psoriasis flakes cause prion disease?” ✗ Collapsed checker Likelihood-NLI (Meditron-3-8B) A (394 chars): “Your concerns are valid, but it is highly unlikely that you have prion disease …” Checker: 97% neutral Support 0% \to zero RL gradient; reasonable but unverified. ✗ Strong checker, hacked GPT-NLI, unguarded A (85 chars): “It may not lead to prion disease, but avoid eating psoriasis flakes.” Checker: entail (1/1) Support 86.1% ( higher is not better here ) \to reward hacking, stage 1 of 3: ultra-short answers (§ 6 ); stages 2–3 follow as search avoidance and language collapse. ✓ Calibrated checker (ours) MedNLI-Cls A (full length): “Based on available evidence, no strong support for prion risk from psoriasis flakes …” Checker: entail / neutral mix Support 54.0% \to moderate signal, no GPT dependency, highest answer quality . Figure 1: The same medical question under three checker regimes. Top : log-prob scoring collapses to Neutral , providing no training signal (§ 5 ). Middle : a strong GPT checker provides high support but triggers a three-step reward-hacking cascade (§ 6 )—note that higher support here is not better. Bottom : a local MedNLI classifier produces moderate, non-degenerate verdicts and yields the highest answer quality without GPT dependency. 1 Introduction Consider a patient asking “Could eating psoriasis flakes cause prion disease?” A medical RAG system should retrieve evidence, decompose its draft answer into atomic claims, and verify each claim against the evidence before answering. The intuitive design is to plug a claim-level NLI checker into the RL reward path, so that the policy learns to generate evidence-grounded claims through retrieval-augmented RL ( 9 ; 37 ; 43 ) . This design is being explored in concurrent medical RAG-RL systems ( 31 ; 38 ; 30 ; 5 ; 35 ; 36 ; 18 ; 17 ) , and builds on a broader literature where claim-level verification is used for evaluation and post-hoc correction ( 20 ; 33 ; 34 ) , with the headline metric being NLI accuracy on held-out pairs. However, no prior work treats the checker’s output distribution under the policy’s own rollouts —as opposed to its accuracy on a held-out NLI benchmark—as a measurable training-time variable, and we will show that this distinction is where the design fails. In medical RAG, plugging an off-the-shelf NLI checker into the reward path fails in two ways that accuracy does not predict. A biomedical LLM scoring entailment by log-probability ( 26 ) can label > 97% of claims Neutral regardless of evidence (collapsing the per-claim support rate, the fraction labelled Entail , to zero gradient), even though the same model is accurate on balanced NLI tests. A strong proprietary verifier ( 1 ) that does not collapse instead opens a reward-hacking pathway ( 29 ) : ultra-short answers, then search avoidance, then generating non-English answers to evade the English-only NLI prompt; each shortcut closes under a countermeasure but exposes the next. Both failure modes are invisible to held-out NLI accuracy. Our central thesis is that the checker’s output distribution under the policy’s rollouts, not its held-out accuracy, decides whether it provides trainable gradient. We therefore focus on the per-claim support rate—the fraction of generated claims the checker labels Entail —as the primary diagnostic variable throughout the paper. This paper is a diagnosis of the checker-as-reward design space, not a competing system; we make no SOTA claim. We build a multi-turn medical RAG agent on Qwen2.5-7B-Instruct ( 7 ) trained with GRPO ( 27 ) , extending Search-R1 ( 9 ) with a dense retriever over the MedRAG corpus ( 37 ) and a claim-level NLI verification server as process reward. To isolate how the checker’s output distribution shapes training, we instantiate four NLI back-ends as drop-in replacements inside the same reward pathway: Likelihood-NLI (Meditron-3-8B log-prob), MedNLI-Cls (PubMedBERT classifier), Hybrid (GPT atomic extraction + local log-prob scoring), and GPT-NLI (GPT-4o-mini end-to-end). The four span the ( extractor , scorer ) design space along a GPT-dependency gradient (Figure 1 , Table 2 ); the per-claim mechanics are deferred to § 3 . We describe two operating regimes of per-claim support rate during training: moderate ( \lesssim 60%, e.g. MedNLI-Cls at 54%) and strong ( \gtrsim 75%, e.g. Hybrid at 76%, GPT-NLI at 86%). 1 1 1 These are descriptive labels for observed regimes, not strict cutoffs; what matters for the cascade (§ 6 ) is behaviour, not a fixed threshold. We report three diagnostic findings. First , signal collapse is specific to log-probability scoring: MedNLI-Cls produces non-degenerate verdicts on the same pairs Likelihood-NLI labels almost entirely Neutral (§ 5 ). Second , the relation between signal strength and answer quality is non-monotonic: a moderate-signal classifier outperforms a strong-signal proprietary checker (BERTScore 0.600 vs. 0.591) because the latter triggers a three-step cascade of shortcuts (ultra-short answers, search avoidance, language collapse) that each undermine clinical groundedness (§ 6 ). Third , signal strength is a property of the policy–checker pair, not of the checker alone; we support this with one cross-model swap (Qwen2.5-7B \to Qwen3-4B) and treat it as a working hypothesis to be tested at larger scale (§ , App. 25 ). We frame these findings as boundary conditions for concurrent verifier-as-reward systems. 2 Related Work We organise prior and concurrent work along four axes: process rewards in retrieval-augmented RL, claim-level verification for biomedical RAG, NLI as a reward model, and failure modes in LLM RL. Table 1 summarises five concurrent verifier-as-reward systems on seven design dimensions and locates our contribution in that design space. System Med. curriculum. Verifier-as-reward Claim Collapse Cascade Triage Search-R1 ( 9 ) – ✓ F1 only – – partial – EvidenceRL ( 31 ) ✓ ✓ PubMedBERT-MNLI ✓ – – – VERITAS ( 38 ) – † ✓ faithfulness ✓ partial ‡ – – MedRAGChecker ( 8 ) ✓ – ∗ eval-time ✓ – – – Med-R3 ( 19 ) ✓ ✓ retrieval-eff. – – – curr. Ours ✓

Read the complete text