Lessons from Training Grounded LLMs with Verifiable Rewards
Shang Hong Sim, Tej Deep Pala, Vernon Toh, Hai Leong Chieu, Amir Zadeh, Chuan Li, Navonil Majumder, Soujanya Poria · 2025
Read Lessons from Training Grounded LLMs with Verifiable Rewards by Shang Hong Sim, Tej Deep Pala, Vernon Toh, Hai Leong Chieu, Amir Zadeh, Chuan Li, Navonil Majumder, Soujanya Poria (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15522.txt.
Introduction
Shang Hong Sim and co-authors share in 2025 lessons from training grounded language models with checkable rewards. Grounded answers still fail often: models miss stated facts, cite wrongly, or refuse needlessly. Reward training here targets answer truth, citation sufficiency, and refusal quality.
Reasoning-augmented models beat instruction-only ones across four question sets. A two-stage plan, answers and citations first and refusal second, steadies the learning signal. Distilled instruction tuning plus reward training lifts long-form answers further. Unanswerable questions show the largest gains from the reasoning step. No gold reasoning traces or costly hand labels feed the training.
Catalog record
| CREATOR | Shang Hong Sim, Tej Deep Pala, Vernon Toh, Hai Leong Chieu, Amir Zadeh, Chuan Li, Navonil Majumder, Soujanya Poria |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15522
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lessons from Training Grounded LLMs with Verifiable Rewards Shang Hong Sim, Tej Deep Pala, Vernon Toh, Hai Leong Chieu, Amir Zadeh, Chuan Li, Navonil Majumder, Soujanya Poria Abstract Generating grounded and trustworthy responses remains a key challenge for large language models (LLMs). While retrieval-augmented generation (RAG) with citation-based grounding holds promise, instruction-tuned models frequently fail even in straightforward scenarios: missing explicitly stated answers, citing incorrectly, or refusing when evidence is available. In this work, we explore how reinforcement learning (RL) and internal reasoning can enhance grounding in LLMs. We use the GRPO (Group Relative Policy Optimization) method to train models using verifiable outcome-based rewards targeting answer correctness, citation sufficiency, and refusal quality, without requiring gold reasoning traces or expensive annotations. Through comprehensive experiments across ASQA, QAMPARI, ELI5, and ExpertQA we show that reasoning-augmented models significantly outperform instruction-only variants, especially in handling unanswerable queries and generating well-cited responses. A two-stage training setup, first optimizing answer and citation behavior and then refusal, further improves grounding by stabilizing the learning signal. Additionally, we revisit instruction tuning via GPT-4 distillation and find that combining it with GRPO enhances performance on long-form, generative QA tasks. Overall, our findings highlight the value of reasoning, stage-wise optimization, and outcome-driven RL for building more verifiable and reliable LLMs. 1 Introduction The deployment of large language models (LLMs) in information-seeking applications has grown rapidly, yet ensuring that these models generate factually reliable and grounded outputs remains an open challenge. A core concern is the problem of hallucination , where models produce fluent but fabricated or unfaithful content ( Ji et al. 2023 ) . Retrieval-Augmented Generation (RAG) offers a promising solution by conditioning responses on external documents, which improves factual accuracy and enables verification ( Asai et al. 2023 ) . Despite this, even advanced models like GPT-4 often mishandle retrieved information—leading to RAG-specific hallucinations when evidence is misused, omitted, or over-cited ( Gao et al. 2023b ; Song et al. 2025b ) . Recently, several works have begun integrating retrieval more tightly into internal reasoning processes. For example, Search-o1 and RAG-Star embed retrieval into multi-step reasoning pipelines to dynamically acquire and filter supporting evidence ( Li et al. 2025b ; Jiang et al. 2024 ) . Other approaches like ReSearch and R1-Searcher train LLMs to autonomously issue retrieval queries using reinforcement learning (RL) ( Chen et al. 2025 ; Song et al. 2025a ) , while CoT-RAG enhances chain-of-thought prompting with sub-query decomposition and knowledge graph guidance ( Li et al. 2025a ) . While these methods improve retrieval-driven reasoning, their primary focus is on complex QA or planning tasks and not directly on grounding alignment, such as accurate citation or proper refusal when no answer is found. In this paper, we focus specifically on improving grounded response quality in LLMs—ensuring that answers are correct, verifiably cited, and appropriately refused when evidence is lacking. We propose Ground-GRPO , a two-stage reinforcement learning framework based on Group Relative Policy Optimization (GRPO). We propose a novel hierarchial reward system that uses verifiable outcome-based rewards to enhance grounding in 3 directions: Answer Correctness, Citation Quality and Refusal. Ground-GRPO proceeds in two stages: 1. Answer and Citation Optimization: The model is trained on answerable examples using rewards that encourage correct answers and minimal yet sufficient citations. 2. Refusal Learning: The model is then trained on a mixture of answerable and unanswerable examples, using a refusal-specific reward to discourage unsupported answers. Our results show that large reasoning models outperform instruction-tuned variants across multiple QA benchmarks—ASQA, QAMPARI, and ELI5—under the TrustScore metric ( Song et al. 2025b ) , with notable gains in identifying unanswerable questions and providing precise, grounded citations. We find that reasoning-enhanced models benefit significantly more from Ground-GRPO than their instruction-only counterparts, highlighting the synergy between intermediate reasoning and reinforcement learning. Additionally, prior work has shown that GPT-4-based distillation improves model groundedness and citation quality. We find that distillation performs well in structured, list-style tasks like QAMPARI, while Ground-GRPO achieves stronger results on open-ended, long-form QA tasks such as ELI5, where generating coherent, well-cited explanations is more challenging. Key Takeaways • Reasoning boosts grounding: Large Reasoning Models generate more grounded answers compared to instruct models. • RL is more effective on reasoning models: Ground-GRPO yields greater improvements on reasoning models than on instruction-tuned models. • Staged training improves alignment: Decomposing rewards across stages—first for answer quality, then for refusal—stabilizes training and leads to better overall grounding. • Distillation and RL are complementary on longform QA datasets: GPT-4 distillation and fine-tuning small models through SFT or DPO improves both the shortform and longform QA performance; GRPO further enhances grounding, especially in open-ended, long-form tasks. Figure 1: Overview of the Ground-GRPO framework for reward-guided reinforcement learning in retrieval-augmented generation (RAG) settings. Given a user query, the Policy Model generates a set of candidate outputs O_{1},\ldots,O_{N} , each of which is evaluated by a Reference Model to compute three distinct reward signals: (1) EM Reward , which checks for exact matches against ground-truth answers and provides a score based on the number of correct hits; (2) Citation Reward , which uses natural language inference (NLI) to verify whether the cited documents support the predicted answer; and (3) Refusal Reward , which incentivizes the model to appropriately abstain from answering unanswerable questions based on the evidence provided. The computed rewards R_{1},\ldots,R_{N} are aggregated using Group Computation to update the policy via reinforcement learning. The lower section of the figure illustrates examples for answerable and unanswerable queries, showing the model’s internal reasoning, selected answer, and corresponding reward computations. For answerable queries, both EM and citation rewards are used, while for unanswerable queries, a correct refusal is rewarded. Ground-GRPO thus encourages grounded, precise, and abstaining behavior in RAG systems. 2 Problem Definition and Method Given a question Q and a set of relevant documents D , the LLM is instructed to generate a response S consisting of a set of citation-grounded statements \{s_{1},s_{2},...\} . Each statement s_{i} is accompanied by a set of inline citations C_{i}=\{c_{i,1},c_{i,2},..\} that refers to the documents in D , e.g., statement1 [1][2] statement2 [3] . If the relevant documents D is insufficient to answer the question, the question is deemed unanswerable , and the gold response S would be a refusal statement, such as, “I apologize, but I couldn’t find an answer to your question in the search results” . Otherwise, the question is answerable . 2.1 Two-stage Reinforcement Learning To facilitate robust learning of training signals by the LLM, we propose a two-stage RL approach that incorporates verifiable rewards. In Stage 1, the model is trained to accurately answer answerable questions with appropriate citations while adhering to specified output format constraints. In Stage 2, the model is further trained on a mixture of answerable and unanswerable questions, enabling it to provide cited responses when possible and appropriately refuse to answer when questions are unanswerable. Stage 1. The model is trained only on answerable questions in this stage. The reward function consists of three components: exact match (EM), citation, and format rewards. The primary objective is for the model to learn how to effectively utilize the provided documents to generate accurate responses to answerable questions. The EM (Exact Match) reward is computed