Progressive Training for Explainable Citation-Grounded Dialogue: Reducing Hallucination to Zero in English-Hindi LLMs

Vedant Pandya · 2026

Read Progressive Training for Explainable Citation-Grounded Dialogue: Reducing Hallucination to Zero in English-Hindi LLMs by Vedant Pandya (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18911.txt.

Introduction

Pandya presents a four-stage training plan for chat systems that cite their sources in English and Hindi. Stages cover language tuning, cited English chat, two-language chat, and a final reward step. Six models are tested at each stage. Citation training cuts made-up claims to zero for some model types from stage two on. The staged plan also guards Hindi skill while English grows. Three checks trace how citation habits form across training, not just whether they form.

Catalog record

CREATORVedant Pandya
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83756
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18911

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Progressive Training for Explainable Citation-Grounded Dialogue: Reducing Hallucination to Zero in English-Hindi LLMs Vedant Pandya Abstract Knowledge-grounded dialogue systems aim to generate informative, contextually relevant responses by conditioning on external knowledge sources. However, most existing approaches focus exclusively on English, lack explicit citation mechanisms for verifying factual claims, and offer limited transparency into model decision-making. We present XKD-Dial , a progressive four-stage training pipeline for explainable, knowledge-grounded dialogue generation in a bilingual (English–Hindi) setting, comprising: (1) multilingual adaptation, (2) English dialogue SFT with citation grounding, (3) bilingual dialogue SFT, and (4) GRPO alignment with citation-aware rewards. We evaluate six models spanning encoder-decoder (250M–3B) and decoder-only (1B–7B) architectures at every pipeline stage. Our key contributions are: (i) three post-hoc explainability analyses - cross-attention alignment, Integrated Gradients attribution, and occlusion-based causal grounding - applied systematically across the training trajectory to reveal how citation behaviour is learned, not only whether it is learned; (ii) citation-grounded SFT reduces hallucination to 0.0% for encoder-decoder models from Stage 2 onward; (iii) the progressive pipeline prevents catastrophic forgetting while improving Hindi capabilities; (iv) smaller models match larger models on English after SFT; and (v) GRPO provides marginal improvement over well-designed SFT for structured citation tasks. We evaluate across six automatic metrics (BLEU, ROUGE, BERTScore, FactScore, Citation-F1, hallucination rate). [Script=Devanagari, Scale=0.95] Keywords: Knowledge-Grounded Dialogue, Multilingual NLP, Explainability, Large Language Models, Citation Generation, Hindi, GRPO, Hallucination Reduction 1 Introduction Knowledge-grounded dialogue generation has emerged as a critical research direction for building conversational AI systems that produce factually accurate, informative responses [ Dinan et al. 2019 , Rashkin et al. 2021 ] . By conditioning response generation on retrieved knowledge passages, these systems can mitigate the chronic hallucination problem of large language models (LLMs) - where models generate plausible-sounding but factually incorrect information [ Ji et al. 2023 ] . However, current approaches suffer from three fundamental limitations. First, the monolingual bottleneck. The vast majority of knowledge-grounded dialogue research focuses exclusively on English [ Dinan et al. 2019 , Rashkin et al. 2021 , Kim et al. 2020 ] . For languages like Hindi - spoken by over 600 million people - there exists no standard benchmark, no established training methodology, and no systematic study of how knowledge-grounded dialogue systems perform in a bilingual setting. Extending such systems to Hindi is particularly challenging due to: (a) limited availability of Hindi dialogue corpora with knowledge annotations, (b) morphological richness and free word order that complicate both generation and evaluation, and (c) the need to handle code-switching and cross-lingual knowledge transfer. Second, the absence of verifiable citations. While retrieval-augmented generation (RAG) systems retrieve relevant passages [ Lewis et al. 2020 , Guu et al. 2020 ] , the generated response typically does not indicate which passage supports which claim. Without explicit citation markers (e.g., “According to [1], …”), users cannot verify factual claims against their sources, undermining trust and transparency. Recent work on attributed text generation [ Rashkin et al. 2021 ] has highlighted this gap, but citation-grounded training for dialogue remains underexplored. Third, the opacity of model decisions. Even when a model generates a correct, grounded response, it provides no insight into why it selected particular knowledge passages or how it composed the response. This opacity is especially problematic for citation-grounded systems: a model may produce the correct citation marker [1] without genuinely conditioning its output on passage 1, making citation accuracy an unreliable quality signal on its own. Interpretability methods - cross-attention visualization [ Jain and Wallace 2019 , Wiegreffe and Pinter 2019 ] , Integrated Gradients [ Sundararajan et al. 2017 ] , and occlusion-based causal grounding [ Lei et al. 2016 ] - can expose this dissociation, but their systematic application to knowledge-grounded dialogue generation across an entire training trajectory has not been attempted. 1.1 Our Approach We propose XKD-Dial ( E xplainable K nowledge-Grounded D ialogue), a progressive four-stage training pipeline designed to address all three limitations simultaneously. Our key insight is that complex multilingual, knowledge-grounded generation capabilities can be built incrementally, where each training stage adds a specific skill while preserving previously learned capabilities: 1. Stage 1: Multilingual Adaptation. English–Hindi translation training to build bilingual representations, particularly for models with limited Hindi pretraining exposure. 2. Stage 2: English Dialogue SFT. Supervised fine-tuning on English knowledge-grounded dialogue with explicit citation markers, teaching the model to generate responses that attribute claims to specific knowledge passages. 3. Stage 3: Bilingual Dialogue SFT. Extension to Hindi dialogue with citations, leveraging cross-lingual transfer from Stage 2. 4. Stage 4: GRPO Alignment. Reinforcement learning via Group Relative Policy Optimization [ Shao et al. 2024 ] with a composite reward function that incentivizes citation accuracy, factual consistency, and penalizes hallucination. 1.2 Contributions Our main contributions are as follows: 1. A progressive training pipeline for bilingual knowledge-grounded dialogue. We introduce a four-stage methodology that incrementally builds multilingual, citation-grounded dialogue capabilities while preventing catastrophic forgetting. To our knowledge, this is the first systematic pipeline for English–Hindi knowledge-grounded dialogue with citations. 2. Comprehensive cross-architecture empirical study. We evaluate six models across two architecture families (encoder-decoder and decoder-only) spanning 250M to 7B parameters, with each model evaluated at every pipeline stage (30 total evaluation runs). This provides fine-grained ablation of which stage contributes which capability. 3. Citation-grounded hallucination reduction. We observe that training with explicit citation format substantially reduces hallucination rates (reaching 0.0% under automatic NLI-based evaluation from Stage 2 onward for encoder-decoder models), suggesting citation-grounded SFT as a promising anti-hallucination strategy warranting further investigation including human evaluation. 4. Empirical analysis of GRPO for structured tasks. We provide an empirical characterisation of GRPO behaviour in our experimental configuration ( \beta =0.04, 500 steps), finding marginal improvement over SFT. This contributes to the broader discussion of when RL alignment is beneficial, though comprehensive hyperparameter exploration remains future work. 5. Explainability analysis. We apply attention visualization and token attribution methods to analyze how models attend to knowledge passages during generation, providing interpretability insights for knowledge-grounded dialogue. The remainder of this paper is organized as follows. Section 2 surveys related work. Section 3 details our four-stage training pipeline. Section 4 describes the experimental setup including datasets, models, and evaluation metrics. Section 5 presents results and analysis. Section 6 discusses key findings and their implications. Section 7 concludes with future directions. 2 Related Work Our work intersects several active research areas: knowledge-grounded dialogue systems, multilingual language models, reinforcement learning from human feedback, and explainability in natural language generation. We review each in turn, highlighting the gaps that motivate our approach. 2.1 Knowledge-Grounded Dialogue The seminal Wizard of Wikipedia [ Dinan et al. 2019 ] established the paradigm of conditioning dialogue responses on retrieved Wikipedia passages, demonstrating that access to external knowledge significantly improves informativeness and factual accuracy. Subsequent work addressed the critical problem of faithfulness : FaithDial [ Rashkin et al. 2021 ] introduced a benchmark specifically targeting hallucination in knowledge-grounded dialogue, showing that standard models frequently generate claims unsupported by the provided knowledge. The DSTC9 shared task [ Kim et al. 2020 ] extended the challenge to unstructured knowledge access, requiring models to identify relevant knowledge snippets from FAQs and reviews before generating responses. On the modeling side, BlenderBot 2.0 [ Shuster et al. 2022 ] combined internet search with

Read the complete text