Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models
Jaemin Kim, Jong Chul Ye · 2026
Read Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models by Jaemin Kim, Jong Chul Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17677.txt.
Introduction
Kim and Ye add adaptive guidance to retrieval-augmented masked diffusion models. Retrieved text can clash with what a model already knows, and noise then hurts output. Their ARAM framework needs no training and tunes the guidance scale during denoising. It reads the signal-to-noise ratio of the shift that retrieved context causes. Strong evidence gets more weight, while weak or noisy context gets damped. Knowledge-heavy QA scores beat strong RAG baselines, they report.
Catalog record
| CREATOR | Jaemin Kim, Jong Chul Ye |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50132 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17677
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models Jaemin Kim, Jong Chul Ye Abstract Retrieval-Augmented Generation (RAG) improves factual grounding by incorporating external knowledge into language model generation. However, when retrieved context is noisy, unreliable, or inconsistent with the model’s parametric knowledge, it introduces retrieval-prior conflicts that can degrade generation quality. While this problem has been studied in autoregressive language models, it remains largely unexplored in diffusion-based language models, where the iterative denoising process introduces unique challenges for integrating retrieved context. In this work, we propose Adaptive Retrieval-Augmented Masked Diffusion (ARAM) , a training-free adaptive guidance framework for Masked Diffusion Models (MDMs) in RAG settings. ARAM dynamically calibrates the guidance scale during denoising according to the Signal-to-Noise Ratio (SNR) of the distributional shift induced by retrieved context. Intuitively, the model strengthens guidance when the retrieved context provides reliable corrective evidence and suppresses it when the contextual signal is noisy or non-supportive. Extensive experiments on multiple knowledge-intensive QA benchmarks show that ARAM improves overall QA performance over competitive RAG baselines. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks 9 ; 11 . However, their reliance on static parametric knowledge often leads to hallucinations and outdated responses when dealing with dynamic or knowledge-intensive queries. Retrieval-Augmented Generation (RAG) 14 ; 30 addresses this limitation by grounding generation in external knowledge retrieved at inference time. Recently, diffusion-based language generation has emerged as a promising alternative to autoregressive models. In particular, Masked Diffusion Models (MDMs) generate text through an iterative denoising process that progressively fills masked tokens while leveraging bidirectional attention 19 ; 27 . Compared to autoregressive models, this generation paradigm enables parallel token updates and iterative refinement of intermediate predictions. Figure 1: Overview of ARAM. ARAM mitigates knowledge conflict via step-wise and token-wise adaptive guidance driven by the Signal-to-Noise Ratio. While recent research has focused on improving the generative capabilities of MDMs to narrow the gap with autoregressive models 18 , applying RAG to MDMs remains notably underexplored. Preliminary studies indicate that MDM-based RAG demonstrates remarkable in-context learning capabilities that often surpass those of autoregressive models 6 ; 28 . Despite this potential, existing approaches predominantly assume that the retrieved documents are reliable. In realistic RAG settings, however, the retrieved context is often noisy, partially irrelevant, or even contradictory to the model’s internal knowledge. A fundamental challenge is how to effectively ground the iterative denoising process in retrieved knowledge, especially when this external information conflicts with the model’s parametric prior. When such heterogeneous context is injected into the generation process, the model must dynamically decide how much it should trust the retrieved context versus its parametric prior. We refer to this tension between retrieved context and parametric knowledge as knowledge conflict 22 ; 24 ; 29 . This raises a key question: How should a diffusion language model determine when to trust retrieved context and when to rely on its parametric knowledge? To address this question, we propose Adaptive Retrieval-Augmented Masked Diffusion (ARAM) , an adaptive guidance framework that self-calibrates the guidance scale at each denoising step and token position. To effectively inject conditional information and control the generation quality in MDMs, we adopt Classifier-Free Guidance (CFG) 10 for RAG. However, applying a static guidance scale under retrieved contexts leads to a critical dilemma: (1) over-guidance on noisy context, which causes hallucinations, and (2) under-guidance on valid context, which fails to correct the flawed prior. Our key insight is that the influence of retrieved context can be interpreted as a distributional shift from a retrieval-free prior to a retrieval-conditioned posterior. Specifically, we analyze the interaction through an information-theoretic lens, and derive a Signal-to-Noise Ratio (SNR) of this distributional shift to adapt the guidance strength. Consequently, when retrieved context consistently supports the conditional distribution, guidance can be strengthened; when the retrieved context is uncertain or non-supportive, guidance can be attenuated to prevent the over-amplification of noise. We illustrate the overview of our proposed method in Fig. 1 . We evaluate our adaptive guidance mechanism on knowledge-intensive QA benchmarks. Empirical results demonstrate that our method outperforms both static CFG and baseline MDMs, achieving significant improvements in exact matching (EM) and F1 scores for MDM-based RAG. Our main contributions are as follows: • We propose ARAM, an adaptive CFG framework for MDMs in RAG, which dynamically modulates guidance scale based on the SNR. • We analyze the interaction between retrieval context and parametric prior in diffusion-based generation through an information-theoretic lens and derive a guidance rule motivated by signal and uncertainty. • Extensive experiments across multiple QA benchmarks validate that our method improves EM and F1 over other baselines and mitigates retrieval-prior conflicts. 2 Related Work 2.1 Knowledge Conflict in RAG While Retrieval-Augmented Generation (RAG) improves factual grounding by incorporating external context, it can also introduce knowledge conflict 17 between the model’s internal knowledge and the retrieved external context, leading to degraded performance and hallucinations 4 . To mitigate these conflicts, current approaches rely on external context assessors 5 , multi-agent debates 24 , or contrastive decoding 22 ; 23 . Among these, CAD 22 is most closely related to our work, as it contrasts output distributions with and without external context to penalize reliance on parametric priors. However, applying a static contrastive penalty often overcorrects when the retrieved context already aligns with the model’s parametric knowledge, inadvertently degrading performance on non-conflicting data 29 . To address this overcorrection, dynamic decoding strategies like AdaCAD 23 and COIECD 29 have emerged. While effective, these approaches are fundamentally designed for autoregressive language models and cannot be trivially extended to the non-causal, iterative denoising dynamics of diffusion-based language models. Motivated by the need for dynamic adjustment 23 ; 29 , we introduce a discrete-diffusion-specific framework that modulates the guidance scale at each denoising step using a Signal-to-Noise Ratio (SNR). 2.2 Classifier-Free Guidance in MDMs Classifier-Free Guidance (CFG) 10 is widely used to strengthen conditional generation by combining conditional and prior score functions. Since CFG was originally formulated for continuous spaces, defining an analogous procedure for discrete diffusion is nontrivial, as the discrete state space is inherently non-differentiable. Recent theoretical works 21 ; 20 have formalized the validity of CFG at the logit level in discrete diffusion models, but standard CFG applies a static guidance scale throughout the entire denoising process. Recognizing the dynamic uncertainty during generation, recent methods like A-CFG 15 dynamically alter the unconditional input by re-masking low-confidence tokens. Distinct from modifying the input formulation, our approach directly calibrates the guidance itself, preventing over-guidance on noisy context and under-guidance on valid corrections. 2.3 RAG in Masked Diffusion Models Recent studies demonstrate the potential of integrating RAG with MDMs to enhance knowledge utilization beyond autoregressive (AR) models. For example, recent analysis on Attention Floating 6 reveals that MDMs exhibit distinctive attention dynamics, where dominant attention tokens are dispersed across positions and denoising steps. This behavior allows MDMs to effectively track relevant context, resulting in stronger utilization of external knowledge than AR models. SPREAD 28 further shows that MDMs exhibit a strong reliance on contextual information and improve answer precision by enforcing query-level semantic coherence. Despite this potential, integrating RAG into MDMs remains unexplored. To the best of our knowledge, our work is