Large Language Models as Discounted Bayesian Filters

Jensen Zhang, Jing Yang, Keze Wang · 2025

Read Large Language Models as Discounted Bayesian Filters by Jensen Zhang, Jing Yang, Keze Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18489.txt.

Introduction

Jensen Zhang, Jing Yang, and Keze Wang ask how large language models update their beliefs as new evidence streams in. Earlier work mostly tested static tasks, but agents and world models must revise their views step by step. Their paper, posted to arXiv in December 2025, builds a Bayesian filtering framework to probe this online reasoning.

Their test suite covers discrete cases like dice rolls and continuous ones like Gaussian processes, with true values that shift over time. They find that model updates resemble Bayesian reasoning but with a twist: each model acts like an exponential forgetting filter, discounting older evidence by a factor below one. The built-in priors are often off, yet the updating itself stays orderly. Prompting tricks that reset the priors fix much of the miscalibration at little cost.

Catalog record

CREATORJensen Zhang, Jing Yang, Keze Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18489

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models as Discounted Bayesian Filters Jensen Zhang, Jing Yang, Keze Wang Abstract Large Language Models (LLMs) demonstrate strong few-shot generalization through in-context learning (ICL), yet their reasoning in dynamic and stochastic environments remains opaque. Prior studies mainly address static tasks, overlooking the online adaptation required when beliefs must be continuously updated—a key capability for LLMs as world models or agents. We introduce a Bayesian filtering framework to evaluate online inference in LLMs. Our probabilistic probe suite spans multivariate discrete (e.g., dice rolls) and continuous (e.g., Gaussian) distributions, where ground-truth parameters shift over time. We find that while LLMs’ belief updates resemble Bayesian posteriors, they are more accurately described by an exponential forgetting filter with a model-specific discount factor γ < 1 \gamma<1 . This reveals systematic discounting of older evidence, varying significantly across architectures. Although inherent priors are often miscalibrated, the updating mechanism itself is structured and principled. We validate these findings in a simulated agent task and present prompting strategies that effectively recalibrate priors with minimal cost. † † address: Sun Yat-sen University 1 Introduction In-context learning (ICL) represents a remarkable capability of large language models (LLMs) [ 23 , 20 , 18 , 17 ] , enabling rapid adaptation to novel tasks based solely on a handful of examples provided in their prompts, without explicit gradient updates. While the empirical success of ICL underpins modern prompting techniques, its fundamental mechanism remains largely opaque. A key open question is whether ICL constitutes structured statistical reasoning analogous to Bayesian inference or is merely sophisticated pattern recognition. A promising perspective frames ICL as implicit Bayesian inference, where models iteratively update latent belief states based on contextual evidence [ 21 , 19 , 24 ] . However, existing foundational studies predominantly investigate static environments, assuming stationary data-generating distributions. Such an assumption neglects a crucial aspect of real-world intelligence: the necessity to operate effectively within non-stationary environments. In these dynamic contexts, agents must continuously integrate new information while systematically discounting—or “forgetting”—outdated evidence whose relevance diminishes over time. This capacity for online adaptation is critical for deploying LLMs as reliable world models or autonomous agents, yet remains under-explored. Figure 1: Belief trajectory on the Biased Die task: LLM estimates (solid) vs. discounted filters (dashed), with \gamma^{*} giving the closest match. In this work, we propose a new theoretical perspective: we conceptualize ICL in Transformers as online Bayesian filtering characterized by systematic discounting of past evidence . Central to our thesis is the assertion that LLMs, when confronted with sequences of evolving evidence, do not function as ideal Bayesian observers. Instead, we hypothesize that their behavior incorporates an intrinsic forgetting mechanism, likely arising from architectural elements of Transformers [ 16 , 22 ] . To formalize and empirically evaluate this hypothesis, we introduce an analytical framework built around fitting a discount factor \gamma\in(0,1] that minimizes the Kullback-Leibler (KL) divergence between a model’s predictive distribution and a theoretical Bayesian filter’s. We investigate this dynamic behaviour through a controlled probabilistic probe suite [ 16 , 6 , 5 ] , involving tasks such as biased die rolling and Gaussian mean estimation, where the ground-truth parameters undergo sudden shifts, compelling models to adapt their posterior beliefs. Through our integrated experimental and analytical approach, we present the following contributions: 1. We establish a novel framework for interpreting ICL as online Bayesian filtering. Our analysis provides the first quantitative demonstration that LLMs systematically discount historical information, a behaviour accurately modelled by fitting a stable, model-specific discount factor \gamma via a principled optimization process. 2. The proposed framework enables a nuanced decomposition of predictive errors into what we term Update Divergence and Model Misspecification Divergence . We show that observed performance limitations are predominantly attributable to the latter, which stems from miscalibrated priors and intrinsic discounting, rather than a flawed updating process itself. 3. We further investigate the internal mechanisms underlying this discounting behaviour, revealing through correlation analysis that inferential quality is decoupled from the raw magnitude of attention scores. This finding points to a complex architectural basis for evidential forgetting that extends beyond straightforward attention allocation. 2 Related Work In-context learning (ICL) in large language models (LLMs) [ 4 ] has been extensively studied as a form of implicit statistical inference, particularly through a Bayesian lens. Foundational works, such as those by Xie et al. [ 16 ] and Aky"urek et al. [ 2 ] , frame ICL as approximating Bayesian posterior updates or gradient descent on latent functions, enabling few-shot generalization in stationary environments. However, these perspectives assume fixed data distributions, overlooking the challenges of non-stationary settings where beliefs must adapt online to shifting parameters—a gap our discounted filtering framework addresses by introducing systematic evidence forgetting. Recent efforts have begun exploring dynamic adaptation in LLMs, including continual learning approaches [ 10 ] that mitigate catastrophic forgetting via architectural modifications or replay buffers. For instance, methods like self-synthesized rehearsal [ 8 ] generate synthetic data to preserve knowledge across tasks. Yet, these often focus on task-specific retention rather than principled probabilistic updating under uncertainty.Our work diverges by providing a unified Bayesian filtering model with a fitted discount factor \gamma<1 , quantifying deviations from ideal inference and linking them to Transformer attention mechanisms, thus bridging static ICL theory with online reasoning. 3 Methodology Our methodology is designed to empirically test if LLM in-context learning emulates online Bayesian inference with evidence discounting. We first define our theoretical model and the non-stationary tasks used for probing, then detail the quantitative analysis, summarized in Algorithm 1 . 3.1 Framework and Experimental Probes Discounted Bayesian Filtering. We model the "forgetting" of past evidence using a discounted Bayesian filtering framework [ 15 ] . This introduces a discount factor, \gamma\in(0,1] , which tempers the posterior belief p_{t-1}(\theta|D_{1:t-1}) before it is updated with new evidence D_{t} : p^{\prime}_{t-1}(\theta)\propto[p_{t-1}(\theta|D_{1:t-1})]^{\gamma} (1) Here, \gamma=1 represents a standard Bayesian filter with perfect memory, while \gamma\to 0 approaches a memoryless state. Probabilistic Probe Suite. To test online adaptation, we designed two non-stationary tasks over a sequence of T=100 observations, with a parameter changepoint at t=51 : Biased Die (Discrete): A 6-sided die where the dominant face abruptly changes from one to another. Gaussian Mean (Continuous): Samples from \mathcal{N}(\mu,1) , where the mean \mu shifts from 2.0 to -2.0 . At each timestep t , we elicit the LLM’s predictive distribution \hat{\mathbf{p}}_{\text{LLM},t} by providing the history D_{1:t-1} and applying softmax to the output logits for all possible next outcomes. 3.2 Quantitative Analysis Our analysis quantitatively connects the LLM’s behavior to our framework. The core procedure is outlined in Algorithm 1 . 1. Quantifying Discounting ( \gamma^{*} ): We fit an optimal discount factor \gamma^{*} by minimizing the total KL divergence between the LLM’s predictive sequence and that of the discounted filter: \gamma^{*}=\arg\min_{\gamma\in(0,1]}\sum_{t=1}^{T}\mathcal{D}_{\text{KL}}\left(\hat{\mathbf{p}}_{\text{LLM},t}\parallel\mathbf{p}_{\text{Bayes},t}(\gamma)\right) (2) We solve this using the L-BFGS-B algorithm. A fitted \gamma^{*}<1 provides direct evidence of discounting. 2. Decomposing Predictive Error: We diagnose the source of errors by decomposing the total divergence into two components: Update Divergence ( \mathcal{D}_{\text{Update}} ): \mathcal{D}_{\text{KL}}(\hat{\mathbf{p}}_{\text{LLM}}\parallel\mathbf{p}_{\text{Bayes}}(\gamma^{*})) . Measures if the LLM is a principled discounted updater. Model Misspecification Divergence ( \mathcal{D}_{\text{ModelSpec}} ): \mathcal{D}_{\text{KL}}(\mathbf{p}_{\text{Bayes}}(\gamma^{*})\parallel\mathbf{p}_{\text{Truth}}) . Measures the inherent error of the best-fit model. 3. Analyzing Architectural Basis: We investigate the role of attention by computing the

Read the complete text