Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise
Bo Long, Deepak Agarwal, Jelena Markovic-Voronov, Yi Wang, Liuqing Li · 2026
Read Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise by Bo Long, Deepak Agarwal, Jelena Markovic-Voronov, Yi Wang, Liuqing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18832.txt.
Introduction
Bo Long and four colleagues give transformers a clear handle on uncertainty through filtering that tracks belief. Every token now carries equal trust, a flaw that bites on cold-start items, uneven signal quality, and attention sinks. In their Bayesian Filtering Transformer, attention becomes precision-weighted kriging, the shortcut link becomes a Kalman update, and the feed-forward block carries that weight onward. A free gauge supplies the seen-data weight. Tests show gains for item suggestions and noisy fine-tuning, largest where uncertainty runs highest.
Catalog record
| CREATOR | Bo Long, Deepak Agarwal, Jelena Markovic-Voronov, Yi Wang, Liuqing Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100725 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise Bo Long, Deepak Agarwal, Jelena Markovic-Voronov, Yi Wang, Liuqing Li Abstract The Transformer is the foundational building block of modern AI, yet offers no principled handling of uncertainty , which is prevalent in real applications: cold-start tokens with sparse histories in sequential recommendation, heterogeneous signal quality in language models, and attention sinks induced by unconstrained softmax. Every token is treated with uniform confidence. We show this uniformity is a degenerate case of our Bayesian Filtering Transformer (BFT): attention becomes precision-weighted kriging, the residual connection becomes a Kalman update with adaptive gain, and the FFN becomes a dynamics model propagating precision via a Jacobian–plus–process-noise rule. Observation precision comes from a parameter-free Restricted Maximum Likelihood (REML) estimator with a conjugate Bayesian prior. BFT replaces any Transformer layer with negligible overhead. On sequential recommendation, BFT applied to three major architectures yields significant gains on six benchmarks, with the largest improvements on cold-start users and rare items where uncertainty is highest. On supervised fine-tuning of large language models with noisy data, BFT improves robustness in two regimes: noisy supervision (token-label corruption in question answering) and noisy context (retrieval-augmented QA with real RAG distractors). A single principled modification—restoring precision—unlocks substantial headroom across both classical sequence-modeling and modern LLM regimes. 1 Introduction The Transformer ( Vaswani et al., 2017 ) is the dominant architecture in modern AI: language models, recommenders, vision encoders, multimodal systems. Its self-attention mechanism computes a weighted combination of values mediated by query–key similarities. Although this simple design has proven remarkably universal, it makes an implicit assumption that is limiting: every token is equally reliable . Real applications routinely violate this assumption. In generative recommendation ( Zhai et al., 2024 ) , user sequences contain a mixture of high-signal interactions (careful purchases) and noisy clicks, while items range from popular products with rich histories to cold-start items observed only a handful of times ( Chen et al., 2022 ; Wang et al., 2021 ) . The current Transformer does not have a principled, model-based way to down-weight a noisy item or to express uncertainty about a cold item. Heuristics applied at the input-construction stage are widely employed but hard to tune. This is one of the main reasons why Transformer-based sequential recommenders have struggled to deliver strong performance in cold-start scenarios. In LLM fine-tuning on real-world data , domain corpora contain tokens of uneven quality; a standard loss weights every token identically, giving low-signal tokens the same gradient pull as high-signal ones ( Li et al., 2020 ) . The same uniform-confidence pathology shows up elsewhere in modern LLMs: attention sinks ( Xiao et al., 2024 ; Sun et al., 2024 ; Qiu et al., 2025 ) concentrate attention mass on uninformative early tokens, and position uncertainty grows with retrieval depth in long-context RAG ( Liu et al., 2024 ) —in both cases, the architecture has no native channel through which to say “this position is unreliable.” Bespoke fixes have been proposed for each of these phenomena—denoising masks, attention calibration, sigmoid gates, sink tokens—but they remain isolated patches. We show that a single Bayesian formulation that propagates uncertainty through estimates provides a holistic solution and addresses these failure modes together. Contributions. 1. A unifying framework (Section 2 ). We formulate the Transformer as a three-step Bayesian filter— observe, update, predict —and identify the precision \lambda{=}1 degeneracy as the common structural cause of attention sinks, noisy-feedback corruption, and cold-start degradation. 2. An efficient, drop-in algorithm (Section 3 ). We derive BFT’s observation precision from Restricted Maximum Likelihood with a conjugate Bayesian prior— parameter-free and content-aware. Precision propagation through the FFN combines the delta method with per-dimension process noise, and an expanded-square identity makes the whole update scale at standard attention’s peak memory. Drop-in integration requires replacing a single layer class. 3. Empirical validation across two impactful application domains (Section 5 ). On generative recommendation across three architecturally distinct backbones—SASRec and BERT4Rec under softmax attention, HSTU under SiLU-gated attention, each instantiating BFT with the variance estimator the framework prescribes for its weighting scheme (REML for softmax-normalized weights, sandwich for non-normalized; Section 4.1 )—and six standard benchmarks, BFT delivers significant gains, with the largest improvements where the framework predicts they should be: rare items and cold users . On supervised fine-tuning of TinyLlama-1.1B, BFT improves robustness over standard SFT under both noisy supervision (token-label corruption on SQuAD) and noisy context (Contriever-retrieved RAG distractors on NQ-Open). The pattern is consistent: precision tracking helps most where signal-to-noise is lowest. 2 The Bayesian Filtering Transformer: Framework We first present BFT at the level of the framework —what each layer does, in Kalman-filter terms, without specifying how individual quantities are estimated. Section 3 then instantiates the framework with concrete estimators. 2.1 Bayesian filtering at a glance A Bayesian filter maintains, at each step, a belief about a latent state. The belief is a distribution with a mean (point estimate) and a precision (inverse variance, quantifying confidence). Each step consumes an observation and produces an updated belief via three operations: observe \to update \to predict . The observe step produces an estimate e from the current observation and assesses its precision \lambda^{\mathrm{obs}} . The update step merges e with the prior belief — written as a (mean, precision) pair (h,\lambda) — via the Kalman gain K=\lambda^{\mathrm{obs}}/(\lambda+\lambda^{\mathrm{obs}}) , yielding posterior h^{\prime}=h+K\cdot e and posterior precision \lambda^{\prime}=\lambda+\lambda^{\mathrm{obs}} . The predict step evolves the state forward through a dynamics model f ; in precision form, the prior precision at the next step is \lambda/(\partial f/\partial h)^{2} , reduced by process noise. This is the textbook form of the (Extended) Kalman Filter ( Kalman, 1960 ) . Its merits are well-known: the gain K adapts automatically between “trust the prior” ( K\to 0 when observations are unreliable) and “trust the observation” ( K\to 1 when the prior is uncertain), with no learned gate; precision propagates through the dynamics without ad hoc normalization; and noise accumulates and dissipates at principled rates. 2.2 Mapping the Transformer to the filter A pre-norm Transformer layer computes self-attention and an FFN, each preceded by a normalization \mathrm{Norm}(\cdot) and followed by a residual connection: h^{\prime}=h+\mathrm{Attn}(\mathrm{Norm}(h)),\qquad h^{\prime\prime}=h^{\prime}+\mathrm{FFN}(\mathrm{Norm}(h^{\prime})). (1) We identify each operation with a Kalman-filter step: • Self-attention is the observe step. Attention pools values v_{j} from surrounding positions via weights \alpha_{tj}=\mathrm{softmax}_{j}(q_{t}^{\top}k_{j}/\sqrt{d}) . This is a local weighted estimate of the state at position t using neighboring observations—precisely the role of a kriging predictor in geostatistics ( Krige, 1951 ; Mathéron, 1963 ) or a Nadaraya–Watson regressor in statistics ( Nadaraya, 1964 ) . In both cases, the estimate carries not only a mean but a variance, which the standard Transformer discards. • The residual connection is the update step. The standard residual h^{\prime}=h+e with e=\mathrm{Attn}(\mathrm{Norm}(h)) is the K{=}1 instance of the Kalman update h^{\prime}=h+K\cdot e . With K<1 , the model can partially trust attention evidence. • The FFN is the predict step. The FFN transforms h^{\prime} to produce the input for the next layer. In the filter view, this is the dynamics model that evolves the state forward one step; its Jacobian should propagate precision to the next layer. In this mapping, what is missing from the standard Transformer is