Belief-State RWKV for Reinforcement Learning under Partial Observability

Liu Xiao · 2026

Read Belief-State RWKV for Reinforcement Learning under Partial Observability by Liu Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09671.txt.

Introduction

Liu Xiao reframes reinforcement learning on RWKV-style models. The fixed recurrent state becomes a belief state, not an opaque hidden vector. The method keeps a compact uncertainty-aware state of mean and spread. Control then depends on memory and confidence together. A pilot with hidden episode-level noise shows belief-state policies nearly matching the best recurrent baseline. They slightly improve returns where noise is hardest and under a held-out shift. Simple belief readout also beats two fancier extensions.

Catalog record

CREATORLiu Xiao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22674
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Belief-State RWKV for Reinforcement Learning under Partial Observability Liu Xiao Abstract We propose a stronger formulation of RL on top of RWKV-style recurrent sequence models, in which the fixed-size recurrent state is explicitly interpreted as a belief state rather than an opaque hidden vector. Instead of conditioning policy and value on a single summary h t h_{t} , we maintain a compact uncertainty-aware state b t = ( μ t , Σ t ) b_{t}=(\mu_{t},\Sigma_{t}) derived from RWKV-style recurrent statistics and let control depend on both memory and uncertainty. This design targets a key weakness of plain fixed-state policies in partially observed settings: they may store evidence, but not necessarily confidence. We present the method, a theoretical program, and a pilot RL experiment with hidden episode-level observation noise together with a test-time noise sweep. The pilot shows that belief-state policies nearly match the best recurrent baseline overall while slightly improving return on the hardest in-distribution regime and under a held-out noise shift. Additional ablations show that this simple belief readout is currently stronger than two more structured extensions, namely gated memory control and privileged belief targets, underscoring the need for richer benchmarks. Code is available at https://github.com/xiaol/Autoresearch_ideas . 1 Introduction RWKV shows that a recurrent architecture can retain constant-space inference while still supporting transformer-like parallel training 17 ; 18 . This suggests an appealing direction for reinforcement learning: use an RWKV-style recurrent state as the sole interface between long-horizon history and decision making. Our earlier formulation attached policy and value heads directly to a generic linear RNN state h_{t} . While simple, that view leaves a core question unresolved: if the agent is uncertain about the latent state of the environment, where is that uncertainty represented? That question is especially important because RL generalization often creates implicit partial observability even when the nominal task description appears fully observed 4 . In other words, a recurrent policy is not only compressing history; it is implicitly constructing belief. We argue that the next step is to reinterpret the RWKV state as a belief state . Concretely, instead of a single hidden summary, we maintain two coupled fixed-size components: a location statistic \mu_{t} and an uncertainty statistic \Sigma_{t} . Policy and value are conditioned on both. This creates a compact interface between memory, uncertainty, and control while retaining the efficiency of RWKV-style recurrence. Contributions. • We introduce a belief-state variant of RL-conditioned RWKV-style models that conditions policy and value on (\mu_{t},\Sigma_{t}) . • We formalize proposition-level statements around approximate sufficiency, stability, and low-rank reward-relevant state structure. • We provide a pilot partially observed RL experiment with hidden observation noise, showing promising gains in the hardest and shifted regimes. • We add ablation and calibration diagnostics showing that the plain belief-state readout is the strongest simple out-of-distribution variant in the current suite. 2 Related Work Our work sits at the intersection of RWKV-style recurrent sequence modeling, recurrent RL under partial observability, and state representation learning. RWKV reframes the efficient-sequence-model tradeoff by combining parallelizable training with recurrent inference 17 , and Eagle/Finch extend the architecture with matrix-valued states and dynamic recurrence 18 . In RL, Decision Mamba adapts selective state spaces to sequence modeling for control 15 , while KalMamba moves closer to our view by combining probabilistic state-space models with efficient sequence backbones for RL under uncertainty 1 . On the broader RL side, recent surveys emphasize that representation quality and uncertainty handling are central to sample efficiency and generalization 3 . Generalization-focused analyses further argue that many practical RL problems should be treated as epistemic POMDPs 4 , and recurrent model-free agents remain strong baselines on such tasks 6 ; 10 ; 14 . Transformer baselines such as GTrXL show that better long-range sequence processing can matter substantially in RL, but they typically sacrifice the constant-state deployment story that makes RWKV attractive 16 . Our proposal is closer in spirit to predictive-state approaches that try to expose belief-like recurrent structure directly 7 ; 2 and to probabilistic latent-state methods such as DVRL 9 , but we focus on the control interface of an RWKV-like backbone itself. This complements benchmark efforts such as POPGym and newer memory-improvable suites 13 ; 21 , along with model-based work showing strong efficiency gains from learned world models 5 ; 11 . Figure 1: The proposed RWKV-first interface. Long observation history is compressed into a fixed-size belief state (\mu_{t},\Sigma_{t}) . Policy and value read from this uncertainty-aware state, while reward drives parameter updates rather than directly defining the instantaneous recurrent state. 3 Method 3.1 Belief-State RWKV Recurrence We replace the opaque hidden state with a structured fixed-size belief state b_{t}=(\mu_{t},\Sigma_{t}), (1) where \mu_{t} is a location statistic and \Sigma_{t} is an uncertainty statistic. In the simplest case, both are produced by linear recurrent accumulators: \displaystyle s^{(1)}_{t} \displaystyle=A_{1}s^{(1)}_{t-1}+B_{1}x_{t}, (2) \displaystyle s^{(2)}_{t} \displaystyle=A_{2}s^{(2)}_{t-1}+B_{2}\phi(x_{t}), (3) followed by a deterministic map \mu_{t}=f_{\mu}(s^{(1)}_{t}),\qquad\Sigma_{t}=f_{\Sigma}(s^{(1)}_{t},s^{(2)}_{t}). (4) The key design principle is that the state remains fixed-size and recurrent, but it is now explicitly tasked with representing both what the agent believes and how certain it is . 3.2 RWKV Instantiation In a full RWKV instantiation, the belief-state mechanism would sit on top of the time-mix/channel-mix backbone rather than replacing it. Concretely, one can use the RWKV recurrent state emitted after the time-mix update as the carrier of history, then learn lightweight readouts that produce \mu_{t} and \Sigma_{t} from that state. The actor and critic then consume (\mu_{t},\Sigma_{t}) instead of a raw hidden vector. This keeps the computational advantages of RWKV while making uncertainty explicit at the control interface. At the block level, let h_{t-1} denote the incoming feature state and let s_{t-1} denote the RWKV temporal memory. We write the time-mix update abstractly as (u_{t},s_{t})=\mathrm{TimeMix}(x_{t},h_{t-1},s_{t-1}), (5) where u_{t} is the time-mixed feature and s_{t} is the updated recurrent memory. A belief readout then branches directly from the temporal pathway: z_{t}=\psi(u_{t},s_{t}),\qquad\mu_{t}=W_{\mu}z_{t},\qquad\log\Sigma_{t}=W_{\Sigma}z_{t}. (6) The feature state passed onward through the backbone is produced by the channel-mix block, h_{t}=\mathrm{ChannelMix}(u_{t}). (7) This placement is intentional: the time-mix state is where RWKV aggregates history, so it is the natural location from which to expose uncertainty-aware belief for RL. Figure 2: RWKV-specific control interface. The belief readout branches from the temporal state produced by RWKV time-mix, then emits (\mu_{t},\Sigma_{t}) for policy and value heads. Channel-mix continues to refine the backbone feature stream, but the actor-critic reads belief rather than the raw hidden state. The same design extends naturally to Eagle/Finch-style RWKV variants with matrix-valued states 18 : the belief readout can be applied to vector summaries of the matrix state, diagonal uncertainty statistics, or low-rank projections when a full covariance representation would be too costly. 3.3 Policy and Value Conditioning We define \pi(a_{t}\mid\mu_{t},\Sigma_{t}),\qquad V(\mu_{t},\Sigma_{t}), (8) and train with a standard actor-critic objective 19 ; 20 : \mathcal{L}=\mathcal{L}_{\text{policy}}+c_{v}\mathcal{L}_{\text{value}}-c_{e}\mathcal{H}(\pi). (9) This formulation is intentionally lightweight: no attention over full history is required at decision time, and no external memory is assumed beyond the RWKV state itself. 3.4 Belief-Conditioned Memory Control The belief readout need not be purely passive. RWKV-style recurrence naturally supports an additional control path in which belief statistics modulate memory retention itself. Let s_{t}^{\text{carry}} and s_{t}^{\text{write}} denote the carry and write components exposed by the temporal update. We can define g_{t}=\sigma\!\left(W_{g}[\mu_{t};\log\Sigma_{t}]\right),\qquad

Read the complete text