Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions
Jingyang You, Hanna Kurniawati · 2025
Read Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions by Jingyang You, Hanna Kurniawati (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20974.txt.
Introduction
Jingyang You and Hanna Kurniawati remove the approximation error from a popular way of teaching agents to adapt. Meta-reinforcement learning infers a hidden model of each new task, but most deep Bayesian approaches lean on variational posteriors, whose looseness distorts the task representation. Their framework, GLiBRL, instead models tasks with generalized linear models over learnable nonlinear basis functions, a pairing chosen so the Bayesian updates stay exact.
Posterior updates arrive in closed form, sequentially, and identically regardless of the order experiences arrive in, and the method plugs into both off-policy and on-policy algorithms. A learned task representation obeys an exact kernel identity linking distances between tasks to kernel discrepancies. Against eight rival methods, GLiBRL posts the best aggregate zero-shot scores on the MuJoCo locomotion and MetaWorld manipulation benchmarks.
Catalog record
| CREATOR | Jingyang You, Hanna Kurniawati |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20974
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalised Linear Models in Deep Bayesian RL with Learnable Basis Functions Jingyang You, Hanna Kurniawati Abstract Bayesian Reinforcement Learning (BRL) provides a framework for generalisation of Reinforcement Learning (RL) problems from its use of Bayesian task parameters in the transition and reward models. However, classical BRL methods assume known forms of transition and reward models, reducing their applicability in real-world problems. As a result, recent deep BRL methods have started to incorporate model learning, though the use of neural networks directly on the joint data and task parameters requires optimising the Evidence Lower Bound (ELBO). ELBOs are difficult to optimise and may result in indistinctive task parameters, hence compromised BRL policies. To this end, we introduce a novel deep BRL method, G eneralised Li near Models in Deep B ayesian RL with Learnable Basis Functions ( GLiBRL ), that enables efficient and accurate learning of transition and reward models, with fully tractable marginal likelihood and Bayesian inference on task parameters and model noises. On challenging MetaWorld ML10/45 benchmarks, GLiBRL improves the success rate of one of the state-of-the-art deep BRL methods, VariBAD, by up to 2.7 × 2.7\times . Comparing against representative or recent deep BRL / Meta-RL methods, such as MAML, RL 2 , SDVT, TrMRL and ECET, GLiBRL also demonstrates its low-variance and decent performance consistently. 1 Introduction Reinforcement Learning (RL) algorithms have great potentials to enable robots to act intelligently without supervisions from humans. A well-known issue with RL algorithms is they generalise poorly to unseen tasks. Most standard RL algorithms by their designs do not consider possible variations in the transition and reward models, hence fail to adapt to new tasks whose models might be different from that of training environments. Bayesian Reinforcement Learning (BRL) is an effective framework that can be used to improve the generalisation of RL. Instead of ignoring possible variations in transition and reward models, BRL methods explicitly take them into considerations by assuming parametric distributions of models and performing Bayesian inference on the parameters ( Ghavamzadeh et al. 2015 ) . Different parameters indicate different transition and reward models, hence implicitly encode various tasks. To solve BRL problems, many previous works use planners ( Poupart et al. 2006 ; Guez et al. 2013 ) that search for Bayes-optimal policies. These methods are often limited in their scalability. Moreover, they require full information about the forms of transition and reward models, which restricts generalisation across different tasks. Hence, recent deep BRL methods ( Rakelly et al. 2019 ; Zintgraf et al. 2021 ) enable model learning by optimising the marginal likelihood of the data. However, most of the deep BRL methods do not support tractable Bayesian inference on the task parameters, because of the direct use of neural networks on the joint data and parameters. As a result, the exact marginal likelihood of the data is also not tractable and cannot be optimised directly. To this end, deep BRL methods adopt variational inference to optimise the evidence lower bound (ELBO) instead. However, the optimisation of ELBO is not an easy task as it may face issues such as high-variance Monte Carlo estimates, amortisation gaps ( Cremer et al. 2018 ) and posterior collapse ( Bowman et al. 2016 ; Dai et al. 2020 ) . Such issues can preclude BRL methods from obtaining meaningful and distinctive distributions of task parameters, which are crucial to smooth Bayesian learning. To alleviate the above issues, we propose G eneralised Li near Models in Deep B ayesian RL with Learnable Basis Functions ( GLiBRL ). GLiBRL assumes a generalised linear relation between task parameters and features of the data, computed via learnable basis functions. Such modelling allows exact posterior inference and marginal likelihood to be computed, avoiding optimising the challenging ELBO objective. Furthermore, GLiBRL generalises previous works such as ( Harrison et al. 2018b ) in that GLiBRL also performs inference on the noise, reducing the error of predictions in unseen tasks. The performance of GLiBRL is evaluated with the challenging MetaWorld ( Yu et al. 2021 ; McLean et al. 2025 ) benchmark and compared with standard baselines and recent methods, including deep BRL methods, VariBAD ( Zintgraf et al. 2021 ) and SDVT ( Lee et al. 2023 ) , and Meta-Reinforcement Learning ( Beck et al. 2023b ) methods, including RL 2 ( Wang et al. 2016 ; Duan et al. 2016 ) , MAML ( Finn et al. 2017 ) , TrMRL ( Melo 2022 ) and ECET ( Shala et al. 2025 ) . GLiBRL has improved the success rate of one of the state-of-the-art deep BRL methods, VariBAD, by up to 2.7\times , and has consistently outperformed all listed methods and demonstrated low-variance behaviours in the most complex subset of MetaWorld, ML45. 2 Background 2.1 Markov Decision Processes and Reinforcement Learning Markov Decision Processes (MDPs) are 5-tuples defined as \mathcal{M}=\left(\mathcal{S},\mathcal{A},R,T,\gamma\right) , where \mathcal{S} is the set of states ; \mathcal{A} , the set of actions ; R(s_{t},a_{t},s_{t+1},r_{t+1})=p(r_{t+1}|s_{t},a_{t},s_{t+1}) , the reward function; T(s_{t},a_{t},s_{t+1})=p(s_{t+1}|s_{t},a_{t}) , the transition function; and \gamma\in(0,1] a discount factor. In the above definition, s_{t}\in\mathcal{S},s_{t+1}\in\mathcal{S},a_{t}\in\mathcal{A} and r_{t+1}\in\mathbb{R} . The goal of solving an MDP is to find a policy \pi(a_{t}|s_{t}):\mathcal{S}\rightarrow\mathcal{A} , such that the expected return for a finite horizon H>0 , \mathcal{J}(\pi)=\mathbb{E}_{T,\pi,R}[\sum_{t=0}^{H}\gamma^{t}R(s_{t},a_{t},s_{t+1},r_{t+1})] , is maximised. While MDPs assume fully known transition and reward functions T and R , standard Reinforcement Learning (RL) problems optimise the same objective, however, with at least one of the T and R unknown but can be learnt from data samples. 2.2 Bayes-Adaptive MDPs and Bayesian Reinforcement Learning Bayes-Adaptive MDPs (BAMDPs) ( Duff 2002 ; Ghavamzadeh et al. 2015 ) is a Bayesian framework for solving RL. Compared to standard MDPs, BAMDPs assume known forms of functions of T and / or R , but parameterised by unknown parameters \theta_{T}\in\Theta_{T} and / or \theta_{R}\in\Theta_{R} . In BAMDPs, distributions (or, beliefs ) b_{t}=p({\theta_{T}}_{,t},{\theta_{R}}_{,t})\in\mathcal{B}_{T}\times\mathcal{B}_{R} are placed on the unknown parameters, and updated to posteriors b_{t+1}=p({\theta_{T}}_{,t+1},{\theta_{R}}_{,t+1}) with Bayesian inference. To efficiently use existing MDP frameworks, the beliefs can be absorbed into the original state space to form hyper-states \mathcal{S}^{+}=\mathcal{S}\times\mathcal{B}_{T}\times\mathcal{B}_{R} . Hence, BAMDPs can be defined as 5-tuple (\mathcal{S}^{+},\mathcal{A},R^{+},T^{+},\gamma) MDPs, where \displaystyle\begin{split}T^{+}(s^{+}_{t},a_{t},s_{t+1}^{+})&=p(s_{t+1},b_{t+1}|s_{t},b_{t},a_{t})\\ &=\mathbb{E}_{{\theta_{T}}_{,t}\sim b_{t}}\left[p(s_{t+1}|s_{t},a_{t},{\theta_{T}}_{,t})\right]\cdot\delta(b_{t+1}=p({\theta_{T}}_{,t+1},{\theta_{R}}_{,t+1}))\end{split} (1) \displaystyle\begin{split}R^{+}(s^{+}_{t},a_{t},s_{t+1}^{+},r_{t+1})&=p(r_{t+1}|s_{t},b_{t},a_{t},s_{t+1},b_{t+1})\\ &=\mathbb{E}_{{\theta_{R}}_{,t+1}\sim b_{t+1}}\left[p(r_{t+1}|s_{t},a_{t},s_{t+1},{\theta_{R}}_{,t+1})\right]\end{split} (2) The hyper-transition function (Equation 1 ) consists of the \theta_{T} -parameterised expected regular MDP transition and a deterministic posterior update specified by the Dirac delta function \delta(b_{t+1}=p({\theta_{T}}_{,t+1},{\theta_{R}}_{,t+1})) . The hyper-reward function consists of the \theta_{R} -parameterised expected regular MDP reward function. Accordingly, the expected return to maximise becomes J^{+}(\pi^{+})=\mathbb{E}_{T^{+},\pi^{+},R^{+}}[\sum_{t=0}^{H^{+}}\gamma^{t}R^{+}(s^{+}_{t},a_{t},s^{+}_{t+1},r_{t+1})] , where H^{+}>0 is the BAMDP horizon, and \pi^{+}:\mathcal{S}^{+}\rightarrow\mathcal{A} is the policy of BAMDPs. Traditionally, problems that require solving BAMDPs are named Bayesian Reinforcement Learning (BRL). Aside from their generalisability, BRL methods are also recognised for offering principled approaches to the exploration–exploitation problem in RL ( Ghavamzadeh et al. 2015 ) . However, classical BRL methods ( Poupart et al. 2006 ; Guez et al. 2013 ; Tziortziotis et al. 2013 ) assume that the forms of transition T^{+} and reward R^{+} models are fully known, despite being parameterised by unknown parameters. These methods are not sufficiently flexible in scenarios where the forms of T^{+} and R^{+} are not known a priori. A rough guess of the forms, however, may