BOTS: Batch Bayesian Optimization of Extended Thompson Sampling for Severely Episode-Limited RL Settings
Karine Karine, Susan A. Murphy, Benjamin M. Marlin · 2024
Read BOTS: Batch Bayesian Optimization of Extended Thompson Sampling for Severely Episode-Limited RL Settings by Karine Karine, Susan A. Murphy, Benjamin M. Marlin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00308.txt.
Introduction
A team led by Karine and co-authors adapts a bandit method for settings with very few trials, in a 2024 paper. In health programs that adapt to each person, every trial run costs time or money, so learning must be fast. Classic Thompson sampling picks actions by near-term reward alone, which can miss long-term gains.
Their method adds a learned bias term for each action, tuned between batches of episodes by Bayesian search. That change lets the sampler solve a strictly wider class of decision problems than the classic version. In a simulated adaptive health program, the method earned higher total return than standard sampling. It also needed far fewer episodes than full reinforcement learning methods.
Catalog record
| CREATOR | Karine Karine, Susan A. Murphy, Benjamin M. Marlin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BOTS: Batch Bayesian Optimization of Extended Thompson Sampling for Severely Episode-Limited RL Settings Karine Karine, Susan A. Murphy, Benjamin M. Marlin Abstract In settings where the application of reinforcement learning (RL) requires running real-world trials, including the optimization of adaptive health interventions, the number of episodes available for learning can be severely limited due to cost or time constraints. In this setting, the bias-variance trade-off of contextual bandit methods can be significantly better than that of more complex full RL methods. However, Thompson sampling bandits are limited to selecting actions based on distributions of immediate rewards. In this paper, we extend the linear Thompson sampling bandit to select actions based on a state-action utility function consisting of the Thompson sampler’s estimate of the expected immediate reward combined with an action bias term. We use batch Bayesian optimization over episodes to learn the action bias terms with the goal of maximizing the expected return of the extended Thompson sampler. The proposed approach is able to learn optimal policies for a strictly broader class of Markov decision processes (MDPs) than standard Thompson sampling. Using an adaptive intervention simulation environment that captures key aspects of behavioral dynamics, we show that the proposed method can significantly out-perform standard Thompson sampling in terms of total return, while requiring significantly fewer episodes than standard value function and policy gradient methods. 1 Introduction There is an increasing interest in using reinforcement learning methods (RL) in the healthcare setting, including in mobile health Coronato et al. (2020) ; Yu et al. (2021) ; Liao et al. (2022) . However, the healthcare domain presents a range of challenges for existing RL methods. In mobile health, each RL episode typically corresponds to a human subjects trial involving one or more participants that requires substantial time to carry out (weeks to months) and can incur significant cost. As a result, methods that require many episodes are usually not feasible ( Williams, 1987 ; Mnih et al., 2013 ) . Within the mobile health research community specifically, adaptive intervention policy learning methods have addressed severe episode count restrictions imposed by real-world research constraints by focusing on the use of contextual bandit algorithms ( Tewari and Murphy, 2017 ) . By focusing on maximizing immediate reward, bandit algorithms have the potential to provide an improved bias-variance trade-off compared to policy gradient and state-action value function approaches ( Lattimore and Szepesvari, 2017 ) . Linear Thompson sampling (TS) bandits are a particularly promising approach due to the application of Bayesian inference to capture model uncertainty due to data scarcity in the low episode count setting ( Agrawal and Goyal, 2013 ) . Of course, the main drawback of bandit-like algorithms is that they select actions based on distributions of immediate rewards, thus they do not account for long term consequences of present actions ( Chapelle and Li, 2011 ) . This can lead to sub-optimal performance in real world applications where the environment corresponds to an arbitrary MDP, referred to as the “full RL” setting. In this paper, we propose an approach to extending the linear TS bandit such that actions are selected using a state-action utility function that includes an action bias term learned across episodes using Bayesian Optimization (BO) applied to the expected return of the extended Thompson sampler. This approach retains much of the bias-variance trade-off of the classical TS bandit while having the ability to provide good performance for a strictly larger set of MDPs than the classical TS bandit. Further, we explore the use of batch Bayesian optimization methods, including local methods, that enable multiple episodes to run in parallel while exploring the space of action bias terms in a principled manner. The use of batch BO in our target application domain is critical since real adaptive intervention studies must support multiple simultaneous participants in order to satisfy constraints on total study duration. To improve the BO exploration, we also investigate setting the TS prior parameters using a small micro-randomized trial (MRT). We explore the above issues in the context of a recently proposed physical activity intervention simulation where the reward is in terms of step count and the actions correspond to the selection of different forms of contextualized motivational messages ( Karine et al., 2023 ) . The simulation captures key aspects of the physical activity intervention domain including a habituation process affected by treatment volume and a disengagement process affected by context inference errors. Our results show that optimizing action bias terms using batch BO and selecting actions using the proposed utility function leads to an extended Thompson sampler that outperform standard TS and full RL methods. Using local batch BO instead of global batch BO can further improve the performance. Moreover, modeling the reward variance in addition to the action bias terms can provide further mean performance improvements in some settings. These results suggest that our proposed approach has the potential to enhance the treatment efficacy of TS-based adaptive interventions. Our main contributions are: (1) We introduce a novel extended Thompson sampling bandit model and learning algorithm that can solve a broader class of MDPs than standard Thompson sampling. (2) We show that the proposed extended Thompson sampler outperforms a range of RL baseline methods in the severely episode-limited setting. (3) We provide an implementation of the proposed approach. The code is available at: github.com/reml-lab/BOTS . 2 Methods In this section, we describe our extension to Thompson sampling, our approach to optimizing the additional free parameters introduced, and the JITAI simulation environment. We describe the related work in Appendix A.1 . Extended Thompson Sampling (xTS). Our primary goal is to reduce the myopic nature of standard TS for contextual bandits when applied in the episodic MDP setting while maintaining low variance. Our proposed extended Thompson sampling approach is based on selecting actions according to a state-action utility that extends the linear Gaussian reward model used by TS as shown below. \displaystyle u_{ta} \displaystyle=r_{ta}+\beta_{a} (1) \displaystyle p(r_{ta}|a,\mathbf{s}_{t}) \displaystyle=\mathcal{N}(r_{ta};\theta_{ta}^{\top}\mathbf{s}_{t},\sigma_{Ya}^{2}) (2) \displaystyle p(\theta_{ta}|\mu_{ta},\Sigma_{ta}) \displaystyle=\mathcal{N}(\theta_{ta};\mu_{ta},\Sigma_{ta}) (3) where at each time t , \mathbf{s}_{t} is the state (or context) vector, r_{ta} is the reward for taking action a , \theta_{ta} is a vector of weights for action a . We refer to the additional term \beta_{a} included in the utility as the action bias for action a . The action bias values are fixed within each episode of xTS, but are optimized across episodes to maximize the total return of the resulting policy \pi_{\textit{xTS}} . The policy \pi_{\textit{xTS}} is similar in form to that of standard TS except that actions are selected according to the probability that they have the highest expected utility. The policy \pi_{\textit{xTS}} maintains the same posterior distribution over immediate rewards used by standard TS. We provide the pseudo code in Algorithm 2 in Appendix A.2.3 . The basic intuition for the xTS model is that the action bias parameters enable penalizing or promoting the selection of each action a (regardless of the current state), based on action a ’s average long term consequences effect on the total return. The inclusion of action bias parameters in the policy \pi_{\textit{xTS}} provides access to a strictly larger policy space than standard TS. At the same time, the number of additional