Portfolio Beam Search: Diverse Transformer Decoding for Offline Reinforcement Learning Using Financial Algorithmic Approaches
Dan Elbaz, Oren Salzman · 2025
Read Portfolio Beam Search: Diverse Transformer Decoding for Offline Reinforcement Learning Using Financial Algorithmic Approaches by Dan Elbaz, Oren Salzman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10473.txt.
Introduction
Elbaz and Salzman (2025) propose Portfolio Beam Search, a decoding method for transformer models used in offline reinforcement learning. Offline methods learn from a fixed dataset and then act in the world, where the model may meet state and action sequences the data never covered. The standard beam search decoder ignores that uncertainty, and its greedy left-to-right habit produces near-identical candidate plans, missing better options.
Borrowing from finance, the authors build an uncertainty-aware diversification step into the search, balancing exploration against exploitation while the sequence is decoded. On a standard locomotion benchmark, their method earned higher returns than beam search and cut the variability of outcomes sharply. They present it as a simple, drop-in change that makes transformer policies safer and more reliable when they step beyond their training data.
Catalog record
| CREATOR | Dan Elbaz, Oren Salzman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-10473
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Portfolio Beam Search: Diverse Transformer Decoding for Offline Reinforcement Learning Using Financial Algorithmic Approaches Dan Elbaz, Oren Salzman Abstract Offline Reinforcement Learning (RL) algorithms learn a policy using a fixed training dataset, which is then deployed online to interact with the environment. Transformers, a standard choice for modeling time-series data, are gaining popularity in offline RL. In this context, Beam Search ( BS ), an approximate inference algorithm, is the go-to decoding method. In offline RL, the restricted dataset induces uncertainty as the agent may encounter unfamiliar sequences of states and actions during execution that were not covered in the training data. In this context, BS lacks two important properties essential for offline RL: It does not account for the aforementioned uncertainty, and its greedy left-right search approach often results in sequences with minimal variations, failing to explore potentially better alternatives. To address these limitations, we propose Portfolio Beam Search ( PBS ), a simple-yet-effective alternative to BS that balances exploration and exploitation within a Transformer model during decoding. We draw inspiration from financial economics and apply these principles to develop an uncertainty-aware diversification mechanism, which we integrate into a sequential decoding algorithm at inference time. We empirically demonstrate the effectiveness of PBS on the D4RL locomotion benchmark, where it achieves higher returns and significantly reduces outcome variability. 1 Introduction Reinforcement Learning (RL) is concerned with an agent learning how to take actions in an environment to maximize the total reward it obtains. An RL agent typically learns by trial and error, involving online interaction with the environment Sutton and Barto (1998) . However, such online interaction may incur undesirable costs, time or risks. This is relevant to many fields such as healthcare, autonomous driving, and recommendation systems Gottesman et al. (2018) ; Wang et al. (2018) ; Yu et al. (2019) ; Sallab et al. (2017) ; Strehl et al. (2010) ; Covington et al. (2016) . In contrast, offline RL allows agents to avoid undesirable online interactions by using pre-collected data (batch) from other agents or human demonstrations. While this approach mitigates risks associated with online exploration, it also introduces significant challenges. In offline RL, there are no guarantees regarding the quality of the batch data, as it may have been collected by an agent following a suboptimal policy. Therefore, the goal for the learned policy is to outperform the policy that generated the data. Consequently, it is necessary to execute a different sequence of actions than those stored in the batch, leading to encounters with unfamiliar state-action sequences. This situation introduces subjective or epistemic uncertainty—uncertainty arising from a lack of knowledge or statistical evidence due to the limited size of the training batch or due to its quality. This uncertainty can result in erroneous value estimations, increasing variability in the results, which decreases stability and reliability. This phenomenon, known as distributional shift Kumar et al. (2020a) , is one of the central challenges in offline RL. Offline RL with Transformers Transformers are a deep-learning architecture originally proposed for natural language processing (NLP) tasks Vaswani et al. (2017) . They have demonstrated powerful expressive capabilities, leading to their adoption in various domains where processing sequential information is crucial Radford et al. (2023) ; Liu et al. (2023) . Recently, Transformer models have been successfully applied to various RL benchmarks, such as language instructions for vision tasks Yao et al. (2023) , the Atari games suite Reed et al. (2022) , and continuous control locomotion tasks Wu et al. (2024) . While earlier studies employed traditional RL algorithms, merely substituting recurrent networks with Transformer architecture Parisotto and Salakhutdinov (2021) ; Kumar et al. (2020d) , our work builds on a more recent approach wherein Transformers are used as trajectory models. Pioneered by Decision Transformer ( DT ) Chen et al. (2021) and Trajectory Transformer ( TT ) Janner et al. (2021) , these techniques utilize Transformers to learn dynamics and reward structures, framing offline RL as a conditional sequence-modeling problem. With trajectory models at our disposal, we can simulate potential future scenarios to formulate plans, a principle known as model-based planning Sutton and Barto (1998) . Our work targets real-world applications, particularly locomotion problems involving high-dimensional continuous domains. As a concrete example, consider the task of bipedal locomotion. This problem supports multiple solutions, as it can encompass various motion styles using different postures, stride patterns, or gaits. Such problems pose significant challenges for offline RL, as they are often sensitive to even minor variations, especially when such variations are not explicitly included in the offline data batch. Moreover, in such problems, the vast number of possible trajectories makes exhaustive model exploration impractical. In fact, similar to NLP tasks, the number of possible sequences grows exponentially with sequence length, making exact inference NP-hard. Thus, designing algorithms that can efficiently explore these overwhelmingly large domains remains a significant challenge. As finding an exact solution to these sequential-decision problems is typically intractable, approximate inference algorithms such as Beam Search ( BS ) are typically used. BS is a heuristic-search algorithm that explores the search space in a greedy left-right fashion retaining only the top b scoring candidates sequences. In offline RL, BS is repurposed as a planning or decoding algorithm that integrates reward signals, evaluating candidate trajectories based on their cumulative return. Despite the success of this approach in some offline RL tasks, naively modifying BS to to consider only the maximum predicted returns has several disadvantages that may be detremental in certain applications. Here, we highlight the two most critical ones: (i) Ignoring the Distributional Shift : When used as a planning algorithm, BS is not designed to be robust against the distributional shift typical in offline RL. Pursuing the maximal reward trajectory without accounting for uncertainty can result in suboptimal outcomes (ii) Insufficient Diversity : Greedy approaches, such as BS , tend to follow the highest rewarding trajectories, which can result in limited exploration and a lack of diversity in decoded solutions. In locomotion problems, identifying various motion modalities, including suboptimal ones, can be advantageous. While a cluster of similar motions might fail if the predicted trajectories prove infeasible, a diverse portfolio of behaviors can adapt to new situations by employing alternative strategies. Overview and Contributions To address the shortcomings of BS when decoding Transformers for offline RL, we propose Portfolio Beam Search ( PBS ), an uncertainty-aware strategy to decode a set of diverse sequences. Our decoding approach is inspired by the finance sector, where investors demand a reward for bearing risk when making investment decisions. This approach was formalized by Harry Markowitz [ 1952 ] who introduced this risk-expected return relationship in the mean-variance model, for which he was awarded a Nobel Prize in economics. The mean-variance model addresses the portfolio-optimization problem (formally defined in Sec. 2.3 ) by identifying the optimal allocation of wealth across different assets to achieve a diversified portfolio by taking into account both the mean (expected return) and the variance (risk). Building on portfolio theory, our primary technical contribution is the application of these principles to develop PBS , a Transformer decoding algorithm for offline RL. PBS mitigates the adverse effects of the distributional-shift by balancing the desire to maximize rewards with the