True Online TD-Replan( λ \lambda ) Achieving Planning through Replaying
Abdulrahman Altahhan · 2025
Read True Online TD-Replan( λ \lambda ) Achieving Planning through Replaying by Abdulrahman Altahhan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19027.txt.
Introduction
Abdulrahman Altahhan wrote this 2025 paper on a learning method that lets an agent replay its own past. In reinforcement learning, an agent improves by updating its value guesses step by step. The true online TD method does those updates in a clean, efficient way, but it cannot revisit old experience the way planning methods do.
The extension by Altahhan, True Online TD-Replan, replays past steps in their original order, either at every step or sparsely under the same lambda dial that sets update depth. On problems where replay helps, the method beats plain true online TD, though it costs quadratic time. It also beat rival methods of similar cost, including Dyna planning. Tests span a random walk task and a muscle-signal control task.
Catalog record
| CREATOR | Abdulrahman Altahhan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19027
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
True Online TD-Replan( λ \lambda ) Achieving Planning through Replaying Abdulrahman Altahhan Abstract In this paper, we develop a new planning method that extends the capabilities of the true online TD to allow an agent to efficiently replay all or part of its past experience, online in the sequence that they appear with, either in each step or sparsely according to the usual λ \lambda parameter. In this new method that we call True Online TD-Replan( λ \lambda ), the λ \lambda parameter plays a new role in specifying the density of the replay process in addition to the usual role of specifying the depth of the target’s updates. We demonstrate that, for problems that benefit from experience replay, our new method outperforms true online TD( λ \lambda ), albeit quadratic in complexity due to its replay capabilities. In addition, we demonstrate that our method outperforms other methods with similar quadratic complexity such as Dyna Planning and TD(0)-Replan algorithms. We test our method on two benchmarking environments, a random walk problem that uses simple binary features and on a myoelectric control domain that uses both simple sEMG features and deeply extracted features to showcase its capabilities. I Introduction Experience replay plays an important role in the context of reinforcement learning algorithms. In this paper we tackle the issue of building a robust method that allows the agent to maximize its experience replay capability with relatively cheap complexity. We will tackle multi-step sequential replay algorithm where the agent replays a sequence of past experience steps in the order they appeared with. This issue have been partially attempted in [ 1 ] where the algorithm used TD(0) update rules as its basis. In this work, we will extend the ideas developed in [ 1 ] to the true online TD( \lambda ) updates. In particular, we will build a new method based on three requirements. First, we would like to be able to utilise a multi-step targets for each replay update instead of the one step target update of TD(0), this allows the method to choose how deep its targets are going to be for each replay update. The second requirement, is that we want to allow the algorithm designer to choose how much of the past experience updates he/she needs to incorporate and how frequently this replay process will be performed. Thirdly, regardless of the replay depth and the target’s depths, the method should be efficient even on the limit when the replay depth is maximal. To achieve these goals we first introduce a method, namely online \lambda -return TD-Replan, that takes experience replay to its extreme by allowing the agent to replay all of its past experience online in every time step. Unlike previous work, this method allows us to utilize the multi-step interim \lambda -return targets for each replay update instead of the one-step target of TD( 0 ). We show how to deduce an online efficient incremental method, namely TD( \lambda )-Replan, that is equivalent to online \lambda -return TD-Replan, but has a complexity that is not related to the time step, we prove the equivalency mathematically in Theorem 1. We show then that the true online TD algorithm becomes a special case of the true online TD-Replan algorithm in Theorem 2. We then soften the replay maximal replay requirement by utilizing a parameterization that interpolates between no-replay and replay-all, effectively allowing the designer to choose the online replay depth similar to how we choose the depth of \lambda -return in the true online TD( \lambda ). Finally, the target’s depth and the replay process depth of the resultant TD( \lambda )-Replan( \acute{\lambda} ) method, can be aligned to formulate TD-Replan( \lambda ). Reinforcement learning deals with Markov Decision Process (\mathcal{S,A},p,r,\gamma) where we have a set of discrete actions \mathcal{A} that are available for the agent to pick from at each state s . We denote the feature representation of state s as {\boldsymbol{\phi}}(s) and the number of features |{\boldsymbol{\phi}}|=n . r(s,a,\acute{s}) is the expected reward signal for taking action a at state s then moving to state \acute{s} . The feature representation can be complex. For example, the features can be obtained from a deep architecture such as from the encoder of an auto-encoder [ 2 ] or a set of stacked auto encoders [ 3 ] . We can then employ whatever algorithm we have on the resultant features [ 4 ] . In this case each learning stage is dissected from the other stages. In this paper we take this approach due to its simplicity and theoretical guarantees. Alternatively, one can take a more integrated end-to-end learning approach. An example of such integrated approach is to build a deep feature extraction model and append a fully connected layer with linear activation function, we then can backpropagate the error coming out of the last layer further into previous layers [ 5 ] . However, despite the impressive empirical achievement of such models convergence guarantees do not apply straight on a non-linear model [ 4 ] . The high performance in the second approach can mainly be attributed to the stability and agility provided by the replay process [ 6 ] . By building the targets in the method itself we conjecture that we can avoid having to use a separate network to represents the targets as in [ 5 ] . The better performance of an intermediate replay depth value can be attributed to the ability to balance the importance of sequence of events and to break the strong correlation that can lead to instability when non-linear function approximation is used. In this work we build an algorithm that is particularly suitable for deep learning whether the learning process is dissected or integrated. Throughout our presentation we will assume that the behaviour policy is derived directly from the learning weights for action-values and we do not consider policy gradient methods. Our focus will be on multi-steps policy evaluation methods that involves looking back at multiple updates for steps that happened in the past and repeat their respective updates, i.e. replay them. II Bundled Experience Replay Replay can be categorized as sequential with specific frequency (ex. replaying past sequence of 10 steps every other step), which is the topic of this paper, and non-sequential, for example the one used in [ 7 ] . When sequential replay is incorporated in the method we can systematically reduce its complexity to be related to the number of used features, whether a deep or a shallow model is used, rather than in the number of time steps of an experience. More recently, the replay process has been taken from a different perspective through the work of [ 8 ] to mean re-evaluating past target updates using what has been recently learned by the agent, they tagged it as replay. In this sense their algorithms redo the same set of past updates, with the same initial learning weights in each step, using updated targets, in order to benefit from past experience. In their setting each bundle of updates always starts from the same initial weight’s values, which is a key issue. Although this simplifies finding incremental forms for the learning process,