Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games

Ke Yan The University of Edinburgh [email protected] · 2024

Read Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games by Ke Yan The University of Edinburgh [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00725.txt.

Introduction

Ke Yan wrote this 2024 study comparing two sequence models that learn to play Atari games from past play records. One model, based on the Transformer design, is set against a newer one based on the Mamba design. Early tests showed a split: the Mamba-based model won at Breakout and Qbert, while the Transformer-based model won at harder games such as Hero and Kung Fu Master.

Yan widened the test to twelve games and measured traits such as the size of the action set, the visual load, and the wait for the first reward. The gap between the models traced back mostly to action set size and visual load. The Mamba-based model did best where actions and screens were simple, and the other model did best where both were rich. The findings can guide which design to pick for a given task.

Catalog record

CREATORKe Yan The University of Edinburgh [email protected]
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00725

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games Ke Yan The University of Edinburgh [email protected] Abstract This work analyses the disparity in performance between Decision Transformer (DT) and Decision Mamba (DM) in sequence modelling reinforcement learning tasks for different Atari games. The study first observed that DM generally outperformed DT in the games Breakout and Qbert, while DT performed better in more complicated games, such as Hero and Kung Fu Master. To understand these differences, we expanded the number of games to 12 and performed a comprehensive analysis of game characteristics, including action space complexity, visual complexity, average trajectory length, and average steps to the first non-zero reward. In order to further analyse the key factors that impact the disparity in performance between DT and DM, we employ various approaches, including quantifying visual complexity, random forest regression, correlation analysis, and action space simplification strategies. The results indicate that the performance gap between DT and DM is affected by the complex interaction of multiple factors, with the complexity of the action space and visual complexity (particularly evaluated by compression ratio) being the primary determining factors. DM performs well in environments with simple action and visual elements, while DT shows an advantage in games with higher action and visual complexity. Our findings contribute to a deeper understanding of how the game characteristics affect the performance difference in sequential modelling reinforcement learning, potentially guiding the development of future model design and applications for diverse and complex environments. † † footnotetext: Project code available: Github 1 Introduction The field of reinforcement learning (RL) has seen significant advancements in recent years, particularly in the domain of sequential decision-making tasks. One notable approach has emerged as a powerful tool for these challenges: the Decision Transformer (DT) [ 1 ] . Recently, Ota introduced Decision Mamba (DM), which replaces DT’s causal self-attention blocks with the new Mamba architecture [ 2 , 3 ] . Both DT and DM have shown promise in various RL tasks. However, to the best of our knowledge, a comprehensive analysis of their relative performance across different environments remains an open question. This study aims to fill this gap by thoroughly comparing DT and DM across a wide range of Atari games. Our investigation begins with an initial observation: while DM outperforms DT in some games, such as Breakout, DT demonstrates superior performance in others, notably Hero and KungFuMaster. This gap in performance across different games raises questions about the factors that influence the effectiveness of these models. We seek to understand whether certain architectural features of DT and DM are better suited to specific types of games, and how various game characteristics impact their relative performance. Based on previous observations, we perform a sequence of analyses that integrate empirical evaluation with an in-depth study of game characteristics. This investigation begins with a comprehensive survey of various game characteristics, including action-space complexity and visual complexity. To quantify the visual complexity, we introduce three metrics: image entropy, compression ratio, and feature count. These metrics, along with other game characteristics such as the number of actions and average trajectory length, form the basis for detailed analysis using random forest regression and correlation studies. The analysis results show that the action space complexity is indeed an important factor in determining the performance of the model, but at the same time, the visual complexity also plays a crucial role. Finally, to isolate the effect of action space complexity and explore further, two action space simplification strategies are implemented and evaluated: simple action fusion and frequency-based action fusion. These studies provide more insight into the influence of action space complexity and expose the limitations of simplification strategies in maintaining game dynamics. The key contributions of this study are: 1. A thorough comparison of Decision Transformer (DT) and Decision Mamba (DM) in a variety of Atari games, providing insights into their relative strengths and weaknesses. 2. The identification and analysis of key game characteristics that influence model performance, with a focus on visual and structural elements of the games. 3. A quantitative assessment of the relative importance of various game characteristics in determining the performance gap between DT and DM, using Random Forest regression and correlation analysis. 4. The development and evaluation of novel strategies to simplify the action space in complex games. Our findings reveal that while action space complexity is indeed a significant factor in determining model performance, visual complexity, particularly as measured by compression ratio, also plays a crucial role. We observe that DM tends to excel in visually simpler environments, while DT shows advantages in games with higher visual complexity. These insights not only contribute to our understanding of the strengths and limitations of DT and DM but also have implications for the design and application of sequence modelling approaches in RL. The rest of the dissertation is structured as follows: Section 2 provides background information on Transformer, State Space Models, basic concepts of Reinforcement Learning, and related work on applying these sequence modelling architectures to reinforcement learning tasks. Section 3 presents our approach in detail, including dataset preparation, model architecture, and experimental setup. Section 4 focuses on our experiments and analysis. It begins with a basic comparison and extended measurements of the game’s action space complexity and visual complexity. This is followed by the application of random forest, correlation analysis, and action fusion to further explore the effect of action complexity. Section 5 discusses potential directions for future work, and Section 6 summarizes the findings of this paper. Through these analyses, this study aims to provide valuable insights into the factors that affect the performance of sequence modelling in reinforcement learning tasks, possibly guiding future development of model design and applications in more complex environments. 2 Background 2.1 Transformer The Transformer introduced by Vaswani et al. revolutionized the field of sequence modelling [ 4 ] . The architecture consists of an encoder and decoder, each of which contains multiple layers of self-attention and fully connected neural networks, with a cross-attention connecting the encoder and decoder [ 4 ] . Compared to Recurrent Neural Networks (RNNS), Transformer can process the entire sequence simultaneously, and without suffering long-range dependency problems. This is achieved through the self-attention mechanism. It allows the model to weigh the importance of different parts of the input sequence, captures complex relationships inside the data, and can be computed in parallel on modern GPUs. The Transformer show better performance on various tasks while significantly reducing training time [ 4 , 5 ] . However, they may encounter difficulties when dealing with very long sequences during inference due to their quadratic computational complexity, which leads to a large video RAM cost. [ 3 ] . 2.2 State Space Models State Space Models (SSMs) have emerged as a promising alternative for long sequential modelling tasks. An SSM, or state space model, utilises a mathematical framework to convert a one-dimensional input into an n-dimensional latent state, which is then projected back to a one-dimensional output [ 3 , 6 ] . The process can be expressed by the following equations: \displaystyle h^{\prime}(t) \displaystyle=Ah(t)+Bx(t) (1) \displaystyle y(t) \displaystyle=Ch(t) (2) where h(t) represents the N -D latent state, x(t) is the

Read the complete text