N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs

Ilya Zisman, Alexander Nikulin, Viacheslav Sinii, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov · 2024

Read N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs by Ilya Zisman, Alexander Nikulin, Viacheslav Sinii, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01958.txt.

Description

Abstract In-context learning allows models like transformers to adapt to new tasks from a few examples without updating their weights, a desirable trait for reinforcement learning (RL). However, existing in-context RL methods, such as Algorithm Distillation (AD), demand large, carefully curated datasets and can be unstable and costly to train due to the transient nature of in-context learning abilities.

Catalog record

CREATORIlya Zisman, Alexander Nikulin, Viacheslav Sinii, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs Ilya Zisman, Alexander Nikulin, Viacheslav Sinii, Denis Tarasov, Nikita Lyubaykin, Andrei Polubarov, Igor Kiselev, Vladislav Kurenkov Abstract In-context learning allows models like transformers to adapt to new tasks from a few examples without updating their weights, a desirable trait for reinforcement learning (RL). However, existing in-context RL methods, such as Algorithm Distillation (AD), demand large, carefully curated datasets and can be unstable and costly to train due to the transient nature of in-context learning abilities. In this work, we integrated the n-gram induction heads into transformers for in-context RL. By incorporating these n-gram attention patterns, we considerably reduced the amount of data required for generalization and eased the training process by making models less sensitive to hyperparameters. Our approach matches, and in some cases surpasses, the performance of AD in both grid-world and pixel-based environments, suggesting that n-gram induction heads could improve the efficiency of in-context RL. 1 Introduction In-context learning is a powerful ability of pretrained autoregressive models, such as transformers ( Vaswani et al. 2023 ) or state-space models ( Gu et al. 2022 ) . In contrast to fine-tuning, in-context learning is able to effectively solve downstream tasks on inference without explicitly updating the weight of a model, making it a versatile tool for solving wide range of tasks ( Agarwal et al. 2024 ) . Originated in the language domain ( Brown et al. 2020 ) , the in-context ability has quickly found its applications in Reinforcement Learning (RL) for building agents that can adaptively react to the changes in the dynamics of the environment. This trait allows researchers to use In-Context Reinforcement Learning (ICRL) as a backbone for the embodied agents ( Elawady et al. 2024 ) or to benefit from its adaptation abilities for domain recognition in order to build generalist agents ( Grigsby et al. 2024 ) . In-context reinforcement learning methods that learn from offline datasets were first introduced by Laskin et al. 2022 and Lee et al. 2023 . In the former work, Algorithm Distillation (AD), authors propose to distill the policy improvement operator from a collection of learning histories of RL algorithms. After pretraining a transformer on these learning histories, an agent is able to generalize to unseen tasks entirely in-context. In the latter approach, the authors show it is possible to train adaptive models on datasets that contain interactions collected with expert policies, provided that the optimal actions for each state are also available. Figure 1: Performance comparison for different number of training goals between our method and Algorithm Distillation (AD), an in-context reinforcement learning method ( Laskin et al. 2022 ) . Our method demonstrates similar performance with less training goals (128 vs. 512) and in general outperforms the baseline. See Section 4 for results. Both methods require specifically curated data, which can be demanding to obtain ( Nikulin et al. 2024b ) . In addition, the in-context ability itself is transient ( Singh et al. 2024 ) and it is difficult to predict its emergence from cross-entropy loss alone ( Agarwal et al. 2024 ) , making the training of such models unstable and expensive in terms of training budget. Our work takes initial steps toward addressing these challenges by introducing modifications to the transformer’s attention heads, which can accelerate training and reduce the amount of data required for in-context learning to emerge. Induction heads have been shown to be a central mechanism that allows in-context learning in transformers ( Olsson et al. 2022 ) . Edelman et al. 2024 studied the emergence of these statistical induction heads on synthetic data and concluded that transformers obtain a simplicity bias towards plain uni-grams. Akyürek et al. 2024 take a step forward in this direction, demonstrating that in the in-context learning setting, the attention mechanism develops higher-order induction heads. These heads are responsible for recognizing and capturing n-grams within a sequence. Authors propose to hardcode this mechanism into a transformer, creating an n-gram layer which is used as a drop-in replacement for the multi-head attention mechanism. Intuitively, a transformer benefits from it by not learning this complicated behavior by itself; rather, it straightforwardly receives an inductive bias that n-gram heads provide. This approach significantly decreases perplexity even when applied to recurrent sequential models, indicating that n-grams play a major role in building effective in-context learning models. In our work, we propose integrating an n-gram induction head into the ICRL model. As we demonstrate, these heads can improve model performance in low-data settings and reduce hyperparameter sensitivity while introducing only a few additional hyperparameters that are straightforward to optimize. We provide experimental evidence on Dark Room, Key-to-Door and Miniworld environments, covering both discrete and visual observation spaces. To summarize our main contributions, in this paper we show that N-Gram attention heads : • Decrease the amount of data needed for generalization on novel tasks. By utilizing n-gram heads, it is possible to reduce the total number of transitions in training data by a maximum of 27x compared to the original method of Laskin et al. 2022 . The results are presented in Section 4.1 . • Help mitigate hyperparameter sensitivity in ICRL models, contributing to more stable training. By employing n-gram heads, one may need less time searching for a good set of hyperparameters. The results are presented in Section 4.2 . • Can be used in the environments with visual observations. However n-grams are originally found in discrete structures (e.g. natural language texts), we show it is possible to detect repeating patterns in the sequences of images. The details of the implementation are presented in Section 2.4 and the results of the experiment are shown in Section 4.3 . 2 Method 2.1 Algorithm Distillation We build our method on Algorithm Distillation ( Laskin et al. 2022 ) and use it as our baseline. It is an in-context reinforcement learning algorithm that distills the policy improvement operator by training a transformer model on specifically acquired data. As training data, the authors propose to use the learning histories of many RL algorithms that are trained to solve different tasks in the multi-task environment. After pretraining on such data, the model is able to solve unseen tasks entirely in-context by interacting with an environment without explicitly updating weights of the model. More formally, if we assume that a dataset \mathcal{D} consists of learning histories , then \mathcal{D}:=\Bigl\{\left(\tau_{1}^{g},...,\tau_{n}^{g}\right)\sim\left[\mathcal{A}^{source}_{g}|g\in\mathcal{G}\right]\Bigl\}, where \tau_{i}^{g}=\left(o_{1},a_{1},r_{1},...,o_{T},a_{T},r_{T}\right) is a trajectory generated by a source algorithm from \mathcal{A}^{source}_{g} for a goal g from a set of all possible goals \mathcal{G} , and o_{i},a_{i},r_{i} are observations, actions and rewards, respectively. Such data might be difficult to obtain, since the aforementioned process requires training thousands of RL algorithms solving different tasks to obtain enough learning histories. In addition, AD suffers the same problems as any in-context algorithm. Learning the optimal solution can be delayed by a tendency of transformers to learn simple structures at first ( Edelman et al. 2024 ) . Moreover, the nature of in-context ability is unstable and can fade into in-weights regime as the training progresses, considerably complicating the emergence of adaptation ability ( Singh et al. 2024 ) . 2.2 N-Gram Attention To address simplicity bias

Read the complete text