BabyHGRN : Exploring RNNs for Sample-Efficient Training of Language Models

Patrick Haller, Jonas Golde, Alan Akbik · 2024

Read BabyHGRN : Exploring RNNs for Sample-Efficient Training of Language Models by Patrick Haller, Jonas Golde, Alan Akbik (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15978.txt.

Introduction

Patrick Haller, Jonas Golde, and Alan Akbik test whether older-style networks can rival transformers for language models. Their 2024 paper builds BabyHGRN on a recent recurrent design, HGRN2. Recurrent networks read text step by step rather than all at once. The test ground is low-resource training, where data and compute are scarce.

BabyHGRN beat transformer models on both the ten-million and hundred-million word tracks of a public challenge. Scores came from tests of grammar, world knowledge, and language understanding. Extra training by imitation of a bigger model helped further. The results push back on the field's near-total focus on transformers. They suggest recurrent models deserve a fresh look when resources are tight.

Catalog record

CREATORPatrick Haller, Jonas Golde, Alan Akbik
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15978

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BabyHGRN : Exploring RNNs for Sample-Efficient Training of Language Models Patrick Haller, Jonas Golde, Alan Akbik Abstract This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource language modeling scenarios. We utilize HGRN2 ( Qin et al. 2024 ) , a recently proposed RNN-based architecture, and comparatively evaluate its effectiveness against transformer-based baselines and other subquadratic architectures (LSTM, xLSTM, Mamba). Our experimental results show that BabyHGRN , our HGRN2 language model, outperforms transformer-based models in both the 10M and 100M word tracks of the challenge, as measured by their performance on the BLiMP, EWoK, GLUE and BEAR benchmarks. Further, we show the positive impact of knowledge distillation. Our findings challenge the prevailing focus on transformer architectures and indicate the viability of RNN-based models, particularly in resource-constrained environments. 1 Introduction In recent years, natural language processing (NLP) has been revolutionized by transformer-based language models (LMs), like BERT ( Devlin et al. 2019 ) or GPT ( Brown et al. 2020 ) and their derivatives, achieving state-of-the-art results ( Touvron et al. 2023 ; Abdin et al. 2024 ) across a wide range of tasks such as machine translation, question answering, and text generation. However, despite their dominance, transformers come with notable limitations: they require extensive training data ( Hoffmann et al. 2022 ) and enormous computational resources, which pose challenges for their use in resource-constrained environments. These limitations led to an increasing interest in more sample-efficient alternatives and approaches with lower computational requirements ( Wang et al. 2020b ) . The shared task of the BabyLM Challenge Warstadt et al. 2023a systematically explores this trend by training LMs on datasets of limited size (10M words in the " strict-small " and 100M words in the " strict " setup). The resulting models are then evaluated on linguistic and general language understanding tasks. While most participants in BabyLM Challenge focus on adapting transformers to low-resource settings, we propose revisiting recurrent neural networks (RNNs). Once foundational to sequence modeling tasks ( Lample et al. 2016 ; Howard and Ruder 2018 ) , RNNs have been largely overshadowed by transformers due to their sequential nature which does not easily allow for parallelization. Potential of RNN-architectures. In this paper, we investigate whether the inductive biases of RNN architectures, such as their sequential processing and memory states, provide advantages in data-constrained settings. This question is especially relevant given that state-of-the-art transformer models depend on quadratic self-attention, which requires calculating the inner product between all tokens. In particular, we investigate the potential of the HGRN2 ( Qin et al. 2024 ) , a novel subquadratic RNN-based architecture based on hierarchical gating. We train our model using knowledge distillation ( Hinton et al. 2015 ) and evaluate our approach, BabyHGRN , against state-of-the-art transformer models and other efficient RNN architectures (e.g. xLSTM ( Beck et al. 2024 ) or Mamba ( Gu and Dao 2024 ) ). Our experiments demonstrate that our resulting model yields better performance compared to both transformer-based and other RNN-based architectures. We summarize our contributions as follows: 1. We conduct an exploratory evaluation of transformer-based and other RNN-based architectures (HGRN2, LSTM, xLSTM, Mamba), contributing to the ongoing research on sample-efficient language modeling. 2. We present a comprehensive evaluation of our proposed HGRN2 language model BabyHGRN . We show the impact of knowledge distillation and the choice of dataset. 3. We release all code, datasets, and experimental setups to the research community to facilitate reproducibility and further research 1 1 1 https://github.com/HallerPatrick/BabyLM-2024 . Our results show that BabyHGRN outperforms transformer-based baselines on both tracks of the BabyLM challenge. Dataset Count Ratio (%) Pile-CC 4,900,155 49.00 OpenWebText2 3,078,791 30.79 FreeLaw 946,382 9.46 USPTO Backgrounds 261,159 2.61 Wikipedia (en) 187,094 1.87 PubMed Central 142,698 1.43 PubMed Abstracts 118,427 1.18 Others 365,188 3.65 Total 9,999,894 Dataset Count Ratio (%) Pile-CC 49,214,555 49.21 OpenWebText2 30,344,790 30.34 FreeLaw 9,471,436 9.47 USPTO Backgrounds 2,519,390 2.52 Wikipedia (en) 1,855,709 1.86 PubMed Central 1,449,273 1.45 PubMed Abstracts 1,175,838 1.18 Others 3,968,870 3.97 Total 99,999,861 Table 1: Composition of the 10M (left table) and 100M (right table) word datasets (word counts and ratio per domain) we created from the Pile to train BabyHGRN . 2 BabyHGRN We utilize HGRN2 as our backbone architecture with a hidden size of 2048 and 18 layers, resulting in a total parameter count of 330M. We train our model either with ( 1 ) the default dataset of the BabyLM Challenge or ( 2 ) a sub-sampled split of ThePile ( Gao et al. 2020 ) . Further, we employ knowledge distillation training using a teacher-student setup. In the following, we will discuss the details of our design choices. 2.1 Training Dataset We curate our own training datasets for the strict and strict-small tracks by sub-sampling the Pile dataset (see Table 1 ). The Pile consists of 22 smaller datasets that cover a variety of domains, including books, web pages, scientific literature, and programming code. The main motivation behind choosing the Pile dataset is its diverse composition, which may offer several advantages for language model training. Approximately 14% of the original BabyLM dataset consists of child-related text (e.g., the Children’s Book Test ( Hill et al. 2016 ) , Children’s Stories Text Corpus 2 2 2 https://www.kaggle.com/datasets/edenbd/children-stories-text-corpus , and CHILDES project ( Macwhinney 2000 ) ), which may limit its generalizability across diverse domains. In contrast, the broader scope of the Pile dataset could improve resilience in zero-shot tasks and potentially enhance adaptability for fine-tuning on specific areas of interest. We create the splits by randomly sampling from each chosen subset until we reached the pre-defined thresholds. We depict details on our selected subsets and corresponding word counts in Table 1 . To minimize computational overhead, we concatenate all samples and segment them into uniform chunks of 512 tokens. Subsequently, each input sample is tokenized using Byte-Pair Encoding (BPE), employing a vocabulary size of 16,000 tokens. We chose the BabyLlama tokenizer provided with the baseline models by the organizers 3 3 3 https://huggingface.co/babylm/babyllama-100m-2024 . 2.2 Training Objectives We use standard next-token prediction as the language modeling task and employ token-level cross-entropy loss for training our models. For a sequence of tokens x=(x_{1},...,x_{N}) , the loss is calculated as: \mathcal{L}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\log P(x_{i}|x_{1},...,x_{i-1};\theta) where \theta represents the model parameters and P(x_{i}\mid x_{1},...,x_{i-1};\theta) is the probability the model assigns to the i-th token given all previous tokens. We further improve our model through knowledge distillation ( Bucila et al. 2006 ; Hinton et al. 2015 ) , where we train a second HGRN2 model (student) using predictions from our initially trained model (teacher). While knowledge distillation traditionally transfers knowledge from larger to smaller models, using same-sized teacher and student models has proven effective in recent work - notably in the previous BabyLM Challenge where an ensemble of teachers was used for knowledge transfer ( Timiryasov and Tastet 2023 ) . The training process for the student model incorporates an additional loss term based on soft labels produced by the teacher model. The total loss function for the student model can be expressed as: \mathcal{L}_{\text{total}}=(1-\alpha)\mathcal{L}_{\text{CE}}+\alpha\mathcal{L}_{\text{KD}} where \mathcal{L}_{\text{CE}} is the standard cross-entropy loss for the student model, \mathcal{L}_{\text{KD}} is the knowledge distillation loss,

Read the complete text