Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models

Yuval Weiss, David Demitri Africa, Richard Diehl Martinez · 2025

Read Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models by Yuval Weiss, David Demitri Africa, Richard Diehl Martinez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12960.txt.

Introduction

Yuval Weiss and 3 co-authors investigate ReLoRA in small language models in a 2025 paper. LoRA is a parameter-efficient method that fine-tunes models through small low-rank adapters. ReLoRA extends the idea to pretraining by repeatedly merging and reinitializing those adapters, increasing cumulative rank while keeping updates cheap.

Small language models, by contrast, may under-use their available dimensionality and suffer rank deficiencies. The authors ask whether ReLoRA's rank-expanding rule can steer small models toward healthier learning dynamics. They present the first systematic study of ReLoRA in models of tens of millions of parameters, evaluating performance and learning dynamics across loss, perplexity, and a linguistic benchmark. ReLoRA underperformed full training in their results, with the gap growing at larger scales.

Catalog record

CREATORYuval Weiss, David Demitri Africa, Richard Diehl Martinez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-12960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models Yuval Weiss, David Demitri Africa, Richard Diehl Martinez Abstract Parameter-efficient methods like LoRA have revolutionised large language model (LLM) fine-tuning. ReLoRA extends this idea to pretraining by repeatedly merging and reinitialising low-rank adapters, increasing cumulative rank while keeping updates cheap. This aligns well with observations that high-capacity models learn through locally low-rank trajectories that expand over time. By contrast, recent work suggests that small language models (SLMs) exhibit rank deficiencies and under-utilise their available dimensionality. This raises a natural question: can ReLoRA’s rank-expanding update rule steer SLMs toward healthier learning dynamics, mitigating rank bottlenecks in a capacity-constrained regime? We argue SLMs are an ideal testbed: they train quickly, enable controlled ablations, and make rank phenomena more measurable. We present the first systematic study of ReLoRA in SLMs (11M–66M parameters), evaluating both performance and learning dynamics. Across loss, Paloma perplexity, and BLiMP, we find that ReLoRA underperforms full-rank training, with gaps widening at larger scales. Analysis of proportional effective rank and condition numbers shows that ReLoRA amplifies existing rank deficiencies and induces ill-conditioned updates early in training. Our results suggest that while ReLoRA’s merge-and-restart strategy can expand ranks in larger models, it does not straightforwardly translate to capacity-limited SLMs, motivating adaptive-rank or hybrid-rank approaches for low-compute pretraining. yuvalw/pico-relora Yu-val-weiss/pico-relora 1 Introduction Contemporary work on language modelling has continually prioritised ever-larger scales, delivering remarkable capability gains ( Chowdhery et al., 2023 ; Grattafiori et al., 2024 ; OpenAI, 2024 ) . However, the computational and environmental costs of large-scale language modelling are substantial ( Chowdhery et al., 2023 ; Grattafiori et al., 2024 ; Morrison et al., 2024 ; OpenAI, 2024 ) . Small language models (SLMs) offer a complementary path: they are cheap to train and deploy, easier to study, and attractive for settings where safety, privacy, or energy constraints dominate ( Schwartz et al., 2020 ) . However, SLMs lag in quality due to tight capacity limits and brittle optimisation ( Mielke et al., 2019 ; Ettinger, 2020 ; Kaplan et al., 2020 ) . A rank-centric view of learning. A growing body of work frames transformer learning through the lens of the rank of model weights. In high-capacity models, training often proceeds via low-rank updates that gradually expand effective rank ( Aghajanyan et al., 2021 ; Boix-Adsera et al., 2023 ) . Conversely, SLMs show rank deficiencies and anisotropic representations that restrict gradient flow and under-utilise model dimensionality ( Noci et al., 2022 ; Diehl Martinez et al., 2024 ; Godey et al., 2024a ) . From this perspective, methods that shape the rank profile of updates could directly influence optimisation quality and final capability. LoRA. LoRA ( Hu et al., 2022 ) has been widely used due to its effectiveness and applicability to any model with matrix computations, enabling straightforward fine-tuning of even the largest models ( Blattmann et al., 2023 ; Dettmers et al., 2023 ; Fomenko et al., 2024 ) . LoRA introduces low-rank adaptation, where the pretrained model weights are frozen, and fine-tuning is performed with trainable rank decomposition matrices at each layer of the architecture, thus significantly reducing the number of trainable parameters. ReLoRA. ReLoRA, proposed by Lialin et al. (2023) , aims to leverage LoRA’s overwhelming success in low-rank adaptation for fine-tuning to advance efficient pretraining. It does this by injecting low-rank LoRA-style matrices into a language model, then repeatedly merging and reinitialising them throughout the training process, thereby parametrising gradient updates through the merge operations at each restart. The method is tested on models with 60M to 1.3B parameters, improving training speeds and reducing GPU memory footprints, especially at the higher end. However, research for models even smaller than this is limited. Why ReLoRA might help SLMs. Small transformer models suffer from rank deficiencies in their weight matrices, limiting the subspace they can explore during training ( Diehl Martinez et al., 2024 ) . ReLoRA ( Lialin et al., 2023 ) periodically merges low-rank adapters into the base weights and reinitialises them. Summing distinct low-rank updates increases the rank of the cumulative update, potentially widening the subspace explored over training. If SLMs struggle because their updates remain trapped in low-rank bottlenecks, then an explicit rank-expansion mechanism could steer them toward healthier dynamics, improving sample efficiency and language competence in the low compute regime. This provides the intuition behind our methodology. Why test this in SLMs? SLMs are fast to train, enabling careful ablations across model structure alongside detailed diagnostics such as effective rank and condition number that would otherwise be prohibitively costly in billion-parameter models. Their capacity limits also amplify rank phenomena, making both benefits and failure modes easier to detect. Therefore, SLMs are a sensitive and economical sandbox for understanding how rank-structured updates shape learning. This work: research question . In this work, we ask whether ReLoRA’s rank-expanding merge-and-restart methodology actually helps in the capacity-constrained regime. In other words, does ReLoRA boost performance or drag it down? The case for either can be summarised as follows: Boost : By strategically resetting and merging LoRA matrices, ReLoRA aggregates multiple low-rank steps, which may widen those bottlenecks and enable smaller models to capture more complex patterns Drag : However, ReLoRA may also have the opposite effect: further reducing the (already limited) representational capacity of SLMs. This work: outline of methodology and results . We extend a Llama-style SLM ( Touvron et al., 2023 ; Diehl Martinez, 2025 ) with ReLoRA and run matched ablations at 11M and 66M parameters. We evaluate training loss, Paloma perplexity ( Magnusson et al., 2024 ) , and BLiMP ( Warstadt et al., 2020 ) , and additionally probe learning dynamics via proportional effective rank (PER) and condition numbers of both weights and updates. Contrary to the motivating intuition, ReLoRA does not boost SLMs: it underperforms full-rank training and reinforces rank deficiencies, with early training marked by highly ill-conditioned updates. We thus conclude that the mechanism that helps larger models does not straightforwardly transfer to SLMs. This work: contributions . Our contributions are as follows: (1) Novel systematic evaluation of ReLoRA for small language models (SLMs). We investigate ReLoRA on 11M and 66M parameter models, filling a critical gap in understanding its behaviour for low-compute domains. We make two public code contributions: a public HuggingFace spaceto be used for evaluating LMs on the BLiMP task through the evaluate library and a public fork, pico-relora , of pico-train ( Diehl Martinez, 2025 ) extending pico-decoder with ReLoRA. (2) Comprehensive analysis of learning dynamics. We compute proportional effective rank (PER) and condition numbers of weights and gradient updates throughout training, revealing that ReLoRA consistently reduces rank and induces highly ill-conditioned updates in SLMs. (3) Drag: Evidence of performance degradation and training instability. Across loss, Paloma perplexity, and BLiMP evaluation, ReLoRA underperforms its full-rank baseline, with widening gaps as model size increases from the tiny to small scales. Our findings suggest that low-rank strategies do not trivially transfer to SLMs and thus motivate hybrid or adaptive-rank approaches for future low-resource model training. Figure 1: LoRA decomposition ( Hu et al., 2022 ) , which can be applied to any linear operation parameterised by a matrix 2 Background: ReLoRA We now

Read the complete text