The 4th Dimension for Scaling Model Size

Ruike Zhu 1‡ , Hanwen Zhang 1‡ , Tianyu Shi 2∗ , Chi Wang 3 , Tianyi Zhou 4 , Zengyi Qin 5∗† 1 University of Illinois at Urbana-Champaign 2 University of Toronto 3 Google DeepMind 4 University of Maryland, College Park 5 Massachusetts Institute of Technology {ruikez2, hanwen6}@illinois.edu , ty.shi@ · 2025

Read The 4th Dimension for Scaling Model Size by Ruike Zhu 1‡ , Hanwen Zhang 1‡ , Tianyu Shi 2∗ , Chi Wang 3 , Tianyi Zhou 4 , Zengyi Qin 5∗† 1 University of Illinois at Urbana-Champaign 2 University of Toronto 3 Google DeepMind 4 University of Maryland, College Park 5 Massachusetts Institute of Technology {ruikez2, hanwen6}@illinois.edu , ty.shi@ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18233.txt.

Introduction

Ruike Zhu and co-authors propose a fourth way to scale large language models, beyond depth, width, and parameter count, published in 2025. Their idea, virtual logical depth, reuses the same weights to deepen the model's internal steps. That raises effective depth without adding any new parameters at all.

Controlled tests gave three main findings. First, at a fixed parameter count, reusing weights left knowledge capacity near unchanged, while capacity across models still tracked parameter count. Second, well-built reuse boosted reasoning a great deal, splitting reasoning gains from model size. Third, those gains held across designs and reuse schedules. The authors ask whether future gains need ever-larger models, or deeper reuse of the models we already have.

Catalog record

CREATORRuike Zhu 1‡ , Hanwen Zhang 1‡ , Tianyu Shi 2∗ , Chi Wang 3 , Tianyi Zhou 4 , Zengyi Qin 5∗† 1 University of Illinois at Urbana-Champaign 2 University of Toronto 3 Google DeepMind 4 University of Maryland, College Park 5 Massachusetts Institute of Technology {ruikez2, hanwen6}@illinois.edu , ty.shi@
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18233

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The 4th Dimension for Scaling Model Size Ruike Zhu 1‡ , Hanwen Zhang 1‡ , Tianyu Shi 2∗ , Chi Wang 3 , Tianyi Zhou 4 , Zengyi Qin 5∗† 1 University of Illinois at Urbana-Champaign 2 University of Toronto 3 Google DeepMind 4 University of Maryland, College Park 5 Massachusetts Institute of Technology {ruikez2, hanwen6}@illinois.edu , [email protected] , [email protected] , [email protected] , [email protected] Abstract Scaling the size of large language models typically involves 3 dimensions: depth, width, and the number of parameters. In this work, we explore the 4th dimension, virtual logical depth (VLD), which allows increasing the effective algorithmic depth without changing the overall parameter count by reusing parameters within the model. Although parameter reuse itself is not new, its intriguing potential and characteristics in model scaling have not been thoroughly studied. We carefully design controlled experiments and have the following key discoveries on VLD scaling: 1. VLD scaling forces the knowledge capacity of the model to stay almost constant , though with some non-significant variations. 2. VLD scaling enables the reasoning capability to be significantly improved , if the scaling method is properly implemented. 3. The number of parameters is proportional to knowledge capacity , but not reasoning capability. Under certain conditions, it is not necessary to increase parameter count to improve reasoning. 4. The above observations hold for various model configurations and are likely to be generally true under the scope of our experiments. These findings not only provide useful insights on the future model scaling strategies, but also introduces a even deeper question: Do we really need a lot of parameters to get a really intelligent model? We believe there are many unknown dynamics inside model scaling that needs exploration, so we fully open-source our models and code ( https://vldscaling.ngrok.io ) to ensure reproducibility and facilitate future explorations. 1 1 footnotetext: Equal advising. 2 2 footnotetext: Corresponding author. † † footnotetext: \ddagger Equal contribution. Figure 1: Comparison between classical model size scaling vs. VLD scaling. Bubble size proportional to total effective depth (base layer depth + virtual logical depth). Blue bubbles represent standard models without VLD scaling but following the classical model size scaling, while green bubbles show models with VLD scaling applied, demonstrating near-vertical scaling paths. VLD scaling significantly enhances reasoning capability (y-axis) while keeping the knowledge capacity (x-axis) almost constant without significant variations. 1 Introduction Scaling up large language models (LLMs) is widely recognized as a key factor in enhancing their general capabilities ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ; Brown et al. 2020 ; Wei et al. 2022 ) . The general concept of model scaling several directions, including model size scaling, data scaling, compute scaling ( Wei et al. 2022 ) , and the recent inference scaling ( Snell et al. 2024 ; Jaech et al. 2024 ) . The model size scaling direction typically occurs along three dimensions: depth, width, and the total number of parameters 1 1 1 Since depth and width directly influence the number of parameters, these three dimensions are not entirely independent. However, the total parameter count can indeed be adjusted independently by modifying the inner dimension of the MLP layer within a transformer block. . Contemporary LLMs often contain an extremely large number of parameters, sometimes exceeding 100 billion ( Wei et al. 2022 ) . While such models can store vast amounts of knowledge, this does not necessarily equate to strong reasoning capabilities. In contrast, the human brain exhibits the opposite characteristics. Human memory is relatively limited—hence the invention of tools like pen and paper to compensate for this weakness—yet humans excel at reasoning ( Chollet 2019 ; Xu & Poo 2023 ; Wang & Li 2024 ) . This cognitive ability is largely supported by a specialized reasoning core located in the prefrontal cortex ( Raichle 2009 ) . This fundamental difference in the strengths and weaknesses of language models and the human brain raises critical questions about the efficacy of the current scaling paradigm in achieving super-intelligence. Ideally, a truly intelligent system would consist of a powerful reasoning core complemented by a selective yet essential memory, with additional domain-specific knowledge retrieved externally as needed ( Kumaran et al. 2016 ) . This insight motivates us to explore new scaling dimensions that intentionally limits the knowledge capacity to force it towards improving reasoning capabilities . We find that simple weight-sharing within the standard transformer architecture performs surprisingly well. This introduces the core concept of this manuscript: virtual logical depth (VLD). VLD is achieved by repeating transformer layers while sharing weights across these repetitions. Although not entirely new , the intriguing properties of VLD in model scaling is not thoroughly explored. Previous studies ( Lan et al. 2020 ; Dehghani et al. 2019 ; Bai et al. 2019 ; Hay & Wolf 2024 ; Bhojanapalli et al. 2021 ; Reid et al. 2021 ; Takase & Kiyono 2021 ; Liu et al. 2023 ) have discussed parameter reuse or layer repetition in different contexts, but their primary focus was not on understanding its dynamics in scaling. The looped transformers ( Saunshi et al. 2025 ) is a pioneering work that studies the reasoning capability among the related work, but we focus more on the model scaling strategy and provide a set of more comprehensive experiments and new conclusions on the VLD scaling law. Through carefully controlled experiments, we investigated the characteristics of VLD scaling, specifically measuring reasoning capabilities and knowledge capacity under various configurations. We discovered that VLD scaling shows very different properties from classical model size scaling. In brief, VLD scaling forces the knowledge capacity of the model to stay almost constant (with non-significant variations), while significantly improving the reasoning capability. We illustrate a part of our main findings in Figure 1 , which compares the two scaling strategies. These findings hold for various model and experiment configurations, and are likely to be universal under our scope of experiments. Under reasonable conditions, we don’t have to increase the number of parameters in order to increase the inherent reasoning capability of the model . Beyond the scope of this work, we would like to raise a even deeper question: Do we really need a lot of parameters to achieve super-intelligence? Of course this first depends on how we define super-intelligence. Humans have poor memory but good reasoning capability and agency capability (e.g., subjective initiatives). Under limited human brain capacity, why did the natural evolution chose to allocate more capacity on reasoning capability and agency capability, but only some essential capacity for memory? It is not impossible that under a small but essential amount of parameters and memory capacity, we will be able to push the reasoning capability and agency capability to the extreme and achieve super-intelligence in the far future. We would like to move a small step towards this ultimate goal, and open-source all source code of our current work to facilitate future research. Figure 2: Different patterns of parameter reuse to increase VLD while keeping the total number of parameters constant. (a) an example of a standard transformer with 3 layers. (b) sequentially repeat neighboring layers. (c) cycle-repeat the layers. (d) repeat layers in an inverse-cycle order. Under

Read the complete text