Scalable Complexity Control Facilitates Reasoning Ability of LLMs
Liangkai Hang, Junjie Yao, Zhiwei Bai, Tianyi Chen, Yang Chen, Rongjie Diao, Hezhou Li, Pengxiao Lin, Zhiwei Wang, Cheng Xu, Zhongwang Zhang, Zhangchen Zhou, Zhiyu Li, Zehao Lin, Kai Chen, Feiyu Xiong, Yaoyu Zhang, Weinan E, Hongkang Yang, Zhi-Qin John Xu · 2025
Read Scalable Complexity Control Facilitates Reasoning Ability of LLMs by Liangkai Hang, Junjie Yao, Zhiwei Bai, Tianyi Chen, Yang Chen, Rongjie Diao, Hezhou Li, Pengxiao Lin, Zhiwei Wang, Cheng Xu, Zhongwang Zhang, Zhangchen Zhou, Zhiyu Li, Zehao Lin, Kai Chen, Feiyu Xiong, Yaoyu Zhang, Weinan E, Hongkang Yang, Zhi-Qin John Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23013.txt.
Introduction
Liangkai Hang, Junjie Yao, Zhiwei Bai, and sixteen co-authors show that taming model complexity, a cheap training knob, lifts language model reasoning and scaling at once. Their paper from twenty twenty-five adjusts two levers: the initialization rate and the weight decay coefficient. Complexity control in this form needs no new data or architecture. The scaling law improves across model sizes and data sizes alike.
A sharper test compares 2.4-billion-parameter models pretrained on one trillion tokens under varied complexity settings. Holding the initialization standard deviation fixed loses to a constant initialization rate. With the rate held constant, the scaling curve drops faster as either size or data grows. The team frames complexity control as a steady, general path for pushing large models further.
Catalog record
| CREATOR | Liangkai Hang, Junjie Yao, Zhiwei Bai, Tianyi Chen, Yang Chen, Rongjie Diao, Hezhou Li, Pengxiao Lin, Zhiwei Wang, Cheng Xu, Zhongwang Zhang, Zhangchen Zhou, Zhiyu Li, Zehao Lin, Kai Chen, Feiyu Xiong, Yaoyu Zhang, Weinan E, Hongkang Yang, Zhi-Qin John Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23013
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Complexity Control Facilitates Reasoning Ability of LLMs Liangkai Hang, Junjie Yao, Zhiwei Bai, Tianyi Chen, Yang Chen, Rongjie Diao, Hezhou Li, Pengxiao Lin, Zhiwei Wang, Cheng Xu, Zhongwang Zhang, Zhangchen Zhou, Zhiyu Li, Zehao Lin, Kai Chen, Feiyu Xiong, Yaoyu Zhang, Weinan E, Hongkang Yang, Zhi-Qin John Xu Abstract The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their generalizability. This work demonstrates that model complexity control, conveniently implementable by adjusting the initialization rate and weight decay coefficient, improves the scaling law of LLMs consistently over varying model sizes and data sizes. This gain is further illustrated by comparing the benchmark performance of 2.4B models pretrained on 1T tokens with different complexity hyperparameters. Instead of fixing the initialization std, we found that a constant initialization rate (the exponent of std) enables the scaling law to descend faster in both model and data sizes. These results indicate that complexity control is a promising direction for the continual advancement of LLMs. 1 Introduction Figure 1: Next-token prediction accuracy with varying model complexity. The colors indicate the probability of each token predicted by the models Model complexity decreases from left to right, as the initialization rate \gamma=0.1,0.5,1 (see Section 3 for the definition of \gamma ). These models have the same shape (180M parameters) and are trained the same dataset (40B tokens). In recent years, large language models (LLMs) have achieved unprecedented progress, demonstrating impressive performance on a wide range of tasks [ 2 , 36 , 58 , 66 , 70 , 77 ] . The key to this success is the improved generalizability of these models, in particular, their reasoning ability. Various approaches have been explored to enhance reasoning, such as post-training with reinforcement learning [ 30 , 21 ] , high-quality math and code data with reasoning traces [ 21 ] , chain-of-thought and related prompting strategies [ 67 ] , and the separation of reasoning and memory in pretraining [ 71 ] . This work identifies model complexity as the key factor in the development of the reasoning ability of LLMs. Complexity control can be implemented through various means, such as adjusting the rate of parameter initialization [ 39 , 88 , 89 , 84 , 73 ] or applying stronger penalties on parameter norms [ 84 ] . Similar to the gene that determines the characteristics of an organism, these designs directly regulate the reasoning ability of LLMs. An illustration is provided in Figure 1 . On the left, a pretrained model with large complexity (large initialization scale) fails to make meaningful next-token predictions on unseen test data. In the middle, with moderate complexity (the commonly used initialization scale), the model demonstrates basic knowledge of grammar (e.g. “finds that ”) and vocabulary (e.g. “Jess ica ”), as well as the induction head mechanism [ 43 ] (e.g. the repeated phrase “September 12, 2024 ”). On the right, the low-complexity model (i.e. with small initialization scale) further captures sophisticated semantic reasoning, successfully predicting “Plaintiff Michael Anderson ” given the context “Anderson v. Carter” despite that the full name “Michael Anderson” has not appeared before, and also predicting “required medical treatment ” given that Anderson was punched by Carter. Intuitively, smaller complexity forces the model to compress data into a smaller set of production rules, revealing the deep dependency among the tokens and preventing plain memorization. Previous studies [ 39 , 88 ] have shown that with a smaller initialization scale, neurons within each layer tend to evolve within a few groups, a phenomenon known as condensation, which limits the effective number of neurons. Readers are referred to an overview of condensation [ 69 ] . Likewise, models trained with stronger penalty on parameter norm may converge to solutions with smaller norms, resulting in lower-complexity outputs. Thus, it is promising that these techniques can enhance the reasoning ability of LLMs. As a verification of this intuition, our experiments exhibit improved scaling laws in data size and model size, as well as higher scores over almost all benchmarks (+4.6% for 0.9B model with 600B data, and +3.4% for 2.4B model with 1T data, averaged over 15 tasks). Compared with the standard deviation (std) of parameter initialization, the initialization rate (the exponent of std as a function of network width) turns out to be the right invariant for the scaling laws. This is in accordance with previous works [ 39 , 88 ] on the phase diagram of neural network training. We provide some heuristic calculations to explain how complexity control facilitates the learning of multi-step reasoning. 2 Related works LLMs reasoning ability Even advanced LLMs such as GPT-4 often struggle with implicit reasoning over parametric knowledge [ 56 , 28 , 47 , 45 , 3 , 72 ] , revealing their limited ability to internalize structured facts and rules. Verbalized reasoning strategies such as chain-of-thought can substantially boost performance, particularly for large models [ 67 , 75 , 54 , 37 , 74 , 18 , 33 ] . However, understanding the underlying capacity for implicit reasoning remains a critical challenge, often studied by controlled experiments [ 46 , 13 , 63 ] . This work takes the more intrinsic perspective of complexity control. Effect of initialization Parameter initialization is known to be influential to the training and performance of classical neural networks [ 4 , 10 , 80 , 14 , 27 , 40 , 50 , 52 , 68 ] . For instance, distinct phases (the linear and condensed regimes) can be induced in wide ReLU networks by varying initialization rates [ 39 , 88 ] , and training in the condensed regime tend to fit data with lower-complexity functions [ 81 , 85 , 82 , 86 ] . The particular case of Transformer language models has also been studied [ 25 , 38 , 59 , 61 , 78 , 91 , 83 , 84 ] , with particular interest on the impact of initialization on training stability and efficiency. Experiments on toy datasets [ 83 , 84 ] show that small initialization scales assist Transformers to identify the elementary functions when fitting synthetic compositional data. However, the influence of initialization on the reasoning ability of Transformers trained on natural language data remains to be explored. Weight decay [ 29 ] introduced weight decay as a method to improve the generalization of neural networks. Many subsequent works have explored its role in controlling model complexity and enhancing generalization [ 8 , 7 , 5 , 42 , 20 , 65 ] . More recently, weight decay has been shown to be particularly important for achieving better generalization [ 44 , 60 ] . In this work, we systematically investigate how controlling pre-training complexity, via initialization strategies and weight decay, affects the performance of LLMs. Unlike prior studies that focus on training stability or synthetic tasks, we evaluate the impact across a wide range of downstream benchmarks and analyze the underlying mechanisms, aiming to offer practical guidance for large-scale model pre-training. 3 Complexity control In this section, we introduce the approach to modulate the model complexity. Initialization rate \gamma Given