Law of Neural Interaction: Depth–Width Shape, Interaction Efficiency, and Generalization
Wenjie Sun, Jinning Yang, Shuai Zhang, Mengnan Du · 2026
Read Law of Neural Interaction: Depth–Width Shape, Interaction Efficiency, and Generalization by Wenjie Sun, Jinning Yang, Shuai Zhang, Mengnan Du (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27989.txt.
Introduction
Sun and colleagues ask how a fixed parameter budget should split between a network's depth and width. Tracing a gradient-based measure of response strength, they find it tracks test error closely, including a shared peak at interpolation. The work links scaling behavior to amplified error responses, offering a practical guide to shaping models.
Catalog record
| CREATOR | Wenjie Sun, Jinning Yang, Shuai Zhang, Mengnan Du |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87355 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27989
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Law of Neural Interaction: Depth–Width Shape, Interaction Efficiency, and Generalization Wenjie Sun, Jinning Yang, Shuai Zhang, Mengnan Du Abstract The guidance of scaling laws has increased the resource demands of modern large language models (LLMs), yet it remains questionable whether these models utilize resources effectively under a fixed budget. Previous research has proved superposition as a key contributor to loss. By leveraging the Neural Feature Ansatz, we extend superposition from parameter space to gradient space and define it as neural interaction. We find that under a fixed budget, good generalization is usually accompanied by efficient neural interactions, and the model can be placed in an efficient interaction interval by adjusting its depth-width ratio ( R D / W R_{D/W} ). In addition, as the budget scales up, the efficient interaction interval of the model remains relatively stable. By comparing existing small scale dense LLMs, we observe that models operating near this interval tend to perform better on the MMLU-Pro benchmark. Our findings reveal that the R D / W R_{D/W} influences resource utilization efficiency and thereby affects generalization, providing insights into model shape initialization and the understanding of model generalization mechanisms. Code for Neural Interaction Law is available at: https://anonymous.4open.science/r/Neural_Interaction_Law-D788 . 1 Introduction Scaling has become the dominant paradigm for modern machine learning. In language modeling, performance improves predictably as model size, dataset size, and compute increase, giving rise to empirical scaling laws that have shaped the design of contemporary large language models (LLMs) [ 1 , 2 , 3 , 4 ] . Subsequent work further argued that compute-optimal training requires scaling model parameters and training tokens in roughly equal proportion, suggesting that many models are undertrained relative to their size [ 5 , 6 , 7 , 8 ] . The findings collectively establish a compelling framework for advancement: larger models trained on more data tend to generalize better. However, they also raise a more fundamental question: if scaling laws achieve performance by allocating more parameters and data budgets, then how effectively networks utilize these resources under fixed resource budgets? Resolving this question is crucial for addressing the resource demands of modern LLMs and for understanding the generalization mechanisms of neural networks. Existing explanations of scaling laws primarily include asymptotic statistics, geometry, training dynamics, and feature learning [ 9 , 10 , 11 , 12 , 13 ] . However, these perspectives mainly explain scaling in terms of asymptotic behavior, data geometry, or optimization, and say less about how limited representational resources are organized inside trained networks. Recently, Liu et al. 2025 proposed an explanation grounded in representation: neural scaling may be driven by superposition. Under this framework, models pack more features than available representational dimensions, so that interference among overlapping representations may become an important contributor to loss. This superposition perspective is compelling because it shifts the focus from raw capacity to representational efficiency. In terms of mathematics, this interference manifests as off-diagonal structure in the feature interaction matrix W^{\top}W . However, its current formulation relies on toy models constrained by a single hidden layer and tied weights, making it difficult to generalize to deep multilayer networks or other architectures [ 15 , 16 , 17 , 18 ] . To bridge this gap, we turn to the Neural Feature Ansatz (NFA). Radhakrishnan et al. 2024 demonstrated that the Average Gradient Outer Product (AGOP) captures the features learned by neural networks. From this viewpoint, NFA links the feature geometry in the weight space to gradient space through AGOP, providing an architecture-agnostic route from the original weight-based intuition behind superposition to a quantity that can be measured in deep networks [ 19 , 20 ] . In this work, we generalize superposition into the notion of gradient superposition, or more broadly, neural interaction, defined through the off-diagonal structure of AGOP. We quantify this interaction using two complementary metrics: the AGOP Off-diagonal Frobenius Energy (AOFE) for the absolute interactive energy, and the AOFE-ratio for the fraction of the network’s total sensitivity contributed by interaction. This allows us to move from asking whether interaction exists to asking whether it is efficient. To study interaction efficiency under fixed budgets, we need a controlled way to redistribute limited resources, and the model’s structure (depth and width) naturally offers a choice. We find that, under fixed budgets, generalization is strongly associated with interaction efficiency, which can be systematically reshaped by the depth-width ratio ( R_{D/W} ). The best generalizing configurations typically achieve a high interaction contribution (high AOFE-ratio) with a low absolute interaction energy (low AOFE), a regime we refer to as benign superposition . Based on these observations, we propose the Law of Neural Interaction : under fixed budgets, generalization depends not only on how much capacity a network has, but also on how efficiently it converts limited representational resources into reusable structure shared across inputs. From this perspective, depth and width matter because they reshape interaction efficiency in different ways, trading off compositional hierarchy against parallel representational freedom, so that the optimal model shape is not an isolated hyperparameter choice, but an architectural signature of efficient neural interaction. Our core contributions include: • We generalize the definition of superposition to multilayer networks and arbitrary architectures through NFA and AGOP, defining it as "neural interaction". We propose two complementary metrics to quantify this interaction: absolute interaction energy (AOFE) and interaction contribution (AOFE-ratio). Based on the replication results of the Double Descent experiment, we propose the Law of Neural Interaction: under a fixed budget, generalization depends not only on the model’s capacity, but also on the efficiency with which it converts parameters into reusable feature structures. • By fixing the budget across different network architectures and sweeping various depth-to-width ratios, we find that tuning a network’s depth-width shape can directly impact interaction efficiency, thereby indirectly affecting generalization. Furthermore, an interaction efficient interval exists under a fixed budget. • Empirically, by scaling up parameter and data budgets, we find that this interaction efficient interval remains relatively stable. Additionally, by comparing existing small dense LLMs, we observe that models closer to this interval generally perform better on the MMLU-Pro benchmark. 2 Preliminaries 2.1 Neural Feature Ansatz (NFA) A central perspective on feature learning is provided by the NFA [ 19 ] , which posits that the feature geometry learned by a network layer is encoded in a neural feature matrix (NFM), and that this matrix is proportional to a positive power of the corresponding AGOP. Specifically, consider a trained network, a layer l with weight matrix W^{(l)} , and the corresponding local input u^{(l)} . The NFA states that W^{(l)\top}W^{(l)}\propto\left(\frac{1}{N}\sum_{p=1}^{N}\nabla_{u^{(l)}}\hat{f}(\mathbf{x}^{(p)})\nabla_{u^{(l)}}\hat{f}(\mathbf{x}^{(p)})^{\top}\right)^{\alpha}, (1) where N is the number of samples and \alpha>0 is the exponent in the ansatz. For vector-valued outputs, we use the input-gradient matrix whose columns are the gradients of the output coordinates. Equation ( 1 ) is central for our purposes because it connects the classical toy-model object W^{\top}W to the gradient-based operator AGOP. In the tied-weight toy model, the relevant NFM numerically coincides with W^{\top}W . However, the two viewpoints carry different interpretations. In the toy autoencoder model, W^{\top}W is used to measure overlap among feature directions in the bottleneck representation. By contrast, under the NFA viewpoint, the corresponding AGOP characterizes how different