Revisiting Gradient Descent: A Dual-Weight Method for Improved Learning
XI WANG · 2025
Read Revisiting Gradient Descent: A Dual-Weight Method for Improved Learning by XI WANG (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11965.txt.
Introduction
Xi Wang rethinks how neural networks learn, by giving every neuron two weight sets instead of one. Standard training crams what a neuron should respond to and what it should ignore into a single set of numbers, which blurs fine distinctions. Wang keeps the two apart during learning: one set gathers the target features, the other the rest, and the working weight is their difference.
On house prices, wine scores, and three image benchmarks, the split weights generalized better and resisted overfitting, above all when training data was sparse or noisy. At use time the method costs exactly the same as the standard setup. Wang presents it as a drop-in improvement to a workhorse algorithm.
Catalog record
| CREATOR | XI WANG |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11965
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Gradient Descent: A Dual-Weight Method for Improved Learning XI WANG Abstract **** 1 Introduction In neural networks, the weight vector W of a neuron plays a crucial role in transforming input features into outputs. While representing synaptic weights of postsynaptic neurons from presynaptic neurons, W can also be viewed as the neuron’s encoding of the target concept it aims to represent. However, defining a target concept independently from other concepts often results in insufficient representation; rather, effective learning necessitates contrasting the target with non-targets. For instance, to accurately define a ”dog,” it is essential not only to understand the characteristics of dogs but also to distinguish them from non-dog entities. Without this contrast, differentiation remains incomplete. Similarly, when a neuron learns, it should capture the differences between the features of the target class (hereafter termed positive examples) and those of non-target classes (negative examples). This suggests that the weight vector W should represent the contrast between the target features ( W_{1} ) and the non-target features ( W_{2} ), effectively modeling W=W_{1}-W_{2} . Traditional methods, which store and update this contrast within a single weight vector W using gradient descent, as described by LeCun et al. (1989) [ 1 ] , may not accurately capture this distinction. In this study, we propose an alternative to the conventional gradient descent approach by decomposing the weight vector into two separate components, W_{1} and W_{2} , which are updated independently. By explicitly modeling the contrast between target and non-target features, our method adjusts weights more accurately during training. Experimental results demonstrate that models trained using the formulation, (W_{1}-W_{2})X+\text{bias} , generalize better than those trained with the traditional WX+\text{bias} approach. While our method introduces a slight increase in computational cost during training due to the separate updates of W_{1} and W_{2} , the overall computational complexity remains equivalent to the traditional method in terms of Big-O notation. Moreover, once training is completed, we can directly compute and store W=W_{1}-W_{2} , ensuring that the computational cost during the inference phase is identical to that of the conventional approach. In summary, this research advocates for replacing the traditional single-weight gradient descent method with the W_{1}-W_{2} approach, offering a more accurate and effective framework for training neural networks. 2 Related Work The idea that neurons should learn by contrasting positive and negative features has its roots in the broader field of contrastive learning [ 2 , 3 , 4 ] . Classical contrastive learning frameworks typically focus on comparing different input samples (e.g., pairs or triplets) to encourage representations that capture meaningful distinctions between similar and dissimilar instances. In contrast, our approach shifts the focus from sample-level comparisons to neuron-level weight contrasts. Instead of promoting discrimination by directly comparing input samples, we advocate comparing the weights a neuron learns under different conditions—i.e., how well it encodes target features relative to non-target features. This weight-centric perspective provides a more fine-grained mechanism for achieving contrastive learning within a neuron’s parameters. We will further elaborate on how our method differs from traditional contrastive learning methodologies in Appendix [A]. Our proposed update strategy also bears some conceptual resemblance to L2 regularization [ 5 , 6 ] , which is commonly employed to curb overfitting by penalizing large weight magnitudes. However, unlike L2 regularization—which uniformly applies a penalty to all weights regardless of their role or importance—our approach functions as a neuron-specific, adaptive form of regularization. This dynamic procedure allows each neuron to determine which time and how much to adjust its weights, thereby avoiding the indiscriminate suppression of valuable feature representations. By allowing neurons to dynamically control their own weight adjustments, our method avoids the potential downsides of blanket regularization, such as suppressing useful feature representations. Subsequent sections, including those dedicated to the comparison with L2 regularization and our experimental evaluation, will provide a more detailed discussion of these differences. 3 Methodology Understanding how neurons encode information is crucial for improving neural network training methods. Traditional gradient descent methods update the synaptic weight vector W without explicitly distinguishing between target and non-target features. In this study, we propose a novel approach that decomposes the weight vector into two separate components, W_{1} and W_{2} , to more accurately model the contrast between these features. 3.1 Traditional gradient descent In the following, we focus on the computation performed by a single neuron. In traditional neural networks, the output of a neuron a is computed through a two-step process: 1. Linear Combination: \displaystyle z=WX+b. (1) Here, W is the weight vector. X is the input vector. b is the bias term. 2. Activation: \displaystyle a=f(z), (2) where f is the activation function applied to z . Our objective is to adjust the weights W to minimize a loss function E . An example of E is the mean squared error, which measures the discrepancy between the network’s predictions and the actual targets. To minimize E , we use gradient descent to update the weights in the direction that reduces the loss: \displaystyle\Delta W=-\eta\frac{\partial E}{\partial W}, (3) where \Delta W is the change in weights. \eta is the learning rate, controlling the step size of each update. \frac{\partial E}{\partial W} is the gradient of the loss function with respect to the weights. Using the chain rule, the gradient \frac{\partial E}{\partial W} can be written as: \displaystyle\frac{\partial E}{\partial W} \displaystyle=\frac{\partial E}{\partial a}\cdot\frac{\partial a}{\partial z}\cdot\frac{\partial z}{\partial W} \displaystyle=\frac{\partial E}{\partial a}\cdot f^{\prime}(z)\cdot X (4) Hence, the weight update rule becomes: \displaystyle W_{\text{new}}=W_{\text{old}}+\Delta W, (5) where \displaystyle\Delta W=-\eta\cdot\frac{\partial E}{\partial a}\cdot f^{\prime}(z)\cdot X. (6) The case with ReLU activation function: When the activation function f is the Rectified Linear Unit (ReLU), defined as f(z)=\max(0,z) , its derivative f^{\prime}(z) simplifies significantly: \displaystyle f^{\prime}(z)=\begin{cases}1,&\text{if }z>0\\ 0,&\text{if }z\leq 0\end{cases} (7) Substituting f^{\prime}(z) into the weight update rule: \displaystyle\Delta W=\begin{cases}-\eta\cdot\frac{\partial E}{\partial a}\cdot X,&\text{if }z>0\\ 0,&\text{if }z\leq 0\end{cases} (8) This means that only neurons with a positive pre-activation value z (i.e., active neurons) will have their weights updated. Inactive neurons (where z\leq 0 ) do not contribute to the weight update. 3.1.1 Cumulative weight updates over iterations Considering multiple training iterations, the cumulative weight becomes: \displaystyle W=W_{\text{init}}-\eta\sum_{i=1}^{n}\text{grad}_{i}\cdot X_{i}, (9) where the gradient is defined as \displaystyle\text{grad}_{i}\equiv\frac{\partial E}{\partial a}f^{\prime}(z_{i}) (10) while W_{\text{init}} is the initial weight vector, and X_{i} and z_{i} represent an input vector at iteration i and weighted linear sum of its elements, respectively. After numerous updates, the influence of W_{\text{init}} diminishes, allowing us to focus on the aggregated updates. Since W_{\text{init}} is negligibly smaller than the second term, here we remove it in the following equations. Since gradients \text{grad}_{i} can be positive or negative, we can partition the summation based on the sign of the gradients: \displaystyle W=\eta\left(\sum_{\text{grad}_{i}<0}(-\text{grad}_{i})X_{i}\right)-\eta\left(\sum_{\text{grad}_{i}>0}\text{grad}_{i}X_{i}\right). (11) This formulation clarifies that the weight vector W is effectively the difference between two aggregated terms: one associated with negative gradients (reinforcing the neuron’s response to positive examples) and one with positive gradients (reducing the response to negative examples). 3.2 Proposed weight decomposition and its implementation Recognizing the inherent structure with respect to the positive or negative gradient associated with each input in the traditional weight update rule (Eq. 11 ), we propose to extend it to perform the updates separately. Namely, we want the weight vector to be explicitly composed of the