CGD: Modifying the Loss Landscape by Gradient Regularization
Shikhar Saxena, Tejas Bodas, Arti Yardi · 2025
Read CGD: Modifying the Loss Landscape by Gradient Regularization by Shikhar Saxena, Tejas Bodas, Arti Yardi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16182.txt.
Introduction
Shikhar Saxena, Tejas Bodas, and Arti Yardi propose a new way to climb down a loss curve in a 2025 paper on optimization. Line search methods pick a downhill path at each step. The plainest follows the slope; Newton's method adds curve data but costs more.
Their method, constrained gradient descent, adds a cap tied to the slope itself. Solving the capped problem reshapes the hill in a helpful way. Steps then land on points with smaller slopes than plain descent finds. The team proves steady, fast closing on the best point in theory. Tests on made-up problems back the claim. Two cheap variants skip the costly curve data, one by finite steps and one by a near-Newton trick.
Catalog record
| CREATOR | Shikhar Saxena, Tejas Bodas, Arti Yardi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CGD: Modifying the Loss Landscape by Gradient Regularization Shikhar Saxena, Tejas Bodas, Arti Yardi Abstract Line-search methods are commonly used to solve optimization problems. The simplest line search method is steepest descent where one always moves in the direction of the negative gradient. Newton’s method on the other hand is a second-order method that uses the curvature information in the Hessian to pick the descent direction. In this work, we propose a new line-search method called Constrained Gradient Descent (CGD) that implicitly changes the landscape of the objective function for efficient optimization. CGD is formulated as a solution to the constrained version of the original problem where the constraint is on a function of the gradient. We optimize the corresponding Lagrangian function thereby favourably changing the landscape of the objective function. This results in a line search procedure where the Lagrangian penalty acts as a control over the descent direction and can therefore be used to iterate over points that have smaller gradient values, compared to iterates of vanilla steepest descent. We establish global linear convergence rates for CGD and provide numerical experiments on synthetic test functions to illustrate the performance of CGD. We also provide two practical variants of CGD, CGD-FD which is a Hessian free variant and CGD-QN, a quasi-Newton variant and demonstrate their effectiveness. 1 Introduction 3D surface plots showing the "original loss function f₁(x)" (a blue, shallow curve) and the "steeper loss function f₂(x)" (a brown, steeper curve). A contour plot and gradient norm heatmap of the original loss function f₁(x) overlaid with two 5-step GD trajectories, one over f₁ and the other over f₂. Both start from the same point, but the path over f₂ reaches the local minimum more directly. Figure 1: (Left) Loss function f_{1}({\mathbf{x}})=x_{1}^{2}+2x_{2}^{2} and a steeper loss function f_{2}({\mathbf{x}}) . (Right) 5 steps of GD on functions f_{1}({\mathbf{x}}) and f_{2}({\mathbf{x}}) with a fixed step-size \alpha=0.05 . The contours and the gradient norm heatmap are over the function f_{1}({\mathbf{x}}) . Line-search methods such as gradient descent and Newton’s method have been extremely popular in solving unconstrained optimization problems [ 17 ] . However, vanilla version of such algorithms typically produce unsatisfactory results when applied to large scale optimization problems such as those involving deep neural networks. In such problems, the objective/loss function has a very non-linear landscape that is embedded with several local maxima and minima making them difficult to optimize [ 12 ] . To tackle this problem, one of the approaches used is to modify the objective/loss function in such a way that the new function is easier to optimize. This is typically done by adding a regularization term that will make the loss landscape more smooth, and which would in turn reduce the chances of the optimizer reaching sub-optimal minima, thereby improving generalizability [ 7 , 5 , 23 , 1 , 22 , 9 ] . In this work, we focus on this idea of modifying the objective function for effective numerical optimization and propose an algorithmic paradigm to achieve it. To illustrate the key idea, consider minimizing a loss function f_{1}(\cdot) over \mathbb{R}^{2} as shown in Fig. 1 (Left) . Additionally consider a steeper loss function f_{2}(\cdot) such that for every {\mathbf{x}}\in\mathbb{R}^{2} we have f_{1}({\mathbf{x}})\leq f_{2}({\mathbf{x}}) . Note that the functions are such that their minima coincides. Now perform gradient descent (GD) with a fixed step-size on both these functions. Their respective trajectories are plotted in Fig. 1 (Right) on the contours of f_{1} . It is clear from the figure that the iterates of GD on f_{2} are much closer to the minima as compared to iterates of GD on f_{1} . We make this observation a focal point of this work and investigate if one can achieve the latter iterates (iterates from GD on f_{2} ) on the former ( f_{1} ). Towards this, we propose Constrained gradient descent (CGD), a variant of the GD algorithm which achieves this by constraining the gradient norm to be appropriately small. Instead of solving this constrained optimization problem, we consider the Lagrangian of this function and perform GD on it. The Lagrangian parameter \lambda controls the penalty on gradient norm and thereby controls the steepness of the modified function. It is on this modified function that we seek to apply GD to possibly attain iterates that are much closer to the local minima as compared to GD iterates. From Fig. 1 (Right) we additionally see that compared to GD, the CGD iterates (which is nothing but GD iterates on f_{2} ) have a lower gradient norm. Since the path to minima is over points with lower gradient norm, this we believe is an attractive feature and even amounts to better generalization properties for high dimensional functions such as loss landscapes of neural networks. The idea of flat minima and their attractiveness goes long back to Hochreiter and Schmidhuber [ 8 ] : An optimal minimum is considered “flat” if the test error changes less in its neighbourhood. Keskar et al. [ 11 ] and Chaudhari et al. [ 4 ] observe better generalization results for neural networks at flat minima. Gradient regularization has only been recently explored, that too from a numerical perspective for deep neural networks (DNNs) where it has been shown that fixed learning rates result in implicit gradient regularization [ 22 , 9 , 1 , 19 ] . A key aim of this work is to understand gradient regularization from the perspective of a numerical optimization algorithm and identify properties and features that may not have been obvious earlier. We believe CGD and its variants discussed in this work have the ability to find flatter minima, something which is useful while training neural networks. We summarize our contributions below: • We propose a new line-search procedure called Constrained Gradient Descent (CGD) that performs gradient descent on a gradient regularized loss function. We also provide global linear convergence for CGD in Theorem 3.1 . • While CGD requires the Hessian information, we also propose a first-order variant of CGD using finite-difference approximation of the Hessian called CGD-FD . We define appropriate stopping criteria in CGD-FD for settings where gradient computation can be expensive. • We conduct experiments over synthetic test functions to compare the performance of CGD and its variants compared to standard line-search procedures. • Our work also provides new insights to existing gradient regularization based methods. In fact, we re-interpret and identify pitfalls in the Explicit Gradient Regularization (EGR) Method [ 1 ] using our formulation. The remainder of the paper is organized as follows. In the next section, we recall some preliminaries on line-search methods. We then discuss the CGD algorithm and propose its variants. We then illustrate the performance of our algorithm on several test functions and conclude with a discussion on future directions. 2 Notation and Preliminaries The set of real numbers and non-negative real numbers is denoted by \mathbb{R} and \mathbb{R}^{+} respectively. We consider a vector \mathbf{x}\in\mathbb{R}^{n} as a column vector given by \mathbf{x}=\begin{bmatrix}x_{1}&x_{2}&\ldots&x_{n}\end{bmatrix}^{T} . We use a boldface letter to denote a vector and lowercase letters (with subscripts) to denote its components. L^{p} -norm of a vector \mathbf{x} is defined as \|\mathbf{x}\|_{p}=\left(\sum_{i}|{x_{i}}|^{p}\right)^{1/p} . Setting p=2 gives us L^{2}