Gradient Descent Methods for Regularized Optimization

Filip Nikolovski, Irena Stojkovska, Katerina Hadži-Velkova Saneva, Zoran Hadži-Velkov · 2024

Read Gradient Descent Methods for Regularized Optimization by Filip Nikolovski, Irena Stojkovska, Katerina Hadži-Velkova Saneva, Zoran Hadži-Velkov (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20115.txt.

Description

Filip Nikolovski and three co-authors review step by step search methods for optimization problems that add a penalty term, called regularization. Such terms can smooth a goal, shape the valid set, or stop a model from fitting noise. Their work came out in twenty twenty-four.

Plain gradient descent works poorly when the penalty makes the goal sharp at zero, since steps can swing or stall. A near variant fixes this with soft thresholding, which shrinks steps toward zero and yields sparse answers. The authors survey both methods, then add a new variant whose step size adapts at each pass. It gauges the local slope bound on the spot rather than using one fixed global value.

Catalog record

CREATORFilip Nikolovski, Irena Stojkovska, Katerina Hadži-Velkova Saneva, Zoran Hadži-Velkov
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20115

Browse the full library · About the Tower of Babel