Gradient Regularized Newton Boosting Trees with Global Convergence

Nikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon · 2026

Read Gradient Regularized Newton Boosting Trees with Global Convergence by Nikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00581.txt.

Description

Nikita Zozoulenko and 3 co-authors put Newton boosting, the second-order engine inside the dominant tree-boosting libraries, on firm theoretical ground. Those libraries fit decision trees stage by stage, and second-order steps are their secret, yet global convergence of Newton boosting was poorly understood compared with first-order methods. The authors frame the problem as convex optimization with Newton's method on function spaces, allowing inexact steps.

Within that framework, ordinary Newton boosting provably converges linearly for smooth, strongly convex losses meeting a Hessian condition. For general convex losses, they add a gradient-regularized Newton scheme whose adaptive penalty scales with the square root of the gradient's size. That variant converges at a quadratic-in-denominator rate, matching first-order boosting with momentum, and is therefore a globally convergent second-order boosting algorithm. In experiments their scheme converged on problems where the vanilla method could diverge.

Catalog record

CREATORNikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00581

Browse the full library · About the Tower of Babel