Gradient Regularized Newton Boosting Trees with Global Convergence
Nikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon · 2026
Read Gradient Regularized Newton Boosting Trees with Global Convergence by Nikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00581.txt.
Description
Nikita Zozoulenko and 3 co-authors put Newton boosting, the second-order engine inside the dominant tree-boosting libraries, on firm theoretical ground. Those libraries fit decision trees stage by stage, and second-order steps are their secret, yet global convergence of Newton boosting was poorly understood compared with first-order methods. The authors frame the problem as convex optimization with Newton's method on function spaces, allowing inexact steps.
Within that framework, ordinary Newton boosting provably converges linearly for smooth, strongly convex losses meeting a Hessian condition. For general convex losses, they add a gradient-regularized Newton scheme whose adaptive penalty scales with the square root of the gradient's size. That variant converges at a quadratic-in-denominator rate, matching first-order boosting with momentum, and is therefore a globally convergent second-order boosting algorithm. In experiments their scheme converged on problems where the vanilla method could diverge.
Catalog record
| CREATOR | Nikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00581