Self-Scaled Broyden Family of Quasi-Newton Methods in JAX
Ivan Bioli, Mikel Mendibe Abarrategi · 2026
Read Self-Scaled Broyden Family of Quasi-Newton Methods in JAX by Ivan Bioli, Mikel Mendibe Abarrategi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10599.txt.
Introduction
Ivan Bioli and Mikel Mendibe Abarrategi document a JAX package for quasi-Newton methods in the self-scaled Broyden family, built on Optimistix. It covers BFGS, DFP, and Broyden methods with their self-scaled variants, plus L-BFGS, and adds a zoom line search meeting the strong Wolfe conditions. The authors bill it as a short technical note rather than a research paper: no new method is claimed. Its aim is to ease adoption of these optimizers in the JAX community, with installs via PyPI.
Catalog record
| CREATOR | Ivan Bioli, Mikel Mendibe Abarrategi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 7713 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Scaled Broyden Family of Quasi-Newton Methods in JAX
Ivan Bioli, Mikel Mendibe Abarrategi
Abstract
We present a JAX implementation of the Self-Scaled Broyden family of quasi-Newton methods, fully compatible with JAX and building on the Optimistix [ 4 ] optimisation library. The implementation includes BFGS, DFP, Broyden and their Self-Scaled variants (SSBFGS, SSDFP, SSBroyden), together with a Zoom line search satisfying the strong Wolfe conditions. This is a short technical note, not a research paper, as it does not claim any novel contribution; its purpose is to document the implementation and ease the adoption of these optimisers within the JAX community. The code is available at https://github.com/IvanBioli/ssbroyden_optimistix.git .
1
Introduction
Optimistix
[
4
]
is a JAX library for nonlinear solvers
providing modular and composable optimisation algorithms. While Optimistix
includes a standard BFGS implementation paired with a backtracking Armijo line
search, it lacks both the Zoom line search, which satisfies the strong Wolfe
conditions, and the broader family of Self-Scaled Broyden
methods
[
1
,
3
,
2
]
. This work provides a pure-JAX implementation of these
methods, designed to be fully compatible with the Optimistix solver interface:
the new solvers can be used as drop-in replacements, composed with existing
Optimistix descents and searches, and benefit from all JAX transformations.
Specifically, our implementation addresses the following gaps in the current
Optimistix offerings:
1.
Zoom line search.
We integrate the Zoom line search
(Algorithm 3.6 in
[
5
]
) into Optimistix,
ensuring that the strong Wolfe conditions are satisfied at each step.
The implementation is adapted from
bagibence/zoom_linesearch
1
1
1
https://github.com/bagibence/optimistix/tree/zoom_linesearch
with minor modifications to fit the new Optimistix interface.
2.
Self-Scaled Broyden family.
We implement the full Self-Scaled
Broyden family of quasi-Newton Hessian updates, encompassing Broyden,
BFGS, DFP, and their Self-Scaled variants SSBroyden, SSBFGS, and
SSDFP.
3.
Iteration counting.
We provide a wrapper that distinguishes
between actual quasi-Newton iterations and internal line search steps,
which Optimistix does not separate by design choice. This allows for
more refined comparisons between solvers.
2
The Self-Scaled Broyden Family
The Self-Scaled Broyden family of quasi-Newton methods generalises the classic
Broyden, BFGS, and DFP updates to minimize a function
f:\mathbb{R}^{N}\to\mathbb{R}
[
1
,
3
,
2
]
. At each iteration
k
, these methods maintain
an approximation
H_{k}
of the inverse Hessian of
f
and compute a search
direction
d_{k}=-H_{k}\nabla f(x_{k})
. After a line search determines a step size
\alpha_{k}
, the iterate is updated as
x_{k+1}=x_{k}+\alpha_{k}d_{k}
. The
inverse Hessian approximation is then updated using the step
s_{k}=x_{k+1}-x_{k}
and the gradient difference
y_{k}=\nabla f(x_{k+1})-\nabla f(x_{k})
.
The Self-Scaled Broyden family update is parameterised by two scalars,
\theta_{k}
and
\tau_{k}
. In its most general form, the update is given by
H_{k+1}=\frac{1}{\tau_{k}}\left(H_{k}-\frac{H_{k}y_{k}y_{k}^{\top}H_{k}}{y_{k}^{\top}H_{k}y_{k}}+\phi_{k}\,(y_{k}^{\top}H_{k}y_{k})\,v_{k}v_{k}^{\top}\right)+\rho_{k}\,s_{k}s_{k}^{\top},
(1)
where
\begin{split}\rho_{k}&=\frac{1}{y_{k}^{\top}s_{k}},\\
v_{k}&=\frac{s_{k}}{y_{k}^{\top}s_{k}}-\frac{H_{k}y_{k}}{y_{k}^{\top}H_{k}y_{k}},\\
\phi_{k}&=\frac{1-\theta_{k}}{1+(h_{k}b_{k}-1)\theta_{k}},\\
b_{k}&=\frac{s_{k}^{\top}B_{k}s_{k}}{y_{k}^{\top}s_{k}},\\
h_{k}&=\frac{y_{k}^{\top}H_{k}y_{k}}{y_{k}^{\top}s_{k}}.\end{split}
The parameter
\theta_{k}
interpolates between the BFGS (
\theta_{k}=0
) and DFP
(
\theta_{k}=1
) updates, with the more general Broyden family obtained by
computing
\theta_{k}
dynamically at each iteration as:
\theta_{k}=\max\left(\theta_{k}^{-},\,\min\left(\theta_{k}^{+},\,\frac{1-b_{k}}{b_{k}}\right)\right),
where, letting
a_{k}=b_{k}h_{k}-1
and
c_{k}=\left(\frac{a_{k}}{1+a_{k}}\right)^{1/2}
,
\rho_{k}^{-}=\min\bigl(1,\,h_{k}(1-c_{k})\bigr),\qquad\theta_{k}^{-}=\frac{\rho_{k}^{-}-1}{a_{k}},\qquad\theta_{k}^{+}=\frac{1}{\rho_{k}^{-}}.
The parameter
\tau_{k}
controls the Self-Scaled variant (
\tau_{k}=1
means no
scaling) and is computed as
\tau_{k}=\begin{cases}\min\bigl(\rho_{k}^{+}\,\sigma_{k}^{(1-N)},\,\sigma_{k}\bigr),&\text{if }\theta_{k}\leq 0,\\[4.0pt]
\rho_{k}^{+}\,\min\bigl(\sigma_{k}^{(1-N)},\,\tfrac{1}{\theta_{k}}\bigr),&\text{otherwise,}\end{cases}
where
\rho_{k}^{+}=\min\left(1,\,\frac{1}{b_{k}}\right),\qquad\sigma_{k}=1+\theta_{k}\,a_{k},\qquad\sigma_{k}^{(1-N)}=|\sigma_{k}|^{\frac{1}{1-N}}.
Table
1
summarises the six concrete solvers
obtained by choosing
\theta_{k}
and
\tau_{k}
.
Table 1:
Solvers implemented as special cases of the Self-Scaled Broyden family.
Solver
\theta_{k}
\tau_{k}
Description
BFGS
0
1
Classic BFGS
SSBFGS
0
computed
Self-Scaled BFGS
DFP
1
1
Classic DFP
SSDFP
1
computed
Self-Scaled DFP
Broyden
computed
1
Broyden family (no scaling)
SSBroyden
computed
computed
Self-Scaled Broyden family
2.1
Software Design
The implementation follows a class hierarchy that mirrors the mathematical
structure of the update family, building on top of the
AbstractQuasiNewton
base class already present in Optimistix:
1.
AbstractSSBroydenFamily
implements the shared logic: Hessian
initialisation, computation of the auxiliary quantities, and the
dispatch to subclass-specific update terms. It exposes two hooks:
_compute_thetak
and
_compute_tauk
, which
subclasses override to fix or compute
\theta_{k}
and
\tau_{k}
.
2.
AbstractSSBroyden
implements the general update
term (
1
) and computes both
\theta_{k}
and
\tau_{k}
dynamically.
AbstractBroyden
inherits from it but overrides
_compute_tauk
\equiv 1
.
3.
AbstractSSBFGS
fixes
\theta_{k}=0
, which simplifies the
update to the BFGS Woodbury form.
AbstractBFGS
specialises
further with
\tau_{k}=1
.
4.
AbstractSSDFP
fixes
\theta_{k}=1
, eliminating the
v_{k}
term.
AbstractDFP
specialises further with
\tau_{k}=1
.
Each abstract class has a concrete counterpart (e.g.
BFGS
,
SSBFGS
) that binds a default descent (
NewtonDescent
) and a
default search (
Zoom
line search). Users can subclass the abstract
variants to plug in alternative descents or searches.
3
Numerical Example: PINNs for the 3D Poisson Equation
The SSBroyden family of optimizers has recently shown improved performance over
BFGS for Physics Informed Neural Networks (PINNs)
[
2
]
.
In this numerical example, available as
example.py
in our repository,
we solve the Poisson equation
-\Delta u=f
on
\Omega=[0,1]^{3}
with
Dirichlet boundary conditions, where the exact solution is
u^{*}(x)=\prod_{i=1}^{3}\sin(\pi x_{i})
. The solution is approximated by a fully connected
neural network (3 hidden layers of 32 units with
\tanh
activations). The loss
function is
\mathcal{L}(\theta)=\frac{1}{2N_{\Omega}}\sum_{i=1}^{N_{\Omega}}\bigl(\Delta u_{\theta}(x_{i})+f(x_{i})\bigr)^{2}+\frac{1}{2N_{\Gamma}}\sum_{j=1}^{N_{\Gamma}}\bigl(u_{\theta}(x_{j})-u^{*}(x_{j})\bigr)^{2},
with
N_{\Omega}=5000
interior and
N_{\Gamma}=800
boundary collocation
points, sampled uniformly at random.
Figure
1
compares the convergence of the implemented solvers
(BFGS, SSBFGS, Broyden, SSBroyden). The self-scaled variants converge
notably faster in terms of both loss reduction and relative
\mathrm{L}^{2}
and
\mathrm{H}^{1}
errors.
Figure 1:
Convergence of quasi-Newton solvers on the 3D Poisson PINN problem.
The self-scaled variants (SSBFGS, SSBroyden) achieve lower errors in
fewer iterations compared to the standard BFGS and Broyden methods.