Self-Scaled Broyden Family of Quasi-Newton Methods in JAX

Ivan Bioli, Mikel Mendibe Abarrategi · 2026

Read Self-Scaled Broyden Family of Quasi-Newton Methods in JAX by Ivan Bioli, Mikel Mendibe Abarrategi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10599.txt.

Introduction

Ivan Bioli and Mikel Mendibe Abarrategi document a JAX package for quasi-Newton methods in the self-scaled Broyden family, built on Optimistix. It covers BFGS, DFP, and Broyden methods with their self-scaled variants, plus L-BFGS, and adds a zoom line search meeting the strong Wolfe conditions. The authors bill it as a short technical note rather than a research paper: no new method is claimed. Its aim is to ease adoption of these optimizers in the JAX community, with installs via PyPI.

Catalog record

CREATORIvan Bioli, Mikel Mendibe Abarrategi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE7713
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10599

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Scaled Broyden Family of Quasi-Newton Methods in JAX
Ivan Bioli, Mikel Mendibe Abarrategi

Abstract
We present a JAX implementation of the Self-Scaled Broyden family of quasi-Newton methods, fully compatible with JAX and building on the Optimistix [ 4 ] optimisation library. The implementation includes BFGS, DFP, Broyden and their Self-Scaled variants (SSBFGS, SSDFP, SSBroyden), together with a Zoom line search satisfying the strong Wolfe conditions. This is a short technical note, not a research paper, as it does not claim any novel contribution; its purpose is to document the implementation and ease the adoption of these optimisers within the JAX community. The code is available at https://github.com/IvanBioli/ssbroyden_optimistix.git .



1
Introduction

Optimistix
[
4
]
is a JAX library for nonlinear solvers
providing modular and composable optimisation algorithms. While Optimistix
includes a standard BFGS implementation paired with a backtracking Armijo line
search, it lacks both the Zoom line search, which satisfies the strong Wolfe
conditions, and the broader family of Self-Scaled Broyden
methods
[
1
,
3
,
2
]
. This work provides a pure-JAX implementation of these
methods, designed to be fully compatible with the Optimistix solver interface:
the new solvers can be used as drop-in replacements, composed with existing
Optimistix descents and searches, and benefit from all JAX transformations.
Specifically, our implementation addresses the following gaps in the current
Optimistix offerings:

1.

Zoom line search.
We integrate the Zoom line search
(Algorithm 3.6 in
[
5
]
) into Optimistix,
ensuring that the strong Wolfe conditions are satisfied at each step.
The implementation is adapted from

bagibence/zoom_linesearch
1
1

1

https://github.com/bagibence/optimistix/tree/zoom_linesearch

with minor modifications to fit the new Optimistix interface.

2.

Self-Scaled Broyden family.
We implement the full Self-Scaled
Broyden family of quasi-Newton Hessian updates, encompassing Broyden,
BFGS, DFP, and their Self-Scaled variants SSBroyden, SSBFGS, and
SSDFP.

3.

Iteration counting.
We provide a wrapper that distinguishes
between actual quasi-Newton iterations and internal line search steps,
which Optimistix does not separate by design choice. This allows for
more refined comparisons between solvers.

2
The Self-Scaled Broyden Family

The Self-Scaled Broyden family of quasi-Newton methods generalises the classic
Broyden, BFGS, and DFP updates to minimize a function

f:\mathbb{R}^{N}\to\mathbb{R}

[
1
,
3
,
2
]
. At each iteration
k
, these methods maintain
an approximation
H_{k}
of the inverse Hessian of
f
and compute a search
direction
d_{k}=-H_{k}\nabla f(x_{k})
. After a line search determines a step size

\alpha_{k}
, the iterate is updated as
x_{k+1}=x_{k}+\alpha_{k}d_{k}
. The
inverse Hessian approximation is then updated using the step
s_{k}=x_{k+1}-x_{k}
and the gradient difference
y_{k}=\nabla f(x_{k+1})-\nabla f(x_{k})
.

The Self-Scaled Broyden family update is parameterised by two scalars,

\theta_{k}
and
\tau_{k}
. In its most general form, the update is given by

H_{k+1}=\frac{1}{\tau_{k}}\left(H_{k}-\frac{H_{k}y_{k}y_{k}^{\top}H_{k}}{y_{k}^{\top}H_{k}y_{k}}+\phi_{k}\,(y_{k}^{\top}H_{k}y_{k})\,v_{k}v_{k}^{\top}\right)+\rho_{k}\,s_{k}s_{k}^{\top},

(1)

where

\begin{split}\rho_{k}&=\frac{1}{y_{k}^{\top}s_{k}},\\
v_{k}&=\frac{s_{k}}{y_{k}^{\top}s_{k}}-\frac{H_{k}y_{k}}{y_{k}^{\top}H_{k}y_{k}},\\
\phi_{k}&=\frac{1-\theta_{k}}{1+(h_{k}b_{k}-1)\theta_{k}},\\
b_{k}&=\frac{s_{k}^{\top}B_{k}s_{k}}{y_{k}^{\top}s_{k}},\\
h_{k}&=\frac{y_{k}^{\top}H_{k}y_{k}}{y_{k}^{\top}s_{k}}.\end{split}

The parameter
\theta_{k}
interpolates between the BFGS (
\theta_{k}=0
) and DFP
(
\theta_{k}=1
) updates, with the more general Broyden family obtained by
computing
\theta_{k}
dynamically at each iteration as:

\theta_{k}=\max\left(\theta_{k}^{-},\,\min\left(\theta_{k}^{+},\,\frac{1-b_{k}}{b_{k}}\right)\right),

where, letting
a_{k}=b_{k}h_{k}-1
and
c_{k}=\left(\frac{a_{k}}{1+a_{k}}\right)^{1/2}
,

\rho_{k}^{-}=\min\bigl(1,\,h_{k}(1-c_{k})\bigr),\qquad\theta_{k}^{-}=\frac{\rho_{k}^{-}-1}{a_{k}},\qquad\theta_{k}^{+}=\frac{1}{\rho_{k}^{-}}.

The parameter
\tau_{k}
controls the Self-Scaled variant (
\tau_{k}=1
means no
scaling) and is computed as

\tau_{k}=\begin{cases}\min\bigl(\rho_{k}^{+}\,\sigma_{k}^{(1-N)},\,\sigma_{k}\bigr),&\text{if }\theta_{k}\leq 0,\\[4.0pt]
\rho_{k}^{+}\,\min\bigl(\sigma_{k}^{(1-N)},\,\tfrac{1}{\theta_{k}}\bigr),&\text{otherwise,}\end{cases}

where

\rho_{k}^{+}=\min\left(1,\,\frac{1}{b_{k}}\right),\qquad\sigma_{k}=1+\theta_{k}\,a_{k},\qquad\sigma_{k}^{(1-N)}=|\sigma_{k}|^{\frac{1}{1-N}}.

Table
1
summarises the six concrete solvers
obtained by choosing
\theta_{k}
and
\tau_{k}
.

Table 1:
Solvers implemented as special cases of the Self-Scaled Broyden family.

Solver

\theta_{k}

\tau_{k}

Description

BFGS

0

1

Classic BFGS

SSBFGS

0

computed

Self-Scaled BFGS

DFP

1

1

Classic DFP

SSDFP

1

computed

Self-Scaled DFP

Broyden

computed

1

Broyden family (no scaling)

SSBroyden

computed

computed

Self-Scaled Broyden family

2.1
Software Design

The implementation follows a class hierarchy that mirrors the mathematical
structure of the update family, building on top of the

AbstractQuasiNewton
base class already present in Optimistix:

1.

AbstractSSBroydenFamily
implements the shared logic: Hessian
initialisation, computation of the auxiliary quantities, and the
dispatch to subclass-specific update terms. It exposes two hooks:

_compute_thetak
and
_compute_tauk
, which
subclasses override to fix or compute
\theta_{k}
and
\tau_{k}
.

2.

AbstractSSBroyden
implements the general update
term (
1
) and computes both
\theta_{k}
and
\tau_{k}

dynamically.
AbstractBroyden
inherits from it but overrides

_compute_tauk

\equiv 1
.

3.

AbstractSSBFGS
fixes
\theta_{k}=0
, which simplifies the
update to the BFGS Woodbury form.
AbstractBFGS
specialises
further with
\tau_{k}=1
.

4.

AbstractSSDFP
fixes
\theta_{k}=1
, eliminating the
v_{k}

term.
AbstractDFP
specialises further with
\tau_{k}=1
.

Each abstract class has a concrete counterpart (e.g.
BFGS
,

SSBFGS
) that binds a default descent (
NewtonDescent
) and a
default search (
Zoom
line search). Users can subclass the abstract
variants to plug in alternative descents or searches.

3
Numerical Example: PINNs for the 3D Poisson Equation

The SSBroyden family of optimizers has recently shown improved performance over
BFGS for Physics Informed Neural Networks (PINNs)
[
2
]
.
In this numerical example, available as
example.py
in our repository,
we solve the Poisson equation
-\Delta u=f
on
\Omega=[0,1]^{3}
with
Dirichlet boundary conditions, where the exact solution is
u^{*}(x)=\prod_{i=1}^{3}\sin(\pi x_{i})
. The solution is approximated by a fully connected
neural network (3 hidden layers of 32 units with
\tanh
activations). The loss
function is

\mathcal{L}(\theta)=\frac{1}{2N_{\Omega}}\sum_{i=1}^{N_{\Omega}}\bigl(\Delta u_{\theta}(x_{i})+f(x_{i})\bigr)^{2}+\frac{1}{2N_{\Gamma}}\sum_{j=1}^{N_{\Gamma}}\bigl(u_{\theta}(x_{j})-u^{*}(x_{j})\bigr)^{2},

with
N_{\Omega}=5000
interior and
N_{\Gamma}=800
boundary collocation
points, sampled uniformly at random.

Figure
1
compares the convergence of the implemented solvers
(BFGS, SSBFGS, Broyden, SSBroyden). The self-scaled variants converge
notably faster in terms of both loss reduction and relative
\mathrm{L}^{2}
and
\mathrm{H}^{1}

errors.

Figure 1:
Convergence of quasi-Newton solvers on the 3D Poisson PINN problem.
The self-scaled variants (SSBFGS, SSBroyden) achieve lower errors in
fewer iterations compared to the standard BFGS and Broyden methods.

Read the complete text