Factored Levenberg–Marquardt for Diffeomorphic Image Registration: An efficient optimizer for FireANTs

Rohit Jena [email protected], Pratik Chaudhari [email protected], James C. Gee [email protected] · 2026

Read Factored Levenberg–Marquardt for Diffeomorphic Image Registration: An efficient optimizer for FireANTs by Rohit Jena [email protected], Pratik Chaudhari [email protected], James C. Gee [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19371.txt.

Introduction

Jena and co-authors cut the memory cost of aligning medical images with a new optimizer. FireANTs aligns images as a test-time task, but its default Adam optimizer stores costly state. Their factored Levenberg-Marquardt variant keeps just one damping value as state. A trust region rule tunes that value as the fit proceeds. Memory drops by up to 24.6 percent on large volumes. One setting tuned on brain scans transfers to lung and abdomen tasks. It matches or beats Adam on three of four test sets.

Catalog record

CREATORRohit Jena [email protected], Pratik Chaudhari [email protected], James C. Gee [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30529
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19371

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Factored Levenberg–Marquardt for Diffeomorphic Image Registration: An efficient optimizer for FireANTs Rohit Jena [email protected], Pratik Chaudhari [email protected], James C. Gee [email protected] Abstract FireANTs ( Jena et al. 2026 ) introduced a novel Eulerian descent method for plug-and-play behavior with arbitrary optimizers adapted for diffeomorphic image registration as a test-time optimization problem, with a GPU-accelerated implementation. FireANTs uses Adam as its default optimizer for fast and more robust optimization. However, Adam requires storing state variables (i.e. momentum and squared-momentum estimates), each of which can consume significant memory, prohibiting its use for significantly large images. In this work, we propose a modified Levenberg–Marquardt (LM) optimizer that requires only a single scalar damping parameter as optimizer state, that is adaptively tuned using a trust region approach. The resulting optimizer reduces memory by up to 24.6 % 24.6\% for large volumes, and retaining performance across all four datasets. A single hyperparameter configuration tuned on brain MRI transfers without modification to lung CT and cross-modal abdominal registration, matching or outperforming Adam on three of four benchmarks. We also perform ablations on the effectiveness of using Metropolis-Hastings style rejection step to prevent updates that worsen the loss function. 1 Introduction FireANTs ( Jena et al. 2026 ) frames diffeomorphic image registration as GPU-accelerated test-time optimization: a displacement field \mathbf{u}:\Omega\to\mathbb{R}^{d} is optimized by minimizing \mathcal{L}(\mathbf{u})=\mathcal{L}_{\text{sim}}(F,M\circ\phi)+\mathcal{R}(\mathbf{u}), (1) where \mathcal{L}_{\text{sim}} is an image similarity loss and \mathcal{R} regularizes the deformation \phi=\mathrm{Id}+\mathbf{u} . Adam ( Kingma & Ba 2015 ) is the default optimizer, and it works well. But Adam is a quasi-second-order method: it exploits gradient history through exponential moving averages. These moving averages can consume significant additional memory and runtime proportional to the voxels in the image, limiting their use for datasets with large volumes. Although Adam is much cheaper than other second-order methods like L-BFGS ( Liu & Nocedal 1989 ) , it could still introduce a significant overhead for large images. This motivates the use of a more efficient second-order method that is not as memory-intensive. Levenberg–Marquardt (LM) ( Levenberg 1944 ; Marquardt 1963 ) is a well-known algorithm: a damped Gauss-Newton step that interpolates between Newton’s method (fast near optima) and gradient descent (robust far from them). However, the full \mathbf{J}^{\top}\mathbf{J} Hessian is intractable for dense warp fields due to its scale: a 192^{3} displacement field has \sim 5\times 10^{13} parameters. We show that the local structure of the registration loss makes a per-voxel rank-1 Hessian factorization both principled and sufficient, yielding a closed-form update that is as cheap to compute as Adam yet demonstrates second-order behavior. To evaluate our approach, we conduct experiments on four publicly available benchmarks: brain MRI (LUMIR and OASIS), lung CT (NLST), and cross-modal abdominal MR-to-CT registration (Abdomen-L2R). We compare our factored Levenberg–Marquardt (LM) optimizer against Adam, the default optimizer in FireANTs, using the same registration framework and similarity metric across datasets, showing slight improvements in performance without recording any additional state. Our ablation on synthetically generated volumes demonstrate that LM not only reduces memory consumption by up to 24.6\% for large volumes, but also matches or surpasses Adam in registration accuracy using a single hyperparameter configuration transferred across imaging modalities and organs. We additionally provide ablations that highlight the effectiveness of a Metropolis-Hastings-style rejection step, which prevents loss-increasing updates and contributes to the optimizer’s stability. 2 Factored Levenberg–Marquardt for Warp Fields 2.1 Compositive Diffeomorphic Updates with Adaptive Optimization FireANTs maintains diffeomorphism throughout optimization via a compositive update rule. Given a velocity \mathbf{v}^{(t)} produced by the optimizer, the warp is updated as \mathbf{u}^{(t+1)}=\epsilon\mathbf{v}^{(t)}+\mathbf{u}^{(t)}\circ(\mathrm{Id}+\epsilon\mathbf{v}^{(t)}), (2) where \epsilon is a small positive constant to ensure that \mathrm{Id}+\epsilon\mathbf{v}^{(t)} is diffeomorphic. We defer the reader to Jena et al. 2026 for the choice of \epsilon for maintaining diffeomorphic behavior. The quantity \mathbf{u}^{(t)}\circ(\mathrm{Id}+\mathbf{v}^{(t)}) resamples the current warp at locations displaced by \mathbf{v}^{(t)} . Optimization proceeds in a coarse-to-fine pyramid; at each level the warp inherits the solution from the previous level, consistent with recommended practice in literature on multi-resolution optimization ( Gee et al. 1991 ; Jena et al. 2026 ; Avants et al. 2008 ; Jena et al. 2025 ) . Since diffeomorphisms lie on a Riemannian manifold and admit a group structure, deploying adaptive optimization (that are generally defined for Euclidean spaces) of \mathbf{v}^{(t)} is a non-trivial problem. Jena et al. 2026 also proposes a novel Eulerian descent method for plug-and-play behavior of arbitrary optimizers for diffeomorphic image registration bypassing the need for expensive parallel transport operations by leverging the group structure of diffeomorphisms. 2.2 Factored Levenberg–Marquardt for Diffeomorphic Registration Classical LM and the scale problem. For a least-squares residual \mathbf{r}(\bm{\theta}) with Jacobian \mathbf{J} , the LM step solves \bigl(\mathbf{J}^{\top}\mathbf{J}+\lambda\mathbf{I}\bigr)\Delta\bm{\theta}=-\mathbf{J}^{\top}\mathbf{r}, (3) with \lambda\geq 0 the damping parameter. For \lambda=0 this is the Gauss–Newton step; as \lambda\to\infty the step shrinks toward -\mathbf{g}/\lambda , recovering gradient descent. For image registration, we typically optimize a single scalar loss function (ignoring regularization terms) \mathcal{L}(\mathbf{u}) which does not necessarily admit a squared loss form required for LM. However, one can convert the problem into a least-squares form by rewriting commonly used loss functions in a squared loss form. 1. Mean Squared Error (MSE): This is already in a squared loss form. 2. Local Normalized Cross Correlation (LNCC): This can be rewritten as a squared loss form by minimizing (1-\text{LNCC})^{2} instead of minimizing -\text{LNCC} . 3. Mutual Information (MI): This can be rewritten as a squared loss form by minimizing (\log_{2}(B)-\text{MI})^{2} instead of minimizing -\text{MI} , where B is the number of bins for the parzen windowing method. Note that \log_{2}(B) upper-bounds the discrete mutual information loss, so the squared loss function will always maximize the mutual information as well. Therefore, we only have a single scalar residual {r}(\mathbf{u}) (abbreviated as r for brevity) and the Jacobian \mathbf{J}(\mathbf{u}) is a vector of the same length as the displacement field (and is actually the gradient of the loss function with respect to the displacement field). Forming \mathbf{J}^{\top}\mathbf{J} for a 3D displacement field with more than \sim 10^{7} parameters is intractable. Rank-1 factorization per voxel. The key observation is that the registration gradient \mathbf{g}\in\mathbb{R}^{H\times W\times D\times d} at each voxel \mathbf{x} depends primarily on local image content. We therefore approximate the Gauss–Newton Hessian independently per voxel by the rank-1 outer product \hat{\mathbf{H}}_{\mathbf{x}}=\mathbf{g}_{\mathbf{x}}\mathbf{g}_{\mathbf{x}}^{\top} . The damped inverse of this 3\times 3 matrix has a closed form (proof in appendix A ): \Delta\mathbf{u}_{\mathbf{x}}=-\frac{r\mathbf{g}_{\mathbf{x}}}{\|\mathbf{g}_{\mathbf{x}}\|^{2}+\lambda}. (4) This is the pointwise update (denoted as tile_size =1 in our implementation): it normalizes each voxel’s gradient step by its magnitude, preserving direction while suppressing updates at voxels with small, unreliable gradients. No matrix storage or inversion is required beyond a scalar norm computation. For richer spatial context, one can pool rank-1 approximations over a window of (2w+1)^{d} voxels (where w is the radius of the window, and d is the dimension of the displacement field) to obtain a d\times d positive semi-definite \hat{\mathbf{H}}_{\text{tile}} , then apply \Delta\mathbf{u}_{\mathbf{x}}=-\bigl(\hat{\mathbf{H}}_{\text{tile}}+\lambda\mathbf{I}\bigr)^{-1}r\mathbf{g}_{\mathbf{x}}, (5) with the 3\times 3 inverse computed via torch.linalg.inv . In practice, tile_size =1 matches or exceeds larger tiles at lower cost (Section 3 ). Adaptive damping. We adapt \lambda after each step by comparing successive losses: \lambda^{(t)}=\begin{cases}\mu^{+}\cdot\lambda^{(t-1)}&\text{if }\mathcal{L}^{(t)}>\mathcal{L}^{(t-1)}\quad\text{(bad step)},\\ \mu^{-}\cdot\lambda^{(t-1)}&\text{otherwise}\quad\text{(good step)},\end{cases} (6) with \mu^{+}>1 and 0<\mu^{-}<1 . This is a simple heuristic for trust-region-based algorithms. The LM step \Delta\mathbf{u} is

Read the complete text