AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning

Ziye Chen Hongbin Lin Chenyu Zhang Xiangda Yan Yongjie Yang Yao Shu · 2026

Read AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning by Ziye Chen Hongbin Lin Chenyu Zhang Xiangda Yan Yongjie Yang Yao Shu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19767.txt.

Introduction

Chen and colleagues resolve a rank paradox in zeroth order tuning with LoRA adapters for large language models. Zeroth order methods avoid storing backpropagation activations, but raising adapter rank blurs the finite difference signal. Their AR1-ZO queries one rank one atom at a time with topology aware scaling, restoring rank invariant signal with no extra passes. Theory proves atom minimality. Tests on OPT and Qwen3 models show high rank LoRA working well under matched budgets.

Catalog record

CREATORZiye Chen Hongbin Lin Chenyu Zhang Xiangda Yan Yongjie Yang Yao Shu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88612
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19767

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning Ziye Chen Hongbin Lin Chenyu Zhang Xiangda Yan Yongjie Yang Yao Shu Abstract Zeroth-order (ZO) optimization enables large-language-model fine-tuning without storing backpropagation activations, while LoRA supplies compact trainable adapters. Combining them creates a rank paradox: increasing LoRA rank improves adapter capacity, but standard two-point ZO either perturbs a rank-dependent number of coordinates or, under atomwise updates, can make the finite-difference signal unobservable. This paper shows that the bottleneck is a measurement-topology problem rather than a need for an external subspace. LoRA already decomposes into matched rank- 1 1 atoms, each a complete factor-coordinate block of dimension d out + d in d_{\text{out}}+d_{\text{in}} . Querying one atom per step keeps the stored adapter rank r r while removing r r from the single-query perturbation dimension. The naive atomwise query is still miscalibrated: if it inherits canonical LoRA scaling α / r \alpha/r , the active finite-difference signal shrinks as 1 / r 1/r and the active finite-difference signal-to-noise ratio (FD-SNR) as 1 / r 2 1/r^{2} , producing directional collapse under a fixed residual evaluation-noise floor. AR1-ZO pairs alternating rank- 1 1 atom queries with topology-aware scaling γ = α ​ r \gamma=\alpha r , restoring rank-invariant active signal without auxiliary bases, activation hooks, curvature estimates, or extra forward queries. Theory proves atom minimality, rank-independent active query dimension, directional collapse and restoration, and the remaining rank dependence as an amortized coverage cost. Experiments on OPT and Qwen3 models validate the signal mechanism and show that AR1-ZO makes high-rank LoRA effective among matched-budget ZO methods under the standard two-forward-pass query budget. 1 Introduction Fine-tuning large language models (LLMs) under tight memory budgets requires controlling both activation storage and trainable degrees of freedom. Zeroth-order (ZO) optimization, exemplified by MeZO mezo , removes the activation-memory cost of backpropagation by estimating descent directions from forward-pass scalar differences. LoRA hu2022lora , together with related low-rank adaptation variants such as QLoRA QLoRA , DoRA DoRA , and AdaLoRA adalora , reduces the number of trainable coordinates by freezing pretrained weights and learning compact low-rank updates. Their combination is therefore natural, but it turns the LoRA rank into a double-edged quantity: the same rank that gives the adapter capacity can also determine what a two-point ZO oracle has to perturb and whether its scalar loss difference remains visible. The usual explanation is the curse of dimensionality. Classical randomized gradient estimators incur variance that grows with the perturbation dimension nesterov2017random ; liu2018zeroth ; if a ZO query perturbs all LoRA factors at once, the dimension grows with rank. Existing remedies reduce the effective dimension through block-coordinate perturbations MeZOBCD , random subspaces yu2025subzero , low-rank estimators LOZO ; seung2025loren ; sun2025tezo , LoRA-manifold tangent updates song2026rozo , activation-guided perturbations lin2026agzo , and second-order reweighting zhao2025secondorder . Recent low-rank and geometry-aware methods such as TeZO and RoZO further show that temporal low-rank structure and LoRA-manifold geometry can improve ZO fine-tuning, but they still leave open the atom-level measurement unit and scale of a two-forward-pass LoRA-ZO oracle. In parallel, LoRA scaling and optimization studies such as rsLoRA kala2023rslora , RoRA liu2025rora , and LoRA+ hayou2024lora refine rank normalization or factor-wise dynamics under first-order joint updates. These lines establish that both query geometry and scaling matter, but they do not settle the measurement unit of black-box LoRA-ZO: Can the rank- 1 atoms already inside LoRA be used as complete two-forward-pass ZO query blocks, and what scaling keeps those atomwise measurements observable at high rank? A fuller discussion of the related literature and novelty boundary appears in Appendix A . Our answer starts by separating three roles that rank often conflates: stored adapter capacity, single-query perturbation dimension, and finite-difference signal scale. The LoRA product can be written as a sum of rank- 1 atoms, but the contribution is not this standard algebraic identity. The key object is the matched column-row pair (\bm{b}_{k},\bm{a}_{k}) , which is the finest complete LoRA-native factor block that can be exposed to a two-point oracle: perturbing only one side is incomplete, while grouping multiple atoms reintroduces a larger query dimension. An atomwise query therefore keeps the adapter rank r in storage and in long-run coverage, but changes the instantaneous ZO problem from a rank- r factor perturbation to a single complete atom perturbation. This removes the first bottleneck without invoking an auxiliary basis, activation hook, curvature estimate, or external subspace. The same native query topology also reveals why the obvious atomwise optimizer fails. Canonical LoRA scaling uses the coefficient \alpha/r , calibrated for joint updates in which all r atoms contribute together. Under a sequential rank- 1 ZO measurement, only one atom differs between the positive and negative endpoints, so inheriting the joint coefficient suppresses the active finite-difference numerator by 1/r . Residual evaluation noise is not reduced by the same factor, making the active FD-SNR decay as 1/r^{2} and eventually erasing directional fidelity before any learning-rate multiplier can act. We call this failure directional collapse induced by a topology-scaling mismatch . AR1-ZO resolves it by assigning the queried atom a rank-invariant active coefficient through the topology-aware calibration \gamma=\alpha r . This is not a new joint LoRA scaling rule; it is a measurement calibration for the sequential two-point oracle. The resulting optimizer, Alternating Rank-1 Zeroth-Order optimization (AR1-ZO), cycles through complete rank- 1 atoms while using the corrected active measurement scale. Each step still uses the standard two forward evaluations, and in transformer implementations the same active rank index can be shared across adapted matrices. The theoretical accounting then becomes precise: rank no longer enters the single-query perturbation dimension or the active FD-SNR; it remains as the amortized cost of revisiting all stored atoms. This yields three contributions. • High rank need not be a single-query ZO dimension tax. We formalize the matched atom (\bm{b}_{k},\bm{a}_{k}) as the minimal complete LoRA-native query block and show that its active two-point estimator has perturbation dimension d_{\text{out}}+d_{\text{in}} , not r(d_{\text{out}}+d_{\text{in}}) , while preserving a rank- r adapter. • The high-rank failure of naive atomwise ZO is a measurement collapse. We prove that inheriting the joint-update scale \alpha/r makes active FD-SNR decay as 1/r^{2} , derive the topology-aware correction \gamma=\alpha r that restores rank-invariant active signal, and extend the accounting to rank- m blocks and other joint-normalized factorizations. • AR1-ZO makes the accounting operational under a pure black-box budget. We instantiate alternating atom queries with two forward passes per step, prove the remaining rank dependence is an amortized coverage cost and characterize when aligned native atoms carry dense descent signal, then validate the mechanism with spectral-alignment diagnostics, rank-dependent active FD-SNR sweeps, downstream tasks, scaling-law controls, and efficiency measurements. Figure 1: The Optimization Pathology of LoRA-ZO and the AR1-ZO Solution. 2 Preliminaries: LoRA and Zeroth-Order Optimization Applying ZO to LoRA is governed by three separable quantities: the trainable low-rank coordinates and output coefficient, the coordinates moved by one two-point query, and the evaluation noise that determines whether the scalar loss difference is observable. A black-box LoRA optimizer therefore faces two constraints: the queried coordinate set should not grow with rank, and the finite-difference numerator should not be scaled below the noise floor. LoRA parameterization and rank scaling. LoRA turns dense adaptation into a factored low-rank parameterization. For

Read the complete text