Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms

Omatharv Vaidya, Connor T. Jerzak, Nhat Ho, Chandrajit Bajaj · 2026

Read Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms by Omatharv Vaidya, Connor T. Jerzak, Nhat Ho, Chandrajit Bajaj (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08423.txt.

Description

Omatharv Bharat Vaidya, Connor Jerzak, Nhat Ho, and Chandrajit Bajaj make low-rank fine-tuning adaptive. A standard LoRA adapter freezes one small update shape per layer. That is efficient but rigid. The right correction can depend on the input. It can also depend on what earlier layers have already computed.

Their queryable design keeps a shared memory of low-rank update atoms. Each block of layers builds a query from its current low-rank state. It adds a running summary of previous blocks. It then retrieves a content-dependent mix of atoms by attention. The mix is applied inside the bottleneck. The model keeps LoRA's efficiency while the update changes with context and depth.

Catalog record

CREATOROmatharv Vaidya, Connor T. Jerzak, Nhat Ho, Chandrajit Bajaj
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE133 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08423

Browse the full library · About the Tower of Babel