Q3R : Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training

Ipsita Ghosh, Ethan Nguyen, Christian Kümmerle · 2025

Read Q3R : Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training by Ipsita Ghosh, Ethan Nguyen, Christian Kümmerle (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04485.txt.

Description

Ipsita Ghosh and two colleagues present Q3R, a regularizer for training networks at low rank. Low-rank training cuts the cost of fine-tuning large models well. Pre-training at low rank is harder, since weights must stay lean while still learning the task.

Q3R adds a quadratic penalty built to stand in for a smoothed rank measure. The idea grows out of iteratively reweighted least squares, a classic fitting scheme. Unlike rival methods, it can drive weights to a chosen target rank directly. Predictive quality stays competitive, which could lower the cost of building large models from scratch.

Catalog record

CREATORIpsita Ghosh, Ethan Nguyen, Christian Kümmerle
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE117 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04485

Browse the full library · About the Tower of Babel