Latent Space Factorization in LoRA
Shashi Kumar, Yacouba Kaloga, John Mitros, Petr Motlicek, Ina Kodrasi · 2025
Read Latent Space Factorization in LoRA by Shashi Kumar, Yacouba Kaloga, John Mitros, Petr Motlicek, Ina Kodrasi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19640.txt.
Description
Shashi Kumar and four co-authors improve on LoRA, the popular low-rank method for fine-tuning large models cheaply. Standard LoRA learns a small update subspace but never sorts out which parts of it carry task-relevant information. Their variant, FVAE-LoRA, adds a variational autoencoder that learns two separate latent spaces.
A new training objective pushes one latent space to hold features that matter for the task and the other to absorb everything else. Across text, audio, and image tasks the factorized version consistently beat ordinary LoRA. Tests with spurious correlations confirmed it isolates the real signal better, making the fine-tuned models more robust when the data distribution shifts.
Catalog record
| CREATOR | Shashi Kumar, Yacouba Kaloga, John Mitros, Petr Motlicek, Ina Kodrasi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19640