Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models
Hoang-Chau Luong Lingwei Chen · 2026
Read Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models by Hoang-Chau Luong Lingwei Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06305.txt.
Description
Fine-tuning a poisoned model on clean data should remove its backdoor, but a popular method fails at that. Hoang-Chau Luong and 1 co-author explain why in a 2026 paper on Low-Rank Adaptation, a technique that updates only a small low-rank piece of a language model's weights.
The weakness, they show, is spectral rather than simply about rank. LoRA updates are far weaker in singular value terms than the pretrained weights, and they align poorly with clean-task directions while still overlapping trigger-sensitive ones. A scaling threshold exists beyond which suppression becomes possible, but standard LoRA rarely reaches it. Their Regularized LoRA strengthens the update and corrects alignment, cutting attack success across benchmarks while preserving clean accuracy.
Catalog record
| CREATOR | Hoang-Chau Luong Lingwei Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06305