Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models

Hoang-Chau Luong Lingwei Chen · 2026

Read Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models by Hoang-Chau Luong Lingwei Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06305.txt.

Description

Fine-tuning a poisoned model on clean data should remove its backdoor, but a popular method fails at that. Hoang-Chau Luong and 1 co-author explain why in a 2026 paper on Low-Rank Adaptation, a technique that updates only a small low-rank piece of a language model's weights.

The weakness, they show, is spectral rather than simply about rank. LoRA updates are far weaker in singular value terms than the pretrained weights, and they align poorly with clean-task directions while still overlapping trigger-sensitive ones. A scaling threshold exists beyond which suppression becomes possible, but standard LoRA rarely reaches it. Their Regularized LoRA strengthens the update and corrects alignment, cutting attack success across benchmarks while preserving clean accuracy.

Catalog record

CREATORHoang-Chau Luong Lingwei Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06305

Browse the full library · About the Tower of Babel