SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models

Yuxuan Zhang · 2025

Read SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models by Yuxuan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18168.txt.

Description

Fine-tuning a large language model on new tasks can erase what it already knew, a failure called catastrophic forgetting. Yuxuan Zhang proposes SECURA, a variant of the popular low-rank adaptation method. It pairs a matrix decomposition that keeps important columns intact with a sigmoid-based scaling that protects large, meaningful weights.

Across math problems, question answering, translation, and logic tasks, SECURA improved fine-tuning scores over a strong rival method on five open models. In continual learning tests it kept more than seventy percent of the models' basic knowledge. That retention beat replay and regularization baselines. The method aims to let small teams adapt big models without retraining them or losing their skills.

Catalog record

CREATORYuxuan Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18168

Browse the full library · About the Tower of Babel