DeltaLLM : Compress LLMs with Low-Rank Deltas between Shared Weights

Liana Mikaelyan, Ayyoob Imani, Mathew Salvaris, Parth Pathak, Mohsen Fayyaz · 2025

Read DeltaLLM : Compress LLMs with Low-Rank Deltas between Shared Weights by Liana Mikaelyan, Ayyoob Imani, Mathew Salvaris, Parth Pathak, Mohsen Fayyaz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18596.txt.

Description

Liana Mikaelyan and co-authors shrink large language models by making their layers share, in a 2025 paper. Their method, DeltaLLM, rebuilds a model so neighboring blocks use the same weights, with a small low-rank correction matrix carrying the difference between layers. Only those small corrections need training, and a few tens of millions of tokens suffice.

Applied to two well-known model families, the method removed about an eighth of the parameters while keeping nine tenths of the original skill on knowledge and reasoning tests. It beat rival pruning methods at the same size, and a leaner variant matched a bigger pruned model that had enjoyed extra recovery training. Memory is the binding constraint on running AI locally, so sharing weights with tiny deltas offers a cheap route to capable models on modest hardware.

Catalog record

CREATORLiana Mikaelyan, Ayyoob Imani, Mathew Salvaris, Parth Pathak, Mohsen Fayyaz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18596

Browse the full library · About the Tower of Babel