DeltaLLM : Compress LLMs with Low-Rank Deltas between Shared Weights
Liana Mikaelyan, Ayyoob Imani, Mathew Salvaris, Parth Pathak, Mohsen Fayyaz · 2025
Read DeltaLLM : Compress LLMs with Low-Rank Deltas between Shared Weights by Liana Mikaelyan, Ayyoob Imani, Mathew Salvaris, Parth Pathak, Mohsen Fayyaz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18596.txt.
Description
Liana Mikaelyan and co-authors shrink large language models by making their layers share, in a 2025 paper. Their method, DeltaLLM, rebuilds a model so neighboring blocks use the same weights, with a small low-rank correction matrix carrying the difference between layers. Only those small corrections need training, and a few tens of millions of tokens suffice.
Applied to two well-known model families, the method removed about an eighth of the parameters while keeping nine tenths of the original skill on knowledge and reasoning tests. It beat rival pruning methods at the same size, and a leaner variant matched a bigger pruned model that had enjoyed extra recovery training. Memory is the binding constraint on running AI locally, so sharing weights with tiny deltas offers a cheap route to capable models on modest hardware.
Catalog record
| CREATOR | Liana Mikaelyan, Ayyoob Imani, Mathew Salvaris, Parth Pathak, Mohsen Fayyaz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-18596