RanDeS: Randomized Delta Superposition for Multi-Model Compression

Hangyu Zhou, Aaron Gokaslan, Volodymyr Kuleshov, Bharath Hariharan · 2025

Read RanDeS: Randomized Delta Superposition for Multi-Model Compression by Hangyu Zhou, Aaron Gokaslan, Volodymyr Kuleshov, Bharath Hariharan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11204.txt.

Description

Hangyu Zhou and co-authors shrink the cost of serving many tuned models in a 2025 paper. Labs often fine tune one base model many ways, and storing each copy eats memory. Merging the copies helps, but the task tweaks clash when summed back at use time.

Their method, RanDeS, recasts merging as squeeze and fetch. Random orthogonal twists scramble the tweaks so stray parts cancel each other out, which cuts the clash a lot. Each twist is set by a seed alone, so new models join with no extra store cost. Gains held across vision and language tasks, and models can join or leave with little compute spent.

Catalog record

CREATORHangyu Zhou, Aaron Gokaslan, Volodymyr Kuleshov, Bharath Hariharan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11204

Browse the full library · About the Tower of Babel