From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression

Elia Cunegatti, Marcus Vukojevic, Erik Nielsen, Giovanni Iacca · 2026

Read From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression by Elia Cunegatti, Marcus Vukojevic, Erik Nielsen, Giovanni Iacca (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02559.txt.

Description

Cunegatti and co-authors compress large language models at a finer grain than past work. Replacement methods swap whole components for fitted modules. Two design limits are standard. Selection works at full-layer granularity. Chosen parts must also sit in one contiguous block. Redundancy in real transformers breaks both assumptions. It scatters across depth and splits unevenly between Attention and FeedForward outputs.

Their method, SubFit, selects submodules instead of layers. Attention and FeedForward units are picked non-contiguously. Each gets its own lightweight fitted residual bypass. The method runs post-training and needs only calibration data. Tests cover ten models, five base and five instruction-tuned. Five sparsity levels run from 12.5 to 37.5 percent. Four replacement baselines compete. SubFit gives the best aggregate perplexity-accuracy trade-off. At 25 percent sparsity, it keeps 84.6 percent of dense downstream accuracy. Inference speed and KV-cache savings also improve.

Catalog record

CREATORElia Cunegatti, Marcus Vukojevic, Erik Nielsen, Giovanni Iacca
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02559

Browse the full library · About the Tower of Babel