Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2
Pere Martra [email protected] · 2025
Read Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2 by Pere Martra [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22671.txt.
Description
Pere Martra reports a split nature inside pruned language models. Shrinking the expansion ratio of the gated MLP layers in Llama-3.2 models damaged exactly the expected abilities. The pruning was guided by a peak-to-peak size rule. Factual knowledge tests, grade-school math, and perplexity all degraded as expected. Instruction-following, however, moved the opposite way, gaining sharply at a 2.4-times balance ratio in both the one-billion and three-billion parameter models.
Seven expansion-ratio setups were benchmarked across knowledge, math, reading, instruction-following, and truthfulness suites. Multi-step reasoning stayed robust throughout. Martra argues the expansion ratio is not merely a compression dial. It is a design parameter that reshapes a model's skill profile in a lopsided way. That challenges the field's assumption that pruning degrades everything across the board.
Catalog record
| CREATOR | Pere Martra [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22671