Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

Pere Martra [email protected] · 2025

Read Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2 by Pere Martra [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22671.txt.

Description

Pere Martra reports a split nature inside pruned language models. Shrinking the expansion ratio of the gated MLP layers in Llama-3.2 models damaged exactly the expected abilities. The pruning was guided by a peak-to-peak size rule. Factual knowledge tests, grade-school math, and perplexity all degraded as expected. Instruction-following, however, moved the opposite way, gaining sharply at a 2.4-times balance ratio in both the one-billion and three-billion parameter models.

Seven expansion-ratio setups were benchmarked across knowledge, math, reading, instruction-following, and truthfulness suites. Multi-step reasoning stayed robust throughout. Martra argues the expansion ratio is not merely a compression dial. It is a design parameter that reshapes a model's skill profile in a lopsided way. That challenges the field's assumption that pruning degrades everything across the board.

Catalog record

CREATORPere Martra [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22671

Browse the full library · About the Tower of Babel