A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions

Tiexin Ding · 2026

Read A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions by Tiexin Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18898.txt.

Introduction

Tiexin Ding uses the two-parameter Weibull distribution as a measuring stick for transformer weights. At initialization, Gaussian weights give a shape value near 1.20, an anchor for reading later shifts. Across twelve models from seven families, feed-forward modules and attention output projections stay near that anchor. Attention input projections drift away instead, with drift shaped by how query and key weights are stored. The scale parameter grows through training with the learning-rate schedule, so shape marks class and scale marks progress.

Catalog record

CREATORTiexin Ding
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66901
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions Tiexin Ding Abstract We apply the Weibull distribution — a two-parameter family from extreme-value theory — as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give | w | ∼ HalfNormal |w|\sim\text{HalfNormal} , which gives k ≈ 1.20 k\approx 1.20 (via middle-80% probability-plot fit, the protocol used throughout this work). This anchor makes k k a principled, architecture-independent measuring stick for training dynamics. Fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that are invisible to aggregate statistics. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W o W_{o} — the Transmission Class — fall in a narrow k k band across architectures: the median terminal k k across our 12 entries falls in [ 1.186 , 1.204 ] [1.186,1.204] with cross-family CV = 0.51 % =0.51\% , shared across SwiGLU and GeLU activations, Pre-LN and QK-Norm placements, and model sizes from 70M to 14B. Second, the attention input projections W q W_{q} and W k W_{k} — the Selection Class — depart from the Weibull family. The severity of this departure is architecture-dependent: models with separately-stored Q/K (OLMo-1, OLMo-2) yield k ∈ [ 0.76 , 0.99 ] k\in[0.76,0.99] (deep Selection), grouped-query attention models (LLaMA-3, Mistral, Qwen2.5/3) yield k ∈ [ 1.10 , 1.16 ] k\in[1.10,1.16] (mild Selection), and Pythia’s merged W qkv W_{\text{qkv}} occupies a transitional zone k ∈ [ 1.05 , 1.18 ] k\in[1.05,1.18] tracking training budget T / τ T/\tau monotonically. Third, the scale parameter λ \lambda grows substantially during training and scales with η / λ wd \sqrt{\eta/\lambda_{\text{wd}}} within the Pythia family (Pearson r = 0.94 r=0.94 across the three Transmission Class kinds), directionally consistent with the AdamW steady-state η / λ wd \sqrt{\eta/\lambda_{\text{wd}}} scaling analysis of Fan et al. 2025 within their validated regime. The two Weibull parameters carry independent information: k k labels the functional class (Transmission vs. Selection), λ \lambda labels training progress. We release npm-weibull-py v0.4 , a pip-installable Python library, together with the benchmark database DATABASE_v9_1 . Both are available at https://github.com/tiexinding/NPM-Weibull-public . 1 Introduction 1.1 Problem and Motivation Understanding what happens during transformer training requires quantitative tools for interrogating learned weight distributions. Existing approaches operate in orthogonal spaces: WeightWatcher ( Martin and Mahoney 2019 ) and HT-SR ( Martin and Mahoney 2020 ) analyze eigenvalue spectra; AlphaDecay ( He et al. 2025 ) tracks eigenvalue drift; massive activation analysis ( Sun et al. 2024 ) measures activation magnitudes. None of these methods directly characterizes the element-wise distribution of weight magnitudes |W_{ij}| — the most granular representation of what a model has learned. This gap matters because element-wise statistics reveal structure that spectral methods cannot. Eigenvalue spectra compress all matrix elements into a single distribution, averaging across component types. If different functional components within the same model — say, FFN layers versus attention projections — follow systematically different distributions, the aggregate spectral statistics obscure rather than reveal this distinction. We apply the Weibull distribution — a two-parameter family from extreme-value theory ( Weibull 1951 ) — as a diagnostic lens on element-wise |W| distributions. The shape parameter k quantifies distributional tailedness; the scale \lambda quantifies magnitude. Critically, the shape parameter has an anchor at initialization: i.i.d. Gaussian weights give |w|\sim\text{HalfNormal} , which gives k\approx 1.20 (via middle-80% probability-plot fit, the protocol used throughout this work). This makes k a precise, dimensionless measuring stick for training dynamics — any departure from k_{0}\approx 1.20 is attributable entirely to training. With this measuring stick, element-wise and per-component, the weight distributions of a transformer model present a picture of heterogeneous, depth-dependent evolution. Different layers of the same model specialize at different rates; within each layer, different projection matrices — W_{q} , W_{k} , W_{o} , the FFN gates — evolve along different trajectories. This depth-dependent structure is invisible to aggregate statistics, but becomes measurable when each projection matrix is examined independently at every layer at every checkpoint. Figure 1 summarizes the resulting diagnostic framework; Section 4 documents the per-layer trajectories for the Selection Class. Figure 1: Diagnostic framework architecture. The framework comprises four interconnected components organized as a layered architecture. Theory (top-left): the Weibull two-parameter primitive f(x;k,\lambda)=(k/\lambda)(x/\lambda)^{k-1}\exp(-(x/\lambda)^{k}) with its \sigma -inverse form \lambda=\sigma/\sqrt{\Gamma(1+2/k)-\Gamma^{2}(1+1/k)} , anchored by two reference scales: the k anchor (Half-Normal initialization, k_{0}\approx 1.20 via middle-80% probability-plot fit — numerically consistent across vendors and \sigma_{\text{init}} scales) and the \lambda reference (steady-state scaling \lambda_{\text{steady}}\propto\sqrt{\eta/\lambda_{\text{wd}}} , \tau_{\text{iter}}=1/(\eta\lambda_{\text{wd}}) , T/\tau as the dimensionless training progress). Two functional classes are defined: Transmission Class ( W_{\text{gate}},W_{\text{up}},W_{\text{down}},W_{o} ) and Selection Class ( W_{q},W_{k} ). Framework (top-right): eight diagnostic functions F1–F8 project the framework onto two classification axes ( k -axis: 3 classes — Transmission, Selection-mild, Selection-deep; \lambda -axis: 4 states — Saturated, Near-saturated, Approaching, Transition), together with the composite diagnose_model() wrapper (universal, with compare_to_benchmark() as a separate optional call). Code : npm-weibull-py v0.4 , implementing the eight functions as an open-source library. Database : DATABASE_v9_1 , 12 entries across 7 architectural families. The framework’s power lies in identifying departures from Weibull: Selection Class departures of W_{q},W_{k} are the primary diagnostic signal, not failure modes. 1.2 Core Findings Applying this measuring stick to 12 model entries spanning 7 architectural families (Pythia 70M–6.9B, OLMo-1, OLMo-2, LLaMA-3, Mistral, Qwen2.5, Qwen3) reveals three robust findings. First, FFN modules and attention output projections fall in a narrow k band across architectures. The median terminal k for these Transmission Class components falls in the interval [1.186,1.204] (CV =0.51\% , n=12 entries; or CV =0.57\% on the depth- \geq 12 subset), shared across SwiGLU and GeLU activations, across Pre-LN and QK-Norm norm placements, and across model sizes from 70M to 14B. Second, the attention input projections W_{q} and W_{k} systematically depart from the initialization anchor, with the magnitude of departure shaped by storage architecture: separately-stored MHA shows the largest drift (median k\in[0.76,0.99] ), GQA and merged W_{\text{qkv}} show smaller drift toward the Transmission band. We document this process in full detail and analyze its evolution mechanism across attention architectures. Third, the scale parameter \lambda grows substantially from initialization to the terminal checkpoint. Within the Pythia family, terminal \lambda across the three Transmission Class kinds ( W_{o} , W_{\text{FFN\_in}} , W_{\text{FFN\_out}} ; W_{\text{qkv}} excluded as Selection) scales with \sqrt{\eta/\lambda_{\text{wd}}} (Pearson r=0.94 , n=5 ), showing directional consistency with the AdamW steady-state \sqrt{\eta/\lambda_{\text{wd}}} scaling analysis of Fan et al. 2025 within their validated regime ( d\leq 2048 ); per-size deviations of 7–36% indicate directional rather than quantitative match. Critically, \lambda grows uniformly across all component types (Pearson r=0.9967 between \lambda_{O} and \lambda_{\text{FFN\_out}} ), while k remains nearly constant within the Transmission Class. The two Weibull parameters carry independent information: k labels the functional class, \lambda labels training progress. 1.3 Contributions This paper makes three contributions. A diagnostic framework. We formalize the framework — the initialization anchor, the noise-optimal middle 80% trim fitting protocol, and the anti-interference property these choices confer. The framework identifies that the attention input projections W_{q} and W_{k} systematically depart from the initialization anchor during training, with the departure magnitude shaped by

Read the complete text