Pre-Warm: Input-Conditioned Weight Initialization for Convolutional Neural Networks
Rowan Martinsn · 2026
Read Pre-Warm: Input-Conditioned Weight Initialization for Convolutional Neural Networks by Rowan Martinsn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25256.txt.
Introduction
How a neural network's weights are set before training shapes everything that follows, yet standard starting schemes ignore the data entirely. Rowan Martinsn proposes Pre-Warm in 2026, a way to seed the first layer of a convolutional network with patterns drawn from real images. Small patches from one training batch are clustered, and the cluster centres become half of the first layer's filters. The rest keep the usual random start, and simple formulas set nearly every knob of the method.
Across five image benchmarks, from handwritten digits to colour photos, Pre-Warm gave statistically significant accuracy gains over the standard Kaiming start in paired tests over eight runs. The strongest result came on street view digits, where it won all eight runs. Setup takes under a tenth of a second and needs no change to the network itself.
Catalog record
| CREATOR | Rowan Martinsn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25256
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pre-Warm: Input-Conditioned Weight Initialization for Convolutional Neural Networks Rowan Martinsn Abstract We introduce Pre-Warm, a simple yet effective zero-training-cost method for data-conditioned initialization of the first convolutional layer. Before the first forward pass, Pre-Warm extracts mean-centered local patches from a single training batch, clusters them with MiniBatchKMeans, applies inverse Manhattan spatial weighting, and uses the resulting centroids to initialize half of the first-layer filters (the remainder retain Kaiming initialization). We derive closed-form rules for all hyperparameters except a single insensitive scale parameter, though we derive a Kaiming parity bound on scale from patch dimensionality. For grayscale datasets we use Otsu’s foreground density; for natural color images we use the mean L2 norm of mean-centered patches. Both rules accurately predict the optimal patch count observed in grid search. Across five standard benchmarks - MNIST, Fashion-MNIST, CIFAR-10, SVHN, and CIFAR-100 - and 8-seed paired experiments, Pre-Warm yields statistically significant accuracy improvements over standard Kaiming initialization (p ¡ 0.05 on all datasets, p = 0.0007 on SVHN with 8/8 wins, p = 0.0033 on CIFAR-100 with 7/8 wins). The method adds negligible overhead, requires no architectural changes, and integrates into existing training pipelines with only a few lines of code. Pre-Warm demonstrates that even a lightweight, input-dependent signal can meaningfully improve optimization trajectories in modern convolutional networks. 1 Introduction The choice of initial weights determines the starting geometry of the loss landscape. Kaiming initialization ( 4 ) and Glorot initialization ( 2 ) are statistically principled - they preserve activation variance across layers - but make no reference to the actual data the network will process. Every dataset, regardless of its local structure, receives an identical initialization drawn from a scaled random distribution. We propose that even a lightweight, zero-cost signal from the data can meaningfully inform the starting state of the first layer. This is not a claim about replacing the optimization process, but about giving it a better starting point in a principled and automatic way. Pre-Warm targets only the first convolutional layer, adds no trainable parameters, requires no gradient steps, and completes in under 0.1 seconds on a T4 GPU. The key insight is that k -means on mean-centered local patches naturally discovers the dominant edge and texture patterns in the image distribution - precisely what the first convolutional layer is known to learn ( 3 ) . Seeding those filters with data-derived centroids reduces the number of gradient steps required to encode this structure. 2 Related Work Data-dependent initialization 9 initialize convolutional weights using statistics from mini-batch forward passes. LSUV ( 13 ) normalizes each layer’s output variance iteratively using real data. Both methods operate at the dataset level and require forward passes through the network. Pre-Warm is per-batch, requires no forward pass, and operates entirely in input space at negligible cost. Unsupervised pre-training Greedy layer-wise pretraining ( 6 ) uses RBMs or autoencoders to provide informed initializations before supervised fine-tuning. These methods are computationally expensive and have been largely supplanted by modern normalization and residual architectures ( 5 ) . Pre-Warm recovers the core intuition - use unsupervised structure to inform initialization at essentially zero cost. Dictionary learning and sparse coding 15 showed that sparse coding on natural image patches produces Gabor-like filters matching V1 receptive fields. K -means on local patches is a tractable approximation of this process. Pre-Warm applies this observation directly to filter initialization, with a principled derivation of how many patches to sample. Meta-learning MAML ( 1 ) learns an initialization that adapts quickly to new tasks through second-order optimization. Pre-Warm can be interpreted as a static, zero-cost approximation of this principle: conditioning initial parameters on input statistics rather than learning the conditioning mechanism. It recovers a fraction of the benefit at orders-of-magnitude lower cost. 3 Experimental Setup We evaluate Pre-Warm against standard Kaiming initialization ( 4 ) across five image-classification benchmarks, using a fully paired design in which the only difference between the two conditions is the initialization of the first convolutional layer. All experiments share a single architecture, optimizer, and training budget; per-dataset quantities (input channels, normalization statistics, learning rate, and the derived Pre-Warm patch count) are summarized in Table 1 . 3.1 Architecture All experiments use a single compact convolutional network, which we refer to as SimpleCNN . It consists of two convolutional blocks followed by a two-layer classifier: \begin{array}[]{c}\text{Conv}(C_{\text{in}}\!\to\!32,\;3\!\times\!3,\;\text{pad }1)\\[2.0pt] \downarrow\\[2.0pt] \text{BatchNorm}\to\text{ReLU}\to\text{MaxPool}(2)\\[2.0pt] \downarrow\\[2.0pt] \text{Conv}(32\!\to\!64,\;3\!\times\!3,\;\text{pad }1)\\[2.0pt] \downarrow\\[2.0pt] \text{BatchNorm}\to\text{ReLU}\to\text{MaxPool}(2)\\[2.0pt] \downarrow\\[2.0pt] \text{Flatten}\\[2.0pt] \downarrow\\[2.0pt] \text{FC}(\!\to\!256)\to\text{ReLU}\\[2.0pt] \downarrow\\[2.0pt] \text{FC}(\!\to\!10)\end{array} Both convolutions use 3\times 3 kernels with padding 1 , so each 2\times 2 max-pool halves the spatial resolution. For the 28\times 28 grayscale datasets this yields a 64\times 7\times 7=3136 -dimensional flattened representation entering the first fully-connected layer; for the 32\times 32 color datasets it yields 64\times 8\times 8=4096 . Batch normalization ( 7 ) is applied after each convolution, including the first - so any initialization signal injected into conv1 must survive normalization to affect training. The first convolutional layer has F=32 output channels and kernel size K=3 throughout; these two quantities determine every Pre-Warm hyperparameter except the scale. 3.2 Datasets and preprocessing We evaluate on MNIST ( 11 ) , Fashion-MNIST ( 17 ) , CIFAR-10 ( 10 ) , SVHN (cropped digits) ( 14 ) , and CIFAR-100 ( 10 ) . CIFAR-100 shares the same 32\times 32 three-channel format as CIFAR-10 but has 100 fine-grained classes, making it a substantially harder classification task at the same input resolution. The two grayscale datasets are single-channel 28\times 28 ; the two natural-image datasets are three-channel 32\times 32 . Pixels are scaled to [0,1] and standardized with per-channel mean and standard deviation. The raw [0,1] images are retained separately from the standardized images: the standardized tensors are used for training, while the raw images are used only to estimate foreground density for the Pre-Warm patch-count rule (Section 3.4 ). All reported metrics use the standard test split of each dataset. 3.3 Training protocol Each run trains for N=5000 optimization steps with Adam ( 8 ) , batch size 256 , and cross-entropy loss. The learning rate follows a cosine annealing schedule ( 12 ) from \eta_{\text{start}} to \eta_{\text{end}} over the full 5000 steps. The grayscale datasets and CIFAR-10 use \eta_{\text{start}}=3\times 10^{-3} , \eta_{\text{end}}=3\times 10^{-5} ; SVHN uses a lower \eta_{\text{start}}=5\times 10^{-4} , \eta_{\text{end}}=3\times 10^{-6} , which we found necessary for stable convergence on that dataset. Every configuration is run over 8 random seeds, \{42,7,13,99,2025,1,8,21\} , with the seed controlling weight initialization, data shuffling, and the k-means initialization so that the Pre-Warm and Kaiming runs at a given seed are exactly paired. All experiments run on a single NVIDIA T4 GPU. 3.4 Pre-Warm configuration The conditioned runs apply Pre-Warm to conv1 before the first forward pass using the design rules derived in Section 4 . Three parameters follow directly from architecture dimensions: n_{\text{clusters}}=F/2=16 (half the filter bank is conditioned, half retains Kaiming initialization), \text{patch\_size}=K=3 (patches must match the filter receptive field), and scale \sigma , which controls the amplitude of the conditioned filters. The fourth parameter, n_{\text{patches}} , is derived from dataset statistics and differs by image type. Grayscale datasets For MNIST and Fashion-MNIST, pixel histograms are strongly bimodal - dark background versus bright foreground strokes - making Otsu’s method ( 16 ) a reliable foreground