Scaling Laws in the Tiny Regime: How Small Models Change Their Mistakes

Mohammed Alnemari, Rizwan Qureshi, Nader Begrazadah · 2026

Read Scaling Laws in the Tiny Regime: How Small Models Change Their Mistakes by Mohammed Alnemari, Rizwan Qureshi, Nader Begrazadah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07365.txt.

Introduction

Alnemari, Qureshi and Begrazadah map scaling laws for tiny vision models, a regime below the sizes prior work covers. Training dozens of small networks at varied widths, they find error still follows a power law, with steeper slopes than those reported for large language models. The law is uneven: local slopes fade with scale, and one architecture saturates near the top of the range. Small models also shift which inputs they miss and concentrate on easy classes, while the tiniest stay best calibrated.

Catalog record

CREATORMohammed Alnemari, Rizwan Qureshi, Nader Begrazadah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45345
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.07365

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Laws in the Tiny Regime: How Small Models Change Their Mistakes Mohammed Alnemari, Rizwan Qureshi, Nader Begrazadah Abstract Neural scaling laws describe how model performance improves as a power law with size, but existing work has focused almost entirely on models above 100M parameters. The regime below 20 million parameters—where TinyML and edge AI systems operate—remains largely unexamined. We train 90 models spanning 22K to 19.8M parameters across two architecture families (a plain ConvNet and MobileNetV2) on CIFAR-100, varying width while holding depth and training protocol fixed. Both architectures follow approximate power laws in error rate, with exponents of α = 0.156 ± 0.002 \alpha=0.156\pm 0.002 (ScaleCNN) and α = 0.106 ± 0.001 \alpha=0.106\pm 0.001 (MobileNetV2), computed across five independent seeds. Because prior scaling laws were fit to cross-entropy loss rather than error rate, and these metrics are nonlinearly related, direct numerical comparison of exponents across regimes is approximate; with that caveat, the small-regime exponents are 1.4–2 × \times steeper than the α ≈ 0.076 \alpha\approx 0.076 reported for large language models. However, the power law does not hold uniformly: local exponents decay with scale, and MobileNetV2 saturates at 19.8M parameters ( α local = 0.006 \alpha_{\mathrm{local}}=0.006 ), reaching a capacity ceiling on this dataset. The structure of errors also changes with scale. The Jaccard overlap between error sets of the smallest and largest ScaleCNN models is only 0.35 (mean across 25 seed pairs, ± 0.004 \pm 0.004 )—compression changes which inputs are misclassified, not merely how many. Small models develop a triage strategy, concentrating capacity on easy classes (Gini of per-class accuracy: 0.26 at 22K params vs. 0.09 at 4.7M) while effectively abandoning the hardest ones (bottom-5 class accuracy: 10% vs. 53%). Counter to the usual trend of overconfidence scaling with capacity, the smallest models are the best calibrated (ECE = 0.013 vs. peak 0.110 at mid-size). Aggregate accuracy alone is therefore a misleading basis for edge deployment decisions; validation must happen at the target model size. Keywords: scaling laws; TinyML; error redistribution; model compression; neural network calibration; edge deployment 1 Introduction Neural scaling laws ( 15 ; 11 ) have become one of the more reliable empirical regularities in deep learning. Loss decreases as a power law with model size N , dataset size D , and compute budget C : L(N)\sim N^{-\alpha} (1) where \alpha\approx 0.076 for language models (measured on cross-entropy loss), with similar values reported for vision transformers ( 26 ) . These laws, spanning seven or more orders of magnitude, enabled the strategic scaling decisions behind GPT-3, PaLM, and subsequent frontier models. Much less is known about the left side of the scaling curve. Below 20 million parameters—the operating point of TinyML systems deployed on microcontrollers with \leq 256 KB RAM and \leq 1 mW power budgets—we lack answers to basic questions: • Does the same power law extend to the small-model regime, or does a different scaling relationship govern? • Does compression merely increase the error rate, or does it fundamentally reshape which inputs the model fails on? • How do calibration and per-class fairness vary with scale? These questions matter for deployment. Edge AI systems run in safety-critical contexts—autonomous vehicles, medical devices, industrial monitoring—where the failure distribution matters as much as the failure rate . A model that maintains 88% accuracy after compression but silently shifts all its errors to a specific subpopulation may be more dangerous than one with 85% accuracy and uniformly distributed errors. The TinyML community typically reports model accuracy at individual operating points. To our knowledge, no prior work has systematically varied model size below 1M parameters across orders of magnitude on a fixed task to characterize both the functional form and the qualitative changes in error behavior. We set out to do exactly that. Contributions. This paper makes three contributions: 1. Systematic scaling law characterization in the sub-20M regime. We measure the accuracy–model-size relationship across nearly three orders of magnitude (22K–19.8M parameters) with two architecture families on CIFAR-100. We report architecture-dependent exponents that are 1.4–2 \times steeper than the large-model regime, though this comparison is approximate because prior work fit to cross-entropy loss while we fit to error rate (Section 5.1 ). We also document a broken power law and architecture-specific saturation. 2. Error redistribution under width scaling. The Jaccard overlap between error sets of the smallest and largest models is only 0.35 ( \pm 0.004 across 25 seed pairs), below the 0.42 expected from pure subset containment and well above the 0.21 expected from independent errors. Compression changes which inputs are misclassified, not merely how many. 3. Class triage and calibration inversion. Small models develop extreme triage strategies (Gini coefficient of per-class accuracy: 0.26 at 22K parameters vs. 0.09 at 4.7M), and the smallest models are the best calibrated, with ECE following an inverted-U pattern in ScaleCNN—contrary to the usual assumption that overconfidence grows monotonically with capacity. 2 Related Work Neural scaling laws. Kaplan et al. ( 15 ) established that language model loss follows power laws in model size, dataset size, and compute, with smooth trends spanning seven orders of magnitude. Hoffmann et al. ( 11 ) refined these findings by characterizing compute-optimal training (Chinchilla scaling). Zhai et al. ( 26 ) extended scaling analysis to vision transformers. Clark et al. ( 5 ) proposed unified scaling laws across modalities. Hestness et al. ( 9 ) demonstrated that deep learning scaling is predictable across domains, reporting exponents in the range \alpha\approx 0.07 – 0.12 for vision tasks above 1M parameters. Rosenfeld et al. ( 20 ) proposed constructive methods for predicting generalization error across scales, including relatively small models. However, all of these works focus primarily on the large-model regime ( > 10M parameters) or treat smaller models as interpolation points rather than as a regime of independent interest. The sub-1M regime relevant to edge deployment has received little attention. Sharma and Kaplan ( 24 ) provided a theoretical grounding for scaling laws by relating the exponent to the intrinsic dimension of the data manifold, predicting \alpha\approx 4/d where d is the manifold dimension. Bahri et al. ( 2 ) offered complementary explanations rooted in statistical mechanics. Caballero et al. ( 4 ) introduced broken neural scaling laws (BNSL), showing that scaling behavior across wide parameter ranges is better described by smoothly bending power laws than by a single exponent. Our empirical observation of decaying local exponents is consistent with the BNSL framework. TinyML and efficient inference. The TinyML community has developed a rich toolkit for deploying neural networks on microcontrollers: knowledge distillation ( 10 ) , structured pruning ( 16 ) , quantization-aware training ( 14 ) , and neural architecture search for constrained devices ( 18 ; 3 ) . These works optimize for accuracy at a specific target size but do not characterize the functional relationship between accuracy and size across scales. We aim to fill that gap by measuring how accuracy, error distribution, and calibration evolve as model size varies continuously. Emergent capabilities and phase transitions. Wei et al. ( 25 ) documented emergent capabilities in large language models that appear discontinuously at critical scales. Schaeffer et al. ( 23 ) argued that some apparent emergence is an artifact of

Read the complete text