Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantization
Ziwei Zheng, Huizhi Liang, Vaclav Snasel, Vito Latora, Panos Pardalos, Giuseppe Nicosia, Varun Ojha · 2026
Read Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantization by Ziwei Zheng, Huizhi Liang, Vaclav Snasel, Vito Latora, Panos Pardalos, Giuseppe Nicosia, Varun Ojha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17987.txt.
Introduction
Ziwei Zheng and co-authors hunt for the smallest neural networks that still do their job well. Their 2026 preprint gives a unified study of minimal deep learning designs. Neural networks excel at classification, yet no one has a clear map of how small a network can be for a set task. The team builds a method to explore that space in a systematic way. They study how three forces interact: convergence, pruning, and quantization.
Pruning cuts parts of a network away. Quantization stores its numbers with less precision. Both shrink a model, but each can break its training. Knowing how the three forces trade off guides lean design. Small, sound models matter for phones, sensors, and low-cost devices.
Catalog record
| CREATOR | Ziwei Zheng, Huizhi Liang, Vaclav Snasel, Vito Latora, Panos Pardalos, Giuseppe Nicosia, Varun Ojha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17987
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantization Ziwei Zheng, Huizhi Liang, Vaclav Snasel, Vito Latora, Panos Pardalos, Giuseppe Nicosia, Varun Ojha Abstract Deep learning networks excel at classification, yet identifying minimal architectures that reliably solve a task remains challenging. We present a computational methodology for systematically exploring and analyzing the relationships among convergence, pruning, and quantization. The workflow first performs a structured design sweep across a large set of architectures, then evaluates convergence behavior, pruning sensitivity, and quantization robustness on representative models. Focusing on well-known image classification of increasing complexity, and across Deep Neural Networks, Convolutional Neural Networks, and Vision Transformers, our initial results show that, despite architectural diversity, performance is largely invariant and learning dynamics consistently exhibit three regimes: unstable, learning, and overfitting . We further characterize the minimal learnable parameters required for stable learning, uncover distinct convergence and pruning phases, and quantify the effect of reduced numeric precision on trainable parameters. Aligning with intuition, the results confirm that deeper architectures are more resilient to pruning than shallower ones, with parameter redundancy as high as 60 % 60\% , and quantization impacts models with fewer learnable parameters more severely and has a larger effect on harder image datasets. These findings provide actionable guidance for selecting compact, stable models under pruning and low-precision constraints in image classification. I Introduction Deep Neural Networks (DNNs) have achieved remarkable success in machine learning, particularly in image recognition [ 10 ] . Beyond standalone multi-layer perceptrons (MLPs), dense layers are central components in modern architectures such as Convolutional Neural Networks (CNNs) and Transformers/Vision Transformers (ViTs) [ 5 ] . Recent studies suggest that carefully designed MLPs can rival more complex models across a range of tasks [ 21 , 17 ] , underscoring the importance of understanding network compactness. Despite this progress, discovering minimal networks that solve a task efficiently remains challenging.Minimal networks are crucial for scalable AI systems as they expose the essential degrees of freedom required for learning. Pruning methods show that large models contain substantial redundancy: simplified subnetworks can retain high accuracy with far fewer parameters [ 1 ] . Related findings in Transformer-like architectures indicate that selectively removing components yields efficiency gains with minimal performance loss [ 5 ] . In parallel, quantization reduces computational and memory costs by lowering arithmetic precision [ 7 , 13 ] . Understanding the characteristics of minimal DNNs for stable learning calls for a systematic study linking architectural design to convergence dynamics, pruning behavior, and quantization robustness. In this paper, we conduct such a study focused exclusively on image datasets of increasing difficulties . Since our computational overhead is extremely expensive due to architectural design variations, we selected the most popular benchmarks. We first perform a structured sweep over architectural shapes —i.e., patterns of increasing/decreasing hidden-layer widths—then extend the analysis from MLP-based classifiers to CNNs and ViTs. For representative models, we examine (i) convergence profiles and their phases, (ii) sensitivity to pruning at different depths and parameter budgets, and (iii) the effects of reduced numeric precision on trainable parameters. Our main contributions are: • A unified methodology for analyzing stable learning in minimal neural networks via joint examination of convergence, pruning, and quantization, applied across MLPs/DNNs, CNNs, and ViTs on image datasets. • A systematic investigation of how architectural depth and hidden-layer shapes influence regime-wise learning dynamics, pruning tolerance, and quantization robustness under varying problem complexities. • Empirical characterization of the relationships among unstable/stable learning, learnable parameter counts, pruning rates, and precision reduction, offering guidance for selecting compact, resource-efficient models. The remainder of the paper is organized as follows. Sec. II reviews related work. Sec. III presents methodology. Sec. IV reports and analyzes experimental results for DNNs, CNNs, and ViTs. Sec. V concludes findings. II Related Work Recent research underscores the rationale for seeking minimal networks by demonstrating that high performance can be achieved with fewer parameters. The lottery ticket hypothesis by Frankle and Carbin [ 3 ] shows that sparse sub-networks, if properly initialized, can perform on par with their larger counterparts. Similarly, Neyshabur et al.’s work on over-parameterization and generalization [ 16 ] shows that excessive parameter redundancy does not necessarily translate into improved network stability or efficiency. These findings align with our pursuit of profiling minimal networks and point to gaps in the research on systematically identifying and characterizing such architectures. A key factor in this exploration is the configuration of a network’s hidden layers, which plays a significant role in learning and overall performance. Neural architecture search (NAS) techniques [ 23 , 12 ] reveal that variations in layer configurations can fundamentally influence the network’s efficiency and generalization capability. Complementary insights from Naitzat et al. [ 15 ] emphasize that the structural design of hidden layers affects how data representations are formed. Thus, investigating hidden layer combinations is crucial in understanding the intrinsic factors that can lead to minimal yet effective network architectures. In studies of rational model minimization, pruning and quantization have emerged as key techniques for reducing model complexity. Molchanov et al. [ 14 ] demonstrated that pruning could eliminate redundant parameters while maintaining performance, and practical low-precision methods show that constraining numerical precision to 8-bit can preserve accuracy while reducing cost [ 8 , 18 ] . These studies give us insights into how to obtain the minimal network structure to explore the stable regimes of learning related to pruning and the impact of quantization on networks with different parameter orders. Kingma and Ba [ 9 ] demonstrated that Adam can substantially reduce the loss within a relatively small number of epochs, indicating that even a short training period (on the order of 20 epochs) may reveal significant convergence trends. In a related study, Smith [ 20 ] introduced cyclical learning rates, showing that performing a learning rate range test over approximately 10--20 epochs is sufficient to capture the overall loss behavior and to identify an optimal learning rate range. Building on these foundations, our work integrates convergence-phase analysis, pruning phases, and quantization into basic networks. We conduct extensive experiments across multiple image datasets under a unified experimental configuration and extend them to MLPs/DNNs, CNNs, and ViTs to explore how architectural decisions and parameter-level configurations affect network convergence and performance, thereby uncovering deeper insights into the design of minimal yet robust models. III Methodology III-A Systematic Characterization We employ a systematic architecture design to analyze three key phases–learning, pruning, and quantization–to yield a minimal neural network architecture (see an example 1-(hidden)layer network in Fig. 1 ). Our approaches enable a systematic evaluation of the topological (network configurations) and parametric factors that influence learning dynamics and network performance. Fig. 1: 1-layer whole network, pruned network, and quantized network (see Sec. III for details). Convergence phases for stable learning Neural network convergence has been a long-standing topic, with theoretical results demonstrating the approximation capabilities of MLPs/DNNs [ 6 ] . We therefore analyze the evolution of network empirical performance in relation to changes in network parameters and topologies. By monitoring the model’s test accuracy across different parameter phases, we identify critical periods that significantly contribute to convergence and generalization. Our analysis of convergence profiles is computationally intensive. Thus, we divided it