Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models

Yonggan Fu*, Xin Dong*, Shizhe Diao, Matthijs Van keirsbilck, Hanrong Ye, Wonmin Byeon, Yashaswi Karnati, Lucas Liebenwein, Hannah Zhang, Nikolaus Binder, Maksim Khadkevich, Alexander Keller, Jan Kautz, Yingyan (Celine) Lin † , Pavlo Molchanov · 2025

Read Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models by Yonggan Fu*, Xin Dong*, Shizhe Diao, Matthijs Van keirsbilck, Hanrong Ye, Wonmin Byeon, Yashaswi Karnati, Lucas Liebenwein, Hannah Zhang, Nikolaus Binder, Maksim Khadkevich, Alexander Keller, Jan Kautz, Yingyan (Celine) Lin † , Pavlo Molchanov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18890.txt.

Introduction

Yonggan Fu and 14 co-authors design Nemotron-Flash, a small language model tuned for low delay on real devices, not just few parameters. Small models serve apps with strict lag limits. Past design chased parameter counts, yet a lean parameter bill does not mean a fast reply on hardware.

Two design facts rule real lag: the depth to width ratio, key at small batch sizes, and operator choice, which shapes both lag and large batch throughput. The team map lag best ratios and build a hybrid model on those rules. Training follows the same lag first lens. The result targets quick replies where users wait. Principles, not one model, are the export.

Catalog record

CREATORYonggan Fu*, Xin Dong*, Shizhe Diao, Matthijs Van keirsbilck, Hanrong Ye, Wonmin Byeon, Yashaswi Karnati, Lucas Liebenwein, Hannah Zhang, Nikolaus Binder, Maksim Khadkevich, Alexander Keller, Jan Kautz, Yingyan (Celine) Lin † , Pavlo Molchanov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18890

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models Yonggan Fu*, Xin Dong*, Shizhe Diao, Matthijs Van keirsbilck, Hanrong Ye, Wonmin Byeon, Yashaswi Karnati, Lucas Liebenwein, Hannah Zhang, Nikolaus Binder, Maksim Khadkevich, Alexander Keller, Jan Kautz, Yingyan (Celine) Lin † , Pavlo Molchanov Abstract Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has primarily focused on reducing the number of parameters to achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. This work aims to identify the key determinants of SLMs’ real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the primary consideration. Specifically, we identify two central architectural factors: depth–width ratios and operator choices. The former is crucial for small-batch-size latency, while the latter affects both latency and large-batch-size throughput. In light of this, we first study latency-optimal depth–width ratios, with the key finding that although deep–thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy–latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency-optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy–latency frontier. In addition to architectural improvements, we further enhance SLM training using a weight normalization technique that enables more effective weight updates and improves final convergence. This technique can serve as a generalizable component for future SLMs. Combining these methods, we introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy–efficiency frontier of state-of-the-art SLMs, e.g., achieving over +5.5% average accuracy, 1.3 × \times /1.9 × \times lower latency, and 18.7 × \times /45.6 × \times higher throughput compared to Qwen3-1.7B/0.6B, respectively. Models on Hugging Face: Nemotron-Flash-1B | 3B | 3B-Instruct 1 Introduction Recent advances in large language models (LLMs) have driven remarkable breakthroughs across a wide range of applications and industries. Despite their impressive capabilities, the substantial computational demands and high latency of LLMs present significant barriers to practical deployment, particularly in latency-sensitive scenarios or on resource-constrained hardware. This challenge has intensified the demand for small language models (SLMs). Most existing SLM designs prioritize parameter reduction to achieve efficiency; however, parameter-efficient models do not necessarily yield proportional latency reductions, especially on hardware AI accelerators like GPUs and TPUs. Additionally, current SLM development processes rely heavily on empirical trial-and-error rather than systematic and principled methodologies. For instance, deep-thin architectures —such as those employed in MobileLLM [ 28 ] and SmolLM [ 3 ] —often result in suboptimal accuracy-latency trade-offs. Furthermore, with the rapid advent of efficient attention operators [ 6 , 24 ] , the potential synergies of combining these operators in hybrid models have not been thoroughly explored [ 26 , 53 , 13 , 9 ] , leading to manual, heuristic-driven architecture decisions that are increasingly costly and less scalable. To address these gaps, this paper introduces generalizable principles and automated methodologies for latency-optimal SLM design and training. We perform a comprehensive study of both architectural choices and training strategies to understand their impact on efficiency and accuracy. Based on these insights, we propose a series of techniques for constructing latency-optimal SLMs and analyze their general applicability and effectiveness. These advancements are integrated into a new family of SLMs, which we refer to as Nemotron-Flash. Figure 1: Visualizing (a) the accuracy–latency trade-off and (b) the accuracy-throughput trade-off of our Nemotron-Flash and SOTA SLMs, where the average accuracy is computed across 16 tasks spanning commonsense reasoning, math, coding, and recall tasks. Latency is measured on an NVIDIA H100 GPU for decoding 8k tokens with a batch size of 1 using CUDA Graph. Decoding throughput is measured with a 32k-token input length using the maximum batch size that does not cause out-of-memory (OOM) errors for each model. The marker size represents the model depth. Specifically, our architectural exploration focuses on two key factors: depth–width ratios and operator selection, where the former is crucial for small-batch-size latency and the latter affects both latency and large-batch-size throughput. Through extensive training and profiling, we show that (1) deep-thin models, while parameter-efficient, yield suboptimal latency–accuracy trade-offs, and (2) the optimal depth–width ratio scales with the target latency constraints. Guided by this general principle, we also extend existing scaling laws [ 21 ] to relate model loss to both depth and width. This allows the sweet-spot depth–width ratio to be determined by profiling a range of configurations and selecting the one that meets the latency constraint while minimizing loss, as predicted by the scaling law. Further, we comprehensively evaluate emerging attention operators for their accuracy-latency trade-offs and potential as SLM building blocks. With these findings, we introduce an evolutionary search framework that efficiently identifies optimal combinations of hybrid attention operators. Our search strategy exploits the early stabilization of performance rankings among LM architectures, using short training runs as reliable proxies for final performance, thereby enabling fast and reliable search. In addition to architectural modifications, based on observations of structural patterns in the weight matrices of trained LMs, we further enhance SLM training by constraining weight norms to increase the effective learning rate. This consistently improves final convergence and downstream accuracy across model families. We also employ learnable meta tokens [ 13 ] for cache initialization. By combining these architectural and training innovations, we develop the Nemotron-Flash model family, which significantly advances the accuracy–latency trade-offs for SLMs. As shown in Fig. 1 , Nemotron-Flash markedly pushes forward the accuracy–efficiency frontier compared to state-of-the-art (SOTA) SLMs. For example, with all models accelerated using TensorRT-LLM’s AutoDeploy kernels [ 32 ] and CUDA Graph, Nemotron-Flash-3B achieves +2.0%/+5.5% higher average accuracy, 1.7 \times /1.3 \times lower latency, and 6.4 \times /18.7 \times higher throughput compared to Qwen2.5-3B/Qwen3-1.7B, respectively. Similarly, Nemotron-Flash-1B achieves +5.5% higher average accuracy, 1.9 \times lower latency, and 45.6 \times higher throughput than Qwen3-0.6B. Figure 2: The accuracy–parameter/latency trade-offs when varying depth and width. While deeper models generally achieve a better accuracy–parameter trade-off, they may not perform as well in the accuracy–latency trade-off and there exists an optimal depth-width ratio for a latency budget. 2 Related Work Small language models. The large model size and computational demands of LLMs [ 60 , 22 , 51 , 2 , 46 , 47 , 45 , 1 ] hinder their efficient deployment on resource-constrained platforms. This has motivated the development of SLMs, such as MobileLLM [ 28 ] , MiniCPM [ 21 ] , PanGu- \pi Pro [ 44 ] , and TinyLlama [ 59 ] . These works aim to deliver parameter-efficient SLMs within a given parameter budget. However, parameter efficiency alone often does not translate into proportional latency reductions on real devices. For example, previous SLMs [ 28 , 44 ] often adopt deep-thin model structures, which may result in suboptimal latency-accuracy trade-offs. Our work targets real-device efficiency and aims to provide insights and methodologies for developing latency-optimal SLMs. Efficient attention alternatives. To address the quadratic computation and linearly increasing memory of attention modules, efficient attention alternatives with sub-quadratic complexity in sequence length have been proposed [ 35 , 43 ,

Read the complete text