SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University School of Artificial Intelligence, Shanghai Jiao Tong University Zenergize AI · 2025
Read SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment by Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University School of Artificial Intelligence, Shanghai Jiao Tong University Zenergize AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20984.txt.
Introduction
A team from Shanghai gives a clan of small language models built from the ground up for home gear, in a 2025 paper. Top models live on cloud rigs with vast chips. Home gear has weak chips, thin store, and slow disks, which crush cloud born designs.
Their build pairs fine expert mix with sparse plain nets to cut work. A pre read router hides slow disk lag by fetch of parts in step with core math. Lean care for past text slashes store needs still more. Both clan sizes beat big peers and run fast on plain home cores.
Catalog record
| CREATOR | Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University School of Artificial Intelligence, Shanghai Jiao Tong University Zenergize AI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20984
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University School of Artificial Intelligence, Shanghai Jiao Tong University Zenergize AI Abstract Abstract While frontier large language models (LLMs) continue to push capability boundaries, their deployment remains confined to GPU-powered cloud infrastructure. We challenge this paradigm with SmallThinker, a family of LLMs natively designed—not adapted—for the unique constraints of local devices: weak computational power, limited memory, and slow storage. Unlike traditional approaches that mainly compress existing models built for clouds, we architect SmallThinker from the ground up to thrive within these limitations. Our innovation lies in a deployment-aware architecture that transforms constraints into design principles. First, We introduce a two-level sparse structure combining fine-grained Mixture-of-Experts (MoE) with sparse feed-forward networks, drastically reducing computational demands without sacrificing model capacity. Second, to conquer the I/O bottleneck of slow storage, we design a pre-attention router that enables our co-designed inference engine to prefetch expert parameters from storage while computing attention, effectively hiding storage latency that would otherwise cripple on-device inference. Third, for memory efficiency, we utilize NoPE-RoPE hybrid sparse attention mechanism to slash KV cache requirements. We release SmallThinker-4B-A0.6B and SmallThinker-21B-A3B, which achieve state-of-the-art performance scores and even outperform larger LLMs. Remarkably, our co-designed system mostly eliminates the need for expensive GPU hardware: with Q4_0 quantization, both models exceed 20 tokens/s on ordinary consumer CPUs, while consuming only 1GB and 8GB of memory respectively. SmallThinker is publicly available at hf.co/PowerInfer/SmallThinker-4BA0.6B-Instruct and hf.co/PowerInfer/SmallThinker-21BA3B-Instruct . \reportnumber 001 Figure 1: Comparative analysis of inference performance and MMLU scores. SmallThinker achieves SOTA performance, outperforming comparable models in both speed and accuracy. 1 Introduction The AI revolution has reached an inflection point. While frontier models like GPT-4, Gemini ( Comanici et al. 2025 ) , and Claude demonstrate unprecedented capabilities, they remain prisoners of the cloud—tethered to massive data centers with unlimited power and memory. Yet the future of AI lies not only in distant servers, but also in the devices we carry every day: smartphones processing sensitive conversations privately, laptops running complex analyses offline, and local devices bringing intelligence to billions without internet access. This vision demands a fundamental rethinking of how we build large language models. The prevailing paradigm for on-device AI involves adapting models originally architected for large-scale, high-memory GPU clusters. This post-hoc adaptation, typically via methods like distillation, or pruning, inherently trades model capability for on-device feasibility. This creates a dichotomy between powerful, cloud-based models and their significantly compromised on-device counterparts. We reject this problematic choice. Instead of asking “how can we squeeze cloud models onto local devices?”, we ask a more fundamental question: “what would an large language model (LLM) look like if designed from first principles for local constraints?” This question led us to train SmallThinker from ground up, a family of LLMs natively architected for local deployment. Rather than treating weak computational power, limited memory, and slow storage as obstacles to overcome, we embrace them as design principles that shape every architectural decision. As shown in Figure 1 , the result is not a compromised cloud model, but a new class of AI system that thrives within local constraints while delivering frontier-level capabilities. Our key insight is that local constraints, when properly understood, reveal opportunities for novel architectural innovations. The limited compute budget motivated our two-level sparse structure that combining fine-grained Mixture-of-Experts (MoE) with sparse feed-forward networks and only activating essential parameters for each token during inference. The slow storage bottleneck inspired our novel pre-attention router, which predicts required experts before attention computation, enabling our inference engine to hide storage I/O latency through intelligent prefetching. To further optimize storage access patterns, we introduce DP-Groups Global Load Balance Loss, which promotes expert specialization while maintaining training stability, allowing predictable activation patterns that enable effective expert caching strategies. The memory limitations drove the development of our NoPE-RoPE hybrid sparse attention, dramatically reducing KV cache requirements without sacrificing long-context understanding. Beyond architecture, we co-design the entire inference stack for local deployment. To fully exploit the inherent sparsity in ReGLU activation, we develop fused sparse FFN kernels that bypass unnecessary computations in up and down projection layers. We design a predictor for the language model head that selectively activates weight rows, dramatically reducing computational overhead. For memory-constrained environments, we leverage expert activation locality to strategically offload expert parameters to SSDs, establishing an efficient I/O-computation pipeline combined with our pre-attention router that completely hides storage latency. This holistic system design transforms theoretical sparsity into real-world speedups, enabling SmallThinker-21B-A3B and SmallThinker-4B-A0.6B to achieve what was previously thought impossible: over 20 tokens per second on ordinary consumer CPUs—without any GPU acceleration—using just 1 GiB and 8 GiB of memory respectively, while achieving state-of-the-art MMLU scores that outperform even larger LLMs. Specifically, SmallThinker-21B-A3B and SmallThinker-4B-A0.6B achieves up to 86× and 19× decoding speed improvement compared to Qwen3-30B-A3B and Qwen3-1.7B, respectively, which matches the speed of in-memory baselines like Gemma3n-E4B and Gemma3n-E2B. SmallThinker represents more than an engineering achievement—it’s a paradigm shift in how we think about AI deployment. By designing natively for local constraints, we demonstrate that the future of AI need not be limited by the reach of cloud infrastructure. Instead, we can bring truly capable AI to billions of devices worldwide, enabling a new era of private, responsive, and universally accessible artificial intelligence. 2 Model Architecture Table 1: Summary of the SmallThinker architectures. Model # of Layers Hidden Dim. Head Dim. FFN Dim. # of Q Heads # of KV Heads # of Experts Top K SmallThinker-4B-A0.6B 32 1536 128 768 12 2 32 4 SmallThinker-21B-A3B 52 2560 128 768 28 4 64 6 Figure 2: SmallThinker model architecture. Residual connections are omitted for clarity. SmallThinker series models are based on standard decoder-only Transformer architecture with several unique variants. Table 1 shows the key information on model architecture and Figure 2 shows the model architecture overview, where we highlight the key architectural decisions below. 2.1 Fine-Grained Mixture of Experts Basic Architecture of SmallThinker For MoE, prior research has demonstrated that fine-grained experts exhibit superior parameter efficiency ( Dai et al. 2024 ) . Accordingly, we adopt a fine-grained Mixture of Experts (MoE) architecture. For our 4B and 21B models, we configure them with 32 and 64 experts, respectively. Sparse ReGLU-Based FFN Although the MoE architecture already introduces structural sparsity by design, we observe that employing activation functions from the ReLU family can induce further neuron-level sparsity within each expert ( Zhang et al. 2024 ; Luo et al. 2024 ; Song et al. 2024b ) . To further reduce computational costs and I/O, we construct each FFN expert using the ReGLU activation function. Pre-Attention Router On-device platforms, such as AI PCs and smartphones, typically need to serve multiple applications concurrently, which leaves a limited memory budget for LLM serving. This constraint necessitates efficient parameter offloading. To create a sufficient time window for prefetching the required expert parameters, we position the MoE router module before the attention block. This architectural design allows us to determine which experts are needed early, enabling the prefetching of their weights to occur in parallel with the attention computation. DP-Groups Global Load Balance Loss Expert specialization refers to the behavior where