OverFill: Two-Stage Models for Efficient Language Model Decoding

Woojeong Kim, Junxiong Wang, Jing Nathan Yan, Mohamed Abdelfattah · 2025

Read OverFill: Two-Stage Models for Efficient Language Model Decoding by Woojeong Kim, Junxiong Wang, Jing Nathan Yan, Mohamed Abdelfattah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08446.txt.

Introduction

Woojeong Kim and co-authors (2025) present OverFill, which makes large language models generate text more cheaply at no quality cost. Running one has two phases. In prefill, the model reads the prompt, work limited by raw computing power. In decoding, it writes one token at a time, work limited by memory bandwidth, and this phase dominates the wait. Standard models use the full network for both phases, though their demands differ.

OverFill splits the job. The model handles prefill, digesting inputs in parallel and building a representation. Generation switches to a pruned copy of the model, which writes tokens faster. Because the heavy model front-loads the thinking, quality holds up. A system shrinking from three billion to one billion parameters beats comparable pruned models by 83 percent on average benchmarks. A larger version improves similarly. The trade offers serving systems better accuracy per cost.

Catalog record

CREATORWoojeong Kim, Junxiong Wang, Jing Nathan Yan, Mohamed Abdelfattah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08446

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OverFill: Two-Stage Models for Efficient Language Model Decoding Woojeong Kim, Junxiong Wang, Jing Nathan Yan, Mohamed Abdelfattah Abstract Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs. LLM inference comprises prefill (compute-bound) and decode (memory-bound) stages, with decode dominating latency particularly for long sequences. Current decoder-only models handle both stages uniformly, despite their distinct computational profiles. We propose OverFill, which decouples these stages to optimize accuracy-efficiency tradeoffs. OverFill begins with a full model for prefill, processing system and user inputs in parallel. It then switches to a dense pruned model, while generating tokens sequentially. Leveraging more compute during prefill, OverFill improves generation quality with minimal latency overhead. Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average across standard benchmarks. OverFill matches the performance of same-sized models trained from scratch, while using significantly less training data. Our code is available at https://github.com/friendshipkim/overfill . 1 Introduction Large language models (LLMs) have achieved remarkable success on a broad spectrum of tasks, from question answering to code generation. Yet, their massive parameter counts pose significant challenges for practical deployment, with inference emerging as a chief bottleneck. In modern LLMs, inference typically comprises two stages: prefill and decode . The prefill stage, where all input tokens are processed in parallel to build a Key-Value (KV) cache, is usually compute-bound : the performance is primarily limited by the utilization of computational units. The subsequent decode stage is memory-bound , where it generates each output token autoregressively. The main bottleneck here is repeatedly loading the model’s large feed-forward (FFN) layers into memory. However, current LLM architectures do not exploit the distinct computational characteristics of these two stages. Our motivation stems from the distinct computational profiles of the prefill and decode stages, which have led to a growing trend of disaggregating them. The first line of work ( Zhong et al. 2024 ; Patel et al. 2024 ) is system-oriented, focusing on stage-specific resource allocation and parallelism while still using a single model across both stages. The second line of work (algorithmic-oriented) ( Nair et al. 2024 ; Bergner et al. 2024 ) explores using models of different sizes for each stage, but often requires complex frameworks or delivers only small accuracy improvements. The question of interest is how to decouple the prefill and decode stages to achieve a stronger balance of accuracy and efficiency. Figure 1: Overview of OverFill. OverFill uses the full model for prefill and a pruned model for sequential decoding. The yellow blocks represent the full model used for prefill. The decoder is initialized by selecting important channels (green blocks) from the full model. The blocks on the decoder side are in darker shades because they are updated during OverFill training. However, the full prefill model is kept frozen. We propose OverFill , a two-stage system that dedicates maximal capacity to the prefill stage, while pruning the costly decode stage to reduce parameter loading. Specifically, the larger model is used only once to process the user input into a vector representation, and a smaller, pruned model subsequently handles token-by-token generation. As a result, OverFill drastically cuts the memory footprint and latency of decoding, especially for longer sequences. While it incurs a small prefill overhead compared to the standalone pruned model, this difference in cost is negligible compared to the dominant decoding latency. Moreover, OverFill starts with a single model and prunes it to build a smaller decoder, eliminating the extensive process of aligning two distinct models. Since only the small decoder is updated during training, OverFill is compatible with the original full model during serving. Importantly, OverFill is end-to-end trainable and does not require additional modules, allowing it to be optimized like any standard transformer architecture. Figure 1 illustrates the overview of our framework. We validate OverFill in diverse decoder scales and compare OverFill accuracy on tasks such as question answering, math, and chain-of-thought reasoning. In a 3B-to-1B configuration, OverFill outperforms standalone pruned models by 83.2% and instruction-tuned model by 52.6% in average. OverFill matches or even outperforms similarly sized models trained from scratch, while using significantly fewer training tokens. We also demonstrate OverFill is pareto-optimal in both scales. OverFill achieves significant accuracy gain over standalone small models while posing minimal latency overhead. These efficiency gains become even more pronounced when long outputs or multiple candidates are generated, as the pruned model remains active throughout the autoregressive process. 2 Related work Model compression. Pruning enhances the efficiency of LLMs by removing model components that contribute the least to the output. Structured pruning eliminates entire groups of parameters, such as channels, attention heads ( Dery et al. 2024 ; Ma et al. 2023 ; Xia et al. 2023b ; Ashkboos et al. 2024 ) or layers ( Men et al. 2024 ; Yang et al. 2024 ; Kim et al. 2024 ) . Structured pruning results in a more compact model while preserving the underlying architecture, keeping it hardware-friendly. In this work, we adopt width pruning ( Dery et al. 2024 ; Ma et al. 2023 ; Xia et al. 2023b ; Ashkboos et al. 2024 ) , which preserves accuracy better than depth pruning. We specifically use the approach from Sreenivas et al. 2024 but without the additional KL loss term, focusing on optimizing pruning to balance performance and computational efficiency. Quantization ( Frantar et al. 2022 ; Lin et al. 2024 ; Xiao et al. 2023a ) is another effective model compression method. Standard quantization methods accelerate both prefill and decode. In our two-stage decoding process, we can also consider high-precision prefill and low-precision decode, which we leave for future work. Decoding targeted speedups. Various methods tackle the serial bottleneck in LLM decoding. Speculative decoding ( Leviathan et al. 2023 ) leverages available compute to propose tokens in parallel using a small draft model. Researchers have explored specialized draft models ( Sun et al. 2021 ; Xia et al. 2023a ) and subnetworks of the target model ( Schuster et al. 2022 ; Elhoushi et al. 2024 ; Zhang et al. 2023a ; Liu et al. 2024a ; Ankner et al. 2024 ) . Among these, Du et al. 2024 ; Li et al. 2024 are particularly relevant to our work as they reuse target model representations to enhance drafting. Unlike speculative decoding, our approach eliminates rollbacks and calls the large model only once during prefill, avoiding parallel execution with the small model during decoding. This significantly reduces memory usage. We provide a theoretical analysis in the Appendix. KV cache compression. Many studies have explored KV cache compression to address memory bottlenecks with heavy batching and long contexts, using methods like token eviction ( Xiao et al. 2023b ; Zhang et al. 2023b ; Adnan et al. 2024 ) , quantization ( Sheng et al. 2023 ; Liu et al. 2024b ) , and prompt compression ( Pan et al. 2024 ; Wingate et al. 2022 ) . Our approach targets scenarios where loading weights is the primary memory bottleneck. While KV cache can dominate memory usage in certain scenarios, this typically occurs only at very long sequence lengths when

Read the complete text