Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems

Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Shao Tang, Hejian Sang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh Pillai, Luke Simon, Rahul Mazumder · 2025

Read Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems by Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Shao Tang, Hejian Sang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh Pillai, Luke Simon, Rahul Mazumder (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14305.txt.

Introduction

Kayhan Behdin and co-authors published a computing paper in 2025 on shrinking large language models for recommendation systems at a big professional network. Large models generalize well but cost too much to serve at scale. The team leans on two tools: teaching small models from large ones, and compressing models by structured pruning plus quantization. Small models keep much of the quality at a fraction of the cost.

The paper details how each trick plays out across many live use cases, from ranking to reasoning features. Hardware tuning adds further speed and throughput. The authors share deployment lessons for teams that need large-model quality under real latency and budget limits.

Catalog record

CREATORKayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Shao Tang, Hejian Sang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh Pillai, Luke Simon, Rahul Mazumder
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14305

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Shao Tang, Hejian Sang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh Pillai, Luke Simon, Rahul Mazumder Abstract Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications, from search and recommendation systems to generative tasks. Although scaling laws indicate that larger models generally yield better generalization and performance, their substantial computational requirements often render them impractical for many real-world scenarios at scale. In this paper, we present a comprehensive set of insights for training and deploying small language models (SLMs) that deliver high performance for a variety of industry use cases. We focus on two key techniques: (1) knowledge distillation and (2) model compression via structured pruning and quantization. These approaches enable SLMs to retain much of the quality of their larger counterparts while significantly reducing training/serving costs and latency. We detail the impact of these techniques on a variety of use cases in a large professional social network platform and share deployment lessons, including hardware optimization strategies that improve speed and throughput for both predictive and reasoning-based applications in Recommendation Systems. 1 Introduction Large language models (LLMs) Dubey et al. (2024) ; Jiang et al. (2023) ; Team et al. (2023) ; Liu et al. (2024) have ushered in a new era in artificial intelligence and machine learning, driving significant improvements in deployed systems across various industries. LLMs suitable for real-world applications come in diverse forms, differing in size (ranging from hundreds of millions to hundreds of billions of parameters), architectural design (e.g., encoder-based models like BERT Devlin (2018) versus decoder-based models like GPT-3 Brown et al. (2020) ), and training paradigms (such as pre-training, instruction tuning, or test-time computation Dubey et al. (2024) ; Team et al. (2023) ; Jiang et al. (2023) ; Mueller et al. (2023) ; Liu et al. (2024) ; Guo et al. (2025) ). Specifically for the Social Network Platforms, LLMs are heavily leveraged and deployed for a multitude of applications: 1. Semantic Search (e.g., embedding generation Wang et al. (2022) ; Wang et al. (2023b) and semantic ranking/matching Qin et al. (2023) ); 2. Recommendation Systems (RecSys), specifically Retrieval and Ranking Zhao et al. (2024) Li et al. (2023b) ; Firooz et al. (2025) ; Li et al. (2023a) ; 3. Generative Use cases, such as chatbots, assistants, image generators, etc. Achiam et al. (2023) ; Dam et al. (2024) ; Ramesh et al. (2022) . Furthermore, scaling laws for LLMs have established a strong correlation between model size, validation loss, and downstream task performance Kaplan et al. (2020) ; Hoffmann et al. (2022) ; Raffel et al. (2020) ; Wei et al. (2022) . As a result, increasing the size of the model is often one of the most effective strategies to enhance performance. Modern LLMs, particularly autoregressive decoder-only models, have expanded to hundreds of billions of parameters. Although large LLMs exhibit extraordinary performance, the deployment of such large models incurs substantial infrastructure costs, especially for latency- or throughput-sensitive tasks in Recommendation Systems (RecSys). However, both academia and industry have developed strategies for creating and deploying efficient small language models (SLMs). Here, we primarily focus on methods that leverage an existing internally trained large LLM to create an efficient SLM that largely maintains the original model’s accuracy. Approaches to achieve this include white-box or black-box distillation Hinton (2015) ; Gu et al. (2024) ; Jin et al. (2021) ; Agarwal et al. (2024) ; Tunstall et al. (2023) , compression techniques such as quantization Frantar et al. (2022) ; Behdin et al. (2023) and sparsification Frantar and Alistarh (2023) ; Meng et al. () ; Sun et al. (2023) ; Meng et al. (2024a) ; Campos et al. (2023) . In this work, we present a suite of insights from the training and deployment of various efficient SLMs in production at a large-scale professional social networking company. We address a wide array of predictive and generative use cases in RecSys (including ranking, recommendations, and reasoning), with inference performance and latency constraints in serving as key considerations. Our contributions are as follows. • We discuss several large-scale RecSys use cases for which language models are useful. • For these use cases, we explore techniques for developing tailored SLMs, with a focus on knowledge distillation and model compression methods such as quantization and structured pruning. • We discuss inference, latency, and other serving considerations, offering insights into the infrastructure required to reliably deploy SLMs in high-throughput or low-latency production environments, and share practical lessons from our real-world deployments. 2 Preliminaries Training details We consider models ranging in size from a billion to \sim 100\text{B} parameters. For all use cases, we appropriately tune the learning rate, learning rate warmup schedule and decay, as well as weight decay. Context length varies from a few hundred tokens to up to 32k, depending on the use case. We provide use-case specific details at the appropriate place. Prompt Structures For predictive tasks, we are mainly interested in ranking use cases. Hence, the use of decoder-based LLMs is prefill-dominant. For generative and reasoning based-tasks, we are also interested in decoding latencies. We relegate the details of the prompt structures to subsections for the specific use cases. Quality metrics We use different accuracy measures across tasks. For predictive tasks, we use area under the curve (AUC). For generative tasks, we rely on validation loss and task-specific metrics. Foundational model for RecSys We base our experiments on an internal foundation model (FM) trained using text, primarily for the purpose of ranking and recommendations Firooz et al. (2025) . The FM is a Mixture-of-Experts (MoE) model with an architecture motivated by the Mixtral family ( Jiang et al., 2024 ) . In particular, each expert is initialized based on a Llama 3.1 8B Instruct model ( Dubey et al., 2024 ) , with 16 experts in total (4 active per token). The FM is trained to approximate the following distribution for a large variety of recommendation tasks where users interact with items: P(m,(e_{1},t_{1}),...,(e_{T},t_{T})), (1) where m represents a user, and each pair (e_{t},i_{t}) for t=1,...,T represents the user’s interaction i_{t} (like or click or equivalent) with an entity e_{t} (such as post on a social media platform) . As mentioned before, the featurization is performed purely via text, allowing the FM to effectively generalize across heterogeneous tasks. The model is then used to estimate the following probabilities for future interactions with entities: P(i_{t},i_{t+1},\dots\mid\text{Task instruction},m,\\ (e_{1},i_{1}),\dots,(e_{t-1},i_{t-1}),e_{t},e_{t+1},\dots) (2) Text-based featurization makes decoder-only LLMs an attractive option for training this model jointly on a large and varied of recommendation tasks. The FM uses single-token generation for pointwise ranking and probability estimation. The reader is encouraged to peruse the paper Firooz et al. (2025) for more details about the FM. Due to the large size of the FM—which contains over 100 billion parameters—serving it online for latency-sensitive applications is challenging. In this work, we present experiments demonstrating how we achieved a more than 20 \times reduction in model size, enabling the online serving of a

Read the complete text