SOMA: Efficient Multi-turn LLM Serving via Small Language Model
Xueqi Cheng 1 Qiong Wu 2 Zhengyi Zhou 2 Xugui Zhou 3 Tyler Derr 4 Yushun Dong 1 1 Florida State University 2 AT&T Chief Data Office 3 Louisiana State University 4 Vanderbilt University {xc25,yushun.dong}@fsu.edu; {qw6547,zz547k}@att.com [email protected]; [email protected] · 2026
Read SOMA: Efficient Multi-turn LLM Serving via Small Language Model by Xueqi Cheng 1 Qiong Wu 2 Zhengyi Zhou 2 Xugui Zhou 3 Tyler Derr 4 Yushun Dong 1 1 Florida State University 2 AT&T Chief Data Office 3 Louisiana State University 4 Vanderbilt University {xc25,yushun.dong}@fsu.edu; {qw6547,zz547k}@att.com [email protected]; [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11317.txt.
Introduction
Cheng and five colleagues propose SOMA, a cheaper way to serve large language models across multi-turn chats. Standard serving re-sends the full chat history each turn, which costs latency, memory and money. SOMA learns from early turns to map the local response space, then adapts a smaller surrogate model to that region for the rest of the chat. Soft prompts, anti-degeneration control and localized fine-tuning let the small model run alone, with a gate that can switch back on drift.
Catalog record
| CREATOR | Xueqi Cheng 1 Qiong Wu 2 Zhengyi Zhou 2 Xugui Zhou 3 Tyler Derr 4 Yushun Dong 1 1 Florida State University 2 AT&T Chief Data Office 3 Louisiana State University 4 Vanderbilt University {xc25,yushun.dong}@fsu.edu; {qw6547,zz547k}@att.com [email protected]; [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75580 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11317
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SOMA: Efficient Multi-turn LLM Serving via Small Language Model Xueqi Cheng 1 Qiong Wu 2 Zhengyi Zhou 2 Xugui Zhou 3 Tyler Derr 4 Yushun Dong 1 1 Florida State University 2 AT&T Chief Data Office 3 Louisiana State University 4 Vanderbilt University {xc25,yushun.dong}@fsu.edu; {qw6547,zz547k}@att.com [email protected]; [email protected] Abstract Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models’ responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://github.com/LabRAI/SOMA . 1 Introduction Large language models (LLMs) such as the GPT series ( radford2019language ; brown2020language ; achiam2023gpt ) , LLaMA ( touvron2023llama ) , Claude ( anthropic_introducing_claude_2023 ) , and DeepSeek ( guo2025deepseek ) have demonstrated strong performance in real-world machine-learning-as-a-service (MLaaS) applications, ranging from chat assistants to code generation ( park2023thinking ; dong2023towards ; liu2024exploring ; li2026llmclinicalgraphstructure ; yu2026health ) . As LLMs are increasingly deployed in interactive settings, multi-turn LLM serving , involving extended interactions between humans and LLMs or among multiple LLM agents, has emerged as a key research focus, as it better reflects real-world usage scenarios ( yi2024survey ; li2025beyond ; zhao2025amplifying ) . Existing research reveals that multi-turn interactions are widespread, underscoring the need for serving systems capable of handling extended conversations in a context-aware manner ( chen2024sharegpt4v ; gao2024cost ) . However, supporting efficient context-dependent multi-turn interaction remains a key challenge, as most LLM serving systems are stateless and require resending the entire conversation history, including all prior queries and responses, with each new query to generate a new response ( ananda2025beyond ; moon2025accelerating ) . This leads to redundant computation, high latency, and rising serving costs as conversations lengthen. Previous work has explored efficient multi-turn LLM serving through two main approaches. One line of work focuses on single-model methods that compress dialogue history ( wang2025recursively ; chen2024compress ; xiao2024efficient ) , retrieve memory from external modules ( melz2023enhancing ; gutierrez2024hipporag ) , or reuse attention computations ( gao2024cost ; jeong2025accelerating ; anthropic_prompt_caching_2024 ) . However, these still rely heavily on large LLMs for every turn, leading to high monetary cost, latency, and GPU usage. They also often truncate or overlook extended context, limiting reasoning over long dialogues. Another line adopts multi-model methods , routing simple queries to smaller models while escalating harder ones to larger LLMs ( behera2025towards ; schick2023toolformer ; ding2024hybrid ; shnitzer2023large ) . Yet, small models struggle to generalize across dialogue complexity, and model switching introduces additional overhead. Moreover, LLMs are known to rely on early turns ( xiao2024efficient ; laban2025llms ) , compounding the difficulty of maintaining coherence in multi-turn settings. Is it possible to achieve an efficient, context-aware multi-turn LLM serving framework that avoids recomputing the full history at every turn while preserving response quality? To achieve this goal, we perform in-depth explorations of real-world multi-turn dialogues across different domains and reveal an interesting long-tail distribution in token counts across turns: early turns are substantially longer, while later turns gradually decrease in length. This phenomenon aligns with the intuition of prior work that early turns typically carry the substantive openings set issues and anchors, including questions, requests, and proposals, while later turns are typically minimal acknowledgments ( he2018decoupling ; stolcke2000dialogue ) . This long-tail trend gives rise to an intuitive idea: since later turns are relatively short, a smaller language model might suffice to generate responses more cost-efficiently. However, the bottleneck lies in the “big head”: if the small model must still process the early, information-dense context, the response quality will be degraded. Specifically, while a small language model may behave reasonably at the start, its responses drift from the large model as the dialogue progresses because most grounding is established early, and later turns remain highly dependent on that context. Therefore, simply handing later turns to a small model without modeling the accumulated context degrades quality. To address this, the small language model must not only process shorter inputs but also approximate the larger model’s behavior within the local manifold of its output or hidden space to capture the contextual dependencies shaped by prior dialogue. This defines a local manifold approximation problem, where the small language model aims to replicate the target model’s behavior induced by the current conversational context. Building on these insights, we propose a novel framework for efficient multi-turn LLM serving that enables a small language model to locally approximate the behavior of a large language model within a constrained region of the reasoning manifold. Specifically, we present SOMA ( S oft-prompts for l O cal M anifold A pproximation) to dynamically adapt the small language model to the local behavior of the larger model conditioned on early turn interactions. This is achieved through a three-stage pipeline: (1) Soft prompt tuning , where we efficiently explore the local reasoning manifold induced by the early conversational context to identify directions of maximal behavioral divergence between the small and large language model ; (2) Localized fine-tuning , where we efficiently fine-tune the small language model on a small number of input–output pairs to align it with the larger model within this context-specific region of the manifold; and (3) Efficiency inference , where we incorporate the extractive summary to minimize computational overhead and the rollback mechanism that monitor potential topic shift to maintain service quality. Together, these components allow the small model to effectively approximate the larger model’s reasoning process within the context of a given session, enabling both cost-effective and context-aware multi-turn serving. Extensive experiments show the effectiveness of our proposed method. Overall, our contributions are: • Long-tail pattern in multi-turn dialogues. We first reveal a previously under-explored long-tail pattern in multi-turn dialogues: the first few turns concentrate heavy context, while later turns are shorter yet more dependent on previous turns. This key empirical characterization suggests that substantial computational and monetary savings can be achieved if a smaller and cheaper language model can replace a large one to process the later turns when given the accumulated context. • SOMA: Efficient multi-turn serving. It first learns soft prompts that expose the largest surrogate–original response dissimilarity, then adapts the surrogate localized LoRA accordingly, enabling prompt-free inference with a simple cosine gate for switching and rollback. • Theory analysis and empirical evaluation. We provide concentration-based bounds for switching, coverage guarantees for prompt-direction search, and suboptimality limits for