Efficient Multi-Model Orchestration for Self-Hosted Large Language Models

Bhanu Prakash Vangala, Tanu Malik · 2025

Read Efficient Multi-Model Orchestration for Self-Hosted Large Language Models by Bhanu Prakash Vangala, Tanu Malik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22402.txt.

Introduction

Bhanu Prakash Vangala and Tanu Malik present Pick and Spin, a system for organizations that run language models on their own hardware. Self-hosting promises privacy, cost control, and customization, but juggling several models strains graphics processor use, request routing, and reliability. Their system is built on Kubernetes with a unified Helm-based deployment setup and automation that scales idle models down to zero.

A hybrid router balances cost, latency, and accuracy using keyword heuristics plus a lightweight DistilBERT classifier. The evaluation covered four large models, from Llama-3 to DeepSeek-R1, across eight public benchmarks and more than 160,000 inference runs. Compared with static deployments, Pick and Spin raised success rates by up to about twenty-two percent while cutting latency and per-query GPU cost by roughly a third.

Catalog record

CREATORBhanu Prakash Vangala, Tanu Malik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22402

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Multi-Model Orchestration for Self-Hosted Large Language Models Bhanu Prakash Vangala, Tanu Malik Abstract Self-hosting large language models (LLMs) is increasingly appealing for organizations seeking privacy, cost control, and customization. Yet deploying and maintaining in-house models poses challenges in GPU utilization, workload routing, and reliability. We introduce Pick and Spin, a practical framework that makes self-hosted LLM orchestration scalable and economical. Built on Kubernetes, it integrates a unified Helm-based deployment system, adaptive scale-to-zero automation, and a hybrid routing module that balances cost, latency, and accuracy using both keyword heuristics and a lightweight DistilBERT classifier. We evaluate four models Llama-3 (90B), Gemma-3 (27B), Qwen-3 (235B), and DeepSeek-R1 (685B) across eight public benchmark datasets, with five inference strategies, and two routing variants encompassing 31,019 prompts and 163,720 inference runs. Pick and Spin achieves up to 21.6% higher success rates, 30% lower latency, and 33% lower GPU cost per query compared with static deployments. The results show that intelligent orchestration and efficient scaling enable enterprise-grade LLM performance on self-hosted infrastructure, bringing high-capacity AI within affordable reach. Introduction Large language models (LLMs) are rapidly transforming applications across domains. Instead of relying on a single general purpose model for all tasks, both research and industry are moving toward a diverse ecosystem of domain tuned models. These specialized models, trained for fields such as scientific research, finance, law, and healthcare, provide greater precision and contextual understanding within their respective areas. However, this specialization also introduces new challenges. Organizations must decide not only which model to use between general purpose and fine tuned variants but also how to deploy and manage them efficiently while maintaining data privacy and minimizing computational cost. This challenge has led to what can be described as the self hosting dilemma. On one side, relying on commercial APIs from providers such as OpenAI, Gemini, or Claude simplifies deployment but introduces vendor lock in, unpredictable costs, and data exposure risks that are unacceptable in sensitive domains such as healthcare, finance, or the life sciences ( 1 ) . On the other side, self hosting preserves privacy and institutional control but comes with operational burdens. Static, always on deployments keep GPUs active even when idle, wasting resources and increasing energy consumption and maintenance overhead ( 22 ) . Although domain tuned or distilled large language models (DT-LLMs) improve efficiency for specific use cases, they remain optimized for narrow objectives and cannot generalize to all query types. In practical applications, prompts vary widely, some require reasoning, others summarization or factual recall. No single model performs best across all these dimensions in there respective domains. For example, a model fine tuned for reasoning may perform poorly on summarization or fact retrieval. This diversity creates a key challenge in managing multiple models so that each input is served by the most suitable one without wasting computational resources or increasing latency. The open source ecosystem, enabled by initiatives such as LLaMA and OPT, has made high quality model weights widely available. However, efficient and affordable deployment of these models remains difficult. Each model behaves differently across tasks, and their varying computational requirements make selection and scheduling complex. Organizations must balance accuracy, responsiveness, and cost, particularly when operating within private or resource constrained environments. These challenges motivate the need for an automated system that can both select the right model for each prompt and allocate resources intelligently In this context, orchestration refers to the automated coordination of models and computational resources. It involves deciding which model to invoke, when to start or stop it, and how to allocate GPUs efficiently. Prior work in distributed systems defines orchestration as the automation and optimization of workflows to ensure scalability and reliability ( 5 ; 27 ) . Extending this principle to multi model inference enables a balance between accuracy, latency, and cost. Instead of keeping all models continuously active, the orchestrator routes simple queries to lightweight models and reserves larger ones for complex tasks. Idle models are scaled to zero, ensuring that GPU resources are used only when needed. Existing research on model serving ( 10 ) , autoscaling ( 4 ; 22 ) , and serverless inference ( 31 ; 28 ) focuses on specific parts of the orchestration pipeline but does not provide an integrated solution. These systems improve efficiency within individual layers such as inference scheduling or container scaling, yet they do not coordinate model selection and resource allocation together. Tools like Helm ( 9 ) and Knative extend Kubernetes with declarative configuration and event driven scaling, offering strong primitives for deployment automation. However, they lack mechanisms for task aware routing or model level coordination, which are essential for managing multiple language models under shared infrastructure. To address this gap, we propose Pick and Spin (PS), a multi model orchestration framework that integrates intelligent routing with orchestration aware scaling. The system’s name encapsulates its dual nature: Pick represents the intelligent routing layer that selects optimal models based on prompt complexity, while Spin represents the dynamic orchestration layer that manages model lifecycles, spinning resources up on demand and down when idle. PS formulates orchestration as a joint optimization problem balancing three objectives: model relevance, latency, and cost. A lightweight routing layer selects the best model for each query using rule based and semantic (DistilBERT) classifiers that estimate prompt complexity and intent. The orchestration layer manages model activation and deactivation using Kubernetes based scaling policies, ensuring efficient GPU utilization and minimal cold start delay. PS is implemented on Kubernetes with Helm based control, using a unified umbrella chart that automates deployment, versioning, and recovery across multiple models and backends such as vLLM, TensorRT LLM, and TGI. This design enables reproducible deployment, zero downtime upgrades, and automatic fault recovery while keeping resource usage cost efficient. By combining content aware routing, dynamic scaling, and fault tolerant orchestration, Pick and Spin transforms static, resource heavy model hosting into a scalable and adaptive workflow. It provides a foundation for private multi model AI deployments that maintain performance, privacy, and cost efficiency within institutional infrastructure. Related Work Pick and Spin (PS) connects three major research directions: efficient LLM inference, multi model orchestration, and serverless computing. Early work such as Megatron LM ( 25 ) explored large scale parallelism for training and serving billion parameter models. Modern systems like Text Generation Inference (TGI) ( 26 ) , vLLM ( 18 ) , and DeepSpeed Inference ( 2 ) focus on runtime efficiency through memory optimization and parallel scheduling. For instance, the PagedAttention mechanism in vLLM manages the KV cache efficiently to reduce fragmentation and increase throughput. FastServe ( 30 ) and SGLang ( 34 ) further streamline inference pipelines but target single model deployments, lacking cross model orchestration. The increasing diversity of models has renewed interest in dynamic routing. Conceptually, this aligns with the Mixture of Experts (MoE) paradigm where a controller routes tokens to specialized submodules within a single network ( 17 ; 23 ) . PS generalizes this concept to the system level by routing full prompts between independently deployed models. Recent systems such as UniRoute ( 15 ) and ModelSAT ( 33 ) use embeddings or instruction tuned heuristics to predict the best model efficiently, while controller based orchestration frameworks ( 29 ) leverage

Read the complete text