Scalable engine and the performance of different LLM models in a SLURM based HPC architecture

Anderson de Lima Luiz, Shubham Vijay Kurlekar, Munir Georges · 2025

Read Scalable engine and the performance of different LLM models in a SLURM based HPC architecture by Anderson de Lima Luiz, Shubham Vijay Kurlekar, Munir Georges (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17814.txt.

Introduction

Anderson de Lima Luiz, Shubham Kurlekar, and Munir Georges describe a computing engine for language models in a paper from two thousand twenty-five. Their system runs many models on a high-performance cluster managed by a standard scheduler. Containerized services and dynamic scheduling divide processors and memory across nodes.

They test four sizes of one model family, from small to very large. Small models handled many parallel requests with very low delay. The largest models saturated with just two users and delays over two seconds. Extra cost from containers and scheduling stayed minimal. The authors also show chatbot setups with retrieval, and they argue the design scales for both batch and live use.

Catalog record

CREATORAnderson de Lima Luiz, Shubham Vijay Kurlekar, Munir Georges
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17814

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable engine and the performance of different LLM models in a SLURM based HPC architecture Anderson de Lima Luiz, Shubham Vijay Kurlekar, Munir Georges Abstract This work elaborates on a High performance computing (HPC) architecture based on Simple Linux Utility for Resource Management (SLURM) [ 1 ] for deploying heterogeneous Large Language Models (LLMs) into a scalable inference engine. Dynamic resource scheduling and seamless integration of containerized microservices have been leveraged herein to manage CPU, GPU, and memory allocations efficiently in multi-node clusters. Extensive experiments, using Llama 3.2 (1B and 3B parameters) [ 2 ] and Llama 3.1 (8B and 70B) [ 3 ] , probe throughput, latency, and concurrency and show that small models can handle up to 128 concurrent requests at sub-50 ms latency, while for larger models, saturation happens with as few as two concurrent users, with a latency of more than 2 seconds. This architecture includes Representational State Transfer Application Programming Interfaces (REST APIs) [ 4 ] endpoints for single and bulk inferences, as well as advanced workflows such as multi-step "tribunal" refinement. Experimental results confirm minimal overhead from container and scheduling activities and show that the approach scales reliably both for batch and interactive settings. We further illustrate real-world scenarios, including the deployment of chatbots with retrieval-augmented generation, which helps to demonstrate the flexibility and robustness of the architecture. The obtained results pave ways for significantly more efficient, responsive, and fault-tolerant LLM inference on large-scale HPC infrastructures. 1 Introduction Large language models (LLMs) have transformed natural language processing by enhancing the understanding and generation of human-like text, which improves information retrieval and facilitates learning, as demonstrated in studies in [ 5 ] and [ 6 ] . However, deployment of such models poses some challenging computational challenges when such a system has to serve a significant number of users concurrently – as the latest models in [ 2 ] and [ 3 ] already require high performance hardware to run individually. This paper provides an in-depth review of a SLURM-based scalable LLM HPC architecture by which such challenges are met through seamless integration of diverse LLM models within an HPC environment. Our architecture leverages the powerful scheduling and resource management capabilities of SLURM to orchestrate inference job execution across heterogeneous HPC clusters. Dynamically generating SLURM batch scripts, it allocates required CPU, GPU, and memory resources for various inference engines such as Text Generation Inferences (TGIs) [ 7 ] and virtual Large Language models (vLLMs) [ 8 ] for their efficient parallel execution and optimal utilization of available hardware. It also allows for the integration of containerized microservices, using RESTful endpoints through FastAPI, to enable scalability and flexibility during client interactions that bridge the gap between hardware intensive HPC operations and user-facing applications. Web applications that include RESTful components (and service for large institutions) such as lecture significance evaluation in [ 9 ] and LLM based chatbots need reliance of robust backends to run their workloads reliably [ 10 ] . The paper is organized as follows: Section 2 surveys the related work in HPC scheduling and LLM inference frameworks. Section 3 describes the scalable engine architecture that includes HPC integration, network coordination, and load balancing. Section 4 focuses on its implementation: code integration, concurrency, and containerization. Section 5 presents experimental results on throughput, concurrency, overhead, and system behaviors across workloads. Section 6 concludes this work with a discussion on performance, robustness, scalability, and future research directions. 2 Related works – HPC and SLURM High Performance Computing (HPC) environments normally consist of large clusters of nodes that are interconnected by high-speed networks. These nodes consist of the conventional CPUs, different types of memory and specialized accelerators like GPUs. HPC systems are set up to maximize throughput, reducing the execution time for tasks that demand intensive hardware use and efficiently manage the available resources among numerous jobs. The concept of job scheduling is an essential aspect of the Simple Linux Utility for Resource Management (SLURM) [ 1 ] operation as it is responsible for allocating the resources available (nodes, GPUs, CPUs, RAM and storage) for each user request or query. Differently from conventional use, priority is based on the workload and queue time among the users. Inference engines such as a TGI [ 7 ] or vLLM [ 8 ] and the required environment variables for the job. Once the job starts, SLURM spawns the requested jobs using the required hardware resources to finish the job. This can also include opening IP and port information for a Representational State Transfer Application Programming Interface (REST API) [ 4 ] endpoint. The sequentially complexities described in sequence are recurrent when implementing the scalable engine with the HPC and SLURM interface. The first one includes network coordination and controller settings for readiness, statuses and intermediate results. Large LLMs need distribution strategies that encompass multiple GPUs on a single node or multiple ones where model weights might require distribution among many GPUs (model parallelism) or each node handling a portion of the inference batch (data parallelism). For these cases, the call on large enough models must ensure correct orchestration for merging results or distributing new requests. Fault tolerance and scaling out are also considered in our architecture. Node failures or canceled jobs in execution might disrupt the system so the structure must be ready to re-queue and move jobs gracefully. While the inference frameworks innately handle low-level memory management and parallelism – the current structure orchestrates the HPC interaction and concurrency logic according to Fig. 1 . Figure 1: Diagram showcasing the activation path designed to provide the LLM endpoints. The user script or service can be any python code or microservice that initiates the scalable engine with a chosen model and required parameters to perform that job. The Scalable engine then reads the template and writes the parameters such as the inference engine, number of GPUs, model name and other hardware resources in the .slurm file. Afterwards, the conventional SLURM batch service is used to queue the job on the HPC system. The scheduler then handles the actual system queuing, resource allocation and dispatch of the jobs into the actual nodes [ 1 ] . The HPC cluster would be granting exclusive or shared GPU nodes specified by the SLURM script according the required resources. Each node then runs a text generation server to handle these requests. The server logs the IPs and ports of such servers upon successful startup. The HPC job logs, titled "hosts file" register the endpoint addresses and allows the scalable engine to monitor which servers are up. The hosts file is parsed and retrieve the addresses for each HPC-based inference engine. If multiple endpoints for different services are found, such as a program using a REST API "server" to deploy another frontend server from gradio [ 11 ] and streamlit [ 12 ] , the scalable engine programmatically creates an NGINX configuration, launching a container that unifies multiple endpoints into one load-balanced address. The final step returns one or more valid inference endpoints for all the created addresses to the user output. The script is then prepared to issue inference requests to any application either synchronously or asynchronously. 3 Scalable Engine and HPC architecture The

Read the complete text