Serving LLMs in HPC Clusters: A Comparative Study of Qualcomm Cloud AI 100 Ultra and High-Performance GPUs Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Col
Mohammad Firas Sada, John J. Graham, Elham E Khoda, Mahidhar Tatineni, Dmitry Mishin, Rajesh K. Gupta, Rick Wagner, Larry Smarr, Thomas A. DeFanti, Frank Würthwein · 2025
Read Serving LLMs in HPC Clusters: A Comparative Study of Qualcomm Cloud AI 100 Ultra and High-Performance GPUs Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Col by Mohammad Firas Sada, John J. Graham, Elham E Khoda, Mahidhar Tatineni, Dmitry Mishin, Rajesh K. Gupta, Rick Wagner, Larry Smarr, Thomas A. DeFanti, Frank Würthwein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00418.txt.
Introduction
Sada and colleagues benchmark in 2025 Qualcomm's Cloud AI 100 Ultra accelerator for serving large language models in computing clusters. They compare it with NVIDIA A100 graphics processors inside a national research platform. A dozen open models, from small to seventy billion parameters, were served with the same software stack. The Qualcomm cards matched on energy efficiency for some models and allowed finer hardware allocation.
Some big models ran on a single Qualcomm card that needed eight graphics cards, at a fraction of the power draw. Small models also used far less power. The authors see promise for energy-limited deployments.
Catalog record
| CREATOR | Mohammad Firas Sada, John J. Graham, Elham E Khoda, Mahidhar Tatineni, Dmitry Mishin, Rajesh K. Gupta, Rick Wagner, Larry Smarr, Thomas A. DeFanti, Frank Würthwein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00418
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Serving LLMs in HPC Clusters: A Comparative Study of Qualcomm Cloud AI 100 Ultra and High-Performance GPUs Conference: Practice and Experience in Advanced Research Computing; July 20–24, 2025; Columbus, OH, USA Practice and Experience in Advanced Research Computing (PEARC ’25), July 20–24, 2025, Columbus, OH, USA DOI: 10.1145/3708035.3736087 ISBN: 979-8-4007-1398-9/2025/07 CCS: Computing methodologies Natural language processing CCS: Networks Cloud computing CCS: Hardware Hardware reliability screening Mohammad Firas Sada, John J. Graham, Elham E Khoda, Mahidhar Tatineni, Dmitry Mishin, Rajesh K. Gupta, Rick Wagner, Larry Smarr, Thomas A. DeFanti, Frank Würthwein Abstract . This study presents a benchmarking analysis of the Qualcomm Cloud AI 100 Ultra (QAic) accelerator for large language model (LLM) inference, evaluating its energy efficiency (throughput per watt) and performance against leading NVIDIA (A100, H200) and AMD (MI300A) GPUs within the National Research Platform (NRP) ecosystem. A total of 15 open-source LLMs, ranging from 117 million to 90 billion parameters, are served using the vLLM framework. The QAic inference cards appears to be energy efficient and performs well in the energy efficiency metric in most cases. The findings offer insights into the potential of the Qualcomm Cloud AI 100 Ultra for high-performance computing (HPC) applications within the National Research Platform (NRP). Bar chart comparing the energy efficiency (tokens generated per watt) of Large Language Models (LLMs) running on Qualcomm Cloud AI 100 Ultra (QAic) accelerators and GPUs. QAic configurations use enough devices to match the throughput (tokens per second) of the selected GPU for each model. For each LLM, the most efficient tested GPU (from NVIDIA or AMD MI300A) is used for comparison. Higher bars indicate greater energy efficiency, with more tokens generated per watt of power consumed. Figure 1. Bar chart comparing the energy efficiency (tokens generated per watt) of Qualcomm Cloud AI 100 Ultra AI Accelerator cards (QAic) and GPUs when running Large Language Models (LLMs). QAic accelerators each include four System-on-Chip (SoC) devices. Each LLM requires a different number of GPUs or QAic devices to run. The figure reports results using the GPU model with the highest tokens per second for each LLM (among tested NVIDIA models and the AMD MI300A) and compares them to QAic. Higher values indicate better energy efficiency. Missing bars for MI300A indicate that a given model could not run on that hardware. Bar chart comparing the energy efficiency (tokens generated per watt) of Large Language Models (LLMs) running on Qualcomm Cloud AI 100 Ultra (QAic) accelerators and GPUs. QAic configurations use enough devices to match the throughput (tokens per second) of the selected GPU for each model. For each LLM, the most efficient tested GPU (from NVIDIA or AMD MI300A) is used for comparison. Higher bars indicate greater energy efficiency, with more tokens generated per watt of power consumed. † † cc-license: by 1. Introduction In recent years, the proliferation of large language models (LLMs) has revolutionized natural language processing, driving innovation across domains. As these models grow in complexity and size, the demand for specialized hardware capable of efficiently running LLMs has intensified, particularly in data center environments where energy efficiency and cost-effectiveness are critical. To address this demand, novel AI accelerators such as the Qualcomm Cloud AI 100 Ultra (QAic) ( Qualcomm 2024 ) , AMD Instinct MI300A, and NVIDIA GH200 have emerged, offering significant improvements in performance and efficiency for Artificial Intelligence (AI) workloads involving LLMs. However, these accelerators are not universally designed for multi-task workloads that include both training and inference. Therefore, a comprehensive evaluation of their real-world capabilities on specific tasks is essential to assess their practical impact and suitability for diverse LLM applications. This study presents a benchmarking analysis of some of these cutting-edge accelerators, assessing their performance, energy efficiency, and suitability for running a diverse set of LLMs for inference applications. We consider factors like model size, parameter count, function calls, and context lengths, while also considering practical usability with state-of-the-art tools like vLLM ( Kwon et al. 2023 ) . Among the accelerators examined, the Qualcomm Cloud AI 100 Ultra is specifically designed and optimized for AI inference. By comparing these accelerators, we aim to provide valuable insights into their relative strengths in terms of energy consumption and throughput. 2. Experimental Setup Experiments are conducted on the National Research Platform (NRP), a distributed, multi-tenant Kubernetes-based infrastructure spanning more than 75 international sites ( Smarr et al. 2018 ) . The NRP integrates diverse computational resources to support advanced AI and machine learning workloads. It currently supports over 1,400 GPUs across 300+ research groups and classrooms, emphasizing flexibility through user-friendly interfaces and leveraging robust automation for streamlined deployment. All GPUs used in this study, except the AMD devices, are part of the NRP. The NVIDIA GH200 GPU is hosted within the Great Plains Extended Network of GPUs for Interactive Experimenters (GP-ENGINE) ( Hurt et al. 2024 ) , while the AMD Instinct MI300A Accelerated Processing Unit (APU) is part of the COSMOS system at the San Diego Supercomputer Center. Our study uses dedicated nodes within the NRP Kubernetes cluster using vLLM as the serving framework, reflecting typical real-world deployment patterns for LLMs in research and education. Each node is fully reserved to avoid interference from other pods or GPU requests, with consistent CPU core allocations across test pods to minimize processor-related variance. One pod per hardware type is deployed, requesting all available GPUs and memory to eliminate resource contention unrelated to AI accelerators ( Abdi et al. 2023 ) . Each pod is configured with 100 GB of shared memory for LLM operations. Local storage is mounted as persistent storage for model data, ensuring efficient loading and retrieval. The vLLM API endpoint is containerized within each pod to eliminate network bottlenecks, ensuring the measurements reflected pure compute performance. We test 15 open-source LLMs ranging in parameters from 117 million (M) to 90 billion (B) from Hugging Face ( Face 2024 ) . Inference tests are run by sending a fixed prompt to each model with a varying number of concurrent requests to the vLLM server. Power consumption is continuously captured using nvidia-smi and amd-smi for NVIDIA and AMD GPUs alike and qaic-util for the QAic cards. 2.1. Qualcomm Cloud AI 100 Ultra The Qualcomm Cloud AI 100 Ultra is a specialized accelerator for large-scale AI and language model operations. Each accelerator card features a multi-chip architecture with four System-on-Chip (SoC) units interconnected through a PCIe switch, collectively providing 576 MB of on-die SRAM and 64 dedicated AI processing cores. Individual SoCs can be allocated as discrete compute units or combined for larger workloads. It supports peer-to-peer communication within and across cards, enabled by the QAic Kubernetes Device Plugin. Running LLM inference on QAic cards is a complex process, requiring models to first be exported to ONNX format and then converted into Qualcomm’s proprietary QPC (Qualcomm Program Container) format. This conversion involves offline, hardware-specific optimizations. Although compilation can take hours, the resulting QPC artifacts are portable and can be deployed across compatible devices. As compilation is hardware-independent, cloud-based workflows are feasible. Once generated, QPC enables instant model loading, following a "compile once, run everywhere" approach ( Qualcomm Innovation Center 2025a ) . The SDK automates retrieval, ONNX export, compilation, and deployment. It also integrates