LLM & HPC:Benchmarking DeepSeek’s Performance in High-Performance Computing Tasks
Noujoud Nader, Patrick Diehl, Steve Brandt, Hartmut Kaiser · 2025
Read LLM & HPC:Benchmarking DeepSeek’s Performance in High-Performance Computing Tasks by Noujoud Nader, Patrick Diehl, Steve Brandt, Hartmut Kaiser (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03665.txt.
Introduction
Noujoud Nader, Patrick Diehl, Steve Brandt, and Hartmut Kaiser tested how well a language model writes code for supercomputers, in a 2025 paper. Such machines run huge science jobs, and their code is hard to write well. The team asked DeepSeek, a recent model, to write five standard test programs, among them a heat-flow solver and matrix routines, in four programming languages.
They checked the code for correctness, speed, and how well it scaled as problems grew. DeepSeek did produce working code for the tasks. Its code lagged behind the code written by GPT-4 in scaling and run speed, though. The authors share the full comparison so labs can judge whether such models are ready to help with high-end computing work.
Catalog record
| CREATOR | Noujoud Nader, Patrick Diehl, Steve Brandt, Hartmut Kaiser |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM & HPC:Benchmarking DeepSeek’s Performance in High-Performance Computing Tasks Noujoud Nader, Patrick Diehl, Steve Brandt, Hartmut Kaiser Abstract Large Language Models (LLMs), such as GPT-4 and DeepSeek, have been applied to a wide range of domains in software engineering. However, their potential in the context of High-Performance Computing (HPC) much remains to be explored. This paper evaluates how well DeepSeek, a recent LLM, performs in generating a set of HPC benchmark codes: a conjugate gradient solver, the parallel heat equation, parallel matrix multiplication, DGEMM, and the STREAM triad operation. We analyze DeepSeek’s code generation capabilities for traditional HPC languages like C ++ , Fortran, Julia and Python. The evaluation includes testing for code correctness, performance, and scaling across different configurations and matrix sizes. We also provide a detailed comparison between DeepSeek and another widely used tool: GPT-4. Our results demonstrate that while DeepSeek generates functional code for HPC tasks, it lags behind GPT-4, in terms of scalability and execution efficiency of the generated code. K eywords LLM ⋅ \cdot DeepSeek ⋅ \cdot GPT-4 ⋅ \cdot HPC ⋅ \cdot Code Performance ⋅ \cdot Programming language processing 1 Introduction Large Language Models (LLMs) have rapidly advanced, potentially approaching Artificial General Intelligence (AGI). Trained on vast textual data, LLMs have shown remarkable capabilities in natural language processing and visualization tasks [ 1 , 15 , 25 ] . This trend has accelerated with the release of the DeepSeek models by the Chinese company DeepSeek [ 7 ] . Their DeepSeek-V3 [ 20 , 17 ] was trained in two months at a cost of $5.6 million, significantly cheaper than similar models. In contrast, the development cost of ChatGPT-4 ranged between $41 million and $78 million [ 3 ] . On January 20th 2025, they launched DeepSeek-R1 [ 16 ] , a reasoning model exhibiting performance comparable to OpenAI’s O1 model, available for open research [ 12 ] . DeepSeek-R1 is trained through large-scale reinforcement learning (RL) without supervised fine-tuning, demonstrating powerful reasoning behaviors [ 16 ] . In the high-performance computing (HPC) domain, LLMs are being explored for tasks such as code analysis, generation, and optimization. However, there is a lack of standardized, reproducible evaluation processes for LLMs in HPC-specific tasks. For this reason, this paper explores the performance of the open-source LLM DeepSeek for simple code generation tasks in HPC. We evaluate code generation using four programming languages (PLs) selected for their prominence in HPC and based on the TIOBE index [ 27 ] . The PLs are C ++ for performance and simulation, Fortran for numerical and array optimizations, Python for flexibility and integration with other HPC languages, and Julia for scientific computing performance with a Python-like syntax. We present five examples to assess the performance of the model. The first example in this study features a conjugate gradient solver that incorporates matrix and vector operations. The second example is a parallel 1D stencil-based heat equation solver [ 10 ] , chosen for its relevance in testing parallel computational workflows and its widespread use in scientific computing. The third example is parallel matrix multiplication, a fundamental operation in scientific computing and machine learning, often accelerated by using HPC systems to enhance computational speed. The fourth example is parallel double-precision general matrix multiply (DGEMM), a widely used operation in numerical linear algebra that is essential for many scientific and engineering applications, testing the system’s computational efficiency for large-scale matrix operations. The final example is the STREAM triad operation, a key benchmark for testing memory bandwidth and data movement between CPU and memory, commonly used to evaluate the memory subsystem’s performance in HPC systems. We summarize the main contributions of this paper as follows: • A comparison of DeepSeek and GPT-4 for HPC code generation, specifically targeting five simple tasks. • A systematic evaluation of the performance and quality of code generated by DeepSeek in when writing code for four widely used HPC programming languages: C ++ , Fortran, Python, and Julia. • A comprehensive analysis of the models’ ability to generate optimized code for common HPC operations, offering insights into the potential of LLMs for high-performance computing tasks. Table 1: Summary of LLM-based HPC related work Paper Main Focus LLM Model Used LLM4HPC [ 5 ] HPC Adaptation LLaMa-2 LLM4VV [ 23 ] OpenACC Testing GPT-4, CodeLlama HPC-GPT [ 11 ] AI Model Management LLaMa-13B Tokompiler LLM [ 18 ] Code Completion GPT-3 HPC-Coder [ 24 ] OpenMP & MPI Handling DeepSpeed Dataset for OpenMP Translation [ 19 ] Code Translation LLaMa-2 LLMs in HPC [ 4 ] LLM-HPC Challenges Various LLMs Godoy et al. (2023) [ 13 ] Kernel Code Generation LLaMa-2 Valero-Lara et al. (2023) [ 28 ] LLaMa-2 Comparison LLaMa-2 Godoy et al. (2024) [ 14 ] Code Parallelization GPT-3 chatHPC [ 29 ] HPC Chatbot GPT-like (based on StarCoder) 2 Related Work LLMs have been widely used across various domains, including natural language processing (NLP) and visualization. Multiple efforts have also been made to benchmark LLMs for tasks such as code generation [ 9 , 8 , 6 ] . However, their application in analyzing and optimizing HPC tasks remains challenging. In Table 1 , we present the related works that use LLMs for HPC tasks. LLM4HPC [ 5 ] represents the first effort to adapt an LLM specifically to the HPC domain. The LLM4HPC framework is specifically designed for HPC, showing success in code similarity analysis, parallelism detection, and OpenMP question-answering tasks. LLM4VV [ 23 ] is a fine-tuned model that uses the capabilities of GPT-4 and CodeLlama (based on Llama-2) to successfully generate OpenACC directives. HPC-GPT introduced by Ding et al. [ 11 ] , was built on LLaMa-13B. HPC-GPT has been successfully applied to managing AI models and datasets, as well as detecting data races. Kadosh et al. [ 18 ] introduced the domain-specific Tokompiler LLM, which outperforms a GPT-3-based model in code completion and semantics for Fortran, C, and C ++ code. Nichols et al. [ 24 ] present HPC-Coder. Their work demonstrated varying success in code completion, including handling OpenMP pragmas and MPI calls. Lei et al. [ 19 ] introduced a dataset designed for fine-tuning models on OpenMP Fortran and C ++ code translation. Their fine-tuned model yielded more accurate results compared to GPT-4. Chen et al. [ 4 ] provide an insightful overview of the challenges and opportunities at the intersection of LLMs and HPC, extending beyond code generation. Notably, the works by Godoy et al. [ 13 ] and Valero-Lara et al. [ 28 ] , which evaluate HPC kernel code generation and results for LLaMa-2, are among the first to apply LLM-based code generation to the domain of HPC software development. Additionally, Godoy et al. [ 14 ] apply LLM capabilities of GPT-3 targeting HPC kernels for code generation, and auto-parallelization of serial code in C ++ , Fortran, Python and Julia. Yin et al. [ 29 ] proposed chatHPC, a chatbot for HPC question answering and script generation. 3 Methodology We used five examples: (1) a conjugate gradient solver, (2) a parallel one-dimensional heat equation solver using finite differencing, (3) parallel matrix multiplication, (4) parallel DGEMM, and (5) the STREAM triad operation. The complexity of the code increases