Profiling and optimization of multi-card GPU machine learning jobs

Marcin Lawenda, Kyrylo Khloponin, Krzesimir Samborski, Łukasz Szustak · 2025

Read Profiling and optimization of multi-card GPU machine learning jobs by Marcin Lawenda, Kyrylo Khloponin, Krzesimir Samborski, Łukasz Szustak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22905.txt.

Introduction

Marcin Lawenda, Kyrylo Khloponin, Krzesimir Samborski, and Łukasz Szustak examine how to profile and speed up machine learning jobs that run on several graphics cards at once, in a 2025 paper. Big training jobs are costly, so both result quality and running cost matter. The team compared ways of splitting image-recognition work across hardware and software setups, and studied several tuning methods for large language models.

Their tests ran on modern NVIDIA H100 cards. They tracked how each choice changed step time, memory use, and data movement between cards. Fine-tuning tricks such as LoRA and its quantized cousins had clear effects on the tuning process. The results show that the best setup depends on the task, and they give engineers measured guidance for multi-card training.

Catalog record

CREATORMarcin Lawenda, Kyrylo Khloponin, Krzesimir Samborski, Łukasz Szustak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22905

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Profiling and optimization of multi-card GPU machine learning jobs Marcin Lawenda, Kyrylo Khloponin, Krzesimir Samborski, Łukasz Szustak Abstract The effectiveness and efficiency of machine learning methodologies are crucial, especially with respect to the quality of results and computational cost. This paper discusses different model optimization techniques, providing a comprehensive analysis of key performance indicators. Several parallelization strategies for image recognition, adapted to different hardware and software configurations, including distributed data parallelism and distributed hardware processing, are analyzed. Selected optimization strategies are studied in detail, highlighting the related challenges and advantages of their implementation. Furthermore, the impact of different performance improvement techniques (DPO, LoRA, QLoRA, and QAT) on the tuning process of large language models is investigated. Experimental results illustrate how the nature of the task affects the iteration time in a multiprocessor environment, VRAM utilization, and overall memory transfers. Test scenarios are evaluated on the modern NVIDIA H100 GPU architecture. \papertype Original Article \paperfield Journal Section \abbrevs HPC, High-Perfornce Computing; LLM, large language model; ML, machine learning. \corraddress Marcin Lawenda, Poznan Supercomputing and Networking Center, Jana Pawła II 10, Poznań, 61-139, Poland \corremail [email protected] \fundinginfo Funded by the European Union, European High Performance Computing Joint Undertaking and Poland, Germany, Spain, Hungary, France and Greece under grant agreement number: 101093457. keywords machine learning, large language models, performance assessment, profiling, multi-card GPU jobs, NVIDIA H100 1 Introduction Artificial intelligence methods, especially machine learning, are the subject of many discussions and studies. Their authors focus mainly on developing new models for analysing phenomena, spending much less time on optimization techniques, especially general-purpose ones that can be used in many implementations. The importance of the issue of efficient use of available resources is evidenced by the fact that the estimated energy consumption for data centres will reach 21% in 2030 (from 2% currently) of the total global energy demand, constituting a significant burden for the electric grid [ 1 ] . This is particularly applicable in domains like machine learning (ML) and large language models (LLM), which have significantly impacted advancements across various fields, including natural language processing, computer vision, and data analysis [ 2 , 3 , 4 ] . These models are characterized by their extensive parametrization and intricate architectures, necessitating substantial computational resources for both training and inference. Multi-card GPU systems, which utilize the parallel processing capabilities of several graphics processing units (GPUs), have become crucial infrastructure to fulfil these computational demands [ 5 , 6 ] . This strategy has led to the creation of code tailored for parallelization, facilitating the accelerated training of significantly larger models by harnessing the enhanced computational power and memory provided by additional accelerators. Nevertheless, the efficient allocation of tasks within multi-card GPU systems continues to pose a considerable challenge, owing to the complex nature of model architectures, data dependencies, and the variability of hardware [ 7 ] . A major challenge for many AI applications is improving their performance, either through scalability or algorithmic improvements [ 8 ] . One effective approach to addressing these challenges and implementing complex tasks is to run machine learning or LLM algorithms on specialized hardware accelerators. Adapting an application to a graphics processing unit (GPU) that was originally designed for the CPU can provide significant performance improvements, but this requires in-depth knowledge of the code itself and the hardware capabilities [ 9 , 10 ] . This involves using various techniques related to process allocation and data management based on lessons learned from previous experiments. Determining the most appropriate method for a particular application workflow while maintaining the integrity of the results presents a significant challenge, particularly in environments utilizing multiple GPU cards across nodes. Consequently, the main aim of this study is to examine various optimization strategies, including data fragmentation, computational precision, memory management, and parameter tuning, in both single and multi-GPU settings. This analysis seeks to improve the processing efficiency of machine learning applications within the realm of AI development. The experiments conducted will assess the scalability and practicality of different techniques, ultimately facilitating enhanced planning and providing recommendations for related applications and advancements. 2 Related works The implementation of effective AI simulations requires a comprehensive understanding of the mechanisms by which they rely. Therefore, different techniques are used to evaluate performance and look for improvements, depending on the architecture of the test environment and the models used. The article [ 11 ] presents an extensive analysis that integrates programming models, architectures, and applications. It compares four precise Spiking Neural Network models (Hodgkin-Huxley, Machine Learning, Wilson, and Izhikevich) within a two-tier character recognition network, distinguished by varying computation to communication ratios. Performance evaluations were conducted on Nvidia Fermi and AMD/ATi Radeon architectures utilizing the OpenCL (Open Computing Language) and CUDA programming models. Subsequently, optimization strategies were implemented, and the problem size was adjusted. The findings indicated that CUDA exhibited superior performance on the Nvidia GPU architecture in comparison to OpenCL. In most instances, both kernel invocations and memory transfers were more efficient with CUDA. The performance disparity is primarily attributed to the differing compilers employed by the two programming models. Nonetheless, there are instances where OpenCL may surpass CUDA, particularly in specific implementations of the Hodgkin-Huxley model. Notably, significant application speedups of up to 1095 times for the most computationally demanding SNN neuron model have been documented when compared to a serial implementation on an Intel Core 2 Quad host. The paper [ 12 ] focuses on an optimization that achieves an order of magnitude speedup in network size by exploiting the parallelism between network operations compared to a naive implementation. The optimization is divided into three steps (first, single-cell optimization, second, single-layer optimization, and third, whole network optimization), which are part of the cuDNN (CUDA Deep Neural Network) library provided by NVIDIA. The effect of various factors on the inference time of a Long Short-Term Memory (LSTM) network was examined, specifically focusing on hidden state size, minibatch size, and the number of layers within the network. The results highlight the influence of optimization techniques on the forward pass of a four-layer LSTM network, which operates over 1000 steps, utilizes a hidden state size of 512, and processes an input minibatch of 64. In this scenario, an approximate speedup of 11 times is realized in comparison to a basic implementation, and a speedup of around 6 times is noted when contrasted with an implementation that employs standard GEMM (GEneral Matrix-matrix Multiplication) clustering optimization. The work [ 13 ] outlines the design, implementation and assessment of the PyTorch distributed data processing module. It discusses the effects of employing native PyTorch methods to enhance distributed data processing, such as gradient clustering, the overlap of computation with communication, and the omission of gradient synchronization. DDP (DistributedDataParallel), which facilitates parallel training across multiple processes and machines, accelerates this process by aggregating gradients into communication containers, overlaying communication with computations, and skipping synchronization. The findings indicate that, when configured appropriately, the PyTorch distributed data processing module can attain nearly linear scalability with the utilization of 256 GPUs. Furthermore, the experiments conducted indicated that the backpropagation step in DDP represents the most resource-intensive phase of training. This necessitates collaboration from both framework developers, who must facilitate the optimization algorithms, and application developers,

Read the complete text