Efficient allocation of image recognition and LLM tasks on multi-GPU system

Marcin Lawenda, Krzesimir Samborski, Kyrylo Khloponin, Łukasz Szustak · 2025

Read Efficient allocation of image recognition and LLM tasks on multi-GPU system by Marcin Lawenda, Krzesimir Samborski, Kyrylo Khloponin, Łukasz Szustak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15252.txt.

Introduction

Marcin Lawenda and co-authors study how to split machine learning work across many graphics cards, in a 2025 paper. Training image models and tuning large language models both take heavy compute. How tasks are shared across cards, and what data is fed, can change each step's speed a lot.

The team compared simple and spread-out sharing methods for image training. They also tested how data set type changed tuning time for language models. Built-in tools from a common framework handled the sharing, and profiling tracked memory and transfer costs. Tests on high-end cards showed which setups ran best and where time was lost.

Catalog record

CREATORMarcin Lawenda, Krzesimir Samborski, Kyrylo Khloponin, Łukasz Szustak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15252

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient allocation of image recognition and LLM tasks on multi-GPU system Marcin Lawenda, Krzesimir Samborski, Kyrylo Khloponin, Łukasz Szustak Abstract Abstract This work is concerned with the evaluation of the performance of parallelization of learning and tuning processes for image classification and large language models. For machine learning model in image recognition, various parallelization methods are developed based on different hardware and software scenarios: simple data parallelism, distributed data parallelism, and distributed processing. A detailed description of presented strategies is given, highlighting the challenges and benefits of their application. Furthermore, the impact of different dataset types on the tuning process of large language models is investigated. Experiments show to what extent the task type affects the iteration time in a multi-GPU environment, offering valuable insights into the optimal data utilization strategies to improve model performance. Furthermore, this study leverages the built-in parallelization mechanisms of PyTorch that can facilitate these tasks. Furthermore, performance profiling is incorporated into the study to thoroughly evaluate the impact of memory and communication operations during the training/tuning procedure. Test scenarios are developed and tested with numerous benchmarks on the NVIDIA H100 architecture showing efficiency through selected metrics. 1 Introduction The rise of Machine Learning (ML) along with Large Language Models (LLMs) has revolutionized various domains such as natural language processing, computer vision, and data analysis. Known for their extensive parametrization and complex structures, these models require significant computational resources for both training and inference. Multi-card GPU systems, which leverage the parallel processing capabilities of multiple graphics processing units (GPUs), have become the essential infrastructure to meet these computational requirements [ 1 ] . This approach has resulted in the development of code specifically designed for parallelization, enabling faster training of much larger models by leveraging the increased computational power and memory of additional accelerators. However, efficient task allocation in multi-card GPU systems remains a significant challenge due to the inherent complexity of the model architecture, data dependencies, and hardware heterogeneity [ 2 ] . The primary objective of this work is to analyse the distribution of data chunks in single and multi-GPU environments to enhance the processing efficiency of machine learning applications. Through experiments, different techniques are evaluated for scalability and execution capabilities, leading to improved planning and recommendations for future applications and enhancements. The study involves GPU benchmarking with neural models and data distribution strategies to optimize training processes. The aim is to assess the scalability in a multi-GPU setup to understand the performance of hardware-software configurations in large-scale computing. Selecting the most suitable approach for a specific application’s workflow is a key challenge, especially when utilizing multiple GPU cards across nodes. The findings from these tests will offer valuable insights for refining models and tools in the future. 2 Experimental environment and methodology 2.1 Hardware infrastructure The hardware platform utilized for this research is built on the HPE Cray XD665 system, which comprises of 2 AMD EPYC 9334 CPUs and 4 NVIDIA H100-94 SXM5 GPUs. There are a total of 64 CPU cores, based on Zen 3 architecture and 2.7 GHz clock speed. The system’s RAM capacity per node amounts to 768 GB, complemented by NVIDIA H100-94 SXM5 GPUs, each equipped with 94GB of HBM2e memory, resulting in a total HBM2e memory capacity of 376 GB per node. The total GPU computational power across all four GPUs in each node is rated at 268 TFLOPS. The GPUs are interconnected via NVIDIA NVLink NV6. The environment is running on Rocky Linux 9.3 with a kernel version of k08r01s02.novalocal (version 5.14.0-362.24.1.el9_3.0.1) with CUDA 12.4, Nsight system version 2024.4.1 and SLURM version 23.11.6. 2.2 Benchmarking software and procedures Implementation frameworks. The fine-tuning of Large Language Models was conducted utilizing TorchTune - a library within the PyTorch ecosystem. Torchtune uses recipes for single-device and distributed tuning, depending on whether one or more GPUs were used. These recipes can be subsequently modified using custom configuration files in order to perform a wide array of tests [ 3 ] . Profiling tools. NVIDIA Nsight Systems is a comprehensive system-wide performance analysis tool designed to optimize and profile applications running on NVIDIA GPUs. It provides detailed insights into CPU and GPU interactions, memory usage, and the performance of CUDA kernels and APIs. By capturing a holistic view of the system’s behavior, nsys helps identify bottlenecks, analyze workload distribution, and optimize both CPU and GPU performance. 2.3 Optimization techniques Precision. The investigated models support variable precision in computations. For our experiments, we utilized both double (64-bit), float (32-bit) and half (16-bit) precision. When reducing the precision of computations from double to float or half, several significant changes occur in the system’s performance and resource utilization. The most immediate impact is a reduction in memory usage, as each value now occupies half the space. This reduction allows for larger batch sizes or higher resolution images to be processed simultaneously, enhancing the throughput of the model. Pin_memory. During the execution of the algorithm, there is active communication and data exchange between the CPU and GPU. Among the investigated parameters pin_memory showed positive effects on the algorithm’s performance. This is a parameter of DataLoader [ 4 ] that forces the system to use only page-locked memory and prevents intermediate data from being swapped to disk. By locking the memory pages in RAM, it allows for faster and more efficient data transfers between the CPU and GPU. 2.4 Selected profiler metrics As a result of the conducted tests, a comprehensive review of the metrics offered by the profiling tools was undertaken, leading to the selection of several key metrics. Below are some selected metrics that have the greatest impact on the performance of the tested applications. Table 1: Selected profiler metrics used for profiling Metric Description CUDA memcpy Host-to-Device time taken for data transfer from the CPU memory to the GPU memory cudaLaunchKernel time taken to launch a GPU kernel from the host cudaStreamSynchronize function from the CUDA API designed to synchronize task execution on the GPU ncclDevKernel_AllGather time taken by the NCCL to perform the All-Gather operation using the Ring algorithm with the Low-Latency protocol cudaEventSynchronize function is used to synchronize CPU thread with the GPU by forcing it to wait until all operations linked to a particular event are fully completed cudaEventDestroy function is employed to properly release the memory and other resources associated with an event, once it is no longer needed cudaMemcpyAsync asynchronous memory copy function that enables non-blocking data transfers between different memory spaces, such as from CPU to GPU, or between device memories 3 Profiling of image recognition 3.1 Benchmarking model Mobilenet v2. MobileNet v2 [ 5 ] is a convolutional neural network with a depth of 53 layers, specifically designed for efficient performance in resource-constrained environments, such as mobile devices. The architecture is characterized by an inverted residual structure, wherein the input and output of each residual block are narrow bottleneck layers, while channel expansion occurs within the intermediate layers. This design contrasts with traditional residual models, where expanded representations are utilized at the input and output. MobileNet v2 also employs depthwise separable convolutions, significantly reducing the computational load and the number of parameters, while maintaining high accuracy. In addition, the removal of non-linearities in the narrow bottleneck layers

Read the complete text