Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project

Carolin Penke, Chelsea Maria John, Jan Ebert, Stefan Kesselheim, Andreas Herten · 2025

Read Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project by Carolin Penke, Chelsea Maria John, Jan Ebert, Stefan Kesselheim, Andreas Herten (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10013.txt.

Introduction

Carolin Penke and co-authors share in 2025 best practices for training large language models on supercomputers, drawn from a German project for open multilingual models for European languages. Such training needs huge computing, complex software, and careful workflows. The report details work on a national supercomputer to train a seven billion parameter model, from system layout to daily operations.

Topics include the training infrastructure, software choices, and tools for profiling and benchmarking. Measured throughput is given for varied ways of splitting work across machines, and for features such as flash attention. Engineering and operational problems are shared openly, so other teams can avoid them when training large models.

Catalog record

CREATORCarolin Penke, Chelsea Maria John, Jan Ebert, Stefan Kesselheim, Andreas Herten
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10013

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project Carolin Penke, Chelsea Maria John, Jan Ebert, Stefan Kesselheim, Andreas Herten Abstract ash]bashbreaklines=true,fontsize= ,autogobble=true March 31, 2025 1. Introduction The field of Artificial Intelligence (AI) has experienced a rapid boost with the advent of performant Large Language Models (LLMs). These have transformed natural language processing (NLP), driving advancements in machine translation, text summarization, and conversational AI. Training these models requires vast computational resources, necessitating the use of high-performance computing (HPC) systems. Within the OpenGPT-X 1 1 1 https://opengpt-x.de/en/ project —an initiative focused on developing open and sovereign European AI language models— computational resources required for training were provided by the supercomputer JUWELS Booster Krause 2019 at the Jülich Supercomputing Centre (JSC), facilitating a scalable and optimized LLM training. A key milestone of the OpenGPT-X project is the release of Teuken- 7B Ali et al. 2024 , a 7-billion-parameter model designed to support multiple European languages with an emphasis on transparency, efficiency, and open access. Training Teuken-7B on JUWELS Booster leveraged the supercomputer’s GPU-accelerated architecture to efficiently manage large-scale data processing, optimize memory usage, and ensure high training throughput. The model’s development required advanced parallelization strategies, including data parallelism, pipeline, tensor and sequence parallelism, as well as mixed-precision training to fully utilize JUWELS Booster’s computational power. In this work, we detail the methodologies employed for training Teuken-7B on JUWELS Booster, addressing challenges related to scalability, resource efficiency, and model convergence. We also present a benchmarking framework and insights gained from this large-scale training effort. 2. Background In this section, we provide a concise overview of key developments in the field of Natural Language Processing (NLP) during the duration of the OpenGPT-X project (2022–2025). The scope of the project is outlines, emphasizing the role of HPC centers as critical partners. As a best practice, this section stresses the importance of closely following rapid advancements in NLP to maintain state-of-the-art relevance. Furthermore, given the rapid expansion of the field, we highlight the necessity for focused, specialized projects rather than attempting to address all possible areas simultaneously. 2.1. Large Language Models Given a sequece of text tokens, a language model predicts the next most-probable token and can be used for autonomous text generation in a variety of tasks. Most modern language models are deep neural networks based on the transformer architecture published in 2017 Vaswani et al. 2023 , and are trained on vast amounts of text data. Previously, variants of recurrent neural networks (RNNs), such as long-short-term memory (LSTM) networks dominated the field of sequence learning. The attention mechanism, orignally proposed as a feature of RNNs, is the main building block of the transformer architecture. This architecture avoids recurrence and convolutions, and thus training and inference can be highly parallelized. By mainly relying on matrix-matrix products, operations can be performed in an efficient manner on modern hardware such as GPUs Penke 2022 . In 2018, OpenAI published GPT (later to be known as GPT-1), a language model based on a simplified decoder-only transformer architecture, with an increased number of layers, using 117 million parameters. It introduced the paradigm of unlabeled pre-training on large corpora of text, followed by task-specific fine-tuning. The BERT model (up to 340 million parameters) was published by Google in 2019 Devlin et al. 2019 and employs a masked bidirectional pre-training objective, making it suitable as a text encoder. GPT-2 Radford et al. 2019 followed in 2018 and further increased the number of parameters to 1.5 billion, based on the transformer architecture with slight modifications. With its increased capabilities. It paved the way to move from task-specific models trained on labeled data, towards general models, that have been trained unsupervised. With GPT-3 Brown et al. 2020 , the number of parameters rose to 175 billion, and once more a drastic jump in capabilities was observed. Under this backdrop, the OpenGPT-X project was devised. During the project runtime, InstructGPT Ouyang et al. 2022 applied Reinforcement Learning from Human Feedback (RLHF) Ziegler et al. 2020 ; Stiennon et al. 2022 in order to turn the pre-trained text completion models into helpful assistants, aligned with human preferences. Based on InstructGPT, ChatGPT was published in 2022, making the impressive capabilities available to the general public in an intuitive user interface. Further models by OpenAI, such as GPT-4 followed and a number of commercial competitors started to offer similar systems, among them Google with Gemini, Athropic with Claude, and Groq. Due to commercial interests, these models, i.e. their pre-trained weights, are not publicly available (closed source). The level of detail in published specifications varies. For example, the number of parameters of GPT-4 is not disclosed, but multiple sources have stated GPT-4 to be based on a mixture-of-experts model employing 1.8 trillion parameters 2 2 2 https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/ , accessed 2025-03-31 . Further ChatGPT competitors arose in the open-source space. Meta published a series of Llama models Touvron et al. 2023 . Most recently, Deepseek DeepSeek-AI et al. 2024 published an open-weights model with 671 billion parameters on par with the capabilities of recent OpenAI models. 2.2. OpenGPT-X OpenGPT-X is a German collaborative initiative aimed at developing large-scale, multilingual language models tailored to business and research needs. It emphasizes transparency, trustworthiness, and open-source principles. Funded by the German Federal Ministry for Economic Affairs and Climate Action (BMWK), the project spanned from January 2022 to March 2025 and released Teuken-7B, a 7-billion-parameter model optimized for European languages, along with several research publicaitons Penke et al. 2022 ; Ostendorff & Rehm 2023 ; John et al. 2023 ; Ali et al. 2024a ; John & Herten 2023 ; Weber et al. 2024 ; Mirza et al. 2024 ; Courtois et al. 2024 ; Ostendorff et al. 2024 ; John et al. 2024 ; Ali et al. 2024 ; Thellmann et al. 2024 ; Brandizzi et al. 2024 ; Ebert et al. 2025 . The initiative is part of the broader Gaia-X ecosystem, a European project dedicated to creating a secure, transparent, and sovereign data infrastructure. Gaia-X fosters collaboration across various industries, supporting digital sovereignty and innovation by establishing common standards and trusted data spaces for sharing data securely and transparently across Europe. OpenGPT-X addresses multiple aspects necessary for developing and deploying advanced language technologies. It encompasses utilizing scalable GPU-based computing infrastructures, creating compatible data environments and MLOps workflows, developing innovative AI language services, and ensuring interoperability with established AI platforms. Additionally, it focuses on real-world prototyping and application testing across diverse domainsm thereby validating practical applicability for industry and research. The focus of this report is the establishment of a highly scalable GPU-based computing infrastructure, central to the OpenGPT-X project. By integrating the computational resources from key consortium partners, including Fraunhofer, IONOS, Forschungszentrum Jülich, and TU Dresden, the project has created an environment that supports large-scale training of AI language models. Furthermore, this infrastructure has served as a testing ground for evaluating and optimizing emerging GPU and accelerator technologies, reinforcing Europe’s technological sovereignty in AI. 2.3. Current Trends and Developments In this subsection, we outline and briefly describe significant trends in language model research and development that emerged during the OpenGPT-X project and are expected to continue influencing the field. (1) Number of parameters: While leading pre-trained flagship

Read the complete text