Beyond Pre-Training: The Full Lifecycle of Foundation Models on HPC Systems Conference: Cray User Group; April 26-30, 2026; Nice, France

Dino Conciatore, Elia Oggian, Federico Da Forno, Stefano Schuppli, Jerome Tissieres, Joost VandeVondele, Maxime Martinasso · 2026

Read Beyond Pre-Training: The Full Lifecycle of Foundation Models on HPC Systems Conference: Cray User Group; April 26-30, 2026; Nice, France by Dino Conciatore, Elia Oggian, Federico Da Forno, Stefano Schuppli, Jerome Tissieres, Joost VandeVondele, Maxime Martinasso (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12599.txt.

Introduction

Dino Conciatore and six colleagues map the full AI lifecycle onto national supercomputing. Pre-training put supercomputers at the heart of sovereign AI efforts. Fine-tuning and inference follow different operational patterns. They need heavy compute but use hardware unlike pre-training. The team presents a hybrid cloud-native platform at the Swiss National Supercomputing Centre. Diskless GPU nodes join virtualized commodity infrastructure under Kubernetes. Early work covers tuning pipelines and always-on inference services. The design offers a blueprint for AI Factory services on supercomputers.

Catalog record

CREATORDino Conciatore, Elia Oggian, Federico Da Forno, Stefano Schuppli, Jerome Tissieres, Joost VandeVondele, Maxime Martinasso
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78874
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12599

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Pre-Training: The Full Lifecycle of Foundation Models on HPC Systems Conference: Cray User Group; April 26-30, 2026; Nice, France Dino Conciatore, Elia Oggian, Federico Da Forno, Stefano Schuppli, Jerome Tissieres, Joost VandeVondele, Maxime Martinasso Abstract . Large-scale pre-training of Foundational Models (FM) constitutes a computationally intensive first phase for enabling AI across diverse scientific and societal applications. This first phase has positioned High-Performance Computing (HPC) facilities as indispensable backbones of “Sovereign AI” initiatives. While the massive throughput requirements of FM pre-training align with the traditional capability-oriented mission of HPC, subsequent phases of the AI lifecycle, typically referred to as fine-tuning and inference, introduce operational paradigms that can conflict with established batch-processing environments. Moreover, these phases are not computationally trivial: they often require substantial high-end compute resources while exhibiting hardware utilization patterns that differ significantly from those of pre-training. This paper addresses the architectural and strategic challenges of operationalizing a complete AI lifecycle within a national supercomputing facility. We present a hybrid cloud-native platform being developed and deployed at the Swiss National Supercomputing Centre (CSCS) that combines diskless GPU-enabled HPE Cray EX compute nodes with virtualized commodity infrastructure. Orchestrated by Kubernetes, this novel service architecture bridges the gap between HPC batch processing and service-oriented workflows. We report our initial investigations into fine-tuning pipelines and highly available inference services, analyzing the associated trade-offs while improving user productivity. Our findings offer a blueprint for enabling supercomputers to integrate “AI Factories” services and workflows, supporting AI innovations into end-to-end scientific and industrial use cases. 1. Introduction The proliferation of Large Language Models (LLMs) represents the most visible facet of a broader adoption of Machine Learning (ML) Foundation Models (FMs) across society and science. Beyond natural language, data-driven approaches are increasingly augmenting traditional methodologies across a wide range of domains. For example, in weather and climate science ( 10 ) , they provide a complementary paradigm to conventional numerical simulations. This technological adoption has prompted governments to pursue sovereign AI capabilities, aiming not only at regulatory compliance and cultural representation, but also at strategic independence of nation-critical computational and informational assets. Large models development such as the Apertus model ( 1 ) inherently requires state-of-the-art HPC resources because of the computational intensity and data scale involved, especially in their early stages. Consequently, for sovereign and non-commercial initiatives, it is natural to approach national facilities such as the Alps Research Infrastructure (RI) operated by the Swiss National Supercomputing Centre (CSCS) in Lugano, Switzerland. These represent the natural ecosystem for such undertakings, providing both the capital-intensive hardware, primarily represented by massive GPU partitions, high-end interconnects, and suitable storage systems, as well as the world-class engineering talent required to execute such specialized programs. Although Foundation Models pre-training aligns with the capabilities-oriented mission of HPC facilities (i.e., specialized infrastructures dedicated to enable otherwise intractable computational problems), pre-training alone is insufficient for developing AI sovereignty. Realizing the value of these models requires, among other things, post-training adaptation (fine-tuning) and deployment (inference). However, these downstream tasks introduce a conflict: while fine-tuning may still demand significant infrastructure resources, inference often entails high-capacity workloads of independent instances that do not strictly require a supercomputer architecture. This creates tension between the typical capabilities-oriented mandate and the heterogeneous requirements of the typical AI project lifecycle, raising a strategic dilemma: should HPC facilities broaden their operations to support these workflows, or is this a substantial deviation from their core mission? The necessity to resolve this dilemma is heightened by the emerging European framework of AI Factories (AIFs) and their associated Antennas ( 6 ) . These initiatives aim to democratize access to supercomputing, extending the infrastructure’s reach beyond traditional academic research to include key industrial sectors where AI can have transformative impact, such as Life Sciences & Healthcare, Weather Prediction, as well as enabling SMEs and startups. To fulfill this vision, HPC facilities have to support the research and development activities of these diverse stakeholders in a secure and cost-effective way with the least amount of friction possible. Accommodating this broader community introduces distinct architectural challenges. Regarding fine-tuning, while current workflows are Slurm-centric, we anticipate a shift toward demand for “one-click” solutions, particularly from new user communities (including, SMEs involved in R&D) who may lack either the capabilities or the time to deal with the intrinsic complexity of HPC systems, and will therefore require higher levels of abstraction. However, implementing such level of abstraction is non-trivial; fine-tuning involves multiple stages (e.g., SFT, RL) and techniques (e.g., LoRA) that require proper calibration to prevent damaging the underlying model capabilities ( 21 ) . Regarding inference, the need for long-running, highly available services conflicts with the standard batch-processing model. Emulating such features via job schedulers is operationally brittle, relying on, e.g., resubmission workarounds. Furthermore, long-running inference workloads create a distinct optimization challenge: preventing high-end HPC nodes from idling during low-traffic periods. Crucially, these integration hurdles are not exclusive to LLMs, but apply equally to other scientific domains where the software product ecosystem is often not yet as established. In response, CSCS is actively exploring architectural solutions to address these challenges. This effort is anchored in a co-design process involving several key representative groups, who have been granted early access to help define the necessary service models. Therefore, our work presented in this paper provides a technical and operational framework for bridging the gap between high-performance “capability” computing and cloud-native “service” environments in the context of AI services. In particular, our contributions are: • A hybrid architecture for an AI Factory that integrates diskless HPE Cray EX nodes and commodity hardware (VMs) into one Kubernetes-managed control plane. • A methodology for operationalizing the “Full AI Lifecycle”, extending HPC services from the batch model used for pre-training, to a set of services suitable for supporting fine-tuning and high-availability inference. • A set of initial performance results together with technical trade-off of running distributed training and high-throughput inference within a containerized, service-oriented ecosystem on Slingshot-interconnected hardware. This paper is thus organized as follows. After further expanding on the core motivations in Section 2 , and reviewing related work in Section 3 , we outline the ongoing efforts to provide end-to-end AI lifecycle support within our HPC context in Section 4 . By presenting our initial findings and the architectural trade-off encountered, our aim is to contribute to the solution on how HPC centres can empower a sovereign AI ecosystem, and ultimately define a blueprint of those solutions to share across the HPC community. We conclude by discussing results obtained so far in Section 5 , ongoing efforts and anticipated future work in Section 6 . 2. Motivations The strategic dilemma outlined in the introduction, whether HPC facilities should expand beyond their traditional capability-oriented mandate, relies on a historical dichotomy between “batch processing” and “interactive services.” However, as AI advances to become intrinsic to the scientific method, this dichotomy is increasingly difficult to sustain. The motivation to support the full AI lifecycle at CSCS is driven not merely by customer demand, but by the necessity to support modern scientific workflows and the operational efficiency of the infrastructure itself. A primary example of this shift is found in numerical weather prediction (NWP). For decades, CSCS has provided computational capability for weather forecasting

Read the complete text