Rethinking cloud abstractions for tenant-provider cooperative optimization of AI workloads
Marco Canini, Ricardo Bianchini, Íñigo Goiri, Dejan Kostić, Peter Pietzuch · 2025
Read Rethinking cloud abstractions for tenant-provider cooperative optimization of AI workloads by Marco Canini, Ricardo Bianchini, Íñigo Goiri, Dejan Kostić, Peter Pietzuch (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09562.txt.
Introduction
Marco Canini and co-authors set out a 2025 vision for how cloud systems should support artificial intelligence work. Large AI jobs run on shared cloud hardware and need vast computing power. Yet tenants know little about the hardware, and providers know little about the jobs, so both sides lose chances to save time and energy.
Their proposal, called HarmonAIze, would let tenants and providers share the right details and tune jobs together. Gains could come in speed, cost, fault handling, and lower energy use. The authors map the main chances and the hard problems that such a change would face. The piece is a position paper, not a test of a built system.
Catalog record
| CREATOR | Marco Canini, Ricardo Bianchini, Íñigo Goiri, Dejan Kostić, Peter Pietzuch |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking cloud abstractions for tenant-provider cooperative optimization of AI workloads Marco Canini, Ricardo Bianchini, Íñigo Goiri, Dejan Kostić, Peter Pietzuch Abstract . AI workloads, often hosted in multi-tenant cloud environments, require vast computational resources but suffer inefficiencies due to limited tenant-provider coordination. Tenants lack infrastructure insights, while providers lack workload details to optimize tasks like partitioning, scheduling, and fault tolerance. We propose the HarmonAIze project to redefine cloud abstractions, enabling cooperative optimization for improved performance, efficiency, resiliency, and sustainability. This paper outlines key opportunities, challenges, and a research agenda to realize this vision. 1. Introduction Modern AI workloads are widely compute-intensive, driven by the growing adoption of generative AI (GenAI), agentic applications, and large language models (LLMs). These workloads require large clusters of specialized hardware accelerators, such as GPUs and TPUs, to efficiently train, fine-tune or serve models that contain billions or even trillions of parameters. Training and serving state-of-the-art AI models consumes substantial amounts of power, contributing to carbon emissions and raising sustainability challenges ( Wu et al. 2024a ; Wu et al. 2022 ) . Optimizing the efficiency of AI workloads is essential to mitigate their environmental impact. As these workloads scale to meet growing demands, their computational requirements outpace traditional hardware capabilities, necessitating not only energy-efficient algorithms ( Chung et al. 2024 ; You et al. 2023 ) but also innovative infrastructure designs that maximize resource utilization. To maximize returns on costly hardware accelerators (e.g., GPUs costing tens of thousands of dollars), AI workloads are increasingly hosted in multi-tenant cloud environments. These environments enable resource sharing, offering tenants scalability, flexibility, and cost-efficiency while minimizing idle hardware and maximizing infrastructure utilization. However, tenants operating in these environments face a fundamental challenge: they lack visibility into the infrastructure details (such as network topology, job placement, resource availability, etc.) that could be used to optimize their workloads. Providers often withhold such information to maintain commercial competitiveness or to preserve the flexibility to make changes to their infrastructure. This lack of awareness creates a barrier to fine-grained optimizations that could benefit both tenants and providers. Providers are ideally placed to optimize infrastructure-level tasks critical to AI workloads, such as collectives, job scheduling and scaling, and fault tolerance ( Wu et al. 2024b ; He et al. 2024 ; Jiang et al. 2024 ; Duan et al. 2024 ) . Yet, without insights into tenant workloads – such as communication patterns ( Zheng et al. 2022 ; Li et al. 2022 ) or hyperparameter adaptations ( Mai et al. 2020 ; Qiao et al. 2021 ) – providers cannot optimize these tasks effectively. The root cause of these inefficiencies lies in the lack of abstractions enabling cooperative optimization between tenants and providers ( Section 2 ), a challenge shared across AI and other cloud applications ( Huang et al. 2024 ; Bilal et al. 2023 ; Chen et al. 2022 ) . Such abstractions would need to provide provide semantic information necessary for cross-layer control, allowing tenants and providers to jointly optimize performance, efficiency, and resilience, achieving outcomes that neither could accomplish alone. To address this gap, we propose widening traditional cloud abstractions to enable tenant-provider collaboration. Specifically, these abstractions expand cloud interface boundaries to provide tenants with additional information typically unavailable to them and enable providers to optimize workloads more effectively. Fig. 1 outlines the architecture of our proposal, HarmonAIze , and illustrates interactions between tenants and providers to enable cross-layer optimization opportunities ( Section 3 ) via micro- and macro- level abstractions ( Section 4 ). Compared to the status quo, HarmonAIze introduces two key innovations: (1) tenants’ control loops are informed by infrastructure-level updates, and (2) providers’ control loops are guided by workload-specific requirements, integrating optimizations that tenants could only achieve with perfect infrastructure visibility in an ideal scenario. Recent work has introduced ad hoc approaches for optimizing AI workloads in specific cases ( Wu et al. 2024b ; Mai et al. 2020 ; Duan et al. 2024 ) . In contrast, HarmonAIze advocates a general framework for cooperative optimization. Workload Intelligence ( Huang et al. 2024 ) proposes dynamic, bi-directional communication for a generic cloud cooperation framework. While HarmonAIze shares some goals with WI, it departs from preserving traditional VM abstractions and VM-oriented mechanisms. Instead, it elevates its abstractions closer to the workload and refactors functionality to leverage unique workloads characteristics, such as their accelerator-driven, iterative nature, performance predictability, proximity to the network, and tight synchronization. This paper aims to spark discussion in the systems research community on rethinking cloud abstractions tailored to AI workloads and their potential benefits. We highlight cross-layer optimization opportunities these abstractions could unlock. We also explore the challenges and trade-offs, such as standardization, privacy, and collaboration incentives, and propose a research agenda to bridge the tenant-provider gap, fostering efficient, sustainable, and scalable AI workloads in cloud environments. 2. AI workloads in the cloud Today, tenants have multiple options for running AI workloads on cloud providers, depending on their specific requirements, levels of expertise, and available resources. Broadly, tenants can choose from managed Machine Learning as a Service (MLaaS) platforms, use Infrastructure as a Service (IaaS) to set up customized environments, or opt for hybrid configurations that combine on-premises and cloud resources. Figure 1. Architecture of HarmonAIze and overview of interactions between tenant and provider to realize cross-layer optimization opportunities via micro- and MACRO- level cloud abstractions. Dashed call-outs indicate potential realization loci (e.g., at IPUs ( Intel 2023 ) ) of optimized control loops. MLaaS platforms such as Google Vertex AI ( Google Cloud 2024a ) and AWS SageMaker ( Amazon Web Services 2024 ) offer managed services that streamline large-scale ML pipelines. These platforms provide two primary options: (1) AutoML tools, which deliver a code-free experience but limit customization over model architecture and infrastructure, and (2) custom training interfaces that allow tenants to use ML frameworks of their choice (e.g., PyTorch or TensorFlow), run jobs, and utilize select optimizations like Google’s Reduction Server ( Google Cloud 2024b ) . However, despite these advantages, MLaaS platforms lack the cooperative mechanisms needed for deeper collaboration between tenants and providers, which we envision as essential for effective workload optimization. Additionally, proprietary APIs reinforce vendor lock-in, restricting flexibility and making it costly for tenants to migrate to other platforms. For greater control and flexibility, tenants may opt for IaaS solutions, provisioning cloud infrastructure to create tailored environments for AI workloads. Using IaaS, tenants can deploy their own ML pipelines, libraries and containers on virtual machines, attach high-performance GPUs or TPUs as needed, and orchestrate resources with Kubernetes clusters. This approach is suitable for users who need specific configurations, such as optimized distributed ML frameworks or customized distributed execution strategies. However, IaaS solutions hinder cooperative optimization as current deployments do not expose sufficient semantic information about AI workloads to enable cross-layer optimization. Thus, tenants must manually handle resource configuration, scheduling, and optimize ML pipelines — tasks that could be optimized more effectively with cross-layer cooperation. 3. Cross-layer opportunities To make our discussion concrete, we illustrate several cross-layer optimization opportunities that could benefit from the proposed abstractions. These opportunities span various aspects of AI workload management, including communication, scheduling, fault tolerance, and resource scaling. O1-Runtime adaptation: In cloud