Ultra Ethernet’s Design Principles and Architectural Innovations
Torsten Hoefler, Karen Schramm, Eric Spada, Keith Underwood, Cedell Alexander, Bob Alverson, Paul Bottorff, Adrian Caulfield, Mark Handley, Cathy Huang, Costin Raiciu, Abdul Kabbani, Eugene Opsasnick, Rong Pan, Adee Ran, Rip Sohan · 2025
Read Ultra Ethernet’s Design Principles and Architectural Innovations by Torsten Hoefler, Karen Schramm, Eric Spada, Keith Underwood, Cedell Alexander, Bob Alverson, Paul Bottorff, Adrian Caulfield, Mark Handley, Cathy Huang, Costin Raiciu, Abdul Kabbani, Eugene Opsasnick, Rong Pan, Adee Ran, Rip Sohan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08906.txt.
Introduction
Torsten Hoefler and co-authors wrote a paper from 2025 explaining the design of Ultra Ethernet, a new high-performance networking standard for artificial intelligence and supercomputing. Ethernet runs the world's ordinary networks, but the largest training clusters need faster, more reliable links than it has offered. The Ultra Ethernet 1.0 specification rebuilds the whole stack for that scale, and this paper, written by the standard's own authors, gives the reasoning.
Its centerpiece is a new transport protocol built to run largely in hardware. It promises reliable delivery at extreme scale without the stalls that plague older designs. The authors contrast their path with the last big standard in this space, set more than two decades ago. They lean on Ethernet's vast ecosystem and on huge gains in computing efficiency per bit moved. The paper frames the standard as the base for the next era of large machine clusters.
Catalog record
| CREATOR | Torsten Hoefler, Karen Schramm, Eric Spada, Keith Underwood, Cedell Alexander, Bob Alverson, Paul Bottorff, Adrian Caulfield, Mark Handley, Cathy Huang, Costin Raiciu, Abdul Kabbani, Eugene Opsasnick, Rong Pan, Adee Ran, Rip Sohan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08906
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ultra Ethernet’s Design Principles and Architectural Innovations Torsten Hoefler, Karen Schramm, Eric Spada, Keith Underwood, Cedell Alexander, Bob Alverson, Paul Bottorff, Adrian Caulfield, Mark Handley, Cathy Huang, Costin Raiciu, Abdul Kabbani, Eugene Opsasnick, Rong Pan, Adee Ran, Rip Sohan Abstract . The recently released Ultra Ethernet (UE) 1.0 specification defines a transformative High-Performance Ethernet standard for future Artificial Intelligence (AI) and High-Performance Computing (HPC) systems. This paper, written by the specification’s authors, provides a high-level overview of UE’s design, offering crucial motivations and scientific context to understand its innovations. While UE introduces advancements across the entire Ethernet stack, its standout contribution is the novel Ultra Ethernet Transport (UET), a potentially fully hardware-accelerated protocol engineered for reliable, fast, and efficient communication in extreme-scale systems. Unlike InfiniBand, the last major standardization effort in high-performance networking over two decades ago, UE leverages the expansive Ethernet ecosystem and the 1,000x gains in computational efficiency per moved bit to deliver a new era of high-performance networking. 1. Introduction Ultra Ethernet (UE) standardizes a new protocol to support high-performance Artificial Intelligence (AI) and High-Performance Computing (HPC) networking over Ethernet. This paper, written by UE’s authors, supplements the full specification by highlighting historical and innovative technical aspects of our nearly 2.5-year journey. It is designed to be approachable to a general audience and, thus, abstracts many details while using intuitive wording and explanations. The final authority for questions regarding UE is the full 562-page specification ( Ultra Ethernet Consortium 2025 ) . In 2022, as the world was rapidly moving into the new age of massive computation required to satisfy the demands of artificial intelligence systems, various datacenter providers recognized the limitations posed by InfiniBand and its sister protocol Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE). At the same time, Ethernet’s success as generic interconnect was hard to overstate with hundreds of millions of ports deployed per year and earning it’s inventors the Turing award in that year. RoCE version 2 embedded InfiniBand’s transport in a routable Ethernet (OSI Layer 3) to enable datacenter deployment nearly a decade earlier. RoCE adopted InfiniBand’s transport protocol largely unchanged, requiring lossless transport with strict in-order packet delivery. This lossless in-order packet delivery is guaranteed by converged Ethernet using priority flow control (PFC) as a main mechanism. Yet, PFC requires a separate traffic class with substantial headroom buffer and suffers from congestion spreading and head-of-line blocking. Furthermore, in-order delivery limits the choice of paths leading to potentially suboptimal performance. Those and further limitations of RoCE are summarized in ( Hoefler et al. 2023 ) . The original InfiniBand transport was designed more than a quarter century ago in an environment where architects had to exploit high bandwidth with limited computation ( InfiniBand Trade Association 2000 ) . After 25 years of exponential Moore’s Law scaling, the cost per transistor and, therefore, computation was reduced by more than 100,000 times while the usable bandwidth grew only by a factor of 100 from Single Data Rate (SDR) to eXtended Data Rate (XDR). Thus, not only did the accelerators at the endpoints get a significant performance boost, but also network architects have more than 1,000x more computation per moved bit. This prompted many to rethink the network stack in internal AI product lines ( Shalev et al. 2020 ; Goel et al. 2024 ; Jouppi et al. 2017 ) and HPC deployments ( De Sensi et al. 2020 ) . Some early on recognized that datacenter and HPC networks are bound to converge into a single technology ( Hoefler et al. 2022b ) and discussions among a handful of companies began to make this happen. The initial small group formed in the first quarter of 2022 including AMD, Broadcom, HPE, Intel, and Microsoft, who agreed to build an open standard for next generation Ethernet inspired by the various parallel internal developments and the opportunity to create a bigger market. Initially, the effort was called HiPER and later renamed to Ultra Ethernet (UE). By July 2022, a new consortium was formed, and we met face to face for the first time in September 2022. That first meeting would bring out a wide range of opinions. From “let’s standardize one of those RoCE variants” to “let’s build a new standard from the many ingredients we have”, the energy was high and the group was engaged. By January 2023, we had our path: we would combine capabilities designed for HPC and security and congestion management techniques designed for the datacenter to build something new: a highly scalable transport to run over plain old Ethernet. Shortly after, in July 2023, the Ultra Ethernet Consortium (UEC) was officially announced by AMD, Arista, Broadcom, Cisco, Eviden (Atos), HPE, Intel, Meta, and Microsoft. As an open Linux Foundation Joint Development Foundation (JDF) project, it quickly grew to more than 100 member companies than 1,500 participants at the end of 2024. UEC set out to define an open next-generation HPC and AI network compatible with existing Ethernet deployments and interoperable between vendors. The discussions focused on several foundational tenets: Massive scalability enables large deployments that are needed for future AI systems. UE is designed to support millions of network endpoints in flexible arrangements with a connectionless API. Initially, it focuses on supporting traditional fat tree deployments, while not preventing other optimized topologies such as HammingMesh ( Hoefler et al. 2022a ) , Dragonfly ( Kim et al. 2008 ) , or Slim Fly ( Besta and Hoefler 2014 ) , which the consortium did not test. High-Performance is achieved by efficient protocols designed for large-scale deployments. For example, UE’s connection-less API is supported by a mechanism to establish a peer-to-peer reliability context without additional latencies, that is, the initial packet arriving establishes a context, potentially in nanoseconds, even for massive out-of-order delivery. Furthermore, UE supports optional extensions such as packet trimming for fast packet loss detection and reaction. Compatibility with existing Ethernet datacenter deployments is achieved by imposing minimal requirements on the switch infrastructure to allow easy deployment and rolling extension of existing infrastructures. UE switches are only required to support Equal-Cost Multi-Pathing (ECMP) and basic Explicit Congestion Notification (ECN) marked at egress, but they may (optionally) support packet trimming to improve network performance. UE does not require changes in either the Physical PHY Layer (OSI Layer 1) or the Link Layer (OSI Layer 2) but defines several optional extensions to improve the performance of the Link Layer in new deployments to foster vendor differentiation. UE is fully Ethernet-compatible, which enables users to use existing tools for operational management, debugging, and deployment. Vendor differentiation is maximized within the limits of interoperability of the specification. This enables the existing Ethernet vendor ecosystem to drive quick innovation cycles and development to its full extent in an active and sizeable market. In many cases, such as packet load balancing or fast loss detection, the specification proposes a set of options to implement compatible protocols but does not mandate any. Vendors could use one of the proposed approaches or invent their own to innovate. This enables architects to design their system to support important optimization objectives. For example, a vendor could choose specific load balancing and loss detection schemes to support the most reliable