Streaming Tensor Program: A streaming abstraction for dynamic parallelism DOI: XXXXXXX.XXXXXXX ISBN: 978-X-XXXX-XXXX-X/XX/XX
Gina Sohn, Genghan Zhang, Konstantin Hossfeld, Jungwoo Kim, Nathan Sobotka, Nathan Zhang, Olivia Hsu, Kunle Olukotun · 2025
Read Streaming Tensor Program: A streaming abstraction for dynamic parallelism DOI: XXXXXXX.XXXXXXX ISBN: 978-X-XXXX-XXXX-X/XX/XX by Gina Sohn, Genghan Zhang, Konstantin Hossfeld, Jungwoo Kim, Nathan Sobotka, Nathan Zhang, Olivia Hsu, Kunle Olukotun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07776.txt.
Introduction
Gina Sohn, Genghan Zhang, and Konstantin Hossfeld give spatial dataflow accelerators a language for programs that change shape at run time. Machine learning models increasingly use data-dependent tensor sizes and control flow. Older abstractions for these chips force dynamic behavior into static, unoptimized molds.
Streaming Tensor Programs introduce flexible routing operators, an explicit memory hierarchy, and symbolic-shape semantics. Dynamic data rates and dimensions become visible to the compiler instead of hidden. Programs can then adapt tile sizes and flows as data demands. Efficient dynamic execution could let accelerators serve modern models without falling back to general processors for every irregular step.
Catalog record
| CREATOR | Gina Sohn, Genghan Zhang, Konstantin Hossfeld, Jungwoo Kim, Nathan Sobotka, Nathan Zhang, Olivia Hsu, Kunle Olukotun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07776
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Streaming Tensor Program: A streaming abstraction for dynamic parallelism DOI: XXXXXXX.XXXXXXX ISBN: 978-X-XXXX-XXXX-X/XX/XX Gina Sohn, Genghan Zhang, Konstantin Hossfeld, Jungwoo Kim, Nathan Sobotka, Nathan Zhang, Olivia Hsu, Kunle Olukotun Abstract . Dynamic behaviors are becoming prevalent in many tensor applications. In machine learning, for example, the input tensors are dynamically shaped or ragged, and data-dependent control flow is widely used in many models. However, the limited expressiveness of prior programming abstractions for spatial dataflow accelerators forces the dynamic behaviors to be implemented statically or lacks the visibility for performance-critical decisions. To address these challenges, we present the Streaming Tensor Program (STeP), a new streaming abstraction that enables dynamic tensor workloads to run efficiently on spatial dataflow accelerators. STeP introduces flexible routing operators, an explicit memory hierarchy, and symbolic shape semantics that expose dynamic data rates and tensor dimensions. These capabilities unlock new optimizations—dynamic tiling, dynamic parallelization, and configuration time-multiplexing—that adapt to dynamic behaviors while preserving dataflow efficiency. Using a cycle-approximate simulator on representative LLM layers with real-world traces, dynamic tiling reduces on-chip memory requirement by 2.18×, dynamic parallelization improves latency by 1.5×, and configuration time-multiplexing improves compute utilization by 2.57× over implementations available in prior abstractions. 1. Introduction Abstraction Data Flow Explicit Data Rate Explicit Memory Hierarchy Dynamic Routing & Merging Dynamic Tiling Spatial ( 14 ) ✗ ✗ ✓ ✗ ✗ Revet ( 26 ) ✗ ✗ ✓ ✓ (limited) ✗ StreamIt ( 33 ) ✓ ✓ ✗ ✗ ✗ SAM ( 12 ) ✓ ✗ ✗ ✓ (limited) ✓ (limited) Ripple ( 8 ) ✓ ✗ ✗ ✓ ✓ STeP ✓ ✓ ✓ ✓ ✓ Table 1. Landscape of programming abstractions for SDAs Spatial dataflow accelerators (SDAs) ( 23 ; 36 ; 22 ; 24 ; 9 ; 15 ; 37 ) are reconfigurable architectures with spatially distributed compute and memory units. Due to their high performance and energy efficiency, SDAs have gained popularity as an alternative to general-purpose processors and GPUs across several applications, mainly focusing on tensor applications ( 24 ; 22 ; 26 ; 9 ; 15 ) . Prior work on the compiler and the architecture of SDAs demonstrates that several static tensor applications can be effectively mapped to SDAs ( 41 ; 14 ; 9 ; 23 ) . However, the need to support dynamic tensor applications is rapidly growing. As an example, the input data has dynamic or ragged shapes in machine learning (ML) applications due to varying batch sizes, image resolutions, and prompt lengths ( 21 ; 28 ; 38 ; 20 ) . This makes efficient parallelization and spatial mapping challenging. Moreover, dynamic control flow is also heavily used in applications with the advent of dataflow applications that sparsely activate parts of the computation graph ( 4 ; 18 ; 39 ) . Many such dynamic workloads can be characterized as asynchronously running blocks communicating with each other. This characteristic aligns well with the execution model of SDAs, where compute and memory units run asynchronously and communicate via hardware FIFOs. However, unlike the ample software support for expressing static workloads on SDAs, current programming abstractions for SDAs have limited support for accelerating dynamic workloads, as shown in Table 1 , leaving performance on the table. Many prior SDAs use an imperative language ( 14 ; 41 ; 26 ; 9 ) as the programming abstraction. While this approach is general, it requires transforming the imperative code to a dataflow representation to match the hardware’s execution model. This makes it challenging to exploit the inherent parallelism in the application, as imperative code enforces a sequential order between instructions ( 41 ) . Furthermore, imperative languages lack explicit primitives to express asynchronous execution or queueing, which are crucial for optimizing dynamic workloads. The lack of explicit support for asynchrony and queueing forces their implementation with complex control and memory instructions, even though they can be directly mapped to SDA hardware FIFOs with minimal overhead ( 8 ) . On the other hand, dataflow abstractions, such as SAM ( 12 ) Ripple ( 8 ) , and StreamIt ( 33 ) , expose queues at the abstraction level and enable exploiting the inherent parallelism in the application. However, none of them model an explicit memory hierarchy, and many were designed for a specific domain, limiting their ability to capture the broader range of dynamic tensor workloads. SAM is limited to sparse tensor algebra kernels, and StreamIt adopts a synchronous dataflow model, making it challenging to express dynamic behaviors. Ripple adopts a design of asynchronous blocks that can contain any imperative code. However, Ripple makes the memory hierarchy implicit in the abstraction. This makes it difficult to express and discover efficient implementations of many widely used dynamic tensor applications, such as ML, where the performance is dictated by the data movement across the memory hierarchy. Furthermore, the opaque data rate at the abstraction level requires lifting the imperative code in each asynchronous block to analyze the program using the data rates. To address the limitations of prior SDA dataflow abstractions in expressing and optimizing dynamism, we propose the Streaming Tensor Program (STeP), a new streaming abstraction for accelerating dynamic tensor applications on SDAs. STeP expresses data as streams, where tiles and buffers in the stream can have dynamic shapes. It consists of asynchronous dataflow blocks that provide three key properties: explicit memory hierarchy, symbolic data consumption and production rate, and data-dependent control flow operators. These properties give STeP unique capabilities that were unavailable in prior abstractions for SDAs. First, it enables capturing performance-critical metrics such as off-chip traffic, on-chip memory requirement, and operational intensity at the abstraction level. We show how STeP provides insight into memory-intensive tensor applications and validate the captured metrics with a cycle-accurate simulator ( Section 4 ). STeP also enables expressing optimizations such as dynamic parallelization, configuration time-multiplexing, and dynamic tiling ( Section 5 ), which were not expressible in prior abstractions for SDAs. We evaluate each optimization on representative layers from open-source large language models (LLMs) with real-world traces using a cycle-approximate simulator. Our evaluations show a geometric mean of 2.1\times less on-chip memory requirement with dynamic tiling, 2.57\times improved compute utilization with configuration time-multiplexing, and 1.5\times speedup with dynamic parallelization when compared to the implementations available in prior abstractions. Lastly, we discuss various ways and trade-offs for supporting dynamic features of STeP on SDAs ( Section 6 ). Overall, our contributions are: • An asynchronous dataflow abstraction for SDAs (STeP) with full support for dynamism ( Section 3 ). • A symbolic system based on STeP’s shape semantics to extract performance-critical metrics ( Section 4 ). • Optimizations that exploit the dynamic features and explicit memory hierarchy of STeP ( Section 5 ) and an outline of how those abstract dynamic features would be supported in SDA hardware ( Section 6 ). • A performance and resource utilization investigation on the impact of dynamic optimizations enabled by STeP on representative LLM applications ( Section 5 ). 2. Background and Related Work This section provides background on the application, hardware, and related programming abstractions discussed in this paper. First, we explain representative dynamic behaviors in tensor applications, using ML workloads as an example. We then explain the architecture and