ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing Conference: Proceedings of the 48th International ACM ICMR Conference on Research and Development in Information Retrieval; June 30–July 3, 2025; Chicago, IL, USA. Proceedings of the 48th International ACM ICMR Conference on Rese
Yisu Wang, Yixiang Zhu, Xinjiao Li, Yulong Zhang, Ruilong Wu, Dirk Kutscher · 2025
Read ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing Conference: Proceedings of the 48th International ACM ICMR Conference on Research and Development in Information Retrieval; June 30–July 3, 2025; Chicago, IL, USA. Proceedings of the 48th International ACM ICMR Conference on Rese by Yisu Wang, Yixiang Zhu, Xinjiao Li, Yulong Zhang, Ruilong Wu, Dirk Kutscher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16258.txt.
Introduction
Yisu Wang and five co-authors present a system that speeds video feature indexing inside datacenters in this 2025 paper. Indexing huge video libraries depends on moving feature data efficiently. Doing computation inside the network can help, but varied hardware and tangled paths make it hard.
Their framework, ViFusion, merges many small feature tensors into bigger units as they travel the network. A fusion module at switches and hosts streamlines the flow for indexing jobs. In deployment, the system raised retrieval throughput eight to twenty-two fold at the same latency as leading systems. The authors credit the gains to communication-aware merging rather than faster models. They aim the design at large multimedia datacenters.
Catalog record
| CREATOR | Yisu Wang, Yixiang Zhu, Xinjiao Li, Yulong Zhang, Ruilong Wu, Dirk Kutscher |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-16258
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing Conference: Proceedings of the 48th International ACM ICMR Conference on Research and Development in Information Retrieval; June 30–July 3, 2025; Chicago, IL, USA. Proceedings of the 48th International ACM ICMR Conference on Research and Development in Information Retrieval (ICMR ’25), June 30–July 3, 2025, Chicago, IL, USA ISBN: 979-8-4007-1592-1/25/07 DOI: 10.1145/3731715.3733461 CCS: Computing methodologies CCS: Distributed computing methodologies CCS: Distributed algorithms Yisu Wang, Yixiang Zhu, Xinjiao Li, Yulong Zhang, Ruilong Wu, Dirk Kutscher Abstract . Large-scale video feature indexing in datacenters is critically dependent on efficient data transfer. Although in-network computation has emerged as a compelling strategy for accelerating feature extraction and reducing overhead in distributed multimedia systems, harnessing advanced networking resources at both the switch and host levels remains a formidable challenge. These difficulties are compounded by heterogeneous hardware, diverse application requirements, and complex multipath topologies. Existing methods focus primarily on optimizing inference for large neural network models using specialized collective communication libraries, which often face performance degradation in network congestion scenarios. To overcome these limitations, we present ViFusion, a communication aware tensor fusion framework that streamlines distributed video indexing by merging numerous small feature tensors into consolidated and more manageable units. By integrating an in-network computation module and a dedicated tensor fusion mechanism within datacenter environments, ViFusion substantially improves the efficiency of video feature indexing workflows. The deployment results show that ViFusion improves the throughput of the video retrieval system by 8–22× with the same level of latency as state-of-the-art systems. Figure 1 . ViFusion Processing Pipeline. Multimodal video features are processed through a combination of tensor fusion, LLM inference, and in-network optimization. 1. Introduction The rapid expansion of video-centric digital services has placed tremendous pressure on large-scale feature indexing systems to efficiently process increasingly complex queries. Modern video understanding models, from temporal action detection frameworks ( He et al. 2022 ) to multimodal architectures ( Wadekar et al. 2024 ) , require high computational throughput and extensive memory bandwidth ( Arnab et al. 2021 ; Peebles and Xie 2023 ) . Despite significant advances in GPU acceleration, single-node deployment remains impractical for models with billions of parameters ( OpenAI 2024b ; Song and Ermon 2020 ) , which necessitates distributed execution across clusters. Compared with traditional document or image retrieval, video retrieval presents unique and significant challenges. First, there is an inherent semantic gap between video and text. Text is highly abstract symbolic information, whereas video comprises sequences of visual frames rich in spatio-temporal signals. These differences in representation and semantic distribution make cross-modal alignment particularly difficult. Second, the massive amount of video data introduces substantial computational overhead and communication costs. Retrieving a specific segment from a long video may involve decoding and analyzing tens of thousands of frames. When processing such data in a distributed setting, additional overhead arises from data transmission and synchronization across nodes, which can significantly impact system throughput and response latency. Thus, efficient handling of long videos while achieving precise cross-modal alignment and temporal grounding is a core challenge in video retrieval. To address these issues, researchers are exploring large-scale pre-trained models to improve video retrieval ( Peebles and Xie 2023 ; Arnab et al. 2021 ) . Traditional methods relying on manual tags or rules cannot fully capture video semantics or user intent. In contrast, multimodal pre-trained models and LLMs can bridge the video-text gap by leveraging representations learned from massive data. For example, CLIP ( Radford et al. 2021 ) aligns images and text via contrastive learning, while GPT-4 ( OpenAI 2024a ) and DeepSeek-R1 ( DeepSeek-AI 2025 ) demonstrate strong language understanding and reasoning. Integrating these models into video indexing goes beyond simple keyword matching and more accurately captures the deeper semantics of user queries. However, deploying large models directly in video-retrieval services is challenging. The state-of-the-art LLMs are extremely large: DeepSeek-R1 ( DeepSeek-AI 2025 ) , for example, contains 671 billion parameters, requiring the model to be partitioned between multiple GPUs during both training and inference. Similarly, large multimodal models used for video understanding are often too large to fit or be inferred efficiently on a single GPU ( Guo et al. 2024 ; He et al. 2024 ; Rajbhandari et al. 2022 ) . Moreover, multi-machine cooperative inference introduces imbalances in computing and communication resources. Differences in computational speed and network bandwidth among nodes can lead to additional synchronization overhead, and the cost of communication may offset the benefits of parallelism, resulting in limited throughput gains or even degraded performance with increased system latency ( Liu et al. 2024 ) . In this paper, we propose ViFusion, a system that addresses these challenges by combining batch-based inference with in-network computation. We fuse video feature queries into a single large tensor and offload partial computations to in-network computation nodes. Figure 1 illustrates the architecture of our proposed system, which integrates tensor fusion and in-network optimization to enhance video retrieval performance. Multiple text-based inputs are received via the network and aggregated through the tensor fusion module. The fused representation is processed by the LLM backbone and enriched with relevant video features from the video feature database. To enhance efficiency, the system employs in-network AllReduce for optimized communication. Finally, the processed data is output as videos, providing multimodal retrieval results. By dynamically batching incoming requests and fusing them into unified tensors, ViFusion maximizes the throughput of deep learning accelerators, reduces per-query memory overhead, and amortizes inference costs across multiple queries. Even small requests are processed as part of a larger fused batch, significantly improving GPU utilization and overall system throughput. Beyond batching optimization, in-network aggregation can accelerate distributed feature indexing in ViFusion. Instead of performing all tensor aggregation on end-host GPUs, part of the reduction process is offloaded to middle host computation node. This in-transit aggregation minimizes communication overhead, alleviates network congestion, and improves scalability, while maintaining correct results within network hardware constraints. In addition, ViFusion seamlessly integrates with popular deep learning frameworks, such as PyTorch ( Paszke et al. 2019 ) . The contributions of this study are as follows. • A communication-aware tensor fusion framework that jointly optimizes batch sizes, network utilization, and GPU occupancy for video indexing workloads. • An in-network computation strategy that offloads and optimizes AllReduce operations to boost overall performance. • Comprehensive experimental validation demonstrating that, compared to existing methods, our approach achieves significant improvements in both retrieval accuracy and system efficiency, validating the effectiveness and practical value of the proposed system. 2. Related Work and Motivation 2.1. LLM for Video Search Integration Researchers are currently exploring the application of LLMs in video retrieval to enhance the semantic capabilities of video indexing and understanding. One approach is to leverage multimodal models to connect textual queries with video content. For example, OpenAI’s CLIP model ( Radford et al. 2021 ) , which was originally designed for image-text tasks, has been extended to video retrieval. A representative method, CLIP4Clip ( Luo et al. 2021 ) , utilizes CLIP’s visual-semantic representations to map video frame features into a shared embedding space with text, enabling text-to-video retrieval. By transferring CLIP’s knowledge to video-language tasks, CLIP4Clip achieved state-of-the-art performance on multiple video retrieval benchmarks. Beyond embedding-based matching, another approach is