SWAN : World-Aware Adaptive Multimodal Networks for Runtime Variations

Jason Wu, Shir-Kang Scott Jin, Yuyang Yuan, Maggie Wigness, Lance M. Kaplan, Hang Qiu, Mani Srivastava · 2026

Read SWAN : World-Aware Adaptive Multimodal Networks for Runtime Variations by Jason Wu, Shir-Kang Scott Jin, Yuyang Yuan, Maggie Wigness, Lance M. Kaplan, Hang Qiu, Mani Srivastava (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26181.txt.

Introduction

Wu and six colleagues present SWAN, an adaptive multi-mode network built for runtime shifts in input quality, input load, and compute. A quality-aware control assigns power among inputs within a user-set maximum budget. A gating module scales layer use to sample load inside that budget. A token-dropping module masks spare bits before spotting. On multi-object 3D spotting for self-driving cars, it cuts floating-point operations by up to about half with only minimal accuracy loss.

Catalog record

CREATORJason Wu, Shir-Kang Scott Jin, Yuyang Yuan, Maggie Wigness, Lance M. Kaplan, Hang Qiu, Mani Srivastava
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39656
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26181

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SWAN : World-Aware Adaptive Multimodal Networks for Runtime Variations Jason Wu, Shir-Kang Scott Jin, Yuyang Yuan, Maggie Wigness, Lance M. Kaplan, Hang Qiu, Mani Srivastava Abstract Multimodal deep neural networks deployed in realistic environments must contend with runtime variations: changes in modality quality, overall input complexity, and available platform resources. Current networks struggle with such fluctuations – adaptive networks cannot adhere to a strict compute budget, controller-based networks neglect to consider input complexity, and statically provisioned networks fail at all the above. Consequently, they do not extract maximum utility from the expended computational resources. We present SWAN ( S ample and W orld- A ware Multimodal N etwork), the first adaptive multimodal network that accomplishes all three goals. SWAN employs a quality-aware controller to assign resources among modalities according to a variable user-specified maximum budget. Within this budget, an adaptive gating module further optimizes efficiency by scaling layer utilization according to sample complexity. For further gains, SWAN also employs a token dropping module that masks semantically irrelevant multimodal features before performing detections. We evaluate SWAN in the domain of autonomous driving with complex multi-object 3D detection, reducing FLOPs by up to 49% with minimal degradation. 1 Introduction Real world systems frequently contend with runtime variations as the world changes during system operation. A camera-based object detector may encounter varying environmental conditions that drastically alter the utility of its input modality. Similarly, a system deployed on a mobile robot will encounter diverse computational resource availability owing to thermal throttling or power limitations. Despite the large class of runtime variations, the most prevalent variations can be largely grouped into three categories – input modality quality of information (QoI) ( e.g . sensor failure, dynamic weather), sample complexity ( e.g . more or less sensing targets), and platform dynamics ( e.g . thermal throttling). A system’s ability to adjust to such variations determines its utility in realistic deployments. One particularly relevant scenario involves the broad space of autonomous vehicles (AVs), ranging from small mobile robots to road-certified vehicles. The majority of AVs will encounter different environmental lighting or weather, and must exhibit robustness to resulting variations in input QoI. The AV’s platform capabilities can also vary, where faster traveling speeds, low battery capacity, or thermal throttling can affect the amount of computation available for the current input sample. Finally, adapting computation based on sample complexity is also an important consideration. Since AVs operate on edge devices with limited resources, reducing computation on “easier” samples can translate to meaningful reductions in latency and energy consumption. The inability to adapt to any one of these variations can severely hamper the safety or utility of a given AV. Existing works often deal with each of these variations separately. First, deep multimodal neural networks have emerged as a promising method for combating variable QoI. By virtue of aggregating information across multimodal sensors, such networks mitigate environmental corruptions in the input data through redundancy. Thus, multimodal networks are the de facto standard in AVs [ 18 , 26 , 1 ] . Although they provide robustness against variable QoI, the additional complexity of these networks can complicate adaptation to other runtime variations. Next, to meet dynamic computational budgets, reconfigurable neural networks modulate network computation during inference time. Once-For-All [ 6 ] , LayerDrop [ 8 ] , and ADMN [ 25 ] train adaptive networks where modules or layers can selectively be omitted at inference time for budget flexibility. Finally, input-adaptive systems incorporate knowledge of the complexity of the input to flexibly increase or reduce the computation performed on each sample [ 21 , 12 , 13 ] . For instance, AdaViT [ 21 ] dynamically removes attention heads, tokens, and layers from the network according to the nature of the input. However, current literature falls short of proposing a system that jointly addresses these dynamics. The key difficulty lies in the compounding complexity that arises when these strategies are combined. To illustrate, once multimodality is introduced to combat dynamic QoI, adapting network utilization to input complexity requires consideration of shared information across modalities. Similarly, techniques that adapt network computation to a maximum budget are difficult to incorporate with those implementing per-sample complexity adaptation. We propose SWAN , an adaptive multimodal network that addresses all three runtime variations – changes in modality QoI, maximum compute, and sample complexity. Although this problem is universal across any domain deploying multimodal neural networks in real-world environments, we focus specifically on autonomous vehicles (AVs) due to the task complexity (multimodal, multi-object 3D detection), robust evaluation frameworks, and high quality datasets. First, we train a layer-wise adaptive multimodal network based on the existing CMT [ 26 ] AV detection framework to allow for flexible allocation of layers during inference-time. Next, we introduce a controller that determines the optimal allocation of layers across modalities for the current budget and relative modality QoI. The controller is trained through NeuralSort [ 10 ] gradient approximation, enabling an understanding of relative importance among layers which existing controller-based techniques [ 25 ] fail to model. Finally, SWAN also minimizes resource usage within the user-specified maximal budget, enabling additional reductions in compute and latency. We present two complementary innovations – a feature-aware SkipGate module that conditionally executes the next layer, and a token dropout mechanism that filters out redundant tokens before the detection head. Both methods are conditioned upon the input, enabling efficiency on simpler inputs while retaining flexibility on more complex samples. We evaluated SWAN on the popular nuScenes dataset [ 5 ] with modality QoI degradation simulated through the MultiCorrupt [ 2 ] pipeline. Across various maximum allowable budgets, SWAN outperforms related baselines by up to 11.2 NDS and 13.7 mAP while achieving up to a 49% reduction in FLOPs compared to a fully-provisioned network. SWAN ’s controller correctly prioritizes high-QoI modalities under compute constraints, while the SkipGate and token pruning modules significantly reduce network computation. Additionally, we benchmarked SWAN on an Nvidia Jetson Orin edge device and with real-world data to further demonstrate its practicality. In summary, our main novel contributions are as follows: 1. We introduce the first multimodal network that modulates resource allocation across modalities in accordance to modality QoI, sample complexity, and a user-defined maximum budget 2. SWAN ’s lightweight, QoI-aware controller selects an optimal layer configuration given a specified budget, while retaining end-to-end differentiability through the NeuralSort gradient estimation technique 3. We additionally integrate a feature-based layer dropping module ( SkipGate ) that is conditioned on the output of the controller, and filter out tokens prior to the detection head for further efficiency gains 4. We perform evaluations on synthetically corrupted and real-world data, while also running on edge hardware to verify efficiency gains 2 Related Work Multimodal AV Neural Networks. Owing to strict robustness and accuracy demands of autonomous driving, many AV perception stacks rely on multimodal deep neural networks [ 18 , 1 , 14 , 7 , 27 ] . For example, BEVFusion [ 18 ] combines information across LiDAR and camera in the BEV space, improving detection performance by over 19 NDS and 5 NDS relative to the best LiDAR and camera unimodal detectors, respectively. Additionally, these multimodal networks also showcase greater robustness to environmental noise when compared

Read the complete text