MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception

Wenzhuo Liu Wenshuo Wang Yicheng Qiao Qiannan Guo Jiayin Zhu Pengfei Li, Zilong Chen Huiming Yang Zhiwei Li Lening Wang Tiao Tan Huaping Liu · 2025

Read MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception by Wenzhuo Liu Wenshuo Wang Yicheng Qiao Qiannan Guo Jiayin Zhu Pengfei Li, Zilong Chen Huiming Yang Zhiwei Li Lening Wang Tiao Tan Huaping Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02264.txt.

Introduction

Wenzhuo Liu and co-authors build one system that reads the driver, the car, and the road at the same time, in work from 2025. Driver assistance tools need all three views, yet most systems learn each task in isolation.

Their framework, MMTL-UniAD, jointly recognizes driver behavior, driver emotion, vehicle behavior, and traffic context from multiple sensor streams. Two additions fight negative transfer, where tasks drag each other down: an attention network that picks out task-relevant features, and a dual-branch embedding that balances shared and task-specific learning. On a standard benchmark dataset the system beat prior methods on all four tasks, and the code is public.

Catalog record

CREATORWenzhuo Liu Wenshuo Wang Yicheng Qiao Qiannan Guo Jiayin Zhu Pengfei Li, Zilong Chen Huiming Yang Zhiwei Li Lening Wang Tiao Tan Huaping Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception Wenzhuo Liu Wenshuo Wang Yicheng Qiao Qiannan Guo Jiayin Zhu Pengfei Li, Zilong Chen Huiming Yang Zhiwei Li Lening Wang Tiao Tan Huaping Liu Abstract Advanced driver assistance systems require a comprehensive understanding of the driver’s mental/physical state and traffic context but existing works often neglect the potential benefits of joint learning between these tasks. This paper proposes MMTL-UniAD, a unified multi-modal multi-task learning framework that simultaneously recognizes driver behavior (e.g., looking around, talking), driver emotion (e.g., anxiety, happiness), vehicle behavior (e.g., parking, turning), and traffic context (e.g., traffic jam, traffic smooth). A key challenge is avoiding negative transfer between tasks, which can impair learning performance. To address this, we introduce two key components into the framework: one is the multi-axis region attention network to extract global context-sensitive features, and the other is the dual-branch multimodal embedding to learn multimodal embeddings from both task-shared and task-specific features. The former uses a multi-attention mechanism to extract task-relevant features, mitigating negative transfer caused by task-unrelated features. The latter employs a dual-branch structure to adaptively adjust task-shared and task-specific parameters, enhancing cross-task knowledge transfer while reducing task conflicts. We assess MMTL-UniAD on the AIDE dataset, using a series of ablation studies, and show that it outperforms state-of-the-art methods across all four tasks. The code is available on https://github.com/Wenzhuo-Liu/MMTL-UniAD . 1 Introduction Over the past decade, Advanced Driver Assistance Systems (ADAS) — such as automatic emergency braking, lane keeping assist, and surround view monitoring — have significantly enhanced driving safety through the advances in monitoring driver states and surrounding traffic wang2023openoccupancy ; zhang2023oblique ; li2024mipdmultisensoryinteractiveperception ; chen2024fw . Despite these advancements, approximately 1.35 million fatalities occur annually in traffic accidents world2019global , with human drivers’ abnormal mental or physical states contributing to over 65% of these incidents tian2013studying . Consequently, accurate identification of driver states is essential for ADAS martin2019drive ; li2021spontaneous but remains a complex challenge due to the intricate causal relationships between driver states and traffic context yang2023aide (see Fig. 1 ). For instance, traffic congestion can induce driver anxiety, effecting driving behavior yang2023aide . Figure 1 : Traffic context and driver states interaction diagram. Tasks (a), (b), (c), and (d) represent traffic context recognition, vehicle behavior recognition, driver behavior recognition, and driver emotion recognition, respectively. These tasks comprehensively demonstrate the complex and closely interconnected relationships between the driver and traffic. Most existing approaches to driver state and traffic context recognition focus on isolated tasks, such as driver behavior or emotion recognition saleh2017driving ; gong2023sifdrivenet ; li2021cogemonet ; zepf2020driver ; yang2023robust ; mou2023driver , traffic environment recognition guo2023temporal ; zhi2021bigru ; martin2018dynamics ; wang2023path . However, they fail to exploit the interaction between these tasks, which can limit the potential for cross-task learning qian2019dlt . In reality, these tasks are not independent when driving in real-world traffic yang2023aide ; martin2019drive . For example, lane-change behavior relies on the interaction between the current traffic context (e.g., congestion) and the driver’s emotional state guo2023temporal ; xing2021multi . Multi-task learning (MTL) offers an opportunity to enhance task recognition accuracy by enabling information sharing between related tasks chowdhuri2019multinet ; ishihara2021multi . However, current research is often hindered by negative transfer, a phenomenon where performance degrades due to information sharing between weakly related tasks. Many existing works focus on tasks with fewer differences. For instance, studies often combine similar tasks like lane detection, depth estimation, and drivable area segmentation wu2022yolop ; qian2019dlt ; teichmann2018multinet to improve traffic context understanding, while others focus on joint learning for driver-related tasks such as behavior, emotion, intention recognition xing2021multi ; kim2020multi ; xun2020multitask . However, neglecting the interaction between driver-related tasks and traffic context recognition limits the integration of driver state and environmental information, thus restricting ADAS’s ability to fully understand driving context yang2023aide ; martin2019drive . Another significant limitation of current MTL models is their reliance on single-modal inputs. Achieving satisfactory recognition performance typically requires the complementarity of multimodal data li2021cogemonet ; du2020convolution ; guo2023temporal ; liu2024glmdrivenet ; liu2024fmdnet ; cui2024textnerf ; gong2022multi . For instance, traffic context recognition often relies on multi-view driving scene images, while driver behavior and state recognition necessitate comprehensive in-vehicle images, fine-grained driver images, and joint data for driver posture and gesture. Although leveraging these multimodal inputs can enhance the accuracy rong2020driver ; guo2023temporal , most existing MTL models rely on a single input type (e.g., driving scene images or driver images) choi2024multi ; qian2019dlt ; teichmann2018multinet , severely limiting their practical application in ADAS. To address these challenges, we propose a Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception (MMTL-UniAD). Our framework leverages multimodal data to simultaneously recognize driver behavior, emotion, traffic context, and vehicle behavior. First, inspired by wang2020axial ; zhu2023biformer , we designed a multi-axis region attention network for multi-view images from the driving environment and the human driver. This network captures global context through horizontal-vertical attention and then uses region attention to extract interest-triggering features, selecting task-related high-level semantic information. This approach mitigates negative transfer between tasks. Additionally, we introduce a dual-branch multimodal embedding that uses soft parameter sharing ma2018modeling to adaptively adjust task-shared and task-specific parameters. This design enhances task-specific learning while promoting information sharing and positive transfer between tasks. We validate our approach on the publicly available AIDE dataset, demonstrating superior performance over state-of-the-art methods. Our contributions are: \bullet We propose MMTL-UniAD, a novel framework that addresses the challenges of multimodal multi-task learning (MMTL) in assistive driving. \bullet We designed a multi-axis region attention network to effectively extract features from multi-view images of the driving environment and human drivers. \bullet We introduce a dual-branch multimodal embedding to extract task-shared and task-specific features, mitigating negative transfer caused by task differences while enhancing cross-task knowledge learning. 2 Related Work 2.1 Multi-task Learning Recent advancements in deep neural networks have made it feasible to learn multiple tasks jointly, improving the performance of individual tasks through shared learning. A key strategy for achieving multi-task learning is the sharing of parameters across tasks. Two primary approaches for parameter sharing are hard parameter sharing and soft parameter sharing. • Hard parameter sharing limits the sharing of parameters to the output layer of the neural network li2020knowledge ; xu2018pad ; cao2023relational ; al2022zero ; ghiasi2021multi ; gan2024segmentation ; huang2024mfe . This method effectively reduces overfitting between tasks and improve learning efficiency. However, it lacks flexibility, as it heavily depends on the correlations between tasks. If the task has significantly different objectives or conflicts, the shared parameters may harm the performance of individual tasks misra2016cross . • Soft parameter sharing, on the other hand, extends sharing to both output and internal layers of the network gao2023enhanced ; chen2023adamv ; xu2023demt ; xu2022mtformer . This approach helps reduce conflicts between tasks, even when the tasks have weak correlations. By allowing more flexible adaptation, soft parameter sharing can enhance the performance and adaptability of each individual task, particularly in settings where tasks may differ in complexity or relevance. In the context of MTL for ADAS, soft parameter sharing has become an attractive choice for improving the overall performance across multiple driving-related tasks by allowing tasks to learn shared representations while maintaining flexibility in

Read the complete text