TEM 3 -Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving

Wenzhuo Liu, Yicheng Qiao, Zhen Wang, Qiannan Guo, Zilong Chen, Meihua Zhou, Xinran Li, Letian Wang, Zhiwei Li, Huaping Liu, Wenshuo Wang · 2025

Read TEM 3 -Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving by Wenzhuo Liu, Yicheng Qiao, Zhen Wang, Qiannan Guo, Zilong Chen, Meihua Zhou, Xinran Li, Letian Wang, Zhiwei Li, Huaping Liu, Wenshuo Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18084.txt.

Introduction

Wenzhuo Liu and ten co-workers wrote a 2025 paper on a learning frame for driver assistance that handles four tasks at once. The tasks cover driver emotion, driver behavior, road context and car behavior. Past multi-task methods used one sense type and ran too slowly for live use. Their frame reads several camera views over time.

A first part pulls time and space features from the views at low cost. A second part gates features per task, so each task takes what helps it most. That gating also curbs harm from task clashes. On a named data set, the model topped all four tasks while staying small and fast.

Catalog record

CREATORWenzhuo Liu, Yicheng Qiao, Zhen Wang, Qiannan Guo, Zilong Chen, Meihua Zhou, Xinran Li, Letian Wang, Zhiwei Li, Huaping Liu, Wenshuo Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18084

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TEM 3 -Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving Wenzhuo Liu, Yicheng Qiao, Zhen Wang, Qiannan Guo, Zilong Chen, Meihua Zhou, Xinran Li, Letian Wang, Zhiwei Li, Huaping Liu, Wenshuo Wang Abstract Multi-task learning (MTL) can advance assistive driving by exploring inter-task correlations through shared representations. However, existing methods face two critical limitations: single-modality constraints limiting comprehensive scene understanding and inefficient architectures impeding real-time deployment. This paper proposes TEM 3 -Learning ( T ime- E fficient M ulti m odal M ulti-task Learning), a novel framework that jointly optimizes driver emotion recognition, driver behavior recognition, traffic context recognition, and vehicle behavior recognition through a two-stage architecture. The first component, the mamba-based multi-view temporal-spatial feature extraction subnetwork (MTS-Mamba), introduces a forward-backward temporal scanning mechanism and global-local spatial attention to efficiently extract low-cost temporal-spatial features from multi-view sequential images. The second component, the MTL-based gated multimodal feature integrator (MGMI), employs task-specific multi-gating modules to adaptively highlight the most relevant modality features for each task, effectively alleviating the negative transfer problem in MTL. Evaluation on the AIDE dataset, our proposed model achieves state-of-the-art accuracy across all four tasks, maintaining a lightweight architecture with fewer than 6 million parameters and delivering an impressive 142.32 FPS inference speed. Rigorous ablation studies further validate the effectiveness of the proposed framework and the independent contributions of each module. The code is available on https://github.com/Wenzhuo-Liu/TEM3-Learning . I INTRODUCTION Advanced Driver Assistance Systems (ADAS) can improve driving safety by continuously monitoring the driver’s state and traffic environment [ 14 , 54 ] . However, existing research is mostly limited to the solution of single tasks, such as driver emotion/behavior, traffic environment recognition, without addressing the inherent interdependencies between these tasks [ 38 ] . Studies show a significant coupling relationship between the drivers’ state and the traffic environment [ 51 , 36 ] . For example, drivers frequently adjust their behavior, such as changing lanes based on surrounding traffic conditions, while traffic congestion can induce driver anxiety. Integrating driver state and traffic environment recognition into a unified multi-task learning (MTL) framework can provide a more holistic understanding of driving scenarios and enhance the safety performance of ADAS [ 51 ] . Fig. 1: Comparison of four mainstream algorithm frameworks: (a) single-modal single-task, (b) multimodal single-task, (c) single-modal multi-task, and (d) the proposed multimodal multi-task. As the number of tasks and modalities increases, model complexity rises significantly, making it crucial to balance computational efficiency with accuracy in multimodal multi-task learning. Compared to single-task learning, MTL reduces overfitting by sharing features across tasks, thereby improving the performance of each individual task [ 9 , 24 ] . In the context of ADAS, existing MTL models generally focus on related sub-tasks. For example, Wu proposed a model that jointly learns multiple traffic environment-related tasks, enabling simultaneous recognition of lane markings, drivable areas, and object detection in driving scenes [ 47 ] . Similarly, Xing et al. focus on driver-related MTL to recognize the driver’s emotions and behaviors [ 48 ] . While these approaches have yielded performance improvements, they often struggle with negative transfer, where task performance deteriorates due to task-related conflicts or differences [ 27 ] . MTL models tend to have more complex structures compared to single-task learning, as they must simultaneously handle multiple tasks [ 47 , 52 ] (Fig. 1 (a) and (c)). Many MTL methods rely on inputs from a single modality such as driving scene images [ 8 , 6 ] . However, optimal task performance requires complementary multimodal inputs [ 17 , 28 ] . For example, combining driver images with eye movement data can improve emotion recognition [ 37 ] , while fusing image and point cloud data can enhance traffic object recognition [ 19 , 1 ] . Unfortunately, many multimodal input algorithms still rely on independent feature extraction [ 45 , 30 , 29 , 23 ] , which significantly increases model parameters, reduces inference speed, and compromises practical scalability (Fig. 1 (b)). Therefore, a key challenge lies in effectively leveraging multimodal information while optimizing individual task performance, enhancing task-level synergy, and maintaining high inference speed with a minimal parameter count. To address these challenges, we propose a Time-efficient multimodal multi-task learning network that simultaneously performs four tasks — driver emotion recognition ( der ), driver behavior recognition ( dbr ), traffic context recognition ( tcr ), and vehicle behavior recognition ( vbr ) — using multimodal data (Fig. 1 (d)). This is achieved through two key components: the Mamba-based multi-view temporal-spatial feature extraction subnetwork (MTS-Mamba) and the multi-task learning-based gated multimodal feature integrator (MGMI). MTS-Mamba introduces a forward-backward temporal scanning mechanism and a global-local spatial feature extraction strategy, enabling efficient extraction of temporal-spatial features from multi-view sequential images at low computational cost. This approach provides richer and more robust feature representations for subsequent multi-task recognition. MGMI, inspired by previous work [ 34 ] , incorporates a multi-gating mechanism that dynamically adjusts the weights of modality features based on task-specific attention. This selective reinforcement of important features helps alleviate negative transfer. We validated the effectiveness of our method using the publicly available AIDE dataset. Experimental results show that our model outperforms previous methods on all four tasks, with fewer than 6M parameters and exceptionally fast inference speed, demonstrating its efficiency and practicality. Our contributions include: \bullet A time-efficient multimodal MTL framework that provides a new paradigm for multimodal MTL in ADAS. \bullet The MTS-Mamba subnetwork, which effectively extracts temporal-spatial features from multi-view sequential images across multiple dimensions. \bullet The MGMI mechanism, which adaptively adjusts attention to different modalities for each task, mitigating negative transfer and enhancing task-specific feature extraction. II Related Work II-A Multimodal Learning ADAS systems that rely solely on single-modality data often struggle to handle the complex and dynamic challenges encountered during driving [ 37 , 15 , 12 , 40 , 46 , 3 , 1 ] . Each modality offers unique advantages and limitations, and leveraging multiple modalities can provide a more comprehensive understanding of the driving environment. For example, combining multi-view driving scene images with LiDAR data enhances environmental perception [ 49 , 26 ] . Similarly, integrating driver images and joint information can effectively capture facial expressions and behavioral states [ 51 ] . As a result, many studies have increasingly turned to multimodal data to enhance task accuracy. For instance, Zhou et al. [ 55 ] used front-view driving scene images, driver images, and vehicle speed data for driver behavior recognition, while Liu et al. [ 33 ] combined camera images and LiDAR data for 3D object detection. However, multimodal learning in ADAS faces two key challenges. First, many existing models rely on independent feature extraction branches for each modality, even when processing multi-view images [ 29 , 17 , 42 , 55 , 31 ] . This approach significantly increases the model’s parameter count and overlooks potential intermodal interactions, leading to inefficient information use. Second, multimodal fusion methods are often limited to combining only a few modality features, restricting their generalization ability and scalability. II-B Multi-task Learning MTL improves model performance on each task by sharing features across tasks [ 9 , 24 ] . The two predominant strategies for parameter sharing

Read the complete text