3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding

Xiaoye Wang, Chen Tang, Xiangyu Yue, Wei-Hong Li · 2025

Read 3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding by Xiaoye Wang, Chen Tang, Xiangyu Yue, Wei-Hong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20646.txt.

Introduction

Wang, Tang, Yue, and Li inject 3D awareness into networks juggling several vision tasks at once. Their 2025 paper tackles multi-task learning for dense scene understanding. One network predicting segmentation and depth together saves compute. Most prior designs relate tasks only in flat image space. The resulting features lack a sense of real geometry.

Their Cross-view Module shares information between different camera views. Cost volumes capture how views geometrically agree with each other. The module bolts onto existing multi-task encoders unchanged. Single-view and multi-view inputs are both supported. A spatial-aware encoder with multi-view transformer does the work. Tests ran on the NYUv2 and PASCAL-Context benchmarks. Geometric consistency improved results across attached base methods. Strong backbones like SAK and DINOv3 variants all benefited. Static scenes remain the comfort zone. Moving objects and camera motion remain future work.

Catalog record

CREATORXiaoye Wang, Chen Tang, Xiangyu Yue, Wei-Hong Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D-Aware Multi-Task Learning with Cross-View Correlations for Dense Scene Understanding Xiaoye Wang, Chen Tang, Xiangyu Yue, Wei-Hong Li Abstract This paper addresses the challenge of training a single network to jointly perform multiple dense prediction tasks, such as segmentation and depth estimation, i.e., multi-task learning (MTL). Current approaches mainly capture cross-task relations in the 2D image space, often leading to unstructured features lacking 3D-awareness. We argue that 3D-awareness is vital for modeling cross-task correlations essential for comprehensive scene understanding. We propose to address this problem by integrating correlations across views, i.e., cost volume, as geometric consistency in the MTL network. Specifically, we introduce a lightweight Cross-view Module (CvM), shared across tasks, to exchange information across views and capture cross-view correlations, integrated with a feature from MTL encoder for multi-task predictions. This module is architecture-agnostic and can be applied to both single and multi-view data. Extensive results on NYUv2 and PASCAL-Context demonstrate that our method effectively injects geometric consistency into existing MTL methods to improve performance. 1 Introduction One central focus of multi-task learning (MTL) in computer vision 59 is to jointly solve multiple visual tasks within a single network. By sharing the majority of model parameters, MTL models effectively reduce computational cost and memory capacity while exploiting cross-task inductive biases 8 ; 59 ; 75 ; 32 . This multi‑task collaborative framework resonates deeply with a variety of real‑world applications 75 , such as robotic automation, where depth estimation and spatial awareness for obstacle avoidance are combined with semantic segmentation and other scene understanding tasks for object localization 1 . However, building an MTL model that performs consistently well for all desired tasks remains a challenging problem as it requires the MTL model to maintain a good balance between shared and task-specific features 32 . Modern deep learning-based MTL methods have explore various architectures innovations to address this: Liu et al . 37 introduce task-specific attention modules with a fully shared feature encoder for more flexible feature sharing; Vandenhende et al . 59 and Bruggemann et al . 59 design cross-task attention modules to capture inter-task relations from multi-scale features; Recent transformer-based methods have advanced MTL through techniques such as high-resolution multi-scale decoding 71 , prompt learning 73 ; 62 , task-specific experts 72 ; 69 ; 15 ; 18 and multi-teacher knowledge distillation 48 ; 23 ; 42 . Despite advancements, most existing MTL methods predominantly rely on mapping 2D images to high-dimensional features and per-pixel supervision (as shown in (a)), resulting in unstructured features. This unstructured feature space, coupled with a lack of explicit mechanisms for modeling spatial consistency, can lead to noisy inter-task relations and degraded performance 32 ; 81 ( (c) top). In response, pioneering work like 3DMTL 32 explores integrating 3D regularization into MTL by proposing a structured 3D-aware regularizer that projects shared features into a 3D feature space and decodes multiple tasks via differentiable rendering. Another approach, MuvieNeRF 81 recasts multi-task dense prediction as multi-view synthesis, embedding both cross-view and cross-task attention within a NeRF 44 backbone to synthesize multiple scene properties. However, despite utilizing multi-view data for 3D-awareness via feature projection and rendering, 3DMTL 32 does not directly extract and integrate multi-view geometric cues into its shared representations. Meanwhile, MuvieNeRF 81 requires multi-view data and camera parameters during inference, which limits its scalability for real-world MTL applications. On the other hand, recent work on multi-view scene reconstruction, such as VGGT 61 , MVSplat 12 , and DepthSplat 65 , demonstrates the success of leveraging multiple views for building robust 3D representations. MVSplat 12 effectively reconstructs high-fidelity 3D scenes by efficiently processing sparse multi-view images with 3D Gaussian Splatting. Complementing this, DepthSplat 65 integrates depth information from a pretrained depth estimation model with Gaussian Splatting to achieve superior 3D reconstruction quality. VGGT 61 leverages a unified Transformer architecture enhanced with visual geometry grounding to improve 3D understanding from multi-view inputs and multiple downstream 3D tasks. However, these methods primarily focus on scene reconstruction or 3D representation and are not directly designed to enhance multi-task learning that jointly performs multiple dense prediction tasks from single view image input, including depth estimation, boundary detection, surface normal estimation, and semantic segmentation, making their integration into MTL unclear. This leads us to a critical question for current MTL frameworks: Can we introduce cross-view correlations into MTL to ensure high geometric consistency across vision tasks? Motivated by this insight, we propose a 3D-aware multi-task learning framework that tightly integrates the design principles of MTL and 3D reconstruction, as shown in (b). Our approach augments monocular multi-task learning feature encoder with multi-view geometric cues via a dedicated geometric pathway, which we term the Cross-view Module (CvM), allowing the model to learn representations that are simultaneously task-aligned and geometry-aware ( (c)). The CvM consists of three sequential components: (i) a spatial-aware encoder for extracting geometric and spatial primitives, (ii) a multi-view transformer that ingests these encoded features for relational interactions, and (iii) a cost volume module that reconstructs cross-view correlations as a geometric representation. Crucially, the spatial-aware encoder operates independently of the main monocular MTL pathway, allowing it to leverage strong inductive biases for spatial locality to explicitly capture geometry-rich cues. Its features are then passed into the multi-view transformer, where intra- and cross-view attention forge robust geometry-aware representations, culminating in the construction of a cost volume that establishes dense correspondence across views. These 3D-aware features complement the original monocular MTL features and are concatenated with them before being passed into lightweight, task-specific decoder heads. Our method supports both MTL training on multi-view data (or video inputs) and single-view, while only a single image is needed for inference, making it broadly applicable in practice. To summarize, our main contributions are as follows: • Unlike prior work that primarily focuses on learning direct mappings between input images and desired task ground-truths, we propose to enable 3D-aware MTL by integrating cross-view correlation, i.e . , cost volume, as a geometric consistency into multi-task learning through an introduced multi-view module. • Our method is architecture-agnostic, allowing seamless plugged into various existing MTL architectures to enhance their performance. • Our approach is applicable to both single and multi-view data during training, yet requires only a single image for inference, eliminating the need for camera parameters at deployment. • Extensive experimental results demonstrate the effectiveness and superiority of our proposed 3D-aware multi-task learning framework on standard NYUv2 and PASCAL-Context benchmarks. 2 Related Work 2.1 Multi-task Learning Multi-task Learning (MTL) 8 aims at learning a single network that jointly estimates accurate predictions for multiple desired tasks. Recent research of multi-task learning in computer vision can be broadly divided into two categories 59 ; 50 . The first group aims at addressing the unbalanced optimization issues - each task’s loss function often exhibits distinct scales and convergence behaviors, jointly minimizing them can lead to optimization conflicts and performance degradation. To address this issue, prior work proposes to estimate loss weights dynamically ( 25 ; 37 ; 20 ; 13 ; 34 ; 52 ; 36 ) , mitigating conflicts between gradient conflicts 76 ; 35 ; 14 ; 16 ; 57 or aligning features with single-task models ( 30 ; 31 ) . Our work is more related

Read the complete text