Depth Edge Alignment Loss: DEALing with Depth in Weakly Supervised Semantic Segmentation
Patrick Schmidt, Vasileios Belagiannis, Lazaros Nalpantidis · 2025
Read Depth Edge Alignment Loss: DEALing with Depth in Weakly Supervised Semantic Segmentation by Patrick Schmidt, Vasileios Belagiannis, Lazaros Nalpantidis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17702.txt.
Introduction
Patrick Schmidt, Vasileios Belagiannis, and Lazaros Nalpantidis sharpen weakly supervised image segmentation with depth cues. Their 2025 arXiv paper serves robots entering new domains, where pixel-level labels are pricey and image-level labels are all a trainer may have. Weak supervision builds pixel maps from those cheap labels, but the maps stay rough at object borders.
Their loss, called DEAL, aligns the predicted segments with depth edges, a signal robots commonly carry. It is model-agnostic, so it can wrap many segmentation designs, and it stacks with other losses for further gains. On PASCAL VOC, MS COCO, and a HOPE onboarding split, the loss lifted mean overlap scores, with the largest jump on the last set. Code is public. The authors frame depth, often treated as a side input, as a direct teacher for where one object ends and another begins.
Catalog record
| CREATOR | Patrick Schmidt, Vasileios Belagiannis, Lazaros Nalpantidis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Depth Edge Alignment Loss: DEALing with Depth in Weakly Supervised Semantic Segmentation Patrick Schmidt, Vasileios Belagiannis, Lazaros Nalpantidis Abstract Autonomous robotic systems applied to new domains require an abundance of expensive, pixel-level dense labels to train robust semantic segmentation models under full supervision. This study proposes a model-agnostic Depth Edge Alignment Loss to improve Weakly Supervised Semantic Segmentation models across different datasets. The methodology generates pixel-level semantic labels from image-level supervision, avoiding expensive annotation processes. While weak supervision is widely explored in traditional computer vision, our approach adds supervision with pixel-level depth information, a modality commonly available in robotic systems. We demonstrate how our approach improves segmentation performance across datasets and models, but can also be combined with other losses for even better performance, with improvements up to + 5.439 +5.439 , + 1.274 +1.274 and + 16.416 +16.416 points in mean Intersection over Union on the PASCAL VOC / MS COCO validation, and the HOPE static onboarding split, respectively. Our code will be made publicly available. I Introduction Autonomous robotic systems rely on robust perception capabilities to ensure smooth and safe operation without human intervention. A key component to this is semantic segmentation for precise localization of objects in a scene. These models are usually trained fully-supervised with a plethora of images and pixel-level, dense labels which are expensive to obtain, and for new applications often not readily available. Weakly Supervised Semantic Segmentation (WSSS) addresses this by obtaining pixel-level dense labels from weaker forms of supervision, such as Class Activation Maps (CAM) extracted from image-level classification labels [ 1 , 2 , 3 , 4 , 5 , 6 ] . While WSSS shows promising results on general-purpose datasets like PASCAL VOC [ 7 ] or MS COCO [ 8 ] , it has not been widely explored in a robotics context yet. We argue that WSSS can be extended to operate in real-world robotics scenarios, by incorporating additional information such as depth—commonly available to robots due to the omnipresence of RGB-D sensors. We therefore propose a new, model-agnostic Depth Edge Alignment Loss (DEAL), which enforces edges extracted from a CAM to align with edges extracted from the depth map of the current scene. DEAL is added on top of other losses, and can be combined with other improving losses like Importance Sampling Loss (ISL) and Feature Similarity Loss (FSL) [ 9 ] , boosting those even further. Fig. 1: A graphical summary of the intuition behind our proposed Depth Edge Alignment Loss. Alignment of object boundaries in RGB images and edges extracted at depth discontinuities can improve the results of WSSS. We observe that including depth information in the training process of existing WSSS methods can improve the segmentation performance in terms of mean Intersection over Union (mIoU) from weak supervision. Depth provides pixel-level dense information about the spatial layout of a scene and—as opposed to pixel-level semantic labels—is inexpensive to obtain through RGB-D sensors commonly used in autonomous robotic systems. In Multi-Task Learning, where multiple down-stream tasks share one common feature extraction backbone and thus a common feature representation, research has shown high task affinity between semantic segmentation and monocular depth estimation, provided sufficient network capacity [ 10 ] . Our added loss shows that it can leverage this affinity, as it improves mIoU performance across datasets and models. In addition, even when using real-life noisy RGB-D data, our proposed DEAL shows improvements, showing it is robust to noise. While our approach prerequisites CAM-based WSSS models, which is the most-explored mode of generating dense pixel-level labels from image-level supervision, it can inspire recent WSSS approaches using Vision Language Models to include depth information in their training and inference processes. Our contributions can be summarized as follows: • We introduce our novel Depth Edge Alignment Loss (DEAL) which improves mIoU segmentation performance in CAM-based WSSS frameworks. • We demonstrate how DEAL can improve segmentation performance across datasets and models . • We show that our depth-guided WSSS can yield strong improvements for robotic applications, even in the presence of noisy depth data. II Related Work II-A Weakly Supervised Semantic Segmentation (WSSS) WSSS has been proposed as an alternative to Fully Supervised Semantic Segmentation (FSSS). FSSS requires pixel-level semantic labels, which are expensive to obtain. In contrast, in the most common case, WSSS uses much easier to obtain image-level category labels for training. Other forms of weak supervision in the WSSS task include scribbles, bounding box, or foreground/background point annotations [ 11 ] . To highlight the added time requirements to obtain pixel-level labels, on the MS COCO dataset, it took workers about 85,000 hours to obtain these for 328,000 images [ 8 ] , including 20,000 hours for image-level labeling and 10,000 hours for foreground point annotations. Commonly, image-level labels are used to train an image classification network, then generate a coarse localization of objects using Class Activation Mappings (CAM) which are further refined using dense Conditional Random Fields (CRF) [ 12 ] and finally used to train a segmentation network on the extracted, pixel-level pseudo labels [ 1 , 2 , 3 , 4 , 5 , 6 ] . As outlined, WSSS usually builds upon a multi-stage framework of classification training, pseudo-label generation and refinement, and segmentation training. Single-stage, end-to-end approaches do exist [ 13 , 14 , 15 , 16 , 17 , 11 ] , but they generally do not match the performance of multi-stage approaches. In our experiments, we focus on approaches using image-level classification labels and on the first two stages of the framework, classification network training and CAM generation. Recent approaches leverage foundation models like the Segment Anything (SAM) model [ 18 , 19 , 20 , 21 ] , however, since SAM was trained with mask supervision where human mask refinement was involved [ 22 ] , we argue that these frameworks are not WSSS approaches, given the influence of pixel-level labels through SAM. With the rise of Vision Transformer (ViT)-based architectures, some works leverage the self-localizing properties of this architecture [ 23 , 24 , 25 ] . The similarity to the Transformer architecture commonly used in Natural Language Processing (NLP) enabled the development of Vision-Language models like Contrastive Language-Image Pretraining (CLIP) [ 26 ] . In a WSSS context, recent approaches leveraged CLIP to generate initial pseudo-mask seeds, departing from CAM-based approaches [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ] . In our work, we demonstrate our method both on a ViT-based and a ResNet-based WSSS framework, showcasing its model-agnostic property. II-B WSSS on non-standard datasets Typically, studies on WSSS present their approaches on two standard, general-purpose datasets: PASCAL VOC and MS COCO [ 1 , 2 , 3 , 4 , 5 , 6 ] . These datasets have certain characteristics that do not necessarily hold true in a real-life robotics scenario: they contain close to no object co-occurrence, a small number of classes per image and a large number of negative examples per class [ 36 ] . Opposed to the standard WSSS studies, [ 36 ] shows a CLIP-based approach on driving scenes from the CityScapes dataset under high object co-occurrence ratios. Akiva and