Weakly Supervised Segmentation as Semantic-Based Regularization

Stefano Colamonaco, Andrei-Bogdan Florea, Jaron Maene · 2026

Read Weakly Supervised Segmentation as Semantic-Based Regularization by Stefano Colamonaco, Andrei-Bogdan Florea, Jaron Maene (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13674.txt.

Introduction

Stefano Colamonaco, Andrei-Bogdan Florea and Jaron Maene address weakly supervised semantic segmentation, training dense pixel-level models from partial or coarse annotations. The authors take a neurosymbolic view, joining differentiable fuzzy logic with deep segmentation nets. Weak annotations and domain priors become continuous logical constraints tuning SAM under weak supervision. The refined model yields better pseudo-labels, and a second-stage prompt-free model reaches state-of-the-art accuracy, often beating densely supervised baselines.

Catalog record

CREATORStefano Colamonaco, Andrei-Bogdan Florea, Jaron Maene
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58544
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13674

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Weakly Supervised Segmentation as Semantic-Based Regularization Stefano Colamonaco, Andrei-Bogdan Florea, Jaron Maene Abstract Weakly supervised semantic segmentation (WSSS) trains dense pixel-level segmentation models from partial or coarse annotations such as bounding boxes, scribbles, or image-level tags. While recent work leverages foundation models such as the Segment Anything Model (SAM) to generate pseudo-labels, these approaches typically depend on heuristic prompt choices and offer limited ways to incorporate prior knowledge or heterogeneous labels. We address this gap by taking a neurosymbolic perspective: integrating differentiable fuzzy logic with deep segmentation models. Weak annotations and domain-specific priors are unified as continuous logical constraints that fine-tune SAM under weak supervision. The refined foundation model then produces improved pseudo-labels, from which we train a second-stage prompt-free segmentation model. Experiments on Pascal VOC 2012 and the REFUGE2 optic disc/cup segmentation dataset show that our logic-guided fine-tuning yields higher-quality pseudo-labels, leading to state-of-the-art segmentation accuracy that often exceeds densely supervised baselines. 1. Introduction Semantic segmentation is a fundamental task in computer vision, enabling a pixel-level understanding of visual scenes. Despite remarkable progress, state-of-the-art methods still depend on large datasets with densely annotated masks, which is an expensive and time-consuming requirement that limits scalability. Producing pixel-perfect labels demands substantial manual effort, creating a major bottleneck for deploying segmentation models in new domains. To reduce this dependence on dense supervision, Weakly Supervised Semantic Segmentation (WSSS) has emerged as a practical alternative [ 8 ] . Instead of relying on full masks, WSSS methods learn from cheaper, indirect cues such as image-level tags [ 54 ] , bounding boxes [ 26 , 40 ] , or sparse scribbles [ 29 ] . These weak labels lower annotation costs but introduce ambiguity about object boundaries and shape, making it difficult for models to produce coherent segmentations without additional structural guidance [ 52 ] . Recently, the landscape of WSSS has been reshaped by foundation models such as the Segment Anything Model (SAM) [ 24 ] . Thanks to their prompt-based design and extensive pretraining, foundation models can generate high-quality pseudo-masks from simple user inputs, providing a powerful prior that many WSSS pipelines now exploit. However, this paradigm is limited by its reliance on fixed, pre-trained models. When applied to specialized domains (e.g., medical imaging) these models often struggle with domain shifts and object appearances unseen during their original training. Moreover, their prompt-driven nature provides no principled way to incorporate domain knowledge or symbolic constraints that could guide or correct the predictions. As a result, several current approaches still depend on heuristic prompt design rather than a general mechanism for reasoning about spatial and semantic structure. Stage 1 : Logic-guided SAM finetuning Stage 2 : Pseudo-label supervised training Weak labels Structural priors Smoothness, superpixel boundaries, etc. SAM Pseudo labels Fuzzy Loss (Eq. 2 ) Segmentation model Segmentation Loss Prompt Constraints Figure 1 : Overview of the proposed neurosymbolic weakly supervised segmentation framework. The method operates in two stages: First, the Segment Anything Model (SAM) is fine-tuned using differentiable logic constraints derived from weak annotations and general structural priors to generate refined pseudo-labels. Second, these pseudo-labels are used as ground truth to train a standard segmentation network for final deployment. Neurosymbolic learning offers a promising direction for addressing this gap by combining the flexibility of deep networks with the interpretability and compositionality of symbolic reasoning [ 36 ] . Through differentiable logic or constraint-based formulations, neurosymbolic models can integrate structured knowledge directly into the learning process, enforcing consistency, relational structure, or expert priors in a way that remains trainable end-to-end. While most existing efforts apply such reasoning at the object or scene level [ 35 , 11 ] , extending it to the pixel level opens the door to a neurosymbolic segmentation guided by explicit relational rules, a direction that has remained largely unexplored. Building on this idea, we introduce a neurosymbolic approach for weakly supervised semantic segmentation that fine-tunes a foundation segmentation model using multiple weak annotation types simultaneously, expressed as differentiable fuzzy logic constraints. In contrast to prompt-based adaptation or single-label weak supervision, our approach unifies heterogeneous supervision signals , such as bounding boxes and scribbles, within a single, end-to-end optimization scheme. By grounding logical constraints on these weak cues, the model learns not only to fit annotations but also to satisfy high-level spatial and relational priors. Practically, the method follows a two-stage pipeline. In the first stage, SAM is fine-tuned through differentiable logic losses derived from weak annotations, yielding refined and structurally consistent pseudo-masks. In the second stage, these pseudo-masks are used as training data for a prompt-free segmentation network, enabling scalable deployment without human prompts. In summary, the contributions of this work are (i) a neurosymbolic formulation of weakly supervised segmentation using differentiable fuzzy logic, (ii) a general training scheme that fine-tunes SAM to integrate multiple weak annotations and priors jointly, and (iii) empirical evidence that such reasoning-guided learning can achieve state-of-the-art performance on standard segmentation benchmarks. 1 1 1 The source code is available at https://github.com/StefanoColamonaco/Logic-Guided-Segmentation 2. Related Work Weak supervision. WSSS has been studied extensively with different annotation types. Early work explored bounding boxes [ 26 , 40 , 46 , 21 ] , scribbles [ 29 ] , or point-level cues [ 51 ] as alternatives to dense masks. Moreover, many methods exploited class activation maps (CAMs) for image-level labels [ 7 ] or to propagate labels from sparse scribbles [ 39 ] . For instance, TEL [ 27 ] optimizes a tree energy loss for scribble supervision, while AGMM [ 51 ] employs adaptive Gaussian mixtures. Concurrently, the field has seen a surge in methods leveraging vision-language models such as CLIP to extract dense pixel-level affinities from simple Image-Level Tags (ILT), as demonstrated by CLIP-ES [ 30 ] and CLIP-CPAL [ 45 ] . However, a primary limitation of these approaches is that they are highly tailored to a single, specific type of weak supervision (e.g., only image-level tags or only scribbles), making them inflexible when diverse annotation types are available. To integrate multiple forms of weak supervision, Xu et al. [ 52 ] proposed a unified framework that leverages multiple weak cues (tags, bounding boxes, and partial labels) via a latent variable Conditional Random Field (CRF). However, the complexity of its alternating optimization made it difficult to scale, leading later WSSS methods to revert to disjointed pipelines. We overcome this bottleneck by replacing intractable structured prediction with differentiable fuzzy logic. Foundation models. The recent shift of task-specific models to foundation models has also been adopted in image segmentation. A notable example is the Segment Anything Model (SAM) [ 24 ] , which was trained on a billion-mask dataset and generates accurate segmentation masks from simple prompts (points or boxes). Jiang and Yuqi [ 22 ] showed that prompting SAM with weak labels yields high-quality pseudo-labels that can be used to train a segmentation network in a weakly supervised fashion. Although promptable foundation models often outperform traditional pipelines, reliance on a fixed pre-trained model can limit adaptability to out-of-distribution domains [ 37 ] . Because of this, several works have fine-tuned SAM using large-scale, densely annotated datasets. For instance, MedSAM [ 31 ] and SAM-Med2D [ 10 ] adapt SAM to a wide variety of medical

Read the complete text