SHED Light on Segmentation for Dense Prediction
Seung Hyun Lee, Sangwoo Mo, Stella X. Yu · 2026
Read SHED Light on Segmentation for Dense Prediction by Seung Hyun Lee, Sangwoo Mo, Stella X. Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22529.txt.
Introduction
Seung Hyun Lee, Sangwoo Mo, and Stella Yu built SHED, a model that reads every pixel of an image while honoring scene structure. Dense prediction, as this task is called, feeds 3D sensing and robotics. Their 2026 preprint notes that most tools treat each pixel alone, which yields ragged, clashing maps.
SHED bakes a geometric prior — a built-in sense of shape — into its encoder-decoder frame. The result should be maps that hang together as real scenes do. Cleaner maps help robots move and grasp with trust. The work is a design fix for a core vision task.
Catalog record
| CREATOR | Seung Hyun Lee, Sangwoo Mo, Stella X. Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22529
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SHED Light on Segmentation for Dense Prediction Seung Hyun Lee, Sangwoo Mo, Stella X. Yu Abstract Dense prediction infers per-pixel values from a single image and is fundamental to 3D perception and robotics. Although real-world scenes exhibit strong structure, existing methods treat it as an independent pixel-wise prediction, often resulting in structural inconsistencies. We propose SHED, a novel encoder-decoder architecture that enforces geometric prior explicitly by incorporating segmentation into dense prediction. By bidirectional hierarchical reasoning, segment tokens are hierarchically pooled in the encoder and unpooled in the decoder to reverse the hierarchy. The model is supervised only at the final output, allowing the segment hierarchy to emerge without explicit segmentation supervision. SHED improves depth boundary sharpness and segment coherence, while demonstrating strong cross-domain generalization from synthetic to the real-world environments. Its hierarchy-aware decoder better captures global 3D scene layouts, leading to improved semantic segmentation performance. Moreover, SHED enhances 3D reconstruction quality and reveals interpretable part-level structures that are often missed by conventional pixel-wise methods. 1 Introduction Dense prediction tasks, such as monocular depth estimation and semantic segmentation, serve as the foundation for spatial reasoning in robotics ( Liu et al. 2024 ) . At their core, these tasks require an understanding of how 2D image projections relate to the underlying 3D structure of the world. However, modern state-of-the-art models based on Vision Transformers (ViTs) ( Dosovitskiy 2020 ) and pixel-wise regression often treat these tasks as a collection of independent point estimations rather than a coherent structural reconstruction. This disconnect frequently results in structural leakage, where predicted boundaries do not align with physical objects, leading to blurred edges and inconsistent surface geometry ( Figure 1 , row 1). In contrast, the human visual system does not process images as an array of independent points. Instead, it employs a sophisticated bidirectional hierarchy ( Hochstein & Ahissar 2002 ) . Our perception integrates depth and segmentation through a feedback loop where a global scene layout is first inferred by grouping fine-grained textures into parts and wholes. This high-level structural scaffold then guides the refinement of local details. This part-whole reasoning ensures that local pixel-level predictions are constrained by the global geometric context. This organization results in depth maps characterized by sharp, high-fidelity boundaries and smooth intra-object variations ( Figure 1 , row 2). Figure 1 : Segment hierarchy for dense prediction (SHED). Conventional methods such as DPT ( Ranftl et al. 2021 ) perform pixel-wise prediction without considering structure, often resulting in blurry object shapes. SHED addresses this by leveraging a hierarchy of segment tokens to guide prediction. Unlike DPT, which uses fixed grid tokens across all layers, we adapt its ViT ( Dosovitskiy 2020 ) blocks into two stages: the encoder pools superpixel tokens into coarser segment tokens, and the decoder progressively refines predictions from coarse to fine segments, producing depth maps with structural coherence. To realize this idea, we propose SHED , S egment H i e rarchy for D ense Prediction, a novel architecture that re-conceptualizes dense prediction as a process of hierarchical structural reasoning. Unlike traditional models that decode latent features directly back to a rigid pixel grid, SHED learns an implicit hierarchy of segments that adaptively decompose the scene. Note that these segments are not provided via external supervision. They are learned end-to-end, guided solely by the dense prediction objective. The technical core of SHED lies in its symmetric bidirectional flow. While the encoder iteratively groups pixels into increasingly abstract segment tokens, our contribution focuses on the structural inversion of this process in the decoder. To produce a structured dense map, SHED’s decoder unpools features from coarser segments to finer ones using probabilistic assignments computed during the encoding phase. This reverse hierarchy, which has no counterpart in encoder-only segmentation frameworks, enables coarse structural context to directly constrain fine-grained predictions. By distributing segment-level features across associated regions, the model naturally enforces sharp boundaries between distinct objects and maintains geometric smoothness within a single surface. This mechanism allows high-level geometric priors to strictly inform the pixel-level output, preserving the global layout while capturing fine-grained structural detail. Integrating hierarchical segmentation into the dense prediction pipeline grants SHED three key advantages: • SHED produces depth maps with high-fidelity boundary alignment. This inherent structural bias significantly enhances robustness in cross-domain transfer, such as from synthetic to real-world environments, where traditional pixel-wise models often fail to maintain consistency. • We demonstrate that depth supervision induces structured representations that capture the 3D scene layout. Consequently, SHED refines semantic segmentation by ensuring that category labels strictly adhere to geometrically consistent spatial boundaries. • Since SHED maintains a symbolic hierarchy of segments, it enables the discovery of object parts in 3D space without any part-level labels. This provides a level of structural interpretability that holistic, black-box architectures like DPT ( Ranftl et al. 2021 ) fundamentally lack. 2 Related Work Dense prediction is a core problem in computer vision, aiming to assign pixel-level outputs across an image ( Forsyth & Ponce 2002 ) . It includes tasks such as semantic segmentation ( Long et al. 2015 ) , depth estimation ( Eigen et al. 2014 ) , optical flow ( Teed & Deng 2020 ) , and image editing ( Isola et al. 2017 ) . Modern approaches typically adopt encoder-decoder architectures, such as U-Net ( Ronneberger et al. 2015 ) and DPT ( Ranftl et al. 2021 ) , trained using task-specific supervision. These models perform well on benchmarks focused on per-pixel accuracy, as demonstrated by large-scale systems like Segment Anything ( Ravi et al. 2025 ) and Depth Anything ( Yang et al. 2024 ) . However, recent studies show that even top-performing models often lack structural consistency ( El Banani et al. 2024 ; Man et al. 2024 ) . We argue that dense prediction should move beyond local estimation toward structured reasoning guided by region-level abstraction. Figure 2 : SHED introduces a bidirectional segment hierarchy for dense prediction. SHED decomposes dense prediction into two complementary processes: structural abstraction and structural inversion. 1) The encoder constructs a forward hierarchy by grouping superpixel tokens into increasingly abstract segments. 2) The decoder explicitly inverts this hierarchy by unpooling segment representations from coarse to fine, enabling global structural context to directly constrain pixel-level predictions. Segment tokens are projected into region-aligned 2D feature maps and fused across hierarchy levels, together with early convolutional features, to recover fine details and produce the final dense map. Monocular depth estimation is a representative dense prediction task, that infers per-pixel depth from a single image. It is widely used in 3D reconstruction ( Song et al. 2017 ) , autonomous driving ( Geiger et al. 2012 ) , and robotic perception ( Tateno et al. 2017 ) . Structural cues in depth estimation have been extensively explored to enhance geometric coherence. Existing approaches can be broadly categorized into four types: 1) Representation approaches modify how depth is encoded, such as by discretizing depth values ( Fu et al. 2018 ; Bhat et al. 2021 ; Li et al. 2024 ) or modeling spatial dependencies ( Liu et al. 2015 ; Cheng et al. 2018 ; Yuan et al. 2022 ) . 2) Regularization imposes geometric constraints through loss functions that promote