Lighting-aware Unified Model for Instance Segmentation
Qisai Liu, Alloy Das, Zhanhong Jiang, Joshua R. Waite, Aditya Balu, Adarsh Krishnamurthy, Soumik Sarkar · 2026
Read Lighting-aware Unified Model for Instance Segmentation by Qisai Liu, Alloy Das, Zhanhong Jiang, Joshua R. Waite, Aditya Balu, Adarsh Krishnamurthy, Soumik Sarkar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20436.txt.
Introduction
Liu and six co-workers make instance segmentation robust to lighting with a Lighting Convolutional-Attention adapter. Foundation models like SAM generalize well but degrade under varied illumination, and the adapter avoids retraining the heavy backbone. Their dual-branch adapter reads RGB features beside contrast maps, sensing structural change rather than lighting artifacts. Pairwise training penalizes gaps between clean images and their lighting variants, and a new Unity synthetic set tests complex real-world lighting.
Catalog record
| CREATOR | Qisai Liu, Alloy Das, Zhanhong Jiang, Joshua R. Waite, Aditya Balu, Adarsh Krishnamurthy, Soumik Sarkar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62742 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20436
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Lighting-aware Unified Model for Instance Segmentation Qisai Liu, Alloy Das, Zhanhong Jiang, Joshua R. Waite, Aditya Balu, Adarsh Krishnamurthy, Soumik Sarkar Abstract Foundation models like the Segment Anything Model (SAM) demonstrate impressive zero-shot generalization but frequently degrade under diverse real-world illumination, particularly for instance segmentation. In this work, we address this limitation by developing Lighting Convolutional-Attention (LCA) , an adapter module that enhances segmentation robustness without fine-tuning the heavy backbone. LCA employs a dual-branch architecture to process RGB features alongside contrast maps, enabling physically motivated sensitivity to structural changes rather than illumination artifacts. We optimize LCA through a pairwise training strategy, introducing a targeted loss term that explicitly penalizes discrepancies between clean images and their corresponding illumination variants. To evaluate and support this architecture, we conduct a comprehensive empirical study across multiple existing benchmarks and present a novel Unity-based synthetic dataset specifically designed to accurately replicate complex real-world lighting conditions. Extensive experimental results demonstrate that our approach successfully bridges the domain gap, delivering superior lighting-robust segmentation. 1 Introduction Instance segmentation has achieved remarkable progress through foundation models such as the Segment Anything Model (SAM) [ 16 ] , which demonstrates impressive zero-shot generalization across diverse visual domains. However, because these models are predominantly trained on well-lit and high-quality imagery, they frequently suffer significant performance degradation when deployed in environments with adverse illumination. This creates a critical gap in real-world applications such as nighttime driving, indoor low-light scenes, overexposed outdoor settings, and mixed artificial and natural lighting conditions. In safety-critical applications such as autonomous driving [ 22 ] , robotic manipulation, and medical imaging [ 1 ] , lighting variation is the norm rather than the exception. Similarly, in agricultural applications, plant segmentation in outdoor scenes is subject to drastic illumination shifts from morning to night, leading to significant model degradation [ 23 ] . A segmentation model that performs well under controlled laboratory conditions but degrades under a cloudy sky or in a dimly lit warehouse is fundamentally unreliable. Yet, the dominant paradigm in segmentation research treats lighting variation as a secondary concern, addressed primarily through generic data augmentation [ 24 ] or domain adaptation [ 27 ] , rather than as a first-class architectural and training problem. We argue that addressing lighting robustness requires intervention at three complementary levels: (1) the data level , where training images must systematically span the complex space of realistic lighting conditions; (2) the architectural level , where the model must possess explicit mechanisms for reasoning about structural integrity independent of illumination; and (3) the training level , where the learning objective must explicitly force the extraction of lighting-invariant representations. Contributions. To this end, we propose Pairwise Lighting Augmentation Pipeline with Lighting Convolutional Attention (PLAP-LCA) , a framework that equips the Segment Anything Model (SAM) with robust lighting invariance without requiring computationally prohibitive fine-tuning of its heavy backbone. At the core of our approach is the LCA module, a specialized adapter injected into SAM’s frozen encoder. LCA employs a dual-branch architecture that explicitly separates standard spatial features from illumination-sensitive structural edge information, leveraging Laplacian-filtered contrast maps. We propose a pairwise training strategy that optimizes the network using standard segmentation supervision alongside a targeted feature consistency loss, penalizing representational discrepancies across lighting variations. This strategy is evaluated on both real-world public benchmarks and a new synthetic dataset generated from a Unity environment, covering a wide range of physically accurate illumination. In summary, our main contributions are as follows: (1) We introduce the LCA module, a lightweight dual-branch adapter that equips foundation models with illumination-robust structural representations. (2) We propose a curriculum-based pairwise training strategy and a novel feature consistency loss to explicitly bridge the representational gap between ideal and adverse lighting conditions. (3) We present a comprehensive empirical study on lighting vulnerabilities, supported by a novel Unity-based synthetic dataset for physically accurate illumination evaluation as well as three diverse public benchmarks: COCO [ 18 ] , VOC [ 9 ] , and Cityscapes [ 5 ] , demonstrating consistent improvements across all illumination conditions. Figure 1 : PLAP-LCA. Clean and variant images share a SAM ViT-B encoder with LCA modules injected into the final two blocks. Only LCA weights, scalar gates, and the mask decoder are trained through supervise and consistency losses. Please see the Supplementary Materials for concrete details of the whole framework. 2 Related Work In this section, we review primarily related work on foundation model for segmentation and refer interested readers to Sec. 0.A.1 for attention mechanism and data augmentation. Segmentation Models. The Segment Anything Model (SAM) [ 16 ] represents a paradigm shift in image segmentation, leveraging a dataset of over 1 billion masks to achieve strong zero-shot generalization through a promptable architecture. While SAM’s core design consists of a ViT-based image encoder, a lightweight prompt encoder, and a lightweight mask decoder is highly effective. Subsequent work has explored SAM adaptation for specialized domains. For instance, SAM-Adapter [ 4 ] incorporates task-specific tuning, while HQ-SAM [ 15 ] introduces an additional token to enhance mask boundary details. More recently, the community has expanded SAM’s capabilities; EfficientSAM [ 30 ] drastically reduces computational overhead via masked image pretraining, PerSAM [ 31 ] enables training-free, one-shot personalization for specific visual concepts, and SAM 2 [ 20 ] extends the foundation model into real-time video segmentation using a streaming memory architecture. However, despite these rapid architectural evolutions, existing adaptations primarily optimize for computational efficiency, semantic tracking, or boundary precision under standard conditions. They leave the foundation model vulnerable to low-level physical degradations like extreme lighting. Our work complements these efforts by addressing the orthogonal challenge of illumination invariance, introducing a lightweight adapter that secures lighting resilience without updating SAM’s pre-trained weights. Robustness to Environmental Conditions. Segmentation under adverse conditions has been extensively studied, primarily within the autonomous driving domain. While Cityscapes [ 5 ] established the standard benchmark for urban scene understanding under controlled, clear-weather conditions, ACDC [ 22 ] , which captures fog, rain, snow, and night scenarios, alongside Dark Zurich [ 21 ] , which specifically isolates nighttime degradation, have driven significant progress. To tackle these domain shifts, existing paradigms typically rely on unsupervised domain adaptation (UDA) [ 27 , 11 ] , image-to-image style transfer [ 13 ] , or environment-specific architectural designs [ 29 ] . However, these approaches generally necessitate full-model retraining or require direct access to target-domain data. In contrast, our approach addresses environmental degradation by enhancing a frozen foundation model through a lightweight adapter. By relying entirely on synthetically augmented data and a unity-based physical engine, we eliminate the dependency on target-domain collection while preserving the zero-shot capabilities of the base architecture. However, these approaches generally necessitate full-model retraining or require direct access to target-domain data. In contrast, our approach addresses environmental degradation by enhancing a frozen foundation model through a lightweight adapter. We eliminate the dependency on costly real-world data collection under adverse lighting conditions, while maintaining both the computational efficiency and the generalization capabilities of the base architecture 3 Dataset Generation A main contribution of our approach is that segmentation models can be trained to be lighting-invariant, with structured pairwise training on clean and lighting-variant images that share identical ground-truth. This requires robust methods for