Robust Onion: Peeling Open Vocab Object Detectors Under Noise *Equal contribution work done as intern *Equal contribution
Priyank Pathak*, Mukilan Karuppasamy* †, Aaditya Baranwal, Shruti Vyas, and Yogesh S Rawat · 2026
Read Robust Onion: Peeling Open Vocab Object Detectors Under Noise *Equal contribution work done as intern *Equal contribution by Priyank Pathak*, Mukilan Karuppasamy* †, Aaditya Baranwal, Shruti Vyas, and Yogesh S Rawat (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26734.txt.
Introduction
Priyank Pathak, Mukilan Karuppasamy, and three colleagues dissect how real-world noise degrades open-vocabulary detectors. Such detectors find objects described in words, not just fixed categories. Their internal complexity hid where robustness fails. The study, called Robust Onion, applies controlled damage to images and peels the models layer by layer to watch features collapse.
Models sharing a vision backbone degraded alike, with features collapsing at similar layers. How the models were first trained mattered little by comparison. The image domain, not the labelling style, governed robustness. That explains why some benchmark data sets flatter detectors with large, isolated objects. The team validated fixes on driving, face, and drone data sets. A light plug-in method improved robustness while training vastly fewer parameters than full retraining.
Catalog record
| CREATOR | Priyank Pathak*, Mukilan Karuppasamy* †, Aaditya Baranwal, Shruti Vyas, and Yogesh S Rawat |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26734
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust Onion: Peeling Open Vocab Object Detectors Under Noise *Equal contribution work done as intern *Equal contribution Priyank Pathak*, Mukilan Karuppasamy* †, Aaditya Baranwal, Shruti Vyas, and Yogesh S Rawat Abstract The impact of real-world noise on O pen V ocabulary O bject D etectors (OV-ODs) remains poorly understood due to their architectural complexity. We present our comprehensive analysis, Robust Onion , an empirical study that uses controlled synthetic visual degradations to peel OV-ODs layer-by-layer, revealing how, why , and where robustness degrades, systematically analyzing feature collapse . Our findings reveal that models with similar vision backbones exhibit comparable robustness, driven by similar feature collapse at similar layers, while factors such as pretraining strategy, architectural nuances, and caption supervision contribute little. Robustness is primarily governed by the image domain rather than annotations, explaining the similar robustness impact on COCO and LVIS, and why datasets like ODinW-13 can give an impression of inflated robustness due to large, isolated objects. Finally, we validate our insights by improving robustness on real-world BDD-100K, WiderFace, and VisDRONE via our lightweight plug-and-play NN & TK0 approach, using 96 × \times fewer trainable parameters than end-to-end training. We also explain the prior works’ robustness observations. Figure 1 : Effect of Noise: GLIP [ 31 ] (above) & MM-GDINO [ 75 ] (bottom) performance on COCO [ 33 ] for noises like turbulence, pixelation, and motion blur. 1 Introduction Vision Language Models (VLMs) have shown strong generalization in tasks like image-to-text retrieval [ 52 , 46 ] , open-vocabulary classification [ 1 ] , image captioning [ 11 ] , visual-question-answering [ 25 ] etc . The ability to adapt without fine-tuning makes VLMs highly lucrative for applications where zero-shot is not just a convenience but a necessity. VLM based O pen V ocabulary O bject D etectors ( OV-ODs ) have gained attention for their easy adaptability in security [ 24 ] , medical imaging [ 69 ] , environmental monitoring [ 63 ] , and self-driving cars [ 59 ] . (a) Observable Collapse (b) Minimum collapse (c) Pixelation Severity Figure 2 : Real-World & Synthetic: GLIP-T synthetic COCO noisy features collapse against clean image aligns with BDD-100K real-world collapse. BDD-100K doesnt have explicit clean images, hence all noisy categories are in blue , except the highlighted . Real-world deployment of OV-ODs requires a critical understanding of their robustness against visual distortions and noise. However, their enormous complexity from various moving parts like vision backbone, text backbones, fusion network, box predictors, alignment networks etc. , makes them one of the most obscure deep learning models. Additionally, studying and isolating the true impact of low quality (LQ) noise requires matching high quality (HQ) counterpart images; and such well-annotated ‘noise-paired’ real-world datasets are nearly impossible to find. As a result, despite their growing use, the effect of out-of-distribution noise on VLM based OV-ODs remains largely underexplored. Addressing the gap in robustness against real-world distortions ( e.g . BDD-100K [ 68 ] ), our novel analysis, Robust Onion broadly categorizes noise-induced feature (variance) collapse [ 34 , 8 , 45 , 3 ] into two categories: 1. Observable ( Fig. 2(a) ): noisy features form distinct clusters separate from clean HQ features ( e.g . cloudy, overcast), 2. Minimal ( Fig. 2(b) ): little or no observable collapse ( e.g . snow, rain). By carefully tuning synthetic noises, we mimic these feature collapses, providing a practical proxy for real-world degradations ( e.g . turbulence for observable, motion blur for minimal). Increasing noise severity can change the collapse from minimal to observable ( Fig. 2(c) ). Robust Onion then empirically peels each component of OV-OD under these controlled noises. To answer: “How do visual distortions impact complex models such as OV-ODs?" and “What are the most effective directions to improve their robustness?" , We present an interpretability-driven analysis of robustness in OV-ODs. Rather than simply evaluating performance, our goal is to explain why failures occur, where they originate in the architecture, and how robustness can be improved. Our analysis revolves around five key questions: (1) Do bells and whistles like pretraining, finetuning, and architecture modules impact the robustness? (2) Which part of the model architecture (Backbone, FPN, RPN, etc) induces the robustness? (3) Are larger models inherently more robust, or are other factors decisive? (4) Is robustness solely determined by the model, or do input images play a role? (5) Does language prompting improve robustness under visual noise? Our analysis frames robustness as an explainable phenomenon rather than a black box outcome. We peel the layers of OV-ODs to reveal the following: • Vision Backbone drives robustness: Similar backbones exhibit comparable robustness due to resembling feature collapses at similar depths , regardless of bells and whistles of pretraining or overall architecture. • Layer-wise robustness: Shallow layers are adversely affected by the noise. Further, cross-exchanging the backbone layer can improve robustness. • Dataset bias: ODinW-13 can give a false impression of robustness because it features large and isolated objects. • Domain over annotation: Detecting on what (domain) matters more than detecting what (annotation), thus similar robustness of COCO & LVIS. • Minimal language influence: Once the visual features are degraded, language/captions contribute little to recover lost robustness. Each analysis includes [Takeaways & Model Design] highlighting key actionable insights for designing robust real-world OV-OD . We conclude by validating our analysis on real-world autonomous driving datasets like BDD-100K, DAWN, Foggy Cityscapes, and Virtual KITTI 2, WiderFace, and VisDRONE via our NN & TK0 approach, a lightweight technique to induce robustness in object detectors (open vocabulary). NN & TK0 uses 96 \times fewer trainable parameters than end-to-end training with similar robustness. Our analysis also explains several of the previously published robustness-related observations. 2 Related Work VLMs and OV-ODs: VLM generalizability [ 51 , 43 ] has progressed into object detection [ 57 , 19 , 71 ] . Advances in large-scale multimodal training, have further enhanced open-vocabulary object detectors (OV-OD) [ 2 , 60 , 9 , 42 , 74 ] . Versatility of OV-ODs [ 30 , 16 ] makes them ideal for real-world ( e.g . surveillance [ 15 ] , industrial-inspection [ 4 ] , satellite imagery [ 48 ] , autonomous-driving [ 59 ] , etc ; thus understanding their limitations [ 6 , 72 , 3 ] against noise is crucial. Robustness against Noise: Weather ( e.g . rain, fog, snow) & common artifacts ( e.g . jpg compression) pose significant challenges in object detection [ 39 , 50 , 13 , 73 , 67 ] . These distortions lose discriminative features, fundamentally affecting almost all models [ 58 ] . Robustness methods often use synthetic noises to improve on real-world tasks, like person re-id [ 47 ] , self-driving in fog/rain [ 21 ] . Despite the prevalence of noises, their impact on VLMs remains largely unexplored [ 12 , 32 ] , with most recently LR0.FM [ 45 ] analyzing robustness of VLMs in image classification. On the contrary, OV-ODs are far more complex than simple VLMs. Here, we present a comprehensive analysis of SOTA OV-ODs, revealing bottlenecks and critical factors in their robustness. More works in Table 3 . 3 Analysis Setup Models : We