Exploiting Contextual Uncertainty of Visual Data for Efficient Training of Deep Models DOI: 10.475/123_4 ISBN: 123-4567-24-567/08/06 Conference: Indian Conference on Computer Vision, Graphics and Image Processing; December 2024; Bangalore, India
Sharat Agarwal sharat29ag.github.io · 2024
Read Exploiting Contextual Uncertainty of Visual Data for Efficient Training of Deep Models DOI: 10.475/123_4 ISBN: 123-4567-24-567/08/06 Conference: Indian Conference on Computer Vision, Graphics and Image Processing; December 2024; Bangalore, India by Sharat Agarwal sharat29ag.github.io (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01925.txt.
Description
Abstract . Objects, in the real world, rarely occur in isolation and exhibit typical arrangements governed by their independent utility, and their expected interaction with humans and other objects in the context. For example, a chair is expected near a table, and a computer is expected on top.
Catalog record
| CREATOR | Sharat Agarwal sharat29ag.github.io |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-01925
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploiting Contextual Uncertainty of Visual Data for Efficient Training of Deep Models DOI: 10.475/123_4 ISBN: 123-4567-24-567/08/06 Conference: Indian Conference on Computer Vision, Graphics and Image Processing; December 2024; Bangalore, India Sharat Agarwal sharat29ag.github.io Abstract . Objects, in the real world, rarely occur in isolation and exhibit typical arrangements governed by their independent utility, and their expected interaction with humans and other objects in the context. For example, a chair is expected near a table, and a computer is expected on top. Humans use this spatial context and relative placement as an important cue for visual recognition in case of ambiguities. Similar to human’s, DNN’s exploit contextual information from data to learn representations. Our research focuses on harnessing the contextual aspects of visual data to optimize data annotation and enhance the training of deep networks. Our contributions can be summarized as follows: (1) We introduce the notion of contextual diversity for active learning CDAL ( Agarwal et al. 2020 ) and show its applicability in three different visual tasks semantic segmentation, object detection and image classification, (2) We propose a data repair algorithm ( Agarwal et al. 2022 ) to curate contextually fair data to reduce model bias, enabling the model to detect objects out of their obvious context, (3) We propose Class-based annotation ( Agarwal et al. 2023 ) , where contextually relevant classes are selected that are complementary for model training under domain shift. Understanding the importance of well-curated data, we also emphasize the necessity of involving humans in the loop to achieve accurate annotations and to develop novel interaction strategies that allow humans to serve as fact-checkers. In line with this we are working on developing image retrieval system for wildlife camera trap images and reliable warning system for poor quality rural roads. For large-scale annotation, we are employing a strategic combination of human expertise and zero-shot models, while also integrating human input at various stages for continuous feedback. 1. Introduction “For me context is the key - from that comes the understanding of everything.” - Kenneth Noland Objects in the actual world exhibit typical arrangements, giving information of their interaction with other objects and the context of the overall scene. Humans use this spatial context as an essential cue for visual recognition in natural settings ( Biederman et al. 1982 ) . When we look at a complex scene, we perceive it effortlessly and identify the objects even without recognizing them. Context of the objects in the real world helps us solve perceptual inference faster and more accurately. It’s generally observed that objects appearing in a consistent or familiar background are detected more accurately than objects in an inconsistent environment. For instance, we perceive the round disc-like object on the dining table from a distance as a diner plate. Spatial context is an essential factor in facilitating scene understanding and object recognition for both machines and humans. Similar to how humans learn concepts and objects by observing their surroundings, deep learning models leverage large amounts of data, often annotated with labels, to learn diverse representations that can generalize to unseen data. One of the reasons why deep neural networks have done exceptionally well in the past decade is the availability of large diverse datasets. While data is crucial for deep learning, it is often overlooked in a learning setting. The more quality data a model is exposed to, the better it can capture intricate patterns, relationships, and nuances, leading to higher accuracy and robustness. While this works in theory, the sheer scale of data for practical applications comes at a high labor cost to label, especially in very specialized fields like medical or autonomous driving domains where the cost of running simulations to produce ground truth is very expensive. Possible algorithmic solutions like active learning comes to let the algorithm iteratively pick most informative data examples to be labeled from unlabeled datasets in a manner such that it is representative of the underlying data distribution to a near-optimal learner ( Settles 2012 ) . Traditional AL techniques ( Lewis and Catlett 1994 ; Lewis and Gale 1994 ; Joshi et al. 2009 ) have mostly been based on uncertainty and have exploited the ambiguity in the predicted output of a model. Existing approaches that leverage these cues are still insufficient in adequately capturing the spatial and semantic context within an image and across the selected set. Considering the critical role that data plays in model training, “we argue that along with the quantity of data, the quality of data needs attention.” To this end, we propose that model training should be designed so that models are trained using contextually diverse data to ensure they are accurate and unbiased while being efficiently trained. We thus investigate different aspects of contextual information from the available data and need of human in the loop for efficient annotation. 2. Progress and Results As motivated in previous section contextual information is a crucial part for humans visual understanding and inspired deep networks. In this section, we focus on the main contribution of this manuscript, discussing the importance of contextual information in selecting data for visual recognition in different applications. Contextual Diversity for Active Learning ( Agarwal et al. 2020 ) , ECCV20 : State of the art Active Learning approaches typically rely on measures of visual diversity or prediction uncertainty, which are unable to effectively capture the variations in spatial context. On the other hand, modern CNN architectures make heavy use of spatial context for achieving highly accurate predictions. Since the context is difficult to evaluate in the absence of ground-truth labels, we introduce the notion of contextual diversity that captures the confusion associated with spatially co-occurring classes. Contextual Diversity (CD) hinges on a crucial observation that the probability vector predicted by a CNN for a region of interest typically contains information from a larger receptive field. Such a measure would help select a training set that is diverse enough to cover a variety of object classes and their spatial co-occurrence and thus improve generalization of CNNs. The objective of this paper was to achieve this goal by designing a novel measure for active learning which helps select frames having objects in diverse contexts and background. Contextually Fair Data To Reduce Model Bias ( Agarwal et al. 2022 ) , WACV22 : Co-occurrence bias in the training dataset may hamper a DNN model’s generalizability to unseen scenarios in the real world. For example, in COCO ( Lin et al. 2014 ) dataset, many object categories have a much higher co-occurrence with men compared to women, which can bias a DNN’s prediction in favor of men. Recent works have focused on task specific training strategies to handle bias in such scenarios, but fixing the available data is often ignored. We propose a novel and more generic solution to address the contextual bias in the datasets by selecting a subset of the samples, which is fair in terms of the co-occurrence with various classes for a protected attribute. We introduce a data repair algorithm using the coefficient of variation, which can curate fair and contextually balanced data for a protected class(es). This helps in training a fair model irrespective of the task, architecture or