Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition

Sayanta Adhikari, Rishav Kumar, Konda Reddy Mopuri, Rajalakshmi Pachamuthu · 2024

Read Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition by Sayanta Adhikari, Rishav Kumar, Konda Reddy Mopuri, Rajalakshmi Pachamuthu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02833.txt.

Description

A 2024 arXiv research paper by Sayanta Adhikari, Rishav Kumar, Konda Reddy Mopuri, Rajalakshmi Pachamuthu. Contextual information plays a critical role in object recognition models within computer vision, where changes in context can significantly affect accuracy, underscoring models’ dependence on contextual cues. This study investigates how context manipulation influences both model accuracy and feature attribution, providing insights into the reliance of object recognition models on contextual information as understood through the lens of feature attribution methods.

Catalog record

CREATORSayanta Adhikari, Rishav Kumar, Konda Reddy Mopuri, Rajalakshmi Pachamuthu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition Sayanta Adhikari, Rishav Kumar, Konda Reddy Mopuri, Rajalakshmi Pachamuthu Abstract . Contextual information plays a critical role in object recognition models within computer vision, where changes in context can significantly affect accuracy, underscoring models’ dependence on contextual cues. This study investigates how context manipulation influences both model accuracy and feature attribution, providing insights into the reliance of object recognition models on contextual information as understood through the lens of feature attribution methods. We employ a range of feature attribution techniques to decipher the reliance of deep neural networks on context in object recognition tasks. Using the ImageNet-9 and our curated ImageNet-CS datasets, we conduct experiments to evaluate the impact of contextual variations, analyzed through feature attribution methods. Our findings reveal several key insights: (a) Correctly classified images predominantly emphasize object volume attribution over context volume attribution. (b) The dependence on context remains relatively stable across different context modifications, irrespective of classification accuracy. (c) Context change exerts a more pronounced effect on model performance than Context perturbations. (d) Surprisingly, context attribution in ‘no-information’ scenarios is non-trivial. Our research moves beyond traditional methods by assessing the implications of broad-level modifications on object recognition, either in the object or its context. Code available at https://github.com/nineRishav/Lost-In-Context Conference: Indian Conference on Computer Vision Graphics and Image Processing; December 13–15, 2024; Bengaluru, India Indian Conference on Computer Vision Graphics and Image Processing (ICVGIP 2024), December 13–15, 2024, Bengaluru, India DOI: 10.1145/3702250.3702254 ISBN: 979-8-4007-1075-9/24/12 CCS: Information systems Digital libraries and archives CCS: Computing methodologies Object detection 1. Introduction In Computer Vision, Context refers to information or signals not part of an object’s appearance, including visual scenes and other objects. It encompasses various types of information, such as the presence and relationships of co-occurring objects, the overall scene type, ambient lighting conditions, and spatial arrangements, all of which contribute to a comprehensive understanding of the visual scene. Context plays a vital role in humans’ ability to comprehend scene information. In situations where something is unclear, humans rely on contextual cues to understand the scene, such as when driving in foggy or rainy conditions, where visibility is low, but contextual information like the road layout and surrounding vehicles helps navigate safely. Figure 1. The predictions emphasize contextual information, while feature attribution for the top predicted class highlights the object in the image. Predictions were generated using a pre-trained ResNet50 model, and feature attribution was performed with GradCAM . An image showing the predictions emphasizing contextual information with feature attribution highlighting the object in the image. The visual context of an object is an important source of information for recognition tasks in both human and computer vision. Understanding the importance of context in computer vision has become a significant field of study. One of the primary tasks in computer vision is Object Recognition, where deep learning models tend to surpass human-level performance ( He et al., 2015 ) , making it an ideal area to study the effects of context as perceived by these models. With the introduction of Deep Neural Networks (DNNs) in computer vision, their utility has grown exponentially. DNN models have surpassed traditional methods in various applications, including autonomous vehicles ( Huang et al., 2020 ; González et al., 2016 ) , medical imaging ( Litjens et al., 2017 ; Patrício et al., 2023 ) , face recognition ( Schroff et al., 2015 ) , and motion detection ( Wang et al., 2016 ) . In most cases, DNN models are used as black box models ( Adadi and Berrada, 2018 ) , with users primarily concerned about their performance. However, understanding how or why a model provides certain outputs is crucial for critical applications, such as medical imaging ( Samek et al., 2017 ) . For example, in Figure 1 (top row), feature attribution focuses on the object, a sofa, but the model’s prediction is influenced by the context, i.e., lakeside, coast. These concerns have led to the introduction of explainability in machine learning and deep learning models. Many works ( Adebayo et al., 2020 ; Srinivas and Fleuret, 2019 ; Yang and Kim, 2019 ; Alsallakh et al., 2021 ) have been conducted in the visual explanation domain for computer vision applications. These visual explanation methods provide the regions the model looks at to generate a particular output prediction. Some past works show the effect of context on the performance of models. For instance, Xiao et al. ( Xiao et al., 2021 ) have shown how context, or image background, plays a major role in the performance of models. Many past works ( Chen et al., 2016 ; Divvala et al., 2009 ; Galleguillos et al., 2008 ; Mottaghi et al., 2014 ; Wang and Zhu, 2023 ) have shown how context helps provide information in different ways to improve the model’s accuracy. This paper provides insights into the effect of contextual information on the model’s prediction from the perspective of feature attribution methods. We extend this view by proposing methods and metrics to understand context effects using explainability methods. Explainability methods allow us to identify important areas, making it natural to use these methods to understand better how models perceive contextual information ( Patrício et al., 2023 ) . We have tried to interpret context effects such as context change and context perturbation from an explainability point of view and provide deeper insights using ImageNet-9 and ImageNet-CS (Sec 4.1.2 ), where perturbation is added to the context. In this paper, we propose a simple metric to compute the importance of context as provided by popular feature attribution methods in the literature. The major contributions of this paper are: • We propose a novel approach to understand the effect of context on object recognition models through feature attribution methods. By leveraging feature attribution maps and segmentation maps, we introduce a new metric for quantitatively analyzing the impact of context (Section 4.2 ). • Our experimental results demonstrate that context change significantly affects model performance more than context perturbation (Section 5.1 ). Additionally, we show that models trained on larger datasets exhibit reduced reliance on contextual pixels (Section 5.2 ). • Our study on object size reveals that contrary to expectations, context attribution does not vary significantly between larger and smaller objects, indicating that object size alone does not predict context dependence. This challenges the assumption that more context inherently increases context reliance, highlighting the complexity of these interactions (Section 5.4 ). • We reveal that misclassification is closely linked to context change or perturbation. Furthermore, our experiments indicate that ‘no information’ contexts tend to attract higher attribution than expected, highlighting the nuances of context influence (Section 5.5 ). 2. Preliminaries 2.1. Models In Computer Vision, convolutional neural networks (CNNs) are fundamental, serving as the foundational components in prominent DNN models across diverse applications. The literature has witnessed significant advancements, leading to the introduction of various architectural designs that facilitate the extraction of discriminative features for accurate classification. Among these architectures, InceptionNet ( Szegedy et al., 2015 ) , VGG ( Simonyan and Zisserman, 2014 ) , ResNet ( He et al., 2016 ) , and EfficientNet ( Tan and Le, 2019

Read the complete text