Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models
Mateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh, Varun Jampani, Guha Balakrishnan · 2024
Read Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models by Mateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh, Varun Jampani, Guha Balakrishnan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19920.txt.
Introduction
A photograph taken from an odd angle can fool even strong vision models, and Mateusz Michalkiewicz and co-authors measure how unstable today's foundation models are to viewpoint. They define instability as large feature swings from tiny camera moves, and they hunt especially for accidental viewpoints, angles where an object's true shape hides. Nine leading models were probed using their internal features alone.
All models consistently registered accidental viewpoints, but they differed wildly on out-of-distribution angles, sometimes misclassifying by geometric resemblance. The team could sort viewpoints into stable, accidental, and alien classes from features without seeing the images. Tests on classification, question answering, and three-dimensional reconstruction showed the instability's cost. Robustness to viewpoint, they argue, deserves benchmark status.
Catalog record
| CREATOR | Mateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh, Varun Jampani, Guha Balakrishnan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models Mateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh, Varun Jampani, Guha Balakrishnan Abstract In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object’s true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object misclassifications based on geometric resemblance. Through quantitative and qualitative evaluations on three downstream tasks—classification, VQA, and 3D reconstruction—we illustrate the impact of viewpoint instability and underscore the importance of feature robustness across diverse viewing conditions. 1 Introduction A common assumption in visual reasoning is that scene content in the 3D world is independent of camera viewpoint. While implicitly assumed in classical reconstruction and rendering algorithms [ 31 ] , this assumption is not explicitly enforced in feature representation models now used in a range of computer vision applications. Popular foundation representation models, e.g., DINO [ 5 ] and CLIP [ 36 ] , simply leverage large datasets and self-supervised training objectives to transform image pixels into a different feature space amenable to semantic interpretation. Although effective, this approach can introduce generalization gaps, particularly related to 3D reasoning [ 16 , 46 ] and spatial context [ 51 ] , resulting in poor performance when faced with unusual viewpoints or atypical test cases. Hence, while the success of foundation models are impressive, they lack certain fundamental reasoning capabilities, such as content-viewpoint disentanglement. Figure 1 : Viewpoint stability of a featurizer. The blue camera represent small perturbations in camera space that result in correspondingly small changes in a feature space of a featurization function, indicating a stable viewpoint. In contrast, the red camera represent small perturbations in camera space that lead to large changes in feature space, signifying an unstable viewpoint. In the light of the above discussion, we take a deeper look in this study on how popular vision foundation models are affected by changes to viewpoint. In particular, we propose studying their viewpoint stability, or their rate of change of their output features with respect to changes in viewpoint. We define a model’s feature output from a certain viewpoint of a scene as unstable if minor changes in the viewing angle lead to significant change in the returned features (see Fig. 1 ). A robust representation should ideally exhibit minimal changes in high-level scene content with varying viewpoints. Viewpoint instability may be caused by several factors. As explored in a recent study named OmniView [ 41 ] , vision language models (VLMs) such as CLIP [ 37 ] and BLIP [ 25 ] struggle with out-of-distribution (OOD) viewpoints: object viewpoints that are rare in their training distributions. For instance, a mattress could be erroneously perceived as a closed laptop from a frontal viewpoint due to the abundance of closed laptops compared to mattresses in the training data from that particular viewpoint (see Fig. 7 ). However, instability is unavoidable in some cases, such as with accidental viewpoints, where a specific camera orientation can obscure the true 3D structure of an object [ 52 , 47 , 19 ] . In these cases, there is a fundamental geometric ambiguity that is not explicitly conveyed by featurizers. Understanding when a feature representation is unstable, and whether it coincides with known accidental viewpoints, can improve our grasp of deep learning models’ limitations, suggest pathways for enhancing their robustness, and use them in more responsible ways for downstream applications. Existing works with foundation models such as OmniView notably miss critical discussions on accidental viewpoints - a topic generally neglected since the foundational studies of the late 20th century [ 52 ] . In this study, we introduce a methodology for recognizing viewpoint instabilities directly from the output feature representations of foundational models, without the need to access the input images. This capability is particularly beneficial to model featurizer-specific instabilities, and to support privacy-sensitive applications where image access is restricted. We also propose a method to further classify unstable viewpoints into accidental versus other varieties from their features. Through extensive experiments on downstream tasks such as classification, VQA, and monocular 3D reconstruction, we illustrate the influence of viewpoint stability on model performance and identify specific conditions under which model robustness may falter due to viewpoint-related biases. Our findings first reveal that foundation models, regardless of their structural differences, exhibit a notable consistency in encoding accidental unstable viewpoints. However, their interpretations of OOD viewpoints are variable, which we attribute to inherent model biases. This underscores the need for robust feature representations to support stable performance across diverse viewing conditions. 2 Related Works Viewpoint-invariant reasoning is a fundamental challenge in computer vision. Classical algorithms to address viewpoint variability depend on explicit geometric constraints or multiple view representations in the image space [ 2 , 19 , 20 , 54 ] . Of particular relevance to our study are accidental viewpoints [ 52 , 19 , 47 ] , or unstable viewpoints caused by the camera orientation obscuring an object’s true 3D structure [ 52 ] , which present significant barriers to robust recognition. In contrast to classical approaches, this work forms the first comprehensive study of modern foundation models for viewpoint instability, rather than from images cues directly. 2.1 Viewpoint Biases in Neural Networks Deep neural network classifiers have been known to have viewpoint biases, i.e., performance gaps depending on viewpoint. Popular CNNs used for classification tasks exhibit a noticeable drop in performance for images taken from underrepresented viewpoints [ 1 , 29 ] . Other studies [ 57 , 14 ] investigated the biases of models towards canonical viewpoints and introduced datasets and techniques for evaluating viewpoint robustness, focusing on uncommon (but not explicitly accidental) viewpoints. Specifically, ViewFool [ 14 ] proposed a technique to generate adversarial viewpoints that push the models towards poor performance, effectively revealing their weaknesses in handling OOD viewpoints. Another study demonstrates predicting OOD viewpoint instabilities of deep networks and achieving viewpoint-invariant visual recognition through adversarial training [ 40 ] . Several other studies also propose network training paradigms [ 6 , 32 , 22 , 45 , 53 ] to encourage viewpoint-agnostic learning representations for specific applications. Though inspired by these studies, our work differs in several ways. First, we focus our attention on a range of modern foundation models rather than traditional classifiers. Second, we aim to predict viewpoint instability directly from features outputted by networks, which offers potential means to flag uncertain responses from these algorithms. Third, we provide a distinct emphasis on understanding how networks react to accidental viewpoints. 2.2 Robustness Analysis of Foundation Models Foundation models, such as CLIP [ 37 ] , DINOv2 [ 34 ] , and Llama [ 49 ] , are now