\systemname : A Visual Analytics Workflow for Subgroup-based Semantic Error Analysis of CVML Models
Jun Yuan, Kevin Miao, Heyin Oh, Isaac Walker, Zhouyang Xue, Tigran Katolikyan, Marco Cavallo · 2025
Read \systemname : A Visual Analytics Workflow for Subgroup-based Semantic Error Analysis of CVML Models by Jun Yuan, Kevin Miao, Heyin Oh, Isaac Walker, Zhouyang Xue, Tigran Katolikyan, Marco Cavallo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20112.txt.
Introduction
Jun Yuan and six co-authors build a tool for understanding model errors in twenty twenty-five. Error analysis is critical when vision and language models are deployed. One way to learn from errors is to find what failed examples share. With images, those shared traits are hard to see, and labels may not exist to guide the search.
Category-based analysis needs analysts to guess error causes in advance. Their workflow, VibE, works at subgroup level instead. It supports semantic error analysis without demanding hypotheses up front. Analysts can surface coherent subgroups of failures from unstructured data. Readers who ship computer vision models get a structured way to ask where, and for whom, the model goes wrong.
Catalog record
| CREATOR | Jun Yuan, Kevin Miao, Heyin Oh, Isaac Walker, Zhouyang Xue, Tigran Katolikyan, Marco Cavallo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20112
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
\systemname : A Visual Analytics Workflow for Subgroup-based Semantic Error Analysis of CVML Models Jun Yuan, Kevin Miao, Heyin Oh, Isaac Walker, Zhouyang Xue, Tigran Katolikyan, Marco Cavallo Abstract . Effective error analysis is critical for the successful development and deployment of CVML models. One approach to understanding model errors is to summarize the common characteristics of error samples. This can be particularly challenging in tasks that utilize unstructured, complex data such as images, where patterns are not always obvious. Another method is to analyze error distributions across pre-defined categories, which requires analysts to hypothesize about potential error causes in advance. Forming such hypotheses without access to explicit labels or annotations makes it difficult to isolate meaningful subgroups or patterns, however, as analysts must rely on manual inspection, prior expertise, or intuition. This lack of structured guidance can hinder a comprehensive understanding of where models fail. To address these challenges, we introduce VibE, a semantic error analysis workflow designed to identify where and why computer vision and machine learning (CVML) models fail at the subgroup level, even when labels or annotations are unavailable. VibE incorporates several core features to enhance error analysis: semantic subgroup generation, semantic summarization, candidate issue proposals, semantic concept search, and interactive subgroup analysis. By leveraging large foundation models (such as CLIP and GPT-4) alongside visual analytics, VibE enables developers to semantically interpret and analyze CVML model errors. This interactive workflow helps identify errors through subgroup discovery, supports hypothesis generation with auto-generated subgroup summaries and suggested issues, and allows hypothesis validation through semantic concept search and comparative analysis. Through three diverse CVML tasks and in-depth expert interviews, we demonstrate how VibE can assist error understanding and analysis. Conference: 30th International Conference on Intelligent User Interfaces; March 24–27, 2025; Cagliari, Italy 30th International Conference on Intelligent User Interfaces (IUI ’25), March 24–27, 2025, Cagliari, Italy DOI: 10.1145/3708359.3712147 ISBN: 979-8-4007-1306-4/25/03 CCS: Human-centered computing Visual analytics CCS: Computing methodologies Computer vision tasks Figure 1. \systemname enables CVML model developers to analyze the semantic distribution of model errors with the support of large foundation models. It offers two main pages: (A) the Data Overview page, where users can explore error samples and review system-generated clusters of semantically similar samples; and (B) the Subgroup Analysis page, which helps users generate and validate hypotheses about error-related semantic features. The user interface of \systemname{} to assist semantic error analysis of a 3d asset generation model. † † cc-license: by 1. Introduction In recent years, Computer Vision and Machine Learning (CVML) algorithms have advanced rapidly, transforming fields like healthcare, retail, and finance ( Szeliski, 2022 ) . As these models become more complex, ensuring their performance, reliability, and fairness at scale has increasingly become challenging. A critical aspect of evaluating these models is subgroup error analysis , examining model performance across specific subgroups—distinct subsets of data. Each subgroup is defined by common characteristics within the group, like demographics (e.g., age, gender, ethnicity) or pre-defined labels (e.g., image classes). The goal is to uncover biases or disparities in model performance that may not be apparent at the model level. This analysis ultimately helps uncover issues, such as overfitting to certain patterns, poor generalizability, or unfair outcomes in minority groups, which are key factors in improving model robustness and fairness. Traditional subgroup error analysis uses techniques such as confusion matrices and per-class accuracy ( Naidu et al., 2023 ) . However, as we discuss in our formative study (Sec. 3 ), these methods depend heavily on the presence of complete metadata, clean labels, and well-defined taxonomies. In the absence of such data, these conventional techniques frequently fail to capture the nuanced factors driving performance variations. This is particularly true when samples contain anomalies or unknown characteristics, so-called “aggressors”. For example, a dataset with demographic metadata might not necessarily contain features like tattoos or head coverings, which could have a big influence on the final model performance. This underscores the first major challenge: identifying and analyzing meaningful subgroups when traditional methods struggle with incomplete data or unexpected characteristics. Our study also highlights a second challenge: validating error hypotheses . Once a problematic subgroup is identified, determining whether the errors are specific to that subgroup or more widespread is difficult. Traditional workflows offer limited tools to explore correlations between subgroups and error patterns, making it hard to isolate the root causes of the model’s shortcomings. While methods such as TCAV ( Kim et al., 2018 ) and self-supervised visual concept segments ( He et al., 2022 ) are able to assist users in exploring data samples with specific concepts, they depend on pre-generated concepts and, in some cases, require access to model internals, which may not always be available for error analysis. To address these challenges, we introduce a more semantic approach to subgroup analysis, leveraging advances in self-supervised learning and large foundation models. Multimodal models like CLIP ( Radford et al., 2021 ) and BLIP ( Li et al., 2022 ) can extract rich, latent features from raw data, even without explicit labels. These models allow us to align visual and textual data, creating semantically meaningful subgroups, and enabling a more nuanced analysis of subgroup errors. We further leverage the state-of-the-art multimodal large language models (LLMs), such as GPT-4 ( GPT, [n. d.] ) , to assist users to understand the problematic subgroups and candidate issues with these subgroups. More specifically, our work builds on a formative study with seven CVML engineers (Sec. 6 ) and a hierarchical task analysis of their debugging workflows (Fig. 2 ). From this, we derived design goals to develop semantic-focused methodologies for subgroup analysis (Sec. 4 ). We introduce \systemname , an intelligent workflow powered by large foundation models that facilitates the identification of semantic subgroups, discovery of potential data issues, retrieval of relevant samples, and recommendation of similar subgroups for hypothesis validation. In Sec. 5 , we present the implementation of this workflow as a visual analytics system. This system supports developers in investigating where their models make errors, forming hypotheses about why these errors occur, and verifying whether the identified factors are statistically likely to contribute to the mistakes. We evaluate our approach through three case studies involving expert interviews with 3D asset generation, face reconstruction, and object detection models. Our findings show that foundation models enhance error analysis by providing meaningful data summaries and supporting semantic exploration. Additionally, visual analytics techniques help mitigate the limitations of foundation models, such as uncertainty or ambiguity in their outputs. We also discuss lessons learned and offer design guidelines for integrating foundation models into visual analytics systems while addressing our system’s limitations for real-world use. In summary, our contributions are: (1) an initial study with seven CVML engineers, which informed key design principles; (2) a novel analytic workflow that supports semantic subgroup identification and error validation; (3) the development of a visual analytics system implementing this workflow; (4) three case studies, with expert interviews, each focusing on a different CVML task; and (5) insights gained from integrating foundation models into a visual analytics system. 2. Background and Related Work In this work, we propose a data-centric visual analytics workflow that empowers model developers to formulate and validate hypotheses, leading to insights into model