An Empirical Study of Bugs in Data Visualization Libraries DOI: 10.1145/3729363 Journal: PACMSE Volume: 2 FSE FSE093 7 CCS: General and reference Empirical studies CCS: Software and its engineering Software libraries and repositories CCS: Software and its engineering Software testing and debugging
Weiqi Lu, Yongqiang Tian, Xiaohan Zhong, Haoyang Ma, Zhenyang Xu, Shing-Chi Cheung, Chengnian Sun · 2025
Read An Empirical Study of Bugs in Data Visualization Libraries DOI: 10.1145/3729363 Journal: PACMSE Volume: 2 FSE FSE093 7 CCS: General and reference Empirical studies CCS: Software and its engineering Software libraries and repositories CCS: Software and its engineering Software testing and debugging by Weiqi Lu, Yongqiang Tian, Xiaohan Zhong, Haoyang Ma, Zhenyang Xu, Shing-Chi Cheung, Chengnian Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15084.txt.
Introduction
Weiqi Lu and six co-authors report, in a 2025 study, the first broad look at bugs in data visualization libraries. These libraries turn data into charts, so a bug rarely crashes a program. Instead, it can draw a wrong chart that misleads the reader and skews choices. The team wanted to know what such bugs look like and what causes them.
They gathered 564 bugs from five widely used libraries and sorted them by symptom and root cause. Wrong or imprecise plots were common, and faulty graphic math was the top cause. They also list eight steps that can trigger such bugs and two checks that suit chart testing. A side test asked whether vision-language models can spot bad plots.
Catalog record
| CREATOR | Weiqi Lu, Yongqiang Tian, Xiaohan Zhong, Haoyang Ma, Zhenyang Xu, Shing-Chi Cheung, Chengnian Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15084
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Study of Bugs in Data Visualization Libraries DOI: 10.1145/3729363 Journal: PACMSE Volume: 2 FSE FSE093 7 CCS: General and reference Empirical studies CCS: Software and its engineering Software libraries and repositories CCS: Software and its engineering Software testing and debugging Weiqi Lu, Yongqiang Tian, Xiaohan Zhong, Haoyang Ma, Zhenyang Xu, Shing-Chi Cheung, Chengnian Sun Abstract . Data visualization (DataViz) libraries play a crucial role in presentation, data analysis, and application development, underscoring the importance of their accuracy in transforming data into visual representations. Incorrect visualizations can adversely impact user experience, distort information conveyance, and influence user perception and decision-making processes. Visual bugs in these libraries can be particularly insidious as they may not cause obvious errors like crashes, but instead mislead users of the underlying data graphically, resulting in wrong decision making. Consequently, a good understanding of the unique characteristics of bugs in DataViz libraries is essential for researchers and developers to detect and fix bugs in DataViz libraries. This study presents the first comprehensive analysis of bugs in DataViz libraries, examining 564 bugs collected from five widely-used libraries. Our study systematically analyzes their symptoms and root causes, and provides a detailed taxonomy. We found that incorrect/inaccurate plots are pervasive in DataViz libraries and incorrect graphic computation is the major root cause, which necessitates further automated testing methods for DataViz libraries. Moreover, we identified eight key steps to trigger such bugs and two test oracles specific to DataViz libraries, which may inspire future research in designing effective automated testing techniques. Furthermore, with the recent advancements in Vision Language Models (VLMs), we explored the feasibility of applying these models to detect incorrect/inaccurate plots. The results show that the effectiveness of VLMs in bug detection varies from 29% to 57%, depending on the prompts, and adding more information in prompts does not necessarily increase the effectiveness. Our findings offer valuable insights into the nature and patterns of bugs in DataViz libraries, providing a foundation for developers and researchers to improve library reliability, and ultimately benefit more accurate and reliable data visualizations across various domains. † † cc-license: by 1. Introduction Data visualization (DataViz) is a representation technique employing various visual encodings, such as size, angle, color, and shape, to convey information derived from data effectively. This technique is extensively utilized across multiple domains, including presentation ( Kosara 2016 ; Knaflic 2015 ) , data analysis ( Kehrer and Hauser 2013 ; Clarinval and Dumas 2022 ; Wu et al. 2024 ) , and web applications ( Harrison et al. 2018 ; Burger et al. 2013 ; Patterson 2016 ) . To facilitate the transformation of raw data into meaningful graphic representations, various DataViz libraries have been developed, such as matplotlib ( matpltlib 2023 ) , ggplot2 ( ggplot 2023 ) , and Chart.js ( Chart.js 2023 ) . These libraries are sophisticated visualization tools that offer a wide range of functionalities, including the capability to create different kinds of charts, 2D and 3D plots, images, and their combinations. These tools are designed to accurately depict insights, such as patterns and trends, extracted from raw data. Despite the popularity of these DataViz libraries, their inherent complexity often leads to the emergence of bugs, which pose significant threats to both functional correctness and user experience. These bugs are generally identified by users and subsequently reported in the respective repositories of the libraries. A considerable number of these bugs can lead to incorrect functionalities, resulting in the misrepresentation of data. Such inaccuracies may foster false perceptions of the presented information, potentially leading to erroneous decision-making ( Nguyen et al. 2021 ) . fig. 1 shows an example of data misrepresentation arising from a real bug in ggplot2. Users invoke the ggplot2 APIs to draw a hexagonal heatmap to show a two-dimensional distribution of 53,940 diamonds, with price in US dollars and carat weight as the dimensions of interest. This example demonstrates a typical use of DataViz libraries to visualize complex datasets that are difficult to interpret in text. However, the ggplot2 bug uses incorrect values to update the graphic parameters ( e.g. , colors) of hexagons. The resulting plot ( fig. 1(a) ) incorrectly colors many deep blue hexagons as light blues. After the bug is fixed, the data are correctly visualized in fig. 1(b) , which contains only five light blue hexagons. This bug results in a misleading graphic representation of the relationship between diamond’s price and carat. The diamonds dataset is large, comprising 53,940 entries. Users may not easily notice the anomaly, leading to incorrect data interpretation, analysis, and decision-making. For instance, diamond sellers may conclude from fig. 1(a) that diamonds ranging from 0.5 to 2.5 carats and priced between 5,000 and 7,500 are the most popular among consumers, suggesting a business strategy to prioritize importing these diamonds. However, the conclusion is incorrect and can lead to misguided inventory decisions and financial losses. (a) Faulty Figure (b) Figure After Bug Fix Figure 1 . An Illustrative Example of Data Misrepresentation (ggplot2 Issue #5037 ( git 2022f ) ). Some hexagons in the heatmap are incorrectly colored, leading to the misrepresentation of the two-dimensional distribution. Data Misrepresentation https://github.com/tidyverse/ggplot2/issues/5037 The example above underscores the severe consequences that bugs in DataViz libraries can cause, highlighting the need for an empirical study to learn how these bugs can be better detected and prevented. DataViz libraries are distinct from other software libraries in their support of comprehensive data conversion into visual representations, requiring full integration of data processing, graphic computations, and rendering. Therefore, the insights from prior bug characterization studies in other domains ( Sun et al. 2016 ; Humbatova et al. 2020 ; Shen et al. 2021 ) are mostly inapplicable, necessitating a focused investigation into the specific nature of bugs in DataViz libraries. Existing research on DataViz libraries has primarily addressed data misrepresentation or misleading visualizations resulting from improper usage, with efforts focused on identifying the characteristics and impacts of such flawed visualizations ( Barcellos et al. 2018 ; Szafir 2018 ) , promoting best practices ( Midway 2020 ) , and enhancing the design of visualization tools to minimize syntactic errors and improve expressive flexibility ( Wilkinson 2012 ; Satyanarayan et al. 2014 ; Walny et al. 2019 ) . However, no prior studies examine the data misrepresentation caused by bugs in DataViz libraries . Inspired by the bug characterization studies in other software domains ( Shen et al. 2021 ; Guan et al. 2023a ; Zhang et al. 2018 ; Liu et al. 2014 ; Liu et al. 2021 ) , we study whether DataViz library bugs exhibit specific characteristics that facilitate their better detection and prevention by tool automation. We gathered a dataset comprising 564 bugs reported across the five most widely used open-source DataViz libraries from September 2021 to August 2023. We carefully reviewed the associated bug reports, discussions between users and developers, patches, and test cases to identify the symptoms, root causes, and correlations between these factors, and make suggestions to prevent such bugs. Additionally, we analyzed the key steps necessary to reproduce the bugs in the reports and classify the types of test oracles employed. This comprehensive investigation aims to deepen our understanding of effective, efficient, and varied