Making AI Agents Evaluate Misleading Charts without Nudging

Swaroop Panda · 2026

Read Making AI Agents Evaluate Misleading Charts without Nudging by Swaroop Panda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05662.txt.

Introduction

AI agents are cheap stand ins for human testers in early chart evaluation, but Swaroop Panda asks whether they notice when a chart is designed to mislead. In an initial study, deliberately flawed charts were shown to an agent with no prompt to hunt for errors. The charts carried decorative clutter, manipulated axes, and distorted proportional cues.

Using two established rating scales for beauty and readability, the agent scored the charts. Ratings for aesthetic appeal and perceived readability often stayed high even when graphical integrity was compromised. The author concludes that unnudged agent evaluation can underweight defects tied to honesty, unless such checks are asked for directly. The single author study is a caution for teams that swap human reviewers for agents too early. It suggests integrity checks must be built into the evaluation task itself, not left for the agent to raise on its own.

Catalog record

CREATORSwaroop Panda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05662

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Making AI Agents Evaluate Misleading Charts without Nudging Swaroop Panda Abstract AI agents are increasingly used as low-cost proxies for early visualization evaluation. In an initial study of deliberately flawed charts, we test whether agents spontaneously penalise chart junk and misleading encodings without being prompted to look for errors. Using established scales (BeauVis and PREVis), the agent evaluated visualizations containing decorative clutter, manipulated axes, and distorted proportional cues. The ratings of aesthetic appeal and perceived readability often remained relatively high even when graphical integrity was compromised. These results suggest that un-nudged AI agent evaluation may underweight integrity-related defects unless such checks are explicitly elicited. 1 Introduction AI agents for usability testing [ 4 ] represent an emerging area within HCI-AI research [ 6 ] . These agents are designed to complement rather than replace traditional user evaluations, serving as a cost-effective preliminary assessment tool [ 5 ] before conducting comprehensive user studies. This approach enables teams to identify potential usability issues efficiently during early development stages. Shao et al. [ 8 ] find that “Agents can simulate ratings relatively aligned with human responses.” This alignment suggests that AI agents can provide meaningful preliminary insights into user experience quality before engaging actual participants. The capability to generate human-like evaluative responses enables design teams to iterate rapidly on interface concepts and identify critical usability concerns early in the development cycle. Furthermore, this preliminary screening can help researchers allocate their limited user testing resources more strategically by focusing human evaluation efforts on the most refined design iterations. Chart junk [ 3 ] refers to unnecessary decorative elements in visualizations that obscure data comprehension without adding informational value. Beyond mere aesthetic clutter, visualizations can deliberately mislead audiences through manipulated axes, truncated scales, inappropriate chart types, or distorted proportions. These deceptive practices exploit visual perception to misrepresent data relationships, magnify trivial differences, or obscure unfavorable trends. Such manipulations undermine data integrity and can significantly influence decision-making processes based on flawed interpretations. We investigate how AI agents evaluate visualizations containing chart junk through the application of established assessment scales. Our methodology does not explicitly inform the agents about the presence of decorative or misleading elements. Instead, we prompt agents to respond to scale-based evaluation questions, allowing us to observe whether they organically identify and account for these problematic design features in their assessments. We attempt to infer these organic identifications through systematic analysis of the scale results and response patterns. 2 Method We employ deliberately flawed visualizations in our study because their known deficiencies provide a controlled testing environment for evaluating AI agents’ critical assessment capabilities. Importantly, we do not inform the agents that these visualizations contain errors, nor do we explicitly prompt them to identify flaws. Instead, we assess these visualizations through established evaluation frameworks to observe how agents rate them organically. While Panda [ 7 ] deployed a similar methodology, our approach differs in that we do not provide user personas to the AI agents, allowing them to evaluate visualizations independently without predetermined contextual constraints. We utilize two validated evaluation instruments for this assessment: the BeauVis [ 2 ] scale and the PREVIS Scale [ 1 ] , which together enable systematic examination of how AI agents perceive and rate visualization quality across multiple dimensions. For the visualisations we randomly pick ten visualisations from the corpus provided by [ 3 ] . We used a single AI agent based on GPT-5.2, accessed via the ChatGPT Plus interface, to evaluate each visualization using the BeauVis and PREVis rating items. Across all the visualizations, the agent was not alerted to potential flaws and provided scale ratings in a single pass per visualization. 3 Results & Analysis Table 1: BeauVis aesthetic ratings of ten visualisations. Ratings use a seven point Likert scale from 0 (strongly disagree) to 6 (strongly agree). Visualisation Enjoyable Likable Pleasing Nice Appealing 1 3 4 4 4 4 2 3 3 3 3 3 3 2 2 2 2 2 4 4 4 4 4 4 5 5 5 5 5 5 6 3 3 3 3 3 7 2 2 2 2 2 8 6 6 6 6 6 9 2 2 2 2 2 10 3 3 3 3 3 Table 2: Understand and Layout scales of the PREVis instrument for ten visualisations (ratings 0–7; 0 = strongly disagree, 7 = strongly agree). Visualisation obvious represent understand messy crowd distract 1 2 2 2 1 1 2 2 6 6 6 6 6 6 3 4 4 4 5 4 4 4 6 6 6 6 6 6 5 6 6 6 6 6 6 6 5 5 5 6 6 5 7 1 1 1 0 0 0 8 5 5 5 6 5 6 9 3 3 3 2 2 2 10 5 5 5 6 6 6 Table 3: Reading data and Reading features scales of the PREVis instrument (ratings 0–7; 0 = strongly disagree, 7 = strongly agree). Visualisation find identify information visible see 1 2 2 2 0 0 2 7 6 6 5 5 3 4 4 4 3 3 4 7 6 6 6 6 5 6 6 6 4 4 6 5 5 5 3 3 7 0 1 1 0 0 8 5 5 5 4 4 9 2 2 2 2 2 10 6 6 6 5 5 On the BeauVis aesthetic ratings, the ten visualisations show a clear tiering. Visualisation 8 is a ceiling case (all items = 6), followed by Visualisation 5 (all = 5) and Visualisation 4 (all = 4), whereas Visualisations 3, 7, and 9 cluster at uniformly low ratings (all = 2). Across the dataset, responses are highly consistent across the five BeauVis descriptors, suggesting that the agent treated them as a largely unitary judgement of aesthetic appeal rather than as distinct dimensions. On PREVis (perceived readability), Visualisation 4 is the most consistently readable, with Visualisations 2, 5, and 10 forming a strong second tier across the subscales. Visualisation 5 is particularly informative: despite very high scores on Understand, Layout, and Reading Data, it is comparatively lower on Reading Features (mean = 4.0), indicating that extracting values may be easier than perceiving higher-level visual patterns (e.g., trends or salient deviations). Visualisations 6 and 8 show a similar, though less pronounced, “values > features” profile, while Visualisations 1 and especially 7 underperform across subscales, indicating difficulties both in interpreting the encoding (Understand) and in extracting information (Reading Data/Reading Features). Finally, the PREVis Layout items are typically framed in terms of the absence of clutter (e.g., not messy, not crowded, not distracting); accordingly, higher layout ratings should be interpreted as clearer layouts rather than greater messiness. 4 Discussion Although all ten visualizations contained known design flaws - ranging from chart-junk embellishments to integrity-relevant distortions (e.g., problematic encodings, axis treatments, or rhetorical framing) - the ratings illustrate how apparent quality can decouple from graphical integrity. BeauVis is explicitly an aesthetic instrument rather than a correctness check, and the clear tiering (e.g., Visualisation 8 > 5 > 4, with 3/7/9 low) shows that the agent produced coherent aesthetic judgements across the five descriptors even when the underlying visualization was flawed. PREVis likewise operationalises perceived readability; consequently, strong PREVis profiles (e.g.,

Read the complete text