Exploring Interaction Paradigms for LLM Agents in Scientific Visualization
Jackson Vonderhorst Univ. Notre Dame, Kuangshi Ai Univ. Notre Dame, Haichao Miao LLNL, Shusen Liu LLNL, Chaoli Wang Univ. Notre Dame · 2026
Read Exploring Interaction Paradigms for LLM Agents in Scientific Visualization by Jackson Vonderhorst Univ. Notre Dame, Kuangshi Ai Univ. Notre Dame, Haichao Miao LLNL, Shusen Liu LLNL, Chaoli Wang Univ. Notre Dame (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27996.txt.
Introduction
Vonderhorst and colleagues compare agent designs for turning plain-language requests into scientific visualisation workflows across fifteen benchmark tasks. General coding agents succeed most often but cost more, domain-specific agents run cheaper and steadier yet flex less, and computer-use agents handle single steps but falter on longer workflows. Persistent memory helps over repeated trials, though its value depends on interaction mode and feedback quality.
Catalog record
| CREATOR | Jackson Vonderhorst Univ. Notre Dame, Kuangshi Ai Univ. Notre Dame, Haichao Miao LLNL, Shusen Liu LLNL, Chaoli Wang Univ. Notre Dame |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29268 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27996
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Interaction Paradigms for LLM Agents in Scientific Visualization Jackson Vonderhorst Univ. Notre Dame, Kuangshi Ai Univ. Notre Dame, Haichao Miao LLNL, Shusen Liu LLNL, Chaoli Wang Univ. Notre Dame Abstract This paper examines how different types of large language model (LLM) agents perform on scientific visualization (SciVis) tasks, where users generate visualization workflows from natural-language instructions. We compare three primary interaction paradigms, including domain-specific agents with structured tool use, computer-use agents, and general-purpose coding agents, by evaluating eight representative agents across 15 benchmark tasks and measuring visualization quality, efficiency, robustness, and computational cost. We further analyze interaction modalities, including code scripts and model context protocol (MCP) or API calls for structured tool use, as well as command-line interfaces (CLI) and graphical user interfaces (GUI) for more general interaction, while additionally studying the effect of persistent memory in selected agents. The results reveal clear tradeoffs across paradigms and modalities. General-purpose coding agents achieve the highest task success rates but are computationally expensive, while domain-specific agents are more efficient and stable but less flexible. Computer-use agents perform well on individual steps but struggle with longer multi-step workflows, indicating that long-horizon planning is their primary limitation. Across both CLI- and GUI-based settings, persistent memory improves performance over repeated trials, although its benefits depend on the underlying interaction mode and the quality of feedback. These findings suggest that no single approach is sufficient, and future SciVis systems should combine structured tool use, interactive capabilities, and adaptive memory mechanisms to balance performance, robustness, and flexibility. \onlineid 0 \vgtccategory Research Introduction Scientific visualization (SciVis) tools such as ParaView [ 2 ] are essential for analyzing complex, high-dimensional data, yet they remain difficult to use, particularly for new users. Effective visualization often requires detailed knowledge of rendering pipelines, filter semantics, and scripting APIs, making even simple exploratory tasks time-consuming and error-prone. Recent advances in large language models (LLMs) have enabled a new generation of autonomous agents capable of translating natural language intent into visualization actions. Systems such as ChatVis [ 24 ] , ParaView-MCP [ 19 ] , and AVA [ 20 ] demonstrate that language-driven interfaces can reduce the cognitive burden of interacting with SciVis software. More broadly, multimodal agents now exhibit the ability to perceive visual interfaces, reason about task goals, and execute actions across GUIs designed for human use [ 21 , 32 ] . These developments suggest a paradigm shift from manual visualization workflows toward AI-assisted interaction. Despite this progress, current systems remain unreliable on realistic SciVis tasks, which often require multi-step reasoning, parameter tuning, and perceptual verification. Recent work has called for community collaboration to develop benchmarks that enable meaningful evaluation of agents for scientific data analysis and visualization [ 5 ] . SciVisAgentBench [ 3 ] addresses this need by providing representative tasks and evaluation protocols, which we adopt to compare interaction paradigms under shared conditions. Existing approaches to LLM-based SciVis agents can be organized along two orthogonal dimensions: agent paradigms (i.e., what capabilities the agent provides) and interaction modality (i.e., how the agent interfaces with the system). Agent paradigms capture the agent’s primary design objective, and prior work spans several representative paradigms. Domain-specific agents (e.g., ChatVis [ 24 ] , ParaView-MCP [ 19 ] ) operate directly on visualization APIs to achieve efficient and deterministic execution. Computer-use agents (e.g., UFO [ 34 , 33 , 35 ] , Open Interpreter [ 22 ] ) interact with existing software environments to enable flexible adaptation and error recovery. General-purpose coding agents (e.g., Codex CLI [ 23 ] , Claude Code [ 7 ] ) leverage broad programming capabilities to construct end-to-end workflows with minimal domain constraints. Additionally, some agents (e.g., Agent S [ 1 ] , Letta [ 18 ] ) further incorporate persistent memory and reflection to improve performance over repeated trials. Interaction modality, in contrast, describes how agents execute actions, including code scripts and model context protocol (MCP) or API calls for structured tool use, as well as command-line interfaces (CLI) and graphical user interfaces (GUI) for more general interaction. Interaction modalities are orthogonal to agent paradigms: domain-specific agents primarily use scripts or APIs, computer-use agents operate through GUIs, and general-purpose agents typically interact via CLI. While these paradigms and interaction modalities offer complementary strengths, their relative effectiveness and tradeoffs for SciVis remain poorly understood. In this paper, we present a comparative study of interaction paradigms for LLM-driven SciVis agents. Rather than proposing a new agent architecture, we focus on evaluating representative systems under realistic tasks derived from SciVisAgentBench. Our goal is to understand why different paradigms succeed or fail, and what design principles may guide future visualization agents. Specifically, this work makes the following contributions: • We provide one of the first side-by-side analyses of domain-specific tool-use, GUI-based computer-use, and CLI-based coding agents on realistic SciVis tasks, along with an analysis of the impact of persistent memory and adaptive learning. • We identify key behavioral tradeoffs, including determinism vs. robustness and efficiency vs. adaptability, that shape agent effectiveness. • We adopt a task-driven evaluation that measures success, consistency, failure modes, and recovery behaviors. • We derive design insights suggesting that hybrid agents integrating deterministic execution, perceptual grounding, and adaptive learning may outperform any single paradigm. 1 Related Work Recent SciVis research increasingly treats LLMs as agentic systems that execute workflows rather than serve as passive interfaces [ 14 ] . Existing approaches can be organized by agent paradigms, alongside differences in interaction modality and adaptation capability. Domain-specific agents translate natural language into structured API calls within visualization systems, as seen in ChatVis [ 24 ] , ParaView-MCP [ 19 ] , and InferA [ 26 ] . These systems enable efficient, reproducible execution. However, their tight coupling to predefined tools limits robustness in the face of ambiguous intent or unforeseen states. Computer-use agents interact directly with software interfaces via perception and low-level actions. Domain-specific systems such as AVA [ 20 ] , NLI4VolVis [ 4 ] , and TexGS-VolVis [ 27 ] also incorporate visual feedback for iterative refinement, while general frameworks including UFO series [ 34 , 33 , 35 ] and Anthropic’s computer-use [ 6 ] enable flexible adaptation across applications. This flexibility improves error recovery but incurs higher computational cost and variability. Recent work further highlights that effective human oversight in such agents depends on how supervision is structured, rather than simply increasing user control [ 11 ] . Persistent memory and reflection mechanisms can improve performance over time, as in Agent-S [ 1 ] and OS-Copilot [ 28 ] . In SciVis, VizGenie [ 9 ] demonstrates reusable visualization skills. Broader studies emphasize the importance of learning in long-horizon tasks [ 16 , 25 ] , though evaluation remains difficult due to delayed and noisy feedback. General-purpose coding agents such as Codex [ 23 ] and Claude Code [ 7 ] leverage broad programming capabilities to construct workflows end-to-end. While flexible, they often incur higher costs and lack the structured guarantees of domain-specific systems. Evaluating agent behavior remains challenging. Prior work considers execution success, perceptual similarity, and user studies. CoDA [ 13 ] , MatPlotBench [ 31 ] , and VisEval [ 12 ] propose complementary metrics, while benchmarks such as VisualWebArena [ 17