See or Say Graphs: Agent-Driven Scalable Graph Understanding with Vision-Language Models
Shuo Han, Yukun Cao, Zezhong Ding, Zengyi Gao, S. Kevin Zhou, Xike Xie · 2025
Read See or Say Graphs: Agent-Driven Scalable Graph Understanding with Vision-Language Models by Shuo Han, Yukun Cao, Zezhong Ding, Zengyi Gao, S. Kevin Zhou, Xike Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16769.txt.
Introduction
Shuo Han and five co-authors present GraphVista, a framework that helps vision-language models understand the structure of very large graphs. Graphs encode networks of nodes and links. Language models read them poorly at scale because long inputs overflow token limits, and text and image views of a graph are hard to combine.
GraphVista stores graph facts in a light retrieval base and pulls only the text and high-detail subgraphs a task needs. A planning agent then routes each sub-task to text or image, whichever fits best. Experiments show the system handles graphs far larger than older benchmarks and beats text-only, image-only, and fusion methods. The design targets tasks where both global facts and local shape matter.
Catalog record
| CREATOR | Shuo Han, Yukun Cao, Zezhong Ding, Zengyi Gao, S. Kevin Zhou, Xike Xie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16769
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
See or Say Graphs: Agent-Driven Scalable Graph Understanding with Vision-Language Models Shuo Han, Yukun Cao, Zezhong Ding, Zengyi Gao, S. Kevin Zhou, Xike Xie Abstract Vision-language models (VLMs) have shown promise in graph understanding, but remain limited by input-token constraints, facing scalability bottlenecks and lacking effective mechanisms to coordinate textual and visual modalities. To address these challenges, we propose GraphVista, a unified framework that enhances both scalability and modality coordination in graph understanding. For scalability, GraphVista organizes graph information hierarchically into a lightweight GraphRAG base, which retrieves only task-relevant textual descriptions and high-resolution visual subgraphs, compressing redundant context while preserving key reasoning elements. For modality coordination, GraphVista introduces a planning agent that routes tasks to the most suitable modality—using the text modality for simple property reasoning and the visual modality for local and structurally complex reasoning grounded in explicit topology. Extensive experiments demonstrate that GraphVista scales to large graphs, up to 200 × \times larger than those used in existing benchmarks, and consistently outperforms existing textual, visual, and fusion-based methods, achieving up to 4.4 × \times quality improvement over the state-of-the-art baselines by fully exploiting the complementary strengths of both modalities. * * footnotetext: These authors contributed equally to this work. † † footnotetext: Corresponding author. 1 Introduction Recently, vision-language models (VLMs) have shown potential for general-purpose understanding of graph data Tang et al. 2024a ; Kong et al. 2025b ; Ding et al. 2025b , offering new paradigms for solving real-world problems naturally represented as graphs. A central research focus of this emerging field is to enhance the foundational capability of VLMs to understand graph structures Ren et al. 2024 , particularly for large-scale graphs, without relying on external tools such as code execution or software systems Ding et al. 2025a ; Feng et al. 2025 . To this end, existing studies have explored two primary graph input modalities for VLMs: textual graph descriptions 1 1 1 Many early studies on text-modality-based graph understanding were developed on LLMs. As VLMs extend LLMs with visual capabilities, this paper presents a unified discussion from the perspective of graph understanding with VLMs. , which convert structural information into natural language for indirect memory and reasoning over node properties and relations Jiang et al. 2023 ; Jin et al. 2024 ; Zhang et al. 2024a ; and visual graph representations, which render nodes and edges as structured images to support direct structural perception Li et al. 2024c ; Zhu et al. 2025 . More recent efforts further combine these modalities to improve structural understanding and complex reasoning on graphs Wei et al. 2024 . Figure 1: Comparison of textual and visual modalities across different types of graph understanding tasks However, despite these advances, current VLMs still lack a unified framework that systematically addresses the scalability challenges and the difficulties in coordinating textual and visual modalities inherent in graph understanding. Specifically: Scalability remains a fundamental limitation for both textual and visual graph inputs in VLMs. Given their limited input token capacities, textual graph descriptions are constrained by sequence length, making it difficult to encode the full structure of large graphs Chen et al. 2024a ; Yuan et al. 2025a . Visual graph inputs, on the other hand, are restricted by image resolution, resulting in the loss of structural details as graph size increases Zhu et al. 2025 . In addition, VLMs face the lack of an effective mechanism to coordinate different input modalities for graph understanding, which prevents them from fully leveraging the distinct strengths and complementary advantages of textual and visual modalities. As illustrated in Figure 1 , under the token length limits of VLMs, the text modality is more effective for simple property tasks such as counting nodes, measuring degrees, and assessing connectivity. In these cases, the information can be directly located and reasoned over with minimal effort. By contrast, the visual modality often performs worse because the limited resolution of global visual graphs blurs structural details, causing VLMs to misidentify relevant elements and leading to performance degradation. For tasks that require local and structurally complex reasoning, such as shortest-path computation or cycle detection, high-resolution local visual subgraphs offer a clear advantage for VLMs. They provide a “what-you-see-is-what-you-get” view of the topology and naturally support stepwise reasoning. In contrast, textual descriptions are often lengthy and lack structural clarity, leaving VLMs with only indirect language cues and thus little structural grounding for effective complex graph reasoning. Therefore, a natural conclusion is that realizing the full potential of VLMs in graph understanding requires addressing these two challenges. This entails, on the one hand, preserving as much task-relevant graph information as possible within limited input tokens, and on the other hand, assigning tasks according to modality strengths: letting the text modality efficiently address simple property tasks, while leveraging the visual modality to directly support local and structurally complex tasks. To this end, we propose GraphVista , a unified framework for graph understanding with VLMs. For the scalability challenges, GraphVista is inspired by the retrieval-augmented generation (RAG) paradigm Lewis et al. 2020 . It organizes graph information hierarchically according to structural importance and stores multi-granularity textual descriptions in a lightweight GraphRAG base. For a given graph understanding task, GraphVista retrieves only the relevant information before input, thereby compressing redundant context and preserving key reasoning elements. When necessary, it can also generate high-resolution visual subgraphs from the retrieved local information to support complex reasoning tasks. To enable effective coordination across modalities, GraphVista introduces a planning agent that parses graph understanding tasks and routes them to the appropriate modality. The textual modality branch, enhanced with the RAG techniques, efficiently handles simple property tasks by retrieving and reasoning over concise textual descriptions. In contrast, the visual modality branch employs the Visual Graph Thoughts Agent , which constructs a multimodal chain of thought called Visual Graph Thoughts over high-resolution visual subgraphs, enabling VLMs to perform stepwise visual reasoning grounded in explicit structural evidence for structurally complex tasks. The main contributions of this paper are listed as follows: \bullet We present the first systematic analysis of the scalability challenges and coordination gaps that hinder VLMs in graph understanding, providing both conceptual insights and empirical evidence. \bullet We propose GraphVista , the first unified framework for graph understanding with VLMs, designed around two objectives: improving scalability and enabling coordination across modalities. To this end, GraphVista integrates a set of innovative techniques, such as a hierarchical GraphRAG base, a planning agent for task routing, and a Multimodal Graph Thoughts agent for complex reasoning. \bullet We introduce Grena , a scalable benchmark that supports step-level training of Visual Graph Thoughts and evaluates graph understanding from simple property to complex local tasks. \bullet Extensive experiments across diverse graph understanding tasks demonstrate the effectiveness of GraphVista, highlighting its ability to scale to large graphs and to exploit the complementary strengths of textual and visual modalities. 2 Related Work 2.1 Text Modality-based Graph Understanding Most existing studies convert graphs into textual descriptions to enable VLM-based graph understanding, primarily focusing on benchmark construction and capability analysis. Recent benchmarks evaluate VLMs on basic graph element comprehension Wang et al. 2023 ; Guo et al. 2023 ; Wu et al. 2024 and graph-theoretic reasoning tasks Chen et al. 2024a ;