A Graph Talks, But Who’s Listening? Rethinking Evaluations for Graph-Language Models
Soham Petkar, Hari Aakash K, Anirudh Vempati, Akshit Sinha, Ponnurangam Kumaraguru, Chirag Agarwal · 2025
Read A Graph Talks, But Who’s Listening? Rethinking Evaluations for Graph-Language Models by Soham Petkar, Hari Aakash K, Anirudh Vempati, Akshit Sinha, Ponnurangam Kumaraguru, Chirag Agarwal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20583.txt.
Introduction
Soham Petkar and co-authors wrote a 2025 paper on the benchmarks used to grade graph-language models, tools that blend graph neural networks with large language models. Such models aim to join the structural reasoning of graph networks with the semantic understanding of language models. The benchmarks in wide use are mostly repurposed node classification data sets.
The team shows those benchmarks can be solved from one modality alone, with no graph-language integration at all. High scores on them therefore tell us little. The authors introduce a new benchmark built from synthetic graphs, paired with questions that require joint reasoning over structure and textual meaning, at varied complexity. In their evaluation, language models with soft prompts performed on par with models carrying a full graph network. Graph-language models also lost much ground when structural reasoning was genuinely required. The work sets a new standard for the field.
Catalog record
| CREATOR | Soham Petkar, Hari Aakash K, Anirudh Vempati, Akshit Sinha, Ponnurangam Kumaraguru, Chirag Agarwal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-20583
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Graph Talks, But Who’s Listening? Rethinking Evaluations for Graph-Language Models Soham Petkar, Hari Aakash K, Anirudh Vempati, Akshit Sinha, Ponnurangam Kumaraguru, Chirag Agarwal Abstract Developments in Graph-Language Models (GLMs) aim to integrate the structural reasoning capabilities of Graph Neural Networks (GNNs) with the semantic understanding of Large Language Models (LLMs). However, we demonstrate that current evaluation benchmarks for GLMs, which are primarily repurposed node-level classification datasets, are insufficient to assess multimodal reasoning. Our analysis reveals that strong performance on these benchmarks is achievable using unimodal information alone, suggesting that they do not necessitate graph-language integration. To address this evaluation gap, we introduce the CLeGR (Compositional Language-Graph Reasoning) benchmark, designed to evaluate multimodal reasoning at various complexity levels. Our benchmark employs a synthetic graph generation pipeline paired with questions that require joint reasoning over structure and textual semantics. We perform a thorough evaluation of representative GLM architectures and find that soft-prompted LLM baselines perform on par with GLMs that incorporate a full GNN backbone. This result calls into question the architectural necessity of incorporating graph structure into LLMs. We further show that GLMs exhibit significant performance degradation in tasks that require structural reasoning. These findings highlight limitations in the graph reasoning capabilities of current GLMs and provide a foundation for advancing the community toward explicit multimodal reasoning involving graph structure and language. † † footnotetext: Affiliations: 1 IIIT Hyderabad, India, 2 University of Virginia, USA, 3 Plaksha University, India CLEGR Dataset rethinking-graph-language-evals Figure 1: Current graph-language benchmarks are insufficient for evaluating multimodal reasoning. We find these benchmarks can be solved using a single modality: linear probing on the graph tokens alone matches GLMs on structurally-sufficient datasets, while a soft-prompted LLM using text alone achieves the same on semantically-sufficient datasets. This shows models are not required to integrate both graph and text to succeed. 1 Introduction The success of Vision-Language Models (VLMs) such as GPT-4V [ 31 ] and LLaVa [ 20 ] demonstrates the transformative potential of integrating different data modalities to improve complex reasoning capabilities through visual question answering, image captioning, and multimodal instruction following [ 5 ] . Inspired by this paradigm, recently there has been an increasing interest in using LLMs for graph-based applications [ 17 ] . Depending on the role of LLMs and their interaction with graph neural networks (GNNs), such techniques can be classified into treating LLMs as the final component for prediction (LLM as Predictor) [ 3 , 9 , 10 , 27 ] , treating LLMs as the feature extractor for GNNs (LLM as Encoder) [ 4 , 8 , 19 ] , or aligning the latent space of LLMs with GNNs (LLM as Aligner) [ 12 , 36 ] . Similarly, according to the relationship between graph and text presented in the application, the scenarios can be categorized into pure graphs, Text-Paired graphs (entire graphs paired with a descriptive text summary) [ 13 ] , and Text-Attributed Graphs (TAGs). TAGs are prevalent in real-world applications where nodes represent textual entities like documents or papers, and edges capture relationships between them [ 11 , 30 ] . The combination of textual attributes with the graph structure has significantly improved representation learning for various applications ranging from recommendation systems to social networks [ 37 ] , with recent work further demonstrating that integrating textual semantics is critical for generalization in foundational models [ 2 ] . To effectively exploit the rich semantic information within TAGs, the LLM-as-predictor framework has emerged as a particularly promising direction [ 24 ] . Notably, LLM-as-predictor approaches have demonstrated zero-shot transfer capabilities across datasets, allowing models trained on a single dataset to generalize effectively to unseen graphs with different features and graph structure [ 27 ] . In this work, we focus primarily on LLM-as-predictor approaches, as they represent a direct application of language models to graph question-answering tasks and align with our goal of assessing graph-language multimodal capabilities. Henceforth, we refer to this approach as Graph-Language Models (GLMs) throughout the rest of the manuscript. We first show that including node classification datasets in graph-language benchmarks does not provide a good proxy for tasks requiring a combination of structural and semantic knowledge. By training unimodal baselines which use either the graph structure, or the text attributes associated with the nodes, we achieve performance comparable to GLMs, questioning the utility of these datasets for assessing multimodal capabilities. To accurately assess these capabilities, we further introduce a new Graph Question-Answering benchmark, CLeGR , specifically designed so that both semantic and structural understanding are necessary to accurately answer the questions. Our contributions are summarized as follows: 1. We demonstrate that current GLMs can achieve strong performance on existing benchmarks by relying solely on graph or language modalities: On semantically-sufficient datasets, we show that using a graph encoder as a backbone does not provide any advantage over using only text attributes for classification. On structurally-sufficient datasets, we use linear probing [ 1 ] to show that graph-encoder representations are sufficient to achieve performance comparable to the full GLM setup, questioning the utility of these datasets in evaluating the multimodal interplay of graphs and language. 2. We introduce CLeGR (Compositional Graph-Language Reasoning), a synthetic benchmark explicitly constructed to assess multimodal graph-language reasoning. CLeGR spans over 1,000 diverse graphs and 54,000 questions designed at multiple levels of structural reasoning and reasoning. We also show that while GLMs saturate performance on CLeGR ’s retrieval tasks, their performance drops significantly on tasks requiring graph reasoning, and is on par with soft-prompted LLMs. 2 Preliminaries 2.1 Graph-Language Models Formally, a graph-language model {\text{GLM}}=(M_{l},M_{g},{M_{P}}) comprises three main components: M_{l} is an LLM, M_{g} is the graph encoder, and {M_{P}} is the linear projector that aligns graph and text representations. Let {\mathcal{G}}=({\mathcal{V}},{\mathcal{E}},{F^{{\mathcal{V}}}},{F^{{\mathcal{E}}}}) be a Text Attributed Graph (TAG), where {\mathcal{V}} is the set of nodes, {\mathcal{E}} is the set of edges between nodes, {N}=|{\mathcal{V}}| is the number of nodes in the graph, {M}=|{\mathcal{E}}| is the number of edges, {F^{{\mathcal{V}}}}=\{f^{{\mathcal{V}}}_{1},f^{{\mathcal{V}}}_{2},\dots,f^{{\mathcal{V}}}_{{N}}\} is the set of textual features of each node, and {F^{{\mathcal{E}}}}=\{f^{{\mathcal{E}}}_{1},f^{{\mathcal{E}}}_{2},\dots,f^{{\mathcal{E}}}_{M}\} is the set of textual features of each edge. For node-level question-answering tasks, given a graph {\mathcal{G}} , a node {n}_{i}\in{\mathcal{V}} , its textual features {f}_{i}\in{F} , and a textual question {q} , the prediction made by the GLM is: \displaystyle\hat{y} \displaystyle=\text{Predict}_{\text{GLM}}({\mathcal{G}},{n}_{i},{f}_{i},{q}) (1) \displaystyle=M_{l}\left({M_{P}}(M_{g}({\mathcal{G}},{n}_{i})){\big\|}{W}({f}_{i}){\big\|}{W}({q}),\right) where {W}\in\mathbb{R}^{|{L}|\times{d}} is the word embedding matrix of the LLM ( {L} denotes the token vocabulary and {d} the LLM’s hidden dimension), and {\big\|} denotes concatenation of token sequences. Here, \hat{y},{q},{f}_{i}\in{L}^{*} where {L}^{*} represents sequences of tokens from the vocabulary. The ground truth {g} also belongs to {L}^{*} . For graph-level question-answering tasks, given a graph {\mathcal{G}} and a textual question {q} , the GLM makes a prediction as: \displaystyle\hat{y} \displaystyle=\text{Predict}_{\text{GLM}}({\mathcal{G}},{q}) (2) \displaystyle=M_{l}\left({M_{P}}(\text{Pool}(M_{g}({\mathcal{G}}))){\big\|}{W}({f}){\big\|}{W}({q})\right) where \text{Pool}(\cdot) aggregates node embeddings from M_{g}({\mathcal{G}},0),\ldots,M_{g}({\mathcal{G}},{N}-1) to produce a single graph-level representation, and {f} represents the concatenated textual features of the entire graph. 2.2 Soft Prompting To isolate the contribution of graph encoders in GLMs, we use soft prompting as a baseline. Soft Prompting [ 16 ] introduces learnable prompt tokens which are concatenated with the embeddings of the input given to the LLM. We note here that a GLM can be viewed as a soft-prompt where a graph encoder is