Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution
Xiaohan Zheng, Lanning Wei, Yong Li, Quanming Yao · 2025
Read Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution by Xiaohan Zheng, Lanning Wei, Yong Li, Quanming Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14529.txt.
Introduction
Xiaohan Zheng, Lanning Wei, Yong Li, and Quanming Yao present LLMNet, a demo system that designs graph neural networks automatically, in a 2025 paper. Graph neural networks drive decisions on network data, but setting them up takes expert tuning. Their agents use language models plus curated knowledge bases instead.
The agents query the knowledge bases about the task and the graph's structure. Retrieval-augmented generation then guides an evolution process that configures and refines the network step by step. Across twelve data sets and three graph learning tasks, the auto-designed models performed strongly. The demo suggests non-experts could get well-tuned graph models without hand-crafting them.
Catalog record
| CREATOR | Xiaohan Zheng, Lanning Wei, Yong Li, Quanming Yao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-14529
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution Xiaohan Zheng, Lanning Wei, Yong Li, Quanming Yao Abstract Effective decision-making on networks often relies on learning from graph-structured data, where Graph Neural Networks (GNNs) play a central role, but they take efforts to configure and tune. In this demo, we propose LLMNet, showing how to design GNN automated through Large Language Models. Our system develops a set of agents that construct graph-related knowlege bases and then leverages Retrieval-Augmented Generation (RAG) to support automated configuration and refinement of GNN models through a knowledge-guided evolution process. These agents, equipped with specialized knowledge bases, extract insights into tasks and graph structures by interacting with the knowledge bases. Empirical results show LLMNet excels in twelve datasets across three graph learning tasks, validating its effectiveness of GNN model designing. 1 Introduction and Related Work Effective decision-making in networks—such as in communication networks, social networks, and transportation networks— often relies on graph-structured data representations. Among the techniques developed for learning from such data, Graph Neural Networks (GNNs) have become widely adopted across diverse domains, including tasks such as anomaly detection and recommendation systems in social networks Hamilton et al. 2017 , as well as for predicting biomedical molecular properties Gilmer et al. 2017 . The majority of existing GNNs are designed for diverse graphs under a specific task Wu et al. 2020 , such as capturing graph-level representations Zhang et al. 2018 ; Ying et al. 2018 . and learning subgraph patterns in link-level tasks He et al. 2020 ; Zhang and Chen 2018 . However, designing effective GNNs for different graph learning problems is challenging, as it requires substantial graph-related knowledge in order to understand the tasks and graphs Hoffman et al. 1995 . Then, there is a natural question: How to integrate graph learning knowledge to design effective GNNs? It is non-trivial to answer this question. Firstly, existing methods have not provided explicit guidelines for utilizing knowledge in designing GNN model architectures. Most GNNs are designed to effectively model graphs for a specific task Wu et al. 2020 ; Hamilton et al. 2017 ; Ying et al. 2018 , based on implicit human expertise, which is difficult to explicitly describe and extract. Therefore, we propose LLMNet, which automates GNN design using LLMs. Specifically, we have designed a Knowledge Agent to extract graph-related knowledge, building knowledge bases covering advanced graph learning research. Then, we have developed a set of agents that use RAG (Retrieval-Augmented Generation) to interact with knowledge bases, designing GNNs step by step in a knowledge-guided manner. Leveraging LLMs’ task analysis, LLMNet streamlines the designing and refinement of GNN model architectures. Extensive experiments on twelve datasets across three tasks demonstrate LLMNet’s superior performance and efficiency, proving the effectiveness of integrating knowledge for automated GNN design. A concrete case demonstrating this process is presented in Section 4 . 2 Method We introduce LLMNet, which prepares and utilizes knowledge to design GNN model architectures for diverse graph learning tasks using LLM-based agents. Firstly, we gather graph-related resources and develop a knowledge agent for knowledge extraction and retrieval. Subsequently, the knowledge is then used by several LLM-based agents step by step to design effective GNN model architectures. Figure 1: System architecture of LLMNet. LLMNet is designed to automate the design of GNN model architectures through a knowledge-guided approach. (a) shows the system inputs: graph need to process, external resources for knowledge base construction. (b) shows how the Knowledge Agent builds the knowledge bases. (c) depicts the automated GNN design and finetuning process, where the constructed knowledge is utilized by a pipeline of LLM-based agents. (d) shows the intermediate response returned to the user and the update of the experimental knowledge base. (e) shows the final results: the designed GNN, performance, and resource usage. 2.1 Knowledge Bases Construction and Utilization Knowledge Bases Construction LLMs face challenges due to outdated knowledge and hallucinations. We address this by creating two knowledge bases, which is currently lacking for designing GNN model architectures. We collect resources and use the Knowledge Agent to manage them. The Knowledge Agent is tasked with acquiring and integrating specialized knowledge tailored to specific user requirements. This agent mainly manages two types of knowledge bases, as shown in Figure 1 : the prior knowledge base and the experiment knowledge base. The prior knowledge base is enriched with task-specific information extracted from sources such as the Open Graph Benchmark (OGB) leaderboards, the PyTorch Geometric (PyG) documentation, and the top-tier conference proceedings that are accessible on Arxiv, ensuring the agent remains at the cutting edge of technology and methodology. The experiment knowledge base archives detailed experimental outcomes such as the benchmark evaluation results, including models setups and their performance on specific datasets, thereby providing insights into their effectiveness and application contexts. The content of papers and reports often overlaps, with redundant background information and methods that can introduce noise and reduce the informativeness of retrieved knowledge. To address this, we employ a two-level knowledge extraction strategy, first, we start by summarizing inputs to obtain coarse-grained knowledge, then refine this into fine-grained details specific to graph learning tasks, such as architecture design and dataset usage. The code and the extended version with more details are available. 1 1 1 https://github.com/lgssstsp/LLMNet . Knowledge Utilization and Update To effectively utilize the constructed knowledge bases, we implement a goal-aware knowledge retrieval mechanism. Utilizing the RAG technique, we enhance the effectiveness of the designing GNN model architectures by retrieving relevant knowledge. The pre-trained model all-MiniLM-L6-v2 encodes both the extracted knowledge and the queries from other agents. We calculate the cosine similarity in the embedding space to identify the most relevant knowledge. To accommodate the varying goals and resource types in graph learning, we apply a post-ranking strategy. The top- k knowledge items from each resource type are initially retrieved and then re-ranked and selected by the knowledge agent based on the query’s context. This refined knowledge is integrated into the graph learning agent’s prompt, facilitating the design of GNN model. LLMNet also incorporates a dynamically knowledge update mechanism. After the evaluation of a GNN model, the experimental summary, including the task plan, designed GNNs, and results, is stored in memory. The planning agent then compiles a report, which is added to the knowledge base, ensuring that the system’s knowledge remains current and applicable for future pipeline runs. This continuous update process allows LLMNet to adapt and improve over time, enhancing its ability to design effective GNN models. 2.2 Knowledge-Guided GNNs Model Designing Cora Photo Actor Genius obgn-arxiv DD Proteins ogbg-molhiv Amazon-Sports (\downarrow) Avg. Rank LLMNet 87.10(0.36) 96.11(0.33) 40.93(0.35) 90.89(0.11) 72.70(0.54) 78.27(2.57) 75.44(0.93) 74.27(1.54) 0.9298(0.0071) 1 LLMNet (GL) 86.68(0.40) 95.50(0.21) 39.59(0.39) 90.33(0.15) 72.30(0.54) 77.69(2.24) 74.88(1.16) 73.37(1.23) 0.9622(0.0103) 2.5 GCN 85.68(0.61) 93.13(0.27) 33.98(0.76) 89.10(0.13) 71.74(0.29) 73.59(4.17) 74.84(3.07) 73.89(1.46) 1.0832(0.0077) 5.25 SAGE 86.18(0.35) 94.60(0.25) 39.28(0.18) 89.71(0.09) 71.49(0.27) 76.99(2.74) 73.87(2.42) 73.46(1.69) 0.9900(0.0125) 4.5 AutoML 86.57(0.32) 95.38(0.30) 40.39(0.03) 90.81(0.04) 72.42(0.37) 77.03(2.48) 74.58(2.61) 73.51(3.21) 0.9327(0.0006) 2.63 LLM-GNN 84.64(1.04) 93.73(0.38) 38.92(0.07) 89.31(0.17) 70.83(0.93) 75.12(3.44) 74.47(3.65) 72.93(0.90) 0.9670(0.0079) 5.13 Table 1: Performance comparisons of the proposed LLMNet and baselines on three tasks. We report the test accuracy and the standard deviation for node and graph classification tasks, and use the common Rooted Mean Square Error (RMSE) for the item ranking task. The