GraphEval : A Lightweight Graph-Based LLM Framework for Idea Evaluation

Tao Feng, Yihang Sun, Jiaxuan You · 2025

Read GraphEval : A Lightweight Graph-Based LLM Framework for Idea Evaluation by Tao Feng, Yihang Sun, Jiaxuan You (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12600.txt.

Introduction

Tao Feng, Yihang Sun, and Jiaxuan You proposed GraphEval in 2025, a framework that uses language models to judge the quality of research ideas. Teachers and reviewers increasingly ask language models to evaluate ideas, but plain prompting is unstable, and the models often miss the tangled meaning inside a complex proposal. Fine-tuned small models have similar limits.

GraphEval first uses prompts to a small language model to break an idea into separate viewpoint statements, then links those viewpoints into a graph using relation extraction and text similarity scores. Scores can then spread across the graph, which steadies the final judgment. The authors present two lightweight methods built on this idea, including a training-free label propagation version, aimed at more robust and higher-quality evaluations.

Catalog record

CREATORTao Feng, Yihang Sun, Jiaxuan You
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12600

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraphEval : A Lightweight Graph-Based LLM Framework for Idea Evaluation Tao Feng, Yihang Sun, Jiaxuan You Abstract The powerful capabilities of Large Language Models (LLMs) have led to their growing use in evaluating human-generated content, particularly in evaluating research ideas within academic settings. Existing solutions primarily rely on prompt-based LLM methods or fine-tuned lightweight language models for idea evaluation. However, these methods are often unstable and struggle to comprehend the complex semantic information embedded in the ideas, impeding their ability to perform high-quality evaluations. To address the above challenges, we propose GraphEval , a lightweight graph-based LLM framework for idea evaluation. Our insight is that a complex idea can be broken down into comprehensible viewpoint-nodes using small prompted LLMs. These viewpoint-nodes can then be linked together through edges created from LLM-based relation extraction and/or BERT similarity scores. The created viewpoint-graph can be used to conveniently propagate scores across viewpoint-nodes to improve the robustness of the idea evaluations. In particular, we propose two lightweight graph-based methods for idea evaluation: (1) GraphEval-LP: a training-free label propagation algorithm that propagates quality labels from known viewpoint-nodes to unknown nodes; (2) GraphEval-GNN: a Graph Neural Network (GNN) that is trained to predict the quality labels given the observed graph with minimal computation resources. Moreover, to overcome LLM’s limitation in objectively assessing the novelty of ideas, we further propose a novelty detection model to GraphEval-GNN to enhance its capability in judging idea novelty. Experiments on two datasets show GraphEval improves F1 scores by at least 14% with low computation and API costs. Additionally, GraphEval can effectively detect plagiarized ideas. Our codes for GraphEval is released at https://github.com/ulab-uiuc/GraphEval . 1 Introduction With the advancement of LLMs, many tasks traditionally performed by humans, such as idea evaluations ( Liang et al., 2024 ; Lin et al., 2023a ) , label annotation ( Wang et al., 2024 ; Goel et al., 2023 ) , or providing feedback to intelligent systems ( Stamper et al., 2024 ; Mai et al., 2023 ) , are now handled by LLMs. Among these applications, the use of LLMs to substitute humans in idea evaluations ( Lu et al., 2024 ; Baek et al., 2024 ) carries substantial potential, where researchers can obtain much faster feedback, as well as considerable risks, where the preference and bias of LLMs could affect the development of a scientific domain. Concretely, it is well known that many reviews for paper submissions are now written with the help of LLMs, which is explicitly allowed by ICLR 2025 as well. Unfortunately, existing LLMs are often biased to be “nice and helpful” while being highly sensitive to the prompt, illustrated by Figure 1 . Therefore, this paper aims to highlight a pressing research question: how do we improve the fidelity of LLM-based idea evaluation? Most existing research attempts to address the problem of LLM-based idea evaluation by designing better prompt strategy ( Brown, 2020 ; Wei et al., 2022 ; Wang et al., 2022 ; Yao et al., 2024 ) , so that more background knowledge, feedback, or inductive bias can be incorporated to an LLM. For example, Research Agent ( Baek et al., 2024 ) evaluates the ideas based on its five criteria added in the prompt. AI Scientist ( Lu et al., 2024 ) introduces some prompt tricks like self-reflection ( Shinn et al., 2024 ) , providing few-shot examples ( Wei et al., 2022 ) , and response ensembling ( Wang et al., 2022 ) to enhance the idea evaluations. However, these prompt-based evaluation methods are still limited because: 1) they are highly sensitive to different prompts ( Errica et al., 2024 ; Zhang et al., 2024a ) and are prone to hallucinations ( Sansford et al., 2024 ; Yao et al., 2023a ) ; 2) they also require LLMs to possess advanced capabilities ( Santu et al., 2024 ; Liang et al., 2024 ) to fully understand and judge a complex research idea, which often requires PhD-level humans in the real-world; 3) they could overlook factual inaccuracies interspersed among the ideas. As illustrated in Figure 2 , existing LLM-based methods directly analyze the entire set of information, therefore easily missing the factual errors within the idea, leading to a biased evaluation. Figure 1: Current LLMs are highly sensitive to prompts and show biases in evaluations. This figure illustrates that even minor variations in the LLM’s prompts (Original Prompt, Positive Prompt, Negative Prompt) for the same idea can lead to drastic changes in the final LLM evaluation results. Moreover, the LLM tends to always give friendly evaluations like ’Accept’ and rarely gives negative evaluations such as ’Reject’. This observation demonstrates that the LLM evaluation is biased. Figure 2: GraphEval performs a better idea evaluation than the existing LLM-based method by focusing on both the global and local information of the idea. In this figure, the part highlighted in red in the idea contain factual errors. The existing LLM-based method shown on the far left focuses solely on the global information of the idea, which often leads to overlooking factual errors interspersed within the idea. In contrast, GraphEval decomposes the idea into viewpoints to obtain scores for each viewpoint, then employs Mean Pooling and Min Pooling to extract global and local information of the idea, respectively. Finally, GraphEval derives a fair and unbiased evaluation based on these two aspects of information. Many studies in human psychology ( Knauff & Wolf, 2010 ; Dijkstra et al., 2014 ) indicate that people often find it difficult to understand abstract ideas, which can lead to random cognition and decision-making. However, two methods can significantly enhance human understanding of abstract ideas: 1) by breaking down complex abstract ideas into simpler viewpoints, it becomes easier for humans to understand ( Cowell et al., 2019 ; Rips et al., 2012 ) ; 2) showing the connections between the complex ideas and other ideas can also improve human’s understanding on these complex ideas ( Huang, 2020 ; Hayes & Kraemer, 2017 ; Khatin-Zadeh & Farsani, 2022 ) . Inspired by the psychological findings above, we propose GraphEval , a lightweight graph-based LLM framework for idea evaluation, which breaks down complex ideas into simple and comprehensible viewpoints, and bridges different viewpoints into a viewpoint-graph. Specifically, we first deconstruct complex and difficult-to-understand ideas into simple viewpoints using a prompt-based approach with a (small) LLM. Then, we treat each viewpoint as a node and construct edges through LLM-based relation extraction and/or BERT similarity scores. Finally, we create a viewpoint-graph by joining the viewpoints and edges across different ideas. Based on this, we propose two lightweight graph-based methods for idea evaluations: 1) GraphEval-LP: It is a training-free framework based on graph label propagation algorithm. It operates by transferring quality labels from labeled nodes to unlabeled nodes through weighted edges, ultimately predicting the final evaluation of an idea based on the labels of the viewpoint-subgraph associated with it. 2) GraphEval-GNN: It is a deep learning framework based on Graph Neural Networks (GNN) that requires minimal training. It models idea evaluations as a graph prediction problem using GNNs, obtaining evaluation results by predicting the attributes or classifications of viewpoint-subgraphs. Moreover, in order to objectively assess the novelty of ideas, we have also

Read the complete text