MIND: Effective Incorrect Assignment Detection through a Multi-Modal Structure-Enhanced Language Model
Yunhe Pang, Bo Chen, Fanjin Zhang, Yanghui Rao, Jie Tang · 2024
Read MIND: Effective Incorrect Assignment Detection through a Multi-Modal Structure-Enhanced Language Model by Yunhe Pang, Bo Chen, Fanjin Zhang, Yanghui Rao, Jie Tang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03930.txt.
Introduction
Yunhe Pang, Bo Chen, Fanjin Zhang, Yanghui Rao, and Jie Tang are named in the catalog for this paper from two thousand twenty-four. Its text adds Evgeny Kharlamov and names the method GuARD. It spots anomalies on text-rich graphs, such as papers filed under the wrong author.
Plain language models can drown in the text and miss the graph's shape, which is key to telling odd nodes from sound ones. GuARD pairs shape features from graph methods with fine text traits drawn by small language models. A staged tuning plan teaches it both streams at once. Tests on four data sets beat graph-only and language-only rivals. It also trains and runs far faster than long-context models on the large WhoIsWho set.
Catalog record
| CREATOR | Yunhe Pang, Bo Chen, Fanjin Zhang, Yanghui Rao, Jie Tang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03930
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MIND: Effective Incorrect Assignment Detection through a Multi-Modal Structure-Enhanced Language Model Yunhe Pang, Bo Chen, Fanjin Zhang, Yanghui Rao, Jie Tang Abstract . The rapid growth of academic publications has exacerbated the issue of author name ambiguity in online digital libraries. Despite advances in name disambiguation algorithms, cumulative errors continue to undermine the reliability of academic systems. It is estimated that over 10 % 10\% paper-author assignments are rectified when constructing the million-scale WhoIsWho benchmark. Existing endeavors to detect incorrect assignments are either semantic-based or graph-based approaches, which fall short of making full use of the rich text attributes of papers and implicit structural features defined via the co-occurrence of paper attributes. To this end, this paper introduces a structure-enhanced language model that combines key structural features from graph-based methods with fine-grained semantic features from rich paper attributes to detect incorrect assignments. The proposed model is trained with a highly effective multi-modal multi-turn instruction tuning framework, which incorporates task-guided instruction tuning, text-attribute modality, and structural modality. Experimental results demonstrate that our model outperforms previous approaches, achieving top performance on the leaderboard of KDD Cup 2024. Our code has been publicly available 1 1 1 https://github.com/THUDM/WhoIsWho/tree/main/mind . 1. Introduction The past decades have witnessed a substantial proliferation of research papers across all fields of science, with platforms like Google Scholar 2 2 2 https://scholar.google.com/ ( Gusenbauer 2019 ) , Semantic Scholar 3 3 3 https://www.semanticscholar.org/ ( Lo et al. 2020 ) , and AMiner 4 4 4 https://www.aminer.cn/ ( Tang et al. 2008 ; Zhang et al. 2019 ; Zhang et al. 2022 ) indexing over 300 million papers. As a result, despite the availability of advanced author name disambiguation algorithms ( Chen et al. 2020 ) , recent studies have highlighted significant error rates of paper-author assignments in digital libraries, with exceeding 10\% error rate when constructing the million-scale WhoIsWho benchmark ( Chen et al. 2023 ; Zhang et al. 2024a ) . These inevitable cumulative errors from imperfect assignment methods significantly impact the efficacy of subsequent assignments, even leading to distorting authors’ citation count and misallocation of credit. Therefore, an inherent self-correcting mechanism to automatically detect and rectify these errors is crucial for maintaining the reliability of academic systems. ((a)) Illustration of the challenges for incorrect assignment detection ((b)) Graph-based method ((c)) LLM-based method Figure 1 . Illustration of incorrect assignment detection task and the common methods. (a) shows the ambiguous name "Li Chen", in which "P4" is wrongly assigned to him. The linkage derived from attributes such as co-author adds extra complexity to the task. (b) and (c) illustrate two common IND methods, Graph-based and LLM-based, respectively. Table 1. Performance comparison of models using different categories of features: Top-3 on the Whoiswho-IND test leaderboard in KDD Cup 2024. Rank Model Man- ual Text- ual Struc- tural AUC (%) Test Time (h) 1 ( Zhang et al. 2024b ) GLM3 ( Zeng et al. 2022 ; GLM et al. 2024 ) - ✓ - 83.45 >10 2 ( Shen 2024 ) GCN ( Kipf and Welling 2016 ) ✓ ✓ ✓ 82.49 <0.1 LGBM ( Ke et al. 2017 ) ✓ ✓ - 3 ( Yan and AsirAsir 2024 ) GLM3 ( Zeng et al. 2022 ; GLM et al. 2024 ) - ✓ - 81.35 >10 GLM4 ( Zeng et al. 2022 ; GLM et al. 2024 ) - ✓ - Mistral ( Jiang et al. 2023 ) - ✓ - LGBM ( Ke et al. 2017 ) ✓ ✓ - - MIND - ✓ ✓ 83.51 1.3 In this paper, we investigate the INcorrect assignment Detection ( IND ) problem ( Chen et al. 2023 ) , which aims to detect incorrect assignments of authors within the paper due to the ambiguity of the author name, as shown in Figure 1(a) . Previous efforts ( Chen et al. 2022 ; Zhang et al. 2024a ) typically employ graph-based algorithms, illustrated in Figure 1(b) , which build paper similarity graphs by considering papers as nodes and establishing edges between papers through key co-occurrence relationships such as co-authorship and co-organization, and then perform node classification to detect outlier papers. However, these methods focus more on key structural features and implicitly incorporate semantic textual features via input node features, but lack characterizing the fine-grained semantic features embodied in rich attributes of papers. Recently, large language models (LLMs) ( Brown et al. 2020 ; Zhao et al. 2023 ) demonstrate their strengths across various natural language understanding and generation tasks, endowed by their intrinsic capability to capture more fine-grained correlations through the self-attention mechanism on the input text. Some researchers ( Zhang et al. 2024a ; Zhang et al. 2024b ; Yan and AsirAsir 2024 ) endorse leveraging LLMs as the backbone and designing appropriate instruction templates to detect incorrect assignments. An example template for addressing the IND task is shown in Figure 1(c) . However, LLM-based methods suffer from modeling long contexts efficiently and effectively, especially when an author has over 1\small{,}000 papers and each paper possesses rich attributes (e.g., title, venue, author list, etc.). Furthermore, LLMs fail to model structural features of the graph natively. Therefore, a unified model that captures long text attributes and graph structural patterns efficiently and effectively is in great demand. Present Work. Inspired by the aforementioned insights, we propose a Multi-modal structural-semantic language model for INcorrect assignment Detection ( MIND ) that combines the strengths of key structural feature extraction endowed by graph-based methods and the fine-grained semantic feature characterized by language models via effective multi-modal fusion for incorrect assignment detection. Practically, we adopt a multi-modal multi-turn instruction tuning framework to incorporate each modality step by step as follows: (1) Task-Guided Multi-Turn Instruction Tuning: In the first stage, we aim to align the language model backbone to tackle the IND task via an instruction-tuning regime. Specifically, we design a task-specific instruction template, as shown in Figure 2 , which takes the target author’s papers and the target papers as input and asks the LLMs to generate the label token supervised by ground truths. To foster context sharing and improve training/inference efficiency, we further design a multi-turn chat instruction template, which significantly improves the accuracy and efficiency of our model. (2) Semantic Embedding Module with Rich Attributes: Limited by the context length, LLMs can only take papers’ key features as input. To involve more informative paper attributes into our framework, we employ a semantic embedding module that extracts and summarizes rich paper attributes through a pre-trained language model, and then utilizes a text projector to obtain a special text token that serves as the summarized semantic feature of each paper. (3) Structural Embedding Module: To endow LLMs with the capacity to capture structural information, we employ a structural embedding module that extracts and summarizes structural features from the graph-based methods, and then utilizes a graph projector to obtain a special graph token that acts as the structural feature of each paper w.r.t. corresponding target author. Through these three successive stages of multi-modal instruction tuning, both structural features and rich semantic characteristics are dynamically and effectively fused to perform accurate and robust incorrect assignment detection. Extensive experimental results highlight the superiority of our proposed method on