SymbolicThought : Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding
Runcong Zhao, Qinglin Zhu, Hainiu Xu, Bin Liang , Lin Gui, Yulan He · 2025
Read SymbolicThought : Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding by Runcong Zhao, Qinglin Zhu, Hainiu Xu, Bin Liang , Lin Gui, Yulan He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04189.txt.
Introduction
Runcong Zhao and co-authors present a human-in-the-loop system for mapping relationships between story characters, in a 2025 paper. Hand labelling is slow and sparse, while language models often invent facts or break logic. Their system pairs model extraction with symbolic rules.
The tool builds editable relationship graphs, then polishes them with seven kinds of logic checks. An interactive screen lets users validate and fix conflicts in real time. A released data set holds 160 relationships with their logic structures for training and testing. Trials show better accuracy and consistency at lower time cost than past routes. The authors aim at story analysis, explainable AI, and model evaluation.
Catalog record
| CREATOR | Runcong Zhao, Qinglin Zhu, Hainiu Xu, Bin Liang , Lin Gui, Yulan He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04189
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SymbolicThought : Integrating Language Models and Symbolic Reasoning for Consistent and Interpretable Human Relationship Understanding Runcong Zhao, Qinglin Zhu, Hainiu Xu, Bin Liang , Lin Gui, Yulan He Abstract Understanding character relationships is essential for interpreting complex narratives and conducting socially grounded AI research. However, manual annotation is time-consuming and low in coverage, while large language models (LLMs) often produce hallucinated or logically inconsistent outputs. We present SymbolicThought , a human-in-the-loop framework that combines LLM-based extraction with symbolic reasoning. The system constructs editable character relationship graphs, refines them using seven types of logical constraints, and enables real-time validation and conflict resolution through an interactive interface. To support logical supervision and explainable social analysis, we release a dataset of 160 interpersonal relationships with corresponding logical structures. Experiments show that SymbolicThought improves annotation accuracy and consistency while significantly reducing time cost, offering a practical tool for narrative understanding, explainable AI, and LLM evaluation. A screencast 1 1 1 https://youtu.be/GmtI4qXumqI and demo 2 2 2 http://113.45.152.150:8888/ are available online. * * footnotetext: Equal contribution. 1 Introduction Recent advancements in artificial intelligence, particularly the emergence of large language models (LLMs), have reignited discussions around the potential for artificial general intelligence. The increasingly human-like capabilities of AI are also drawing significant attention from social science researchers Xu et al. 2024 . One of the most critical functions in social science research is understanding the relationships between individuals. This understanding is foundational to various disciplines, including journalism, legal document analysis, historical studies, and literary interpretation. Many existing studies suggest that social relationships can be modeled as compact graphs, also known as small-world networks. These graphs are characterised by a high clustering coefficient and short path lengths. Consequently, such networks typically contain small but densely connected subgraphs (communities), where nodes within a community are highly likely to be connected. Connections between different communities are often facilitated by a few key nodes (individuals). However, constructing such graphs solely from plain text descriptions presents a significant challenge. This is because the dense local structure of social communities requires a method capable of capturing most intra-community relationships, while the accurate identification of key inter-community connections demands high precision in detecting critical relationships. Traditional methods rely on either crowdsourced annotations or automated approaches based on LLMs. Both methods exhibit substantial limitations. On one hand, human annotation is generally accurate but demands significant cognitive effort and time investment, which usually lead to low coverage in the annotation. In particular, essential cross-sentence and long-distance connections are frequently missed or inaccurately recorded ( Zhao et al. 2024 ; Sancheti and Rudinger 2025 ) . On the other hand, LLM-based methods provide scalability and efficiency but suffer from issues such as hallucinations, semantic inconsistency, and fragmented narrative representations ( Liu et al. 2023 ; Guo et al. 2023 ; West et al. 2023 ; Yuan et al. 2024 ) , which often fail to predict key relations precisely ( Wang et al. 2024 ) . To address these challenges, we propose SymbolicThought , a hybrid framework designed for extracting character relationship graphs from narrative texts, combining the scalability and reasoning ability of LLMs with the precision and interpretability provided by human annotators. SymbolicThought integrates an LLM-based extraction mechanism with interactive human-in-the-loop verification supported by a novel symbolic reasoning module with an editable social knowledge base. Crucially, SymbolicThought leverages an intuitive visual interface that highlights relevant textual evidence and systematically detects logical inconsistencies or conflicts, significantly enhancing annotators’ focus and efficiency. By automatically surfacing critical information and reducing annotators’ cognitive load in searching through extensive narratives, our system empowers humans to direct their attention toward complex judgments and nuanced decisions, which are tasks that current fully automated systems are still unable to handle. Through this structured, human-centered annotation pipeline, SymbolicThought provides a solution that is: • Interpretable , where the symbolic reasoning driven by the LLM is easily traceable, localisable, and understandable by users; • Extendable , where users can edit the rules underlying the symbolic reasoning; • Complete , where the automatic annotation framework is designed to generate all possible relation graphs for a given character list; This enables the construction of an accurate and comprehensive static character relationship graphs, laying a strong foundation for advanced narrative analysis and socially aware AI tools across the humanities and social sciences. 2 Architecture of SymbolicThought Given a narrative document \mathcal{D} , the goal of SymbolicThought is to construct a character relationship graph \mathcal{G}=(\mathcal{V},\mathcal{E}) , where \mathcal{V} is the set of unique, verified character nodes and \mathcal{E} is the set of semantically and logically consistent relationships between them. The pipeline consists of two main stages: (1) Character Extraction: An LLM-based module \mathcal{F}{\text{char}} proposes candidate entities from \mathcal{D} , which are then verified, grouped, or corrected by annotators to yield the finalised node set \mathcal{V} . (2) Relationship Extraction: A downstream extraction module \mathcal{F}{\text{ext}} predicts pairwise relationships conditioned on both \mathcal{D} and \mathcal{V} . These predictions are refined by a symbolic module \mathcal{F}_{\text{ref}} and further reviewed by annotators to produce the final edge set \mathcal{E} . Figure 1: The SymbolicThought annotation workflow. Users (1) upload a narrative document, (2) verify or modify automatically extracted character mentions, and (3) confirm, edit, or refine LLM-generated relationships with real-time symbolic consistency checks. The interface provides visual feedback, alerts for logical conflicts, and suggestions for inferable relations, supporting accurate and efficient human-in-the-loop annotation. 2.1 Character Extraction This step corresponds to the lower-left region of Figure 1 . The goal is to identify a complete and clean set of character entities \mathcal{V} from the input narrative document \mathcal{D} , which serves as the foundation for downstream relation extraction. Explorer: Node Candidate Generation. We run the character extractor \mathcal{F}_{\text{char}} on the corpus \mathcal{D} independently n_{c} times, each with temperature sampling to broaden coverage. Mentions that recur in at least \tau_{c} runs are retained, yielding a consensus set of candidate nodes: \mathcal{V}_{\text{raw}}=\left\{c\;\middle|\;\sum_{i=1}^{n_{c}}\mathbb{I}\!\bigl[c\in\mathcal{F}_{\text{char}}^{(i)}(\mathcal{D})\bigr]\geq\tau_{c}\right\}. This frequency filter maximises recall while suppressing spurious extractions, providing a high-coverage node list for subsequent verification. Verifier: Human Verification. \mathcal{V}_{\text{raw}} is displayed in a front-end interface. Annotators can accept or reject suggestions, highlight new spans, or add missing entries. Mentions shift from red (unconfirmed) to green (confirmed), providing instant feedback. To improve coherence, our interface supports two editing operations: (1) Alias Resolution: Many characters appear under multiple aliases (e.g., “Elizabeth”, “Her Majesty”, or “Queen Elizabeth II” ). Annotators can merge these into a single canonical entity via drag-and-drop operations. (2) Entity Disambiguation: In contrast, distinct characters may share the same surface name (e.g., “Alexandre Dumas” may refer to either the father or the son ). Annotators can split such ambiguous mentions into separate rows and assign more specific names, such as “Alexandre Dumas Sr.” and “Alexandre Dumas Jr.” . 2.2 Relation Extraction This stage corresponds to the right-hand pane of Figure 1 . Its goal is to transform the narrative’s candidate relations \mathcal{E} into a directed graph that captures social, familial, and other types of interactions among the characters, thereby providing a structured representation for subsequent analysis. Explorer: Edge Candidate Generation. With the character inventory fixed, we run the relation extractor \mathcal{F}_{\text{rel}} independently n_{e} times, each with temperature sampling. Analogous to node extraction, we keep only those triples (x,r,y) that occur in at