SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs
2026
Read SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05546.txt.
Introduction
Self-play made AI strong at math and code, where rules make rewards easy to check. Hyobin Park, Taeseop Kim, and Dong-Geol Choi extend the idea to scientific papers with SPARK. Papers hide their relationships across text, figures, and tables, so automatic questions and trustworthy rewards are hard to build.
SPARK first builds one knowledge graph from many documents. Paths through that graph generate relational reasoning questions, and stored facts supply checkable rewards. One small vision-language model alternates between proposing and solving questions. It beat flat-corpus self-play baselines, and the gap grew as questions needed more hops.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs Abstract Self-play reinforcement learning has shown strong performance in domains with formally verifiable structure, such as mathematics and coding, where both problem generation and reward computation can be grounded in explicit rules. Extending this paradigm to scientific literature is more challenging: the relationships among multimodal elements within and across documents are rarely made explicit in text, which makes automatic generation of relational reasoning questions difficult and weakens the reliability of reward signals. We propose SPARK ( S elf- P lay with A symmetric R eward from K nowledge Graphs), a framework that automatically constructs a unified knowledge graph (KG) from multi-document scientific literature and uses it as the structural basis for self-play. KG paths over multimodal nodes serve as a source for generating relational reasoning questions, and structured facts stored in the KG provide a basis for verifiable reward computation. A single small vision-language model (sVLM) alternates between Proposer and Solver roles under information asymmetry against a fixed KG, a design that we believe can be naturally extended toward online adaptation in future work. We evaluate SPARK on public benchmarks and a self-constructed cross-document multi-hop QA dataset. Results show that SPARK consistently outperforms flat-corpus-based self-play baselines, and the performance gap widens as hop count increases, suggesting that KG-structure grounding contributes to relational multi-hop reasoning beyond what unstructured corpus grounding can provide. Hyobin Park 1 Taeseop Kim 2 Dong-Geol Choi 2† 1 ANTLAB, South Korea 2 Hanbat National University, South Korea [email protected] [email protected] [email protected] † Corresponding author 1 Introduction Scientific progress depends on the accumulation and interconnection of knowledge across the literature. Researchers derive new insights not merely by reading individual papers, but by synthesizing the evolution of concepts, divergences in methodology, and comparisons of experimental results spanning multiple documents. Understanding scientific literature is therefore fundamentally a problem of relational reasoning. Within a single document, figures support specific claims, tables ground experimental measurements, and equations formalize methodology. Across multiple documents, the methodology of one paper connects to the experimental design of another, and results from different works are compared and synthesized. How to train a model to automatically understand this complex relational structure is the central question motivating this work. The Gap Between Self-Play RL and Unstructured Domains. Self-play reinforcement learning is a promising paradigm for improving models without external annotation. Systems such as DeepSeek-R1 ( 9 ) and OpenAI o1 ( 13 ) have achieved expert-level performance on mathematics and coding tasks through self-play-based reinforcement learning. The success of this paradigm rests on two conditions. First, formal structure (symbolic equation systems, compilers) enables automatic generation of high-quality problems. Second, that same structure guarantees reliable reward computation that can determine correctness without human involvement. SPICE ( 17 ) attempted to extend this paradigm to document corpora, but treating documents as flat text provides no means to represent relationships among multimodal elements or semantic connections across documents, structurally limiting the satisfaction of both conditions. The relational structure of scientific literature is largely destroyed the moment it is serialized into a linear text sequence. Which claim a figure supports, or how the methodologies of two papers relate, is nowhere explicitly stated in the text. Flat-corpus self-play therefore encounters two fundamental bottlenecks. From the perspective of problem generation, the space of generatable questions is confined to surface-level fact extraction, making automatic generation of relational reasoning questions structurally infeasible. From the perspective of reward computation, when ground-truth answers are scattered across unstructured text, the relational structure grounding those answers remains implicit, making it difficult for any reward scheme to reliably distinguish structurally sound reasoning from plausible-sounding incorrect answers. A Structural Solution via Knowledge Graphs. We argue that knowledge graphs (KGs) provide a principled structural solution to both bottlenecks simultaneously. A KG encodes the multimodal components of a document and the relationships among them as explicit node-edge structures. On this structure, KG paths serve as a source for automatically generating relational reasoning questions (resolving bottleneck 1), and facts stored in the KG provide the basis for verifiable reward computation (resolving bottleneck 2). Furthermore, connecting concept nodes across multiple documents creates cross-document reasoning paths that extend beyond single-document scope, directly incorporating the complex cross-document knowledge synthesis that scientific literature understanding fundamentally requires into the self-play training objective. SPARK. Building on these insights, we propose SPARK . SPARK automatically constructs a three-stage KG from scientific papers via Structural, Reference, and Semantic Relation graphs, and internalizes relational reasoning capabilities through a self-play loop in which a single sVLM alternates between the Proposer role (generating questions conditioned on KG paths) and the Solver role (answering without KG access, under information asymmetry). The reward signal decomposes into three KG-verified components, R_{\text{answer}} , R_{\text{path}} , and R_{\text{consistency}} , directly supervising the structural faithfulness of reasoning paths beyond simple answer matching. Contributions. This paper makes three primary contributions. 1. A KG-Grounded Self-Play Framework. We propose SPARK, which leverages a fixed KG as an offline knowledge structure to simultaneously satisfy the two conditions for successful self-play by automatically generating high-quality problems and computing verifiable rewards in the scientific literature domain. Through the path faithfulness reward R_{\text{path}} , SPARK directly targets multi-hop relational reasoning that flat-corpus self-play cannot supervise. 2. A Multimodal Three-Stage KG Construction Pipeline. For a KG to function as a structural solution, automatically extracting relational structure from raw documents is itself a prerequisite. We present a three-stage pipeline (Structural-Reference-Semantic) that constructs a multimodal KG spanning text, figures, tables, and equations without manual annotation. Cross-Document KG Federation connects concept nodes across papers, enabling cross-document reasoning paths. 3. A Cross-Document Multi-Hop QA Benchmark. The relational multi-hop reasoning capability targeted by SPARK cannot be directly measured by existing benchmarks such as ScienceQA ( 20 ) or DocVQA ( 24 ) . We introduce a cross-document QA dataset of 450 questions derived from 50 arXiv papers. Figure 1 : Overview of SPARK. A three-stage pipeline automatically constructs a KG from scientific documents, and Cross-Document Federation connects concept nodes across papers. A single sVLM alternates between Proposer and Solver roles: the Proposer samples reasoning paths from the KG to generate relational questions with gold answers, while the Solver answers without KG access, under information asymmetry. The reward signal decomposes into three KG-verified components, R_{\text{answer}} , R_{\text{path}} , and R_{\text{consistency}} , where R_{\text{path}} and R_{\text{consistency}} are terms unique to SPARK that cannot be defined without the KG. 2 Related Work 2.1 Self-Play Reinforcement Learning and Language Models Alignment methodologies based on human and AI feedback ( 25 ; 2 ) have established the standard paradigm for language model post-training. Subsequently, self-play-based approaches ( 6 ; 33 ) , where a single model concurrently serves as both generator and discriminator, have been proposed. However, these methods suffer from an information symmetry bottleneck, where it is difficult to generate independent learning signals since both components share the same prior knowledge. While some studies ( 32 ) have formulated this from a game-theoretic perspective, they primarily focus on alignment rather than factual reasoning. On the other hand, reinforcement learning (RL) based on verifiable rewards ( 9 ) has proven effective in inducing long-chain reasoning. Efforts to extend this to zero-data self-play without external data ( 34 ; 11 ) are also underway. More