Peak-Then-Collapse and the Four Interface Channels of Knowledge-Graph Tool Use

Tianda Sun, Dimitar Kazakov · 2026

Read Peak-Then-Collapse and the Four Interface Channels of Knowledge-Graph Tool Use by Tianda Sun, Dimitar Kazakov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26037.txt.

Introduction

Sun and Kazakov test reinforcement learning with verifiable rewards for knowledge-graph tool use, a deliberately tiny API of four Freebase moves. Training Qwen2.5-7B with GRPO lifts tool-grounded answers from 3.8 percent to 9.6 percent over 250 steps, then crashes them to zero within one 50-step window. That peak-then-collapse shape repeats across four seeds. Across seven reward designs, denser or sharper proxy rewards merely shift the failure mode. The authors blame thin interface feedback: an empty result list carries no hint, unlike a Python traceback naming the bad line.

Catalog record

CREATORTianda Sun, Dimitar Kazakov
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64820
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26037

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Peak-Then-Collapse and the Four Interface Channels of Knowledge-Graph Tool Use Tianda Sun, Dimitar Kazakov Abstract We test the standard RLVR tool-use recipe — GRPO on Qwen2.5-7B-Instruct — on a deliberately minimal knowledge-graph tool API: four Freebase navigation verbs over Complex WebQuestions. Under a self-verifiable retrieval reward, the policy’s tool-grounded answer rate climbs from 3.8 % 3.8\% to 9.6 % 9.6\% over 250 steps, then collapses to 0 % 0\% within a single 50-step window — a peak-then-collapse pattern replicated across four seeds. Across seven reward designs, we find four recurring failure modes: adding denser or more targeted proxy rewards shifts the failure mode rather than eliminating it. We argue that a key difference from Python interpreters, web search, and JSON APIs is interface feedback: their failures often leak natural-language signal the model saw in pretraining. A Python traceback names the failing line; an empty Freebase result [] does not. Stripping away that surface exposes a degradation regime that same-family reward redesigns do not fix. A direct oracle ablation rules out relation selection: injecting gold relations at every retrieval call lifts exact-match accuracy by only + 0.20 +0.20 pp, and 95.4 % 95.4\% of retrieval-dependent errors are retrieval-composition failures rather than answer-extraction failures. As a mitigation, one-iteration self-distillation reaches 40.0 % 40.0\% EM at 7B and is capacity-invariant: doubling capacity to 14B improves EM by only 0.25 0.25 pp, and initialization barely matters — the ceiling appears interface-bound within the 7B–14B range tested. 1 Introduction Large language models increasingly act as agents : they call external tools — code interpreters, web search, function APIs — and are trained to use them by reinforcement learning with verifiable rewards (RLVR), where a checkable outcome (did the code run? did the answer match?) supplies the reward ( 11 ; 6 ; 8 ; 16 ) . The dominant recipe — dense process rewards, GRPO ( 18 ) , a 7B-scale policy — is treated as portable across tool families. We test that portability on a tool whose interface is, by construction, the opposite of the pretraining-aligned surfaces RLVR usually enjoys: a knowledge graph (KG). Our benchmark is Complex WebQuestions (CWQ) ( 23 ) , a standard multi-hop question-answering task over the Freebase ( 1 ) knowledge graph, which we re-cast from its usual gold-subgraph/SPARQL form into a deliberately minimal tool-use interface : the policy must reach answers by composing four navigation verbs — the outgoing or incoming relations of an entity, and the entities reachable from it along a relation — where every entity is an opaque machine ID ( m.0d3k14 ) and every miss returns an uninformative empty list. Python tool use leaks natural-language signal the model saw in pretraining — a traceback names the failing line — whereas this KG interface leaks almost none. We ask whether the standard RLVR-tool recipe still converges once that surface is removed. Across seven reward designs trained on Qwen2.5-7B-Instruct ( 24 ) + CWQ, all seven converge into four mechanistically distinct failure modes, none solved by the next reward design in the same family. The signature finding is a peak-then-collapse (§ 4.3 ): a self-verifiable retrieval reward — one that checks a returned entity appears verbatim in the answer, gold-free — climbs grounded retrieval (the correct-via-tool rate, CvT) from 3.8\% to 9.6\% over 250 training steps, then collapses to 0\% in a single 50-step window (Figure 1 , replicated across four seeds). Agentic-RL collapse phenomena have been documented ( 26 ; 8 ; 25 ) , but at the level of action-distribution entropy and reward variance. The specific mechanism we observe — a one-shot quote-and-stop attractor with mean tool calls per question (Tools/Q) dropping 3.0{\to}1.0 in a 50-step window — is a call-volume signature distinct from these precedents and from the standard RLHF reward-hacking framework ( 7 ; 14 ; 21 ) , which reports proxy-overoptimisation at the level of learned reward models. Stripping that pretraining-aligned surface exposes a degradation regime no reward redesign in the same family can fix. Figure 1: Peak-then-collapse under self-verifiable retrieval (R-selfV). Top: CvT (correct-via-tool rate; left, blue) climbs 3.77{\to}9.57\% (steps 0–250) then catastrophically collapses to 0 by step 300; EM (exact match; right, red) follows. Bottom: Tools/Q drops 3.0{\to}1.0 across the shaded 50-step window. self-distill (dashed) reaches 40.0\% EM / 5.81\% CvT in the same reward family. Mechanism: § 4.3 . We run a controlled comparison along a single axis: one model, one task, one algorithm (GRPO), one tool API (the 4-verb Freebase interface), seven reward designs realising orthogonal shape choices. To explain the convergent failures we formalise the tool-API \to policy-gradient pathway as four channels along which a tool API supplies (or fails to supply) RL signal — silent failure ( L_{\mathrm{sig}} ), symbolic schema ( L_{\mathrm{lang}} ), opaque compositional state ( L_{\mathrm{comp}} ), absent pretraining prior ( L_{\mathrm{prior}} ); the KG case is degraded along all four relative to natural-language tool surfaces (§ 5 ). We test the framework’s dominant-bottleneck attribution via a direct oracle ablation: gold-relation injection at the strongest in-family checkpoint lifts EM (exact match) by only +0.20 pp, localising the residual gap downstream of relation selection. Contributions. (C1) A controlled comparison of seven reward designs on a single model \times task \times algorithm \times tool API, organised into a four-mode failure taxonomy (§ 4.3 ). (C2) A signal-theoretic framework (four channels L_{\mathrm{sig/lang/comp/prior}} ) supported by a direct oracle ablation localising the EM gap to channels downstream of relation selection, plus a step-level error decomposition where 95.4\% of 1{,}958 retrieval-dependent errors localise to retrieval composition, not answer extraction (§ 5.3 ). (C3) A one-iteration ReST-EM-style self-distillation recipe (self-distill) reaching EM= 40.0\% at 7B; the 5K rule-based pre-SFT prefix is dispensable (raw-base init reaches the same \approx 40\% ) and 14B matches within 0.25 pp. Concurrent richer pipelines reach higher EM via interface enrichments along channels identified by the framework (§ 4.5 , § 2 ). (C4) Step-resolved (50-step cadence, full CWQ test set) peak-then-collapse on a KG-tool agent — a call-volume signature (Tools/Q 3.0{\to}1.0 ) novel relative to the action-distribution entropy and reward-variance diagnostics of prior agentic-RL collapse ( 26 ; 8 ; 25 ) (§ 4.3 ). We release all code, reward implementations, per-checkpoint full-test evaluations, and the seven-category trajectory classifier at https://anonymous.4open.science/r/KG_GRPO-D47D . 1 1 1 URL anonymised for review. 2 Related Work Process-reward RL for tools and knowledge graphs. Process-reward recipes for tool-using LLMs ( 8 ; 3 ; 4 ; 27 ; 16 ; 34 ) succeed on tool APIs with pretraining-aligned surfaces; we test whether the same recipe transfers to KG tools deliberately stripped of natural-language surface. KG-specific variants using KGs to define rewards (fs1 ( 33 ) ; HyperGraphPro ( 15 ) ; KG-Implicit-RM ( 9 ) ) differ from our setup, which uses the KG to evaluate the policy (under RoG’s ( 12 ) subgraph filter; strict EM motivated by 32 ). On the same Qwen2.5-7B-Instruct + CWQ benchmark, richer-pipeline concurrent systems reach higher numbers by enlarging the interface — GraphWalker ( 29 ) adds \sim 21 K-trajectory SFT + BM25 + LLM reranker ( 79.6\% EM); Graph-RFT ( 22 ) adds web fallback over a 40\% -incomplete KG ( 67.2\% Hits@1) — confirming that interface enrichment, not capacity,

Read the complete text