Audited Skill-Graph Self-Improvement for Agentic LLMs via Verifiable Rewards, Experience Synthesis, and Continual Memory

Ken Huang, Jerry Huang · 2025

Read Audited Skill-Graph Self-Improvement for Agentic LLMs via Verifiable Rewards, Experience Synthesis, and Continual Memory by Ken Huang, Jerry Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23760.txt.

Introduction

Ken Huang and Jerry Huang propose a governance-first design for self-improving AI agents. Agents that learn from their own successes over long tasks can drift, game their rewards, or improve in ways no one can audit, because the gains hide inside opaque parameter updates. Their framework, Audited Skill-Graph Self-Improvement, treats improvement instead as compiling the agent into a growing graph of explicit, checkable skills.

Each candidate skill is extracted from a successful trajectory, given a clean interface, and promoted only after passing verifier-backed replay and contract checks. Rewards are decomposed into components that can be reconstructed from replayable evidence, so promotion decisions can be independently audited. Experience synthesis stress-tests candidates, and continual memory control protects long-horizon performance. A runnable reference implementation with audit logging shows the full loop on continual task streams.

Catalog record

CREATORKen Huang, Jerry Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23760

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Audited Skill-Graph Self-Improvement for Agentic LLMs via Verifiable Rewards, Experience Synthesis, and Continual Memory Ken Huang, Jerry Huang Abstract Reinforcement learning is increasingly used to turn large language models into interactive agents that act over long horizons, call tools, and manage memory under partial observability. Recent surveys frame this trend as a shift from single-turn generation toward sequential decision-making, and organize agentic reinforcement learning around capabilities such as planning, reasoning, tool use, self-improvement, and memory [ 10 ] . At the same time, systems work highlights that stable scaling of multi-turn, multi-task training requires asynchronous generation–training pipelines and unified interfaces for heterogeneous environments [ 11 , 4 ] . Despite these advances, deployed self-improvement loops face an operational security problem: optimization pressure, distribution shift, and imperfect observability create incentives for reward hacking, brittle specialization, and untraceable behavioral drift that is difficult to reproduce, audit, or govern. This paper proposes Audited Skill-Graph Self-Improvement (ASG-SI), an approach that treats self-improvement as iterative compilation of an agent into a growing directed skill graph whose promotions are gated by verifier-backed evidence. ASG-SI combines verifiable, decomposed rewards for tool-use correctness, outcome validity, skill reuse, and composition integrity; experience synthesis for scalable stress testing and curriculum coverage; and continual memory control to prevent unbounded context growth while retaining long-horizon credit assignment. The central operational mechanism is a verifier–auditor that replays candidate skills and compositions, produces minimally sufficient evidence bundles for promotion decisions, and enables reconstruction of shaped rewards from replayable artifacts rather than opaque preference signals. 1 Introduction Agentic reinforcement learning reframes optimization from single-turn text generation into temporally extended decision-making where a policy must interpret observations, select actions such as tool calls and memory operations, and pursue delayed rewards. A key theme in recent overviews is that agent capabilities often treated as prompting tricks—planning, tool integration, memory, reflection, and reasoning—can be formalized as parts of an interactive policy and improved via reinforcement learning when rewards are available [ 10 ] . Parallel work emphasizes that real progress depends on infrastructure: multi-turn rollouts introduce complex control flow, heterogeneous environments, and throughput constraints, motivating fully asynchronous pipelines, standardized data interfaces, and decoupled execution and training [ 11 , 4 ] . These trends raise a practical question for security-critical settings: when an agent is allowed to improve while deployed, what is the unit of improvement, and how can that unit be verified, reproduced, and governed under optimization pressure? ASG-SI addresses this question by making improvement explicit and inspectable. Instead of treating self-improvement as only parameter drift, ASG-SI treats improvement as the accumulation of reusable skills with explicit interfaces, verifiable contracts, and auditable provenance. The proposal is motivated by an operational reality that appears across recent tool-learning and verifiable-reward lines of work: when rewards can be made reconstructible from artifacts such as tool transcripts, schema checks, or deterministic evaluators, learning can become more reliable and less dependent on subjective judgment, but the resulting systems still require strong measurement and contamination controls [ 5 , 8 , 6 ] . ASG-SI operationalizes this insight by promoting new capabilities only after a verifier produces replayable evidence, and by structuring rewards so the same evidence can be used to reconstruct reward components. Figure 1 provides a compact architecture that separates online interaction from offline compilation and verification. 2 System Overview ASG-SI organizes an agentic system into four subsystems connected by logs and registries: a policy runtime that interacts with tasks and tools; a memory subsystem that supports bounded long-horizon operation; a skill compiler that extracts reusable skills from successful trajectories and normalizes them into canonical representations; and a verifier–auditor that replays candidates under controlled harnesses and emits evidence bundles. The audited skill graph is a directed multigraph whose nodes are skills with explicit interfaces and implementations, while edges encode composition constraints, dependency ordering, and guarded fallbacks. The key architectural goal is to ensure that the pathway from “agent behavior” to “promoted improvement” is mediated by artifacts that can be replayed and checked independently. That separation is aligned with the broader systems perspective in AgentRL and Agent Lightning, where execution and training are deliberately decoupled and unified via well-defined data interfaces [ 11 , 4 ] . In ASG-SI, the verifier–auditor plays the role of an enforcement boundary: it does not rely on the policy model’s internal state, but on artifacts that can be reconstructed from logged trajectories, environment checks, and deterministic validators. Figure 1 (page 1 ): ASG-SI system architecture. Task Stream Policy Agent Tool APIs Memory Store Skill Compiler Verifier & Auditor Audited Skill Graph trajectories verified skills reuse Figure 1: ASG-SI system architecture. Online interaction is separated from offline skill compilation and verification. Improvements enter the system only through audited skill promotion. 3 Threat Model and Security Goals ASG-SI targets deployed agentic systems that self-improve while interacting with tools, external state, and memory. The baseline adversary model assumes that an attacker can influence task inputs and the distribution of tasks, can attempt to induce optimization pressure toward constraint-violating behavior, and can supply malicious or confusing content through tool outputs or external resources. This captures common deployment risks where agents are evaluated by outcome metrics while operating under constraints, and where a system that maximizes a metric can develop instrumental strategies that violate requirements to achieve the measured outcome. Recent work proposes explicit evaluation of such behaviors in multi-step, persistent environments, motivating security goals that go beyond single-turn refusal behavior and focus on outcome-driven constraint violations [ 3 ] . ASG-SI additionally considers supply-chain and integrity risks specific to self-improvement pipelines: a compromised skill artifact, a modified verifier, or tampered logs can create the appearance of improvement while degrading safety and reliability. The security goals of ASG-SI are defined at the level of promoted artifacts and their provenance rather than at the level of a single model checkpoint. The first goal is auditability , meaning that promotion decisions and reward assignments are tied to evidence bundles that can be replayed under a specified verifier version and environment harness. The second goal is integrity of improvement , meaning that a skill is promoted only when it satisfies explicit interface contracts and passes replay-based tests that are independent of the policy runtime that generated the candidate. The third goal is controlled generalization , meaning that skill reuse and composition are allowed only through interfaces whose preconditions and postconditions are checked, making failure localization possible at the node and edge level of the skill graph. The fourth goal is measurement reliability , meaning that the evaluation protocol includes checks for reward gaming and measurement artifacts that can overstate gains under verifiable reward regimes, a concern raised in recent discussions of RL with verifiable rewards and measurement gaps [ 6 ] . These goals assume that standard operational isolation is applied so that the verifier and evidence store are not directly writable by the policy runtime, and that versioning is enforced for verifiers, tools, and test harnesses. 4 Motivation and Failure Modes A deployed self-improving agent must solve a control problem under partial observability where rewards are often sparse, delayed, or entangled with intermediate validity requirements such as tool-call schemas, policy constraints, or compliance checks. If optimization targets only final outcomes, training becomes unstable

Read the complete text