AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning

Peilin Wu, Xinlu Zhang, Kun Wan, Wentian Zhao, Gang Wu, Xinya Du, Zhiyu Chen · 2026

Read AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning by Peilin Wu, Xinlu Zhang, Kun Wan, Wentian Zhao, Gang Wu, Xinya Du, Zhiyu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18592.txt.

Introduction

Peilin Wu and six co-authors present AMARIS, a 2026 memory-augmented rubric improvement system for rubric-based reinforcement learning. Rubrics usually update from local evidence like the current batch, discarding diagnostic information from training. AMARIS stores rollout analyses, step summaries, and rubric update records in persistent evaluation memory, then retrieves recent and relevant history to revise rubrics. Across science, medicine, instruction following, and creative writing it beats static, local-adaptive, and memory-ablated baselines, gaining 2.8 points on GPQA-Diamond and 2.2 on IFBench.

Catalog record

CREATORPeilin Wu, Xinlu Zhang, Kun Wan, Wentian Zhao, Gang Wu, Xinya Du, Zhiyu Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105427
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning Peilin Wu, Xinlu Zhang, Kun Wan, Wentian Zhao, Gang Wu, Xinya Du, Zhiyu Chen Abstract Rubric-based reward shaping provides interpretable and editable reward signals for fine-tuning LLMs via reinforcement learning (RL), but existing adaptive rubric methods typically update criteria from local evidence such as the current batch or instance-level comparisons. This local view discards diagnostic information produced during training, making it difficult to track recurring failures, evaluate previous rubric edits, or raise standards once earlier criteria become saturated. We introduce AMARIS, A M emory- A ugmented R ubric I mprovement S ystem that grounds rubric updates in longitudinal training evidence. AMARIS stores rollout analyses, step-level summaries, and rubric update records in a persistent evaluation memory, then retrieves recent and semantically relevant history to revise rubrics. We evaluate AMARIS across science, medicine, instruction following, and creative writing under both global and instance-specific rubric settings. AMARIS improves over static, local-adaptive, and memory-ablated baselines, such as +2.8 points on GPQA-Diamond and +2.2 points on IFBench over the strongest baselines, while analysis shows that memory reduces oscillatory rubric edits and supports a progression from early failure correction to later curriculum advancement. AMARIS runs asynchronously alongside the normal RL loop, reducing blocking latency relative to synchronous rubric updates. The source code of AMARIS is at https://github.com/qualidea1217/AMARIS . Figure 1: The AMARIS system operates asynchronously in parallel with the normal RL pipeline to continuously refine rubrics. 1 Introduction Rubric-based reward shaping has become a practical mechanism for training language models in domains where output quality is multi-dimensional. Rubric-based approaches construct reward signals by decomposing evaluation into explicit natural-language criteria with associated weights ( 9 ; 36 ) . These criteria can cover correctness, helpfulness, safety, style, and instruction following, making the reward function interpretable and editable in open-ended settings where a single ground-truth verifier is hard to define ( 37 ) . Rubrics also allow practitioners to inspect which behaviors are being rewarded and revise the reward specification when training exposes new failure modes, including shortcuts that raise reward without improving output quality ( 27 ; 5 ) . However, a fixed rubric set defined before training cannot anticipate how policy behavior changes during RL training, motivating adaptive rubric updates. Recent work improves rubric flexibility in several ways. Some methods generate criteria for each input ( 7 ) or let the policy model propose rubrics for its own outputs ( 25 ; 32 ) . Others update rubrics from pairwise comparisons between current and reference rollouts ( 22 ) , joint optimization with the judge ( 32 ) , or batch-level selection of criteria that best distinguish current rollouts ( 23 ) . These approaches show that rubrics can be adapted rather than fixed before training. However, most adaptive rubric updates rely on local signals such as the current instance or immediate comparisons between instances, while diagnostic evidence from earlier training steps is discarded after immediate use. As a result, each update has limited access to how policy behavior has changed over time or how previous rubric edits affected the reward signal. Reusing this evidence can make rubric adaptation more consistent across training steps and can support updates that reflect both current rollout quality and earlier evaluation history ( 10 ; 4 ) . We introduce AMARIS, A M emory- A ugmented R ubric I mprovement S ystem for adaptive rubric-based reinforcement learning. AMARIS stores rollout-level diagnostic reports, step-level summaries, and rubric update records in a persistent evaluation memory. At each RL update step, it retrieves both recent summaries and semantically relevant historical evidence, then updates the active rubric set through defensive correction, curriculum advancement, or maintenance. This design turns rubric adaptation from a local per-step heuristic into a longitudinal evaluation process grounded in accumulated training evidence. We evaluate AMARIS across science, medicine, instruction following, and creative writing under both global and per-instance rubric settings. AMARIS improves over static, local-adaptive, and memory-ablated baselines, including +2.8 points on GPQA-Diamond, +3.0 points on HealthBench, and +2.2 points on IFBench over the strongest non-AMARIS baselines. Ablations show that recent-step retrieval and semantic retrieval provide complementary gains, while latency analysis shows that asynchronous execution substantially reduces wall-clock delay relative to synchronous rubric updates. Analyses of rubric evolution and qualitative cases further show how memory supports defensive correction, curriculum advancement, and late-stage maintenance. Our contributions are as follows: • We identify persistent evaluation memory as a useful state variable for adaptive rubric-based reward shaping. • We propose AMARIS, a memory-augmented rubric improvement system that stores rollout analyses, step-level summaries, and rubric update records, and retrieves both recent and semantically matched history to guide rubric updates. • We evaluate AMARIS under global and per-instance rubric settings across four domains and analyze rubric evolution, stability, and pipeline latency, showing how memory supports defensive correction, curriculum advancement, and maintenance. 2 AMARIS AMARIS introduces a memory-augmented rubric improvement loop into rubric-based reinforcement learning. The procedure has three stages. First, AMARIS analyzes individual rollouts as described in Section 2.1 . Second, it summarizes rollout analyses into a step-level batch summary as described in Section 2.2 . Third, it improves rubrics by grounding every change in recent and semantically matched analyses retrieved from memory as described in Section 2.3 . Section 2.4 describes how this procedure can be executed asynchronously and in parallel with the normal RL pipeline. 2.1 Individual Rollout Analysis In a standard rubric-based RL paradigm, the policy model’s rollouts y\sim\pi_{\phi}(\cdot\mid x) , where \pi_{\phi} is the policy model, x is the input, and y is the output, are evaluated against a set of active rubrics. At each RL training step t , AMARIS maintains an active rubric set G_{t}=\{(g_{j},w_{j})\}_{j=1}^{n_{g,t}} where g_{j} is the natural language definition of each individual rubric, w_{j} is the scalar weight, and n_{g,t} is the number of active rubrics. This rubric set can be global, with one set and one evaluation memory shared across all inputs, or per-instance, with each unique input maintaining its own set and memory. We use G_{t} for both rubric scopes in the main text. In the per-instance setting, it denotes the active rubric set associated with the current input. The final scalar reward r_{t} is then generated as r_{t}(x,y)=S_{\theta}(x,y,G_{t}) , where S_{\theta} is the process of scoring using an LLM as a judge (prompt template in Figure 3 - 4 ). The scalar reward is the scoring signal used by RL. Since scalar rewards may hide model behaviors such as reward hacking and rubric limitations, AMARIS also performs a deeper diagnosis of each individual rollout about the model’s behavior and how the current rubrics shape it. The later rubric update stage uses batch summaries and retrieved memory to produce the next active rubric set. For each rollout, AMARIS receives the model’s input and output, the current rubric set, training metadata (the high-level goal, current step, total steps, model size, etc.), and supplemental context such as ground-truth answers, denoted by z , and produces a structured diagnostic report a=A_{\psi}(x,y,G_{t},z) , where A_{\psi} is the LLM for analysis (prompt template in Figures 5 - 8 ). Each analysis provides five types of information. (i) Reward hacking detection reports whether the rollout shows signs of reward hacking, such

Read the complete text