What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents

Xiaozhe Li *, Tianyi Lyu *, Yang Li, Yichuan Ma, Peiji Li, Linyang Li †, Qipeng Guo †, Dahua Lin, Kai Chen · 2026

Read What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents by Xiaozhe Li *, Tianyi Lyu *, Yang Li, Yichuan Ma, Peiji Li, Linyang Li †, Qipeng Guo †, Dahua Lin, Kai Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19447.txt.

Description

Li and team study how to train multi-turn agents from sparse rewards, where one success-or-failure signal must spread across many actions. They test five feedback sources at two insertion granularities and propose SERL, which lets the task reward set update direction while environment feedback tunes placement and size, focusing on critical actions. On ALFWorld and WebShop it reaches 90.0 percent and 80.1 percent success, beating strong reinforcement and distillation baselines. Grounded feedback at meaningful points beats longer, richer context used blindly.

Catalog record

CREATORXiaozhe Li *, Tianyi Lyu *, Yang Li, Yichuan Ma, Peiji Li, Linyang Li †, Qipeng Guo †, Dahua Lin, Kai Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46324
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19447

Browse the full library · About the Tower of Babel