STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
Yao Luan, Ni Mu, Yiqin Yang, Bo Xu, Qing-Shan Jia · 2025
Read STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning by Yao Luan, Ni Mu, Yiqin Yang, Bo Xu, Qing-Shan Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23802.txt.
Description
Learning from human preferences breaks down when the compared clips come from different stages of a task. Yao Luan and 4 co-authors name that flaw stage misalignment and propose STAIR to fix it. Comparing a navigation clip with a grasping clip teaches the reward model little. STAIR first learns stages from temporal distance, using contrastive learning to group nearby states, with no task knowledge given in advance.
Comparisons are then favored inside the same stage, and the stage map adapts as the policy changes. Theory and experiments back the diagnosis. On multi-stage tasks STAIR beat rivals, stayed competitive on single-stage ones, and its stages matched how humans carve up the tasks.
Catalog record
| CREATOR | Yao Luan, Ni Mu, Yiqin Yang, Bo Xu, Qing-Shan Jia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23802