Freeform Preference Learning for Robotic Manipulation

Marcel Torne, Anubha Mahajan, Abhijnya Bhat, Chelsea Finn · 2026

Read Freeform Preference Learning for Robotic Manipulation by Marcel Torne, Anubha Mahajan, Abhijnya Bhat, Chelsea Finn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-32027.txt.

Description

Torne and three colleagues let people teach robots along several quality axes at once, instead of collapsing judgment into a single better-or-worse label. Reward design is the bottleneck for long-horizon manipulation: sparse success labels give too little signal, while binary preferences merge competing notions such as speed, safety and carefulness into one ambiguous verdict.

Freeform Preference Learning has annotators name their own natural-language axes and give pairwise preferences along each one separately. Those annotations train a language-conditioned reward model that scores a trajectory against any chosen axis. A reward-conditioned policy then optimizes across the specified dimensions jointly. Across four real-world and two simulated tasks, performance improves by 38 percentage points over sparse-reward and binary-preference baselines, and users can steer behavior at test time.

Catalog record

CREATORMarcel Torne, Anubha Mahajan, Abhijnya Bhat, Chelsea Finn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.32027

Browse the full library · About the Tower of Babel