AURA : A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents

Takyoung Kim, Janvijay Singh *, Shuhaib Mehri *, Sagnik Mukherjee, Nimet Beyza Bozdag, Sumuk Shashidhar, Dilek Hakkani-Tür · 2025

Read AURA : A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents by Takyoung Kim, Janvijay Singh *, Shuhaib Mehri *, Sagnik Mukherjee, Nimet Beyza Bozdag, Sumuk Shashidhar, Dilek Hakkani-Tür (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01592.txt.

Description

Takyoung Kim and co-authors present AURA, a framework for judging how happy users are with planning agents, in a 2025 paper. AI planning agents chain many steps, from reading context to calling tools and replying. Most benchmarks score only whether the final task got done. The authors argue users judge the whole ride, not just the destination.

AURA breaks agent behavior into stages and scores each with small, clear criteria judged by language models. That detail lets builders spot which stage delights or annoys users. Tests show agents shine at different stages, and satisfaction tracks process as well as outcome. The paper also weighs multi-agent setups and the limits of fake users for testing. It aims to align agent design with what people actually value.

Catalog record

CREATORTakyoung Kim, Janvijay Singh *, Shuhaib Mehri *, Sagnik Mukherjee, Nimet Beyza Bozdag, Sumuk Shashidhar, Dilek Hakkani-Tür
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01592

Browse the full library · About the Tower of Babel