Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence
Sumanth Balaji, Piyush Mishra, Aashraya Sachdeva, Suraj Agrawal · 2026
Read Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence by Sumanth Balaji, Piyush Mishra, Aashraya Sachdeva, Suraj Agrawal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00596.txt.
Description
Sumanth Balaji, Piyush Mishra, Aashraya Sachdeva, and Suraj Agrawal argue that customer support chatbots need a harder test than most benchmarks give them. Old phone-menu systems followed rigid scripts, while language model agents promise flexibility. The open question is whether an agent follows business policy through multi-step tasks, handles dependencies between steps, and stays on track when users behave unpredictably.
Their benchmark, JourneyBench, generates realistic support scenarios from graph representations of customer journeys. A new score, the User Journey Coverage Score, measures how completely an agent adheres to policy across a whole journey rather than on isolated turns. The work targets a gap between agents that merely complete tasks and agents a company can trust with real customers.
Catalog record
| CREATOR | Sumanth Balaji, Piyush Mishra, Aashraya Sachdeva, Suraj Agrawal |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00596