Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System
Kawin Mayilvaghanan, Siddhant Gupta, Ayush Kumar · 2026
Read Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System by Kawin Mayilvaghanan, Siddhant Gupta, Ayush Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14970.txt.
Description
Mayilvaghanan and colleagues test whether language models used to grade contact-center agents treat staff fairly. Such models now grade calls and coach agents at scale. Their web-scale training data raises worries that demographic and behavioral biases may skew workforce reviews. The team builds a counterfactual fairness test across 13 dimensions in three groups: identity, context, and behavioral style.
They score 18 models on 3000 real transcripts, tracking how often a verdict flips and how far coaching scores shift when only a protected trait changes. Flip rates run roughly from 5 to 13 percent, and larger, better-aligned models look fairer, though fairness does not track accuracy. Priming with past performance hurts most, and plain fairness prompts help only a little. The authors call for standard fairness audits before such tools judge workers.
Catalog record
| CREATOR | Kawin Mayilvaghanan, Siddhant Gupta, Ayush Kumar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-14970