Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System

Kawin Mayilvaghanan, Siddhant Gupta, Ayush Kumar · 2026

Read Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System by Kawin Mayilvaghanan, Siddhant Gupta, Ayush Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14970.txt.

Description

Mayilvaghanan and colleagues test whether language models used to grade contact-center agents treat staff fairly. Such models now grade calls and coach agents at scale. Their web-scale training data raises worries that demographic and behavioral biases may skew workforce reviews. The team builds a counterfactual fairness test across 13 dimensions in three groups: identity, context, and behavioral style.

They score 18 models on 3000 real transcripts, tracking how often a verdict flips and how far coaching scores shift when only a protected trait changes. Flip rates run roughly from 5 to 13 percent, and larger, better-aligned models look fairer, though fairness does not track accuracy. Priming with past performance hurts most, and plain fairness prompts help only a little. The authors call for standard fairness audits before such tools judge workers.

Catalog record

CREATORKawin Mayilvaghanan, Siddhant Gupta, Ayush Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-14970

Browse the full library · About the Tower of Babel