FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts DOI: XXXXXXX.XXXXXXX Conference: KDD workshop on Evaluation and Trustworthiness of Agentic and Generative AI Models ; August 4, 2025; Toronto, ON, Canada CCS: Computing methodologies Natural la
Hagyeong Shin, Binoy Robin Dalal, Iwona Bialynicka-Birula, Navjot Matharu, Ryan Muir, Xingwei Yang, Samuel W. K. Wong · 2025
Read FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts DOI: XXXXXXX.XXXXXXX Conference: KDD workshop on Evaluation and Trustworthiness of Agentic and Generative AI Models ; August 4, 2025; Toronto, ON, Canada CCS: Computing methodologies Natural la by Hagyeong Shin, Binoy Robin Dalal, Iwona Bialynicka-Birula, Navjot Matharu, Ryan Muir, Xingwei Yang, Samuel W. K. Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00889.txt.
Introduction
Hagyeong Shin and six co-authors build FECT, a way to check if claims made by smart tools from call center chats are true to the chat. Their paper is from two thousand twenty-five.
Such tools can state things with no base in the source, and in firms that error can hurt real choices. Worse, claims about mood or root cause have no set truth to check. The team first sets out a three-step plan whose name joins the acts of breaking a claim down, parting its strands, and standing back from it. On that base they build their check. The work aims to make machine-made claims from service chats fit for trust.
Catalog record
| CREATOR | Hagyeong Shin, Binoy Robin Dalal, Iwona Bialynicka-Birula, Navjot Matharu, Ryan Muir, Xingwei Yang, Samuel W. K. Wong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00889
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts DOI: XXXXXXX.XXXXXXX Conference: KDD workshop on Evaluation and Trustworthiness of Agentic and Generative AI Models ; August 4, 2025; Toronto, ON, Canada CCS: Computing methodologies Natural language generation CCS: Computing methodologies Model verification and validation Hagyeong Shin, Binoy Robin Dalal, Iwona Bialynicka-Birula, Navjot Matharu, Ryan Muir, Xingwei Yang, Samuel W. K. Wong Abstract . Large language models (LLMs) are known to hallucinate, producing natural language outputs that are not grounded in the input, reference materials, or real-world knowledge. In enterprise applications where AI features support business decisions, such hallucinations can be particularly detrimental. LLMs that analyze and summarize contact center conversations introduce a unique set of challenges for factuality evaluation, because ground-truth labels often do not exist for analytical interpretations about sentiments captured in the conversation and root causes of the business problems. To remedy this, we first introduce a 3D — Decompose, Decouple, Detach —paradigm in the human annotation guideline and the LLM-judges’ prompt to ground the factuality labels in linguistically-informed evaluation criteria. We then introduce FECT , a novel benchmark dataset for F actuality E valuation of Interpretive AI-Generated C laims in Contact Center Conversation T ranscripts, labeled under our 3D paradigm. Lastly, we report our findings from aligning LLM-judges on the 3D paradigm. Overall, our findings contribute a new approach for automatically evaluating the factuality of outputs generated by an AI system for analyzing contact center conversations. 1. Introduction Figure 1. Most factuality evaluation tasks require fact-checking evidence, as shown on the left ( Li et al. 2023 ) . In contrast, our evaluation task on the right requires comprehensive consideration of the conversation context which does not present explicit evidence. Claim 2 is not found in our benchmark, because it is presented solely for the purpose of illustrating a non-factual claim to contrast with Claim 1. For the benchmark, we generated only one claim per conversation which may or may not be considered factual by human evaluators. See Figure 6 for an example of a non-factual claim found in our benchmark dataset. It remains a significant challenge to evaluate the truthfulness of outputs generated by large language models (LLMs) and LLM-based AI agents ( Liu et al. 2023 ; Wei et al. 2025 ) . The most direct way of detecting LLMs’ hallucinations is to have humans manually evaluate (“judge”) every LLM-generated response; however, this is a very labor intensive process, which prohibits scaling to large datasets, or rapidly iterating on the AI system’s quality. A direction that has been explored is the development of automated systems that leverage LLMs themselves—an approach commonly referred to as “LLM-as-a-Judge” or “LLM-Judge” ( Bavaresco et al. 2024 ; Chiang and Lee 2023 ; Liu et al. 2023 ; Zheng et al. 2023 ; Kim et al. 2024 ) . This is a promising direction to take, yet we observed challenges with our domain-specific evaluation tasks, namely that some evaluation materials are inherently ambiguous in the factuality dimension, making it challenging to establish ground-truth factuality labels. Our factuality evaluation task originates from an enterprise AI feature developed to perform business analysis tasks—Cresta’s ai analyst (see the right panel in Figure 1 ). 1 1 1 All conversations presented as examples in this paper are synthetically generated to reproduce patterns of original conversations while adhering to Cresta’s data governance policy. Developed by the authors, AI Analyst leverages LLMs to respond to enterprise users’ research questions about their contact center conversations. The input to AI Analyst is a user-provided analysis task that seeks insightful information from these conversations; for example, Why did the customer choose the plan? asked about selected healthcare-enterprise conversations. Leveraging LLMs, AI Analyst analyzes sampled conversations and generates a single analysis report as output. This report contains LLM-generated claims , which are single-sentence summaries of the conversations referenced for the analysis task. For instance, a claim The customer chose the plan for specific dentist coverage can be the response to the analysis task aforementioned. In other words, enterprise users of our AI Analyst often request analysis tasks that require deep research, and our LLM-generated claims are often neither verbatim nor near-verbatim copies of the conversation; most claims are analytical interpretations made about the conversation. Thus, in the context of our AI Analyst, the factuality evaluation task is to confirm that the claim---an analytical interpretation of the conversation made in response to the analysis task---is grounded in the referenced conversation. 2 2 2 Our definition of factuality only takes into account the claim and the referenced conversation and does not include evaluating whether the claim adequately satisfies the analysis task. The latter is an orthogonal requirement outside the scope of this work. There has been substantial effort to detect hallucinations made by LLMs ( Li et al. 2023 ; Lin et al. 2021 ; Bao et al. 2024 ; Luo et al. 2024 ; Tang et al. 2024 ; Iqbal et al. 2024 ; Wang et al. 2020 ) . However, this body of work focuses on evaluating the truthfulness of LLM outputs by straightforward fact-checking illustrated in the left panel of Figure 1 . Reference materials often contain explicit evidence for LLMs to extract information from, and an evaluator’s task is to confirm whether the explicit evidence verifies the LLM output or not. In contrast, most contact center conversations do not contain explicit evidence to verify the LLM-generated claims. For instance, Claim 1 in Figure 1 The customer chose the plan for specific dentist coverage can be verified only when an evaluator considers the context of the customer’s message Let’s go ahead and make the change now regarding the VivaCare Plus Dental plan and the affirmative message Yes, there’s a specific dentist we want to use . In addition, the evaluator is required to judge whether the link between customer’s messages and the conversational context verifies the relation stated in Claim 1, that is, the customer chose the plan specifically for the specific dentist coverage. Thus, the factuality evaluation of such claims involves evaluating the factuality of subjective analytical interpretations made about the conversation, which cannot be done by straightforward fact-checking. This nature of our AI Analyst therefore introduces a challenge seldom presented in existing hallucination detection tasks. Figure 2. Overview of our methodology to develop a reliable LLM-judge for factuality evaluations. We start with Phase 1 where we align human annotators’ judgments with the guideline of the 3D paradigm (described in Section 2.2 ). We then identify tasks with ambiguous factuality to achieve a dataset with ground-truth labels. In Phase 2, we align LLM-judges with the prompt following the 3D paradigm. Another challenge in our factuality evaluation task is that our LLM-generated claims about analytical interpretations of the conversation often do not yield a ground truth label of factuality. Many evaluation tasks are inherently ambiguous ( Aroyo and Welty 2015 ; Gordon et al. 2022 ; Schaekermann et al. 2020 ; Swayamdipta et al. 2020 ; Zhang et al. 2023 ) ; our tasks exhibit similar ambiguity due to the analytical nature of the claims. To our knowledge, these challenges—factuality evaluations of analytical interpretations and ambiguities in ground truth factuality—have not been addressed directly in the