LLM-Based Insight Extraction for Contact Center Analytics and Cost-Efficient Deployment
Varsha Embar, Ritvik Shrivastava, Vinay Damodaran · 2025
Read LLM-Based Insight Extraction for Contact Center Analytics and Cost-Efficient Deployment by Varsha Embar, Ritvik Shrivastava, Vinay Damodaran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19090.txt.
Introduction
Varsha Embar, Ritvik Shrivastava, and a team of co-authors wrote a 2025 paper on a language model system that mines call center conversations for insight, built to run cheaply in production. The core output is the call driver, the reason each customer called. Call drivers feed topic modeling, call sorting, trend spotting, and FAQ drafting.
The paper lays out the full system design with cost as a first class concern. It compares commercial models, open weight models, and fine tuned ones on quality. It then details thrift strategies, such as routing easy calls to small models, and reports the real cost analysis from production deployment. The lesson for practitioners is how to keep insight flowing without runaway model bills.
Catalog record
| CREATOR | Varsha Embar, Ritvik Shrivastava, Vinay Damodaran |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19090
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-Based Insight Extraction for Contact Center Analytics and Cost-Efficient Deployment Varsha Embar, Ritvik Shrivastava, Vinay Damodaran Abstract Large Language Models have transformed the Contact Center industry, manifesting in enhanced self-service tools, streamlined administrative processes, and augmented agent productivity. This paper delineates our system that automates call driver generation, which serves as the foundation for tasks such as topic modeling, incoming call classification, trend detection, and FAQ generation, delivering actionable insights for contact center agents and administrators to consume. We present a cost-efficient LLM system design, with 1) a comprehensive evaluation of proprietary, open-weight, and fine-tuned models and 2) cost-efficient strategies, and 3) the corresponding cost analysis when deployed in production environments. 1 Introduction The Contact or Call Center (CC) industry has rapidly adopted Large Language Models (LLMs) to extract insights from call streams to efficiently resolve customers’ problems. The recent improvements in LLM quality, performance, and accessibility have facilitated their integration into Interactive Voice Response (IVR) systems and virtual agents or chatbots. This integration has two main objectives: 1) to boost containment rates 1 1 1 Containment rate refers to the fraction of calls handled successfully without human intervention , and 2) to shorten agent handle times by offering real-time assistance via suggested responses linked to relevant documents during calls. In both cases, recognizing topics that customers frequently call about is a critical prerequisite to optimize CC operations. Traditional topic modeling systems provide valuable insights but are limited by their dependence on transcript lexicons, often producing overlapping topics and multiple keywords from a single call. This ambiguity complicates interpretation and typically requires manual curation by CC administrators to meet business needs. To address these limitations, our approach shifts from comprehensive call topics to extracting a customer’s primary contact reason, referred to as a call driver , aiming to reduce ambiguity and improve interpretability in downstream analyses. We leverage the generalization capabilities of instruction-tuned LLMs as our modeling foundation to alleviate the challenges in data availability and human annotation at scale. While some vendors’ generic LLMs can generate quality call drivers via prompt engineering, exclusive reliance on third-party solutions presents several drawbacks: 1) escalated costs due to high call volumes, 2) privacy concerns associated with sending sensitive data externally, and 3) prohibitive expenses for provider managed fine-tuning when applications require extra customization. In this paper, we present an LLM system for a commercial offering that provides contact center insights with the goal to improve the productivity metrics. The key modules are as follows: • Call driver generation (§ 3 ): Extract concise call drivers as primary representations of transcripts, serving as input for all downstream applications. • Topic modeling (§ 4.1 ): Utilize call drivers to derive topic clusters with appropriate labels. • Trend detection (§ 4.3 ): Identify new and emerging topics by maintaining sub-topics derived from our topic models. • Frequently Asked Questions (FAQs) generation (§ 4.4 ): Use call drivers from frequent topics to identify FAQs. Finally, we demonstrate how efficient fine-tuning, input compression, and deployment strategies can reduce LLM production costs compared to employing proprietary vendors, offering a cost-effective solution for organizations seeking to improve their contact center operations. 2 Related Work The advent of transformer-based architectures ( Vaswani et al. 2017 ) led to significant improvements in the Contact Center (CC) industry. Popular CC applications include analyzing mentioned entities with customer sentiment ( Fu et al. 2022 ; Tahmid Rahman Laskar et al. 2022 ) , generating a post call summary ( Zou et al. 2021 ) , and building virtual agents that provide responses grounded in retrieved documents (RAG) ( Bonetta et al. 2021 ) . Instruction-tuned models have expanded these applications. Commercial offerings provide LLM-powered call analytics, features to assist agents during calls and enable data-driven decisions. 2 2 2 Google Cloud CCAI Insights 3 3 3 AWS CC Intelligence . Understanding the customers’ call reasons allows CC admins to equip agents with the necessary training resources to satisfy customers’ needs ( Khasanova et al. 2022 ) . Automatic call classification with predetermined classes has also been explored ( Tang et al. 2003 ) . When predetermined classes are unavailable, traditional topic modeling approaches are employed ( Papadia et al. 2022 ; Hendry et al. 2021 ) . However, a single call may be assigned with multiple labels, due to certain keyphrases. This ambiguity hinders downstream analyses from capturing the essence of the call. 3 Call Driver Generation A call driver is a concise 15-20 word description summarizing the primary reason for a customer’s call. Figure 1 provides an illustrative example. Our objective is to develop a production-ready model that extracts these main reasons from call transcripts, considering various factors including privacy, legal compliance, cost efficiency, and latency optimization. The resulting call drivers will be further consumed by downstream tasks detailed in § 4 . Figure 1: A sample call transcript with call driver generations from various models. 3.1 Datasets & Evaluation Test Set To evaluate the model-generated call drivers, we use two internal real-world CC datasets from shipping (2k calls) and IT help-desk (3k calls) domains. We transcribe these audio datasets using Azure 4 4 4 Azure Speech-to-Text to get diarized transcripts. These transcripts are then human annotated by one professional annotator per call to establish ground truth call drivers. Training Set To comply with our CC product’s privacy policy, we generate 750 synthetic transcripts through human role-playing, avoiding use of customer data for model training. Two annotators simulate calls - as caller and agent - based on a provided domain, problem, and potential solution. A separate set of annotators label the call drivers. Evaluation The shortcomings of standard metrics for evaluating model generated text with ground-truth references like ROUGE, BertScore etc., is well-researched Fabbri et al. 2021 ; Liu et al. 2022 . Previous work Khobragade et al. 2019 has shown that entailment of text correlates better with human judgement in detecting paraphrases. However, entailment models trained on Natural Language Inference (NLI) datasets demonstrate a preference for longer hypotheses ( Gururangan et al. 2018 ; Guo and Vosoughi 2023 ) . Inspired by ( Papineni et al. 2002 ; Dubois et al. 2024 ) , we introduce a length penalty as a remedy. We use nli-deberta-v3-base 5 5 5 SBERT-NLI as our entailment model denoted as \textrm{entails}(\cdot) . For the i -th pair of reference ( ref_{i} ) and hypothesis ( hyp_{i} ) of a dataset of size n , we classify the pair as positive (entailment or neutral) or negative (contradiction): \displaystyle\begin{split}l_{p}&=\min\left(1,\ \alpha\sqrt{\frac{\sum_{i}\mbox{len}(ref_{i})}{\sum_{i}\mbox{len}(hyp_{i})}}\right)\\ S_{\textrm{cd}}&=l_{p}\times\tfrac{1}{n}\sum_{i}\mbox{entails}(ref_{i},hyp_{i}),\end{split} (1) where l_{p} is the corpus length penalty, \alpha is a fixed scaling factor, and \textrm{len}(\cdot) computes the length of text. The corpus length penalty l_{p} is introduced for two reasons: 1) to encourage the generation to focus on a single, or fewer most important call reasons, to disambiguate the downstream tasks for CC insights in § 4 , and 2) to empirically calibrate the call driver score S_{\textrm{cd}} to the end-to-end score S_{\textrm{e2e}} in ( 2 ). This allows the iteration of call driver modeling to be loosely decoupled from end-to-end topic modeling evaluation, to accelerate the experimental time. The scaling factor \alpha is determined to be 1 during our end-to-end topic modeling experiments. 3.2 Zero-shot