LingVarBench: Benchmarking LLM for Automated Named Entity Recognition in Structured Synthetic Spoken Transcriptions

Seyedali Mohammadi, Manas Paldhe, Amit Chhabra · 2025

Read LingVarBench: Benchmarking LLM for Automated Named Entity Recognition in Structured Synthetic Spoken Transcriptions by Seyedali Mohammadi, Manas Paldhe, Amit Chhabra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15801.txt.

Introduction

Seyedali Mohammadi and co-authors present a 2025 paper on pulling structured facts, like names and birth dates, out of messy phone call transcripts. Support and healthcare calls are full of hesitations, interruptions, and overlapping talk, and privacy rules make real recordings hard to share for training. Their benchmark, LingVarBench, instead manufactures synthetic transcripts with varied, realistic speech patterns, sampling entity values with a language model and filtering for consistency.

An automated prompt-optimization tool then builds extractors using only the synthetic data. On real customer transcripts, those extractors beat zero-shot baselines and matched or nearly matched prompts tuned by humans, scoring in the mid-nineties on firm facts. Gains on subjective survey items were large too. The authors offer the pipeline as a cheap, privacy-safe route to deploying call analytics.

Catalog record

CREATORSeyedali Mohammadi, Manas Paldhe, Amit Chhabra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15801

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LingVarBench: Benchmarking LLM for Automated Named Entity Recognition in Structured Synthetic Spoken Transcriptions Seyedali Mohammadi, Manas Paldhe, Amit Chhabra Abstract Phone call transcript labeling is prohibitively expensive ( $2 per minute) due to privacy regulations, consent requirements, and manual annotation costs requiring 3 hours of expert time per hour of audio. Existing extraction methods fail on conversational speech containing disfluencies, interruptions, and speaker overlap. We introduce LingVarBench , a synthetic data generation pipeline that addresses these constraints through automated validation. First, we prompt an LLM to generate realistic structured field values across multiple use cases. Second, we recursively prompt the model to transform these values into thousands of natural conversational utterances containing typical phone call characteristics. Third, we validate each synthetic utterance by testing whether a separate LLM-based extractor can recover the original structured information. We employ DSPy’s SIMBA optimizer to automatically synthesize extraction prompts from validated synthetic transcripts, eliminating manual prompt engineering. Our optimized prompts achieve up to 95% accuracy for numeric fields (vs. 88–89% zero-shot), 90% for names (vs. 47–79%), and over 80% for dates (vs. 72–77%) on real customer transcripts—demonstrating substantial gains over zero-shot prompting. The synthetic-to-real transfer demonstrates that conversational patterns learned from generated data generalize effectively to authentic phone calls containing background noise and domain-specific terminology. LingVarBench provides the first systematic benchmark for structured extraction from synthetic conversational data, demonstrating that automated prompt optimization overcomes cost and privacy barriers preventing large-scale phone call analysis in commercial settings. Introduction Voice-enabled artificial intelligence has emerged as a transformative force in healthcare, with the global AI voice agents in healthcare market reaching $468 million in 2024 and projected to grow at a compound annual growth rate of 37.79% through 2030 ( Research 2024 ) . Healthcare organizations are rapidly adopting voice AI for clinical documentation, patient interaction, and administrative automation, with 72% of patients expressing comfort using voice assistants for healthcare tasks such as appointment scheduling and prescription refills ( Augnito 2024 ) . Clinical documentation alone represents 17.54% of the healthcare voice AI market, as providers seek to streamline workflows and reduce administrative burdens that consume up to 30% of physicians’ time ( Research 2024 ) . At the heart of effective healthcare voice AI lies the challenge of structured information extraction from patient-provider conversations—accurately capturing critical medical details such as patient names, dates of birth, medication names, dosages, and medical record numbers from natural speech patterns ( LLP 2024 ) . Unlike traditional healthcare data entry systems that rely on structured forms and checkboxes, voice AI must process spontaneous patient responses where medical information is expressed with significant linguistic variability. For instance, a patient might describe their birth date as “March third, nineteen seventy-five,” “three three seventy-five,” or “zero three zero three one nine seven five,” each requiring sophisticated extraction capabilities while maintaining HIPAA compliance. Healthcare voice AI faces unique challenges stemming from the sensitive nature of protected health information (PHI) and the complexity of medical conversations. Patient-provider interactions contain medical terminology, emotional expressions, disfluencies during moments of distress, and variations in speech patterns due to age, illness, or medication effects ( Zhan et al. 2024 ) . The cost and complexity of obtaining labeled healthcare data is prohibitive due to HIPAA compliance requirements, Business Associate Agreements, and high potential penalties. Established NLP benchmarks such as CoNLL-2003 and clinical NER datasets fail to address the unique challenges of healthcare conversational speech extraction. These datasets, designed for written medical records or news text, do not capture the linguistic patterns of spontaneous patient-provider conversations, emotional contexts, or the specialized requirements of HIPAA-compliant voice interactions ( Tjong Kim Sang and De Meulder 2003 ) . Healthcare organizations report that 50% experience difficulties integrating voice AI into existing Electronic Health Record (EHR) systems, while maintaining compliance with HIPAA Privacy Rule and Security Rule requirements ( Augnito 2024 ) . Existing voice AI systems often struggle with medical vocabulary recognition, patient privacy protection, and the nuanced communication styles required in healthcare settings. Figure 1: Overview of the LingVarBench Synthetic Data Generation Framework: A three-step pipeline for generating and validating linguistically varied transcripts of structured fields. Canonical values (e.g., phone numbers) are transformed into diverse spoken-style utterances and validated to ensure accurate recovery. To address these fundamental limitations, we introduce a novel approach to semantic synthetic dataset generation —the first systematic framework for creating healthcare conversational data that captures both linguistic variability and semantic structure specific to medical field extraction. To the best of our knowledge, no prior work has developed taxonomic approaches to synthetic data generation that specifically model the semantic relationships between structured medical fields and their diverse spoken expressions in patient conversations. Our core insight leverages large language models to generate realistic patient-provider interactions while employing DSPy’s SIMBA optimizer (Stochastic Introspective Mini-Batch Ascent) ( Khattab et al. 2024 ) for automated prompt optimization, eliminating the need for expensive PHI-based training data. We present LingVarBench (Figure. 1 ), a semantic synthetic dataset generation framework that makes three key technical contributions: (1) A systematic healthcare conversation synthesis pipeline that generates HIPAA-compliant synthetic patient-provider dialogues with controlled linguistic variations and medical context; (2) Automated prompt optimization for healthcare extraction using DSPy and SIMBA to synthesize HIPAA-compliant extraction prompts without requiring access to real patient data as shown in Figure 2 . ; and (3) A healthcare-specific benchmark dataset covering critical medical fields with comprehensive evaluation of extraction robustness under clinical communication scenarios. Our empirical evaluation demonstrates significant improvements over existing approaches in healthcare contexts. Prompts optimized using LingVarBench-generated synthetic data achieve 90+% accuracy for field extraction on real healthcare transcripts, compared to 8̃3% using manually crafted prompts. Our framework successfully generates validated synthetic healthcare conversations covering patient demographics, medical histories, and prescription information while maintaining HIPAA compliance through synthetic data that eliminates PHI exposure risks. Cross-model evaluation across GPT 4, Gemini 2.5 Pro, and Gemini 2.0 Flash demonstrates consistent performance, with our taxonomy-driven approach achieving semantic similarity scores of 0.81±0.13 with authentic patient-provider conversations. LingVarBench addresses a critical bottleneck in healthcare AI adoption by providing the first systematic solution to the dual challenges of data scarcity and HIPAA compliance in medical voice AI development. Our semantic synthetic dataset generation framework enables healthcare organizations to develop robust extraction systems without accessing real patient data, thereby eliminating PHI exposure risks while maintaining clinical accuracy. This work establishes a foundation for HIPAA-compliant AI development methodologies, supporting the projected growth of healthcare voice AI applications from virtual nursing assistants to clinical documentation automation, ultimately improving patient care delivery while protecting sensitive information. Figure 2: Synthetic transcripts data (LingVarBench) are generated and validated to optimize prompts for robust entity extraction in real-world phone calls. Related Work Models trained exclusively on datasets with clean, high-quality transcriptions often fail to generalize effectively in real-world scenario where noise is the norm. Moreover, these datasets frequently lack coverage of domain-specific or emerging entities, limiting their long-term applicability. For example, Davidson et al. 2021 reported low F1 scores for entities such as email address (0.46) and named products (0.65). Kim and Kang 2022 and Eger, Youssef, and Gurevych 2020 show that even modern neural-based NER systems tend to memorize dataset-specific patterns rather than generalize. These models often break down across datasets due

Read the complete text