Comparative Analysis of Large Language Models in Generating Telugu Responses for Maternal Health Queries

Anagani Bhanusree, Sai Divya Vissamsetty, K VenkataKrishna Rao, Rimjhim · 2026

Read Comparative Analysis of Large Language Models in Generating Telugu Responses for Maternal Health Queries by Anagani Bhanusree, Sai Divya Vissamsetty, K VenkataKrishna Rao, Rimjhim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18898.txt.

Introduction

Bhanusree and co-authors compare three chatbots on maternal health questions in Telugu and other languages. Low-resource languages remain little studied for pregnancy advice. A bilingual dataset feeds ChatGPT-4o, Gemini, and Perplexity. BERTScore gives one view of answer quality. Gynecologists also rate accuracy, fluency, relevance, coherence, and completeness by hand. Gemini leads for accurate, coherent Telugu pregnancy answers in their tests. Perplexity does well when prompts are written in Telugu. The authors stress that model choice and prompt language both shape answer quality.

Catalog record

CREATORAnagani Bhanusree, Sai Divya Vissamsetty, K VenkataKrishna Rao, Rimjhim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12567
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparative Analysis of Large Language Models in Generating Telugu Responses for Maternal Health Queries Anagani Bhanusree, Sai Divya Vissamsetty, K VenkataKrishna Rao, Rimjhim Abstract Large Language Models (LLMs) have been progressively exhibiting there capabilities in various areas of research. The performance of the LLMs in acute maternal healthcare area, predominantly in low resource languages like Telugu, Hindi, Tamil, Urdu etc are still unstudied. This study presents how ChatGPT-4o, GeminiAI, and Perplexity AI respond to pregnancy related questions asked in different languages. A bilingual dataset is used to obtain results by applying the semantic similarity metrics (BERT Score) and expert assessments from expertise gynecologists. Multiple parameters like accuracy, fluency, relevance, coherence and completeness are taken into consideration by the gynecologists to rate the responses generated by the LLMs. Gemini excels in other LLMs in terms of producing accurate and coherent pregnancy relevant responses in Telugu, while Perplexity demonstrated well when the prompts were in Telugu. ChatGPT’s performance can be improved. The results states that both selecting an LLM and prompting language plays a crucial role in retrieving the information. Altogether, we emphasize for the improvement of LLMs assistance in regional languages for healthcare purposes. 1 Introduction Advancements in LLMs have greatly expanded their applicability across a variety of languages and specialized fields. However, their capabilities differ significantly when it comes to low resource languages such as Telugu, Hindi, Tamil, Urdu etc. While some models excel in generating fluent and contextually suitable text, others demonstrate stronger abilities in providing precise, domain specific information. The rapid evolution of LLM architectures and the diversity of their training data have led to inconsistent performance across Indic languages. This inconsistency highlights the necessity for comprehensive evaluation methods that account for linguistic nuances. Furthermore, deploying LLMs in sensitive areas like maternal healthcare demands an emphasis on accuracy and trustworthiness. This research seeks to fill these gaps by systematically examining how well, different LLMs understand and generate Telugu language content related to maternal health, considering both bilingual and domain specific factors. Previous studies have explored the accuracy and linguistic capabilities of various LLMs for pregnancy related queries in Telugu and other regional languages. For example, Kishore et al. [ 2 ] found Gemini to have superior grammatical and idiomatic skills, while ChatGPT showed higher factual correctness. Vaidya et al. [ 7 ] highlighted the strong performance of several LLMs in widely spoken Indic languages, although challenges remain for low resource languages such as Telugu. Lima et al. [ 5 ] evaluated LLMs in providing maternal health information in resource-constrained environments, noting both their potential and limitations. Lee et al. [ 4 ] noted readability gaps between ChatGPT and Bard in labor epidural FAQs. Collectively, these works emphasize the need for rigorous evaluation, contextual adaptation, and cautious clinical integration of LLMs, especially in underserved regions where digital tools can greatly enhance healthcare access. Other researchers, including Chalamalasetti et al. [ 3 ] introduced MATA, a benchmark suite for Telugu that assesses various linguistic phenomena and highlights challenges in relying solely on automated metrics. Singh et al. [ 6 ] introduce IndicGenBench, a comprehensive multilingual benchmark designed to assess the text generation capabilities of large language models across diverse Indic languages, emphasizing linguistic diversity and cross-lingual generalization. The study provides critical insights into existing models’ limitations and promotes advancement in Indic NLP evaluation frameworks. Lastly, Aggarwal et al. [ 1 ] introduced IndicXNLI, a benchmark dataset for evaluating multilingual natural language inference across 11 Indian languages, enabling assessment of cross-lingual transfer in models like mBERT, XLM-R, IndicBERT, and MuRIL. 2 Data Collection Here, we formulated a collection of regular pregnancy related questions reflecting typical concerns raised by expectant and lactating mothers. To capture linguistic diversity, these questions were prepared in both English and Telugu. The topics covered, ranged from nutrition, symptom management to fetal development and antenatal care. Each question was presented twice to the LLMs, first in English and then in Telugu. The LLMs were instructed to provide answers in Telugu, aiming to mimic the responses a practicing gynecologist fluent in Telugu might offer. Additionally, a notable expert gynecologists generated reference answers in Telugu to serve as a benchmark for evaluation. This bilingual question and answer dataset forms the foundation for assessing the LLMs performance. 3 Methodology Our evaluation approach combines automated semantic analysis with detailed expert review. Initially, we measured the semantic similarity between the LLM generated answers and the expert responses using BERT Score, a metric that utilizes contextual embeddings to capture meaning beyond exact word matches. This allowed an objective comparison of how closely the LLMs outputs aligned with medically accurate information. Simultaneously, we involved ten practicing gynecologists fluent in Telugu to perform a qualitative evaluation. These experts assessed each answer based on five criteria: accuracy (correctness of medical facts), fluency (grammatical and natural use of Telugu), relevance (appropriateness and focus on the query), coherence (logical structure and flow), and completeness (coverage of the question’s aspects). Each criterion was rated on a Likert scale from 1 (lowest) to 10 (highest). By integrating both quantitative semantic scores and qualitative expert judgments, our methodology captures the multifaceted nature of LLMs performance. We also investigated how the language of the input prompt English versus Telugu affects the quality of Telugu responses. This comprehensive evaluation framework provides valuable insights into the practical utility of LLMs for healthcare communication in regional languages. 4 Results and Discussion Throughout the results and analysis, we use the following abbreviations for clarity: D represents the expert doctor’s reference responses. The abbreviations EP and TP denote responses generated by Perplexity when prompted in English and Telugu, respectively. Similarly, EG and TG correspond to Gemini’s responses for English and Telugu prompts. Lastly, EC and TC indicate ChatGPT’s answers to English and Telugu questions, respectively. These labels are consistently used to facilitate comparison across models and prompt languages. A) Semantic Similarity Analysis Using BERT Score: To objectively assess how closely the responses generated by the LLMs match to the expert gynecologists’ answers, we employed BERT Score, which evaluates semantic similarity using contextual embeddings. Table 1 , summarizes the average precision, recall, and F1 scores for each model’s outputs based on the language of the prompt. Among all configurations, Perplexity prompted in English (EP) achieved the highest F1 score of 0.704, indicating a strong semantic alignment with expert responses. This suggests that while the model’s training and architecture enable it to grasp medical context effectively, the choice of prompt language can subtly influence the outcome. While, Gemini’s responses, whether prompted in English (EG) or Telugu (TG), also demonstrated high semantic similarity, with scores closely trailing Perplexity’s best result. ChatGPT showed a slightly lower semantic overlap, particularly when prompted in English (EC), receiving the lowest F1 score of 0.684. This reflects a moderate but consistent gap in capturing all nuances of maternal healthcare and linguistic expression in Telugu. Table 1: BERT-Score Comparison between Gynecologist Responses and LLM Responses Comparison Precision (Mean) Recall (Mean) F1 Score (Mean) D vs EP 0.692 0.718 0.704 D vs TP 0.688 0.719 0.703 D vs EG 0.694 0.691 0.693 D vs TG 0.689 0.691 0.690 D vs EC 0.662 0.708 0.684 D vs TC 0.685 0.720 0.702 Interestingly, when prompted in Telugu (TC), ChatGPT’s semantic

Read the complete text