RephQA : Evaluating Readability of Large Language Models in Public Health Question Answering

Weikang Qiu †§ , Tinglin Huang † , Ryan Rullo † , Yuchen Kuang ‡ , Ali Maatouk †, S. Raquel Ramos † , Rex Ying † · 2025

Read RephQA : Evaluating Readability of Large Language Models in Public Health Question Answering by Weikang Qiu †§ , Tinglin Huang † , Ryan Rullo † , Yuchen Kuang ‡ , Ali Maatouk †, S. Raquel Ramos † , Rex Ying † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16360.txt.

Introduction

Health answers from AI are useless if lay readers cannot follow them, yet most model studies chase accuracy alone. Weikang Qiu and six co-authors build RephQA, a benchmark of expert-checked public health questions drawn from many sources and topics. It grades models on both reading ease and how much real information their answers carry.

Most of the twenty-five models tested missed reading standards, which exposes a gap between reasoning and clear communication. Four fix strategies were tried, and a token-tuned training method worked best. The benchmark pushes health agents toward answers the public can actually use.

Catalog record

CREATORWeikang Qiu †§ , Tinglin Huang † , Ryan Rullo † , Yuchen Kuang ‡ , Ali Maatouk †, S. Raquel Ramos † , Rex Ying †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-16360

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RephQA : Evaluating Readability of Large Language Models in Public Health Question Answering DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 Thanks: § Correspondence to [email protected] . Weikang Qiu †§ , Tinglin Huang † , Ryan Rullo † , Yuchen Kuang ‡ , Ali Maatouk †, S. Raquel Ramos † , Rex Ying † Abstract . Large Language Models (LLMs) hold promise in addressing complex medical problems. However, while most prior studies focus on improving accuracy and reasoning abilities, a significant bottleneck in developing effective healthcare agents lies in the readability of LLM-generated responses, specifically, their ability to answer public health problems clearly and simply to people without medical backgrounds. In this work, we introduce RephQA , a benchmark for evaluating the readability of LLMs in public health question answering (QA). It contains 533 expert-reviewed QA pairs from 27 sources across 13 topics, and includes a proxy multiple-choice task to assess informativeness, along with two readability metrics: Flesch-Kincaid grade level and professional score. Evaluation of 25 LLMs reveals that most fail to meet readability standards, highlighting a gap between reasoning and effective communication. To address this, we explore four readability-enhancing strategies—standard prompting, chain-of-thought prompting, Group Relative Policy Optimization (GRPO), and a token-adapted variant. Token-adapted GRPO achieves the best results, advancing the development of more practical and user-friendly public health agents. These results represent a step toward building more practical agents for public health. 1. Introduction Large language models (LLMs) have demonstrated remarkable reasoning capabilities in solving complex problems ( OpenAI et al. 2024a ; DeepSeek-AI et al. 2024 ; Team et al. 2024b ; Qiu et al. 2025 ; Zhao et al. 2023 ) . One of the most critical applications is healthcare, which demands strong domain-specific knowledge ( Singhal et al. 2023 ; Thirunavukarasu et al. 2023 ; Nazi and Peng 2024 ) . Previous studies have introduced benchmarks ( Yao et al. 2024 ; Jin et al. 2019 ; Jiang et al. 2024 ; Wang et al. 2024 ; Jin et al. 2019 ) targeting different levels of medical reasoning tasks to be performed by LLMs. Despite their strong potential in addressing complex medical challenges, the readability of LLM-generated responses has been largely overlooked. Users without medical training often struggle to understand responses filled with jargon, even when the information is accurate. In contrast, high-readability responses can deliver the same content in a more accessible form. While recent LLMs are increasingly optimized for advanced reasoning ( Shao et al. 2024 ) , it remains unclear whether this progress compromises readability, as more sophisticated reasoning may produce responses that are overly technical for lay audiences. Table 1. The comparison between RephQA and the existing healthcare-related benchmark datasets. Dataset Answer Format Sources of Questions Evaluation Metrics Target Audience MedQA ( Jin et al. 2021 ) Multiple Choice Medical Examination Accuracy Medical Professionals MedMCQA ( Pal et al. 2022 ) Multiple Choice Medical Examination Accuracy Medical Professionals MMLU ( Hendrycks et al. 2020 ) Multiple Choice Medical Examination Accuracy Academic PubMedQA ( Jin et al. 2019 ) Multiple Choice PubMed Abstract Accuracy Clinical Researchers LiveQA ( Abacha et al. 2017 ) Long Answer National Library of Medicine Human Evaluation Consumers MedicationQA ( Abacha et al. 2019 ) Long Answer MedlinePlus & DailyMed Human Evaluation Medicine Consumers JAMA & Medbullets ( Chen et al. 2025 ) Long Answer & Multiple Choice JAMA Challenge & USMLE NLG Metrics & Accuracy Medical Professionals RephQA Long Answer & Multiple Choice Public Health FAQs & Experts Readability & Accuracy Lay Users / Patients In this study, we introduce RephQA to conduct a comprehensive investigation into the RE adability of LLMs in P ublic H ealth Q uestion- A nswering (QA) tasks. A key strength of RephQA lies in its expert-grounded foundation: it comprises of high-quality test set with 533 QA pairs curated from 27 online public health resources, all reviewed by professionals in nursing, public health, and clinical communication. Different from prior related benchmarks (Table 1 ), RephQA focuses on frequently asked public health questions from lay users or patients, and introduces a novel proxy multiple-choice QA that effectively evaluates the accuracy of generated responses. Besides, we collect a large-scale corpus of 36,060 public health questions to serve as a post-training dataset, supporting the exploration of readability-enhancement strategies. We evaluate 25 LLMs on RephQA using two readability metrics (Flesch-Kincaid grade level ( Kincaid et al. 1975 ) and professional score) and one accuracy metric based on our proxy multiple-choice QA. Additionally, we benchmark four strategies for improving the readability of LLM outputs, including standard prompting ( Ribeiro et al. 2023 ; Malik et al. 2024 ) , chain-of-thought prompting ( Wei et al. 2022 ) , Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) , and our proposed token-adapted GRPO. Experimental results show that most LLMs fail to produce responses at a desirable readability level. Furthermore, post-training with readability-oriented rewards significantly improves readability while maintaining accuracy, where the token-adapted GRPO achieves the best performance. In summary, our contributions are listed as follows: • We introduce RephQA for evaluating the readability of LLM-generated responses in public health QA tasks aimed at lay users and patients, with two readability metrics and one accuracy metric evaluated by our proposed proxy multi-choice QA. • We conduct a comprehensive evaluation of 25 LLMs on RephQA , revealing that most models struggle to generate responses with an expected readability, despite achieving high accuracy. • We explore three existing strategies to improve LLM readability and propose a novel method, the token-adapted GRPO method, which achieves the best performance and provides new insights into readability-aware generation. 2. RephQA Benchmark 2.1. Dataset Curation Expert-Grounded Public Health QA We collected a diverse set of publicly available healthcare-related QA pairs from 27 authoritative online resources covering a broad range of public health topics, including general health, cardiovascular disease, diabetes, HIV, diet, exercise, alcohol, smoking, sleep, mental health, sexual health, and health equity. Sources were selected for their accessibility to lay audiences and their coverage of frequently asked questions by the general public. The constructed datasets are reviewed and refined by domain experts. Based on the question intent, we classify them into four categories: Suggestion , Fact , Definition , Rationale . Proxy Multiple Choice QA Prior studies have attempted to use LLMs as judges when rating the accuracy or relevance of a response relative to a reference answer ( Gu et al. 2024 ) . However, these approaches are not reliable ( Schroeder and Wood-Doughty 2025 ) . To bridge this gap, we further construct a multiple-choice QA based on each reference answer. Specifically, we design an agent workflow that proposes and iteratively refines the proposed QA pairs. This QA task could be an effective proxy for the accuracy evaluation of the original QA: if a model’s response closely matches the original reference answer, an LLM agent will also produce a correct answer for the proxy QA. Metrics We use the Flesch-Kincaid grade level ( Kincaid et al. 1975 ) as one of the readability scores. We also define professional score as the percentage of professional terms in the text. Accuracy is measured through the aforementioned proxy multiple-choice QA task. Post-Training Dataset For the post-training dataset, we curated

Read the complete text