Agentic Artificial Intelligence for the Automated Generation of Accurate Summary Podcasts of Radiology Research Papers Authors: Jeong Hyun Lee, Kyowon Gu, Jaeseung Shin, Ji Hye Min, Jeong Ah Hwang, Ji Yoon Ban, Jihwan Choi, Taewon Han, Dong Young Jeong, Kyung A Kang, Haejung Kim, Harim Kim, Hyun Su Kim, Min Je Kim, Myoung Kyoung Kim, Sanghee Kim, Min Hwan Kwak, Ji Hyun Lee, Sang Yub Lee, Chang Hoon Oh, Sung Goo Park, Yun Hwa Roh, Yongsik Sim, Beomseok Sohn, Siyoun Sung, Min-Hyuk Yu, Seong Ho Park Published: 2026 ABSTRACT ObjectiveTo evaluate whether a custom agentic artificial intelligence (AI) pipeline can overcome the limitations of general-purpose large language model tools, when compared with a generic commercial tool (Google NotebookLM [NBLM]), for generating podcast-style summaries of radiology research articles.Materials and MethodsTwenty-two PDF-format original research articles published in the April 2025 issue of Radiology were processed using our Programmable, Phoneme-Aware PDF-to-Podcast Pipeline (P5) and NBLM to generate 44 audio episodes. P5 utilizes a multi-agent workflow for script generation, quality assurance, pronunciation enhancement, and audio synthesis. Four radiologists from a pool of 25 (7 generalists and 18 specialists) were randomly assigned to evaluate each blinded audio episode, yielding 176 total evaluations. The primary outcomes were the number of hallucinations (factual errors) per episode and the percentage of hallucination-free episodes. Secondary outcomes included the number of inappropriate statements, mispronunciations, and flow disruptions; the composite quality score (Quality Assessment of Educational Podcasts [QAEP]); the key results coverage score; and overall listener preference. Data were analyzed using generalized linear mixed models.ResultsThe P5 method produced significantly fewer hallucinations per episode compared with NBLM (mean, 0.32 vs. 0.93; P < 0.001) and a higher proportion of hallucination-free episodes (71.6% [63/88] vs. 56.8% [50/88]; P = 0.013), consistently across generalists and specialists. P5 demonstrated significantly fewer mispronunciations (mean, 0.11 vs. 1.62; P < 0.001) and flow disruptions (mean, 0.19 vs. 1.06; P < 0.001) per episode, as well as higher mean QAEP composite scores (4.62 vs. 4.28; P < 0.001), compared with NBLM, consistently across generalists and specialists. Raters preferred P5 over NBLM in 72.7% (64/88) of comparisons (P = 0.003).ConclusionOur custom agentic AI pipeline generated podcast-style summaries of radiology research articles with significantly higher quality and greater listener preference than the generic commercial tool. See the invited Editorial “Generative Artificial Intelligence for Empowering Non-Native English Speakers in Verbal Scientific Communication and Dissemination” at https://doi.org/10.3348/kjr.2026.0474. INTRODUCTION Podcasts have become an important vehicle for delivering professional medical information, offering busy clinicians an efficient and flexible means of engaging with current medical knowledge [12]. The audio-only format facilitates learning during commuting or exercise, making podcasts particularly attractive for continuing medical education. High-quality medical podcasts can deliver knowledge comparable to that of conventional reading formats, and studies have shown that many physician listeners modified their clinical practice based on podcast content [2]. Journal-affiliated podcasts, including the Radiology podcast series, play an important role in disseminating published research. Recent evidence has demonstrated that featuring articles on journal podcasts enhances their academic impact, with measurable increases in downloads, citations, and Altmetric Attention Scores [1]. However, producing high-quality podcasts remains time- and labor-intensive. With advances in large language models (LLMs), researchers have explored the use of artificial intelligence (AI) to automate podcast creation from academic articles [3]. Several journals—including the Bone & Joint Journal, the British Journal of Dermatology, the Journal of the American Medical Association, the Korean Journal of Radiology, and Radiology: Artificial Intelligence—have begun experimenting with LLMs for this purpose by directly offering AI-generated audio summaries or by using LLMs to create content summaries and multimedia transcripts [45678]. Among the tools available for this purpose, Google NotebookLM (NBLM)—an AI-powered research assistant that generates podcast-style audio summaries termed “Audio Overviews”—has gained particular attention since its public launch in 2024, garnering millions of monthly visits [9]. However, recent research suggests that such general-purpose tools may have significant limitations when applied to complex clinical research articles, such as those in the medical imaging field [3]. Tejani et al. [3] evaluated NBLM’s Audio Overview feature using 21 articles published in Radiology and found that 71% of the podcasts contained factual errors (i.e., hallucinations), and that the content was overly simplified for specialists. These findings suggest that general-purpose AI tools may not meet the accuracy standards required by medical professionals without further refinement. To address this limitation, we developed a custom pipeline designed to generate more accurate AI-produced podcast-style audio summaries from radiology research papers. This study evaluated whether our custom pipeline could produce higher-quality podcast-style AI-generated audio episodes than NBLM and whether these quality improvements would be consistently recognized by radiologists with different levels of expertise. MATERIALS AND METHODS Ethics and Copyright Protection This nonhuman-subject experimental study did not require institutional review board approval. The study used AI tools under an AI training opt-out policy, ensuring that no proprietary or copyrighted source materials were retained or used for model development. To maintain strict copyright and privacy standards, AI-generated audio files were distributed exclusively to designated evaluators through secure access points. All generated assets were permanently deleted upon completion of the study to prevent potential data leakage or unauthorized use. Permission to use Radiology articles in the AI tools in this study was granted by the Radiological Society of North America. Study Design This study employed a within-subject experimental design to compare our custom Programmable, Phoneme-Aware PDF-to-Podcast Pipeline (P5) with NBLM for generating podcast-style summaries of radiology research articles written in English. We selected NBLM as the benchmark because it was previously evaluated specifically for generating podcast-style audio summaries of radiology research articles [3], in addition to its prominence among similar software tools. The study code is available at https://github.com/sigjhl/p5. This study adheres to the Minimum Reporting Items for Clear Evaluation of Accuracy Reports of Large Language Models in Healthcare (MI-CLEAR-LLM) checklist (Supplementary Table 1) [10]. Agentic Podcast Generation Pipeline We developed an agentic pipeline to convert research papers from PDF format into podcast-style summary audio episodes using AI (Fig. 1). Our pipeline, P5, employs AI agents that perform specific functions at multiple stages, utilizing gemini-2.5-pro (Google) for script generation and quality assurance, o3-2025-04-16 (OpenAI) for pronunciation enhancement, and gemini-2.5-pro-preview-tts (Google) for audio synthesis. All models were accessed via application programming interfaces. Further technical details are provided in Supplement (Technical Implementation Details of the P5 Pipeline) The pipeline incorporates three key components designed to address common limitations of general-purpose AI tools. A quality assurance agent evaluates generated scripts through multi-step verification, including extraction and verification of factual claims against the source document, assessment of content appropriateness, and revision of scripts based on identified issues. A pronunciation enhancement agent analyzes scripts to identify medical terms that are likely to be mispronounced by text-to-speech systems and then inserts appropriate phonetic annotations. An audio generation agent converts phoneme-enhanced scripts into audio, performs quality assurance of the outputs, and automatically regenerates audio when issues are detected. Each PDF was uploaded to gemini-2.5-pro for persistent access, allowing the agents to reference the document’s text and visual elements throughout the generation workflow. Details regarding PDF document processing and content access during generation, the configurations used for script generation, and the roles of the three agents are provided in Supplement (Technical Implementation Details of the P5 Pipeline). Study Materials All original research articles published in the April 2025 issue of Radiology served as source materials; the complete list of articles is provided in Supplementary Table 2. Between July 12 and 18, 2025, a pair of AI-generated audio episodes was generated for each paper, one using P5 and the other using NBLM. Both systems received identical custom instructions emphasizing accurate coverage of key results, appropriate technical depth for radiologist audiences, and an engaging conversational style (Fig. 2). The same PDF files were uploaded to both systems without modification. All generations for both methods were completed successfully without manual intervention. Study Participants as Raters The evaluation was conducted in a randomized, blinded manner. Twenty-five radiologists, who were fluent in English, participated as raters, comprising 7 generalists and 18 specialists. Generalists included radiology residents and board-certified radiologists without fellowship or subspecialty training. Specialists included radiology fellows and attending radiologists with subspecialty expertise. We included a relatively large number of raters (25) rather than adopting the typical design in radiology research involving a small number of raters with similar characteristics. This approach was intended to further mitigate the inherently subjective nature of the evaluation outcomes, beyond the incorporation of a standardized pre-study rater training procedure (explained later), thereby enhancing the reliability of the evaluations. To minimize bias arising from prior experience or familiarity with audio recordings generated by either P5 or NBLM, we limited participation to individuals with less than one hour of experience with NBLM-generated audio. None of the participants had prior experience with audio recordings generated by P5. Study Procedures Each radiologist was assigned to evaluate AI-generated podcast-style summary audio episodes for a subset of papers (one to seven papers per radiologist), with random assignment except for subspecialty matching among specialists. Each paper and its associated summary audio episodes were assessed by four radiologists (two generalists and two specialists matched to the paper’s subspecialty), randomly selected from the 25 raters. This prespecified random rater allocation was chosen to maintain a manageable per-rater listening burden while allowing each episode to be evaluated from multiple independent perspectives drawn from a broader and more heterogeneous rater pool, thereby enabling a more balanced comparison between P5 and NBLM. For each assigned paper, raters were provided with two audio recordings in a single evaluation session—one generated by P5 and the other by NBLM. Raters were blinded to the specific AI tool used for each recording, and the presentation order of the P5 and NBLM versions was randomized for each paper and for each rater to minimize evaluation bias. To reflect typical real-world use, raters were permitted to consult the PDF before, during, or after listening and were allowed to pause or rewind the audio as needed. We adopted this evaluation scheme rather than one involving a washout period to control for carryover effects, such as in a crossover design [11], which is often used for comparisons of paired image sets. This decision was based on the fact that one outcome of this study directly assessed preference between the two recordings, thereby necessitating evaluation of both within the same session (as explained later). In addition, potential carryover effects were considered less influential in our study because the constant element was the underlying PDF article rather than the two generated audio summaries. This differs from paired image comparisons, in which the two images are essentially identical except for the specific technical modifications under evaluation. The most critical measure for bias control in our study was rater blinding to the AI tools, which was rigorously implemented. Raters evaluated episodes using standardized rating criteria for which they had received prior remote training. The training used materials not included in the study evaluation and consisted of distribution of a written rating guide developed by the lead author (J.H.L.), which defined each evaluation item with concrete examples (e.g., hallucination: reporting an area under the curve of “0.93” vs. “0.83” in the source paper), followed by a single round of rater questions and feedback with corresponding revision of the guide, and a recorded walkthrough reinforcing the evaluation criteria. For each audio episode, raters first provided free-text descriptions of identified issues and then submitted category counts (hallucinations, inappropriate statements, mispronunciations, and flow disruptions). A hallucination was recorded when an episode included a factual statement that contradicted the source PDF or asserted a specific fact not supported by it, including fabricated content or numerical inaccuracies. Inappropriate statements were defined as remarks unsuitable for a professional medical education context, and flow disruptions were defined as interruptions that materially degraded the listening experience. Raters also scored key results coverage (0 = none, 0.5 = partial, 1.0 = full). Raters then completed structured quality ratings using the Quality Assessment of Educational Podcasts (QAEP) instrument [212] with a radiology-adapted, modified QAEP scoring form (5-point scale; higher scores indicate better quality) [3]. The modified 20-item QAEP instrument assessed the quality of the generated audio episodes across multiple dimensions, including audio clarity, conversational naturalness, content accuracy, educational value, and appropriateness for radiologist audiences. Finally, raters indicated their overall preference between the two audio recordings for each paired comparison. Further details, including detailed definitions, counting rules, examples, and the full rating guide, are provided in Supplement (Rater Training and Evaluation Protocol). Outcome Measures The primary outcomes were the mean number of hallucination events per audio episode and the percentage of audio episodes that were free of hallucinations. Secondary outcomes included: 1) the number of inappropriate statements per audio episode (any remarks violating professional or sensitivity standards), 2) the number of mispronunciations per audio episode (medical or radiological terms only; repeated or multiple errors on the same word were counted once), 3) the number of flow disruptions per audio episode (irrelevant dialogue, third-party voice intrusion, or significant background noise), 4) the composite Likert score (mean of the 20 modified QAEP items), 5) the key results coverage score (mean completeness across key results items: 0 = no coverage, 0.5 = partial coverage, 1.0 = full coverage; papers without key results were excluded), and 6) overall preference between the two audio episodes. Additionally, individual QAEP items were analyzed separately to identify the specific quality dimensions driving the overall differences. Statistical Analysis All analyses were conducted using generalized linear mixed models (GLMMs) to account for the paired design and repeated evaluations by four readers per AI-generated audio episode. We fitted negative binomial (log link), Gaussian (identity link), and binomial (logit link) GLMMs for count, continuous, and binary outcomes, respectively. All models included fixed effects for AI method (P5 vs. NBLM), rater type (generalist vs. specialist), and listening order, with random effects for paper, rater, and the paper x rater interaction. An additional analysis tested the interaction between AI method and rater type using likelihood ratio tests. False discovery rate (FDR) correction using the Benjamini–Hochberg procedure was applied to account for increased type I error due to multiple outcome comparisons, separately for: 1) primary outcomes (two tests), 2) secondary outcomes (six tests), and 3) individual QAEP items (20 tests), with statistical significance defined as FDR-adjusted P < 0.05. Additionally, generation time and audio duration were compared between methods using a paired t-test. All analyses were conducted in R version 4.5.1 using lme4 (version 1.1-37), MASS (version 7.3-61), emmeans (version 1.11.2-8), and tidyverse (version 2.0.0), with maximum likelihood estimation. RESULTS A total of 22 articles were used as source materials for podcast generation. This process yielded 44 generated audio episodes. Because four evaluations were performed for each article (two by generalists and two by specialists), the total number of evaluations was 176, comprising 88 paired preference judgments between P5 and NBLM (Table 1). Primary Outcomes Rater type did not influence any primary outcomes (P > 0.05), and the results were consistent between generalist and specialist raters. Podcasts generated using the P5 method demonstrated a significantly lower mean number of hallucination events per episode than those generated using NBLM (0.32 vs. 0.93; P < 0.001), representing a 66% reduction (Table 2). The P5 method produced a significantly higher proportion of podcast audio episodes that were entirely free of hallucinations (71.6% [63/88] vs. 56.8% [50/88]; P = 0.013) (Table 2). Secondary Outcomes Rater type did not influence any secondary outcomes (P > 0.05) except for the number of inappropriate statements. A significant interaction was observed between rater type and AI method for the number of inappropriate statements (P = 0.034). Among generalist raters, the mean number of inappropriate statements was significantly lower for P5 (0.14 vs. 0.50; P = 0.005), whereas no significant difference was observed between P5 and NBLM among specialist raters (0.11 vs. 0.07; P = 0.483) (Table 2). For the remaining five secondary outcomes, results were consistent across generalists and specialists. The P5 method demonstrated significantly superior results compared with NBLM for all measures except key results coverage (Table 2). The P5 method showed a significantly lower mean number of mispronunciations (0.11 vs. 1.62; P < 0.001) and flow disruptions (0.19 vs. 1.06; P < 0.001) per audio episode than NBLM. The subjective quality assessment also favored P5. The mean QAEP composite score was significantly higher for P5 (4.62 vs. 4.28; P < 0.001). Raters demonstrated a strong overall preference for the P5 podcasts (72.7% [64/88] vs. 27.3% [24/88]; P = 0.003). Additional exploratory comparisons between P5 and NBLM across the 20 individual QAEP items revealed significantly higher scores for P5 in 13 of the 20 items (all P < 0.05) (Fig. 3). Generation Time The P5 method required significantly more computational time than NBLM (Table 3). The mean generation time for P5 was 11.4 minutes, compared with 3.3 minutes for NBLM (P < 0.001), despite producing audio episodes of similar duration (16.2 minutes for P5-generated podcasts vs. 17.7 minutes for NBLM-generated podcasts; P = 0.062). DISCUSSION This study demonstrated that a custom, agentic AI pipeline can produce podcast-style audio summaries of radiology research papers with significantly higher quality than those produced by a generic AI tool. Compared with NBLM, P5 significantly reduced the number of hallucinated statements by two-thirds (mean, 0.32 vs. 0.93 per episode), increased the proportion of error-free podcasts (71.6% vs. 56.8%), and achieved higher scores on the modified QAEP composite (mean, 4.62 vs. 4.28). Mispronunciations and flow disruptions were also significantly fewer, and listeners preferred the episodes generated by P5 in nearly three-quarters of the cases. These benefits were consistent across generalists and specialists. One goal of scientific journal podcasts is to enhance author engagement—often through interviews that capture behind-the-scenes insights and personal reflections—and to promote deeper academic discourse through expert commentary or panel discussions. Current AI systems cannot replicate this depth of engagement. Consequently, the podcast-style audio summaries generated in this study fall short of the depth and character of a genuine, human-driven journal podcast. Nevertheless, our findings suggest that this approach is feasible as an automated method for generating more accurate summaries that may serve as an effective tool for disseminating new scientific knowledge from research publications. The superior performance of P5 is likely attributable to its multi-agent architecture. A dedicated quality assurance agent cross-references every scripted claim with the original PDF, while a pronunciation enhancement agent supplies phonetic cues that prevent the text-to-speech engine from mispronouncing radiologic terminology. The audio generation agent, in addition to generating the audio file, screens for issues and determines whether the audio should be regenerated. By embedding these checks directly into the generation workflow, P5 addresses shortcomings previously reported for generic AI tools used for podcast-style audio generation. These additional quality control steps extended the generation time compared with NBLM (mean, 11.4 vs. 3.3 minutes). However, this technical cost does not diminish the advantages of the P5 approach. Unlike AI applications in time-sensitive use cases, such as detecting intracranial hemorrhage on head CT, academic journal podcast production is not time-critical. Instead, the accuracy and adequacy of the audio content are of primary importance. Tejani et al. [3] previously evaluated the default “Audio Overview” feature of NBLM for podcast-style audio generation using 21 articles published in Radiology. Our NBLM results differed numerically from theirs, particularly in the proportion of hallucination-free audio episodes (56.8% vs. 28.6%) and in audio duration (mean, 17.7 minutes vs. 9.1 minutes). These discrepancies may be attributed to 1) the custom prompt we supplied, which emphasizes accurate coverage of key results and an audience-appropriate level of detail, 2) the different set of articles used (April 2025 vs. September 2024 issues of Radiology), and 3) interim updates to the NBLM model between the two studies. While our pipeline substantially reduced quality issues, it did not eliminate them entirely—reflecting an inherent challenge in generative AI (Table 4). Minor discrepancies in fine details or nuanced interpretations may still occur occasionally. This finding underscores the importance of a human-in-the-loop approach in AI-driven journal podcast production, analogous to the necessity of human supervision in other areas of AI assistance in scientific publication [1314]. Furthermore, as suggested by the discrepant counts of inappropriate statements per audio episode between generalist and specialist raters, subjective aspects of audio summary quality may require further tailoring to meet the expectations of the primary target audience, for which human oversight may play a more critical role. In this context, the stepwise architecture of the P5 system, which facilitates human modification, represents a distinct advantage over general-purpose tools. Our study was not intended to establish the practical utility of the P5 system. Ultimately, such utility would be reflected in enhanced knowledge dissemination and its subsequent positive impact on radiology practice and research. However, these theoretically desirable outcomes are difficult to measure directly. Unlike studies assessing more immediate educational effects—such as the impact of different teaching approaches on students’ test performance—the influence of research publications on professional practice patterns and knowledge levels is indirect and subject to numerous confounding factors. Consequently, the practical utility of journal podcast–based dissemination is often inferred from surrogate indicators, such as article access, downloads, or citation metrics [1]. Collecting data for such assessments requires long-term collaboration with academic journals, which was beyond the scope of the present study but represents an important direction for future research. This study had several limitations. Although statistically significant differences between P5 and NBLM were detected across multiple outcomes, the dataset—empirically selected without a formal sample size calculation—remained small for individual subspecialty areas and lacked diversity. Therefore, the results should be interpreted with caution. In addition, all included articles were drawn from a single journal (Radiology). We selected Radiology because it is the top-ranked journal in the field of radiology and medical imaging, is known for publishing highly sophisticated research, and enables a more relevant comparison with an existing study [3]. Furthermore, our evaluation was limited to English-language content. Although English is the dominant language of scientific publishing [1516], generalizability to other languages remains uncertain. We did not perform a formal ablation study of the P5 pipeline components or compare P5 with a minimal single-agent Gemini 2.5 Pro baseline to isolate the contribution of the multi-agent architecture. Isolating the multi-agent contribution would be challenging because pronunciation enhancement required the o3 model, and the text-to-speech system itself contributed to audio quality issues independent of script quality. However, NBLM utilizes Gemini 2.5 Pro for Audio Overview generation according to Google’s documentation [17]. Whether it operates as a minimal single-agent wrapper or as its own agentic architecture, our comparison effectively benchmarks P5 against a Gemini 2.5 Pro–based system, with the observed differences attributable to P5’s design choices. Additionally, although raters were limited to those with less than one hour of NBLM experience, prior exposure to its characteristic voice may have introduced subtle recognition bias. Finally, as discussed above, direct clinical or educational utility was not assessed. In conclusion, a custom agentic AI pipeline can generate podcast-style audio summaries of radiology research papers with fewer factual errors and higher overall quality than a generic AI tool. These improvements, driven by specialized agents for quality assurance, pronunciation enhancement, and audio generation, were consistently recognized by listeners across different levels of expertise. This agent-based approach may provide a potential pathway toward more reliable, automated, and scalable production of audio summaries of scientific articles for medical professionals.